System and method for a robotic endoscope using integrated intralesional tool tomosynthesis

Abstract

A method is provided for navigating a robotic endoscope device. The method includes: (a) navigating the robotic endoscope device to a target region within a body part; (b) when a tool is extended into the target region via the robotic endoscope device, acquiring a fluoroscopic image using a fluoroscopic imager, the fluoroscopic image including a plurality of slices stacked in a depth direction; (c) identifying a first slice having a first coordinate corresponding to a center of the target region in the depth direction and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; and (d) determining whether the tool is within the target region based on a difference between the first coordinate and the second coordinate.

Description

Technical Field

[0001] (Cross - reference to related applications)

[0001] This invention claims priority to U.S. Provisional Patent Application No. 63 / 347,203, filed on May 31, 2022. This provisional application is hereby incorporated by reference in its entirety.

Background Art

[0002]

[0002] Early diagnosis of lung cancer is extremely important. Lung cancer is the most lethal form of cancer, causing over 150,000 deaths each year. Compared to CT - guided TTNA (CT - TTNA), navigational bronchoscopy has a favorable safety profile (risk reduction of pneumothorax, life - threatening bleeding, length of hospital stay) and can stage the mediastinum, but is associated with a low diagnostic rate. Endoscopy (e.g., bronchoscopy) can involve accessing and visualizing the inside of a patient's lumen (e.g., airway) for diagnostic and / or therapeutic purposes. During the procedure, a flexible tubular tool, such as an endoscope, can be inserted into the patient's body, and through this endoscope, instruments can be delivered to tissue sites identified for diagnosis and / or treatment.

[0003]

[0003] There is interest in robotic bronchoscopy systems for use in biopsying peripheral lung lesions. The robotic platform provides better stability, distal articulation, and visualization than conventional pre-curved catheters. Some conventional robotic bronchoscopy systems utilize shape sensing technology (SS) for guidance. The SS catheter has embedded fiber optic sensors that measure the shape of the catheter hundreds of times per minute. Other conventional robotic bronchoscopy systems incorporate direct visualization, optical pattern recognition, and geographical position sensing (OPRGPS) for guidance. Both the SS and OPRGPS systems generate electronically generated virtual targets using a pre-planned CT scan. However, the SS and OPRGPS systems are prone to CT-to-body deviation (CT2BD). CT2BD is the mismatch between the electronic virtual target and the actual anatomical position of the peripheral lung lesion. CT2BD can occur for various reasons, including atelectasis, neuromuscular weakness due to anesthesia, tissue distortion by the catheter system, bleeding, ferromagnetic interference, and anatomical disturbances such as pleural effusion. Neither the SS system nor the OPRGPS platform includes intraoperative real-time correction of CT2BD. In particular, CT2BD can increase the length of the procedure, frustrate the operator, and ultimately lead to non-diagnostic procedures.

Summary of the Invention

[0004]

[0004] In recent years, digital tomosynthesis algorithms have been introduced to correct CT2BD. Tomosynthesis is limited-angle tomography, unlike full-angle tomography (e.g., 180-degree tomography). However, tomosynthesis reconstruction does not have uniform resolution. The resolution is lowest in the depth direction. The standard method for presenting 3D volume datasets with three orthogonal planes (e.g., axial, sagittal, and coronal) is not useful. This is because the resolution of two of the orthogonal planes is low. For example, conventional tomosynthesis applied to respiratory medicine has low depth resolution (e.g., anteroposterior direction (AP)), and it is difficult to determine whether there is a tool within the target area (e.g., a lesion). A common method for viewing a tomosynthesis volume is to scroll in the depth direction where each slice has good resolution. In the case of respiratory medicine, the user can view the lesion and the needle in the coronal plane and manually scroll the slice in the anteroposterior (AP) direction to identify a special relationship between the needle and the lesion. However, such a process may be difficult to determine the spatial relationship of the structures in the depth direction due to the cumbersome manual process and human error. In particular, it may be difficult to determine with high confidence and accuracy whether there is a thin tool (e.g., a biopsy needle) within a lesion in the AP direction of tomosynthesis reconstruction.

[0005]

[0005] There is a need for a method and system that can improve accuracy and precision to determine whether a tool is within a target region (e.g., a lesion). The present disclosure addresses this need by providing a method and system for lesion-in-tool determination based on tomography. In particular, the method herein provides the user with quantitative information about the spatial relationship between a thin tool (e.g., a needle) in the depth direction and a target region (e.g., a lesion). The methods and systems herein can identify the positional relationship (in the depth direction) between the tool and the lesion by separately identifying the depths of the tool and the lesion, and can quantitatively determine whether the (thin) tool is within the lesion. As used herein, the term "thin tool" can represent at least a portion (e.g., the distal tip) of a tool having a dimension of 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, or 10 mm or less.

[0006]

[0006] The method herein can be utilized after setting up a robotic platform, identifying and segmenting the target lesion, performing airway alignment, and selecting individual target lesions. The robotic system herein can utilize integrated tomography to improve lesion visualization and in-lesion tool confirmation. For example, during surgery, when navigating an endoscope through the airway to reach the target, the in-lesion tool mode can be activated to verify whether a thin tool (e.g., a needle delivered through the working channel of the endoscope) is within the target. The endoscope navigation system can use various sensing modalities (e.g., camera image data, electromagnetic (EM) position data, robotic position data, etc.). In some examples, the navigation technique can start tracking the tip of the endoscope depending on an initial estimate of where the tip of the endoscope is relative to the airway. Some endoscopy techniques include a three-dimensional (3D) model of the patient's anatomical structure (e.g., a CT image) and can use an EM field and position sensors to guide the navigation.

[0007]

[0007] In some examples, for various purposes, a 3D image of a patient's anatomical structure can be acquired one or more times. For example, prior to a medical procedure, a 3D model of the patient's anatomical structure can be generated to identify a target location. In some examples, the precise alignment (e.g., registration) between the virtual space of the 3D model, the physical space of the patient's anatomical structure represented by the 3D model, and the EM field may be unknown. Thus, prior to generating the alignment, the endoscopic position within the patient's anatomical structure cannot be precisely mapped to the corresponding position within the 3D model. In another example, during surgery, if a target problem or lesion has moved, 3D imaging can be performed to update / confirm the position of the target (e.g., lesion). In some examples, to assist in reaching the position of the target tissue, the position and movement of a medical instrument can be aligned with the intraoperative image of the patient's anatomical structure. By aligning an image-guided instrument with the image, the instrument can navigate through natural or surgically created passages in anatomical systems such as the lungs, colon, intestine, kidney, heart, circulatory system, etc. In some examples, after a medical instrument (e.g., a needle, an endoscope) has reached the target position or after surgery has been completed, 3D imaging can be performed to confirm that the instrument or surgery is at the target position.

[0008]

[0008] In some examples, a fluoroscopic imaging system can be used to determine the position and orientation of medical instruments and the patient's anatomical structures within the coordinate system of a surgical environment. To assist in correctly positioning the medical instrument with the image data, a coordinate system of the imaging system may be required to reconstruct a 3D model. As described above, tomosynthesis or cone beam CT (CBCT) reconstructions can be generated using multiple 2D fluoroscopic images acquired at various orientations / angles to provide better visualization and 3D coordinates of the anatomical structures. However, conventional tomosynthesis has low depth resolution (AP direction) and it is difficult to determine whether a tool is within a target region (e.g., a lesion). The systems and methods herein beneficially improve the accuracy and precision of positioning a tool (e.g., a needle) with respect to a target region by quantitatively providing tool confirmation within the lesion.

[0009]

[0009] In one aspect, a method for navigating a robotic endoscope device is provided. The method includes: (a) navigating the robotic endoscope device to a target region within a body part; (b) when a tool is extended into the target region via the robotic endoscope device, acquiring one or more fluoroscopic images using a fluoroscopic imager and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having a first coordinate corresponding to the center of the target region in the depth direction and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; and (d) determining whether the tool is within the target region based at least in part on a comparison of the difference between the first coordinate and the second coordinate and a threshold value.

[0010]

[0010] In another related aspect, a non-transitory computer-readable storage medium including instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include: (a) navigating a robotic endoscope device to a target region within a body part; (b) when a tool is extended into the target region via the robotic endoscope device, acquiring one or more fluoroscopic images using a fluoroscopic imager and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having a first coordinate corresponding to the center of the target region in the depth direction and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; and (d) determining whether the tool is within the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate with a threshold value.

[0011]

[0011] In some embodiments, the target region is a visible lesion in the 3D fluoroscopic image. In some examples, the first slice is identified by: i) displaying the 3D fluoroscopic image within a graphical user interface (GUI); ii) selecting the first slice from a stack of slices when the lesion is in focus. In some examples, the second slice is identified when the tool is in focus. In some examples, the threshold value is determined based at least in part on the dimensions of the lesion. For example, the dimensions of the lesion are calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a).

[0012]

[0012] In some embodiments, the first slice or the second slice is automatically identified based on a sharpness measure or a contrast measure of each slice in the depth direction. In some examples, the method further includes displaying a 3D fluoroscopic image within a graphical user interface (GUI), and displaying an overlay of the lesion on each slice from a plurality of stacks in the depth direction. In some examples, the overlay is generated based at least in part on a 3D model of the lesion that intersects each slice. The method may further include determining whether the tool is within the target area by identifying whether an overlay of the lesion appears on the second slice.

[0013]

[0013] In some embodiments, the method may further include displaying, on a graphical user interface (GUI), a 3D fluoroscopic image, a first graphic visual indicator representing a first coordinate, and a second graphic visual indicator representing a second coordinate.

[0014]

[0014] In some embodiments, the 3D fluoroscopic image is reconstructed based on the pose of the fluoroscopic imager. In some examples, the pose of the fluoroscopic imager is estimated based on markers included in one or more acquired fluoroscopic images. In some examples, the pose of the fluoroscopic imager is acquired based on position sensor data. In some embodiments, the threshold includes a margin, and the margin is determined based on experimental data.

[0015]

[0015] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, which illustrates only exemplary embodiments of the present disclosure. As will be recognized, the present disclosure is capable of other and different embodiments, and some of its details are capable of various obvious modifications without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

[0016] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference into this application to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein by reference. To the extent that any incorporated publication, patent, or patent application conflicts with the disclosure contained herein, this specification is intended to supersede and / or take precedence over such conflicting material.

Brief Description of the Drawings

[0017]

[0017] The novel features of the present invention are particularly set forth in the appended claims. A further understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description, which describes exemplary embodiments in which the principles of the present invention are utilized, and the accompanying drawings (also referred to herein as "Figures" and "FIGs").

[0018]

Figure 1

[0018] An exemplary workflow for lung cancer diagnosis enabled by the robotic bronchoscopy system described herein is shown.

Figure 2

[0019] Examples of tools within a lesion and tools not within a lesion are schematically shown.

Figure 3

[0020] An example of a graphical user interface (GUI) showing a tomosynthesis image reconstructed with quantitative within-lesion tool information is shown.

Figure 4

[0021] An example of an optimal slice of a tool is shown.

Figure 5

[0022] An example of a GUI for assisting in the confirmation of a within-lesion tool is shown.

Figure 6

[0022] An example of a GUI for assisting in the confirmation of a within-lesion tool is shown.

Figure 7

[0022] An example of a GUI for assisting in the confirmation of a within-lesion tool is shown.

Figure 8

[0022] An example of a GUI for assisting in the confirmation of a within-lesion tool is shown.

Figure 9

[0022] An example of a GUI for assisting in the confirmation of in-lesion tools is shown.

Figure 10

[0023] An example of an in-lesion tool algorithm according to some embodiments of the present disclosure is shown.

Figure 11

[0024] An exemplary workflow for integrating in-lesion tool determination into the navigation of a robotic bronchoscopy system is shown.

Figure 12

[0025] An example of a robotic bronchoscopy system according to some embodiments of the present invention is shown.

Figure 13

[0026] An example of a fluoroscopy (tomosynthesis) imaging system is shown.

Figure 14

[0027] An example of a flexible endoscope is shown.

Figure 15

[0027] An example of a flexible endoscope is shown.

Figure 16

[0028] An example of an instrument drive mechanism that provides a mechanical interface to the handle portion of a robotic bronchoscope is shown.

Figure 17

[0029] An example of the distal tip of an endoscope is shown.

Figure 18

[0030] An exemplary distal portion of a catheter having an integrated imaging device and an illumination device is shown.

Figure 19

[0031] Another example of a GUI that displays quantitative in-lesion tool information is shown.

Figure 20

[0031] Another example of a GUI that displays quantitative in-lesion tool information is shown.

Figure 21

[0032] An exemplary process of tomosynthesis image reconstruction is shown.

Mode for Carrying Out the Invention

[0019]

[0033] Although various embodiments of the present invention are illustrated and described in this specification, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Without departing from the present invention, numerous variations, modifications, and substitutions may occur to those skilled in the art. It should be understood that various alternatives to the embodiments of the present invention described in this specification may be employed.

[0020]

[0034] Exemplary embodiments mainly target bronchoscopes, but those skilled in the art will recognize that this is not intended to be limiting. The devices described in this specification can be used in other therapeutic or diagnostic procedures, and include, but are not limited to, the digestive system including the esophagus, liver, stomach, colon, urinary tract, or the respiratory system including, but not limited to, the bronchi, lungs, and various other parts, and can be used in other anatomical regions of the patient's body.

[0021]

[0035] The embodiments described in this specification can be combined with one or more of many methods to provide improved diagnosis and treatment to patients. The disclosed embodiments can be combined with existing methods and devices to provide improved treatment, which is, for example, a combination with known methods of lung diagnosis and surgery, and surgery of other tissues and organs. Any one or more of the structures and steps described in this specification can be combined with any one or more additional structures and steps of the methods and devices described in this specification, and it will be understood that the drawings and the supporting text provide a description according to the embodiments.

[0022]

[0036] The treatment plans and definitions of the diagnostic or surgical procedures described herein are presented in the context of lung diagnosis or surgery, but the methods and devices described herein can be used to treat any tissue of the body as well as any organ and duct of the body. This includes, for example, the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ears, nose, mouth, soft tissues such as bone marrow, adipose tissue, muscle, gland and mucosal tissue, bone marrow and nerve tissue, cartilage, hard biological tissues such as teeth and bones, etc., as well as body cavities and body passages such as sinuses, ureters, colon, esophagus, lung passages, blood vessels, and larynx.

[0023]

[0037] When the terms "at least", "greater than", or "above" are in front of the first numerical value among a series of two or more numerical values, the terms "at least", "greater than", or "above" apply to each numerical value in that series of numerical values. For example, 1, 2, or 3 or more is synonymous with 1 or more, 2 or more, or 3 or more.

[0024]

[0038] When the terms "below", "less than", or "under" are in front of the first numerical value among a series of two or more numerical values, the terms "below", "less than", or "under" apply to each numerical value in that series of numerical values. For example, 3, 2, or 1 or less is synonymous with 3 or less, 2 or less, or 1 or less.

[0025]

[0039] As used herein, a processor includes one or more processors, which can be, for example, a single processor or multiple processors of a distributed processing system. The controller or processor described herein generally includes a tangible medium that stores instructions for performing steps of a process, and the processor can include, for example, one or more of a central processing unit, programmable array logic, gate array logic, or field programmable gate array. In some examples, the one or more processors can be a programmable processor (e.g., a central processing unit (CPU) or a microprocessor), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more ARM processors. In some examples, the one or more processors can be operably coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium can store logic, code, and / or program instructions executable by one or more processor units to perform one or more steps. The non-transitory computer-readable medium can include one or more memory units (e.g., a removable medium or an external storage device, such as an SD card or a random access memory (RAM)). The one or more methods or operations described herein can be implemented by hardware components or a combination of hardware and software, such as an ASIC, a dedicated computer, or a general-purpose computer.

[0026]

[0040] As used herein, the terms distal and proximal can generally represent positions indicated from a device and can be opposite of anatomical designations. For example, the distal position of a bronchoscope or catheter can correspond to the proximal position of a patient's elongate member, and the proximal position of a bronchoscope or catheter can correspond to the distal position of a patient's elongate member.

[0027]

[0041] The systems described herein include an elongate portion or member, such as a catheter. The terms "elongate member", "catheter", and "bronchoscope" are used interchangeably throughout this specification, unless the context indicates otherwise. The elongate member can be placed directly within a body lumen or cavity. In some embodiments, the system can further include a support device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position, or control the movement and / or operation of the elongate member. Alternatively or additionally, the support device can be a handheld device or other control device that may or may not include a robotic system. In some embodiments, the system can further include peripheral devices and subsystems, such as an imaging system, to assist and / or facilitate the navigation of the elongate member to a target site within a subject's body. Such navigation may require a registration process, as described later herein.

[0028]

[0042] In some embodiments of the present disclosure, a robotic endoscope (e.g., bronchoscopy) system is provided for performing surgery or diagnosis with improved performance and low cost. For example, a robotic bronchoscopy system can include an operable catheter that can be disposable as a whole. This advantageously reduces the need for sterilization or disinfection, which can be ineffective despite being costly and difficult to operate. Further, one challenge in bronchoscopy is to reach the upper lobe of the lung while navigating through the airway. In some examples, the provided robotic bronchoscopy system can be designed to have the function of autonomously or semi-autonomously navigating through an airway with a small bending curvature. Autonomous or semi-autonomous navigation may require a registration process. Alternatively, the robotic bronchoscopy system can be navigated by an operator using a control system with visual guidance.

[0029]

[0043] Typical lung cancer diagnosis and surgical treatment processes can vary widely depending on the techniques, clinical protocols, and clinical settings used by medical institutions. An inconsistent process can lead to delays in early lung cancer diagnosis, high costs for the healthcare system and patients for lung cancer diagnosis and treatment, and a high risk of clinical and technical complications. The provided robotic bronchoscopy system can enable standardized early lung cancer diagnosis and treatment. FIG. 1 shows an exemplary workflow 100 for standardized lung cancer diagnosis enabled by the robotic bronchoscopy system described herein.

[0030]

[0044] As shown in FIG. 1, in some examples, preoperative imaging can be performed to identify a lesion (101) and / or to identify an airway that can be used for alignment and navigation during a procedure. Any suitable imaging modality, such as magnetic resonance (MR), positron emission tomography (PET), X-ray, computed tomography (CT), and ultrasound, can be used to identify a lesion or region of interest. For example, a preoperative CT scan can be performed on a patient suspected of having lung cancer, and a suspected lung nodule can be identified within the CT image. The preoperative imaging process can be performed prior to bronchoscopy. The CT image can be analyzed to generate a map for guiding the navigation of a robotic bronchoscope during bronchoscopy (103). For example, a lesion or region of interest (ROI) can be segmented on the image. When the lung is imaged, a passage or path to the lesion can be highlighted on the reconstructed image to plan a navigation path. The reconstructed image can guide the navigation of a robotic bronchoscope to a target tissue or target site. In some examples, a 3D image data can be used to pre-plan a navigation path. For example, under the robotic control of a robotic bronchoscopy system, a catheter can be advanced towards a target site. The catheter can be manually, autonomously, or semi-autonomously manipulated or advanced towards the target site. In one example, the movement of the catheter can be image-guided so that the insertion and / or operation direction can be automatically controlled. In some examples, preoperative image data can be used to generate a virtual model of the airway with an overlay of the navigation path.

[0031]

[0045] In some cases, due to various reasons such as CT-body separation, the lesion location in preoperative imaging may not be accurate. In such cases, before a surgical procedure (e.g., biopsy or treatment), if the tip of the endoscope is near or adjacent to a target such as a lesion (105), the lesion location can be verified (107). Using a robotic bronchoscopy system, the exact location of the lesion can be verified or updated. For example, the bronchoscopy system can provide an interface to an imaging modality such as real-time fluoroscopy to perform in vivo real-time imaging of the target site and the surrounding area to identify the location of the lesion. In one example, a C-arm or O-arm fluoroscopy imaging system can be used to generate tomosynthesis or cone-beam CT images to verify or update the location of the lesion (107).

[0032]

[0046] Proceed to a surgical procedure such as a biopsy (109), insert various surgical tools such as a biopsy tool, brush, or forceps into the working channel of the catheter, and perform the biopsy or other surgical procedures manually or automatically. In some cases, another fluoroscopy (tomosynthesis) can be performed to confirm that the tool has reached the target site, i.e., for in-lesion tool confirmation. In-lesion tool confirmation may be repeated as required, as further described with reference to FIG. 11. The present disclosure provides a navigation method using integrated in-lesion tool detection based on digital tomosynthesis and tomosynthesis (tomo) reconstruction coordinate techniques. Details of the method are described with reference to FIGS. 2 to 10.

[0033]

[0047] In some examples, once the tool is confirmed to be within the target (e.g., a lesion), a sample of the lesion or any other target tissue can be obtained by the tool inserted into the working channel of the catheter (109). The system can maintain camera visualization throughout the procedure, including during insertion of the tool into the working channel. In some examples, the rapid cytology process can be used to quickly evaluate the tissue sample on-site to determine whether it is necessary to repeat the tissue sampling or to determine another action (111). In some examples, the rapid cytology process can also perform a rapid analysis of the tissue sample to determine the next surgical treatment. For example, if the tissue sample is determined to be malignant by the rapid cytology process, a treatment device can be inserted manually or robotically through the working channel of the robotic bronchoscope to perform endobronchial treatment of lung cancer (113). This advantageously enables diagnosis and treatment to be performed in one session, thereby enabling early stage lung cancer to be treated painlessly, quickly, and with precision.

[0034]

[0048] As described above, once the catheter has been navigated to a target region, such as a lesion, a tool, such as a needle, can be extended onto the catheter and inserted into the target region. For example, after placing a biopsy needle within the corrected target region, such as a lesion, it is desirable to confirm whether the tool is truly within the lesion. In some examples, after the tip of the catheter reaches the target site, a fluoroscopic image can be obtained to confirm the position of the catheter relative to the target position in real time. In some embodiments, the present disclosure provides methods and systems for quantitatively performing in-lesion tool confirmation. The in-lesion tool confirmation method may include a method based on tomosynthesis, which will be described later in this specification. In some examples, once the final position has been confirmed, a cone beam CT scan can be captured (e.g., 8-second sweep, 0.5 projection / degree, 396 projections) and used to perform CBCT in-lesion tool confirmation.

[0035]

[0049] In some examples, intralesional tool confirmation can be defined as the placement of a thin tool (e.g., a needle) within the lesion or along the tangent of the lesion in three orthogonal planes (axial, sagittal, and coronal). Alternatively, the intralesional tool may not include the state along the tangent. FIG. 2 schematically shows an example of an intralesional tool (200) and an example of a tool not within the lesion (210). As shown in these examples, the intralesional tool can be defined as a tool such as a biopsy needle within the lesion or along the tangent of the lesion. Although the method herein is described with respect to determining a biopsy needle within the lesion, it should be noted that the method is not limited to the type, shape, or dimension of the tool, nor to the type, shape, or dimension of the target region. For example, the tool can be any tool other than a needle, and the target region can be a lesion or not a lesion. The intralesional tool confirmation method can be used in any situation where the relative position of the tool with respect to the target region is required.

[0036]

[0050] In one aspect of the present disclosure, a method for real-time intralesional tool confirmation is provided. In some embodiments, the method can be a tomography-based method. As described above, tomosynthesis reconstruction does not have uniform resolution. The sweep angle and the number of projections determine the resolution. Since the sweep angle is limited and the number of projections is small, the tomosynthesis image data in the Fourier domain is incomplete, and it is difficult to generate an accurate tomosynthesis image reconstruction. The anisotropic geometric shape degrades the image resolution in the z-axis (i.e., the depth direction) derived from the Fourier domain without being directly acquired. The z-axis resolution is improved by widening the range of the sweep angle. However, considering the dose limit, there is an optimal number of projections for a specific sweep angle range, and exceeding this, the in-plane (i.e., x, y-axis) image quality degrades with the number of projections without an improvement in the z-axis resolution. As used herein, the term "depth direction" can indicate the vertical direction or the direction of the central ray. The slices stacked in the depth direction can be in-plane slices.

[0037]

[0051] In some examples, the reconstructed 3D volume image is considered as a plurality of thin (e.g., 0.5 - 1 mm) slices parallel to the detector surface. Unlike CT, the slice thickness and reconstruction interval of tomography do not have a one-to-one correspondence with the compression thickness. The standard method for presenting a 3D volume dataset by three orthogonal planes (e.g., axial, sagittal, and coronal directions) is not useful. This is because the resolution of two of the orthogonal planes is low. When a stack of 2D reconstructed image slices is presented on a display screen, the user can view each image slice of the coronal plane and advance the stack of slices in the anteroposterior (AP) direction. Usually, the image stack is in an orientation parallel to the detection surface, and each image is separated by a predetermined depth (e.g., 1 mm).

[0038]

[0052] In some embodiments, the system performs real-time intraoperative imaging to confirm the in-lesion tool and overcome the CT-body interspacing. Digital tomosynthesis in-lesion tool confirmation may require tomosynthesis reconstruction coordinate techniques. In some examples, the reconstructed tomosynthesis image is displayed on the user interface along with quantitative in-lesion tool information. In some examples, the quantitative in-lesion tool information can be based on coordinates representing the depth of the display slice within the reconstruction in the anteroposterior (AP) direction. Details regarding the reconstruction of tomosynthesis images are described later in this specification.

[0039]

[0053] FIG. 3 shows an example of a graphical user interface (GUI) that displays a reconstructed tomosynthesis image along with quantitative in-lesion tool information. The tomosynthesis image can be constructed using any suitable reconstruction algorithm for a plurality of projections (e.g., iterative algorithms such as filtered back projection (FBP), algebraic reconstruction technique (ART), etc.) to generate a 3D volume dataset. As shown in this example, coordinates such as the depth of the display slice in the AP direction, for example, are displayed along with the image.

[0040]

[0054] In the first example, the coordinates of the slice 301 of the lesion (e.g., 143.5 mm) and the coordinates of the slice 303 of the tool (e.g., 145.0 mm) are determined and displayed on the screen. In some examples, the slices 301, 305 of the lesion may be the optimal image slices of the focused lesion. This can be determined based on a measure of the sharpness or contrast of the slice. In some examples, the slices 303, 307 of the tool may be the optimal images of the focused tool. This can also be determined based on the sharpness or contrast of the image slice. FIG. 4 shows an example of the optimal slice of the tool. The optimal slice of the tool (e.g., the tip of the needle) can be the slice 401 showing the entire focused needle, or the slice 403 showing a part of the needle within the focused lesion area.

[0041]

[0055] In some embodiments, the coordinate information is used to calculate the difference between the coordinates of the depths of the respective optimal slices of the needle and the lesion, thereby determining whether the tool is within the lesion. Returning to FIG. 3, the distance 311 between the optimal image slices 301, 305 of the needle and the optimal images 305, 507 of the lesion can be calculated and used to determine whether the needle is within the lesion. The distance 311 can be the difference in depth (e.g., depth / AP direction coordinates) between the slice containing the focused tool and the slice containing the focused lesion.

[0042]

[0056] In some embodiments, the method may include comparing the distance 311 with a threshold value to determine whether the tool is within the lesion. In some examples, the threshold value can be determined based on the nodule size (or lesion size) determined by the average of the longest dimension and the shortest dimension on the pre-planned CT scan of the subject (patient). For example, if the average of the longest dimension and the shortest dimension of the target lesion on the pre-planned CT scan is 4 mm, and the difference in depth between the optimal image 301 of the lesion and the optimal image 303 of the needle is 1.5 mm, which is smaller than the threshold value of 4 mm, the GUI can display the result as a tool within the lesion. In another example, if the distance is 7.6 mm, which is larger than the threshold value, the GUI can display the result as no tool within the lesion.

[0043]

[0057] Thresholds for determining the in-lesion tool can be generated using various methods. In some embodiments, the threshold can be determined at least in part based on the dimensions of the lesion within the same subject. In some examples, the threshold can be the average of the longest and shortest dimensions on a pre-planned CT scan of the subject. In some examples, the threshold can be the radius of the lesion in the AP direction obtained from the subject's pre-planned scan or any scan performed during the procedure (e.g., a 3D scan performed during navigation or when the endoscope is near the target). In some examples, the threshold can include a margin, such that the tool is within the lesion when abs(x - y) < r - e (where r is the radius of the lesion in the AP direction, e is the margin, and x, y are the depth coordinates of the optimal tool slice and the optimal lesion slice), the tool is not within the lesion when abs(x - y) > r + e, and the tool is on the boundary / tangent when abs(x - y) = r + e. In some examples, the margin e can be determined based on experimental data. For example, the margin can be determined using experimental data, data provided by a physician, or historical data regarding the size of similar lesions.

[0044]

[0058] The optimal slice of the tool and / or the optimal slice of the lesion can be determined using various suitable methods. In some embodiments, the user can select the optimal slice of the lesion via the GUI by visually identifying the slice that contains the focused lesion. For example, the user can scroll through the slices in the depth direction and mark the slice in which the lesion or needle has the sharpest edge or the highest contrast as the optimal slice. Alternatively, the optimal slice of the lesion and / or the tool can be determined automatically. For example, software can perform an autofocus analysis of the image stack, such as by defining a measure of sharpness or contrast, and select the slice for which the selected measure is maximum. For example, the sharpness and / or contrast of the lesion area in each slice can be calculated, and the slice with the maximum sharpness measurement can be automatically determined as the optimal slice of the lesion.

[0045]

[0059] In some embodiments, the GUI of this specification can provide visual indicators or interactive visual guidance for the user to interact with the image slices to assist in lesion-internal tool determination. FIGS. 5 to 9 show various examples of the GUI for assisting in lesion-internal tool confirmation. FIG. 5 shows an example of the GUI for lesion-internal tool detection. The GUI can display a tomosynthesis volume image including an interactive graphic element (e.g., a slider bar) 505 that enables the user to scroll through a stack of slices in the AP direction (depth direction). As shown in this example, the depth coordinates 501, 511 associated with the currently displayed slice can also be displayed, for example, by overlaying them on the image slice. The user can scroll the slice by sliding the bar. The user can identify the first slice 500 when focused on the lesion 503 and obtain the associated depth coordinate 501. For example, the user can click on this slice and mark it as the optimal slice, and then the associated depth coordinate is automatically recorded by the system. In some examples, the coordinates of the first slice can be identified as the center of the lesion in the AP direction. Then the user can identify the second slice 510 when focused on the tool 513 and obtain the associated depth coordinate 511. Next, calculate the coordinate difference between the second slice and the center of the lesion in the AP direction (e.g., abs(138mm - 134.5mm)), compare it with a threshold value, and determine whether the tool is within the lesion.

[0046]

[0060] In some examples, on top of the tomosynthesis image, the mask, outer shape, or contour of the lesion within each slice can be overlaid to assist in intra-lesion tool confirmation. FIG. 6 shows an example of a GUI that displays the outer shape or boundary of the lesion within each slice. The user can scroll through the stack of slices, identify the first slice (e.g., the slice with the highest sharpness of the lesion boundary) that contains the focused lesion, and provide an input to the GUI indicating that the coordinates of the first slice are the center of the lesion. For example, the user can click on the first slice 601 and select it as the optimal slice of the lesion. Based on the center coordinates of the lesion, the system can automatically align this image with the 3D model of the lesion and determine the intersection points of the 3D lesion model and each slice. The 3D lesion model can be obtained from a previous scan of the subject, such as in a CT pre-planning process. For example, using preoperative images, a virtual model of the target tissue can be generated by segmenting the target lesion from the image data. Using the intersection points, a mask or contour of the lesions 605, 607 can be generated for each slice and overlaid on the image slice. Then the user can scroll through the slices and identify the second slice 603 when the tool (e.g., a biopsy needle) is in focus or at least a part of the tool is in focus. If a mask or contour of the lesion 607 appears in the second slice, the user can determine that the tool is within the lesion. In another example, if no mask or contour appears in the second slice where the tool is in focus, this may indicate that the tool is not within the lesion. Such visual indicators are advantageously useful for the user to intuitively confirm the intra-lesion tool.

[0047]

[0061] In some examples, rather than displaying the intersection / cross-section of the lesion on each slice, it can be displayed based on the radius of the lesion in the AP direction. FIG. 7 shows another example of the GUI. In this example, once the user identifies the slice 701 of the focused lesion (e.g., the highest sharpness or highest contrast of the contour), the coordinates of this slice can be used as the center of the lesion. Next, on each slice, the radius of the lesion in the AP direction (e.g., R = 6 mm AP) and the distance between the current slice 703 and the center of the lesion (e.g., d = 3.5 mm) can be displayed (705). The user can find the second slice containing the focused tool and check whether the corresponding distance d of the second slice is greater than the radius R. If the distance d is greater than the radius R, this indicates that the tool is not within the lesion.

[0048]

[0062] FIGS. 8 and 9 show examples of the GUI that display the depth difference between the slice of the lesion and the slice of the tool. In the illustrated example, once the user identifies the slice of the focused lesion, i.e., the optimal slice of the lesion, the user can provide input via the GUI, for example, by right-clicking on the image, etc., to mark the current slice as the center of the lesion. A visual indicator representing the lesion center slice (e.g., a thick horizontal line) and an indicator representing the current slice (e.g., a thin horizontal line) can be displayed together with the distance between these two slices. As the user scrolls the slice, the distance and indicator of the current slice can be dynamically rendered.

[0049]

[0063] In some embodiments, the optimal slice of the lesion (i.e., the slice where the focus of the lesion is best aligned) or the optimal slice of the tool (i.e., the slice where the tip of the tool, a part of the tool, or the entire tool is best aligned) can be automatically determined by software. For example, the sharpness or contrast measure of each slice can be calculated, and the slice with the maximum selected measure can be selected. In some examples, the automatically selected slice is presented to the user within the GUI, and the user can confirm this automatically selected slice or select another slice representing the lesion center and / or the tool slice.

[0050]

[0064] Figures 19 and 20 show other examples 1900, 2000 of a GUI that displays quantitative in-lesion tool information. As shown in FIG. 19, an overlay 1901 of the lesion can be displayed on the slice. The overlay 1901 of the lesion can be generated based on the intersection of the 3D model of the lesion and the slice, as described above. The overlay of the lesion can be switched on and off by the user. The user can scroll the slice and select the slice where the focus of the lesion is best as the optimal slice. Then, the depth of the optimal slice can be automatically recorded as the center of the lesion in the depth direction. Then, the user can scroll the slice and select the slice where the focus of the needle tip is best as the optimal slice of the needle. In some examples, when the user scrolls the slice, quantitative information indicating the spatial relationship between the tool and the lesion center in the current slice can be dynamically displayed on the GUI. For example, the dashed line 1907 can represent the depth position of the lesion center, and the solid line 1905 can represent the depth position of the current slice of the needle. The distance between the needle (in the current slice or the optimal slice) and the lesion center can be displayed (e.g., 0.1 mm forward, 3.8 mm forward, etc.). The GUI can further display the AP diameter 1909 (e.g., 10.0 mm) of the target object or lesion, and can also display the target 1907 as viewed from the right-left direction. In some examples, in addition to the quantitative indicator, color coding can be used to indicate whether the tool is inside or outside the lesion. For example, as shown in examples 1900, 1910, when the tool is inside the lesion, the lines 1905, 1907 can be displayed in green, and as shown in example 1920, when the tool is outside the lesion, the line 1921 representing the tool optimal slice can be displayed in red. FIG. 20 shows an example of a GUI 2000 that displays a virtual view of the tip of the target (e.g., lesion) and the tool. As shown in example 2000, a graphic visual indicator 2005 representing the position of the needle tip in the 3D space can be displayed on the virtual view. A second graphic indicator 2003 representing the target or lesion in the 3D space can be displayed. A third indicator 2001 representing the tip of the endoscope can be displayed.The virtual view can be generated based on preoperative image data and / or 3D image data acquired later. The virtual view 2000 with the needle tip position, the lesion position, and the endoscope tip overlaid enables the user to advantageously visualize the spatial relationship between these three objects.

[0051]

[0065] Figure 10 shows an example of a lesion-in tool algorithm 1000 according to some embodiments of the present disclosure. The method may include identifying a first image slice (e.g., a tomosynthesis slice) representing the center of a target such as the center of the lesion (1001). The first image slice can be identified by the user within the GUI as described above. For example, the user can identify the slice focused on the lesion based on the sharpness and / or contrast of the slice. Alternatively or additionally, the first slice may be automatically identified based on sharpness and contrast metrics calculated for each slice. For example, the algorithm can calculate the sharpness and / or contrast within the lesion area of each slice (e.g., the user draws a box or region indicating a possible location of the lesion within the image slice) and select the slice with the maximum metric value as the first slice. Using the coordinates of the first slice in the AP / depth direction, the depth coordinate of the center of the lesion can be marked.

[0052]

[0066] Next, a second image slice (e.g., a tomosynthesis image) corresponding to the optimal focus tool can be identified (1003). The second slice can also be manually selected by the user via the GUI or automatically selected by the system by calculating image metrics such as sharpness or contrast. The difference between the coordinates of the first and second image slices in the AP / depth direction is calculated (1005) and can be used to determine whether the tool is within the lesion. In some examples, this difference can be automatically compared to a threshold (1007). If the difference is below the threshold, the algorithm can output a result indicating that the tool is within the lesion. Alternatively, if the difference is greater than the threshold, the algorithm can output a result indicating that the tool is not within the lesion. The threshold can be determined using any suitable method as described above. For example, the threshold can be the average of the longest and shortest dimensions on the subject's pre-planned CT scan. Alternatively, the threshold can be the radius of the lesion in the AP direction obtained from the subject's pre-planned scan or any scan performed during the procedure. In some examples, the threshold can include a margin, and the tool is within the lesion if abs(x - y) < r - e (where r is the radius of the lesion in the AP direction, e is the margin, and x, y are the depth coordinates of the optimal tool slice and the optimal lesion slice), the tool is not within the lesion if abs(x - y) > r + e, and the tool is on the boundary if abs(x - y) = r + e. In some examples, the margin e can be determined based on experimental data.

[0053]

[0067] Optionally, the algorithm can generate a visual indicator (e.g., a mask or contour of the lesion) of the lesion intersecting each image slice based on the 3D lesion model and the lesion center coordinates determined in step 1001. The user can determine whether the tool is within the lesion by viewing the visual indicator overlaid on the second slice (the slice in focus for the tool).

[0054]

[0068] FIG. 11 shows an exemplary workflow for integrating lesion-instrument determination into robotic bronchoscopy system navigation. As shown in this example, planning software can be used to identify and segment the target lesion and plan the path. The robotic platform can be set up before performing airway alignment and selecting individual target lesions. The catheter system can be guided to the desired target lesion using geographical position guidance. Next, the 2D fluoroscopic c-arm is utilized to perform the lesion-instrument tomosynthesis technique by means of an extended fluoroscopy (TILT+) sweep. In this example, the c-arm sweep includes a limited-angle circular rotation from 30 degrees left anterior oblique to 30 degrees right anterior oblique. Mark the bronchoscope tip position. Based on the reconstruction algorithm, stack the 2D images to generate cross-sectional images. Also, mark the target position on the cross-sectional images. Details regarding the reconstruction of 3D volume tomosynthesis images are described with reference to FIGS. 13 and 21.

[0055]

[0069] Next, navigate the catheter to the corrected target and place the needle. If desired, the operator can utilize extended fluoroscopy to assist in optimizing the position of the bronchoscope and the tool. Perform iterative TILT sweeps to confirm the lesion-instrument placement. Repeated attempts can be made at the discretion of the user until the needle is optimally positioned. Digital tomosynthesis lesion-instrument confirmation is based on the tomosynthesis reconstruction coordinate technique as described above.

[0056]

[0070] Once the final position is confirmed, a cone beam CT scan can be captured. CBCT lesion-instrument confirmation can be defined as the placement of the needle within the lesion or along the tangent of the lesion in three orthogonal planes (axial, sagittal, and coronal).

[0057]

[0071] FIG. 12 shows an example of robotic bronchoscopy systems 1200, 1230 according to some embodiments of the present invention. As shown in FIG. 12, the robotic bronchoscopy system 1200 may include an operable catheter assembly 1220 and a robotic support system 1210 for supporting or carrying the operable catheter assembly. The operable catheter assembly can be a bronchoscope. In some embodiments, the operable catheter assembly can be a single-use robotic bronchoscope. In some embodiments, the robotic bronchoscopy system 1200 may include an instrument drive mechanism 1213 attached to the arm of the robotic support system. The instrument drive mechanism can be provided by any suitable controller device (e.g., a handheld controller) that may or may not include the robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to the operable catheter assembly 1220. The mechanical interface enables the operable catheter assembly 1220 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the operable catheter assembly can be attached to the instrument drive mechanism via a quick installation / release means such as a magnet, a spring level, etc. In some examples, the operable catheter assembly can be manually coupled to or released from the instrument drive mechanism without using tools.

[0058]

[0072] The operable catheter assembly 1220 can include a handle portion 1223 that can include components configured to process image data, supply power, or establish communication with other external devices. For example, the handle portion 1223 can include circuitry and communication elements that enable electrical communication between the operable catheter assembly 1220 and the instrument drive mechanism 1213 and any other external system or device. In another example, the handle portion 1223 can include circuit elements such as a power source for powering the electronics of an endoscope (e.g., a camera and LED lights). In some examples, the handle portion is in electrical communication with the instrument drive mechanism 1213 via an electrical interface (e.g., a printed circuit board), such that image / video data and / or sensor data can be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. Alternatively or additionally, the instrument drive mechanism 1213 may provide only a mechanical interface. The handle portion can be in electrical communication with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals. Details regarding the handle portion are described later in this specification.

[0059]

[0073] The operable catheter assembly 1220 may include a flexible elongate member 1211 coupled to the handle portion. In some embodiments, the flexible elongate member may include a shaft, an operable tip, and an operable portion. The operable catheter assembly can be a single-use robotic bronchoscope. In some examples, only the elongate member can be disposable. In some examples, at least a portion of the elongate member (e.g., the shaft, the operable tip, etc.) can be disposable. In some examples, the entire operable catheter assembly 1220 including the handle portion and the elongate member can be disposable. The flexible elongate member and the handle portion are designed such that the entire operable catheter assembly can be disposed of at low cost. Details of the flexible elongate member and the operable catheter assembly are described later in this specification.

[0060]

[0074] In some embodiments, the provided bronchoscopy system may also include a user interface. As shown in the exemplary system 1230, the bronchoscopy system may include a treatment interface module 1231 (on the user console side) and / or a treatment control module 1233 (on the patient and robot side). The treatment interface module enables an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the treatment control module 1233 can be a handheld controller. The treatment control module can, in some examples, include a proprietary user input device and one or more add-on elements detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or roller can be given a function similar to that of at least one of the virtual graphic elements displayed on a graphical user interface (GUI) (e.g., navigation arrows displayed on a touchpad) to replace or supplement the function of the virtual graphic element. Examples of user devices can include, but are not limited to, mobile devices, smartphones / cell phones, tablets, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, etc. Details regarding the user interface device and user console are described later herein.

[0061]

[0075] The user console 1231 can be mounted on the robot support system 1210. Alternatively or additionally, the user console or a part of the user console (e.g., the treatment interface module) may be mounted on a separate mobile cart.

[0062]

[0076] The present disclosure provides a robotic endoluminal platform using an integrated intralesional tool tomosynthesis technique. In some examples, the robotic endoluminal platform can be a bronchoscopy platform. The platform can be configured to perform one or more operations consistent with the methods described herein. FIG. 13 shows an example of a robotic endoluminal platform and its components or subsystems according to some embodiments of the present invention. In some embodiments, the platform can include a robotic bronchoscopy system and one or more subsystems that can be used in combination with the robotic bronchoscopy system of the present disclosure.

[0063]

[0077] In some embodiments, the one or more subsystems can include an imaging system, such as a fluoroscopy imaging system, for performing real-time imaging of a target site (e.g., including a lesion). Multiple 2D fluoroscopy images can be used to generate tomosynthesis or cone beam CT (CBCT) reconstructions to provide better visualization and 3D coordinates of anatomical structures. FIG. 13 shows an example of a fluoroscopy (tomosynthesis) imaging system 1300. For example, the fluoroscopy (tomosynthesis) imaging system can perform accurate lesion position tracking or intralesional tool confirmation before or during a surgical procedure as described above. In some examples, the lesion position can be tracked based on position data regarding the fluoroscopy (tomosynthesis) imaging system / station (e.g., a C-arm) and image data captured by the fluoroscopy (tomosynthesis) imaging system. The lesion position can be aligned with the coordinate frame of the robotic bronchoscopy system.

[0064]

[0078] In some examples, the position, pose, or movement of a fluoroscopic imaging system can be measured / estimated to align the coordinate frame of an image with a robotic bronchoscopy system or to construct a 3D model / image. The pose or movement of the fluoroscopic (tomosynthesis) imaging system can be measured using any suitable motion / position sensor 1310 disposed on the fluoroscopic (tomosynthesis) imaging system. The motion / position sensor can include, for example, an inertial measurement unit (IMU), one or more gyroscopes, velocity sensors, accelerometers, magnetometers, position sensors (e.g., a global positioning system (GPS) sensor), vision sensors (e.g., an imaging device capable of detecting visible light, infrared light, or ultraviolet light such as a camera), proximity or distance sensors (e.g., ultrasonic sensors, lidar, time-of-flight or depth cameras), altitude sensors, attitude sensors (e.g., compasses), and / or field sensors (e.g., magnetometers, electromagnetic sensors, radio sensors, etc.). In some examples, one or more sensors for tracking the movement and position of the fluoroscopic (tomosynthesis) imaging station can be disposed on the imaging station or positioned remotely from the imaging station such as a wall-mounted camera 1320. A C-arm fluoroscopic (tomosynthesis) imaging system in various (rotational) poses while imaging a subject. The various poses can be captured by one or more of the sensors described above.

[0065]

[0079] In some examples, the pose of a C-arm fluoroscopic (tomosynthesis) imaging system can be estimated using optical methods. FIG. 21 shows an exemplary process 2100 for tomosynthesis image reconstruction. In some examples, the tomosynthesis image reconstruction of process 2100 can include generating a 3D volume using a combination of X-ray projection images acquired at various angles (by any type of C-arm system).

[0066]

[0080] The process may include obtaining C-arm video or image data using an imaging device such as a C-arm imaging system (105). The C-arm imaging system may include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or an X-ray imager). The video or image data that can be generated by the C-arm system can be used for both tomography and enhanced fluoroscopy (e.g., the display of live 2D fluoroscopy examinations enhanced by an overlay of lesions, tools, or other information). In some examples, raw video frames can be used for both tomography and fluoroscopy. However, tomography requires unique frames from the C-arm, while enhanced fluoroscopy can operate using duplicate frames from the C-arm when the frames are live video. The method herein can provide a unique frame check algorithm to process video frames for tomography to ensure uniqueness. For example, when a new image frame is received, if the current mode is tomography, the image frame can be processed to determine whether it is a unique frame or a duplicate. The uniqueness check can be based on an image intensity comparison threshold. For example, if the intensity difference from the previous frame is less than a predetermined threshold, the frame can be identified as a duplicate frame and excluded from use in tomography reconstruction. If the current mode is fluoroscopy, the image frame may not need to be processed to check for uniqueness.

[0067]

[0081] Operation 2110 may include detecting a video or image frame from the C-arm source. The video or image frame may be normalized. Normalization can be an image processing technique that varies the range of pixel intensity values within the video or image frame. Generally, normalization transforms an n-dimensional grayscale image I: {X ⊆ R n} → {Min,…,Max} with intensity values in the range (Min NEW , Max NEW ) into a new image I NEW : {X ⊆ R n} → {Min NEW ,…,Max NEW} can be changed to. Examples of normalization techniques that can be used for C-arm videos or image frames can include linear scaling, clipping, logarithmic scaling, z-score, or any other suitable type of normalization.

[0068]

[0082] For both tomosynthesis image reconstruction and extended fluoroscopy overlay, accurate camera pose and camera parameters are important. In some examples, marker detection can be performed for pose estimation (2115). In some examples, the markers can be 2D or 3D markers on the tomosynthesis substrate, and the X-ray projection of the markers on the tomosynthesis substrate can be the markers detected in the X-ray image. The markers can be detected using any suitable image processing or computer vision techniques (e.g., SfM (Structure from Motion)). For example, using the blob detection algorithm in OpenCV, blob-shaped markers can be detected. In some examples, the detected markers (e.g., blobs) can be detected to have specific characteristics such as the position, shape, size, color, darkness / brightness, opacity, or other suitable characteristics of the markers.

[0069]

[0083] In some examples, method 2100 can include, at 2120, matching the markers to a substrate pattern. The markers detected in the fluoroscopy image can be matched to the tomosynthesis substrate. As described above, the markers exhibit any number of various physical characteristics (e.g., position, shape, size, color, darkness / brightness, opacity, etc.), and these can be detected and used for matching the markers to the marker pattern on the tomosynthesis substrate. For example, the tomosynthesis substrate can have different types of patterns such as large blobs and small blobs. In some examples, the patterns generated by large blobs and small blobs can be used to match the marker pattern in the video or image frame to the pattern on the tomosynthesis substrate.

[0070]

[0084] In some examples, method 2100 may include finding the best marker matching in all video or image frames (2125). The first marker matching may be a match between the in-frame marker and the tomosynthesis substrate. In some examples, the pattern of the matched markers can be compared to the tomosynthesis substrate to find the best match using the Hamming distance. For each frame, a matching with a pattern matching score (e.g., the number of matched markers divided by the total number of detected markers) can be obtained. The best match can be determined as the match with the highest pattern matching score among all frames (2125).

[0071]

[0085] Process 2100 can perform inter-frame tracking (2130). At a high level, inter-frame tracking (2130) may include propagating marker matching from the best match determined in operation 2125 to the remaining tomosynthesis video and image frames by robust tomosynthesis marker tracking. In some examples, (i) first, match the markers in a pair of consecutive frames, (ii) then match each marker in the first frame with the k closest markers in the second frame, (iii) for each of the matched marker pairs, calculate the motion displacement between the two frames, (iv) move all the markers in the first frame to the second frame according to the motion displacement, (v) if the motion displacement between a given moved point from the first frame and a given point position in the second frame is smaller than the threshold and the two given marker types are the same, this match is an inlier, and (vi) the best matching may be the motion with the most inliers. From the calculated tomosynthesis marker tracking (2130), move the existing marker match in the current frame to the marker match in the next frame. In some examples, such a process is repeated for all frames (2135) to match the markers in all frames with the tomosynthesis substrate and find the marker match in all frames.

[0072]

[0086] In some examples, the pose estimation 2140 of the imaging device may include recovering rotation and translation by minimizing the reprojection error from the 3D-2D point correspondence relationship in order to perform pose estimation. In some examples, a camera pose can be recovered from n pairs of point correspondence relationships using PnP (Perspective-n-Point) pose calculation. The minimal form of the PnP problem is P3P, which can be solved by three-point correspondence. There may be multiple marker matches in each tomosynthesis frame, and the RANSAC variant of the PnP solver can be used for pose estimation. In some examples, the pose estimation 2140 can be further improved by minimizing the reprojection error using a non-linear minimization method and starting from an initial pose estimation using a PnP solver.

[0073]

[0087] Tomosynthesis reconstruction 2145 can be based on the pose estimation result. In some examples, the tomosynthesis reconstruction 2145 can be implemented as a model in Python (or other suitable programming language) using the open-source ASTRA (MATLAB and Python toolbox for high-performance GPU primitives for 2D and 3D tomography) toolbox (or other suitable toolbox or package). In tomosynthesis reconstruction, the inputs to the model can be as follows: (i) the corrected projection images without distortion (correction is the process of restoring a damaged image), (ii) the estimated projection matrices such as the pose of each projection, and (iii) the size, resolution, and estimated position of the targeted tomosynthesis reconstruction volume. The output of the model is the tomosynthesis reconstruction (for example, a volume in NifTI format) 2145. For example, the tomosynthesis reconstruction can include 3D volume data of the surgical field or the target scene, and lesions and tools can be seen within this 3D volume data.

[0074]

[0088] Returning to FIG. 13, in some embodiments, the signal processing unit 1330 can be utilized to segment the location of a lesion within the image data (e.g., 3D volume data) captured by a fluoroscopy (tomosynthesis) imaging system. One or more processors of the signal processing unit can be configured to further overlay the treatment location (e.g., the lesion) on a real-time fluoroscopy image / video. For example, the processing unit can be configured to generate an overlay layer that includes augmentation information such as the location of the treatment location or target site. In some examples, the overlay layer can also include graphic markers that indicate the path to this target site. The overlay layer can be a substantially transparent image layer that includes one or more graphic elements (e.g., boxes, arrows, etc.). The overlay layer can be overlaid on the optical view of the optical image or video stream captured by the fluoroscopy (tomosynthesis) imaging system and / or can be displayed on a display device. Due to the transparency of the overlay layer, the user can view the optical image with the graphic elements overlaid thereon. In some examples, both the segmented lesion image and the optimal path for navigating an elongate member to reach the lesion can be overlaid on the real-time tomosynthesis image. Thereby, the operator or user can visualize the exact location of the lesion and the planned path of bronchoscope movement. In some examples, the segmented and reconstructed images (e.g., CT images described elsewhere) provided prior to the operation of the systems described herein can be overlaid on the real-time images.

[0075]

[0089] In some embodiments, one or more subsystems of the platform can include one or more treatment subsystems, such as manual or robotic instruments (e.g., biopsy needles, biopsy forceps, biopsy brushes), and / or manual or robotic treatment instruments (e.g., RF ablation instruments, cryogenic instruments, microwave instruments, etc.).

[0076]

[0090] In some embodiments, one or more subsystems of the platform may include a navigation and localization subsystem. The navigation and localization subsystem may be configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images or tomosynthesis). The navigation and localization subsystem is configured to identify a segmented lesion location within the 3D-rendered airway model, and based on the location of the lesion, the navigation and localization subsystem can generate an optimal path from the main bronchus to the lesion at a recommended approach angle to the lesion for performing a surgical procedure (e.g., a biopsy).

[0077]

[0091] In an alignment step before driving the bronchoscope to the target site, the system can align a virtual view of the rendered airway with the patient's airway. The image alignment can consist of a single alignment step, or a combination of a single alignment step and a real-time sensory update to the alignment information. The alignment process can include finding a transformation that aligns objects (e.g., airway model, anatomical site) between different coordinate systems (e.g., EM sensor coordinates and patient 3D model coordinates based on preoperative CT imaging). Details regarding alignment are described later in this specification.

[0078]

[0092] Once integrated, all airways can be aligned to the preoperatively rendered airways. While driving the bronchoscope towards the target site with a robot, the position of the bronchoscope inside the airway can be tracked and displayed. In some examples, a positioning sensor can be used to track the position of the bronchoscope relative to the airway. Also, sensor fusion techniques can be used to use other types of sensors (e.g., cameras) instead of or together with the positioning sensor. A positioning sensor such as an electromagnetic (EM) sensor can be embedded at the distal tip of the catheter, and the EM field generator can be positioned next to the patient's torso during the procedure. The EM field generator enables identifying the position of the EM sensor in 3D space or the position and orientation of the EM sensor in 5D or 6D space. This can provide a visual guide to the operator when driving the bronchoscope towards the target site.

[0079]

[0093] In real-time EM tracking, an EM sensor composed of one or more sensor coils embedded at one or more positions and orientations on a medical device (e.g., the tip of an endoscopic tool) measures the fluctuations of the EM field generated by one or more stationary EM field generators placed near the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coils of the EM sensor, and by analyzing this, the distance and angle between the EM sensor and the EM field generator can be determined. These distances and orientations can be aligned to the patient's anatomical structure (e.g., 3D model) during the operation to determine a registration transformation that aligns a single position in the coordinate system with a position in the preoperative model of the patient's anatomical structure.

[0080]

[0094] In some embodiments, the platform herein can use a fluoroscopy imaging system to determine the position and orientation of medical instruments and a patient's anatomical structures within the coordinate system of a surgical environment. In particular, the systems and methods herein can employ mobile C-arm fluoroscopy as a low-cost, mobile, real-time quantitative assessment tool. Fluoroscopy is an imaging modality that acquires real-time video of a patient's anatomical structures and medical instruments. A C-arm system that may be included in a fluoroscopy system provides positional flexibility and can be moved in a circular, horizontal, and / or vertical direction by manual or automated control. Fluoroscopy image data from multiple viewpoints in a surgical environment (i.e., by a fluoroscopy imager that has moved between multiple positions) can be compiled to generate two-dimensional or three-dimensional tomographic images. When using a fluoroscopy imager system that includes a digital detector (e.g., a flat panel detector), the generated and compiled fluoroscopy image data enables sectioning of the planar image in parallel planes according to tomosynthesis imaging techniques. A C-arm imaging system can include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or an X-ray imager). The X-ray detector can generate an image representing the intensity of the received x-rays. The imaging system can reconstruct a 3D image based on a plurality of 2D images acquired from a wide range of angles. In some examples, the rotation angle range can be at least 120 degrees, 130 degrees, 140 degrees, 150 degrees, 160 degrees, 170 degrees, 180 degrees, or more. In some examples, the 3D image can be generated based on the pose of the X-ray imager.

[0081]

[0095] The bronchoscope or catheter can be disposable. FIG. 14 shows an example of a flexible endoscope 1400 according to some embodiments of the present disclosure. As shown in FIG. 14, the flexible endoscope 1400 may include a handle / proximal portion 1409 and a flexible elongated member that is inserted into the subject. The flexible elongated member can be the same as those described above. In some embodiments, the flexible elongated member may include a proximal shaft (e.g., insertion shaft 1401), an operable tip (e.g., tip 1405), and an operable portion (active bending portion 1403). The active bending portion and the proximal shaft portion can be the same as those described elsewhere in this specification. Also, the endoscope 1400 may be referred to as an operable catheter assembly as described elsewhere in this specification. In some examples, the endoscope 1400 can be a single-use robotic endoscope. In some examples, the entire catheter assembly can be disposable. In some examples, at least a portion of the catheter assembly can be disposable. In some examples, the entire endoscope can be released from the instrument drive mechanism and disposed of. In some embodiments, the endoscope may include various levels of stiffness along the shaft to improve its functional operation.

[0082]

[0096] The endoscope or operable catheter assembly 1400 can include a handle portion 1409 that can include one or more components configured to process image data, supply power, or establish communication with other external devices. For example, the handle portion can include circuits and communication elements that enable electrical communication between the operable catheter assembly 1400 and an instrument drive mechanism (not shown) and any other external system or device. In another example, the handle portion 1409 can include circuit elements such as a power source for supplying power to the electronics of the endoscope (e.g., camera, electromagnetic sensor, and LED lights).

[0083]

[0097] One or more components positioned on the handle can be optimized to reduce costs and simplify the design of a disposable endoscope by allocating expensive and complex components to a robot support system, a handheld controller, or an instrument drive mechanism. The handle portion or proximal portion can provide electrical and mechanical interfaces to enable electrical and mechanical communication with the instrument drive mechanism. The instrument drive mechanism can include a motor set that is actuated to rotatably drive the pull wire set of the catheter. The handle portion of the catheter assembly can be mounted on the instrument drive mechanism such that its pulley / capstan assembly is driven by the motor set. The number of pulleys can vary based on the configuration of the pull wires. In some examples, one, two, three, four, or more pull wires can be used to bend a flexible endoscope or catheter like a joint.

[0084]

[0098] The handle portion can be designed to enable a low-cost disposable robotic bronchoscope. For example, conventional manual and robotic bronchoscopes can have cables at the proximal end of the bronchoscope handle. Often, the cables can include illumination fibers, camera video cables, and other sensor fibers or cables such as electromagnetic (EM) sensors or shape sensing fibers. Such complex cables are expensive and can increase the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design that allows for the use of simplified structures and components while maintaining mechanical and electrical functionality. In some examples, the handle portion of the robotic bronchoscope can use a cable-free design while providing a mechanical / electrical interface to the catheter.

[0085]

[0099] An electrical interface (e.g., a printed circuit board) can receive image / video data and / or sensor data at a communication module of the instrument drive mechanism and can also transmit it to other external devices / systems. In some examples, the electrical interface can establish electrical communication without using either cables or wires. For example, the interface can include pins soldered onto an electronic device substrate such as a printed circuit board (PCB). For example, a receptacle connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This advantageously enables the endoscope to be quickly plugged into the instrument drive mechanism or robot support without using extra cables. Since such a type of electrical interface can also function as a mechanical interface, both mechanical coupling and electrical coupling can be established when a handle portion is plugged into the instrument drive mechanism. Alternatively or additionally, the instrument drive mechanism may provide only a mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals.

[0086]

[0100] In some examples, the handle portion 1409 can include one or more mechanical control modules such as a luer 1411 for interfacing with a cleaning system / suction system. In some examples, the handle portion can include a lever / knob for joint control. Alternatively, joint control can be located at a separate controller attached to the handle portion via the instrument drive mechanism.

[0087]

[0101] The endoscope can be attached to a robotic support system or a hand-held controller via an instrument drive mechanism. The instrument drive mechanism can be provided by any suitable controller device (such as a hand-held computer) that may or may not include a robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to an operable catheter assembly 1400. The mechanical interface enables the operable catheter assembly 1400 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the operable catheter assembly can be attached to the instrument drive mechanism via a quick installation / release means such as a magnet, a spring level, etc. In some examples, the operable catheter assembly can be manually coupled to or released from the instrument drive mechanism without using tools.

[0088]

[0102] In the example described, the distal tip of the catheter or endoscope shaft is configured to bend / flex like a joint with two or more degrees of freedom to provide a desired camera view or control the direction of the endoscope. As shown in this example, an imaging device (such as a camera) and a position sensor (such as an electromagnetic sensor) 1407 are disposed at the tip of the catheter or endoscope shaft 1405. For example, the line of sight of the camera can be controlled by controlling the joints of the active bending portion 1403. In some examples, the angle of the camera can be adjusted so that the line of sight can be adjusted without or in addition to bending the distal tip of the catheter or endoscope shaft like a joint. For example, the camera can be oriented at an angle (such as a tilt) with respect to the axial direction of the tip of the endoscope using an optimal component.

[0089]

[0103] The distal tip 1405 can be a rigid component that enables the positioning of sensors such as electromagnetic (EM) sensors, imaging devices (such as cameras), and other electronic components (such as LED light sources) embedded in the distal tip.

[0090]

[0104] In real-time EM tracking, an EM sensor composed of one or more sensor coils embedded in a medical device (e.g., the tip of an endoscopic tool) at one or more positions and orientations measures the variations in the EM field generated by one or more static EM field generators placed near the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) is placed near the patient and can generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, and by analyzing this, the distance and angle between the EM sensor and the EM field generator can be determined. For example, the EM field generator can be positioned near the patient's torso during a procedure to enable identifying the EM sensor position in 3D space, or identifying the EM sensor position and orientation in 5D or 6D space. This can provide a visual guide to the operator when driving a bronchoscope towards a target site.

[0091]

[0105] The endoscope can have a unique design in the elongated member. In some examples, the active bending section 1403 and the proximal shaft of the endoscope are composed of a single tube that includes a series of cuts (e.g., reliefs, slits, etc.) along its length, enabling improved flexibility, desirable rigidity, and a retention prevention mechanism (e.g., a mechanism for defining a minimum bending radius).

[0092]

[0106] As described above, the active bending section 1403 can be designed to be bendable with two or more degrees of freedom (e.g., bend like a joint). The unique structure of the active bending section can achieve greater bending degrees such as 180 degrees and 270 degrees (or other joint parameters for clinical applications). In some examples, a variable minimum bending radius along the axial axis of the elongated member can be provided such that the active bending section can include two or more different minimum bending radii.

[0093]

[0107] By applying force to the distal end of the endoscope via one or more pull wires, the joints of the endoscope can be controlled. The one or more pull wires can be attached to the distal end of the endoscope. When there are multiple pull wires, pulling the wires one by one at a time can change the orientation of the distal tip to sway in the up, down, left, right, or any necessary direction. In some examples, the pull wires can be fixed to the distal tip of the endoscope, extend through the bending portion, enter the handle, and be coupled to a drive component (such as a pulley). This handle pulley can interact with the output shaft from the robotic system.

[0094]

[0108] In some embodiments, the proximal end or proximal portion of the one or more pull wires can be operably coupled to various mechanisms (such as gears, pulleys, capstans, etc.) in the handle portion of the catheter assembly. The pull wire can be a metal wire, cable, or thread, or a polymer wire, cable, or thread. The pull wire can also be made of natural or organic materials or fibers. The pull wire can be any suitable type of wire, cable, or thread that can support various types of loads without deformation, significant deformation, or damage. The distal end / distal portion of the one or more pull wires is fixed or integrated with the distal portion of the catheter so that when the control unit operates the pull wire, a force or tension is applied to the distal portion, enabling at least the distal portion of the catheter (such as the flexible portion) to be manipulated or bent like a joint (e.g., in the up, down, pitch, yaw, or any direction between them).

[0095]

[0109] The pull wire can be made of any suitable material such as stainless steel (e.g., SS316), metal, alloy, polymer, nylon, or biocompatible material. The pull wire can be a wire, cable, or thread. In some embodiments, different pull wires may be made of different materials to vary the load-bearing capacity of the pull wire. In some embodiments, different portions of the pull wire can be made of different materials to vary the stiffness and / or load-bearing capacity along the pull. In some embodiments, the pull wire can be utilized for the transfer of electrical signals.

[0096]

[0110] The proximal design can improve the reliability of the device without incurring additional cost, enabling a low-cost single-use endoscope. In another aspect of the invention, a single-use robotic endoscope is provided. The robotic endoscope can be a bronchoscope and can be the same as the operable catheter assembly described elsewhere herein. Conventional endoscopes are complex in design and can typically be designed for reuse after a procedure, which requires thorough cleaning, disinfection, or sterilization after each procedure. In many cases, existing endoscopes are designed with a complex structure to ensure that the endoscope can withstand the cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope can be a single-use endoscope that can advantageously reduce cross-contamination between patients and infectious diseases. In some examples, the robotic endoscope can be sent to the physician in a pre-sterilized package and is intended to be disposed of after single use.

[0097]

[0111] As shown in FIG. 15, the robotic bronchoscope 1510 may include a handle portion 1513 and a flexible elongated member 1511. In some embodiments, the flexible elongated member 1111 may include a shaft, an operable tip, and an operable / active bending section. The robotic bronchoscope 1510 can be the same as the operable catheter assembly described in FIG. 14. The robotic bronchoscope can be a single-use robotic endoscope. In some examples, only the catheter can be disposable. In some examples, at least a portion of the catheter can be disposable. In some examples, the entire robotic bronchoscope can be released from the instrument drive mechanism and disposed of. In some examples, the bronchoscope can include various levels of stiffness along the shaft to improve its functional operation. In some examples, the minimum bending radius along the shaft can vary.

[0098]

[0112] The robotic bronchoscope can be releasably coupled to an instrument drive mechanism 1520. The instrument drive mechanism 1520 can be mounted on an arm of the robotic support system or any actuated support system described elsewhere in this specification. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic bronchoscope 1510. The mechanical interface enables the robotic bronchoscope 1510 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via quick installation / release means such as magnets and spring levels. In some examples, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without using tools.

[0099]

[0113] FIG. 16 shows an example of an instrument drive mechanism 1620 that provides a mechanical interface to the handle portion 1613 of a robotic bronchoscope. As shown in this example, the instrument drive mechanism 1620 can include a motor set that is actuated to rotatably drive a pull wire set of a flexible endoscope or catheter. The handle portion 1613 of the catheter assembly can be mounted on the instrument drive mechanism such that its pulley assembly or capstan is driven by the motor set. The number of pulleys can vary based on the configuration of the pull wires. In some examples, one, two, three, four, or more pull wires can be used to bend the flexible endoscope or catheter like a joint.

[0100]

[0114] The handle portion can be designed to make the robotic bronchoscope low-cost and disposable. For example, conventional manual and robotic bronchoscopes can have a cable at the proximal end of the bronchoscope handle. Often, the cable can include illumination fibers, a camera video cable, and other sensor fibers or cables such as electromagnetic (EM) sensors or shape sensing fibers. Such complex cables are expensive and can increase the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design such that it can utilize simplified structures and components while maintaining mechanical and electrical functionality. In some examples, the handle portion of the robotic bronchoscope can use a cableless design while providing a mechanical / electrical interface to the catheter.

[0101]

[0115] Figure 17 shows an example of the distal tip 1700 of an endoscope. In some examples, the distal portion or tip of the catheter 1700 is substantially flexible and can be manipulated in one or more directions (e.g., pitch, yaw). The catheter can include a tip portion, a bending portion, and an insertion shaft. In some embodiments, the catheter can have a variable bending stiffness along the longitudinal axis. For example, the catheter can include a plurality of portions having different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). To vary the bending stiffness, one can select materials having different stiffness / rigidity, change the structure in different segments (e.g., cuts, patterns), add additional support components, or any combination thereof. In some embodiments, the catheter can have a variable minimum bending radius along the longitudinal axis. By selecting different minimum bending radii at different positions along the catheter, it is possible to advantageously provide a dropout prevention function while enabling the catheter to reach regions that are difficult to reach. In some examples, since the proximal end of the catheter does not need to be highly bent, the proximal portion of the catheter can be reinforced with an additional mechanical structure (e.g., an additional material layer) to achieve a greater bending stiffness. Such a design can provide support and stability to the catheter. In some examples, variable bending stiffness can be achieved by using different materials during the extrusion of the catheter. This can advantageously achieve different stiffness levels along the shaft of the catheter in an extrusion manufacturing process without the need to additionally attach or assemble different materials.

[0102]

[0116] The distal portion of the catheter can be manipulated by one or more pull wires 1705. The distal portion of the catheter can be made of any suitable material such as a copolymer, polymer, metal, or alloy so that it can be bent with the pull wires. In some embodiments, the proximal end or proximal termination of one or more pull wires 1705 can be coupled to a drive mechanism (e.g., gears, pulleys, capstans, etc.) via a fixing mechanism as described above.

[0103]

[0117] The pull wire 1705 can be a metal wire, cable, or thread, or a polymer wire, cable, or thread. The pull wire 1705 can also be made of natural or organic materials or fibers. The pull wire 1705 can be any suitable type of wire, cable, or thread that can support various types of loads without deformation, significant deformation, or damage. The distal end or portion of one or more pull wires 1705 is fixed or integrated with the distal portion of the catheter such that when the control unit operates the pull wire, a force or tension is applied to the distal portion, enabling the operation of at least the distal portion (e.g., the flexible portion) of the catheter or bending it like a joint (e.g., upward, downward, pitch, yaw, or in any direction between them).

[0104]

[0118] The catheter can have dimensions such that one or more electronic components can be integrated into the catheter. For example, since the outer diameter of the distal tip is about 4 - 4.4 millimeters (mm) and the diameter of the working channel is about 2 mm, one or more electronic components can be embedded in the wall of the catheter. However, it should be noted that based on different applications, the outer diameter can be within any range smaller than 4 mm or larger than 4.4 mm, and the diameter of the working channel can be within any range according to the tool dimensions or specific applications.

[0105]

[0119] One or more electronic components may include an imaging device, a lighting device, or a sensor. In some embodiments, the imaging device may be a video camera 1713. The imaging device may include optical elements and an image sensor for capturing image data. The image sensor may be configured to generate image data according to the wavelength of light. Various image sensors such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) can be used to capture image data. The imaging device may be a low-cost camera. In some examples, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include a plurality of electronic elements for processing image signals. For example, the circuit of a CCD sensor may include an A / D converter and an amplifier for amplifying and converting the analog signal provided by the CCD sensor. Optionally, the image sensor can be integrated with an amplifier and a converter to convert the analog signal into a digital signal without requiring a circuit board. In some examples, the output of the image sensor or the circuit board is image data (digital signal), which can be further processed by a camera circuit or a processor of the camera. In some examples, the image sensor may include an array of optical sensors.

[0106]

[0120] The lighting device may include one or more light sources 1711 positioned at the distal tip. The light source may be a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some examples, the light source may be a miniaturized LED or a dual-tone flash LED illumination for a compact design.

[0107]

[0121] The imaging device and the lighting device can be integrated into the catheter. For example, the distal portion of the catheter may include a suitable structure that at least matches the dimensions of the imaging device and the lighting device. The imaging device and the lighting device can be embedded in the catheter. FIG. 18 shows an exemplary distal portion of a catheter with an integrated imaging device and a lighting device. A camera can be positioned at the distal portion. The distal tip may have a structure for receiving the camera, the lighting device, and / or the position sensor. For example, the camera can be embedded within the cavity 1810 at the distal tip of the catheter. The cavity 1810 is integrally formed with the distal portion of the cavity and may have dimensions that match the length / width of the camera so that the camera does not move relative to the catheter. The camera can provide a close-range view of the tissue or organ adjacent to the working channel 1820 of the catheter. In some examples, the posture or orientation of the imaging device can be controlled by controlling the rotational movement (e.g., roll) of the catheter.

[0108]

[0122] Power for the camera can be provided by a wired cable. In some examples, the cable wire can be present within a wire bundle that provides power to the camera and lighting elements or other circuitry at the distal tip of the catheter. Power can be supplied to the camera and / or light source from a power source located in the handle portion via a wire, copper wire, or any other suitable means extending through the length of the catheter. In some examples, real-time images or videos of tissue or organs can be wirelessly transmitted to an external user interface or display. The wireless communication can be WiFi, Bluetooth, RF communication, or other forms of communication. In some examples, images or videos captured by the camera can be multicast to multiple devices or systems. In some examples, image and / or video data from the camera can be transmitted through the length of the catheter to a processor located in the handle portion via a wire, copper wire, or any other suitable means. The image or video data can be transmitted to an external device / system via a wireless communication component within the handle portion. In some examples, the system can be designed such that the wire is not visible to or not exposed to the operator.

[0109]

[0123] In a conventional endoscope, illumination light can be provided by a fiber optic cable that transmits the light of a light source positioned at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, a miniaturized LED light can be used and embedded in the distal portion of the catheter to reduce the complexity of the design. In some examples, the distal portion may include a structure 1430 having dimensions that match the dimensions of the miniaturized LED light source. As shown in the examples described, two cavities 1430 are integrally formed with the catheter and can receive two LED light sources. For example, since the outer diameter of the distal tip is about 4 to 4.4 millimeters (mm) and the diameter of the working channel of the catheter is about 2 mm, two LED light sources can be embedded at the distal end. The outer diameter can be within any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be within any range according to the dimensions of the tool or a particular application. Any number of light sources can be included. The internal structure of the distal portion can be designed to accommodate any number of light sources.

[0110]

[0124] In some examples, each of the LEDs can be connected to a power wire that can extend to the proximal handle. In some embodiments, the LEDs are soldered to separate power wires, and these wires can later be bundled to form a single twisted wire. In some embodiments, the LEDs can be soldered to a pull wire that supplies power. In other embodiments, the LEDs can be directly crimped or connected to a pair of power wires. In some examples, a protective layer such as a thin biocompatible adhesive layer can be applied to the front surface of the LED to provide protection while allowing light emission. In some examples, an additional cover 1431 can be disposed on the front end surface of the distal tip to provide precise positioning of the LED and sufficient spatial clearance for the adhesive. The cover 1831 can be composed of a transparent material that matches the refractive index of the adhesive so that the illumination light is not obstructed.

[0111]

[0125] Preferred embodiments of the present invention have been illustrated and described herein, but it will be apparent to those skilled in the art that such embodiments are provided for illustrative purposes only. Here, numerous variations, modifications, and substitutions will occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed when practicing the present invention. It is intended that the following claims define the scope of the present invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

**Claim 1** A method for navigating a robotic endoscope device, comprising: (a) navigating the robotic endoscope device to a target region within a body part; (b) when a tool is extended into the target region via the robotic endoscope device, acquiring one or more fluoroscopic images using a fluoroscopic imager and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having a first coordinate corresponding to the center of the target region in a depth direction and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; (d) determining whether the tool is within the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate with a threshold value. A method as described above. **Claim 2** The method according to claim 1, wherein the target region is a visible lesion in the 3D fluoroscopic image. **Claim 3** The method according to claim 2, wherein the first slice is identified by: (i) displaying the 3D fluoroscopic image within a graphical user interface (GUI); and (ii) selecting the first slice from a stack of slices when the lesion is in focus. **Claim 4** The method according to claim 3, wherein the second slice is identified when the tool is in focus. **Claim 5** The method according to claim 2, wherein the threshold value is determined based at least in part on the dimensions of the lesion. **Claim 6** The method according to claim 5, wherein the dimensions of the lesion are calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a). **Claim 7** The method according to claim 1, wherein the first slice or the second slice is automatically identified based on a sharpness measure or a contrast measure of each slice in the depth direction. **Claim 8** The method according to claim 2, further comprising: displaying the 3D fluoroscopic image within a graphical user interface (GUI); and displaying an overlay of the lesion on each slice from a plurality of stacks in the depth direction. **Claim 9** The method according to claim 8, wherein the overlay is generated based at least in part on a 3D model of the lesion that intersects each slice. **Claim 10** The method according to claim 8, further comprising determining whether the tool is within the target region by identifying whether the overlay of the lesion appears on the second slice.

11. The method according to claim 1, further comprising displaying, on a graphical user interface (GUI), the 3D fluoroscopic image, a first graphic visual indicator representing the first coordinate, and a second graphic visual indicator representing the second coordinate.

12. The method according to claim 1, wherein the 3D fluoroscopic image is reconstructed based on the pose of the fluoroscopic imager.

13. The method according to claim 12, wherein the pose of the fluoroscopic imager is estimated based on markers included in the acquired one or more fluoroscopic images.

14. The method according to claim 12, wherein the pose of the fluoroscopic imager is acquired based on position sensor data.

15. The threshold includes a margin, The method according to claim 1, wherein the margin is determined based on experimental data.

16. A non-transitory computer-readable storage medium including instructions, wherein the instructions, when executed by one or more processors, (a) navigate the robotic endoscope device to a target region within a body part; (b) when a tool is extended into the target region via the robotic endoscope device, acquire one or more fluoroscopic images using a fluoroscopic imager and reconstruct a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identify a first slice having a first coordinate corresponding to the center of the target region in a depth direction and a second slice having a second coordinate corresponding to the tool in the depth direction; (d) determine whether the tool is within the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate with a threshold; A non-transitory computer-readable storage medium that causes the one or more processors to perform operations including the above.

17. The non-transitory computer-readable storage medium according to claim 16, wherein the target region is a visible lesion in the 3D fluoroscopic image.

18. The first slice is identified by: i) displaying the 3D fluoroscopy image within a graphical user interface (GUI); ii) selecting the first slice from a stack of slices when the lesion is in focus; a non-transitory computer-readable storage medium according to claim 17.

19. The second slice is identified when the tool is in focus; a non-transitory computer-readable storage medium according to claim 18.

20. The threshold is determined based at least in part on the dimensions of the lesion; a non-transitory computer-readable storage medium according to claim 17.

21. The dimensions of the lesion are calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a); a non-transitory computer-readable storage medium according to claim 20.

22. The first slice or the second slice is automatically identified based on a sharpness measure or a contrast measure of each slice in the depth direction; a non-transitory computer-readable storage medium according to claim 16.

23. The operation further includes: displaying the 3D fluoroscopy image within a graphical user interface (GUI); and displaying an overlay of the lesion on each slice from a plurality of stacks in the depth direction; a non-transitory computer-readable storage medium according to claim 17.

24. The overlay is generated based at least in part on a 3D model of the lesion that intersects each slice; a non-transitory computer-readable storage medium according to claim 23.

25. The operation further includes determining whether the tool is within the target area by identifying whether the overlay of the lesion appears on the second slice; a non-transitory computer-readable storage medium according to claim 23.

26. The operation further includes: displaying, on a graphical user interface (GUI), the 3D fluoroscopy image, a first graphic visual indicator representing the first coordinate, and a second graphic visual indicator representing the second coordinate; a non-transitory computer-readable storage medium according to claim 16.

27. The non - transitory computer - readable storage medium according to claim 16, wherein the 3D fluoroscopic image is reconstructed based on the pose of the fluoroscopic imager.

28. The non - transitory computer - readable storage medium according to claim 27, wherein the pose of the fluoroscopic imager is estimated based on markers included in the acquired one or more fluoroscopic images.

29. The non - transitory computer - readable storage medium according to claim 27, wherein the pose of the fluoroscopic imager is acquired based on position sensor data.

30. The threshold value includes a margin, The non - transitory computer - readable storage medium according to claim 16, wherein the margin is determined based on experimental data.

Images