System and computer-readable storage medium for facilitating navigation of an anatomical lumen network

Abstract

A system and a computer-readable storage medium are provided for facilitating navigation of an anatomical lumen network of a patient. The system receives imaging data captured by an imaging device located at a distal end of an instrument when the distal end of the instrument is disposed within the anatomical lumen network, the anatomical lumen network including a right bronchus and a left bronchus; identifies one or more features in the imaging data representing a first disposition of the right bronchus and the left bronchus in the imaging data; accesses one or more features associated with a virtual image that is simulated from a viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network, the virtual lumen network including a virtual right bronchus and a virtual left bronchus, the features associated with the virtual image representing a second disposition of the virtual right bronchus and the virtual left bronchus in the virtual image; calculates a correspondence between the first disposition and the second disposition; and determines a roll of the distal end of the instrument within the anatomical lumen network based on the correspondence.

Description

[0001] This application is a divisional application of a Chinese patent application with an application date of June 6, 2018, an international application number of PCT / US2018 / 036329, an invention title of "Robotic System for Determining the Posture of a Medical Device in a Luminal Network", and an application number of 201880044512.4 when entering the Chinese national phase.

[0002] Cross - reference to related applications

[0003] This application claims the benefit of U.S. Provisional Patent Application 15 / 631,691, filed on June 23, 2017, entitled "AUTOMATICALLY - INITIALIZED ROBOTIC SYSTEMS FOR NAVIGATION OF LUMINAL NETWORKS", the entire content of which is hereby incorporated by reference in its entirety. Technical field

[0004] The systems and methods disclosed herein relate to medical procedures, and more particularly to navigation - assisted medical devices. Background art

[0005] Medical procedures such as endoscopy (e.g., bronchoscopy) can involve accessing and visualizing the interior of a patient's lumen (e.g., airway) for diagnostic and / or therapeutic purposes. For example, during the procedure, a flexible tubular tool such as an endoscope can be inserted into the patient's body, and instruments can pass through the endoscope to a tissue site identified for diagnosis and / or treatment.

[0006] Bronchoscopy is a medical procedure that allows a physician to examine the internal condition of a patient's lung airways, such as the bronchi and bronchioles. During this medical procedure, a thin, flexible tubular tool called a bronchoscope can be inserted into the patient's mouth and directed down the patient's larynx and into his / her lung airways toward a tissue site identified for subsequent diagnosis and treatment. The bronchoscope can have a lumen ("working channel") that provides access to the tissue site, and catheters and various medical tools can be inserted through the working channel to the tissue site. Summary of the invention

[0007] An endoscopic navigation system can use the fusion of different sensing modalities (e.g., video imaging data, electromagnetic (EM) position data, robotic position data, etc.) modeled probabilistically, for example, through adaptive adjustment. Probabilistic navigation methods or other navigation methods can depend on an initial estimate of "where" the end of the endoscope is - for example, an estimate of which airway, how deep into the airway, and how much roll there is in the airway - in order to start tracking the end of the endoscope. Some endoscopic techniques can involve a three-dimensional (3D) model of the patient's anatomy and can use an EM field and position sensors to guide the navigation. At the start of the procedure, the exact alignment (e.g., registration) between the virtual space of the 3D model, the physical space of the patient's anatomy represented by the 3D model, and the EM field may be unknown. As such, the position of the endoscope within the patient's anatomy cannot be precisely mapped to the corresponding location within the 3D model until registration is generated or in cases where the accuracy of an existing registration is in question.

[0008] Typically, the navigation system requires the physician to go through a series of initialization steps to generate this initial estimate. This can involve, for example, instructing the physician (e.g., by touching the main carina, the left carina, and the right carina) to position the bronchoscope at a plurality of specific positions and orientations relative to landmarks within the bronchial tree. Another option requires the physician to perform an initial airway inspection, for example, starting in the mid-trachea and entering each lung lobe while attempting to keep the end of the bronchoscope centered within each airway.

[0009] Such initialization steps can provide an initial estimate of the endoscope's position; however, this approach may have several potential drawbacks, including adding additional time requirements to the start of the procedure. Another potential drawback involves the fact that after initialization has been completed and tracking is occurring, adverse events (e.g., patient coughing, dynamic airway collapse) may create uncertainty about the actual position of the endoscope. This may require determining a new "initial" position, and thus the navigation system may require the physician to navigate back to the trachea to re-perform the initialization steps. This backtracking adds additional time requirements, which can be particularly troublesome if an adverse event occurs after the endoscope has navigated through smaller peripheral airways towards the target site.

[0010] The above problems and others are solved by the endoluminal network navigation systems and techniques described herein. The disclosed techniques can generate a 3D model of a virtual endoluminal network representing a patient's anatomical endoluminal network and can determine multiple positions within the virtual endoluminal network for placement of a virtual camera device. The disclosed techniques can generate a virtual depth map representing distances between the inner surface of the virtual endoluminal network and the virtual camera device placed at the determined positions. Features can be extracted from these virtual depth maps, such as the inter-peak distance in the case of a virtual depth map representing an airway bifurcation, and the extracted features can be stored in association with the positions of the virtual camera device. During a medical procedure, the distal end of an endoscope can be provided with an imaging device, and the disclosed navigation techniques can generate a depth map based on image data received from the imaging device. The disclosed techniques can derive features from the generated depth map, calculate a correspondence between the extracted features and the features extracted from one of the virtual depth maps that are stored, and then use the associated virtual camera device position as an initial position of the distal end of the instrument. Advantageously, such techniques allow a probabilistic navigation system (or other navigation system) to obtain an initial estimate of the scope position without the above-described manual initialization step. Additionally, the disclosed techniques can be used throughout the procedure to improve registration, and in some embodiments, the disclosed techniques can provide an additional "initial estimate" after an adverse event without the need to navigate back to landmark anatomical features through the endoluminal network.

[0011] Accordingly, one aspect relates to a method for facilitating navigation of a patient's anatomical endoluminal network, the method being performed by a collection of one or more computing devices and comprising: receiving imaging data captured by an imaging device located at a distal end of an instrument, the distal end of the instrument being disposed within the anatomical endoluminal network; accessing virtual features derived from a virtual image that is simulated from the viewpoint of a virtual imaging device placed at a virtual position within a virtual endoluminal network representing the anatomical endoluminal network; calculating a correspondence between features derived from the imaging data and the virtual features derived from the virtual image; and determining a pose of the distal end of the instrument within the anatomical endoluminal network based on the virtual position associated with the virtual features.

[0012] In some embodiments, the method further comprises: generating a depth map based on the imaging data, wherein the virtual features are derived from a virtual depth map associated with the virtual image, and wherein calculating the correspondence is at least partially based on correlating one or more features of the depth map with one or more features of the virtual depth map.

[0013] In some embodiments, the method further comprises: generating a depth map by calculating, for each of a plurality of pixels of the imaging data, a depth value that represents an estimated distance between the imaging device and a tissue surface within the anatomical lumen network corresponding to the pixel; identifying a first pixel among the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map; calculating a first value representing the distance between the first pixel and the second pixel; wherein the virtual depth map comprises, for each of a plurality of virtual pixels, a virtual depth value that represents a virtual distance between the virtual imaging device and a portion of the virtual lumen network represented by the virtual pixel, and wherein accessing the virtual features derived from the virtual image comprises: accessing a second value representing the distance between the first depth criterion and the second depth criterion in the virtual depth map; and calculating a correspondence based on a comparison of the first value and the second value.

[0014] In some embodiments, the method further comprises: accessing a plurality of values representing the distance between the first depth criterion and the second depth criterion in a plurality of virtual depth maps, the plurality of virtual depth maps respectively representing different virtual localizations within the virtual lumen network; and calculating a correspondence based on a second value that more closely corresponds to the first value as compared to the other values among the plurality of values. In some embodiments, the anatomical lumen network comprises an airway, and the imaging data depicts a bifurcation of the airway, and the method further comprises: in each of the depth map and the virtual depth map, identifying one of the first depth criterion and the second depth criterion as the right bronchus; and determining a roll of the instrument based on an angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein the posture of the distal end of the instrument within the anatomical lumen network comprises the determined roll.

[0015] In some embodiments, the method further comprises: identifying three or more depth criteria in each of the depth map and the virtual depth map; determining the shape and location of a polygon connecting the depth criteria in each of the depth map and the virtual depth map; and calculating a correspondence based on a comparison of the shape and location of the polygon in the depth map with the shape and location of the polygon in the virtual depth map. In some embodiments, generating the depth map is based on photoclinometry.

[0016] In some embodiments, the method further includes: calculating a probabilistic state of the instrument within the anatomical lumen network based on a plurality of inputs including position; and guiding navigation of the instrument through the anatomical lumen network at least in part based on the probabilistic state. In some embodiments, the method further includes: initializing a navigation system configured to calculate the probabilistic state and to guide navigation of the anatomical lumen network based on the probabilistic state, wherein initializing the navigation system includes: setting a prior of a probability calculator based on position. In some embodiments, the method further includes: receiving additional data representing an updated pose of the distal end of the instrument; setting a likelihood function of the probability calculator based on the additional data; and using the probability calculator to determine the probabilistic state based on the prior and the likelihood function.

[0017] In some embodiments, the method further includes: providing a plurality of inputs to a navigation system configured to calculate a probabilistic state, a first input including a pose of the distal end of the instrument, and at least one additional input including one or both of: robotic position data from a robotic system moving the instrument, and data received from a position sensor at the distal end of the instrument; and calculating a probabilistic state of the instrument based on the first input and the at least one additional input.

[0018] In some embodiments, the method further includes: determining a registration between a coordinate system of a virtual lumen network and a coordinate system of an electromagnetic field generated around the anatomical lumen network, at least in part based on a pose of the distal end of the instrument within the anatomical lumen network determined according to the calculated correspondence. In some embodiments, determining position includes: determining a distance advanced by the distal end of the instrument within a section of the anatomical lumen network.

[0019] On the other hand, there is provided a system configured to facilitate navigation of a patient's anatomical lumen network, the system including: an imaging device located at a distal end of the instrument; at least one computer-readable memory storing executable instructions; and one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least perform the following operations: receiving imaging data captured by the imaging device with the distal end of the instrument disposed within the anatomical lumen network; accessing virtual features derived from a virtual image that is simulated from a viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network representing the anatomical lumen network; calculating a correspondence between features derived from the imaging data and the virtual features derived from the virtual image; and determining a pose of the distal end of the instrument relative to within the anatomical lumen network based on the virtual location associated with the virtual features.

[0020] In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: generate a depth map based on imaging data, wherein the virtual image represents the virtual depth map; and determine a correspondence based at least in part on correlating one or more features of the depth map with one or more features of the virtual depth map. In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: generate a depth map by computing, for each pixel of a plurality of pixels of the imaging data, a depth value that represents an estimated distance between the imaging device and a tissue surface within the anatomical lumen network corresponding to the pixel; identify a first pixel among the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map; compute a first value representing a distance between the first pixel and the second pixel; wherein the virtual depth map includes, for each virtual pixel of a plurality of virtual pixels, a virtual depth value that represents a virtual distance between the virtual imaging device and a portion of the virtual lumen network represented by the virtual pixel, and wherein a feature derived from the virtual image includes a second value representing a distance between the first depth criterion and the second depth criterion in the virtual depth map; and determine the correspondence based on a comparison of the first value and the second value.

[0021] In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: access a plurality of values representing a distance between a first depth criterion and a second depth criterion in a plurality of virtual depth maps, the plurality of virtual depth maps respectively representing different virtual positions within a plurality of virtual positions within the virtual lumen network; and compute a correspondence based on a second value that more closely corresponds to the first value compared to other values among the plurality of values, identifying the second value as the closest match to the first value among the plurality of values. In some embodiments, the anatomical lumen network includes an airway, and the imaging data depicts a bifurcation of the airway, and one or more processors are configured to execute instructions to cause the system to at least perform the following operations: in each of the depth map and the virtual depth map, identify one of the first depth criterion and the second depth criterion as the right bronchus; and determine a roll of the instrument based on an angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein a posture of a distal end of the instrument within the anatomical lumen network includes the determined roll.

[0022] In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: identify three or more depth criteria in each of a depth map and a virtual depth map; determine the shape and positioning of polygons that connect the three or more depth criteria in each of the depth map and the virtual depth map; and calculate a correspondence based on a comparison of the shape and positioning of the polygons in the depth map with the shape and positioning of the polygons in the virtual depth map. In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: generate a depth map based on photogrammetry.

[0023] In some embodiments, one or more processors are configured to communicate with a navigation system, and wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: use the navigation system to calculate a probabilistic state of an instrument within an anatomical lumen network, at least in part based on a plurality of inputs including position; and guide navigation of the instrument through the anatomical lumen network, at least in part based on the probabilistic state calculated by the navigation system. Some embodiments of the system further include a robotic system configured to guide movement of the instrument during navigation. In some embodiments, the plurality of inputs includes robotic position data received from the robotic system, and wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: use the navigation system to calculate the probabilistic state of the instrument, at least in part based on position and the robotic position data. Some embodiments of the system further include a position sensor located at a distal end of the instrument, the plurality of inputs includes data received from the position sensor, and wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: use the navigation system to calculate the probabilistic state of the instrument, at least in part based on position and the data received from the position sensor. In some embodiments, one or more processors are configured to execute instructions to cause the system to at least perform the following operations: determine a registration between a coordinate system of a virtual lumen network and a coordinate system of an electromagnetic field generated around the anatomical lumen network, at least in part based on position.

[0024] On the other hand, it relates to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause at least one computing device to perform at least the following operations: accessing a virtual three-dimensional model of the inner surface of a patient's anatomical lumen network; identifying a plurality of virtual positions within the virtual three-dimensional model; for each virtual position among the plurality of virtual positions within the virtual three-dimensional model: generating a virtual depth map that represents the virtual distance between a virtual imaging device disposed at the virtual position and a portion of the inner surface within the field of view of the virtual imaging device when disposed at the virtual position, and deriving at least one virtual feature from the virtual depth map; and generating a database that associates the plurality of virtual positions with at least one virtual feature derived from the corresponding virtual depth map.

[0025] In some embodiments, the instructions, when executed, cause at least one computing device to perform at least the following operations: providing the database to a navigation system configured to guide an instrument through the anatomical lumen network during a medical procedure. In some embodiments, the instructions, when executed, cause at least one computing device to perform at least the following operations: accessing data representing an imaging device disposed at the distal end of an instrument; identifying image capture parameters of the imaging device; and setting the virtual image capture parameters of the virtual imaging device to correspond to the image capture parameters of the imaging device.

[0026] In some embodiments, the instructions, when executed, cause at least one computing device to perform at least the following operations: generating a virtual depth map based on the virtual image capture parameters. In some embodiments, the image capture parameters include one or more of the following: field of view, lens distortion, focal length, and brightness shading.

[0027] In some embodiments, the instructions, when executed, cause at least one computing device to perform at least the following operations: for each virtual position among the plurality of virtual positions: identifying a first depth criterion and a second depth criterion in the virtual depth map, and calculating a value representing the distance between the first depth criterion and the second depth criterion; and creating a database by associating the plurality of virtual positions with the corresponding values.

[0028] In some embodiments, the instructions, when executed, cause at least one computing device to at least perform the following operations: for each of a plurality of virtual localizations: identify three or more depth criteria in a virtual depth map and determine the shape and location of a polygon connecting the three or more depth criteria; and create a database by associating the plurality of virtual localizations with the shape and location of the corresponding polygon. In some embodiments, the instructions, when executed, cause at least one computing device to at least perform the following operations: generate three-dimensional volume data from a series of two-dimensional images representing an anatomical lumen network of a patient; and form a virtual three-dimensional model of the inner surface of the anatomical lumen network from the three-dimensional volume data. In some embodiments, the instructions, when executed, cause at least one computing device to at least perform the following operations: control a computed tomography imaging system to capture a series of two-dimensional images. In some embodiments, the instructions, when executed, cause at least one computing device to at least perform the following operations: form a virtual three-dimensional model by applying volume segmentation to the three-dimensional volume data.

[0029] Another aspect relates to a method for facilitating navigation of a patient's anatomical lumen network, the method being performed by a collection of one or more computing devices and comprising: receiving a set of stereoscopic images representing the interior of the anatomical lumen network; generating a depth map based on the set of stereoscopic images; accessing virtual features derived from a virtual image that is simulated from the viewpoint of a virtual imaging device positioned at a location within a virtual lumen network; calculating a correspondence between features derived from the depth map and the virtual features derived from the virtual image; and determining the pose of the distal end of an instrument within the anatomical lumen network based on the virtual location associated with the virtual features.

[0030] In some embodiments, generating the set of stereoscopic images includes: positioning an imaging device at the distal end of an instrument located at a first location within the anatomical lumen network; capturing a first image of the interior of the anatomical lumen network while the imaging device is positioned at the first location; robotically controlling the imaging device to move a known distance to reach a second location within the anatomical lumen network; and capturing a second image of the interior of the anatomical lumen network while the imaging device is positioned at the second location. In some embodiments, robotically controlling the imaging device to move a known distance includes one or both of the following: retracting the imaging device and angling the imaging device.

[0031] In addition, there is an aspect related to a system configured to facilitate navigation of a patient's anatomical lumen network, the system comprising: an imaging device located at a distal end of an instrument; at least one computer-readable memory having executable instructions stored thereon; and one or more processors communicatively coupled to the at least one computer-readable memory and configured to execute the instructions to cause the system to at least perform the following operations: receive imaging data captured by the imaging device with the distal end of the instrument disposed within the anatomical lumen network, wherein the anatomical lumen network includes a right bronchus and a left bronchus; identify one or more features from the imaging data, the one or more features identified from the imaging data representing a first disposition of the right bronchus and the left bronchus in the imaging data; access one or more features associated with a virtual image, the virtual image being simulated from a viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network representing the anatomical lumen network, the virtual lumen network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second disposition of the virtual right bronchus and the virtual left bronchus in the virtual image; calculate a correspondence between the first disposition and the second disposition; and determine a roll of the distal end of the instrument within the anatomical lumen network based on the calculated correspondence.

[0032] Moreover, there is an aspect related to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause at least one computing device to at least perform the following operations: receive imaging data captured by an imaging device with the distal end of an instrument disposed within an anatomical lumen network, wherein the anatomical lumen network includes a right bronchus and a left bronchus; identify one or more features from the imaging data, the one or more features identified from the imaging data representing a first disposition of the right bronchus and the left bronchus in the imaging data; access one or more features associated with a virtual image, the virtual image being simulated from a viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network representing the anatomical lumen network, the virtual lumen network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second disposition of the virtual right bronchus and the virtual left bronchus in the virtual image; calculate a correspondence between the first disposition and the second disposition; and determine a roll of the distal end of the instrument within the anatomical lumen network based on the calculated correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The disclosed aspects will be described below in connection with the accompanying drawings and appendices, which are provided for illustration and not limitation of the disclosed aspects, where like reference numerals represent like elements.

[0034] Figure 1 An embodiment of a cart-based robotic system arranged for diagnostic and / or therapeutic bronchoscopy procedures is shown.

[0035] Figure 2 Depicts Figure 1 additional aspects of the robotic system.

[0036] Figure 3 Shows an Figure 1 embodiment of the robotic system arranged for ureteroscopy.

[0037] Figure 4 Shows an Figure 1 embodiment of the robotic system arranged for vascular procedures.

[0038] Figure 5 Shows an embodiment of a table-based robotic system arranged for bronchoscopy procedures.

[0039] Figure 6 Provides Figure 5 an alternative view of the robotic system.

[0040] Figure 7 Shows an example system configured to retract and extend a robotic arm.

[0041] Figure 8 Shows an embodiment of a table-based robotic system configured for ureteroscopy procedures.

[0042] Figure 9 Shows an embodiment of a table-based robotic system configured for laparoscopic procedures.

[0043] Figure 10 Shows an Figures 5 to 9 embodiment of a table-based robotic system having pitch or tilt adjustment.

[0044] Figure 11 Provides Figures 5 to 10 a detailed illustration of the interface between the table and the column of the table-based robotic system.

[0045] Figure 12 Shows an exemplary instrument driver.

[0046] Figure 13 Shows an exemplary medical device having paired instrument drivers.

[0047] Figure 14 Shows an alternative design of an instrument driver and an instrument, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0048] Figure 15 Depicts a block diagram showing a positioning system according to an example embodiment, the positioning system estimating Figures 1 to 10 the positioning of one or more elements of a robotic system, such as Figures 13 to 14 the positioning of an instrument.

[0049] Figure 16A Shows an example operating environment implementing the disclosed navigation systems and techniques.

[0050] Figure 16B Shows an example endoluminal network navigating in an Figure 16A environment.

[0051] Figure 16C Shows an example robotic arm for guiding the movement of an instrument through an Figure 16B endoluminal network.

[0052] Figure 17 Shows an example command console for an example medical robotic system according to one embodiment.

[0053] Figure 18 Shows an example endoscope having imaging and EM sensing capabilities as described herein.

[0054] Figure 19 Depicts a schematic block diagram of a navigation system as described herein.

[0055] Figure 20 Depicts a flowchart of an example process for generating an extracted virtual feature dataset.

[0056] Figure 21 Depicts a flowchart of an example intraoperative process for generating depth information based on a correspondence between captured endoscope images and features of the calculated depth information and an Figure 20 extracted virtual feature dataset. DETAILED DESCRIPTION

[0057] 1. Overview

[0058] Aspects of the present disclosure can be integrated into a robot-enabled medical system capable of performing various medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy. During an endoscopy procedure, the system can perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0059] In addition to performing a wide range of procedures, the system can provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system can provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm movements and positioning. Further still, the system can provide the physician with the ability to perform procedures with improved ease of use such that one or more of the instruments of the system can be controlled by a single user.

[0060] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other implementations of the disclosed concepts are possible and that various advantages can be achieved using the disclosed implementations. Headings are included herein for reference and to assist in locating the various sections. These headings are not intended to limit the scope of the concepts described under the headings. These concepts can have applicability throughout the specification.

[0061] A. Robot System - Cart

[0062] Depending on the specific procedure, a robot-enabled medical system can be configured in various ways. Figure 1 An embodiment of a cart-based robot-enabled system 10 arranged for diagnostic and / or therapeutic bronchoscopy procedures is shown. During bronchoscopy, the system 10 can include a cart 11 having one or more robotic arms 12 to deliver a medical device (e.g., a steerable endoscope 13, which can be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., in this example, the mouth of a patient positioned on a table) to deliver diagnostic and / or therapeutic tools. As shown, the cart 11 can be positioned near the upper torso of the patient to provide access to the entry point. Similarly, the robotic arm 12 can be actuated to position the bronchoscope relative to the entry point. When performing a GI procedure using a gastroscope, a procedure-specific endoscope for gastrointestinal (GI) procedures, the arrangement in Figure 1 can also be utilized. Figure 2 An example embodiment of the cart is depicted in more detail.

[0063] Continuing to refer to Figure 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient automatically, manually, or in combination thereof. As shown, the steerable endoscope 13 can include at least two telescoping portions, such as an inner guide member portion and an outer sheath portion, each portion coupled to a separate instrument driver from the set of instrument drivers 28, each instrument driver coupled to the distal end of a separate robotic arm. This linear arrangement of the instrument drivers 28 that facilitates coaxial alignment of the guide member portion with the sheath portion creates a "virtual track" 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual track described herein is depicted using dashed lines in the figure, and thus the dashed lines do not depict any physical structure of the system. Translation of the instrument drivers 28 along the virtual track 29 causes the inner guide member portion to telescope relative to the outer sheath portion, or causes the endoscope 13 to advance or retract relative to the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on the clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual track 29 shown represents a compromise between providing the physician access to the endoscope 13 while minimizing the friction created by bending the endoscope 13 into the patient's mouth.

[0064] After insertion, the endoscope 13 can be guided along the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide member portion from the outer sheath portion to obtain enhanced engagement and a greater bend radius. Using separate instrument drivers 28 also allows the guide member portion and the sheath portion to be driven independently of each other.

[0065] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as a lesion or nodule within the patient's lung. The needle can be deployed along the working channel to obtain a tissue sample to be analyzed by a pathologist, where the working channel extends along the length of the endoscope. Depending on the pathology results, additional tools can be deployed along the working channel of the endoscope for additional biopsies. After the nodule is identified as malignant, the endoscope 13 can deliver tools endoscopically to excise the potential cancerous tissue. In some cases, the diagnostic and therapeutic procedures may need to be delivered in separate processes. In those cases, the endoscope 13 can also be used to deliver fiducials to also "mark" the location of the target nodule. In other cases, the diagnostic and therapeutic procedures can be delivered during the same process.

[0066] System 10 may also include a movable tower 30 that may be connected to the cart 11 via support cables to provide support for control, electronics, fluidics, optics, sensors, and / or power to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that can be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, the division of functionality between the cart / table and the support tower 30 reduces clutter in the operating room and facilitates improved clinical workflows. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location so as not to be in the way during the procedure.

[0067] To support the above-described robotic system, the tower 30 may include (one or more) components of a computer-based control system that store computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, etc. Execution of these instructions, whether occurring in the tower 30 or in the cart 11, may control the entire system or its (one or more) subsystems. For example, when executed by a processor of the computer system, the instructions may cause components of the robotic system to actuate associated brackets and arm mounts, actuate the robotic arms, and control medical devices. For example, in response to receiving a control signal, motors in the joints of the robotic arm may position the arm in a particular pose.

[0068] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid inlets to provide controlled irrigation and aspiration capabilities to systems that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via (one or more) separate cables.

[0069] The tower 30 may include voltage and surge protectors that are designed to provide filtered and protected power to the cart 11, thereby obviating the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more movable cart 11.

[0070] The tower 30 may also include support devices for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic devices for detecting, receiving, and processing data received from optical sensors or imaging devices throughout the robotic system 10. In combination with the control system, such optoelectronic devices may be used to generate real-time images for display on any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include an electronic subsystem for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position an EM field generator for detection by EM sensors in or on medical devices.

[0071] In addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart), Tower 30 may also include Console 31. Console 31 may include a user interface and a display for a physician operator, such as a touch screen. Consoles in System 10 are generally designed to provide robotic control as well as pre-operative and real-time information of the procedure, e.g., navigation and positioning information of Endoscope 13. When Console 31 is not the only console available to the physician, Console 31 may be used by a second operator, such as a nurse, to monitor the patient's health or vital signs and the operation of the system, as well as to provide procedure-specific data, e.g., navigation and positioning information.

[0072] Tower 30 may be coupled to Cart 11 and Endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from Tower 30 may be provided to Cart 11 via a single cable, thus simplifying the operating room and keeping it uncluttered. In other embodiments, specific functions may be coupled in separate cables and connections. For example, although power may be provided to the cart via a single power cable, support for control, optics, fluidics, and / or navigation may also be provided via separate cables.

[0073] Figure 2 Provided is Figure 1 A detailed illustration of an embodiment of a cart of the cart-based robotic enabling system shown. Cart 11 generally includes an elongate support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of column 14. Column 14 may include one or more brackets, such as bracket 17 (alternatively an "armrest"), for supporting the deployment of one or more robotic arms 12 ( Figure 2 three are shown). Bracket 17 may include individually configurable arm mounts that rotate about a vertical axis to adjust the base of robotic arm 12 to obtain a better placement relative to the patient. Bracket 17 also includes a bracket interface 19 that allows bracket 17 to translate vertically along column 14.

[0074] Bracket interface 19 is connected to column 14 via a slot, such as slot 20, which is disposed on opposite sides of column 14 to guide the vertical translation of bracket 17. Slot 20 contains a vertical translation interface for positioning and holding the bracket at various vertical heights relative to cart base 15. The vertical translation of bracket 17 allows Cart 11 to adjust the reach of robotic arm 12 to accommodate various table heights, patient sizes, and physician preferences. Similarly, the individually configurable arm mounts on bracket 17 allow the robotic arm base 21 of robotic arm 12 to be angled in various configurations.

[0075] In some embodiments, the slot 20 can be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the vertical translation interface and the internal chamber of the column 14 during the vertical translation of the carriage 17. The slot cover can be deployed by a pair of spring reels located near the vertical top and bottom of the slot 20. The cover is coiled within the reel until it is deployed to extend and retract from its coiled state as the carriage 17 translates vertically up and down. When the carriage 17 translates towards the reel, the spring loading of the reel provides the force to retract the cover into the reel while also maintaining a tight seal when the carriage 17 translates away from the reel. The cover can be connected to the carriage 17 using, for example, brackets in the carriage interface 19 to ensure that the cover extends and retracts properly as the carriage 17 translates.

[0076] The column 14 can include mechanisms such as gears and motors internally, which are designed to use a vertically aligned lead screw to translate the carriage 17 in a mechanized manner in response to a control signal generated in response to a user input (e.g., an input from the console 16).

[0077] The robotic arm 12 generally can include a robotic arm base 21 and an end effector 22 separated by a series of links 23, which are connected by a series of joints 24, each joint including an independent actuator, and each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints, thus providing seven degrees of freedom. The multiple joints result in multiple degrees of freedom, thereby allowing for "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arms 12 to position their respective end effectors 22 at specific positions, orientations, and trajectories in space using different link positions and joint angles. This allows the system to position and guide a medical device from a desired point in space while allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create better access while avoiding arm collisions.

[0078] The cart base 15 balances the weight of the column 14, the carriage 17, and the arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart to move and / or be fixed. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart to be easily moved around the room before the procedure. After reaching the appropriate position, the casters 25 can be fixed using wheel locks to hold the cart 11 in place during the procedure.

[0079] The console 16 positioned at the vertical end of the column 14 allows both a user interface for receiving user input and a display screen (or a dual-purpose device, such as the touch screen 26) to provide both pre-operative and intra-operative data to a physician user. Potential pre-operative data on the touch screen 26 can include navigation and mapping data derived from pre-operative computed tomography (CT) scans, pre-operative planning, and / or notes from pre-operative patient interviews. Intra-operative data on the display can include optical information provided by the tool, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. The console 16 can be positioned and tilted to allow the physician to access the console from the side of the column 14 opposite the carriage 17. From this position, the physician can observe the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 for assisting in maneuvering and stabilizing the cart 11.

[0080] Figure 3 An embodiment of a robot-enabled system 10 configured for ureteroscopy is shown. During ureteroscopy, the cart 11 can be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to traverse the patient's urethra and ureter, to the lower abdominal region of the patient. In ureteroscopy, it is desirable for the ureteroscope 32 to be directly aligned with the patient's urethra to reduce friction and forces on sensitive anatomical structures in the area. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 to obtain a direct linear approach to the patient's urethra. From the foot of the table, the robotic arm 12 can insert the ureteroscope 32 directly through the urethra into the lower abdomen of the patient along a virtual track 33.

[0081] After insertion into the urethra, using control techniques similar to those in bronchoscopy, the ureteroscope 32 can be navigated into the bladder, ureter, and / or kidney for diagnostic and / or therapeutic applications. For example, the ureteroscope 32 can be guided into the ureter and kidney to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed along the working channel of the ureteroscope 32. After lithotripsy is completed, the resulting stone fragments can be removed using a basket deployed along the ureteroscope 32.

[0082] Figure 4An embodiment of a robotic enabling system similarly arranged for vascular surgery is shown. In vascular surgery, system 10 can be configured such that the cart 11 can deliver a medical device 34, such as a steerable catheter, to an access point in the femoral artery in a patient's leg. The femoral artery provides both a relatively large diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart (which simplifies navigation). As in the case of ureteroscopy, the cart 11 can be positioned towards the patient's leg and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 with direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 can be guided and inserted by translating the instrument driver 28. Alternatively, the cart can be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0083] B. Robot System - Stage

[0084] An embodiment of a robotic enabled medical system may also include a patient table. Including the table reduces the amount of capital equipment in the operating room by removing the cart, which allows for better access to the patient. Figure 5 An embodiment of such a robotic enabling system arranged for a bronchoscopy procedure is shown. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on the floor. Much like in a cart-based system, the end effector of the robotic arm 39 of system 36 includes an instrument driver 42, which is designed to manipulate an elongate medical device, such as Figure 5 the bronchoscope 40 in []. In fact, a C-arm for providing fluoroscopic imaging can be positioned above the patient's upper abdominal region by placing emitters and detectors around the table 38.

[0085] Figure 6An alternative view of system 36 without patients and medical devices is provided for discussion purposes. As shown, column 37 may include one or more brackets 43 shown as annular in system 36, and one or more robotic arms 39 may be based on the one or more brackets. The brackets 43 may translate along a vertical column interface 44 extending along the length of column 37 to provide different advantageous positions from which the robotic arms 39 may be positioned to reach a patient. The brackets 43 may rotate about column 37 using a mechanical motor located within column 37 to allow the robotic arms 39 to access multiple sides of the table 38, such as both sides of a patient. In embodiments having multiple brackets, the brackets may be separately positioned on the column and may translate and / or rotate independently of other brackets. Although the brackets 43 do not need to be around column 37 or even circular, the shown annular shape facilitates rotation of the brackets 43 about column 37 while maintaining structural balance. The rotation and translation of the brackets 43 allow the system to align medical devices such as endoscopes and laparoscopes to different entry points on a patient.

[0086] The arm 39 may be mounted on the bracket by a set of arm mounts 45 that include a series of joints that may rotate and / or extend telescopically individually, thereby providing additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the bracket 43 such that when the bracket 43 is rotated appropriately, the arm mounts 45 may be positioned on the same side of the table 38 (as Figure 6 shown), on the opposite side of the table 38 (as Figure 9 shown), or on adjacent sides of the table 38 (not shown).

[0087] Column 37 structurally provides support for table 38 and provides a path for the vertical translation of the brackets. Internally, column 37 may be equipped with a lead screw for guiding the vertical translation of the brackets and a motor for mechanizing the translation of the brackets based on the lead screw. Column 37 may also transmit power and control signals to the brackets 43 and the robotic arms 39 mounted on the brackets.

[0088] The table base 46 serves a similar function to the cart base 15 in the cart 11 shown in Figure 2 that is, it houses heavier components to balance the table / bed 38, column 37, brackets 43, and robotic arms 39. The table base 46 may also include rigid casters for providing stability during the procedure. In the case of being deployed from the bottom of the table base 46, the casters may extend in opposite directions on both sides of the base 46 and retract when the system 36 needs to be moved.

[0089] Continuing to refer to Figure 6, the system 36 may also include a tower (not shown) that divides the functionality of the system 36 between the console and the tower to reduce the form factor and volume of the console. As in the earlier disclosed embodiments, the tower may provide various support functions for the console, such as processing, computing, and control capabilities, power, fluid control, and / or optical and sensor processing. The tower may also be movable to be positioned away from the patient to improve the physician's access and keep the operating room uncluttered. Additionally, placing components in the tower may allow for more storage space in the console base for potential stowage of the robotic arm. The tower may also include a console that provides a user interface for user input (e.g., a keyboard and / or a pendant) and a display (or touchscreen) for pre-operative and intra-operative information (e.g., real-time imaging, navigation, and tracking information).

[0090] In some embodiments, the console base may stow and store the robotic arm when not in use. Figure 7 A system 47 for stowing a robotic arm in an embodiment of a console-based system is shown. In system 47, a carriage 48 may be vertically translated into a base 49 to stow the robotic arm 50, arm mount 51, and carriage 48 within the base 49. A base cover 52 may be translated and retracted to open to deploy the carriage 48, arm mount 51, and arm 50 around a column 53, and may be closed to stow and protect them when the carriage, arm mount, and arm are not in use. The base cover 52 may be sealed with a membrane 54 along the edges of the opening of the base cover to prevent dust and fluid from entering when closed.

[0091] Figure 8 An embodiment of a robot-enabled console-based system configured for a ureteroscopy procedure is shown. In ureteroscopy, the console 38 may include a rotating portion 55 for positioning the patient at an angle with respect to the column 37 and the console base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located under the patient's head) to position the bottom portion of the rotating portion 55 away from the column 37. For example, pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned over the patient's lower abdomen without competing for space with a column (not shown) under the console 38. By rotating a carriage 35 (not shown) around the column 37, the robotic arm 39 may insert a ureteroscope 56 directly into the patient's groin area along a virtual track 57 to reach the urethra. In ureteroscopy, stirrups 58 may also be secured to the rotating portion 55 of the console 38 to support the position of the patient's legs during the procedure and allow unobstructed access to the patient's groin area.

[0092] During laparoscopy, minimally invasive instruments (elongated in shape to fit the size of one or more incisions) can be inserted into the patient's anatomy through (one or more) small incisions in the patient's abdominal wall. After the patient's abdominal cavity is inflated, an instrument commonly referred to as a laparoscope can be guided to perform surgical tasks such as grasping, cutting, removing, suturing, etc. Figure 9 An embodiment of a robot-enabled table-based system configured for a laparoscopy procedure is shown. As Figure 9 shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 such that the laparoscope 59 can be positioned through the minimum incisions on both sides of the patient to reach his / her abdominal cavity using the arm mount 45.

[0093] To accommodate the laparoscopy procedure, the robot-enabled table system can also tilt the platform to a desired angle. Figure 10 An embodiment of a robot-enabled medical system with pitch or tilt adjustment is shown. As Figure 10 shown, the system 36 can accommodate tilting of the table 38 to position one part of the table at a greater distance from the ground than another part. Additionally, the arm mount 45 can be rotated to match the tilt such that the arms 39 maintain the same planar relationship with the table 38. To accommodate steeper angles, the column 37 can also include a telescoping section 60 that allows the vertical extension of the column 37 to prevent the table 38 from contacting the ground or colliding with the base 46.

[0094] Figure 11 A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 with multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-table interface, with each axis actuated by a separate motor 2, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment along one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2.

[0095] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position (i.e., a position where the patient's lower abdomen is at a higher position from the ground than the patient's upper abdomen) for lower abdominal surgery. The Trendelenburg position causes the patient's internal organs to slide towards his / her upper abdomen by gravity, thus emptying the abdominal cavity for the entry of minimally invasive tools and performing lower abdominal surgical procedures such as laparoscopic prostatectomy.

[0096] C. Instrument Driver and Interface

[0097] The end effector of the robotic arm of the system includes: (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument manipulator"), which incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device, which may not have any electromechanical components such as a motor. This dichotomy may be driven by the need to disinfect medical devices used in medical procedures and the inability to adequately disinfect expensive capital equipment due to the complex mechanical components and sensitive electronics of such equipment. Thus, the medical device can be designed to be disassembled, removed, and interchanged from the instrument driver (and thus from the system) for separate disinfection or disposal by a physician or the physician's staff. In contrast, the instrument driver does not need to be changed or disinfected and can be covered with a shroud for protection.

[0098] Figure 12 An example instrument driver is shown. The instrument driver 62 disposed at the distal end of the robotic arm includes one or more drive units 63, which are arranged with parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gearhead 65 for converting the rotation of the motor shaft into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (as Figure 12 shown, four) independent drive outputs. In operation, the control circuit 68 will receive control signals, transmit motor signals to the motor 66, compare the resulting motor speed measured by the encoder 67 with the desired speed, and modulate the motor signals to generate the desired torque.

[0099] For processes that require a sterile environment, the robotic system can incorporate a drive interface located between the instrument driver and the medical device, such as a sterile adapter connected to a sterile drape. The primary purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument while maintaining a physical separation between the drive shaft and the drive input and thus maintaining sterility. Accordingly, an example sterile adapter can include a series of rotational inputs and outputs designed to mate with the drive shaft of the instrument driver and the drive input on the instrument. A sterile drape composed of a thin, flexible material (e.g., transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment such as the instrument driver, robotic arm, and cart (in a cart-based system) or table (in a table-based system). Using this drape will allow the capital equipment to be positioned near the patient while still remaining in an area that does not require disinfection (i.e., a non-sterile area). On the other side of the sterile drape, the medical device can dock with the patient in an area that requires sterilization (i.e., a sterile area).

[0100] D. Medical Device

[0101] Figure 13 An example medical device with a mating instrument driver is shown. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongate shaft 71 (or elongate body) and an instrument base 72. The instrument base 72, which is also commonly referred to as an "instrument handle" due to its intended design for manual interaction by a physician, typically can include a rotatable drive input 73 such as a socket, pulley, or spool, which is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mated drive input 73 of the instrument base 72 can share a rotational axis with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 can include splines that are designed to mate with a socket on the drive input 73.

[0102] The elongate shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., in endoscopy) or through a minimally invasive incision (e.g., in laparoscopy). The elongate shaft 66 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a custom combination including both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongate shaft can be connected to an end effector that includes a wrist formed by a U-shaped clip having a rotational axis and a surgical tool (e.g., a grasper or scissors), and the end effector can be actuated based on forces from the tendon portion when the drive input rotates in response to torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongate shaft can include a manipulable or controllable bendable portion that can be engaged and bent based on torque received from the drive output 74 of the instrument driver 75.

[0103] Tendons within the shaft 71 are used to transmit torque from the instrument driver 75 along the elongate shaft 71. These individual tendons (e.g., traction wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. The tendons are guided from the handle 72 along one or more traction lumens within the elongate shaft 71 and anchored at the distal portion of the elongate shaft 71. In laparoscopy, these tendons can be coupled to a distally mounted end effector, such as a wrist, grasper, or scissors. In such an arrangement, torque applied to the drive input 73 transfers tension to the tendons, thereby actuating the end effector in a certain manner. In laparoscopy, the tendons can cause the joint to rotate about an axis, thereby moving the end effector in one direction or the other. Alternatively, the tendons can be connected to one or more jaws of a grasper at the distal end of the elongate shaft 71, where tension from the tendons causes the grasper to close.

[0104] During endoscopy, the tendon portion can be coupled to a bending or articulating section disposed (e.g., at the distal end) along the elongate shaft 71 via an adhesive, a control ring, or other mechanical fasteners. When fixedly attached to the distal end of the bending section, the torque applied on the drive input 73 will be transmitted along the tendon portion, causing the softer bending section (sometimes referred to as the articulable section or region) to bend or articulate. Along the non-bending section, it would be advantageous to helically wind or spiral a separate traction lumen that guides a separate tendon portion along the wall of the endoscope shaft (or inside the wall of the endoscope shaft) to balance the radial forces caused by the tension in the traction wires. For a specific purpose, the spacing therebetween and / or the angle of the spiral can be varied or designed, where a tighter spiral exhibits less shaft compression under a load force, while a smaller amount of spiral causes greater shaft compression under a load force but also exhibits restricted bending. On the other hand, the traction lumen can be oriented parallel to the longitudinal axis of the elongate shaft 71 to allow for controlled articulation in the desired bending or articulable section.

[0105] During endoscopy, the elongate shaft 71 houses a plurality of components to assist in the robotic process. The shaft can include a working channel for deploying surgical tools, irrigation, and / or aspiration to the surgical area at the distal end of the shaft 71. The shaft 71 can also accommodate wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, where the optical assembly can include an optical imaging device. The shaft 71 can also accommodate optical fibers to transmit light from a light source (e.g., a light-emitting diode) located at the proximal end to the distal end of the shaft.

[0106] At the distal end of the instrument 70, the distal tip can also include an opening of a working channel for delivering tools for diagnosis and / or treatment, irrigation, and aspiration to the surgical site. The distal tip can also include a port for a camera device such as a fibroscope or a digital imaging device for capturing images of the internal anatomical space. Correlatively, the distal tip can also include a port for a light source for illuminating the anatomical space when using the camera device.

[0107] In Figure 13 an example, the drive shaft axis and thus the drive input axis are orthogonal to the axis of the elongate shaft. However, this arrangement complicates the rolling ability of the elongate shaft 71. When the tendon portion extends away from the drive input 73 and into the traction lumen within the elongate shaft 71, rolling the elongate shaft 71 along its axis while keeping the drive input 73 stationary can cause an undesired entanglement of the tendon portion. Such an entanglement of the tendon portion can disrupt any control algorithm designed to predict the movement of the flexible elongate shaft during the endoscopy procedure.

[0108] Figure 14An alternative design of the instrument driver and the instrument is shown, where the axis of the drive unit is parallel to the axis of the elongate shaft of the instrument. As shown, the circular instrument driver 80 includes four drive units, the drive outputs 81 of which are aligned in parallel at the end of the robotic arm 82. The drive units and their respective drive outputs 81 are housed in a rotating assembly 83 of the instrument driver 80 driven by one of the drive units within the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along a circular bearing that connects the rotating assembly 83 to the non-rotating portion 84 of the instrument driver. Electrical power and control signals can be transmitted from the non-rotating portion 84 of the instrument driver 80 to the rotating assembly 83 through electrical contacts, which can be maintained by the rotation of a brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can respond to a separate drive unit integrated into the non-rotating portion 84 and thus not be parallel to the other drive units. The rotating mechanism 83 allows the instrument driver 80 to rotate the drive units and their corresponding drive outputs 81 as a single unit about the instrument driver axis 85.

[0109] Similar to the earlier disclosed embodiments, the instrument 86 can include an elongate shaft portion 88 and an instrument base 87 (shown in transparent exterior for purposes of discussion), the instrument base 87 including a plurality of drive inputs 89 (e.g., sockets, pulleys, and spools) configured to receive the drive outputs 81 in the instrument driver 80. Different from the previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, where the axis is substantially parallel to the axis of the drive inputs 89, rather than being orthogonal as in Figure 13 the design.

[0110] When coupled to the rotating assembly 83 of the instrument driver 80, the medical instrument 86 including the instrument base 87 and the instrument shaft 88 rotates about the instrument driver axis 85 in combination with the rotating assembly 83. Since the instrument shaft 88 is positioned at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Thus, the rotation of the rotating assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. Additionally, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive inputs 89 in the instrument base 87 do not become entangled during rotation. Thus, the parallelism of the drive outputs 81, the drive inputs 89, and the axis of the instrument shaft 88 allows for shaft rotation without entangling any control tendons.

[0111] E. Navigation and Control

[0112] Traditional endoscopy can involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operating physician. In contrast, the robotic systems envisioned by the present disclosure can provide non-radiation-based means of navigation and positioning to reduce physician exposure to radiation and reduce the number of devices in the operating room. As used herein, the term "positioning" can refer to determining and / or monitoring the position of an object in a reference coordinate system. Techniques such as pre-operative mapping, computer vision, real-time EM tracking, and robotic command data can be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, pre-operative mapping, computer vision, real-time EM tracking, and robotic command data can be used alone or in combination to improve the information obtained solely by radiation-based imaging modalities.

[0113] Figure 15 is a block diagram of a positioning system 90 that illustrates the positioning (e.g., the positioning of an instrument) of one or more elements of an exemplary robotic system. The positioning system 90 can be a collection of one or more computer devices configured to execute one or more instructions. The computer devices can be implemented by one or more processors and computer-readable memories from among the one or more components discussed above. By way of example and not limitation, the computer devices can be in Figure 1 the tower 30 shown, Figures 1 to 4 the cart shown, Figures 5 to 10 the bed shown, etc.

[0114] As Figure 15 shown, the positioning system 90 can include a positioning module 95 that processes input data 91 to 94 to generate positioning data 96 for the distal end of a medical device. The positioning data 96 can be data or logic representing the positioning and / or orientation of the distal end of the instrument relative to a reference frame. The reference frame can be a reference frame relative to the patient's anatomy or relative to a known object - such as an EM field generator (see the discussion below regarding EM field generators).

[0115] The various input data 91 to 94 are now described in more detail. Pre-operative mapping can be accomplished by using a collection of low-dose CT scans. The pre-operative CT scans generate two-dimensional images, each two-dimensional image representing a "slice" of a cross-sectional view of the patient's internal anatomy. When analyzed as a whole, an image-based model of the anatomical cavities, spaces, and structures of the patient's anatomy (e.g., the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated based on the CT images to form a three-dimensional volume of the patient's anatomy, which is referred to as pre-operative model data 91. The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the entire content of which is incorporated herein by reference. A network topology model can also be derived from the CT images and is particularly suitable for bronchoscopy.

[0116] In some embodiments, the instrument can be equipped with a camera device to provide visual data 92. A positioning module 95 can process the visual data to achieve one or more vision-based positioning and tracking. For example, the pre-operative model data can be used in combination with the visual data 92 to achieve computer vision-based tracking of a medical device (e.g., an endoscope or an instrument advanced through the working channel of the endoscope). For example, using the pre-operative model data 91, a robotic system can generate a library of expected endoscope images based on the expected travel path of the endoscope according to the model, with each image linked to a location within the model. During the surgery, the robotic system can refer to this library to compare the real-time images captured at the camera device (e.g., the camera device at the distal end of the endoscope) with the images in the image library to assist in positioning.

[0117] Other computer vision-based tracking techniques use feature tracking to determine the movement of the camera device and thus the movement of the endoscope. Some features of the positioning module 95 can identify circular geometries in the pre-operative model data 91 corresponding to anatomical lumens and track changes in those geometries to determine which anatomical lumen has been selected, as well as the relative rotational and / or translational movement of the camera device. The use of a topological map can further enhance vision-based algorithms or techniques.

[0118] Optical flow - another computer vision-based technique - can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer the movement of the camera device. By utilizing multiple iterations of comparing multiple frames, the movement and positioning of the camera device (and thus the endoscope) can be determined.

[0119] The positioning module 95 can use real-time EM tracking to generate the real-time positioning of the endoscope in the global coordinate system, where the global coordinate system can be registered to the anatomical structure of the patient represented by the pre-operative model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils embedded with one or more positions and orientations in a medical device (e.g., an endoscopic tool) measures the change in the EM field generated by one or more static EM field generators placed at known positions. The positioning information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensors. The magnetic field induces a small current in the sensor coils of the EM sensor, and this small current can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "registered" to the patient's anatomical structure (e.g., the pre-operative model) during the operation to determine the geometric transformation that aligns the individual position in the coordinate system with the position in the pre-operative model of the patient's anatomical structure. Once registered, the embedded EM tracker at one or more positions of the medical device (e.g., the distal end of the endoscope) can provide a real-time indication of the advancement of the medical device through the patient's anatomical structure.

[0120] The positioning module 95 can also use the robotic commands and kinematic data 94 to provide positioning data 96 for the robotic system. The device pitch and yaw caused by the engagement commands can be determined during pre-operative calibration. During the operation, these calibration measurements can be used in combination with the known insertion depth information to estimate the position of the instrument. Alternatively, these calculations can be analyzed in combination with EM, vision, and / or topological modeling to estimate the position of the medical device within the network.

[0121] As Figure 15 shown, the positioning module 95 can use several other input data. For example, although not shown in Figure 15 , an instrument using shape-sensing optical fibers can provide shape data, and the positioning module 95 can use this shape data to determine the positioning and shape of the instrument.

[0122] The positioning module 95 can use the input data 91 to 94 in combination. In some cases, such a combination can use a probabilistic method, where the positioning module 95 assigns confidence weights to the positions determined according to each of the input data 91 to 94. Therefore, in cases where the EM data may be unreliable (such as in the presence of EM interference), the confidence in the position determined by the EM data 93 may be reduced, and the positioning module 95 may rely more on the visual data 92 and / or the robotic commands and kinematic data 94.

[0123] As discussed above, the robotic systems discussed herein can be designed to incorporate a combination of one or more of the above techniques. A computer-based control system of a robotic system located in a tower, bed, and / or cart can store computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, etc., where the computer program instructions, when executed, cause the system to receive and analyze sensor data and user commands, generate control signals for the overall system, and display navigation and positioning data, such as the position of an instrument within a global coordinate system, an anatomical map, etc.

[0124] 2. Introduction to Automatic Initialization Navigation System

[0125] Embodiments of the present disclosure relate to systems and techniques for facilitating navigation of a medical device through a lumen network, such as a pulmonary airway or other anatomical structure having internal open spaces, by generating and using depth information from endoscopic images to determine an initial endoscopic position, by analyzing multiple navigation-related data sources to increase the accuracy of estimating the position and orientation of the medical device within the lumen network, and by generating and using additional depth information to re-initialize the navigation system after an adverse event.

[0126] A bronchoscope can include a small camera device and a light source that allow a physician to examine a patient's trachea and airways. Patient trauma can occur if the precise positioning of the bronchoscope within the patient's airway is unknown. To determine the positioning of the bronchoscope, an image-based bronchoscopy guidance system can perform local registration (e.g., registration at a specific location within the lumen network) at the bifurcations of the patient's airway using data from the bronchoscope camera device and can thus advantageously be less susceptible to position errors caused by patient respiratory movement. However, since image-based guidance methods rely on bronchoscope video, image-based guidance methods can be affected by artifacts in the bronchoscope video caused by patient coughing or mucus obstruction, etc.

[0127] Electromagnetic navigation-guided bronchoscopy (EMN bronchoscopy) is a bronchoscopy procedure that implements EM technology to localize and guide an endoscopic tool or catheter through the bronchial passages of the lungs. An EMN bronchoscopy system can use an EM field generator that emits a low-intensity, varying EM field and establishes the position of a tracking volume around the patient's internal cavity network. An EM field is a physical field generated by charged objects that affects the behavior of charged objects near the field. EM sensors attached to objects located within the generated field can be used to track the localization and orientation of these objects within the EM field. A small current is induced in the EM sensors by the varying electromagnetic field. The characteristics of these electrical signals depend on the distance and angle between the sensor and the EM field generator. Thus, an EMN bronchoscopy system can include: an EM field generator; a manipulable medical device having an EM sensor at or near its distal end; and a guidance computing system. The EM field generator generates an EM field around the internal cavity network of the patient to be navigated, such as the airway, gastrointestinal tract, or circulatory pathway. The manipulable channel is inserted through the working channel of the bronchoscope and is tracked within the EM field via the EM sensor.

[0128] Prior to beginning an EMN bronchoscopy procedure, a virtual three-dimensional (3D) bronchial map of the patient's specific airway structure can be obtained, for example, from a preoperative CT chest scan. Using this map and the EMN bronchoscopy system, a physician can navigate to a desired location within the lung to perform a biopsy of a lesion, stage lymph nodes, insert a marker to guide radiation therapy, or guide a brachytherapy catheter. For example, registration can be performed at the beginning of the procedure to generate a mapping between the coordinate system of the EM field and the model coordinate system. Thus, when tracking the manipulable channel during bronchoscopy, based on the position data from the EM sensor, the position of the manipulable channel within the model coordinate system is nominally known.

[0129] As used herein, a coordinate system is a reference frame for a particular sensing modality. For example, for EM data, the EM coordinate system is the reference frame defined by the source of the EM field (e.g., the field generator). For CT images and segmented 3D models, this reference frame is based on the reference frame defined by the scanner. This navigation system solves the navigation problem of representing (registering) these different data sources (which are in their own reference frames) to a 3D model (i.e., the CT reference frame), for example, to display the localization of the instrument within the model.

[0130] Thus, as described in more detail below, the disclosed endoluminal network navigation systems and techniques can combine inputs from image-based navigation systems, robotic systems, and EM navigation systems, as well as inputs from other patient sensors, to mitigate navigation problems and achieve a more effective endoscopic procedure. For example, a navigation fusion framework can analyze image information received from an instrument camera device, position information from an EM sensor on the instrument tip, and robotic position information from a robotic system that guides the movement of the instrument. Based on this analysis, the navigation fusion framework can cause the instrument position estimation and / or navigation decision to be based on one or more of these types of navigation data. Some implementations of the navigation fusion framework can further determine the instrument position relative to a 3D model of the endoluminal network. In some embodiments, the initial instrument position for initializing tracking via the navigation fusion system can be generated based on depth information as described herein.

[0131] The disclosed systems and techniques can provide advantages for bronchoscopy guidance systems and other applications including other types of endoscopic procedures for endoluminal network navigation. In anatomy, an "endoluminal" can refer to the internal open space or cavity of an organ, such as an airway, blood vessel, kidney, heart, intestine, or any other suitable organ in which a medical procedure is being performed. As used herein, an "endoluminal network" refers to an anatomical structure having at least one endoluminal that leads to a target tissue site, e.g., the airway of the lung, the circulatory system, the renal calyces, and the gastrointestinal system. Thus, although the present disclosure provides examples of navigation systems related to bronchoscopy, it should be understood that the disclosed aspects of position estimation apply to other medical systems that navigate a patient's endoluminal network. Accordingly, the disclosed systems and techniques can be used with bronchoscopes, ureteroscopes, gastrointestinal endoscopes, and other suitable medical devices.

[0132] 3. Overview of Example Navigation System

[0133] Figure 16A An example operating environment 100 that implements one or more aspects of the disclosed navigation systems and techniques is shown. The operating environment 100 includes a patient 101, a platform 102 that supports the patient 101, a medical robotic system 110 that guides the movement of an endoscope 115, a command center 105 that controls the operation of the medical robotic system 110, an EM controller 135, an EM field generator 120, and EM sensors 125, 130. Figure 16A Also shown is an outline of a region of an endoluminal network 140 within the patient 101, which is shown in more detail in Figure 16B which.

[0134] The medical robotic system 110 can include one or more robotic arms for positioning and guiding the endoscope 115 through the lumen network 140 of the patient 101. The command center 105 can be communicatively coupled to the medical robotic system 110 for receiving position data and / or providing control signals from a user. As used herein, "communicatively coupled" refers to any wired data transmission medium and / or wireless data transmission medium, including but not limited to wireless wide area networks (WWANs) (e.g., one or more cellular networks), wireless local area networks (WLANs) (e.g., configured for standards such as IEEE 802.11 (Wi-Fi)), Bluetooth, data transmission cables, etc. The medical robotic system 110 can be any of the systems described above with respect to Figures 1 to 15 Any of the systems described above. Referring to Figure 16C The embodiments of the medical robotic system 110 are discussed in more detail, and referring to Figure 17 The command center 105 is discussed in more detail.

[0135] The endoscope 115 can be a tubular and flexible surgical instrument that is inserted into the patient's anatomy to capture images of the anatomy (e.g., body tissue) and provide a working channel for inserting other medical devices into the target tissue site. As described above, the endoscope 115 can be a procedure-specific endoscope, such as a bronchoscope, gastroscope, or ureteroscope, or can be a laparoscope or a steerable vascular catheter. The endoscope 115 can include one or more imaging devices (e.g., a camera device or other type of optical sensor) at its distal end. The imaging device can include one or more optical components, such as optical fibers, fiber optic arrays, photosensitive substrates, and / or lenses. The optical components move with the distal end of the endoscope 115 such that movement of the distal end of the endoscope 115 causes a corresponding change in the field of view of the image captured by the imaging device. The distal end of the endoscope 115 can be provided with one or more EM sensors 125 for tracking the position of the distal end in the EM field generated around the lumen network 140. The distal end of the endoscope 115 is further described below with reference to Figure 18 Any of the systems described above. Referring to

[0136] The EM controller 135 can control the EM field generator 120 to generate a varying EM field. Depending on the implementation, the EM field can be time-varying and / or space-varying. In some implementations, the EM field generator 120 can be an EM field generating plate. Some implementations of the disclosed patient navigation system can use an EM field generating plate located between the patient and the platform 102 supporting the patient, and the EM field generating plate can include a thin shield that minimizes any tracking distortion caused by conductive or magnetic materials located beneath it. In other implementations, the EM field generating plate can be mounted on a robotic arm similar to the robotic arm shown in the medical robotic system 110, which can provide flexible setup options around the patient.

[0137] The EM space measurement system incorporated in the command center 105, the medical robotic system 110, and / or the EM controller 135 can determine the positioning of an object with EM sensor coils, such as EM sensors 125, 130, embedded or placed within the EM field. When the EM sensors are placed in a controlled, varying EM field as described herein, a voltage is induced in the sensor coils. The EM space measurement system can use these induced voltages to calculate the position and orientation of the EM sensors, and thus the position and orientation of the object with the EM sensors. Since the magnetic field has a low field strength and can safely penetrate human tissue, the positioning measurement of the object can be performed without the line-of-sight limitations of an optical space measurement system.

[0138] The EM sensor 125 can be coupled to the distal end of the endoscope 115 to track its positioning within the EM field. The EM field is stationary relative to the EM field generator, and the coordinate system of the 3D model of the lumen network can be mapped to the coordinate system of the EM field. A number of additional EM sensors 130 can be placed on the patient's body surface (e.g., in the region of the lumen network 140) to assist in tracking the positioning of the EM sensor 125, for example, by enabling compensation for patient movement, including displacement caused by breathing. Many different EM sensors 130 can be spaced apart on the body surface.

[0139] Figure 16B It is shown that it can be in Figure 16AAn example lumen network 140 navigated within the operating environment 100. The lumen network 140 includes a branched structure of the patient's airway 150, a trachea 154 leading to the main carina 156 (the first bifurcation encountered during bronchoscopy navigation), and nodules (or lesions) 155 that can be accessed as described herein for diagnosis and / or treatment. As shown, the nodule 155 is located at the periphery of the airway 150. The endoscope 115 has a first diameter, and thus the distal end of the endoscope 115 cannot be positioned through the smaller diameter airway around the nodule 155. Accordingly, the steerable catheter 145 extends the remaining distance from the working channel of the endoscope 115 to the nodule 155. The steerable catheter 145 can have a lumen through which instruments such as biopsy needles, cytology brushes, and / or tissue sampling forceps can be passed to the target tissue site of the nodule 155. In such an implementation, the distal ends of both the endoscope 115 and the steerable catheter 145 can be provided with EM sensors for tracking their positions within the airway 150. In other embodiments, the overall diameter of the endoscope 115 can be small enough to reach the periphery without the steerable catheter 155, or can be small enough to approach the periphery (e.g., within 2.5 cm to 3 cm) to deploy medical devices through a non-steerable catheter. Medical devices deployed through the endoscope 115 can be equipped with EM sensors, and when such medical devices are deployed beyond the distal end of the endoscope 115, the position estimation techniques described below can be applied to such medical devices.

[0140] In some embodiments, a two-dimensional (2D) display of the 3D lumen network model or a cross-section of the 3D model as described herein can be similar to Figure 16B . The estimated position information can be overlaid onto such a representation.

[0141] Figure 16C An example robotic arm 175 of a medical robotic system 110 for guiding the movement of an instrument through Figure 16B the lumen network 140 is shown. In some embodiments the robotic arm 175 can be the robotic arms 12, 39 described above and is coupled to a base 180 which, in various embodiments, can be a cart base 15, a column 37 of the patient platform 38, or a ceiling-based mount. As described above, the robotic arm 175 includes a plurality of arm segments 170 coupled at joints 165 that provide the robotic arm 175 with multiple degrees of freedom.

[0142] The robotic arm 175 can be coupled to the instrument driver 190, such as the instrument driver 62 described above, using a mechanism converter interface (MCI) 160. The instrument driver 190 can be removed and replaced with a different type of instrument driver - for example, a first type of instrument driver configured to manipulate an endoscope or a second type of instrument driver configured to manipulate a laparoscope. The MCI 160 includes connectors for transmitting pneumatic pressure, electrical power, electrical signals, and optical signals from the robotic arm 175 to the instrument driver 190. The MCI 160 can be a fixed screw or a baseplate connector. The instrument driver 190 uses technologies including direct drive, harmonic drive, gear drive, pulley, and magnetic drive to manipulate a surgical instrument such as the endoscope 115. The MCI 160 can be interchanged based on the type of the instrument driver 190 and can be customized for a particular type of surgical procedure. The robotic arm 175 can include joint-level torque sensing and a wrist at the distal end.

[0143] The robotic arm 175 of the medical robotic system 110 can use the tendon as described above to manipulate the endoscope 115 to deflect the distal end of the endoscope 115. The endoscope 115 can exhibit non-linear behavior in response to the force applied by the elongate moving member. The non-linear behavior can be based on the stiffness and compressibility of the endoscope 115, as well as the variability of the slack or stiffness between different elongate moving members.

[0144] The base 180 can be positioned such that the robotic arm 175 can be brought close to perform or assist in a surgical procedure on a patient while a user, such as a physician, can control the medical robotic system 110 according to the comfort of the command console. The base 180 can be communicatively coupled to Figure 16A the command console 105 shown in

[0145] The base 180 can include a power supply 182, pneumatic pressure 186, and control and sensor electronics 184 - including components such as a central processing unit, a data bus, control circuits, and a memory - and associated actuators such as motors for moving the robotic arm 175. The electronics 184 can implement the navigation control techniques described herein. The electronics 184 in the base 180 can also process and transmit control signals transmitted from the command console. In some embodiments, the base 180 includes wheels 188 for transporting the medical robotic system 110 and wheel locks / brakes (not shown) for the wheels 188. The mobility of the medical robotic system 110 helps to accommodate space limitations in the surgical operating room and facilitates the proper placement and movement of surgical equipment. In addition, the mobility allows the robotic arm 175 to be configured such that the robotic arm 175 does not interfere with the patient, the physician, the anesthesiologist, or any other equipment. During the procedure, the user can use a control device, such as the command console, to control the robotic arm 175.

[0146] Figure 17 An example command console 200 is shown that can be used, for example, as the command console 105 in the example operating environment 100. The command console 200 includes a console base 201, a display module 202 such as a monitor, and a control module such as a keyboard 203 and a joystick 204. In some embodiments, one or more of the functions of the command console 200 can be integrated into the base 180 of the medical robot system 110 or into another system communicatively coupled to the medical robot system 110. A user 205, such as a physician, uses the command console 200 to remotely control the medical robot system 110 from an ergonomic position.

[0147] The console base 201 can include a central processing unit, a memory unit, a data bus, and associated data communication ports, which are responsible for interpreting and processing signals such as images from the imaging device and for tracking sensor data, for example, from Figures 16A to 16C the endoscope 115 as shown. In some embodiments, both the console base 201 and the base 180 perform signal processing to achieve load balancing. The console base 201 can also process commands and instructions provided by the user 205 through the control modules 203 and 204. In addition to Figure 17 the keyboard 203 and the joystick 204 as shown, the control module can also include other devices, such as a computer mouse, a trackpad, a trackball, a control board, a controller such as a handheld remote controller, and sensors that capture hand and finger gestures (e.g., motion sensors or imaging devices). The controller can include a set of user inputs (e.g., buttons, joysticks, pointing pads, etc.) mapped to operations of the instrument (e.g., engagement, actuation, water flushing, etc.).

[0148] The user 205 can use the command console 200 to control a surgical instrument such as the endoscope 115 in a speed mode or a position control mode. In the speed mode, the user 205 directly controls the pitch and yaw movements of the distal end of the endoscope 115 based on direct manual control using the control module. For example, movement on the joystick 204 can be mapped to the yaw and pitch movements of the distal end of the endoscope 115. The joystick 204 can provide haptic feedback to the user 205. For example, the joystick 204 can vibrate to indicate that the endoscope 115 cannot be further translated or rotated in a certain direction. The command console 200 can also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate that the endoscope 115 has reached its maximum translation or rotation.

[0149] In the position control mode, the command console 200 uses a 3D map of the patient's lumen network as described herein and input from the navigation sensor to control a surgical instrument, such as the endoscope 115. The command console 200 provides control signals to the robotic arm 175 of the medical robotic system 110 to maneuver the endoscope 115 to a target position. Since it relies on the 3D map, the position control mode may require an accurate mapping of the patient's anatomy.

[0150] In some embodiments, the user 205 may manually maneuver the robotic arm 175 of the medical robotic system 110 without using the command console 200. During setup in the surgical operating room, the user 205 may move the robotic arm 175, the endoscope 115 (or endoscopes), and other surgical devices to approach the patient. The medical robotic system 110 may rely on inertial control and force feedback from the user 205 to determine the appropriate configuration of the robotic arm 175 and the devices.

[0151] The display 202 may include an electronic monitor (e.g., an LCD monitor, an LED monitor, a touch-sensitive monitor), a virtual reality viewing device such as goggles or glasses, and / or other display devices. In some embodiments, the display module 202 is integrated with the control module, such as integrated as a tablet device with a touch screen. In some embodiments, one of the displays 202 may display a 3D model of the patient's lumen network and virtual navigation information (e.g., a virtual representation of the end of the endoscope within the model based on the EM sensor position), while the other of the displays 202 may display image information received from a camera device or another sensing device at the end of the endoscope 115. In some implementations, the user 205 may both view data and input commands to the medical robotic system 110 using the integrated display 202 and control module. The display 202 may display 3D images using a stereoscopic device such as goggles or glasses and / or display a 2D rendering of the 3D images. The 3D images provide an "internal view" (i.e., an endoscope view), which is a computer 3D model showing the patient's anatomy. The "internal view" provides a virtual environment inside the patient and the expected positioning of the endoscope 115 within the patient's body. The user 205 compares the "internal view" model with the actual images captured by the camera device to help mentally orient and confirm that the endoscope 115 is in the correct - or approximately correct - position within the patient's body. The "internal view" provides information about the anatomy around the distal end of the endoscope 115, such as the shape of the patient's airway, circulatory blood vessels, or intestine or colon. The display module 202 may simultaneously display a CT scan and a 3D model of the anatomy around the distal end of the endoscope 115. Additionally, the display module 202 may superimpose the determined navigation path of the endoscope 115 on the 3D model and / or the CT scan.

[0152] In some embodiments, a model of the endoscope 115 is displayed together with the 3D model to assist in indicating the status of the surgical procedure. For example, a CT scan identifies lesions in the anatomical structure that may require a biopsy. During the procedure, the display module 202 may show a reference image captured by the endoscope 115 corresponding to the current positioning of the endoscope 115. The display module 202 may automatically display different views of the model of the endoscope 115 according to user settings and the particular surgical procedure. For example, the display module 202 shows a top fluoroscopic view of the endoscope 115 as the endoscope 115 approaches the surgical area of the patient during the navigation step.

[0153] Figure 18 The distal end 300 of an exemplary endoscope (e.g., Figures 16A to 16C the endoscope 115) having imaging and EM sensing capabilities as described herein is shown. In Figure 18 it, the distal end 300 of the endoscope includes an imaging device 315, an illumination source 310, and the end of an EM sensor coil 305. The distal end 300 also includes an opening leading to the working channel 320 of the endoscope through which surgical instruments such as biopsy needles, cytology brushes, and forceps can be inserted along the endoscope axis, thereby allowing access to areas near the end of the endoscope.

[0154] The illumination source 310 provides light to illuminate a portion of the anatomical space. The illumination source may be one or more light-emitting devices each configured to emit light of a selected wavelength or wavelength range. The wavelength may be any suitable wavelength, such as visible spectrum light, infrared light, X-rays (e.g., for fluoroscopy), to name just a few examples. In some embodiments, the illumination source 310 may include a light-emitting diode (LED) located at the distal end 300. In some embodiments, the illumination source 310 may include one or more fiber optic cables that extend through the length of the endoscope to transmit light from a remote light source such as an x-ray generator through the distal end 300. In cases where the distal end 300 includes multiple illumination sources 310, these illumination sources 310 may each be configured to emit light of the same or different wavelengths from one another.

[0155] The imaging device 315 can include any photosensitive substrate or structure configured to convert the energy of received light into an electrical signal, such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) image sensor. Some examples of the imaging device 315 can include one or more optical fibers, such as a fiber bundle, configured to transmit light representative of an image from the distal end 300 of the endoscope to an eyepiece and / or an image sensor near the proximal end of the endoscope. The imaging device 315 can additionally include one or more lenses and / or wavelength-pass or cut-off filters as required by various optical designs. The light emitted from the illumination source 310 allows the imaging device 315 to capture an image inside the patient's internal cavity network. These images can then be transmitted as individual frames or a series of successive frames (e.g., video) to a computer system such as the command console 200 for processing as described herein.

[0156] The electromagnetic coil 305 located on the distal end 300 can be used with an electromagnetic tracking system to detect the position and orientation of the distal end 300 of the endoscope when the distal end 300 of the endoscope is disposed within the anatomical system. In some embodiments, the coil 305 can be angled to provide sensitivity to electromagnetic fields along different axes, thereby endowing the disclosed navigation system with the ability to measure the following complete six degrees of freedom: three degrees of freedom of position and three degrees of freedom of angle. In other embodiments, only a single coil can be disposed on or within the distal end 300, wherein the axis of the single coil is oriented along the endoscope axis of the endoscope. Due to the rotational symmetry of such a system, it is insensitive to rolling about its axis, and thus only five degrees of freedom can be detected in such an implementation.

[0157] Figure 19 A schematic block diagram of an example navigation fusion system 400 as described herein is shown. As described in more detail below, using the system 400, data from multiple different sources are combined and repeatedly analyzed during a surgical procedure to provide an estimate of the real-time movement information and the positioning / orientation information of a surgical instrument (e.g., an endoscope) within the patient's internal cavity network and to make navigation decisions.

[0158] The navigation fusion system 400 includes a plurality of data repositories, which include a depth feature data repository 405, an endoscope EM sensor data repository 415, a registration data repository 475, a model data repository 425, an endoscope imaging data repository 480, a navigation path data repository 445, and a robot position data repository 470. Although for clarity in the following discussion Figure 19are shown separately, but it will be understood that some or all of the data repositories may be stored together in a single memory or a group of memories. System 400 also includes a plurality of processing modules, which include a registration calculator 465, a depth-based position estimator 410, a localization calculator 430, an image analyzer 435, a state estimator 440, and a navigation controller 460. Each module may represent a set of computer-readable instructions stored in a memory, and one or more processors configured by the instructions to perform the features described below together. The navigation fusion system 400 may be implemented as, for example, one or more data storage devices and one or more hardware processors in the control and sensor electronics 184 and / or the console base 201 described above. In some implementations, the navigation fusion system 400 may be an implementation of the positioning system 90.

[0159] Figure 19 Also shown is a modeling system 420 in communication with the navigation fusion system 400. As described in more detail below, using the modeling system 420, data of a plurality of images representing the anatomical lumen network of a patient can be analyzed to establish a three-dimensional model of a virtual representation of the anatomical lumen network, and the virtual anatomical lumen network can be used to establish a depth feature data repository 405. Although shown separately, in some embodiments, the modeling system 420 and the navigation fusion system 400 may be combined into a single system. The modeling system 420 includes a plurality of processing modules, which include a model generator 440 and a feature extractor 450. Although the model data repository 425 and the depth feature data repository 405 are shown within the navigation fusion system 400, in some implementations, these data repositories may alternatively or additionally be located within the modeling system 420.

[0160] The model generator 440 is a module configured to receive data from a medical imaging system (not shown), such as a CT imaging system or a magnetic resonance imaging system. The received data may include a series of two-dimensional images representing the anatomical lumen network of a patient. The model generator 440 may generate three-dimensional volume data based on the series of two-dimensional images, and may form a virtual three-dimensional model of the inner surface of the anatomical lumen network based on the three-dimensional volume data. For example, the model generator may apply segmentation to identify the portions of the data corresponding to the tissue of the anatomical lumen network. In this way, the resulting model may represent the inner surface of the tissue of the anatomical lumen network.

[0161] The model data repository 425 is a data storage device that stores data representing a model of a patient's lumen network—e.g., a model generated by the model generator 440. Such a model can provide 3D information about the structure and connectivity of the lumen network, including, in some examples, the topography and / or diameter of the patient's airway. Some CT scans of the patient's lungs are performed while holding the breath, such that the patient's airway expands to its maximum diameter in the model.

[0162] The endoscopic imaging data repository 480 is a data storage device that stores image data received from an imaging device, such as the imaging device 315, at the distal end of an endoscope. In various embodiments, the image data can be discrete images or a series of image frames in a video sequence.

[0163] The feature extractor 450 is a module configured to receive a model from the model generator 440 and construct a database of depth features corresponding to a plurality of different localizations within the model. For example, the feature extractor 450 can identify a plurality of different localizations within the model, computationally place a virtual imaging device at each localization, generate a virtual image at each localization, and then obtain a specified feature based on the virtual image. A "virtual imaging device" as described herein is not a physical imaging device, but rather a computational simulation of an image capture device. The simulation can generate a virtual image based on virtual imaging device parameters, including field of view, lens distortion, focal length, and luminance shading, which in turn can be based on the parameters of an actual imaging device.

[0164] Each generated virtual image can correspond to a virtual depth map that represents the distance between the tissue of the virtual lumen network within the virtual field of view of the virtual imaging device and the localization of the virtual imaging device. The feature extractor 450 can match the virtual imaging device parameters with the parameters of an actual imaging device that has been identified for use in a medical procedure involving the patient's lumen network. A more detailed example process for constructing the database is described below with reference to Figure 20 more detail.

[0165] The feature extractor 450 can also receive data from the endoscopic imaging data repository 480, generate a depth map representing the distance between the endoscopic imaging device and the imaging tissue represented by the pixels of the image, and obtain features based on the generated depth map. In some embodiments, the feature extractor 450 can use photoclinometry (e.g., shape determination by shading) processing to generate a depth map based on a single image. In some embodiments, the feature extractor 450 can use a stereoscopic image set depicting the imaging region to generate a depth map.

[0166] The depth feature data repository 405 is a data storage device of a database that stores features obtained from depth maps and / or virtual depth maps generated by the feature extractor 450. The features can vary based on the nature of the lumen network and / or the use of the features during navigation. The features can include, for example, the positions of local maxima in the depth map (e.g., representing the farthest virtual tissue visible along a branch of the airway), the positions of the curve peaks along a curve around a local maximum, values representing the distance separating two local maxima (e.g., the number of pixels therebetween), and / or the size, shape, and orientation of a line or polygon connecting multiple local maxima. A curve peak represents a region in the depth map where the depth values of the pixels on one side of the curve peak increase while the depth values on the other side of the curve peak decrease. A curve peak can include a local maximum where the depth associated with a pixel is greater than the depth associated with the pixels on either side of the pixel. For each identified location, the depth feature data repository 405 can store the feature and the associated location within the virtual lumen network, for example, in the form {location n , feature value} as a tuple. As an example, when the location involves a position within the airway and the feature involves the distance between two identified local maxima, the tuple can be generated as {location n (airway segment, depth within the airway segment), feature value (distance)}. In this way, the extracted features in the database can be programmatically evaluated quickly in comparison to the features extracted in real time from the images of the navigated anatomical lumen network, and the location corresponding to the identified best or highly feature-matching can be quickly ascertained.

[0167] The depth-based position estimator 410 is a module configured to compare one or more features extracted in real time from the images of the anatomical lumen network with one or more pre-computed features extracted from the virtual images. The depth-based position estimator 410 can scan the depth feature data repository 405 to find a match between the virtual features and the features extracted from the real image, and can use the location corresponding to the match as the position of the instrument (e.g., an endoscope) within the anatomical lumen network. The match can be an exact match, the best match among the available features in the depth feature data repository 405, a match within a threshold difference from the extracted features. The depth-based position estimator 410 can output the position to the state estimator 440, for example, to be used as an initial position (“prior”) in the probabilistic assessment of the instrument position, or as a prior after an adverse event (e.g., a cough) occurs where the precise positioning of the instrument becomes unknown. The depth-based position estimator 410 can output the position to the registration calculator 465 for generating an initial registration and / or an updated registration between the model and the EM field set up around the patient.

[0168] The endoscopic EM sensor data repository 415 is a data storage device that stores data obtained from an EM sensor at the distal end of an endoscope. As described above, such sensors can include EM sensor 125 and EM sensor coil 305, and the data obtained can be used to identify the position and orientation of the sensor within the EM field. Similar to the data from the EM respiration sensor, the data from the endoscopic EM sensor can be stored as tuples in the form of (x, y, z, t n ), where x, y, and z represent the coordinates of the sensor within the EM field at time t n . Some embodiments may also include roll, pitch, and yaw of the instrument in the EM sensor tuple. The endoscopic EM sensor data repository 415 can store multiple such tuples corresponding to multiple different times for each endoscope-based sensor.

[0169] The registration calculator 465 is a module that can identify the registration or mapping between the coordinate system of a 3D model (e.g., the coordinate system of the CT scanner used to generate the model) and the coordinate system of the EM field (e.g., of the EM field generator 120). To track sensors throughout a patient's anatomy, the navigation fusion system 400 may require a process called "registration," through which the registration calculator 465 finds the geometric transformation that aligns a single object between different coordinate systems. For example, a particular anatomical site on a patient can have a representation in 3D model coordinates and can also have a representation in EM sensor coordinates. To calculate an initial registration, one implementation of the registration calculator 465 can perform the registration as described in the following document: U.S. Application No. 15 / 268,238, entitled "Navigation of Tubular Networks," filed on September 17, 2016, the disclosure of which is hereby incorporated by reference in its entirety. As an example of a possible registration technique, when an endoscope is inserted into a patient's airway, such as when the endoscope reaches various bifurcations, the registration calculator 465 can receive data from the endoscope image data repository 480 and the EM sensor data repository 415 at multiple different points. The image data can be used, for example, to identify when the distal end of the endoscope reaches a bifurcation via automatic feature analysis. When the endoscope is positioned at a bifurcation, the registration calculator 465 can receive data from the endoscope EM sensor data repository 415 and identify the positioning of the EM sensor at the distal end of the endoscope. Some examples can use not only bifurcations but also other points in the patient's airway and can map these points to corresponding points in a "skeleton" model of the airway. The registration calculator 465 can use data that links at least three EM positions to points in the model in order to identify the geometric transformation between the EM field and the model. Another implementation can involve manual registration, such as by taking at least 3 from the first bifurcation of the patient's airway and from two additional bifurcations in the left and right lungs, and corresponding points can be used to calculate the registration. The data used to perform the geometric transformation (also called registration data) can be stored as registration data in the registration data repository 475.

[0170] After determining the initial registration, the registration calculator 465 can update its estimate of the registration transformation based on the received data in order to increase the transformation accuracy and compensate for changes in the navigation system, such as changes due to patient movement. In some aspects, the registration calculator 465 can update the estimate of the registration transformation continuously, at defined intervals, and / or based on the position of the endoscope (or one or more of its components) within the lumen network.

[0171] The registration data repository 475 is a data storage device that stores registration data, which, as just discussed, can be used to perform a geometric transformation from the coordinate system of the EM field to the coordinate system of the model. As discussed above, in some implementations, the registration data can be generated by the registration calculator 465 and can be updated continuously or periodically.

[0172] The localization calculator 430 is a module that receives data from the model data repository 425, the registration data repository 475, and the scope position estimator 420 to convert EM sensor coordinates into 3D model coordinates. As described above, the scope position estimator 420 calculates the initial position of the EM sensor relative to the position of the EM field generator. This position also corresponds to a location within the 3D model. To convert the initial position of the EM sensor from the EM coordinate system to the model coordinate system, the localization calculator 430 can access the mapping (e.g., registration data) between the EM coordinate system and the model coordinate system stored in the registration data repository 475. To convert the scope position to the 3D model coordinate system, the localization calculator 430 receives, as inputs, data representing the topography of the 3D model from the model data repository 425, data representing the registration between the EM field and the coordinate system of the 3D model from the registration data repository 475, and the position of the scope in the EM field from the scope position estimator 420. Some implementations may also receive previously estimated state data from the state estimator 440. Based on the received data, the localization calculator 430 can perform an immediate transformation, for example, of the EM sensor position data to a position within the 3D model. This can represent a preliminary estimate of the position of the distal end of the scope within the topography of the 3D model and can be set as one of the inputs to the state estimator 440 for generating a final estimate of the scope position, as described in more detail below.

[0173] The image analyzer 435 is a module that receives data from the endoscopic imaging data repository 480 and the model data repository 425 and can compare the data to determine the endoscopic positioning. For example, the image analyzer 435 can access the intraluminal images of the volumetric rendering or surface rendering of the airway tree from the model scan and can compare the rendered images with the real-time images or video frames from the imaging device 315. For example, the images can be registered (e.g., using Powell's optimization, simplex or gradient methods, gradient descent algorithms with normalized cross-correlation or mutual information as the cost), and then the registered images obtained from the two sources can be compared using the weighted sum of normalized mutual information and squared difference error. The similarity between the 2D images from the scan and the 2D images received from the endoscope can indicate that the endoscope is located near the position of the image from the scan. Such image-based navigation can perform local registration at the bifurcations of the patient's airway and is thus less susceptible to noise caused by patient respiratory motion compared to the EM tracking system. However, since the image analyzer 435 relies on the endoscopic video, the analysis may be affected by artifacts in the images caused by patient coughing or mucus obstruction.

[0174] In some embodiments, the image analyzer 435 can implement object recognition techniques by which the image analyzer 435 can detect objects present in the field of view of the image data, such as branch openings, lesions, or particles. Using object recognition, the image analyzer can output object data indicating information about what objects are recognized and the position, orientation, and / or size of the objects represented as probabilities. As an example, object recognition can be used to detect objects that can indicate branch points in the lumen network and then determine their position, size, and / or orientation. In one embodiment, in a given image within the lumen network, each branch typically appears as a dark, approximately elliptical region, and these regions can be automatically detected as objects by the processor using region detection algorithms such as maximally stable extremal regions (MSER). The image analyzer 435 can use the light reflection intensity in combination with other techniques to identify the airway. In addition, the image analyzer 435 can also track the detected objects across a set of consecutive image frames to detect which branch of a set of possible branches within the lumen network has been entered.

[0175] The robotic position data repository 470 is a data storage device that stores robotic position data received from the medical robotic system 110 - such as data related to the physical movement of a medical device or a part of a medical device (e.g., the distal end or tip of the device) within the lumen network of the medical robotic system 110. Example robotic position data can include, for example, command data indicating that the tip of the device has reached a specific anatomical site and / or changed its orientation within the lumen network (e.g., for one or both of the guide and sheath of an endoscopic device, by a specific pitch, roll, yaw, insertion, and retraction), insertion data representing the insertion movement of a part of the medical device (e.g., the tip of the device or the sheath), instrument drive data, and mechanical data representing the mechanical movement of an elongate member of the medical device, such as the movement of one or more cables, tendons, or shafts of an endoscope that actuates the actual movement of the endoscope within the lumen network.

[0176] The navigation path data repository 445 is a data storage device that stores data representing a pre-planned navigation path through the lumen network to a target tissue site. Navigation to a specific point within the lumen network of a patient's body may require certain steps to be taken prior to surgery in order to generate the information needed to create a 3D model of the tubular network and determine the navigation path within it. As described above, a 3D model of the topography and structure of a specific patient's airway can be generated. A target can be selected, such as a lesion to be biopsied or a part of an organ tissue to be surgically repaired. In one embodiment, the user is able to select the location of the target by interfacing with a computer display capable of showing the 3D model (e.g., by clicking with a mouse or touching a touch screen). In some embodiments, the navigation path can be programmatically identified by analyzing the model and the identified lesion site to obtain the shortest navigation path to the lesion. In some embodiments, the path can be identified by a physician, or the automatically identified path can be modified by a physician. The navigation path can identify a series of branches within the lumen network to traverse in order to reach the identified target.

[0177] The state estimator 440 is a module that receives inputs and performs an analysis of the inputs to determine the state of the medical device. For example, the state estimator 440 may receive data as inputs from the depth-based position estimator 410, the localization calculator 430, the image analyzer 435, the navigation path data repository 445, and the robotic position data repository 470. Taking into account the provided inputs, the state estimator 440 may implement a probabilistic analysis to determine the state and corresponding probabilities of the medical device within the lumen network. The estimated state may refer to one or more of the following: (1) the x, y, z positions of the device relative to the coordinate system of the lumen network model; (2) whether the device is located in a certain region of the model, such as a specific airway branch; (3) the pitch, roll, yaw, insertion, and / or retraction of the device; and (4) the distance to the target. The state estimator 440 may provide the estimated state of the device (or the distal end of the device) as a function of time.

[0178] In some embodiments, the state estimator 440 may implement a Bayesian framework to determine the state and corresponding probabilities. Bayesian statistical analysis starts with a belief called a prior, and then updates that belief with the observed data. The prior represents an estimate of what the Bayesian model parameters might be and can be represented as a parametric distribution. Observed data can be collected to obtain evidence about the actual values of the parameters. The result of the Bayesian analysis is called the posterior and represents a probability distribution of an event expressed as a confidence. If additional data is obtained, the posterior can be treated as a prior and updated with the new data. This process employs Bayes' rule, which dictates conditional probabilities, such as how likely it is for event A to occur given that event B has occurred.

[0179] Regarding the disclosed navigation fusion system 400, the state estimator 440 may use the previously estimated state data as a prior and may use the inputs from the respiratory rate and / or respiratory phase identifier 410, the scope position estimator 420, the localization calculator 430, the image analyzer 435, the navigation path data repository 445, and / or the robotic position data repository 470 as the observed data. At the beginning of the process, the described vision-based initialization technique can be used to estimate the initial depth and roll in the trachea, and this estimated output from the depth-based position estimator 410 can be used as the prior. The state estimator 440 may perform a Bayesian statistical analysis on the prior and the observed data to generate a posterior distribution representing the probabilities and confidence values for each of the multiple possible states.

[0180] As used herein, the "probability" in "probability distribution" refers to the likelihood of correctly estimating the possible location and / or orientation of a medical device. For example, different probabilities can be calculated by one of the algorithm modules, which indicate the relative likelihood of the medical device being in one of several different possible branches within the lumen network. In one embodiment, the type of probability distribution (e.g., discrete distribution or continuous distribution) is selected to match the characteristics of the estimated state (e.g., the type of the estimated state, such as continuous position information versus discrete branch selection). As an example, the estimated state for identifying which segment a medical device is in for a three-way bifurcation can be represented by a discrete probability distribution and can include three discrete values, 20%, 30%, and 50%, representing the likelihood of being located within each of the three branches as determined by one of the algorithm modules. As another example, the estimated state can include a roll angle of 40 ± 5 degrees of the medical device, and the segment depth of the device tip within the branch can be 4 ± 1 mm, each represented by a Gaussian distribution, which is a type of continuous probability distribution.

[0181] In contrast, the "confidence value" as used herein reflects a measure of the confidence in the estimate of the state provided by one of the Figure 19 modules based on one or more factors. For EM-based modules, factors such as EM field distortion, inaccuracies in EM registration, patient displacement or movement, and patient respiration can affect the confidence in the state estimate. In particular, the confidence value of the state estimate provided by an EM-based module can depend on the patient's specific respiratory cycle, movement of the patient or the EM field generator, and the location within the anatomy where the device tip is located. For the image analyzer 435, example factors that can affect the confidence value of the state estimate include: the lighting conditions of the location of the captured image within the anatomy; the presence of an optical sensor that obscures the captured image or a fluid, tissue, or other obstacle in front of the optical sensor of the captured image; patient respiration; the condition of the tubular network of the patient himself (e.g., the lungs), such as the general fluid within the tubular network and the occlusion of the tubular network; and the specific operating techniques used, for example, in navigation or image capture.

[0182] For example, one factor can be that a particular algorithm has different levels of accuracy at different depths within the patient's lungs, such that in the case of being relatively close to the airway opening, a particular algorithm can have a high confidence in its estimate of the location and orientation of the medical device, but as the medical device further travels into the bottom of the lungs, the confidence value may decrease. Generally, the confidence value is based on one or more system factors related to the processing of the determination result, while the probability is a relative measure that occurs when attempting to determine the correct result based on multiple possibilities using a single algorithm based on the underlying data.

[0183] As an example, a mathematical equation for calculating the result of an estimated state represented by a discrete probability distribution (e.g., branch / segment identification with three values of the estimated state for a three-forked portion) can be as follows:

[0184] S 1 = C EM * P 1，EM + C Image * P 1，Image + C Robot * P 1，Robot ;

[0185] S 2 = C EM * P 2，EM + C Image * P 2，Image + C Robot * P 2，Robot ;

[0186] S 3 = C EM * P 3，EM + C Image * P 3，Image + C Robot * P 3，Robot ;

[0187] In the above example mathematical equation, S i (i = 1, 2, 3) represents possible example values of the estimated state in the case of identifying or having 3 possible segments in the 3D model, C EM , C Image and C Robot represent confidence values corresponding to an EM-based algorithm, an image-based algorithm, and a robot-based algorithm, and P i，EM , P i，Image and P i，Robot represent the probability of segment i. Due to the probabilistic nature of such a fusion algorithm, breathing can be tracked over time and even predicted to overcome latency and outlier interference.

[0188] In some embodiments, the confidence values of the data from the robot position data 470, the localization calculator 435, and the image analyzer 435 can be adaptively determined based on the respiratory period from the respiratory rate and / or respiratory period identifier 410. For example, the robot position data and the image data may be affected by respiratory movement differently from the EM sensor data. In some embodiments, the visual data obtained from the endoscopic imaging data repository 430 can be used to detect certain types of respiratory movement that cannot be detected by sensors external to the lumen network, such as the movement of the airway in the cranio-caudal (posterior-anterior) movement that can be detected by visual processing.

[0189] The navigation controller 460 is a module that receives data from the state estimator 440 and the navigation path data repository 445 and uses this data to guide further operation of the medical robotic system 110. For example, the navigation controller 460 can plot the estimated state along a predetermined navigation path and can determine the next movement of the instrument along the navigation path (e.g., the extension / retraction distance, roll, wire pull, or actuation of other actuation mechanisms). In some embodiments, the navigation controller 460 can automatically control the instrument according to the determined next movement. In some embodiments, the navigation controller 460 can output specific instrument movement instructions and / or instrument driver operation instructions for display to the user, for example, via the workstation 200. In some embodiments, the navigation controller 460 can cause a side-by-side view of a slice of the 3D model at the estimated position and a real-time image received from the scope imaging data repository 480 to be displayed to facilitate user-guided navigation.

[0190] 4. Overview of Example Navigation Technology

[0191] According to one or more aspects of the present disclosure, Figure 20 A flowchart of an example process 500 for generating an extracted virtual feature dataset is depicted. In some embodiments, process 500 can be performed preoperatively, i.e., before the start of a medical procedure that uses the model generated by process 500 and the features extracted by process 500. Process 500 can be implemented in Figure 19 the modeling system 420, Figure 16C the control and sensor electronics 184, and / or Figure 17 the console base 201 or components thereof. The graphical depictions within the provided Figure 20 flowchart are provided to illustrate and not limit the described blocks, and it will be understood that the depicted model 515, depth maps 532, 534, and visual representations of associated features may or may not be generated and displayed during the course of process 500.

[0192] At block 510, the model generator 440 can access image data representing the anatomical lumen network of a patient and generate a three-dimensional model 515. For example, a CT scan or an MRI scan can generate a plurality of images depicting two-dimensional cross-sections of the anatomical lumen network. The model generator 440 can segment these two-dimensional images to isolate or segment the tissue of the anatomical lumen network, and then can construct a three-dimensional point cloud of the data based on the isolated tissue locations in the various images and based on the spatial relationships of the cross-sections depicted in the images. The model generator 440 can generate a model based on the three-dimensional point cloud. The three-dimensional model can model the inner surface of the anatomical lumen network as a virtual lumen network. For example, in some implementations, the model 515 can be a segmented map of a patient's airway generated from a CT scan. The model can be any two-dimensional or three-dimensional representation of the patient's actual lumen network (or a portion of the lumen network).

[0193] At block 520, the feature extractor 450 can identify a plurality of virtual locations 525 within the model 515. As an example, the feature extractor 450 can identify a plurality of locations within the tracheal section of the model representing the patient's airway, e.g., one hundred twenty locations or more or less, depending on the parameters of the navigation system 400. In other examples, the feature extractor 450 can identify locations within other sections of the airway model, e.g., locations along a planned navigation path through the airway model, locations along the planned navigation path and at a predetermined proximity to branches along the planned navigation path, or locations throughout some or all of the airway sections.

[0194] At block 530, the feature extractor 450 can generate a plurality of virtual depth maps 532, 534 corresponding to the identified locations. For example, the feature extractor 450 can use the identified locations to set the location of a virtual imaging device within the virtual anatomical lumen network, and can generate virtual depth maps 532, 534 for each identified location. The example virtual depth maps 532 and virtual depth map 534 depicted illustrate different representations of the same depth information associated with a virtual representation of the main carina 156 in the airway. Each virtual pixel in the two-dimensional representation of the virtual depth map 532 is depicted using a color corresponding to its depth value, while the three-dimensional representation of the virtual depth map 534 depicts a bimodal shape where each virtual pixel is shown at a height along the z-axis corresponding to its depth value. The depicted virtual depth maps are provided to illustrate the concept of block 530, however, in some implementations of process 500, such visual representations may not be generated because process 500 may only require the data representing such depth maps in order to derive features as described below.

[0195] In some implementations, at block 530, the feature extractor 450 may access parameters of an imaging device (e.g., the imaging device 315 at the distal end of an endoscope) that is recognized as being used during a medical procedure during which the lumen network will be navigated. The feature extractor 450 may set virtual parameters of a virtual imaging device to match the parameters of the imaging device. Such parameters may include field of view, lens distortion, focal length, and brightness shading, and may be obtained based on calibration data or data obtained by testing the imaging device. Brightness shading (also known as vignetting) is a position-dependent variation in the amount of light transmitted by an optical system, which causes the image near the edges to darken. Vignetting results in a reduction in the amount of light transmitted by the optical system near the periphery of the lens field of view (FOV), thereby causing the image at the edges to gradually darken. After image capture, vignetting can be corrected by calibrating the lens attenuation distortion function of the imaging device. By matching the virtual parameters with the actual parameters, the resulting virtual depth maps 532, 534 may more closely correspond to the actual depth maps generated based on the images captured by the imaging device.

[0196] At block 540, the feature extractor 450 analyzes the values of the virtual depth map to identify one or more depth criteria. The depth criteria may be, for example, the position of local maxima within the depth map (e.g., pixels representing the farthest virtual tissue visible along a branch of the virtual airway model) or any position within a threshold distance from the local maxima along a curve peak surrounding the local maxima. The described depth criteria positions may be virtual pixel locations within the virtual depth map.

[0197] Block 540 provides a visual illustration of exemplary depth criteria 522 and 544 as local maxima, which correspond to the farthest virtual tissue visible within the virtual left bronchus and virtual right bronchus by the virtual imaging device. As a general rule, due to the typical shape of a human lung, a camera device or virtual camera device located near the main carina will be able to see farther into the right bronchus than into the left bronchus. Thus, depth criterion 544 corresponds to the farthest depicted virtual tissue within the right bronchus because it has a greater value than depth criterion 542, and depth criterion 542 corresponds to the farthest depicted virtual tissue within the left bronchus. Such information may assist in identifying a roll as described herein.

[0198] At block 550, feature extractor 450 derives pre-identified virtual features from the identified depth criteria. For example, as shown, feature extractor 450 may identify the value of distance 555 separating depth criteria 542, 544. The distance value may be represented as the number of pixels in the (x, y) space corresponding to the two-dimensional depth map 532 or as the (x, y, z) vector corresponding to the three-dimensional depth map 544. Feature extractor 450 may additionally or alternatively derive the identification and localization of the right and left bronchi as features. In other implementations, for example, involving depth maps at the localization of the branches of three or more airways, the features may include the size, shape, and orientation of a polygon connecting three or more local maxima.

[0199] At block 560, feature extractor 450 may generate a database of virtual localizations and the associated extracted virtual features. The database may be provided to navigation system 400 for calculating real-time instrument position determination, e.g., to automatically initialize probability state estimation, calculate registration, and perform other navigation-related calculations.

[0200] Figure 21 Described is an example intraoperative process 600 for generating depth information based on the correspondence between captured endoscopic images and an extracted virtual feature dataset of computed, depth information features and Figure 20 The flowchart of the example intraoperative process 600 for generating depth information based on the correspondence between captured endoscopic images and an extracted virtual feature dataset of computed, depth information features and Figure 19 Modeling system 420 and / or navigation fusion system 400, Figure 16C Control and sensor electronics 184 and / or Figure 17 Console base 201 or its components.

[0201] At block 610, feature extractor 450 receives imaging data captured by an imaging device located at the distal end of an instrument disposed within the anatomical lumen network of a patient. For example, the imaging device may be the aforementioned imaging device 315. An example visual representation of the imaging data is shown by image 615 depicting the main carina of the patient's airway. Image 615 depicts the anatomical main carina corresponding to the virtual main carina represented by virtual depth map 532 of Figure 20 Image 615 uses a specific visual representation to depict specific features and is provided to illustrate, and not limit, process 600. Image 615 represents intraluminal image data suitable for use in process 600, and other suitable image data may represent other anatomical structures and / or be depicted as images using different visual representations. Additionally, some embodiments of process 600 may operate on the imaging data (e.g., the values of the pixels received from the image sensor of the imaging device) without generating a corresponding visible representation of the image data (e.g., image 615).

[0202] At block 620, feature extractor 450 generates a depth map 620 corresponding to the imaging data represented by image 615. Feature extractor 450 can compute a depth value for each pixel of the imaging data, where the depth value represents an estimated distance between the imaging device and the tissue surface within the anatomical lumen network corresponding to what the pixel represents. Specifically, the depth value can represent an estimate of the physical distance between the entrance pupil of the optical system of the imaging device and the imaged tissue depicted by the pixel. In some embodiments, feature extractor 450 can generate a depth map based on a single image 615 using photogrammetric methods (e.g., by the shape of shadows) processing. By using photogrammetric methods, feature extractor 450 can be robust to outliers due to differences in reflectivity between parts of tissue that may be covered in fluid (e.g., mucus). In some embodiments, feature extractor 450 can use a set of stereo images depicting the imaged region to generate a depth map. For example, a robotically controlled endoscope can capture a first image at a first location, be retracted, extended, and / or rotated a known distance robotically to a second location, and can capture a second image at the second location. Feature extractor 450 can use the known translation of the robotically controlled endoscope and the differences between the first and second images to generate a depth map.

[0203] At block 630, feature extractor 450 identifies one or more depth criteria in the depth map. As described above with respect to the virtual depth map, depth criteria in a depth map generated based on real image data can be, for example, the location of local maxima within the depth map (e.g., pixels representing the farthest anatomical tissue visible along a patient's airway branch) or any location within a threshold distance from a local maximum along a curve peak surrounding the local maximum. The described depth criteria locations can be pixel locations within image 615. The depth criteria selected for identification at block 630 preferably correspond to the depth criteria identified at block 540.

[0204] For example, feature extractor 450 can identify a first pixel among a plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map, and in some embodiments, each depth criterion can correspond to a local maximum in a region of depth values surrounding the identified pixel. Block 630 provides a visual illustration of exemplary depth criteria 632 and 634 as local maxima, which correspond to the farthest tissue within the left and right bronchi visible to imaging device 315. Specifically, depth criterion 634 corresponds to the pixel representing the farthest imaged tissue within the right bronchus because it has a greater value than depth criterion 632, and depth criterion 632 corresponds to the pixel representing the farthest imaged tissue within the left bronchus. Other airway bifurcations can have similar known depth relationships between different branches.

[0205] At block 640, the feature extractor 450 derives pre-identified features from the identified depth criteria. For example, as shown, the feature extractor 450 can compute the value of a distance 645 (e.g., a separation measure) between pixels corresponding to depth criteria 632 and 634. The distance value can be represented as a number of pixels in the (x, y) space corresponding to a two-dimensional depth map or as an (x, y, z) vector corresponding to a three-dimensional depth map 625, preferably in the same format as the features identified at block 550 of process 500. The feature extractor 450 can additionally or alternatively derive the identification and localization of the right and left bronchi as features. In other implementations, e.g., involving depth maps at the localization of the branching of three or more airways, the features can include the size, shape, and orientation of a polygon connecting three or more local maxima.

[0206] At block 650, the depth-based position estimator 410 computes a correspondence between the features derived from the imaging data and the plurality of features in the depth feature data repository 405. For example, the features derived from the imaging data can be the value of the distance 645 computed based on the identified depth criteria of depth map 625, as described with respect to block 640. The depth-based position estimator 410 can compare the value of the distance 645 with distance values associated with a plurality of localizations in the trachea to identify a distance value in the distance values that corresponds to the value of the distance 645. These distance values can be pre-computed and stored in the data repository 405 as described above, or these distance values can be computed in real time while navigating the patient's anatomy. The values computed in real time can be stored in the working memory during the correspondence computation or can be added to the data repository 405 and subsequently accessed if the localization corresponding to the value involves additional correspondence computations. Figure 20 To determine the correspondence, the depth-based position estimator 410 can identify the value of the distance 555 (discussed above with respect to block 550 of process 500) as an exact match to the value of the distance 645, as the best match (e.g., the closest value) among the options in the depth feature data repository 405 to the value of the distance 645, or as the first match within a predetermined threshold of the value of the distance 645. It will be understood that the navigation system 400 can be pre-configured to look for an exact match, a best match, or a match within a first threshold based on a trade-off between computational speed and accuracy of the position output, or can dynamically look for one of these options based on the current navigation conditions.

[0207]

[0208] ​At block 660, the depth-based position estimator 410 determines an estimated pose of the distal end of the instrument within the anatomical lumen network based on a virtual localization associated with a virtual feature identified in the corresponding relationship at block 650. The pose may include the position of the instrument (e.g., the insertion depth within a section of the airway or other lumen network portion), the roll, pitch, and / or deflection of the instrument, or other degrees of freedom. As described above, the depth feature data repository 405 may store a database of tuples or associated values that includes localizations and features extracted from virtual images generated at those localizations. Thus, at block 660, the depth-based position estimator 410 may access the localization information stored in association with the features identified at block 650 and output that localization as the position of the instrument. In some embodiments, block 660 may include identifying an angular transformation between the positions of the right and left bronchi in the image 615 and the virtual localizations of the virtual right and left bronchi in the virtual depth map 532. The angular transformation may be used to determine the roll of the instrument within the airway.

[0209] At block 670, the depth-based position estimator 410 outputs the identified pose for use within the navigation system 400. As described above, the pose may be output to the state estimator 440 and used as an automatically determined Bayesian prior during initialization, as opposed to an initialization process that requires the user to reposition the endoscope at multiple specified localizations to allow initialization. In some embodiments, the pose may be output to the registration calculator 465 for use in calculating the registration between the model coordinate system and the EM coordinate system. Advantageously, processes 500 and 600 enable such calculations to be performed without requiring the physician to deviate from a predetermined navigation path through the patient's airway to reach the target tissue site.

[0210] 5. Alternatives

[0211] Several alternatives to the subject matter described herein are provided below.

[0212] 1. A method for facilitating navigation of a patient's anatomical lumen network, the method being performed by a collection of one or more computing devices, the method comprising:

[0213] Receiving imaging data captured by an imaging device located at a distal end of an instrument, the distal end of the instrument being disposed within the anatomical lumen network;

[0214] Accessing virtual features derived from a virtual image that is simulated from the viewpoint of a virtual imaging device disposed at a virtual localization within a virtual lumen network representing the anatomical lumen network;

[0215] Calculating a correspondence between features derived from the imaging data and virtual features derived from the virtual image; and

[0216] Determine the pose of the distal end of the instrument within the anatomical lumen network based on a virtual location associated with the virtual feature.

[0217] 2. The method according to alternative 1, further comprising: generating a depth map based on the imaging data, wherein the virtual feature is derived from a virtual depth map associated with the virtual image, and wherein calculating the correspondence is at least partially based on correlating one or more features of the depth map with one or more features of the virtual depth map.

[0218] 3. The method according to alternative 2, further comprising:

[0219] Generate the depth map by calculating, for each pixel in a plurality of pixels of the imaging data, a depth value that represents an estimated distance between the imaging device and a tissue surface within the anatomical lumen network corresponding to the pixel;

[0220] Identify a first pixel among the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map;

[0221] Calculate a first value representing a distance between the first pixel and the second pixel;

[0222] wherein the virtual depth map includes, for each virtual pixel in a plurality of virtual pixels, a virtual depth value that represents a virtual distance between the virtual imaging device and a portion of the virtual lumen network represented by the virtual pixel, and wherein accessing the virtual feature derived from the virtual image includes: accessing a second value representing a distance between a first depth criterion and a second depth criterion in the virtual depth map; and

[0223] Calculate the correspondence based on a comparison of the first value and the second value.

[0224] 4. The method according to alternative 3, further comprising:

[0225] Access a plurality of values representing a distance between a first depth criterion and a second depth criterion in a plurality of virtual depth maps, the plurality of virtual depth maps respectively representing different virtual locations among a plurality of virtual locations within the virtual lumen network; and

[0226] Calculate the correspondence based on the second value that more closely corresponds to the first value compared to other values among the plurality of values.

[0227] 5. The method according to any one of alternative 3 or 4, wherein the anatomical lumen network includes an airway, and the imaging data depicts the bifurcation of the airway, and the method further includes:

[0228] In each of the depth map and the virtual depth map, identifying one of the first depth criterion and the second depth criterion as the right bronchus; and

[0229] Determining the roll of the instrument based on the angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein the pose of the distal end of the instrument within the anatomical lumen network includes the determined roll.

[0230] 6. The method according to any one of alternatives 2 to 5, further including:

[0231] Identifying three or more depth criteria in each of the depth map and the virtual depth map;

[0232] Determining the shape and orientation of polygons connecting the depth criteria in each of the depth map and the virtual depth map; and

[0233] Calculating the correspondence based on a comparison of the shape and orientation of the polygons in the depth map with the shape and orientation of the polygons in the virtual depth map.

[0234] 7. The method according to any one of alternatives 2 to 6, wherein the depth map is generated based on photogrammetry.

[0235] 8. The method according to any one of alternatives 1 to 7, further including:

[0236] Calculating a probability state of the instrument within the anatomical lumen network based on a plurality of inputs including position; and

[0237] Guiding the navigation of the instrument through the anatomical lumen network at least in part based on the probability state.

[0238] 9. The method according to alternative 8, further including: initializing a navigation system configured to calculate the probability state and to guide the navigation of the anatomical lumen network based on the probability state, wherein initializing the navigation system includes: setting a prior of a probability calculator based on the position.

[0239] 10. The method according to alternative 9, further including:

[0240] Receiving additional data representing an updated pose of the distal end of the instrument;

[0241] Set a likelihood function of the probability calculator based on the additional data; and

[0242] Use the probability calculator to determine the probability state based on the prior and the likelihood function.

[0243] 11. The method according to any one of alternatives 8 to 10, further comprising:

[0244] Provide the plurality of inputs to a navigation system configured to calculate the probability state, the first input including the pose of the distal end of the instrument, and at least one additional input including one or both of robot position data from a robot system actuating the movement of the instrument and data received from a position sensor at the distal end of the instrument; and

[0245] Calculate the probability state of the instrument based on the first input and the at least one additional input.

[0246] 12. The method according to any one of alternatives 1 to 11, further comprising: determining a registration between a coordinate system of the virtual lumen network and a coordinate system of an electromagnetic field generated around the anatomical lumen network, at least partially based on a pose of the distal end of the instrument within the anatomical lumen network determined according to the calculated correspondence.

[0247] 13. The method according to any one of alternatives 1 to 12, wherein determining the position includes determining a distance that the distal end of the instrument advances within a section of the anatomical lumen network.

[0248] 14. A system configured to facilitate navigation of a patient's anatomical lumen network, the system comprising:

[0249] An imaging device located at the distal end of the instrument;

[0250] At least one computer-readable memory storing executable instructions; and

[0251] One or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least perform the following operations:

[0252] Receive imaging data captured by the imaging device when the distal end of the instrument is disposed within the anatomical lumen network;

[0253] Access virtual features derived from a virtual image that is simulated from the viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network representing the anatomical lumen network;

[0254] Calculate a correspondence between features derived from the imaging data and virtual features derived from the virtual image; and

[0255] Determine a pose of a distal end of the instrument relative to within the anatomical lumen network based on a virtual localization associated with the virtual feature.

[0256] 15. The system according to alternative 14, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0257] Generate a depth map based on the imaging data, wherein the virtual image represents a virtual depth map; and

[0258] Determine the correspondence at least in part based on correlating one or more features of the depth map with one or more features of the virtual depth map.

[0259] 16. The system according to alternative 15, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0260] Generate the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value that represents an estimated distance between the imaging device and a tissue surface within the anatomical lumen network corresponding to the pixel;

[0261] Identify a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map;

[0262] Calculate a first value representing a distance between the first pixel and the second pixel;

[0263] wherein the virtual depth map includes, for each virtual pixel of a plurality of virtual pixels, a virtual depth value that represents a virtual distance between the virtual imaging device and a portion of the virtual lumen network represented by the virtual pixel, and wherein a feature derived from the virtual image includes a second value representing a distance between the first depth criterion and the second depth criterion in the virtual depth map; and

[0264] Determine the correspondence based on a comparison of the first value and the second value.

[0265] 17. The system according to alternative 16, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0266] Access multiple values representing the distance between a first depth criterion and a second depth criterion in multiple virtual depth maps, where the multiple virtual depth maps respectively represent different virtual positions in a plurality of virtual positions within the virtual lumen network; and

[0267] Calculate the correspondence based on the second value that more closely corresponds to the first value compared to other values among the multiple values, and identify the second value as the closest match to the first value among the multiple values.

[0268] 18. The system according to any one of alternatives 16 to 17, wherein the anatomical lumen network includes an airway, and the imaging data depicts a bifurcation of the airway, and wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0269] In each of the depth map and the virtual depth map, identify one of the first depth criterion and the second depth criterion as the right bronchus; and

[0270] Determine the roll of the instrument based on the angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein the pose of the distal end of the instrument within the anatomical lumen network includes the determined roll.

[0271] 19. The system according to any one of alternatives 15 to 18, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0272] Identify three or more depth criteria in each of the depth map and the virtual depth map;

[0273] Determine the shape and position of a polygon connecting the three or more depth criteria in each of the depth map and the virtual depth map; and

[0274] Calculate the correspondence based on comparing the shape and position of the polygon of the depth map with the shape and position of the polygon of the virtual depth map.

[0275] 20. The system according to any one of alternatives 15 to 19, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations: Generate the depth map based on photogrammetry.

[0276] 21. The system according to any one of alternatives 14 to 20, wherein the one or more processors are configured to communicate with a navigation system, and wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations:

[0277] Calculate a probabilistic state of the instrument within the anatomical lumen network using the navigation system, at least in part based on a plurality of inputs including position; and

[0278] Guide navigation of the instrument through the anatomical lumen network at least in part based on the probabilistic state calculated by the navigation system.

[0279] 22. The system according to alternative 21, further comprising a robotic system configured to guide movement of the instrument during the navigation.

[0280] 23. The system according to alternative 22, wherein the plurality of inputs includes robotic position data received from the robotic system, and wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations: Calculate the probabilistic state of the instrument using the navigation system, at least in part based on the position and the robotic position data.

[0281] 24. The system according to any one of alternatives 21 to 23, further comprising a position sensor located at a distal end of the instrument, the plurality of inputs including data received from the position sensor, and wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations: Calculate the probabilistic state of the instrument using the navigation system, at least in part based on the position and data received from the position sensor.

[0282] 25. The system according to any one of alternatives 14 to 24, wherein the one or more processors are configured to execute the instructions to cause the system to at least perform the following operations: Determine a registration between a coordinate system of the virtual lumen network and a coordinate system of an electromagnetic field generated around the anatomical lumen network, at least in part based on the position.

[0283] 26. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause at least one computing device to at least perform the following operations:

[0284] Access a virtual three-dimensional model of an inner surface of a patient's anatomical lumen network;

[0285] Identify a plurality of virtual localizations within the virtual three-dimensional model;

[0286] For each of the plurality of virtual localizations within the virtual three-dimensional model:

[0287] Generate a virtual depth map that represents the virtual distance between a virtual imaging device positioned at the virtual localization and a portion of the inner surface within the field of view of the virtual imaging device when positioned at the virtual localization, and

[0288] Derive at least one virtual feature from the virtual depth map; and

[0289] Generate a database that associates the plurality of virtual localizations with the at least one virtual feature derived from the corresponding virtual depth map.

[0290] 27. The non-transitory computer-readable storage medium according to alternative 26, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operation: Provide the database to a navigation system configured to guide an instrument through the anatomical lumen network during a medical procedure.

[0291] 28. The non-transitory computer-readable storage medium according to alternative 27, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations:

[0292] Access data representing an imaging device positioned at a distal end of the instrument;

[0293] Identify image capture parameters of the imaging device; and

[0294] Set virtual image capture parameters of the virtual imaging device to correspond to the image capture parameters of the imaging device.

[0295] 29. The non-transitory computer-readable storage medium according to alternative 28, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operation: Generate the virtual depth map based on the virtual image capture parameters.

[0296] 30. The non-transitory computer-readable storage medium according to any one of alternatives 28 to 29, wherein the image capture parameters include one or more of a field of view, lens distortion, focal length, and brightness shading.

[0297] 31. The non-transitory computer-readable storage medium according to any one of alternatives 26 to 30, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations:

[0298] For each of the plurality of virtual localizations:

[0299] Identify a first depth criterion and a second depth criterion in the virtual depth map, and calculate a value representing the distance between the first depth criterion and the second depth criterion; and

[0300] Create the database by associating the plurality of virtual localizations with corresponding values.

[0301] 32. The non-transitory computer-readable storage medium according to any one of alternatives 26 to 31, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations:

[0302] For each of the plurality of virtual localizations:

[0303] Identify three or more depth criteria in the virtual depth map, and

[0304] Determine the shape and location of a polygon connecting the three or more depth criteria; and

[0305] Create the database by associating the plurality of virtual localizations with the shape and location of the corresponding polygon.

[0306] 33. The non-transitory computer-readable storage medium according to any one of alternatives 26 to 32, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations:

[0307] Generate three-dimensional volume data from a series of two-dimensional images representing an anatomical lumen network of the patient; and

[0308] Form a virtual three-dimensional model of the inner surface of the anatomical lumen network from the three-dimensional volume data.

[0309] 34. The non-transitory computer-readable storage medium according to alternative 33, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operation: Control a computed tomography imaging system to capture the series of two-dimensional images.

[0310] 35. The non-transitory computer-readable storage medium according to any one of alternatives 33 to 34, wherein the instructions, when executed, cause the at least one computing device to at least perform the following operation: Form the virtual three-dimensional model by applying volume segmentation to the three-dimensional volume data.

[0311] 36. A method for facilitating navigation of a patient's anatomical lumen network, the method being executed by a collection of one or more computing devices, the method comprising:

[0312] Receive a set of stereoscopic images representing the interior of the anatomical lumen network;

[0313] Generate a depth map based on the set of stereoscopic images;

[0314] Access virtual features derived from a virtual image that is simulated from the viewpoint of a virtual imaging device positioned within a virtual lumen network;

[0315] Calculate a correspondence between features derived from the depth map and virtual features derived from the virtual image; and

[0316] Determine the pose of the distal end of the instrument within the anatomical lumen network based on the virtual location associated with the virtual feature.

[0317] 37. The method according to alternative 36, wherein generating the set of stereoscopic images includes:

[0318] Position an imaging device at the distal end of an instrument located at a first position within the anatomical lumen network;

[0319] Capture a first image of the interior of the anatomical lumen network with the imaging device positioned at the first position;

[0320] Robotically control the imaging device to move a known distance to reach a second position within the anatomical lumen network; and

[0321] Capture a second image of the interior of the anatomical lumen network with the imaging device positioned at the second position.

[0322] 38. The method according to alternative 37, wherein robotically controlling the imaging device to move a known distance includes one or both of: retracting the imaging device, and angling and rolling the imaging device.

[0323] 6. Implementing the System and Terminology

[0324] The implementations disclosed herein provide systems, methods, and devices for improved navigation of a lumen network.

[0325] It should be noted that the terms "couple", "coupling", "coupled", or other variants of the word couple as used herein may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0326] The feature correspondence calculation, position estimation, and robot motion actuation functions described herein can be stored as one or more instructions on a processor-readable medium or a computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or a processor. By way of example and not limitation, such a medium can include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be noted that the computer-readable medium can be tangible and non-transitory. As used herein, the term "code" can refer to software, instructions, code, or data that can be executed by a computing device or a processor.

[0327] The methods disclosed herein include one or more steps or acts for implementing the described methods. Without departing from the scope of the claims, the method steps and / or acts can be interchanged with one another. In other words, unless a specific order of steps or acts is required for the correct operation of the method being described, the order and / or use of specific steps and / or acts can be modified without departing from the scope of the claims.

[0328] As used herein, the term "plurality" means two or more. For example, a plurality of components means two or more components. The term "determine" encompasses a variety of actions and, accordingly, "determine" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, etc. "Determine" can also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. In addition, "determine" can include parsing, selecting, choosing, establishing, etc.

[0329] Unless otherwise expressly stated, the phrase "based on" does not mean "based solely on". In other words, the phrase "based on" describes both "based solely on" and "based at least on".

[0330] A previous description of the disclosed implementations is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these implementations will be apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the scope of the invention. For example, it should be understood that a person of ordinary skill in the art will be able to employ multiple corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling, or joining tool components, equivalent mechanisms for generating a particular actuation motion, and equivalent mechanisms for delivering electrical energy. Accordingly, the present invention is not intended to be limited to the implementations shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system configured to facilitate navigation of a network of anatomical lumens of a patient, the system comprising: an imaging device located at a distal end of an instrument; at least one computer-readable memory having executable instructions stored thereon; and one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least perform the following operations: receive imaging data captured by the imaging device with the distal end of the instrument disposed within the network of anatomical lumens, wherein the network of anatomical lumens includes a right bronchus and a left bronchus; identify one or more features from the imaging data, the one or more features identified from the imaging data representing a first disposition of the right bronchus and the left bronchus in the imaging data; access one or more features associated with a virtual image that is simulated from a viewpoint of a virtual imaging device disposed at a virtual location within a virtual lumen network representative of the network of anatomical lumens, the virtual lumen network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second disposition of the virtual right bronchus and the virtual left bronchus in the virtual image; calculate a correspondence between the first disposition and the second disposition; and determine a roll of the distal end of the instrument within the network of anatomical lumens based on the calculated correspondence, and wherein the network of anatomical lumens includes a trachea leading to the right bronchus and the left bronchus, and wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: determine an insertion depth of the distal end of the instrument within the trachea at least in part based on the virtual location of the virtual image, and wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: generate an electromagnetic field around the network of anatomical lumens; and determine a registration between a coordinate system of the virtual lumen network and a coordinate system of the electromagnetic field at least in part based on the insertion depth and roll of the distal end of the instrument.

2. The system of claim 1, wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: generate a depth map based on the imaging data; and identify the first disposition of the right bronchus and the left bronchus in the imaging data based on the depth map; wherein the one or more features associated with the virtual image include the second disposition derived from a virtual depth map associated with the virtual image.

3. The system of claim 2, wherein the one or more processors are configured to execute instructions to cause the system to at least perform the following operations: The depth map is generated by calculating the following depth value for each of a plurality of pixels of the imaging data: the depth value represents an estimated distance between the imaging device and the tissue surface within the anatomical lumen network corresponding to the pixel; Identify a first pixel among the plurality of pixels corresponding to a first depth criterion in the depth map; Determine a first localization of the right bronchus in the imaging data based on the position of the first pixel; Identify a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map; Determine a second localization of the left bronchus in the imaging data based on the position of the second pixel.

4. The system according to claim 3, wherein, the one or more processors are configured to execute instructions to cause the system to perform at least the following operations: Determine that the first pixel has a greater depth value than the second pixel; and Based on determining that the first pixel has a greater depth value, identify that the first localization corresponds to the right bronchus.

5. The system according to claim 3, wherein, the first depth criterion represents the farthest imaged tissue within the right bronchus, and wherein the second depth criterion represents the farthest imaged tissue within the left bronchus.

6. The system according to claim 3, wherein, the first depth criterion corresponds to a first local maximum in a first region of depth values around the first pixel, and wherein the second depth criterion corresponds to a second local maximum in a second region of depth values around the second pixel.

7. The system according to claim 1, wherein, the one or more processors are configured to execute instructions to cause the system to perform at least the following operation: determine the roll of the instrument based on the angular distance between a first position of the right bronchus in the imaging data and a second position of the virtual right bronchus in the virtual image.

8. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause at least one computing device to perform at least the following operations: Receive imaging data captured by an imaging device with the distal end of an instrument positioned within an anatomical lumen network, wherein, the anatomical lumen network includes a right bronchus and a left bronchus; Identify one or more features from the imaging data, the one or more features identified from the imaging data representing a first placement of the right bronchus and the left bronchus in the imaging data; Access one or more features associated with a virtual image, the virtual image being simulated from the viewpoint of a virtual imaging device positioned at a virtual location within a virtual lumen network representing the anatomical lumen network, the virtual lumen network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second placement of the virtual right bronchus and the virtual left bronchus in the virtual image; Calculate the correspondence between the first placement and the second placement; and Determine the roll of the distal end of the instrument within the anatomical lumen network based on the calculated correspondence, and wherein the anatomical lumen network includes a trachea leading to the right bronchus and the left bronchus, and wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations: determine the insertion depth of the distal end of the instrument within the trachea based at least in part on the virtual positioning of the virtual image, and wherein the instructions, when executed, cause the at least one computing device to at least perform the following operations: Generate an electromagnetic field around the anatomical lumen network; and Determine the registration between the coordinate system of the virtual lumen network and the coordinate system of the electromagnetic field based at least in part on the insertion depth and roll of the distal end of the instrument.

9. The non-transitory computer-readable storage medium according to claim 8, wherein, the instructions, when executed, cause the at least one computing device to at least perform the following operations: Generate a depth map based on the imaging data; and Identify the first placement of the right bronchus and the left bronchus in the imaging data based on the depth map; wherein one or more features associated with the virtual image include the second placement derived from a virtual depth map associated with the virtual image.

10. The non-transitory computer-readable storage medium according to claim 9, wherein, the instructions, when executed, cause the at least one computing device to at least perform the following operations: Generate the depth map by calculating, for each pixel in a plurality of pixels of the imaging data, a depth value that represents an estimated distance between the imaging device and the tissue surface corresponding to the pixel within the anatomical lumen network; Identify a first pixel among the plurality of pixels corresponding to a first depth criterion in the depth map; Determine a first positioning of the right bronchus in the imaging data based on the position of the first pixel; Identify a second pixel among the plurality of pixels corresponding to a second depth criterion in the depth map; Determine a second positioning of the left bronchus in the imaging data based on the position of the second pixel.

11. The non-transitory computer-readable storage medium according to claim 10, wherein, the instructions, when executed, cause the at least one computing device to at least perform the following operations: Determine that the first pixel has a greater depth value than the second pixel; and Identify that the first positioning corresponds to the right bronchus based on determining that the first pixel has a greater depth value.

12. The non-transitory computer-readable storage medium according to claim 10, wherein, the first depth criterion represents the farthest imaged tissue within the right bronchus, and wherein the second depth criterion represents the farthest imaged tissue within the left bronchus.

13. The non-transitory computer-readable storage medium according to claim 10, wherein, the first depth criterion corresponds to a first local maximum in a first region of depth values around the first pixel, and wherein the second depth criterion corresponds to a second local maximum in a second region of depth values around the second pixel.

14. The non-transitory computer-readable storage medium according to claim 8, wherein, when executed, the instructions cause the at least one computing device to at least perform the following operations: determining a roll of the instrument based on an angular distance between a first position of the right bronchus in the imaging data and a second position of the virtual right bronchus in the virtual image.

Images