Systems and methods related to elongate devices

Abstract

To provide improved real-time visualization of a target tissue location to aid an operator during minimally invasive medical techniques.SOLUTION: A medical system includes a catheter and a processor. The catheter includes a first channel and a distal end portion associated with a catheter frame of reference. The processor is configured to receive a first image captured by a first imaging device provided by the catheter within a patient anatomy, and receive a second image of the patient anatomy captured by a second imaging device from outside the patient anatomy. The second image includes the catheter and the first imaging device. The processor is further configured to determine a relative pose between the first imaging device and the distal end of the catheter based on the second image, determine a target location associated with a target structure in the first image, and transform the target location in an image frame of reference associated with the first image into the catheter frame of reference based on the relative pose.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims the benefit of U.S. Provisional Application No. 62 / 662,440, filed April 25, 2018, which is incorporated herein by reference in its entirety.

[0002] The present disclosure is directed to systems and methods for controlling steerable elongate devices. [Background technology]

[0003] Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during a medical procedure, thereby reducing patient recovery time, discomfort, and adverse side effects. Such minimally invasive techniques may be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. An operator may insert a minimally invasive medical instrument (including a surgical, diagnostic, therapeutic, or biopsy instrument) through these natural orifices or incisions to reach a target tissue location. One such minimally invasive technique is the use of a flexible and / or steerable elongated device, such as a flexible catheter, that can be inserted into an anatomical passageway and navigated toward a region of interest within the patient's anatomy. To aid in reaching the target tissue location, the location and movement of the medical instrument may be correlated with pre- or intra-operative images of the patient's anatomy. With image-guided instruments that are correlated with images, the instruments may navigate natural or surgically created passageways within anatomical systems such as the lungs, colon, intestines, kidneys, heart, circulatory system, etc. However, minimally invasive medical devices do not always provide the operator (e.g., a surgeon or other medical personnel) with sufficient imaging capabilities to identify the target tissue location, for example, when the target tissue location is located below the surface of the passageway through which the minimally invasive medical device is introduced. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, it would be advantageous to provide improved real-time visualization of target tissue locations to aid operators during minimally invasive medical procedures. [Means for solving the problem]

[0005] Embodiments of the present invention are best summarized by the claims that follow this description.

[0006] Consistent with some embodiments, a method is performed by a computing system. The method includes receiving a first image of an anatomical structure from an imaging probe inside the patient anatomy. The first image has an image frame of reference. The imaging probe is extendable distally beyond a distal end of a catheter. The distal end of the catheter has a catheter frame of reference. The method further includes receiving a first external image of the patient anatomy from a first external imaging device. The first external image includes the catheter and the imaging probe. The method further includes determining a relative orientation between the imaging probe and the distal end of the catheter based on the first external image, determining a target location associated with the target structure in the first image, and transforming the target location in the image frame of reference to the catheter frame of reference based on the relative orientation.

[0007] Consistent with some embodiments, a medical system includes a catheter including a first channel and a distal end portion. The distal end portion is associated with a catheter frame of reference. The medical system further includes a processor. The processor is configured to receive a first image captured by a first imaging device provided by the catheter within a patient anatomy and a second image of the patient anatomy captured by a second imaging device from outside the patient anatomy. The second image includes the catheter and the first imaging device. The processor is further configured to determine a relative orientation between the first imaging device and the distal end of the catheter based on the second image, determine a target location associated with a target structure in the first image, and transform the target location in the image frame of reference associated with the first image to the catheter frame of reference based on the relative orientation.

[0008] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope thereof. In that regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a simplified diagram of a teleoperated medical system according to some embodiments.

[0010] [Figure 2A] 1 is a simplified diagram of a medical device system according to some embodiments.

[0011] [Figure 2B] 1 is a simplified diagram of a medical instrument with an expanded medical tool according to some embodiments.

[0012] [Figure 3A] FIG. 1 is a simplified side view of a patient coordinate space including a medical instrument attached to an insertion assembly according to some embodiments. [Figure 3B] FIG. 1 is a simplified side view of a patient coordinate space including a medical instrument attached to an insertion assembly according to some embodiments.

[0013] [Figure 4] 1 illustrates a catheter including an imaging system according to some embodiments.

[0014] [Figure 5] 5 illustrates an image produced by the imaging system of FIG. 4 according to some embodiments.

[0015] [Figure 6] 1 illustrates an external view of a patient anatomy image generated by an imaging system outside the patient anatomy according to some embodiments.

[0016] [Figure 7] 1 illustrates a catheter including an imaging system and a tool according to some embodiments.

[0017] [Figure 8] 8 illustrates an image produced by the imaging system of FIG. 7 according to some embodiments.

[0018] [Figure 9] 1 illustrates an external image of a patient anatomy generated by an imaging system outside the patient anatomy according to some embodiments.

[0019] [Figure 10] 1 illustrates a flowchart describing a method for determining the location of a target structure using an imaging system according to some embodiments.

[0020] [Figure 11] 1 illustrates a flowchart describing a method for providing a virtual visualization image on a display system according to some embodiments.

[0021] [Figure 12A] 1 illustrates a virtual visualization image according to some embodiments.

[0022] [Figure 12B] 1 illustrates a virtual visualization image according to some embodiments.

[0023] [Figure 13] 1 illustrates a virtual visualization image according to some embodiments.

[0024] [Figure 14] 1 illustrates a virtual visualization image according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0025] Embodiments of the present disclosure and their advantages are best understood by reference to the following detailed description, in which like reference numerals are used to identify like elements shown in one or more of the figures, and it should be understood that the designations therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting them.

[0026] In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one of ordinary skill in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, not limiting. Those skilled in the art may recognize other elements within the scope and spirit of the present disclosure that are not specifically described herein. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless otherwise specified or unless one or more features render the embodiment non-functional.

[0027] In some instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the embodiments.

[0028] This disclosure describes various instruments and instrument portions with respect to their state in three-dimensional space. As used herein, the term "position" refers to the location of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational configuration of an object or portion of an object (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or portion of an object in at least one translational degree of freedom and the orientation of an object or portion of an object in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0029] 1 is a simplified diagram of a teleoperated medical system 100 according to some embodiments. In some embodiments, the teleoperated medical system 100 is suitable for use in, for example, surgical, diagnostic, therapeutic, or biopsy procedures. Although some embodiments are provided herein with respect to such procedures, any reference to medical or surgical instruments and methods is non-limiting. The systems, instruments, and methods described herein may be used on animals, human cadavers, animal remains, portions of the human or animal anatomy, non-surgical diagnostics, and industrial systems and general-purpose robotic or teleoperated systems.

[0030] As shown in FIG. 1 , medical system 100 generally includes a manipulator assembly 102 for actuating medical instruments 104 in performing various procedures on a patient P. Manipulator assembly 102 may be a teleoperated assembly, a non-teleoperated assembly, or a hybrid teleoperated and non-teleoperated assembly with selectable degrees of freedom that may be motorized and / or teleoperated and selectable degrees of freedom that may be non-motorized and / or non-teleoperated. Manipulator assembly 102 is mounted on or near an operating table T. A master assembly 106 enables an operator (e.g., a surgeon, clinician, or physician as illustrated in FIG. 1 ) to view the intervention site and control manipulator assembly 102.

[0031] The master assembly 106 may be located in an operator console, typically located in the same room as the operating table T, such as to the side of the operating table on which the patient P is positioned. However, it should be understood that the operator O may be located in a different room than the patient P, or in a different building entirely. The master assembly 106 typically includes one or more control devices for controlling the manipulator assemblies 102. The control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controllers, voice recognition devices, body movement or presence sensors, and / or the like. The control devices may have the same degrees of freedom as the associated medical instrument 104 to provide the operator O with a strong sense of direct control over the instrument 104. In this manner, the control devices provide the operator O with a sense of telepresence or the perception that the control device is integral with the medical instrument 104.

[0032] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical instrument 104 and still provide telepresence to the operator O. In some embodiments, the control device may optionally be a manual input device that moves with six degrees of freedom, and the manual input device may include an actuatable handle for actuating the instrument (e.g., closing grasping jaws, applying electrical potentials to electrodes, delivering medical treatment, and / or the like).

[0033] The manipulator assembly 102 supports the medical instrument 104 and may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, commonly referred to as a setup structure) and / or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands from a control system) and kinematic structure of the manipulator. The manipulator assembly 102 may optionally include multiple actuators or motors that drive inputs to the medical instrument 104 in response to commands from a control system (e.g., control system 112). The actuators may optionally include a drive system that, when coupled to the medical instrument 104, may advance the medical instrument 104 naturally or into a surgically created anatomical opening. Other drive systems may move the distal end of the medical instrument 104 in multiple degrees of freedom, including three degrees of linear motion (e.g., linear motion along the X, Y, and Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, and Z Cartesian axes). Additionally, actuators may be used to actuate articulatable end effectors of the medical instrument 104 for grasping tissue within the jaws of a biopsy device and / or the like. Actuator position sensors, such as resolvers, encoders, potentiometers, and other mechanisms, may provide sensor data to the medical system 100 that describes the rotation and orientation of the motor shaft. This position sensor data may be used to determine the movement of an object manipulated by the actuator.

[0034] The teleoperated medical system 100 may include a sensor system 108 comprising one or more subsystems for receiving information regarding the instrument of the manipulator assembly 102. Such subsystems may include a position / location sensor system (e.g., an electromagnetic (EM) sensor system), a shape sensor system for determining the position, orientation, speed, velocity, attitude, and / or shape of the distal end and / or one or more segments along a flexible body that may comprise the medical instrument 104, and / or a visualization system for capturing images from the distal end of the medical instrument 104.

[0035] The teleoperated medical system 100 may include a display system 110 for displaying images or representations of the medical instruments 104 and surgical site generated by subsystems of the sensor system 108. The display system 110 and master assembly 106 may be oriented such that an operator O may control the medical instruments 104 and master assembly 106 with a sense of telepresence.

[0036] In some embodiments, the medical instrument 104 may have a visualization system (discussed in more detail below), which may include a viewing scope assembly that records simultaneous or real-time images of the surgical site and provides the images to the operator or operators O through one or more displays of the medical system 100, such as one or more displays of the display system 110. The simultaneous images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscopic component that may be integrally or removably coupled to the medical instrument 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with the medical instrument 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processor(s) of the control system 112.

[0037] The display system 110 may display images of the medical instrument and surgical site captured by the visualization system. In some examples, the teleoperated medical system 100 may configure the controls for the medical instrument 104 and master assembly 106 so that the relative positions of the medical instrument resemble the relative positions of the eyes and hands of the operator O. In this way, the operator O can operate the medical instrument 104 and manual controls as if viewing the workspace in substantially true presence. True presence means that the image presentation is a true perspective image that simulates the perspective of a physician physically operating the medical instrument 104.

[0038] In some examples, the display system 110 may present images of the surgical site recorded pre- or intra-operatively using image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, and / or the like. The pre- or intra-operative image data may be presented as two-dimensional, three-dimensional, or four-dimensional images (e.g., including time-based or velocity-based information) and / or as images from a model created from the pre- or intra-operative image dataset.

[0039] In some embodiments, often for purposes of image-guided surgical procedures, the display system 110 may display a virtual navigation image in which the actual location of the medical instrument 104 is aligned (i.e., dynamically referenced) with a pre-operative or concurrent image / model. This may be done to present the operator O with a virtual image of the internal surgical site from the perspective of the medical instrument 104. In some examples, the viewpoint may be from the tip of the medical instrument 104. An image of the tip of the medical instrument 104 and / or other graphical or alphanumeric indicators may be overlaid on the virtual image to help the operator O control the medical instrument 104. In some examples, the medical instrument 104 may not be visible in the virtual image.

[0040] In some embodiments, the display system 110 may display a virtual navigation image in which the actual location of the medical instrument 104 is aligned with the pre-operative or concurrent image to present the operator O with a virtual image of the medical instrument 104 within the surgical site from an external perspective. An image of a portion of the medical instrument 104 or other graphical or alphanumeric indicator may be overlaid on the virtual image to aid the operator O in controlling the medical instrument 104. As described herein, visual representations of data points may be rendered on the display system 110. For example, measured data points, moved data points, aligned data points, and other data points described herein may be displayed on the display system 110 in a visual representation. The data points may be visually represented in the user interface by multiple points or dots on the display system 110 or as a rendered model, such as a mesh or wire model, created based on a set of data points. In some examples, the data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed on the display system 110 after each processing operation is performed to modify a data point.

[0041] The teleoperated medical system 100 may include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for controlling among the medical instruments 104, the master assembly 106, the sensor system 108, and the display system 110. The control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing instructions) for implementing some or all of the methods described in accordance with aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112 is shown as a single block in the simplified schematic diagram of FIG. 1 , the system may include two or more data processing circuits, with one portion of the processing optionally performed on or adjacent to the manipulator assembly 102, another portion of the processing performed on the master assembly 106, and / or other. The processor of the control system 112 may execute instructions, including instructions corresponding to processes disclosed herein and described in more detail below. Any of a wide variety of centralized or distributed data processing architectures may be used. Similarly, the programmed instructions may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the remote control system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

[0042] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical instrument 104. In response to the feedback, the control system 112 may send a signal to the master assembly 106. In some examples, the control system 112 may command one or more actuators of the manipulator assembly 102 to move the medical instrument 104. The medical instrument 104 may extend into an internal surgical site within the body of the patient P through an opening in the body of the patient P. Any suitable conventional and / or specialized actuators may be used. In some examples, the one or more actuators may be separate from or integral with the manipulator assembly 102. In some embodiments, the one or more actuators and the manipulator assembly 102 are provided as part of a remotely operated cart that is positioned adjacent to the patient P and the surgical table T.

[0043] The control system 112 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical instrument 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on reference to acquired preoperative or intraoperative datasets of anatomical passageways. The virtual visualization system processes images of the surgical site captured using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, and / or the like. Software, which may be used in combination with manual input, may convert the recorded images into a segmented two-dimensional or three-dimensional composite representation of a partial or entire anatomical organ or anatomical region. An image dataset is associated with the composite representation. The composite representation and image dataset describe the various locations and shapes of passageways and their connectivity. The images used to generate the composite representation may be captured preoperatively or intraoperatively during a clinical procedure. In some embodiments, the virtual visualization system may use standard representations (i.e., non-patient-specific representations) or a hybrid of standard representations and patient-specific data. The composite representations and virtual images generated by the composite representations may represent the resting posture of a deformable anatomical region during one or more stages of motion (e.g., the inhalation / exhalation cycle of the lungs).

[0044] During the virtual navigation procedure, the sensor system 108 may be used to calculate the approximate location of the medical instrument 104 relative to the patient P's anatomy. The location may be used to generate both macro-level (external) tracking images of the patient P's anatomy and virtual internal images of the patient P's anatomy. The system may implement one or more electromagnetic, fiber optic, and / or other sensors to register and display the medical instrument with pre-operatively registered surgical images, such as pre-operatively registered surgical images from a virtual visualization system. For example, U.S. Patent Application Publication No. 2016 / 0124999 (published December 1, 2016, disclosing "Systems and Methods of Registration for Image Guided Surgery"), which is incorporated herein by reference in its entirety, discloses one such system. The teleoperated medical system 100 may further include optional operation and support systems (not shown), such as a lighting system, a navigation control system, an irrigation system, and / or an aspiration system. In some embodiments, the teleoperated medical system 100 may include more than one manipulator assembly and / or more than one master assembly. The exact number of teleoperated manipulator assemblies depends on, among other factors, the surgical procedure and the spatial constraints within the operating room. The master assemblies 106 may be co-located or may be located in separate locations. Multiple master assemblies allow more than one operator to control one or more teleoperated manipulator assemblies in various combinations.

[0045] 2A is a simplified diagram of a medical instrument system 200 according to some embodiments. In some embodiments, the medical instrument system 200 may be used as the medical instrument 104 in an image-guided medical procedure performed by the teleoperated medical system 100. In some examples, the medical instrument system 200 may be used for a non-teleoperated exploratory procedure or in a procedure involving a conventional manually operated medical instrument, such as an endoscope. Optionally, the medical instrument system 200 may be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical passageway of a patient, such as patient P.

[0046] The medical instrument system 200 includes an elongate device 202, such as a flexible catheter, coupled to a drive unit 204. The elongate device 202 includes a flexible body 216 having a proximal end 217 and a distal end or tip portion 218. In some embodiments, the flexible body 216 has an outer diameter of about 3 mm. The outer diameter of other flexible bodies may be larger or smaller.

[0047] Medical instrument system 200 further includes a tracking system 230 for determining the position, orientation, speed, velocity, attitude, and / or shape of one or more segments 224 along flexible body 216 and / or distal end 218 using one or more sensors and / or imaging devices, as described in more detail below. The entire length of flexible body 216 between distal end 218 and proximal end 217 may be effectively divided into segments 224. Tracking system 230 may optionally be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processor(s) of control system 112 of FIG. 1 .

[0048] Tracking system 230 may optionally use shape sensor 222 to track distal end 218 and / or one or more of segments 224. Shape sensor 222 may optionally include an optical fiber aligned with flexible body 216 (e.g., disposed within an internal channel (not shown) or attached externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the dimensions may be larger or smaller. The optical fiber of shape sensor 222 forms a fiber optic bend sensor for determining the shape of flexible body 216. In one alternative, an optical fiber including a fiber Bragg grating is used to provide strain measurements of the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. Patent No. 6,213,629 (filed July 13, 2005) (disclosing “Fiber optic position and shape sensing device and method relating thereto”), U.S. Patent No. 6,213,629 (filed July 16, 2004) (disclosing “Fiber-optic shape and relative position sensing”), and U.S. Patent No. 6,213,629 (filed June 17, 1998) (disclosing “Optical Fiber Bend Sensor”), all of which are incorporated herein by reference in their entirety. In some embodiments, the sensor may utilize other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the elongated device may be determined using other techniques. For example, the history of the orientation of the distal end of the flexible body 216 may be used to reconstruct the shape of the flexible body 216 over a time interval. In some embodiments, tracking system 230 may optionally and / or additionally track distal end 218 using position sensor system 220 .Position sensor system 220 may be a component of an EM sensor system, with the position sensor system 220 including one or more conductive coils that may be exposed to an externally generated electromagnetic field. Each coil in the EM sensor system then generates an induced electrical signal having characteristics that depend on the coil's position and orientation relative to the externally generated electromagnetic field. In some embodiments, position sensor system 220 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, and Z and three orientation angles indicative of the pitch, yaw, and roll of a fiducial, or five degrees of freedom, e.g., three position coordinates X, Y, and Z and two orientation angles indicative of the pitch and yaw of a fiducial. Further description of position sensor systems is provided in U.S. Patent Application Publication No. 2009 / 0129999 (filed August 11, 1999), which discloses "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked," the entire disclosure of which is incorporated herein by reference.

[0049] In some embodiments, tracking system 230 may alternatively and / or additionally rely on stored historical posture, position, or orientation data for known points of the instrument system along a cycle of alternating motion, such as breathing. This stored data may be used to develop shape information about flexible body 216. In some examples, a series of position sensors (not shown), such as electromagnetic (EM) sensors similar to the sensors in position sensor 220, may be positioned along flexible body 216 and then used for shape sensing. In some examples, particularly if the anatomical passage is generally stationary, a history of data from one or more of these sensors taken during a procedure may be used to represent the shape of elongate device 202.

[0050] The flexible body 216 includes a channel 221 sized and shaped to receive a medical instrument 226. FIG. 2B is a simplified diagram of the flexible body 216 with the extended medical instrument 226, according to some embodiments. In some embodiments, the medical instrument 226 may be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical instrument 226 can be deployed through the channel 221 of the flexible body 216 and used at a target location within the anatomy. The medical instrument 226 may include, for example, an image capture probe, a biopsy device, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tool. The medical instrument may include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end effectors may include, for example, forceps, graspers, scissors, a clip applier, and / or the like. Other end effectors may further include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, and / or the like. In various embodiments, the medical instrument 226 is a biopsy instrument that may be used to remove a sampling of sample tissue or cells from a target anatomical location. The medical instrument 226 may also be used with an image capture probe within the flexible body 216. In various embodiments, the medical instrument 226 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera at or near the distal end 218 of the flexible body 216 to capture images (including video images) that are processed by a visualization system 231 for display and / or provided to a tracking system 230 to support tracking of the distal end 218 and / or one or more of the segments 224. The image capture probe may include a cable coupled to the camera for transmitting the captured image data. In some examples, the image capture instrument may be a fiber optic bundle, such as a fiberscope, that couples to the visualization system 231.The image capture instrument may be single or multi-spectral, capturing image data in one or more of the visible, infrared, and / or ultraviolet spectrums, for example. Alternatively, the medical instrument 226 itself may be the image capture probe. The medical instrument 226 may be advanced from the opening of the channel 221 to perform a procedure and then retracted into the channel when the procedure is completed. The medical instrument 226 may be removed from the proximal end 217 of the flexible body 216 or from another optional instrument port (not shown) along the flexible body 216.

[0051] The medical instrument 226 may additionally contain a cable, linkage, or other actuation control device (not shown) extending between its proximal and distal ends for controllably bending the distal end of the medical instrument 226. Steerable instruments are described in detail in U.S. Patent No. 6,226,999 (filed October 4, 2005) (disclosing "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent No. 6,226,999 (filed September 30, 2008) (disclosing "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entireties.

[0052] Flexible body 216 may also house cables, linkages, or other steering controls (not shown) extending between drive unit 204 and distal end 218 to controllably bend distal end 218, for example, as indicated by dashed-line depiction 219 of distal end 218. In some examples, at least four cables are used to provide independent "up / down" steering to control the pitch of distal end 218 and "left / right" steering to control the yaw of distal end 218. Steerable elongate devices are described in detail in U.S. Patent Application Publication No. 2011 / 012999 (filed October 14, 2011, disclosing "Catheter with Removable Vision Probe"), the entire disclosure of which is incorporated herein by reference. In embodiments in which medical instrument system 200 is actuated by a remotely controlled assembly, drive unit 204 may include a drive input that removably couples to and receives power from a drive element, such as an actuator, of the remotely controlled assembly. In some embodiments, medical instrument system 200 may include a gripping feature, manual actuator, or other component for manually controlling the movement of medical instrument system 200. Elongate device 202 may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator control of bending of distal end 218. In some instances, one or more lumens are defined in the wall of flexible body 216 through which medical instruments can be deployed and used at the target surgical site.

[0053] In some embodiments, medical instrument system 200 may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, for use in pulmonary examination, diagnosis, biopsy, or treatment. Medical instrument system 200 is also suitable for navigation and treatment of other tissues, either naturally or via surgically created connected passageways, in any of a variety of anatomical systems, including the colon, intestines, kidneys and renal calyces, brain, heart, circulatory system including vasculature, and / or similar organs.

[0054] Information from tracking system 230 is transmitted to navigation system 232, where it is combined with information from visualization system 231 and / or a pre-operatively derived model (e.g., an anatomical model of the patient anatomy) to provide real-time position information to a physician or other operator. In some examples, the real-time position information may be displayed on display system 110 of FIG. 1 for use in controlling medical instrument system 200. In some examples, control system 116 of FIG. 1 may utilize the position information as feedback to position medical instrument system 200. Various systems for using fiber optic sensors to register and display surgical instruments with surgical images are provided in U.S. Patent Application Publication No. 2011 / 0129999, filed May 13, 2011, which discloses "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery," the entire disclosure of which is incorporated herein by reference.

[0055] In some examples, the medical instrument system 200 may be remotely operated within the medical system 100 of Figure 1. In some embodiments, the manipulator assembly 102 of Figure 1 may be replaced with direct operator controls. In some examples, the direct operator controls may include various handles and operator interfaces for handheld operation of the instrument.

[0056] 3A and 3B are simplified side views of a patient coordinate space including a medical instrument attached to an insertion assembly according to some embodiments. As shown in FIGS. 3A and 3B, a surgical environment 300 includes a patient P positioned on table T of FIG. 1. The patient P may be stationary within the surgical environment in the sense that significant patient movement is limited by sedation, restraints, and / or other means. Periodic anatomical motion, including breathing and cardiac motion, of the patient P may continue unless the patient is asked to temporarily stop breathing by holding their breath. Thus, in some embodiments, data may be collected at specific respiratory phases and tagged and identified with that phase. In some embodiments, the phase at which data is collected may be inferred from physiological information collected from the patient P. Within the surgical environment 300, a point gathering instrument 304 is coupled to an instrument carriage 306. In some embodiments, the point gathering instrument 304 may use EM sensors, shape sensors, and / or other sensor modalities. The instrument carriage 306 is attached to an insertion stage 308 that is fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable but have a known position within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument carriage 306 may be a component of a manipulator assembly (e.g., manipulator assembly 102) that couples to the point collection instrument 304 to control the insertion motion (i.e., movement along the A-axis) and, optionally, movement of the distal end 318 of the elongate device 310 in multiple directions, including yaw, pitch, and roll. The instrument carriage 306 or insertion stage 308 may include actuators, such as servo motors (not shown), that control the movement of the instrument carriage 306 along the insertion stage 308.

[0057] The elongate device 310 is coupled to an instrument body 312. The instrument body 312 is coupled to and fixed to the instrument carriage 306. In some embodiments, a fiber optic shape sensor 314 is fixed on the instrument body 312 at a proximal point 316. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may be movable with the instrument body 312, although the location of the proximal point 316 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point, such as the distal end 318 of the elongate device 310. The point collector 304 may be substantially similar to that of the medical instrument system 200.

[0058] A position measuring device 320 provides information regarding the position of the instrument body 312 as it moves on the insertion stage 308 along the insertion axis A. The position measuring device 320 may include resolvers, encoders, potentiometers, and / or other sensors that determine the rotation and / or orientation of the actuators that control the movement of the instrument carriage 306 and, consequently, the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be curved or have a combination of curved and linear segments.

[0059] 3A shows the instrument body 312 and instrument carriage 306 in a retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is at a position L0 on axis A. At this position along the insertion stage 308, the A component of the location of the proximal point 316 may be set to zero and / or another reference value to provide a base reference describing the position of the instrument carriage 306, and thus the proximal point 316, on the insertion stage 308. With this retracted position of the instrument body 312 and instrument carriage 306, the distal end 318 of the elongate device 310 may be positioned just inside the entrance orifice of the patient P. Also, at this position, the position measuring device 320 may be set to zero and / or another reference value (e.g., I=0). 3B , instrument body 312 and instrument carriage 306 are advanced along the linear track of insertion stage 308, with distal end 318 of elongate device 310 advanced into patient P. In this advanced position, proximal point 316 is at position L1 on axis A. In some examples, the position L of proximal point 316 relative to position L0 is determined using one or more actuators controlling the movement of instrument carriage 306 along insertion stage 308 and / or encoders and / or other position data from one or more position sensors associated with instrument carriage 306 and / or insertion stage 308. x In some examples, the position L x may further be used as an indicator of the distance or insertion depth that the distal end 318 of the elongate device 310 is inserted into the passageway of the patient P's anatomy.

[0060] With reference to Figures 4, 5, 6, 7, 8, 9, and 10, in some embodiments, the location of a target structure may be confirmed / determined using an imaging probe (e.g., an ultrasound probe) integrated into or delivered by a catheter. In various embodiments, the use of ultrasound technology to determine and / or confirm the target structure location may enable greater accuracy in biopsies or other focal therapies directed at small diameter nodules (e.g., about 10 mm or less). Figures 4, 5, and 6 illustrate catheters including channels that may be used to insert either an imaging probe or a tool. Figures 7, 8, and 9 illustrate catheters configured to simultaneously deliver an imaging probe and a tool. Figure 10 illustrates a method for determining the location of a target structure using an imaging probe.

[0061] The example in Figure 4 shows the target structure P M 1 illustrates a virtual image 400 of a target structure P (e.g., a tumor, lesion, nodule) and a nearby anatomical passageway 402. The passageway 402 includes a passageway wall 404. The passageway 402 is positioned within a patient reference frame having a coordinate system (Xp, Yp, Zp). The patient reference frame may be a fixed reference frame (i.e., a reference frame that does not move during the medical procedure). Multiple images of the passageway 402 (e.g., obtained from pre-operative or intra-operative modeling) may be registered to the patient reference frame using the methods described above with reference to FIG. 1. M The location of is determined in pre-operative or intra-operative imaging and is thus known in the patient's frame of reference. In this embodiment, the anatomical passageway is a bronchial passageway of the lung, although the systems and methods of this disclosure may be suitable for use in other natural or surgically created passageways in anatomical systems, such as the circulatory, digestive, renal, and reproductive systems, including anatomical structures such as the colon, intestines, kidneys, heart, and / or similar organs.

[0062] A flexible catheter body 408 (substantially similar to the flexible body 216) may be navigated (advanced) to a catheter park location that allows access to the target structure. The catheter may be positioned at the catheter park location in an orientation that allows the distal end portion of the catheter to be pointed in a direction toward the target structure. In this manner, a tool inserted through the flexible catheter body 408 is inserted along a pointing vector to the target structure. The catheter may be navigated using, for example, a visual endoscope, an EM sensor, and / or a fiber optic shape-based navigation technique. An imaging probe 406 may be inserted through the flexible catheter body 408. In one embodiment, the imaging probe 406 is an ultrasound probe 406. The ultrasound probe 406 may use an ultrasound transducer, such as a side-facing transducer, a forward-facing transducer, a curved transducer, a radial transducer, and / or the like. In one example, the ultrasound probe uses a side-imaging transducer, including a rotational ultrasound transducer for imaging in a direction generally perpendicular to the transducer's axis of rotation. A side-imaging probe generates cross-sectional (i.e., radial) images along an imaging plane 412. Optionally, the ultrasound probe 406 may be integral with the catheter, rather than being replaceable, as shown in FIG. 4. A catheter reference frame and coordinate system (Xc, Yc, Zc) is defined at the distal end portion 410 of the catheter 408 and is aligned with the patient coordinate system using the method described above with reference to FIG. 1. In the embodiment of FIG. 4, the transducer's axis of rotation is generally along the Zc direction of the catheter reference frame.

[0063] In some aspects, a side-view imaging probe generates images of tissue at a radial distance from the axis of rotation, including tissue located outside the anatomical passageway. In other embodiments, a forward-looking ultrasound probe may be used to image tissue distal to the imaging transducer. The ultrasound probe may be relatively small to navigate narrow anatomical passageways. For example, the ultrasound probe may have a distal tip diameter of approximately 1.4 mm.

[0064] FIG. 5 illustrates an image 500 produced by the ultrasound probe 406 in the imaging plane 412 shown in FIG. 4. The image 500 is based on a reference coordinate system and a coordinate system (X I , Y I , Z I ) has a target structure 502 (P I ) is identified within the image frame of reference. It may be identified, for example, by a clinician, by an image analysis algorithm, or by a combination of the two. The ultrasound scan may be gated for the respiratory cycle and / or the cardiac cycle. Image 500 is two-dimensional, although a three-dimensional image may be constructed from multiple two-dimensional ultrasound scans. The target structure P within the image frame of reference I The data associated with the location of P is stored in the catheter coordinate system or the patient coordinate system (aligned with the catheter coordinate system). P The target structure P determined by the ultrasound probe is P is determined from pre- or intra-operative imaging in the catheter or patient frame of reference to determine the correction vector 414. M The correction vector 414 may be compared to the location of the target structure P M Location and target structure P P is the offset value between the location of

[0065] In various embodiments, to transform the image coordinate system to the catheter coordinate system, the relative three-dimensional pose between the image coordinate system and the catheter coordinate system is determined. In some embodiments, it is measured directly by sensors on the imaging probe and catheter. Such sensors may include EM sensors, fiber optic shape sensors, or the like. The sensors on the imaging probe and catheter need not be of the same type.

[0066] In some embodiments, the relative orientation between the image coordinate system (XI, YI, ZI) and the catheter coordinate system (XC, YC, ZC) can be calculated by measuring the insertion length L and / or roll angle of the image coordinate system with respect to the catheter coordinate system, assuming that the imaging probe extends linearly beyond the tip of the catheter and that the orientation of the catheter tip is known. In some embodiments, the relative orientation is determined based on an assumption of the relative position of the target with respect to the catheter distal tip (e.g., the catheter tip is pointing at the target). In these embodiments, the imaging probe delivered by the catheter is assumed to be located along the aiming direction of the catheter, and thus only the distance from the catheter distal tip needs to be determined. Such distance may be an assumed distance or may be determined using external imaging. In one example, after determining the target location in the catheter reference system and the target location in the image reference system (e.g., from a captured ultrasound image), the transformation between the image reference system and the catheter reference system may be determined.

[0067] In some examples, the insertion length L may be determined using, for example, an encoder or a stepper motor. The roll angle may be determined using various techniques. For example, the ultrasound probe may include a roll alignment configuration that is keyed to the catheter, causing the ultrasound probe to maintain a fixed orientation about a catheter roll axis (Zc) that extends through the catheter. The roll alignment configuration may be used to determine the image reference frame and coordinate system (X I , Y I , Z I) roll angle is calculated based on the catheter reference frame and the coordinate system (X C , Y C , Z C ) to allow the probe to be aligned with the catheter roll axis. In some embodiments, the roll alignment feature is a shaped protrusion keyed to match a similarly shaped channel in the catheter. In an alternative embodiment, the roll alignment feature may be a channel shaped to match a protrusion in the catheter. More than one roll alignment feature may be used to maintain the probe in a fixed orientation about the catheter roll axis. In another alternative embodiment, the roll angle of the image coordinate system relative to the catheter coordinate system may be determined by a roll sensor positioned outside the patient anatomy. In yet another alternative embodiment, the roll angle may be determined by one or more markers or other features with a known angle relative to the catheter in images recorded by the imaging probe. For example, the features or markers may be positioned around the catheter and may have contrast (e.g., ultrasound contrast) relative to the catheter. In another alternative embodiment, the combined roll and insertion length may be determined by observing a pattern on the probe around the proximal end of the catheter, or by observing a pattern on the catheter around the proximal end of the probe. Various registration techniques (eg, optical flow) may be used to register the image coordinate system and the catheter coordinate system.

[0068] Referring to FIG. 6 , in some embodiments, the relative orientation between the image coordinate system and the catheter coordinate system is determined using one or more external images, where each external image includes a catheter image and an imaging probe image. Such external images may be provided by computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, and / or the like (e.g., fluoroscopy). In the example of FIG. 6 , the display system 110 displays a simultaneous or real-time external image 602 of the patient anatomy from an external fluoroscopic imaging device. The external image 602 includes images of the catheter 408, the ultrasound probe 406, and bones within the patient anatomy (e.g., ribs 604-1, 604-2). In some embodiments, an operator may provide input (e.g., using an input device) to identify the location of the distal end 410 of the ultrasound probe 406 and the catheter 408. 6, the display system 110 includes a touchscreen, and the operator may use the touchscreen to identify the distal end 410 of the catheter 408 (e.g., by using the marker "X") and the ultrasound probe 406 (e.g., by using the marker "Δ") in the external image 602. The operator may transmit the identified locations to the control system by selecting button 606.

[0069] In some embodiments, the control system may determine the relative orientation between the image coordinate system and the catheter coordinate system by using an image analysis algorithm that analyzes the image 602 and automatically detects the positions of the ultrasound probe 406 and the distal end 410 of the catheter 408 within the image 602. In some examples, known anatomical landmark dimensions within the image 602 (e.g., the distance d1 between successive clavicles 604-1 and 604-2) may be used to determine the distance d2 between the positions of the ultrasound probe 406 and the distal end 410 of the catheter 408 within the image 602.

[0070] In some embodiments, two or more external images are used to determine the location and / or relative positions of the distal end 410 of the ultrasound probe 406 and the guide catheter 408. By using two or more external images, a three-dimensional image of the patient anatomy, including the ultrasound probe 406 and the catheter 408, may be constructed from the multiple two-dimensional external images based on the location of the distal end 410 of the ultrasound probe 406 and the catheter 408 on each external image. The location of the distal end 410 of the ultrasound probe 406 and the catheter 408 on each external image may be identified based on either operator input or digital image processing performed by a control system. In some embodiments, the two or more external images are provided by the same external imaging device (e.g., a fluoroscopic imaging device) from different viewing directions. In an alternative embodiment, the two or more external images are provided by two or more external imaging devices using different imaging techniques (such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube x-ray imaging).

[0071] 7, 8, and 9, in some embodiments, the catheter may provide an imaging probe (e.g., an ultrasound probe) simultaneously with the tool. In these embodiments, a first image from the imaging probe may include both the tool and the target structure, which may be used to confirm or determine the location of the target structure and the tip of the tool. In some embodiments, additional location data for the tip of the tool may be determined based on sources other than the first image (e.g., an external image of the patient anatomy, a known orientation of the distal end of the catheter), which may be used to determine the location of the target structure. By using the first image from the imaging probe and the location data for the tip of the tool, the location of the target structure may be determined during a medical procedure (e.g., biopsy, resection, etc.). In one example, a virtual visualization image of the patient anatomy may be provided to the operator during a biopsy procedure, which includes an image of the biopsy instrument and an image of the target structure, providing visualization of the insertion of the biopsy instrument into the target structure.

[0072] Referring to the example of FIG. 7 , a virtual image 800 of a target structure 810 and a nearby anatomical passageway 802 is illustrated. The passageway 802 includes a passageway wall 804. As shown in FIG. 7 , a biopsy instrument 806 is inserted into a channel of a catheter 808 during a biopsy procedure. The catheter 808 is steered to a position and / or orientation that provides access to the target structure 810. Steering of the catheter may be achieved directly through active steering controls of the catheter or indirectly by navigating a steerable biopsy instrument 806. The biopsy instrument 806 is extended from the catheter 808 through the wall 804 of the anatomical passageway 802 to contact the target structure 810 to enable a tissue sample to be taken, although in some instances, extension of the biopsy instrument 806 may be delayed until the distal end 814 of the catheter 808 is properly positioned. In alternative examples, the biopsy instrument 806 may be replaced with a treatment device, such as a resection or dissection tool.

[0073] 7 , catheter 808 includes an imaging probe 812 (e.g., an ultrasound probe) integrated into a distal end 814 of catheter 808. In an alternative embodiment, the imaging probe (e.g., an ultrasound probe) may be simultaneously inserted through a second channel of the catheter, while a biopsy instrument is inserted through a first channel of the catheter. Imaging probe 812 is positioned within passageway 802 to acquire intraoperative and real-time images of biopsy instrument 806 and target structure 810. In various embodiments, these images of biopsy instrument 806 and target structure 810 may be used to register images captured by imaging probe 812 and / or further assist in localizing imaging probe 812 relative to target structure 810, biopsy instrument 806, and tool tip 816 of biopsy instrument 806.

[0074] 7, the imaging probe 812 is a forward-facing ultrasound probe that is used to capture images of both the target structure 810 and the biopsy instrument 806 distal to the imaging probe 812 (e.g., as the biopsy instrument 806 penetrates the target structure 810). In an alternative embodiment, the imaging probe 812 includes a lateral imaging probe for capturing images of both the target structure 810 and the biopsy instrument 806 by generating cross-sectional (i.e., radial) images along the imaging plane of the lateral imaging probe 812 (e.g., imaging plane 412 of imaging probe 406 in FIG. 4). In various embodiments, the ultrasound probe may be used to capture images of objects located outside the anatomical passageway (e.g., the target structure 810, a portion of the biopsy instrument 806).

[0075] Referring to the example of Figure 8, an image 850 is illustrated that is generated by the forward-looking ultrasound probe 812 shown in Figure 7. The image 850 is based on an image reference frame and coordinate system (X I , Y I , Z I) and includes an image of the target structure 810 and an image of the biopsy instrument 806, including the tool tip 816. In various embodiments, the target structure 810, biopsy instrument 806, and tool tip 816 are identified within an image reference system. They may be identified, for example, by an operator (e.g., a clinician), by an image analysis algorithm, or by a combination of the two. The image 850 is two-dimensional, although a three-dimensional image may be constructed from multiple two-dimensional ultrasound images. Data associated with the location of the tool tip 816 and target structure 810 within the image reference system is transformed to a catheter coordinate system or a patient coordinate system (aligned with the catheter coordinate system). The location data of the target structure 810 and tool tip 816 within the catheter coordinate system or the patient coordinate system may be used to provide a visual visualization image on the display system 110.

[0076] In various embodiments, in addition to first location data of the tool tip 816 determined based on the image 850 provided by the imaging probe 812, second location data of the tool tip 816 based on a source (e.g., a known orientation of the distal end 814 of the catheter 808, an external image from an external imaging device) may be used to determine the location of the target structure 810. In some embodiments, such second location data of the tool tip 816 is determined based on the known orientation of the distal end 814 of the catheter 808. For example, the second location data of the tool tip 816 may indicate that the tool tip 816 is along the same direction as the distal end 814 of the catheter 808. Further, for example, the second location data of the tool tip 816 may indicate that the tool tip 816 is a predetermined distance (e.g., 15 mm) from the distal end 814 of the catheter 808.

[0077] 9, in some embodiments, second location data for the tool tip 816 may be determined based on an external image of the patient anatomy provided by an external imaging device. Figure 9 illustrates a display system 110 displaying a simultaneous or real-time external image 902 of the patient anatomy from an external fluoroscopic imaging device. The external image 902 includes an image of the catheter 808, the ultrasound probe 812 integrated into the distal end 814 of the catheter 808, the biopsy instrument 806, and the tool tip 816 of the biopsy instrument 806.

[0078] In some embodiments, an operator (e.g., a clinician) may provide input (e.g., using an input device) to identify the location of the distal end 814 of the catheter 808 and the tool tip 816 in the external image 902. In the example of FIG. 9 , the display system 110 includes a touchscreen, and the operator may use the touchscreen to identify the location of the distal end 814 of the catheter 808 (e.g., using the marker "X") and the location of the tool tip 816 (e.g., using the marker "Δ") on the external image 902. The operator may then submit the identified locations to the control system (e.g., using button 906).

[0079] In some embodiments, the control system may use image analysis algorithms to identify the location of the distal end 814 of the catheter 808 and the tool tip 816 in the external image 902. For example, the image analysis algorithm may be used to automatically detect the distal end 814 of the catheter 808 and the tool tip 816 in the external image 902. In some examples, known anatomical landmark dimensions in the external image 902 (e.g., the distance d3 between consecutive clavicles 904-1 and 904-2) may be used to determine the distance d4 between the distal end 814 of the catheter 808 and the tool tip 816.

[0080] Referring to the example of FIG. 10 , a method 1000 of determining a target structure location using an imaging probe is illustrated according to some embodiments. Method 1000 is depicted as a series of operations or processes 1002-1014. Not all of the illustrated processes 1002-1014 may be performed in all embodiments of method 1000. Additionally, one or more processes not explicitly shown in FIG. 10 may be included before, after, between, or as part of processes 1002-1014. In some embodiments, one or more of the processes may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when executed by one or more processors (e.g., processors of control system 112), may cause the one or more processors to perform one or more of the processes.

[0081] Method 1000 begins with process 1002, in which a first image of an anatomical structure is received from an imaging probe (e.g., an ultrasound probe) positioned inside the patient anatomy. In some embodiments, as shown in FIG. 4, the imaging probe 406 is extendable distally beyond the distal end 410 of the catheter 408. In process 1002, the control system 112 receives the image 500 of FIG. 5 of the anatomical structure including the target structure PI from the imaging probe 406 of FIG. 4. In an alternative embodiment, as shown in FIG. 7, the imaging probe 812 is integrated with the catheter 808 and positioned at the distal end 812 of the catheter 808. In process 1002, the control system 112 receives the image 850 of FIG. 8 of the anatomical structure including the target structure 810 captured by the imaging probe 806 of FIG. 7. In yet another alternative embodiment, an instrument (eg, a biopsy instrument) is inserted through a first channel of the catheter while an imaging probe may simultaneously be inserted through a second channel of the catheter.

[0082] In process 1004, control system 112 receives one or more external images of the patient anatomy from an external imaging device located outside the patient anatomy, where such external images include images of a catheter, imaging probe, and / or instrument. In some embodiments, as shown in FIG. 6, control system 112 receives one or more external images 602 of FIG. 6 from an external imaging device located outside the patient anatomy. Each of the external images includes an image of catheter 408 and an image of imaging probe 406 of FIG. 4. In some embodiments, as shown in FIG. 9, control system 112 receives one or more external images 902 of FIG. 9 from an external imaging device located outside the patient anatomy. Each of the external images 902 includes an image of catheter 808, an image of imaging probe 812, and tool 806 of FIG. 8.

[0083] Method 1000 then proceeds to process 1010, where a relative orientation between an image frame of reference associated with the imaging probe and a catheter frame of reference associated with the distal end of the catheter is determined. Such relative orientation may be used to transform location data of the target structure and instrument (e.g., tool tip) in the first image of the image frame of reference to the catheter frame of reference. In some embodiments, the relative orientation may be measured directly by sensors on the imaging probe and catheter. Such sensors may include EM sensors, fiber optic shape sensors, or the like. In alternative embodiments, the relative orientation may be determined by using two or more external images (e.g., provided by external imaging devices having different viewing directions relative to the patient anatomy). In these embodiments, the three-dimensional positions of both the imaging probe and catheter may be determined using two or more external images, and the relative orientation may be determined using those three-dimensional positions of the imaging probe and catheter.

[0084] In some embodiments, process 1010 uses process 1006 to determine the relative position between the imaging probe and the distal end of the catheter based on one or more external images, and process 1008 to determine the relative orientation between the imaging probe and the distal end of the catheter. In those embodiments, process 1010 determines the relative pose using the relative position determined in process 1006 and the relative orientation determined in process 1008.

[0085] In some embodiments, in process 1006, the relative position between the imaging probe and the distal end of the catheter is provided by assuming a fixed distance (e.g., 15 mm) between the imaging probe and the distal end of the catheter and by assuming the imaging probe is aligned in the same direction as the distal end of the catheter. In another alternative embodiment, as shown in FIGS. 6 and 9, the relative position between the imaging probe and the distal end of the catheter is determined based on one or more external images (e.g., using operator input, image analysis algorithms, and / or a combination thereof). For example, as shown in FIG. 6, the positions of the distal end 410 of the catheter 408 and the imaging probe 406 in the external image 602 may be identified by operator input, image analysis algorithms, and / or a combination thereof.

[0086] In process 1008, a relative orientation between the imaging probe and the distal end of the catheter is determined. In some embodiments, in process 1008, the catheter is navigated to a catheter parking location, where the distal end of the catheter is aligned with an anatomical landmark (e.g., a bifurcation). In these embodiments, the relative orientation between the imaging probe and the distal end of the catheter is determined based on the same anatomical landmark detected in the first image from the imaging probe (e.g., using operator input, an image analysis algorithm, and / or a combination thereof). In alternative embodiments, the imaging probe is integrated directly into the catheter to maintain a fixed orientation. In one example, the imaging probe may be embedded at the distal end of the catheter. In another example, the imaging probe may include a roll alignment feature keyed to the catheter to cause the imaging probe to maintain a fixed orientation about a catheter roll axis (Zc) extending through the catheter. In yet another example, the imaging probe is at least partially surrounded by a sheath that is rotatably fixed (e.g., keyed) relative to the catheter. The keyed sheath may include one or more markers (e.g., echogenic features) to create a distinguishable region in the first image, which may then be used as a reference roll angle between the imaging probe and the distal end of the catheter. In yet another alternative embodiment, the roll angle between the imaging probe and the catheter may be determined using a roll sensor positioned outside the patient anatomy. In another embodiment, the catheter is oriented so that the target is approximately centered in the image. In that embodiment, the catheter is oriented so that the target is straight ahead along the catheter shaft, and a relative orientation between the imaging probe and the distal end of the catheter is not required.

[0087] In process 1012, for a catheter that simultaneously delivers an imaging probe and a tool (e.g., during a biopsy), first and second location data for the tip of the tool are determined. As shown in Figure 8, the first location data of the tool tip is determined using an image 850 provided by the imaging probe 812 of Figure 7. As shown in Figure 9, the second location data of the tool tip is determined based on an external image 902 provided by an external imaging device (e.g., using operator input, an image analysis algorithm, and / or a combination thereof).

[0088] In process 1014, location data of the target structure in the catheter reference system and / or the patient reference system may be determined based on the location data of the target structure in the image reference system, the relative orientation between the imaging probe and the distal end of the catheter, the first tool tip location data, the second tool tip location data, and / or a combination thereof. For example, the location data of the target structure in the first image may be transformed from the image reference system to the catheter reference system based on the relative orientation. Further, for example, the location data of the target structure in the catheter reference system may be transformed to the patient reference system. Using the location data of the target structure in the catheter reference system and / or the patient reference system, a virtual visualization of the patient anatomical structure including the target structure may be provided to the operator.

[0089] With reference to Figures 11, 12A, 12B, 13, and 14, various visual visualization images may be provided to an operator using location data of a target structure within the catheter reference frame and / or the patient reference frame. Figure 11 illustrates a method 1100 for presenting a target structure in a virtual visualization image on a display system using location data of the target structure determined based on a first image from an imaging probe. Figures 12A-12B illustrate various virtual visualization images including a virtual representation of the target structure in an anatomical model of the patient anatomy. Figures 13 and 14 illustrate various virtual visualization images including a virtual representation of the target structure in an external image of the patient anatomy.

[0090] 11 , illustrated is a method 1100 for presenting a target structure in a virtual visualization image on a display system using location data of the target structure determined based on a first image from an imaging probe. Method 1100 is illustrated as a series of operations or processes 1102-1110. Not all of the illustrated processes 1102-1110 may be implemented in all embodiments of method 1100. Additionally, one or more processes not explicitly illustrated in FIG. 11 may be included before, after, between, or as part of processes 1102-1110. In some embodiments, one or more processes may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when executed by one or more processors (e.g., processors of control system 112), may cause the one or more processors to perform one or more of the processes.

[0091] Method 1100 begins with process 1102, in which location data of a target structure transformed into a visualization frame of reference (e.g., a model frame of reference associated with an anatomical model or a patient frame of reference associated with an external image) is received. The location data of the target structure in the visualization frame of reference may be determined based on images provided by an imaging probe, as described in method 1000.

[0092] In process 1104, in some embodiments, a virtual visualization image including an image of a patient anatomical model of the patient anatomy and an image of the target structure is provided to the operator. In these embodiments, the visualization frame of reference is a model frame of reference. Referring to FIG. 12A , in some examples, the virtual visualization image includes an overall view of the anatomical model. Specifically, as shown in FIG. 12A , in some examples, the display system 110 displays a virtual visualization image 1202 of the patient anatomy as an anatomical model 1204, including a passageway. The anatomical model 1204 can be generated pre-operatively or intra-operatively from data collected using an external imaging source, as described above. The virtual visualization image 1202 can additionally provide a visual representation of a simultaneous, updated, or real-time image of the target structure 1214 using location data of the target structure 1214 within the model frame of reference. In some embodiments, the image of the target structure 1214 is a three-dimensional image generated using multiple images provided by an imaging probe. By projecting an image of the target structure 1214 with its correct location and orientation in the virtual visualization image 1202 based on one or more images provided by the imaging probe, the virtual visualization image 1202 provides the operator with improved spatial awareness of the location of the target structure where the lesion is located. Additionally, the virtual visualization image 1202 can include a simultaneous or real-time virtual image of the catheter 1206 within the passageway 1208 of the anatomical model 1204 and the tool 1212 extending from the distal end 1210 of the catheter 1206.

[0093] 12B, in some examples, the display system 110 displays a simultaneous or real-time virtual visualization 1250 of the patient anatomy, including an interior view of the passageway 1208 of the anatomical model 1204 of FIG. 12A. The virtual visualization 1250 provides a virtual image of the catheter 1206 and the tool 1212 extending from the distal end 1210 of the catheter 1206.

[0094] In process 1106, in some embodiments, the virtual visualization image includes an image on an anatomical model of the patient anatomy that includes an updated target location of the target structure determined based on the image provided by the imaging probe. Such an updated target location may further provide a better aiming direction for optimal access to the target structure.

[0095] In some embodiments, in process 1106, the virtual visualization image includes an indication of an optimal catheter parking location for optimal access to the target structure based on images provided by the imaging probe. In some embodiments, the optimal catheter parking location is determined based on multiple sequential images captured by the imaging probe when the imaging probe is at multiple different locations in the corridor 1208 (e.g., different locations of a segment of the corridor 1208 that allows for capturing images of the target structure) and / or different orientations. For each of those sequential images, a target structure surface area measurement is calculated (e.g., using the ratio of the number of target structure pixels over the total number of pixels in the image). In some embodiments, the target structure surface area measurement may guide the imaging probe to an optimal orientation that maximizes the target structure surface area measurement. In some embodiments, an optimal catheter parking location and / or an optimal catheter orientation (e.g., for performing a biopsy) is determined based on the optimal imaging probe orientation.

[0096] In some embodiments, automated optimization may be used to generate a catheter drive path that is determined based on the target structure surface area measurement. The catheter drive path allows the catheter to be driven along the gradient of the target structure surface area measurement to achieve an optimal target structure surface area measurement. In some examples, the optimal catheter parking location and optimal catheter orientation can be used to drive automated scanning with the catheter tip within the region.

[0097] In process 1108, a virtual visualization image is provided to the operator, including an external image from an external imaging device outside the patient anatomy and an image of the target structure. The image of the target structure is provided in the virtual visualization image using location data of the target structure in the patient reference frame. For example, as shown in FIG. 13 , the display system 110 includes a virtual visualization image 1300 including a real-time external image 1302 during a biopsy. The real-time external image 1302 includes an image of the catheter 808, the imaging probe 812 integrated into the distal end 814 of the catheter 808, the tool 806, and the tool tip 816. An image of the target structure 1304 is provided in the virtual visualization image using location data of the target structure provided by the imaging probe in the patient reference frame. In the example of FIG. 13 , virtual visualization of the insertion of the tool tip 816 into the target structure 1304 during a biopsy is achieved by using the imaging probe to capture images of both the target structure 1304 and the tool tip 816.

[0098] In process 1110, a virtual visualization image is provided to the operator, including a reference catheter image based on imaging probe location data and an external image from an external imaging device outside the patient anatomy. In the example of FIG. 14 , the display system 110 includes a virtual visualization image 1402. The virtual visualization image 1402 includes a real-time external image 1404 during the biopsy, including a real-time image of a catheter 1406. In process 1110, the imaging probe is removed from the catheter channel, and a tool 1410 (e.g., a biopsy needle) is inserted into the channel. In the example of FIG. 14 , an image of a target structure 1408 is provided in the external image 1404 based on location data of the target structure in the patient reference frame. Additionally, a reference image of a catheter 1412 is provided in the external image 1404 based on pre-acquired location data of the imaging probe and catheter when the imaging probe is actuated to capture an image of the target structure.

[0099] The disclosed systems and methods may be used for the navigation and treatment of connected bronchial passages in the lungs. The systems and methods may also be suitable for the navigation and treatment of other tissues through natural or surgically created connected passages in any of a variety of anatomical systems, including the colon, intestines, kidneys, brain, heart, circulatory system, or equivalent organs. The systems and methods may also be suitable for navigation around traceable surfaces of organs. The disclosed methods and embodiments are also suitable for non-surgical applications.

[0100] One or more elements of an embodiment of the present invention may be implemented in software to run on a processor of a computer system such as the control system 112. When implemented in software, the elements of an embodiment of the present invention are essentially code segments to perform the necessary tasks. Programs or code segments can be stored in a processor-readable storage medium or device that may be downloaded by a computer data signal embodied in a carrier wave over a transmission medium or a communication link. Processor-readable storage devices may include any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy diskettes, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments may be downloaded via a computer network, such as the Internet, an intranet, or the like.

[0101] It should be noted that the processes and displays presented may not inherently relate to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the described operations. The required structure for a variety of these systems appears as elements in the claims. Additionally, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0102] While certain exemplary embodiments of the present invention are described and shown in the accompanying drawings, it is to be understood that such embodiments are by way of example only and are not intended to limit the broad invention, and that embodiments of the present invention are not limited to the specific configurations and arrangements shown and described, since various modifications may occur to those skilled in the art. [Prior art documents] [Patent documents]

[0103] [Patent Document 1] International Publication No. 2016 / 191298 [Patent Document 2] U.S. Patent Application Serial No. 11 / 180,389 [Patent Document 3] U.S. Patent Application Serial No. 12 / 047,056 [Patent Document 4] U.S. Patent No. 6,389,187 [Patent Document 5] U.S. Patent No. 6,380,732 [Patent Document 6] U.S. Patent No. 7,316,681 [Patent Document 7] U.S. Patent Application Serial No. 12 / 286,644 [Patent Document 8] U.S. Patent Application Serial No. 13 / 274,208 [Patent Document 9] U.S. Patent Application Serial No. 13 / 107,562

Claims

1. a catheter including a first channel and a distal end, the distal end associated with a catheter reference frame; a processor, the processor is configured to receive a first image captured by a first imaging device provided by the catheter and positioned within a patient anatomy, the first image associated with an image reference frame; the processor is configured to receive a second image of the patient anatomy captured by a second imaging device, the second imaging device being positioned outside the patient anatomy, the second image including the catheter and the first imaging device; The processor is configured to determine a relative orientation between the first imaging device and the distal end of the catheter, the relative orientation being determined by identifying a location of the first imaging device and a location of the distal end of the catheter in the second image, the location of the first imaging device and the location of the distal end of the catheter being determined by: performing an image analysis of the second image; or receiving a user input identifying the location of the first imaging device in the second image and receiving a user input identifying the location of the distal end of the catheter in the second image; is determined by the processor is configured to determine a target location associated with a target structure in the first image; the processor is configured to transform the target location in the image frame of reference to the catheter frame of reference based on the determined relative pose. Healthcare system.

2. The medical system of claim 1 , wherein the first imaging device includes an ultrasound probe.

3. The medical system of claim 2 , wherein the ultrasound probe comprises a side-imaging ultrasound probe, a forward-looking ultrasound probe, a curved ultrasound transducer, or a radial ultrasound transducer.

4. The medical system of claim 2 , wherein the ultrasound probe is configured to be slidably received within the first channel of the catheter.

5. The medical system of claim 4 , wherein a tool is configured to be slidably received within the second channel of the catheter.

6. The medical system of claim 5 , wherein the tool comprises a biopsy needle, an excision device, or an endoscope.

7. The processor: determining a tool tip location of the tip of the tool in the first image; transforming the tool tip location from the image frame of reference to the catheter frame of reference; further configured as follows: The medical system of claim 5 .

8. The medical system according to any one of claims 2 or 3 to 7, wherein the ultrasound probe is positioned near the distal end of the catheter.

9. Determining the relative orientation between the first imaging device and the distal end of the catheter includes: determining a first distance between the distal end of the catheter and the first imaging device using the second image; The medical system according to any one of claims 1 or 2 to 7.

10. 10. The medical system of claim 9, wherein the first distance between the distal end of the catheter and the first imaging device is measured in the second image based on input associated with the second image from an operator.

11. 10. The medical system of claim 9, wherein the first distance between the distal end of the catheter and the first imaging device is automatically measured by performing image analysis on the second image.

12. The medical system of claim 9 , wherein the first distance is determined based on an anatomical landmark dimension in the second image.

13. The medical system of claim 12 , wherein the anatomical landmark dimension comprises a second distance of a given spacing between given points of two consecutive clavicles of the patient anatomy.

14. Determining the relative orientation between the first imaging device and the distal end of the catheter includes: receiving a third image of the patient anatomy captured by the second imaging device from outside the patient anatomy, the third image including the catheter and the first imaging device; determining a relative position between the first imaging device and the distal end of the catheter based on the second image and the third image. The medical system of claim 11.

Images