Graphical User Interface for Monitoring Image-guided Procedures

Abstract

Systems and methods for monitoring a procedure include a medical device including an elongate device including a flexible body, a tracking system disposed along at least a portion of the flexible body, and one or more processors communicatively coupled to the tracking system, wherein the one or more processors are configured to receive a route to a target location within an anatomical structure, determine one or more characteristics of the route based on a first anatomical representation, generate a reduced-scale anatomical representation based on the one or more characteristics of the route, receive real-time position information from the tracking system, associate the real-time position information with the reduced-scale anatomical representation, and dynamically display the reduced-scale anatomical representation along with the associated real-time position information.

Description

[Technical field]

[0001] Related Applications This disclosure claims priority to U.S. Provisional Patent Application No. 62 / 486,879, entitled “Graphical User Interface for Monitoring an Image-guided Procedure,” filed April 18, 2017, which is incorporated by reference in its entirety.

[0002] The present disclosure relates to systems and methods for performing medical procedures, and more particularly, to systems and methods for monitoring image-guided procedures using a graphical user interface. [Background technology]

[0003] Minimally invasive medical techniques are intended to reduce patient recovery time, discomfort, and adverse side effects by reducing the amount of tissue damaged during medical procedures. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, or biopsy instruments) to reach the location of the target tissue. One such minimally invasive technique is the use of flexible and / or steerable elongated devices, such as catheters, that can be inserted into anatomical passages and navigated toward the region of interest in the patient's anatomy. Control of such elongated devices by medical personnel during image-guided procedures involves management of several degrees of freedom, including at least management of the insertion and pullback of the elongated device, as well as management of the steering and / or bending radius of the device. Additionally, different modes of operation may also be supported.

[0004] It would therefore be advantageous to provide a graphical user interface that supports intuitive control and management of medical instruments, including flexible and / or steerable elongated devices, such as steerable catheters, suitable for use in minimally invasive medical techniques. Summary of the Invention

[0005] Embodiments of the present invention are best summarized by the claims that follow the detailed description.

[0006] According to some embodiments, a method for displaying an anatomical structure may include providing a graphical user interface; receiving a first anatomical representation including a 3D representation of a plurality of passageways in the anatomical structure and a route to a target location within the plurality of passageways; generating a reduced anatomical representation based on a subset of the plurality of passageways, the subset of the plurality of passageways including a path passageway directly connected to the route; and displaying the reduced anatomical representation as a linear anatomical representation. Implementations may include one or more of the following features: In the method, the anatomical structure corresponds to a lung and the plurality of passageways in the anatomical structure correspond to airways of the lung. In the method, the target location includes one or more of a lesion, a nodule, and a tumor. In the method, a width of the path passageway is indicated in the linear anatomical representation by vertically spaced lines with graduated spacing. In the method, the graduated spacing of the vertically spaced lines is tiered for path passageways with higher branching generations. In the method, a location of a branch off the path passageway is included in the linear anatomical representation. In the method, the bifurcation location includes a truncated bifurcation that does not have a complete bifurcation structure of the bifurcation. The method further includes displaying an alternate route indicator representing an alternate route to the target location. The method further includes displaying a target icon when one or more of the bifurcations lead to a second target location. The method further includes displaying an insertion trajectory from the end of the route to the target location. The method further includes identifying hazards associated with the route and displaying the location of the hazards relative to the insertion trajectory. In the method, the hazards include one or more of a pleura, a blood vessel, a large blister, and a heart. The method further includes receiving real-time location information associated with the instrument during traversal of the route; mapping the real-time location information to a reduced-scale anatomical representation; and dynamically displaying the real-time location information with the reduced-scale anatomical representation.In the method, dynamically displaying real-time location information includes displaying one or more indicators when an anomaly is detected. In the method, the one or more indicators include a turn error indicator when the anomaly causes the instrument to be steered down an incorrect path. In the method, the one or more indicators include a reverse indicator when the anomaly causes the instrument to be driven beyond the end of the route. In the method, the one or more indicators include an over bend indicator when the anomaly includes an instrument with a tight bend radius. In the method, the appearance of the instrument in the reduced anatomical model is altered when the anomaly includes an instrument with a tight bend radius.

[0007] According to some embodiments, a medical device may include an elongated device including a flexible body, a tracking system disposed along at least a portion of the flexible body, and one or more processors communicatively coupled to the tracking system. The one or more processors are configured to receive a route to a target location in an anatomical structure, determine one or more characteristics of the route based on a first anatomical representation, generate a reduced anatomical representation based on the one or more characteristics of the route, receive real-time location information from the tracking system, associate the real-time location information with the reduced anatomical representation, and dynamically display the reduced anatomical representation with the associated real-time location information. Implementations may include one or more of the following features: In the medical device, the anatomical structure corresponds to a lung, and the first anatomical representation includes a plurality of passageways in the anatomical structure corresponding to airways of the lung. In the medical device, the reduced anatomical representation is displayed as a linear anatomical representation. In the medical device, the one or more characteristics of the route include a location of a bifurcation along the route. In the medical device, the location of the bifurcation in the reduced anatomical representation includes a truncated bifurcation that does not have a complete bifurcation structure of the bifurcation. In the medical device, an alternate route indicator is displayed representing an alternate route to the target location. In the medical device, a target icon is displayed when one or more bifurcations lead to a second target location. In the medical device, the one or more characteristics of the route include an insertion trajectory from an end of the route to the target location. In the medical device, the one or more characteristics of the route include a risk associated with the route. In the medical device, dynamically displaying the real-time location information includes displaying one or more indicators when an anomaly is detected. In the medical device, the one or more indicators include one or more of a turn error indicator, a reversal indicator, and an overbend indicator.

[0008] According to some embodiments, a method of displaying a target in an anatomical structure may include providing a graphical user interface; receiving an anatomical representation via the graphical user interface; determining a target relative to the anatomical representation; determining an uncertainty zone associated with the target; and displaying the target relative to the anatomical representation, the uncertainty zone being displayed to at least partially surround the target. Implementations may include one or more of the following features: In the method, the anatomical representation is registered to the anatomical structure, and the uncertainty zone is determined based on an uncertainty of the registration. In the method, the uncertainty zone is determined based on a predetermined size of the target. In the method, the uncertainty zone is determined based on an accessibility of the target. The method further includes determining a location of a potential hazard. In the method, the location of the potential hazard is related to a location of the target. In the method, the potential hazard is displayed relative to the target. In the method, the potential hazard includes one or more of a pleura, a blood vessel, and a large blister in a lung of the patient.

[0009] According to some embodiments, the system may include a display system, a user input device, and one or more processors configured to perform operations. The operations include receiving an anatomical representation, receiving a user input associated with the target on the anatomical representation via the user input device, determining an uncertainty zone associated with the target, and displaying the target relative to the anatomical representation via the display system, the uncertainty zone being displayed to at least partially surround the target. Implementations may include one or more of the following features: In the system, the anatomical model is registered to the anatomical structure, and the uncertainty zone is determined based on an uncertainty of the registration. In the system, the uncertainty zone is determined based on a predetermined size of the target. In the system, the uncertainty zone is determined based on an accessibility of the target. The system further includes identifying a location of a potential hazard. In the system, the location of the potential hazard is related to a location of the target. In the system, the potential hazard is displayed relative to the target. In the system, the potential hazard includes one or more of a blood vessel, a large blister, and a pleura in the patient's lungs.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. [Brief description of the drawings]

[0011] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the agency upon request and payment of the necessary fee. [Figure 1] FIG. 1 is a schematic diagram of a remote medical system according to some embodiments. [Figure 2A] 1 is a schematic diagram of a medical instrument system according to some embodiments. [Figure 2B] 1 is a schematic diagram of a medical instrument including an expanded medical tool, according to some embodiments. [Figure 3A] FIG. 1 is a schematic diagram of a side view of a patient coordinate space including a medical instrument attached to an insertion assembly, according to some embodiments. [Figure 3B] FIG. 1 is a schematic diagram of a side view of a patient coordinate space including a medical instrument attached to an insertion assembly, according to some embodiments. [Figure 4] 1 is a schematic diagram of a graphical user interface displayable on a display system according to some embodiments. [Figure 5A] 1 is a schematic diagram of a graphical user interface in one mode according to some embodiments. [Figure 5B] 1 is a schematic diagram of a graphical user interface in one mode according to some embodiments. [Figure 5C] 1 is a schematic diagram of a graphical user interface in one mode according to some embodiments. [Figure 5D] 1 is a schematic diagram of a graphical user interface in one mode according to some embodiments. [Figure 6A] 1A-1C are schematic diagrams of dynamic point cloud views at two different times during the registration process, according to some embodiments. [Figure 6B] 1A-1C are schematic diagrams of dynamic point cloud views at two different times during the registration process, according to some embodiments. [Figure 7A] 1A-1C are schematic diagrams of dynamic alignment guidance views at two different times, according to some embodiments. [Figure 7B] 1A-1C are schematic diagrams of dynamic alignment guidance views at two different times, according to some embodiments. [Figure 8] 1 is a schematic diagram of a driving force indicator according to some embodiments. [Figure 9] 1 is a schematic diagram of a bend indicator according to some embodiments. [Figure 10]1 is a schematic diagram of a global anatomical representation (e.g., a model) according to some embodiments. [Figure 11A] 1A-1C are schematic diagrams of reduced anatomical representations (e.g., models) at three different times, according to some embodiments. [Figure 11B] 1A-1C are schematic diagrams of reduced anatomical representations (e.g., models) at three different times, according to some embodiments. [Figure 11C] 1A-1C are schematic diagrams of reduced anatomical representations (e.g., models) at three different times, according to some embodiments. [Figure 12A] 1A-1C are schematic diagrams of live camera feeds at two different times, according to some embodiments. [Figure 12B] 1A-1C are schematic diagrams of live camera feeds at two different times, according to some embodiments. [Figure 13A] 1A-1C are schematic diagrams of virtual tip views at two different times, according to some embodiments. [Figure 13B] 1A-1C are schematic diagrams of virtual tip views at two different times, according to some embodiments. [Figure 13C] 1A-1C are schematic diagrams of virtual tip views at two different times, according to some embodiments. [Figure 13D] 1A-1C are schematic diagrams of virtual tip views at two different times, according to some embodiments. [Figure 14] 1A-1C are schematic diagrams of a series of views during a scenario in which a catheter is inserted beyond the end of the planned route, according to some embodiments. [Figure 15A] 1A-1C are schematic diagrams of remote image views in certain modes according to some embodiments. [Figure 15B] 1A-1C are schematic diagrams of remote image views in certain modes according to some embodiments. [Figure 15C] 1A-1C are schematic diagrams of remote image views in certain modes according to some embodiments. [Figure 16]FIG. 1 is a schematic diagram of a dual screen display for displaying a graphical user interface, according to some embodiments. [Figure 17] 1 is a schematic diagram of a method for monitoring a medical procedure according to some embodiments. [Figure 18] 1 is a schematic diagram of a method for monitoring a medical procedure using a graphical user interface, according to some embodiments. [Figure 19] 1 is a schematic diagram of a method for displaying a patient's anatomy using a graphical user interface, according to some embodiments. [Figure 20] FIG. 1 is a schematic diagram of a method for displaying targets within a patient's anatomy using a graphical user interface, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description, in which like reference numerals have been used to identify like elements shown in one or more of the figures, and it should be understood that the representations in the figures are for purposes of illustrating, and not for purposes of limiting, embodiments of the present disclosure.

[0013] In the following description, certain details are described to describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative and not limiting. One skilled in the art may recognize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. Furthermore, to avoid unnecessary repetition, one or more features illustrated and described in connection with one embodiment may be incorporated into other embodiments, unless otherwise specified or unless one or more features render the embodiment non-functional.

[0014] In some instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the embodiments.

[0015] This disclosure describes various instruments and parts of instruments with respect to the state of those instruments and parts of instruments in three-dimensional space. As used herein, the term "position" refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along orthogonal x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational configuration of an object or part of an object (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0016] 1 is a schematic diagram of a teleoperated medical system 100 according to some embodiments. In some embodiments, the teleoperated medical system 100 may be suitable for use, for example, in surgical, diagnostic, therapeutic, or biopsy procedures. As shown in FIG. 1, the medical system 100 generally includes a teleoperated manipulator assembly 102 for manipulating a medical instrument 104 in performing various procedures on a patient P. The teleoperated manipulator assembly 102 is mounted on or near an operating table T. A master assembly 106 allows an operator (e.g., a surgeon, clinician, or physician O, as shown in FIG. 1) to view the intervention site and control the teleoperated manipulator assembly 102.

[0017] The master assembly 106 may be located in a surgeon console, which is typically located in the same room as the operating table T, such as beside the operating table on which the patient P is positioned. However, it should be understood that the physician O may be located in a different room than the patient P, or in a completely different building. The master assembly 106 generally includes one or more controllers for controlling the teleoperated manipulator assembly 102. The controllers may include any number of different input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controls, voice recognition devices, motion sensors, or presence detection sensors. The controllers may be provided with the same degrees of freedom as the associated medical instrument 104, to provide the physician O with a strong sense of direct control of the instrument 104. In this way, the controllers provide the physician O with telepresence, i.e., the perception that the controllers are one with the medical instrument 104.

[0018] In some embodiments, the controller may have more or fewer degrees of freedom than the associated medical instrument 104 and still provide telepresence to the physician O. In some embodiments, the controller may optionally be a manual input device that moves with six degrees of freedom, which may also include an actuatable handle for actuating the instrument (e.g., to close gripping jaws, apply electrical potentials to electrodes, deliver therapy, etc.).

[0019] The teleoperated manipulator assembly 102 supports the medical instrument 104 and may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that can be manually positioned and fixed in place, commonly referred to as a setup structure) and a teleoperated manipulator. The teleoperated manipulator assembly 102 may optionally include a number of actuators or motors that actuate inputs on the medical instrument 104 in response to commands from a control system (e.g., control system 112). The actuators may optionally include a drive system that, when coupled to the medical instrument 104, can advance the medical instrument 104 into a natural or surgically created anatomical orifice. Other drive systems can move the tip of the medical instrument 104 with multiple degrees of freedom, which may include three degrees of freedom of linear motion (e.g., linear motion along X, Y, and Z coordinate axes) and three degrees of freedom of rotational motion (e.g., rotation about X, Y, and Z coordinate axes). Additionally, these actuators can be used to actuate an articulatable end effector of the medical instrument 104 to grasp tissue in jaws, such as a biopsy device. Actuator position sensors, such as resolvers, encoders, potentiometers, and other mechanisms, can provide sensor data to the medical system 100 indicative of the rotation and orientation of the motor shaft. This position sensor data can be used to determine the movement of an object manipulated by the actuator.

[0020] The teleoperated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information regarding the instrument of the teleoperated manipulator assembly 102. Such subsystems may include a position / localization sensor system (e.g., an electromagnetic (EM) sensor system), a shape sensor system for determining the position, orientation, speed, velocity, attitude, and / or shape of the tip and / or one or more segments along a flexible body that may comprise the medical instrument 104, and / or a visualization system for capturing images from the tip of the medical instrument 104.

[0021] The teleoperated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and the medical instruments 104 generated by the subsystems of the sensor system 108. The display system 110 and the master assembly 106 may be oriented such that the physician O can control the medical instruments 104 and the master assembly 106 using a telepresence perception.

[0022] In some embodiments, the medical instrument 104 may include a visualization system (described in more detail below) that may include a field scope assembly that records concurrent or real-time images of the surgical site and provides the images to the operator or physician O via one or more displays of the medical system 100 (such as one or more displays of the display system 110). The concurrent images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope element that may be integrally or removably coupled to the medical instrument 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with the medical instrument 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors (which may include the processors of the control system 112).

[0023] The display system 110 may also display images of the surgical site and medical instruments captured by the visualization system. In some examples, the teleoperated medical system 100 may configure the controls of the medical instruments 104 and master assembly 106 such that the relative positions of the medical instruments are similar to the relative positions of the eyes and hands of the physician O. In this way, the physician O can manipulate the medical instruments 104 and hand controls as if he were viewing the workspace in substantially true presence. True presence means that the presentation of images is a true perspective image that simulates the perspective of a physician physically maneuvering the medical instruments 104.

[0024] In some examples, the display system 110 may present images of a surgical site recorded pre- or intra-operatively using image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, etc. The pre- or intra-operative image data may be presented as two-dimensional, three-dimensional, or four-dimensional (e.g., including time-based or velocity-based information) images and / or as images from representations (e.g., models) created from the pre- or intra-operative image datasets.

[0025] In some embodiments, for purposes of image-guided medical procedures, in most cases, the display system 110 may display a virtual navigation image in which the actual position of the medical instrument 104 is aligned (i.e., dynamically referenced) with a pre-operative or simultaneous image from a representation (e.g., a model). This may be done to present the physician O with a virtual image of the internal surgical site from the perspective of the medical instrument 104. In some examples, the perspective may be from the tip of the medical instrument 104. An image of the tip of the medical instrument 104 and / or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist the physician O in controlling the medical instrument 104. In some examples, the medical instrument 104 may not be visible in the virtual image.

[0026] In some embodiments, the display system 110 may display a virtual navigation image that aligns the actual position of the medical instrument 104 with the preoperative or concurrent images to present the physician O with a virtual image of the medical instrument 104 within the surgical site from an external perspective. Images of parts of the medical instrument 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist the physician O in controlling the medical instrument 104. As described herein, visual representations of the data points may be rendered on the display system 110. For example, measured data points, moved data points, aligned data points, and other data points described herein may be displayed in visual representations on the display system 110. The data points may be visually represented in a user interface by a number of points or dots on the display system 110 or as a rendered representation (e.g., a model), such as a mesh or wire model created based on a set of data points. In some examples, the data points may be color-coded according to the data they represent. In some embodiments, the visual representations may be refreshed on the display system 110 after each processing operation is performed to modify the data points.

[0027] The teleoperated medical system 100 may also include a control system 112. The control system 112 includes at least one computer processor (not shown) and at least one memory for controlling between the medical instruments 104, the master assembly 106, the sensor system 108, and the display system 110. The control system 112 also includes program instructions (e.g., a non-transitory machine-readable medium storing instructions) for implementing some or all of the methods described according to the aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112 is shown as a single block in the schematic diagram of FIG. 1, the system may include two or more data processing circuits, some of which processing is optionally performed on or near the teleoperated manipulator assembly 102 and another part of which is performed on the master assembly 106, etc. The processor of the control system 112 may execute instructions including instructions corresponding to the processes disclosed herein and described in more detail below. Any of a wide variety of centralized or distributed data processing architectures may be used. Similarly, the program instructions may be implemented as separate programs or subroutines, or the instructions may be integrated into other aspects of the remote control system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA, Home RF, IEEE 802.11, DECT, and wireless telemetry.

[0028] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical instrument 104. In response to the feedback, the control system 112 may send a signal to the master assembly 106. In some examples, the control system 112 may send a signal to command one or more actuators of the teleoperated manipulator assembly 102 to move the medical instrument 104. The medical instrument 104 may extend into an internal surgical site within the body of the patient P through an orifice in the patient P's body. Any suitable conventional and / or dedicated actuators may be used. In some examples, the one or more actuators may be separate from or integrated with the teleoperated manipulator assembly 102. In some embodiments, the one or more actuators and the teleoperated manipulator assembly 102 are provided as part of a teleoperated cart that is positioned adjacent to the patient P and the surgical table T.

[0029] The control system 112 may optionally further include a virtual visualization system for providing navigational assistance to the physician O when controlling the medical instrument 104 during an image-guided medical procedure. The virtual navigation using the virtual visualization system may be based on reference to acquired pre-operative or intra-operative data sets of anatomical passageways. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, etc. Software, which may be used in combination with manual input, is used to convert the recorded images into a segmented two-dimensional or three-dimensional synthetic representation of the partial or entire anatomical organ or anatomical region. The image data set is associated with the synthetic representation. The synthetic representation and the image data set show the various positions and shapes of the passageways and their connectivity. The images used to generate the synthetic representation may be recorded pre-operatively or intra-operatively during clinical diagnosis. In some embodiments, the virtual visualization system may use a standard representation (i.e., not patient-specific) or a hybrid of standard representation and patient-specific data. The synthetic representation and any virtual image generated by this synthetic representation may represent the static posture of the deformable anatomical region during one or more motion phases (e.g., during the inhalation / exhalation cycle of the lungs).

[0030] During the virtual navigation procedure, the sensor system 108 can be used to calculate the approximate position of the medical instrument 104 relative to the anatomy of the patient P. This position can be used to generate both a macro-level (external) tracking image of the anatomy of the patient P and a virtual internal image of the anatomy of the patient P. The system can implement one or more electromagnetic (EM), fiber optic, and / or other sensors to register and display the medical instrument with the pre-operatively recorded medical images. For example, see U.S. Patent Application Serial No. 13 / 107,562 (filed May 13, 2011) ("Medical System Providing Dynamic Registration of a Model of an Anatomy of the Patient P"), which is incorporated herein by reference in its entirety. One such system is disclosed in the publication "Anatomical Structure for Image-Guided Surgery" by Johns Hopkins University Press, 1999, which is incorporated herein by reference in its entirety. The teleoperated medical system 100 may further include optional operation and support systems (not shown), such as a lighting system, a navigation control system, an irrigation system, and / or an aspiration system. In some embodiments, the teleoperated medical system 100 may include two or more teleoperated manipulator assemblies and / or two or more master assemblies. The exact number of teleoperated manipulator assemblies will depend on the medical procedure and spatial constraints within the operating room, among other factors. The master assemblies 106 may be side-by-side or the assemblies may be located in separate locations. Multiple master assemblies allow two or more operators to control one or more teleoperated manipulator assemblies in various combinations.

[0031] 2A is a schematic diagram of a medical instrument system 200 according to some embodiments. In some embodiments, the medical instrument system 200 may be used as the medical instrument 104 in an image-guided medical procedure performed using the teleoperated medical system 100. In some examples, the medical instrument system 200 may be used in non-teleoperated exploratory procedures or procedures associated with traditional manually operated medical instruments, such as endoscopy. Optionally, the medical instrument system 200 may be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical passageway of a patient, such as patient P.

[0032] The medical instrument system 200 includes an elongate device 202 coupled to a drive unit 204. The elongate device 202 includes a flexible body 216 having a proximal end 217 and a distal or tip portion 218. In some embodiments, the flexible body 216 has an outer diameter of about 3 mm. Other flexible bodies may have an outer diameter that is larger or smaller.

[0033] The medical instrument system 200 further includes a tracking system 230 that uses one or more sensors and / or imaging devices to determine the position, orientation, speed, velocity, attitude, and / or shape of the flexible body 216 at the distal end 218 and / or one or more segments 224 along the flexible body 216, as described in more detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 may be effectively divided into multiple segments 224. The tracking system 230 may be a component of the sensor system 108 when the medical instrument system 200 is matched to a medical instrument 104 of the teleoperated medical system 100. The tracking system 230 may be implemented as hardware, firmware, software, or a combination thereof that optionally interacts with or is otherwise executed by one or more computer processors (which may include the processors of the control system 112 of FIG. 1 ).

[0034] The tracking system 230 may optionally use a shape sensor 222 to track the tip 218 and / or one or more segments 224. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or attached externally). In one embodiment, the optical fiber has a diameter of about 200 μm. In other embodiments, the dimensions may be larger or smaller. The optical fiber of the shape sensor 222 forms a fiber optic bend sensor for determining the shape of the flexible body 216. In one alternative, optical fibers including fiber Bragg gratings (FBGs) are used to obtain one or more dimensional strain measurements within the structure. Various systems and methods for monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. patent application Ser. No. 11 / 180,389 (filed Jul. 13, 2005), entitled "Fiber optic position and shape sensing device and method relating thereto"; U.S. patent application Ser. No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic No. 6,389,187 (filed Jun. 17, 1998) (disclosing an “Optical Fiber Bend Sensor”), all of which are incorporated herein by reference in their entireties. The sensor in some embodiments may use other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of flexible body 216 may be determined using other techniques. For example, the pose history of the tip of flexible body 216 may be used to reconstruct the shape of flexible body 216 over time. In some embodiments, tracking system 230 may optionally and / or additionally determine the position, The sensor system 220 may be used to track the tip 218. The position sensor system 220 may be a component of an EM sensor system with the position sensor system 220 including one or more conductive coils that may be exposed to an externally generated electromagnetic field. Each coil of the EM sensor system 220 then generates an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the position sensor system 220 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z, and three orientation angles indicative of the pitch, yaw, and roll of a fiducial, or five degrees of freedom, e.g., three position coordinates X, Y, Z, and two orientation angles indicative of the pitch and yaw of a fiducial. Further description of position sensor systems may be found in U.S. Pat. No. 6,380,732 (filed Aug. 11, 1999) entitled "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked), which is incorporated herein by reference in its entirety.

[0035] In some embodiments, the tracking system 230 may alternatively and / or additionally rely on stored past posture, position, or orientation data for known points of the instrument system along a cycle of alternating motion, such as breathing. This stored data may be used to develop shape information for the flexible body 216. In some examples, a series of position sensors (not shown), such as electromagnetic (EM) sensors similar to those of the position sensor 220, may be positioned along the flexible body 216 and then used for shape sensing. In some examples, a history of data from one or more of these sensors acquired during a procedure may be used to represent the shape of the elongated device 202, especially when the anatomical passageway is substantially static.

[0036] The flexible body 216 includes a channel 221 sized and shaped to receive a medical instrument 226. FIG. 2B is a schematic diagram of the flexible body 216 including the medical instrument 226 in an extended state according to some embodiments. In some embodiments, the medical instrument 226 may be used for procedures such as surgery, biopsy, cauterization, illumination, irrigation, or aspiration. The medical instrument 226 may be deployed through the channel 221 of the flexible body 216 and used at a target location within the anatomy. The medical instrument 226 may include, for example, an image capture probe, a biopsy instrument, a laser cauterization fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical tool may include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, etc. Other end effectors may include, for example, forceps, graspers, scissors, clip appliers, etc. Other end effectors may further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, etc. In various embodiments, the medical instrument 226 is a biopsy instrument that may be used to remove a sample of tissue or cells from a location of a target anatomical structure. The medical instrument 226 may also be used with an image capture probe within the flexible body 216. In various embodiments, the medical instrument 226 may be an image capture probe that includes a tip portion having a stereoscopic or monoscopic camera at or near the tip 218 of the flexible body 216, which captures images (including video images) that are processed by a visualization system 231 that are displayed and / or provided to a tracking system 230 to support tracking of the tip 218 and / or one or more segments 224. The image capture probe may include a cable coupled to the camera to transmit the captured image data. In some examples, the image capture instrument may be a fiber optic bundle, such as a fiberscope, that couples to the visualization system 231. The image capture instrument may be single or multi-spectral, for example capturing image data in one or more of the visible, infrared, and ultraviolet spectrums. Alternatively, the medical instrument 226 itself may be an image capture probe.A medical instrument 226 may be advanced through the opening of channel 221 to perform a procedure and then retracted back into the channel when the procedure is completed. The medical instrument 226 may be removed from the proximal end 217 of flexible body 216 or from another optional instrument port (not shown) along flexible body 216.

[0037] The medical instrument 226 may further contain cables, linkages, or other actuation controls (not shown) extending between its proximal and distal ends for controllably bending the distal end of the medical instrument 226. Steerable instruments are described in U.S. Pat. No. 7,516,681 (filed Oct. 4, 2005) ("Articulated Surgical Instrument for Performing Minimally Invasive Instruments"). No. 12 / 286,644 (filed Sep. 30, 2008) (disclosing “Passive Preload and Capstan Drive for Surgical Instrument”), which are incorporated herein by reference in their entireties.

[0038] The flexible body 216 may also house cables, linkages, or other steering controls (not shown) that extend between the drive unit 204 and the tip 218 to controllably bend the tip 218, for example as indicated by the dashed line representation 219 of the tip 218. In some examples, at least four cables are used to provide independent "up / down" steering to control pitch movement of the tip 218 and "left / right" steering to control yaw movement of the tip 281. Steerable catheters are described in detail in U.S. Patent Application Serial No. 13 / 274,208 (filed October 14, 2011), which discloses "Catheter with Removable Vision Probe", and which is incorporated herein by reference in its entirety. In embodiments in which the medical instrument system 200 is actuated by a remotely operated assembly, the drive unit 204 may include a drive input that removably couples to and receives power from a drive element, such as an actuator, of the remotely operated assembly. In some embodiments, the medical instrument system 200 may include a gripping mechanism, manual actuator, or other components for manually controlling the movement of the medical instrument system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable, not including an integrated mechanism for operator control of the bending of the tip 218. In some examples, one or more lumens are defined within the wall of the flexible body 216 through which medical instruments can be deployed and used at a targeted surgical location.

[0039] In some embodiments, the medical instrument system 200 may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, for use in lung examination, diagnosis, biopsy, or treatment. The medical instrument system 200 is also suitable for navigation and treatment of other tissues via naturally or surgically created connecting passageways in any of a variety of anatomical systems, including the colon, intestines, kidneys and renal calyces, brain, heart, circulatory system including vasculature, etc.

[0040] Information from the tracking system 230 is sent to a navigation system 232 where it can be combined with information from the visualization system 231 and / or a pre-operatively derived representation (e.g., a model) to provide real-time location information to a doctor, clinician, surgeon, or other operator. In some examples, the real-time location information can be displayed on the display system 110 of FIG. 1 for use in controlling the medical instrument system 200. In some examples, the control system 116 of FIG. 1 can utilize the location information as feedback to position the medical instrument system 200. Various systems for displaying a surgical instrument in registration with a surgical image using fiber optic sensors are described in U.S. patent application Ser. No. 13 / 107,562 (filed May 13, 2011) ("Medical System Providing Dynamic Registration of a Model of an Instrument," in which: For purposes of this disclosure, reference is made to "Using MRI to diagnose and treat vascular endoscopic procedures," which discloses "Using MRI to diagnose and treat vascular endoscopic procedures," in "Image-Guided Surgery," which is incorporated herein by reference in its entirety.

[0041] In some examples, the medical instrument system 200 may be remotely operated within the medical system 100 of Figure 1. In some embodiments, the teleoperated manipulator assembly 102 of Figure 1 may be replaced by direct operator controls. In some examples, the direct operator controls may include various handles and operator interfaces for handheld operation of the instrument.

[0042] 3A and 3B are schematic diagrams of a side view of a patient coordinate space including a medical instrument attached to an insertion assembly, according to some embodiments. As shown in FIGS. 3A and 3B, a surgical environment 300 includes a patient P positioned on a platform 302. The patient P may be stationary within the surgical environment in the sense that the patient's overall motion is limited by sedation, restraints, and / or other means. The patient P's cyclical anatomical motion, including breathing and cardiac motion, may continue unless the patient is asked to hold his / her breath to temporarily halt the act of breathing. Thus, in some embodiments, data may be collected at specific stages of breathing and tagged and identified at those stages. In some embodiments, the stage at which data is collected may be inferred from physiological information collected from the patient P. Within the surgical environment 300, a point gathering instrument 304 is coupled to an instrument carriage 306. In some embodiments, the point gathering instrument 304 may use EM sensors, shape sensors, and / or other sensor modalities. The instrument carriage 306 is attached to an insertion stage 308 that is fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable within the surgical environment 300 but have a known position (e.g., via a tracking sensor or other tracking device). The instrument carriage 306 is a component of a teleoperated manipulator assembly (e.g., teleoperated manipulator assembly 102) that couples to the point collection instrument 304 and controls the insertion motion (i.e., motion along the A-axis) and optionally motion in multiple directions including yaw, pitch, and roll of the tip 318 of the elongated device 310. The instrument carriage 306 or insertion stage 308 may include actuators such as servo motors (not shown) that control the movement of the instrument carriage 306 along the insertion stage 308.

[0043] The elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled and fixed to the instrument carriage 306. In some embodiments, a fiber optic shape sensor 314 is fixed to a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may be moveable with the instrument body 312, but the position of the proximal point 316 may be known (e.g., by a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point, such as a distal end 318 of the elongated device 310. The point collection instrument 304 may be substantially similar to the medical instrument system 200.

[0044] The position measurement device 320 provides information regarding the position of the instrument body 312 as it moves on the insertion stage 308 along the insertion axis A. The position measurement device 320 may include resolvers, encoders, potentiometers, and / or other sensors that determine the rotation and / or orientation of actuators that control the movement of the instrument carriage 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be curved or have a combination of curved and linear portions.

[0045] FIG. 3A shows the instrument body 312 and instrument carriage 306 in a retracted position along the insertion stage 308. In this retracted position, the proximal end point 316 is at a position L0 on the axis A. At this position along the insertion stage 308, the A component of the position of the proximal end point 316 may be set to zero and / or another reference value to provide a base reference and indicate the position of the instrument carriage 306 on the insertion stage 308 and thus the proximal end point 316. In this retracted position of the instrument body 312 and instrument carriage 306, the distal end 318 of the elongated device 310 may be positioned just inside the entrance orifice of the patient P. Also in this position, the position measurement device 320 may be set to zero and / or another reference value (e.g., I=0). In FIG. 3B, the instrument body 312 and instrument carriage 306 advance along the linear trajectory of the insertion stage 308 and the distal end 318 of the elongated device 310 has advanced into the patient P. In this advanced position, proximal point 316 is at position L1 on axis A. In some examples, encoders and / or other position data from one or more actuators controlling movement of instrument carriage 306 along insertion stage 308 and / or one or more position sensors associated with instrument carriage 306 and / or insertion stage 308 are used to determine a position Lx of proximal point 316 relative to position L0. In some examples, position Lx may further be used as an indicator of the distance or insertion depth that distal end 318 of elongate device 310 is inserted into a passage of the patient P's anatomy.

[0046] In an exemplary application, a medical instrument system, such as medical instrument system 200, may include a robotic catheter system for use in a lung biopsy procedure. The catheter of the robotic catheter system provides a conduit for tools, such as an endoscope, an endobronchial ultrasound (EBUS) probe, a therapeutic tool, and / or a biopsy tool, to be delivered to a location within the airway where one or more targets for a lung biopsy, such as a lesion, nodule, tumor, etc., reside. Typically, the endoscope is positioned so that a clinician, such as a surgeon O, can monitor a live camera feed of the tip of the catheter as the catheter is driven through the anatomy. The live camera feed and / or other real-time navigation information may be displayed to the clinician via a graphical user interface.

[0047] Prior to performing a biopsy procedure using the robotic catheter system, pre-operative planning steps may be performed to plan the biopsy procedure. The pre-operative planning steps may include segmenting a patient's CT scan to create a three-dimensional representation (e.g., a 3D model) of the anatomy, selecting a target within the 3D model, determining airways within the model, growing the airways to form a connection tree of the airways, and planning a path to the target through the connection tree. One or more of these steps may be performed on the same robotic catheter system used to perform the biopsy, on a different medical instrument system, and / or on a standalone processor such as a workstation dedicated to pre-operative planning, etc. The plan for the biopsy procedure may be saved (e.g., as one or more digital files) and transferred to the robotic catheter system used to perform the biopsy procedure. The saved plan may include the 3D model, identification of the airways, target location, path to the target location, etc. An example of a graphical user interface supporting the pre-operative planning steps is described in co-pending U.S. provisional patent application (Attorney Docket No. ISRG10600PROV / US / 70228.613PV01), which is incorporated herein by reference.

[0048] When the plan is transferred to the robotic catheter system, the 3D model of the anatomy is registered to the actual patient anatomy and / or the catheter within the patient anatomy. As a result, the real-time position and orientation of the catheter can be projected onto the 3D model and displayed via the graphical user interface. The clinician can then proceed with driving the catheter through the anatomy while monitoring the navigation progress in the graphical user interface. For example, the clinician can drive the catheter along a predefined path in the saved plan to navigate to a target location and / or to perform a biopsy at the target location.

[0049] Provided below are exemplary embodiments of a graphical user interface for monitoring a medical procedure, including but not limited to the lung biopsy procedure described above. The graphical user interface may include an alignment mode used to monitor alignment of a 3D model to an anatomical structure, a navigation mode used to monitor navigation of a medical instrument to a target location within the anatomical structure, and a performance mode used to monitor performance of an interventional step at the target location. Some aspects of the graphical user interface are described in detail in the accompanying application, “Graphical User Interface for Displaying Guidance Information During Intervention,” filed June 30, 2016. No. 62 / 357,217, entitled “Graphical User Interface for Displaying Guidance Information in a Computer-Aided System and Image-Guided Procedure,” filed June 30, 2017, No. 62 / 357,258, entitled "Plurality of Modes During and Image-Guided Procedure," which are incorporated herein by reference in their entireties.

[0050] FIG. 4 is a schematic diagram of a graphical user interface 400 displayable on a display system, such as display system 110, according to some embodiments. The graphical user interface 400 displays information related to a medical procedure in one or more views viewable by a clinician, such as surgeon O. Although an exemplary arrangement of views is shown in FIG. 4, it should be understood that the graphical user interface 400 can display any suitable number of views in any suitable arrangement and / or on any suitable number of screens. In some examples, the number of views displayed simultaneously may be changed by opening and closing views, minimizing and maximizing views, moving views between the foreground and background of the graphical user interface 400, switching between screens, and / or otherwise completely or partially obscuring views. Similarly, the arrangement of the views (size, shape, orientation, order (in the case of overlapping views), etc.) may be changed and / or may be user configurable.

[0051] In some examples, the views displayed in the graphical user interface 400 may be arranged in an organized scheme to facilitate quick access to relevant information. FIG. 4 shows an illustrative example of one such organization scheme, although many other organization schemes are possible. As shown in FIG. 4, the graphical user interface 400 includes an upper portion that displays one or more global views 410, a middle portion that displays one or more compact views 420, and a lower portion that displays one or more local views 430. The global views 410 generally display overall aspects of the medical procedure providing the clinician with a detailed picture of the current state of the medical procedure. The compact views 420 generally display a reduced set of information about the medical procedure in a simplified and uncluttered format to facilitate quick understanding by the clinician. The local views 430 generally display local aspects of the medical procedure monitoring the movements and / or intervention steps being performed in real time by the medical instruments. Examples of global, compact, and local views 410-430 are discussed in more detail below with reference to Figures 5A-5D.

[0052] In some examples, the global view, compact view, and local views 410-430 may be arranged in various configurations other than those shown in FIG. 4. For example, the graphical user interface 400 may have a landscape layout in which the global view 410 is positioned on the left, the compact view 420 is oriented vertically in the center, and the local view 430 is positioned on the right. In some examples, the global view, compact view, and local views 410-430 may span the entire graphical user interface 400, as shown in FIG. 4, thereby preventing the graphical user interface 400 from being divided into dedicated regions. In some examples, the graphical user interface 400 may include various views, controls, indicators, etc. in addition to those shown in FIG. 4. For example, the graphical user interface 400 may include a header, a footer, one or more sidebars, a message bar, a pop-up window, a background, an overlay, etc.

[0053] The graphical user interface 400 may be operated in different modes at various stages of a medical procedure. In some examples, the organization scheme may change based on the mode of the graphical user interface 400. In each mode, an arrangement of views may be selected to convey information available and / or relevant to the current stage of the medical procedure. In some examples, the modes may include an alignment mode, a navigation mode, and / or a performance mode, as discussed below. In some examples, the various modes may overlap with each other and / or seamlessly transition into each other to operate as a single mode. For example, the navigation mode and the performance mode may seamlessly transition into each other such that the modes are considered a single hybrid navigation and performance mode.

[0054] 5A-5D are schematic diagrams of a multi-mode graphical user interface 500 according to some embodiments. According to some embodiments consistent with FIG. 4, the graphical user interface 500 may correspond to the graphical user interface 400. Thus, the graphical user interface 500 displays information related to a medical procedure in one or more views viewable by a clinician, such as a surgeon O. In some examples, the graphical user interface 500 and / or selected windows or views within the global view 510, compact view 520, and local view 530 may be displayed on an I / O device, such as a touch screen, to receive user input for controlling and / or configuring the graphical user interface. For example, the user input may allow a clinician to control the positioning, zoom, viewpoint, rotation, color scheme, and / or other aspects of the configuration and / or appearance of the graphical user interface 500. In some examples, the user input may be received via another input device, such as a stand-alone control console. In some examples, the user input device may be omitted, such as when the graphical user interface 500 is not user configurable.

[0055] FIG. 5A illustrates a graphical user interface 500 in a registration mode. The registration mode is used to monitor and assist in the registration of a 3D model to an anatomical structure at the beginning of a medical procedure. In an exemplary registration process, a clinician drives an instrument such as a catheter that includes a localization sensor (e.g., a shape sensor, an electromagnetic (EM) sensor, etc.) to various portions of the anatomy to collect registration data. For example, if the anatomy corresponds to a lung, the clinician may drive the catheter to portions of the lung corresponding to the right middle lobe, the left middle lobe, the main bronchus, etc. A localization sensor such as the shape sensor 222 is used to determine a series of points that define the position and orientation of the catheter in the sensor reference frame as the catheter is driven to various portions of the anatomy. The series of points may additionally or alternatively be referenced to the robot sensor frame using, for example, encoder positions of the insertion, elevation, and / or setup joint axes. The series of points forms a point cloud that is registered to the 3D model using a registration technique such as an iterative nearest neighbor (ICP) algorithm and / or other similar algorithms. In some embodiments, guidance information can be provided to the clinician as to where to drive the catheter to facilitate and / or improve the accuracy of the registration.

[0056] Thus, the registration mode graphical user interface 500 may show one or more views that facilitate monitoring of the registration process. In some examples, the graphical user interface 500 may display one or more global views 510 that correspond to the global view 410. As shown in FIG. 5A, the registration mode global view 510 includes a pair of dynamic point cloud views 511 and 512 from a frontal and a side perspective, respectively. It should be appreciated that other perspectives may be used, including user-configurable perspectives and / or clinician adjustable in real time (e.g., using controls to adjust zoom, rotation, appearance, etc.). The dynamic point cloud views 511 and 512 display a three-dimensional representation of the point cloud and are continually updated as more points are added to the point cloud. In some examples, a three-dimensional representation of the point cloud may be rendered in the dynamic point cloud views 511 and 512 when the point cloud includes a sufficient amount of data to align the representation to the anatomical structures with reasonable accuracy. An example of a dynamic point cloud view is discussed in more detail below with reference to Figures 6A-6B.

[0057] In some examples, the alignment mode graphical user interface 500 may not display a compact view that corresponds to the compact view 420. In particular, one or more portions of the compact view 420 may be unavailable until the alignment is complete. Instead, the alignment mode graphical user interface 500 may display status information 521. As shown in FIG. 5A, the status information 521 includes a message that reads "Building airway information" and a wait icon.

[0058] In some examples, the alignment mode graphical user interface 500 may display one or more local views 530 corresponding to the local view 410. As shown in FIG. 5A, the local view 530 includes a live camera feed 531 and a dynamic alignment guidance view 532. The live camera feed 531 displays real-time images received from an endoscope positioned at the distal end of the medical instrument. An example of the dynamic alignment guidance view is discussed in more detail below with reference to FIGS. 7A-7B. An example of the live camera feed is discussed in more detail below with reference to FIGS. 12A-12B.

[0059] As shown in FIG. 5A, the graphical user interface 500 further includes a header 541 and a message bar 542. The header 541 displays a title (e.g., "Performing Airway Alignment") and patient information. In some examples, the header 541 may include mode transition buttons to transition between modes of the graphical user interface 500. The message bar 542 displays one or more messages that may be used to inform the clinician of the next step of the procedure (e.g., "Confirm alignment on controller screen and begin navigating to target"), send a reminder to the clinician (e.g., "Use breath-hold timer during alignment"), warn the clinician of hazards, etc. In some embodiments, the appearance (e.g., color, size, texture, font, etc.) of the message bar 542 may vary depending on the type of message. For example, a warning message may be displayed in red. The message bar 542 may be hidden when there are no messages to display.

[0060] In some examples, the graphical user interface 500 may include a driving force indicator 543 and a bend indicator 544. In some embodiments, the bend indicator 544 may be used to alert a clinician when the catheter is positioned within the anatomy such that one or more segments of the catheter are bent at a radius that is too tight for a tool, such as a biopsy needle, to pass through the catheter. For example, the bend indicator 544 may be displayed when a tight bend radius is detected in the catheter (e.g., when the bend radius is below a predetermined threshold) and hidden otherwise. In some embodiments, the bend indicator 544 may be displayed in a normal state as an indication of the current shape of the tip of the catheter to enhance the user's awareness of the tip configuration. In this regard, the shape of the bend indicator 544 may be calculated from the actual measured shape of the catheter and / or may be an abstract iconic indicator. Examples of driving force indicators and bend indicators are discussed in more detail below with reference to Figures 8 and 9, respectively.

[0061] When alignment is complete, the graphical user interface 500 transitions from alignment mode to navigation mode. In some examples, the graphical user interface 500 may transition from alignment mode to navigation mode automatically in response to detecting that alignment is complete. In some examples, the transition may be performed manually in response to a clinician clicking a button on the graphical user interface 500 (e.g., a mode transition button in header 541), initiating input on another input device (e.g., a touch screen on a control console) to proceed to navigation mode, etc.

[0062] 5B illustrates a navigation mode graphical user interface 500 according to some embodiments. In some examples, the navigation mode can be used to monitor and assist in the navigation of the catheter to a target location. For example, the navigation mode can be used when a clinician is driving a catheter along a route selected during planning of a medical procedure. The navigation mode can further include an indicator that provides the clinician with directional guidance along the route to the target.

[0063] In the example shown in FIG. 5B, the navigation mode global view 510 includes a pair of global anatomical models 513 and 514 from a frontal and a side perspective, respectively. In another example, the perspective of the global anatomical models 513 and / or 514 can be rotated by the clinician using an input device, such as a scroll wheel, mouse, or touch screen, to drag the global anatomical models 513 and / or 514 to a desired orientation. In some embodiments, the clinician can rotate one view (e.g., the global anatomical model 513) and in response, another view (e.g., the global anatomical model 514) can automatically rotate to provide an orthogonally oriented view, respectively. Examples of global anatomical models are discussed in more detail below with reference to FIG. 10. In some examples, the navigation mode graphical user interface 500 can display one or more compact views 520 corresponding to the compact view 420. As shown in FIG. 5B, the compact view 520 includes a reduced anatomical model 522. The scaled anatomical model 522 displays an elongated representation of the planned route to the target location, displaying various features along the route, including the target location, in simplified form. Examples of scaled anatomical models are discussed in more detail below with reference to FIGS. 11A-11C. The local view 530 in navigation mode includes a live camera feed 531, similar to the registration mode. In some examples, in navigation mode, supplemental guidance information can be superimposed on the live camera feed 531, as discussed below with reference to FIGS. 12A-12B. The local view 530 can include a virtual tip view 533 that displays a rendering of the 3D model from the perspective of the tip of the catheter. Examples of virtual tip views are discussed in more detail below with reference to FIGS. 13A-13D.

[0064] As the tip of the catheter approaches the target location (e.g., when the tip is within a certain range of the target location), an alternative configuration of the graphical user interface 500 may be displayed, as shown in FIG. 5C. In some examples, the graphical user interface 500 may automatically transition from the configuration shown in FIG. 5B to the alternative configuration shown in FIG. 5C in response to detecting that the catheter is in proximity to and / or aligned with the target location within a threshold. In some examples, the transition may be performed manually, such as in response to a clinician clicking a button on the graphical user interface 500 and / or another input device to advance to a performance mode.

[0065] As shown in FIG. 5C, the global view 510 and compact view 520 are similar to FIG. 5B, but various adjustments can be made to the displayed views (e.g., displaying a zoomed inversion of the global view 510). However, various configurations of local views 530 can be displayed to assist the clinician in aligning the catheter to the target location when the catheter is in the vicinity of the target location. For example, as described in more detail below, a remote image view 534, such as a fluoroscopy live view, can be displayed instead of and / or in addition to the endoscope live view. To accommodate the remote image view 534, the endoscope live view 531 can be resized and / or repositioned, for example, to the bottom left corner of the display. In some examples, the appearance of the virtual tip view 533 can be modified to assist the clinician in identifying and aligning the target location, such as displaying a translucent passageway wall, target indicator, uncertainty zone, and / or crosshair indicator.

[0066] When the clinician is ready to perform an interventional step at the target location, the graphical user interface 500 transitions from the navigation mode to the performance mode. In some examples, the graphical user interface 500 may transition from the navigation mode to the performance mode automatically in response to detecting that the camera probe has been removed from the catheter (thus eliminating the endoscopic live view) and / or that a medical instrument for performing the interventional step is being inserted into the catheter. In some examples, the transition may be performed manually, such as in response to the clinician clicking a button on the graphical user interface 500 and / or a separate input device.

[0067] FIG. 5D illustrates a performance mode graphical user interface 500 according to some embodiments. The performance mode is used to monitor the performance of an interventional step at a target location. For example, an interventional step may correspond to a biopsy, cauterization, delivery of a chemical substance, physical manipulation of tissue, placement or removal of a biodevice, etc. In some embodiments, an endoscope inserted through the catheter to provide a live camera feed during the navigation process is removed during the performance of the interventional step and replaced with the tool used to perform the interventional step. When the endoscope is removed, the catheter remains parked near the target site and provides a conduit for the tool to reach the target site. In some embodiments, a probe, such as an endobronchial ultrasound (EBUS) probe, may be inserted into the parked catheter before, during, and / or after insertion of the tool, but after removal of the endoscope. For example, a radial EBUS probe may be extended just past the distal tip of the catheter to provide a 360 degree ultrasound view. The ultrasound view may be used by the clinician to confirm the location of the target location relative to the tip of the catheter. For example, in scenarios where the alignment was inaccurate and / or the anatomy has shifted and / or deformed since alignment was performed, the catheter may be misaligned relative to the target location. In this regard, ultrasound views can be used to reposition the catheter to improve alignment with the target. When the clinician is ready to perform the interventional step, the probe used for navigation and / or fine-tuning alignment (e.g., endoscope or EBUS probe) can be withdrawn and replaced with a procedure-specific tool, such as a biopsy needle.

[0068] As shown in FIG. 5D, the global view 510 and compact view 520 in the performance mode are similar to the navigation mode, but various adjustments can be made to the displayed views (e.g., displaying a zoomed inversion of the global view 510 in the performance mode). The adjustments can be made manually by the clinician and / or automatically by the display system. As shown in FIG. 5D, the local view 530 in the performance mode includes a remote image view 534. In this example, the remote image view 534 includes a fluoroscopic image oriented to capture the anatomical structures near the tip of the catheter. Examples of remote image views are discussed in more detail with reference to FIGS. 15A-15C. The other views included in the local view 530 in the performance mode are generally similar to the navigation mode, but can be resized and / or repositioned to accommodate the remote image view 534.

[0069] 6A-6B are schematic diagrams of a dynamic point cloud view 600 at two different times during an alignment process according to some embodiments. According to some embodiments consistent with FIG. 5A, the dynamic point cloud view 600 may correspond to one or more of the dynamic point cloud views 511 and / or 512 of the graphical user interface 400. However, it should be understood that the dynamic point cloud view 600 may be displayed in contexts other than the graphical user interface 500, as a stand-alone graphical user interface element, and / or in combination with views other than those shown in the graphical user interface 500.

[0070] The dynamic point cloud view 600 shows a point cloud 610 that corresponds to registration data captured using a position probe 620. As shown in FIGS. 6A-6B, the point cloud 610 is displayed using blue dots and the position probe 620 is displayed as a green line. An orientation icon 630 in the upper right corner indicates the orientation of the plot relative to the anatomy. FIG. 6A shows the dynamic point cloud view 600 at the beginning of the registration process when the point cloud 610 corresponds to registration data captured along a single, unbranched path in the anatomy. In some instances, a single, unbranched path in the anatomy may be insufficient to perform registration with reasonable accuracy. Thus, no 3D model is rendered in FIG. 6A.

[0071] FIG. 6B illustrates a dynamic point cloud view 600 at a later stage in the registration process, when the point cloud 610 has grown to span multiple branches (e.g., multiple airways) and / or multiple partitions (e.g., multiple lung lobes). When the point cloud 610 contains a sufficient amount of registration data to perform the registration, an estimated position of the model 630 relative to the point cloud 610 is calculated using a registration algorithm such as ICP. Once the estimated position is calculated, the model 630 may be rendered in the dynamic point cloud view 600. As shown in FIG. 6B, the model 630 is displayed in a semi-transparent gray color. As the point cloud continues to grow, the estimated position of the model 630 may be continually updated. The color and / or translucency of the model 630 may also be adjusted depending on the level of completion of the registration process, e.g., the model becomes more opaque as the registration quality improves and / or the point cloud coverage of the model increases.

[0072] The clinician can visually assess the accuracy of the registration by identifying anomalies in the dynamic point cloud view 600. For example, in FIG. 6B , the clinician can determine that the registration is inaccurate when portions of the point cloud 610 and / or location probe 620 are not positioned inside the model 630 because the point cloud 610 and location probe 620 are generally expected to be within the boundaries of the model 630. In scenarios where the accuracy of the registration is not satisfactory to the clinician, the clinician can choose to continue to capture additional points to try to improve the accuracy of the registration, restart the registration process, and / or proceed to the next stage of the procedure.

[0073] 7A-7B are schematic diagrams of a dynamic alignment guidance view 700 at two different times according to some embodiments. According to some embodiments consistent with FIG. 5A, the dynamic alignment guidance view 700 may correspond to the dynamic alignment guidance view 532. However, it should be understood that the dynamic alignment guidance view 700 may be displayed in contexts other than the graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in the graphical user interface 500.

[0074] The dynamic registration guidance view 700 displays information and / or instructions that are dynamically updated to reflect the progress of the registration. In some examples, the dynamic registration guidance view 700 may include instructions 710 that describe the current step of the registration process and / or provide recommended areas of the anatomical structure where additional points should be collected. When the current step is completed, the content of the instructions 710 may be automatically updated to describe the next step of the registration process.

[0075] In some examples, the dynamic alignment guidance view 700 may include a schematic representation 720 of an anatomical structure. Each segment of the schematic representation 720 corresponds to a portion of the anatomical structure. As shown in FIGS. 7A-7B, the segments include various lung lobes, major airways (e.g., main bronchial trunks), and areas around the main carina of the lungs. Once a sufficient amount of alignment data is obtained from a particular segment of the anatomical structure, one or more attributes of the segment (e.g., color, shape, size, texture, text label, etc.) may be updated to indicate that alignment is complete for that portion of the anatomical structure. For example, a segment may transition from an unfilled state (e.g., no color) to a filled state (e.g., one color) when a threshold amount of alignment data is collected from the segment. Additionally or alternatively, the appearance of the segment may gradually transition as more alignment data is collected from the segment. In some examples, one or more segments that include a minimum amount of alignment data may be identified. By monitoring the changes in the appearance of the segments in the schematic representation 720, the clinician is notified when registration is completed for each segment and can take steps to move the location probe to parts of the anatomy that do not yet have a sufficient amount of registration data. As a result, the registration process may be accelerated because the clinician is instructed to grow more registration points in areas where the registration data is sparsest and / or not yet sufficient to perform an accurate registration.

[0076] FIG. 7A shows a schematic representation 720 at an early stage of the registration process before sufficient alignment data has been captured from any of the segments. Thus, none of the segments are filled in. In contrast, FIG. 7B shows a schematic representation 720 at a later stage of the registration process after sufficient alignment data has been captured from all of the segments. Thus, each segment of the schematic representation 720 is solid blue. A target indicator 730 appears in FIG. 7B to indicate which segment contains the target location of the medical procedure. Multiple target indicators may appear if the medical procedure includes multiple target locations. In an alternative example, the target indicators may be overlaid on the segments and / or the location of the target may be indicated by changing the color, weight, hue, and / or transparency of the corresponding segment.

[0077] 8 is a schematic diagram of a driving force indicator 800 according to some embodiments. According to some embodiments consistent with FIGS. 5A-5, driving force indicator 800 may correspond to driving force indicator 543. However, it should be understood that driving force indicator 800 may be displayed in a context other than graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in graphical user interface 500.

[0078] The driving force indicator 800 displays a visual and / or alphanumeric representation of the axial driving force applied to the catheter during operation (e.g., during insertion and / or withdrawal of the catheter from the patient's anatomy). For example, a large driving force may be applied when the catheter is inserted into a passageway narrower than the diameter of the catheter and / or when the catheter is otherwise obstructed by an obstacle. Many approaches to monitoring the applied driving force are possible. As shown in FIG. 8, the driving force is plotted as a function of time using a symmetrical bar plot. The most recent driving force measurement is on the right and historical measurements are plotted on the left. In some embodiments, various other plotting techniques may be used, such as one-sided bar graphs, line graphs, scatter plots, etc. Alternatively or additionally, the current driving force may be displayed using a meter, alphanumeric text, etc. In some examples, the appearance (e.g., color, texture, size, etc.) of the driving force indicator 800 may change as a function of driving force and / or time. As shown in FIG. 8, the color of the bar changes between gray, yellow, or red based on the magnitude of the driving force, and the thickness of the bar gradually decreases going back in time. In some examples, the driving force indicator 800 may be positioned adjacent to and / or within a view showing a representation of the catheter, such as the scaled anatomical view 522. In this arrangement, the driving force indicator 800 may convey the visual impression of pushing the catheter into the anatomical structure. Various examples of driving force indicators are further described in U.S. Provisional Patent Application No. 62 / 357,258, which is incorporated by reference above.

[0079] 9 is a schematic diagram of a bend indicator 900 according to some embodiments. According to some embodiments consistent with FIGS. 5A-5D, the bend indicator 900 may correspond to the bend indicator 544. However, it should be understood that the bend indicator 900 may be displayed in a context other than the graphical user interface 500, as a standalone view, and / or in combination with other views other than those shown in the graphical user interface 500. As previously discussed, the bend indicator 900 may be displayed when a tight bend radius is detected in the catheter (e.g., when the bend radius falls below a predetermined threshold) and may be hidden otherwise. Alternatively, a selected portion of the bend indicator 900 may be hidden when no tight bend is present, e.g., the bend radius value 910.

[0080] The bend indicator 900 provides a schematic bend representation 910 of the catheter. When the distal tip of the catheter is bent, a bend line 925 appears indicating the direction the distal tip of the catheter will bend. For example, as shown in FIG. 9, a bend line 925 appears at the top right of the ring 915, indicating that the catheter is bent to the right. Thus, to straighten the catheter, the catheter can be steered down and to the left to reduce the bend. In some examples, when the distal tip of the catheter is straight, the bend line 925 may be hidden.

[0081] In some examples, the schematic bend representation 910 may include a rendering of the tip of the catheter from a perspective looking back up the catheter tube through the distal tip of the catheter (from the distal tip towards the proximal portion of the catheter). Consistent with such examples, the ring 915 may be interpreted as corresponding to the distal tip of the catheter. As the catheter is bent, a portion of the catheter becomes visible behind the distal tip (i.e., ring 915). As a result, the bend line 925 corresponds to a portion of the distal end of the catheter that becomes visible behind the distal tip (i.e., ring 915) due to bending of the catheter.

[0082] In an alternative example, the schematic bend representation 910 may include a rendering of the distal end of the catheter from a perspective looking forward down the catheter tube from a proximal position along the catheter toward the distal tip. Consistent with such an example, the ring 915 may be interpreted as corresponding to a cut section of the catheter at the proximal position. As the catheter is bent, a portion of the tip becomes visible behind the cut section (i.e., the ring 915). As a result, the bend line 925 corresponds to a portion of the catheter that becomes visible behind the cut section (i.e., the ring 915) due to bending of the catheter.

[0083] In some examples, the bend indicator 900 may display a visual and / or alphanumeric representation of the minimum or smallest bend radius detected along the catheter. When the minimum bend radius falls below a threshold, the bend indicator 900 may alert the clinician that a predefined threshold has been violated by displaying an alphanumeric value and / or otherwise changing appearance. In some embodiments, the threshold may be determined based on whether a tool can be passed through the catheter. In some embodiments, the threshold may be determined based on a radius at which catheter buckling and / or damage may occur. The threshold may be manually selected, automatically determined, determined based on the type of catheter and / or tool, and / or set using general rules of thumb. As shown in FIG. 9, when the minimum bend radius detected is below a threshold, the bend indicator 900 includes a number 920 indicating a real-time value of the minimum bend radius, and a portion of the bend indicator 900 turns a different color, such as red, as shown in FIG. 9. In some embodiments, the location of the red portion may reflect the amount of force applied by one motor to the catheter pull wire for that section of the catheter. For example, in FIG. 9, the top left pull wire is pulled harder as indicated by the red wedge that appears in the schematic bend representation 910. In some examples, the bend indicator 900 may include an outer ring 930 that dynamically changes color based on whether the minimum bend radius is approaching or exceeding a threshold. In some examples, the dynamic change may be represented by a change in appearance, such as transparency, texture, line width, and / or color, of portions of the bend indicator 900. Various examples of bend indicators and associated indicators of monitored parameters other than bend are further described in U.S. Provisional Patent Application No. 62 / 357,217, which is incorporated by reference above.

[0084] 10 is a schematic diagram of a global anatomical model 1000 according to some embodiments. According to some embodiments consistent with FIGS. 5B-5D, the global anatomical model 1000 may correspond to one or more of the global anatomical models 513 and / or 514. However, it should be understood that the global anatomical model 1000 may be displayed in contexts other than the graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in the graphical user interface 500.

[0085] The global anatomical model 1000 displays a 3D model 1010 of an anatomical passageway. In some examples, the 3D model 1010 may be surrounded by a boundary 1020 of a relevant portion of the anatomy (in this case, the lung). For example, the boundary 1020 may be displayed using a semi-transparent fill and / or a wire grid. As shown in FIG. 10, the boundary 1020 is displayed as a horizontal wire grid. The appearance of the boundary 1020 (e.g., color, texture, spacing of hash marks, etc.) may vary to indicate potential dangers. As shown in FIG. 10, the dangers 1025 correspond to a portion of the boundary 1020 near the target location 1030, which is colored red to alert the clinician to the risk of puncturing the pleura of the lung when navigating and / or performing an interventional step at the target location 1030. In some examples, the uncertainty zone 1035 may be displayed around the target location 1030, for example as a semi-transparent sphere as shown in FIG. 10, or in another example in a different color. The size of the uncertainty zone 1035 may be fixed and / or determined based on one or more factors including uncertainties in the alignment of the target relative to the anatomical structures, the location of the target, the expected difficulty in accessing the target, etc. The clinician may use the uncertainty zone 1035 to perform an intervention step, such as a biopsy, at different distances from the estimated center of the target. For example, if the uncertainty zone 1035 is large (e.g., due to large alignment uncertainties), the clinician may be advised to take biopsy samples farther apart from each other than they would be if taken to increase the chances of successfully sampling the lesion.

[0086] In some embodiments, the global anatomical model 1000 may further include a virtual image of the catheter 1040 based on the shape sensing data and / or the planned route 1050 to the target location 1050. As shown in FIG. 10, the catheter 1040 is shown as a green line and the planned route 1050 is shown as a blue line. In some examples, one or more attributes of the catheter 1040 (e.g., color, line type, texture, size, etc.) may change based on measurement data related to the catheter 1040, such as bend radius, temperature, strain, etc. An orientation icon 1060 is located at the top right and indicates, in this example, that the global anatomical model 1010 is viewed from a frontal perspective. Various examples of global anatomical models are further described in U.S. Provisional Patent Application No. 62 / 357,217 and U.S. Provisional Patent Application No. 62 / 357,258, which are incorporated by reference above.

[0087] 11A-11C are schematic illustrations of a reduced anatomical model 1100 at three different time points during a procedure and / or at different catheter positions of an anatomical structure, according to some embodiments. According to some embodiments consistent with FIGS. 5B-5D, the reduced anatomical model 1100 may correspond to the reduced anatomical model 522. However, it should be understood that the reduced anatomical model 1100 may be displayed in a context other than the graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in the graphical user interface 500.

[0088] The reduced anatomical model 1100 displays one or more features of a planned route to a target location. As shown in Figures 11A-11C, the features include a simplified route path 1110, a width of a passageway 1112, a location of a branch passageway 1114, an alternative route indicator 1115, a target location 1116, a hazard location 1118, and an insertion trajectory 1120. The features are extracted from a larger anatomical model, such as a full 3D anatomical model used to determine the route during the planning stage. In some examples, the reduced anatomical model 1100 may exclude various details from the larger anatomical model.

[0089] In some examples, the reduced anatomical model 1100 may be displayed as a linear anatomical representation (e.g., model), where the 3D aspects of the planned route and / or anatomical structures may be reduced (straightened, flattened, clipped, resized, simplified, etc.) to fit within an elongated region. Consistent with such an embodiment, the simplified route path 1110 may be represented as a straight line. The width of the passageway 1112 is indicated using a pair of vertically spaced horizontal lines centered on the simplified route path 1110. In some examples, the width of the passageway 1112 may be rounded to the nearest tiered level, such that the vertically spaced lines have a tiered spacing that generally correlates with the generation of the branch. The location of the branching passageway 1114 is indicated relative to the simplified route path 1110 using a pair of horizontally spaced vertical lines branching off from the passageway 1112. In some examples, the complete branching structure of the branching passageway 1114 (e.g., various sub-branches and / or the 3D shape of the branching passageway) may be omitted from the reduced anatomical model 1100. For example, the branching passageway 1114 may be displayed as a pruned branch cut near where it intersects with the passageway 1112. If the branching passageway provides an alternative route to the target location 1116, the passageway may be labeled using an alternative route indicator 1115. Rather than showing the full 3D path of the alternative route, the alternative route indicator 1115 may include a simplified indication that an alternative route is available, such as a vertical dotted line. In one example, the clinician may select or click on the alternative route indicator 1115. In response, the reduced anatomical model 1100 may be updated to display the alternative route instead of the route originally displayed. In another example, the reduced anatomical model 1100 may be automatically updated to display the alternative route as the clinician begins to traverse a branching passageway corresponding to the alternative route.

[0090] An insertion trajectory 1120, representing the puncture trajectory of the instrument through the lumen of the passageway 1112, is shown as a dotted line connecting the end of the simplified route path 1110 to the target location 1116. If a hazard (e.g., a blood vessel, a large blister, an organ such as the heart, and / or the pleura of the lung) is located behind and / or otherwise adjacent to the target location 1116, the insertion trajectory 1120 may extend past the target location 1116 to show the distance between the target location 1116 and the hazard 1118, which is shown in this example as a diagonal line. In some examples, various features including the simplified route path 1110, the branching passageway 1114, the alternative route indicators 1115, the target location 1116, the hazard 1118, and / or the insertion trajectory 1120 may be displayed with contrasting colors, transparencies, textures, and / or line widths. Generally, the reduced anatomical model 1100 shows one route and target location at a time. When a medical procedure includes multiple target locations, the additional target locations can be marked by labeling the corresponding branch (through which the additional target locations can be reached) using target icon 1122, as shown in FIG. 11C.

[0091] The reduced anatomical model 1100 is continually updated to reflect the progress of the catheter 1130 along the simplified route path 1110. For example, the catheter 1130 may be overlaid on the simplified route path 1110 in a contrasting color or shade, such as green. Various indicators and / or warnings may be displayed when anomalies are detected. For example, when the catheter 1130 makes a wrong turn, a wrong turn indicator 1142 may appear, as shown in FIG. 11B. In some examples, a bend indicator 1144 may appear when an excessive bend is detected. The bend indicator 1144 may include a display value representing the bend radius. In some examples, the appearance of the catheter 1130 may be modified to draw attention to a tight bend radius, for example, by changing from a solid green line to a dotted red line. In some examples, one or more bookmarks 1150 may be placed along the route to indicate locations and / or times of particular interest (e.g., locations where the clinician captured a snapshot and / or otherwise observed a feature of interest for later review). For example, the bookmark 1150 may be shown as a bracket superimposed on the catheter 1130.

[0092] In some examples, the reduced anatomical model 1100 may be displayed within a geometrically constrained region of a graphical user interface. For example, the geometrically constrained region may correspond to a horizontally elongated region, a vertically elongated region, an "L" shaped region, and / or various other regions having an irregular shape. The ability to display the reduced anatomical model 1100 in a geometrically constrained region may correspond to tight packing of the reduced anatomical model 1100 with other views of the graphical user interface. Similarly, the ability to fit the reduced anatomical model 1100 into a geometrically constrained region may correspond to an irregularly shaped display screen, such as an elongated display screen.

[0093] Additionally, the reduced anatomical model 1100 may provide the clinician with quick and organized access to information related to a medical procedure. For example, the reduced anatomical model 1100 may display a selected subset of information related to a medical procedure extracted from a more detailed anatomical model. In some examples, the reduced anatomical model 1100 may maintain a consistent appearance across different medical procedures. For example, the target location 1116 may appear in substantially the same location (e.g., the rightmost portion of the reduced anatomical model 1100) regardless of the actual 3D location of the target and / or the actual 3D route to the target. The consistent appearance of the reduced anatomical model 1100 may allow the clinician to reduce the time it takes to search for and / or understand information displayed on the reduced anatomical model 1100 across different medical procedures and / or different routes within a medical procedure.

[0094] 12A-12B are schematic illustrations of a live camera feed 1200 at two different time points during a procedure and / or at different catheter positions in an anatomical structure, according to some embodiments. According to some embodiments consistent with FIGS. 5A-5, the live camera feed 1200 may correspond to the live camera feed 531. However, it should be understood that the live camera feed 1200 may be displayed in a context other than the graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in the graphical user interface 500.

[0095] The live camera feed 1200 displays an image from a camera, such as an endoscope at the tip of a catheter. In some examples, supplemental guidance information 1210 may be overlaid on the camera image. For example, the supplemental guidance information 1210 may be generated by using image processing techniques to identify features in the image, and / or by using shape data from the catheter to determine the location of the camera in the anatomy, etc. As shown in FIGS. 12A-B, the supplemental guidance information 1210 is displayed using a reticle that includes a circle with an arrow. As the camera approaches a branch, the arrow points to the branch that corresponds to the planned route. In some examples, the arrow may be displayed as the camera approaches a branch and hidden otherwise. In some examples, the reticle may be color coded to indicate which branch to steer toward and / or which branch to steer away from. For example, a red portion of the circle may indicate that the catheter should not be navigated down the displayed branch. In another example, the size of the color-coded portion may indicate the degree of deviation from the planned route to the target, including a circle that is filled with red when the catheter is in the airway (not on route).

[0096] 13A-13D are schematic illustrations of virtual tip view 1300 at four different times according to some embodiments. According to some embodiments consistent with FIGS. 5B-5D, virtual tip view 1300 may correspond to virtual tip view 537. However, it should be understood that virtual tip view 1300 may be displayed in a context other than graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in graphical user interface 500.

[0097] The virtual tip view 1300 is generated based on a 3D model of the anatomy. With the 3D model registered to the anatomy, the position of the catheter within the 3D model can be determined based on real-time shape sensing. The virtual tip view 1300 is then generated by rendering the 3D model from the perspective of the tip of the catheter (i.e., looking toward the tip).

[0098] In some examples, the rendered 3D model image may include supplemental guidance information 1310. Similar to the supplemental guidance information 1210, the supplemental guidance information 1310 may include one or more directional indicators of various forms to aid in navigating the branch. For example, the correct branch may be indicated by lighting (or otherwise changing the appearance of) the branch to be driven to, providing a line indicating which branch to drive to, displaying a reticle similar to FIGS. 12A-12B, etc. In another example, the view may include a reticle in the center of the view to aid in precisely aiming the direction of the catheter, e.g., aligning the catheter to the center of the target. In the example shown in FIG. 13A, the branch to be driven to is highlighted using contour lines that selectively extend beyond the correct branch. In some embodiments, a blockage indicator (e.g., a stop sign, a no entry sign, etc.) may appear above the incorrect branch. Additionally, the virtual tip view 1300 may include a target distance indicator 1320 and a danger distance indicator 1330. A target distance indicator 1320 can provide the distance from the catheter tip to the target. A danger distance indicator 1330 can provide the distance from the catheter tip to the nearest danger, such as the pleura, heart, large blisters, and / or blood vessels. Various examples of virtual tip views are further described in U.S. Provisional Patent Application No. 62 / 357,258, which is incorporated by reference above.

[0099] When the catheter is positioned within a predetermined threshold distance from the target, the appearance and / or configuration of the virtual tip view 1300 can shift to provide directional guidance towards the target. In particular, the target may be embedded in the passageway wall relative to the tip of the catheter and thus is not visible as shown in FIG. 13A. FIG. 13B displays an example of a virtual tip view when the tip of the catheter is positioned within the threshold distance to the target as indicated by the distance to the target indicator 1320. In this example, 3.1 cm is within the threshold where the virtual tip view 1300 is modified to display the passageway wall as transparent (or semi-transparent) so that the target is visible. In other examples, the threshold distance may vary based on the procedure, type of tool, target size, target location, etc. As shown in FIG. 13B, the target is shown as a semi-solid sphere 1340 while the uncertainty zone 1345 is shown as a semi-transparent sphere. Additionally, crosshairs 1350 can be provided to indicate the direction the catheter is pointing. In some examples, the crosshairs 1350 indicate where an instrument such as a biopsy needle would be inserted if inserted through the central lumen of the catheter. As shown in FIG. 13B, the crosshairs 1350 include a circle and a cross in the center of the circle. In an alternative embodiment, the crosshairs 1350 may include only a circle with a small dot in the center. The crosshairs 1350 may be automatically centered in the view so that the view always displays a tip-wise view from the catheter in the center of the image. In one example, the crosshairs 1350 are always displayed to indicate a forward direction of the center of the catheter. In an alternative example, the crosshairs 1350 are automatically displayed when the catheter approaches the target within a predefined threshold. In yet another example, the user may be provided with an input to show or hide the crosshairs 1350 as desired.

[0100] Figure 13C shows an alternative example of a virtual tip view 1300, which is similar to the view of Figure 13C and shows a target 1340, a zone of uncertainty 1345, a crosshair 1350, and also provides a view of a danger 1360. In the example of Figure 13C, a virtual image of a catheter 1370 is also displayed. In an alternative embodiment, the catheter image 1370 is not shown.

[0101] FIG. 13D displays another example of a virtual tip view 1300 that may be displayed during a medical procedure, such as a biopsy. The virtual tip view 1300 shown in FIG. 13D may be similar to that described above, except that it provides a dynamic representation of interventional steps previously performed during a medical procedure. As shown in FIG. 13D, after a series of biopsies have been performed, the virtual tip view may be dynamically updated to indicate various locations of each biopsy using labels 1380. This may assist the user in selecting the next biopsy location. Although the example of FIG. 13D describes a biopsy procedure, it should be understood that the virtual tip view 1300 may be used to indicate the historical application of any type of procedure, including therapeutic procedures, such as ablation, chemotherapy, and / or other types of diagnostic procedures. In some examples, additional labels corresponding to the label 1380 may be updated in various other views of the graphical user interface 500, such as the global anatomical model 513, 514 shown in FIG. 5B.

[0102] FIG. 14 is a schematic diagram of a set of views 1400 during a scenario in which a catheter is inserted beyond the end of a planned route, according to some embodiments. The views 1400 include a zoomed-out anatomical view 1410, a live camera feed 1420, and a virtual tip view 1430. These generally correspond to the similar labeled views shown in FIGS. 5-13. The views 1400 illustrate various techniques for alerting the clinician that the catheter has been inserted beyond the end of the route and should be withdrawn. In the zoomed-out anatomical view 1410, a reverse indicator 1415 appears at the tip of the catheter. In the live camera feed 1420, a reticle 1425 turns completely red to indicate that none of the bifurcations displayed in the image are correct. In the virtual tip view 1430, the contour lines disappear and a reverse indicator 1435 appears. It should be understood that these alerting measures are merely exemplary and that many other signaling techniques may be used to alert the clinician that the catheter has been inserted beyond the end of the route.

[0103] 15A-15C are schematic diagrams of multiple modes of remote image view 1500 according to some embodiments. According to some embodiments consistent with FIG. 5D, remote image view 1500 may correspond to remote image view 538. However, it should be understood that remote image view 1500 may be displayed in a context other than graphical user interface 500, as a standalone view, and / or in combination with views other than those shown in graphical user interface 500.

[0104] FIG. 15A illustrates a remote image view 1500 in a live mode. In the live mode, the remote image view 1500 displays a live remote image 1510, such as a fluoroscopic image of a relevant portion of the anatomy (e.g., lungs). The live remote image 1510 can be adjusted to zoom in, zoom out, pan the view of the image, etc. The clinician can provide input via a remote control or use a mouse or scroll ball to drag the live image in the fluoroscopic view. Additionally, if the system is connected to a mechanically adjustable fluoroscopy arm, the system can rotate the fluoroscopy arm to provide an alternative live remote image 1510, possibly providing a better view of the target or catheter. Additionally, the remote image view 1500 displays a live probe image 1520 from the tip of the catheter, such as an image generated by an endoscope and / or an EBUS probe. The clinician can adjust the position of the catheter while monitoring the live remote image 1510 and / or the live probe image 1520. At various times during the live mode, the clinician can capture and save snapshots of one or more images displayed in the remote image view 1500. Thus, a series of snapshots can be saved over time as the probe is repositioned while attempting to place and / or align the catheter to the target location. Once satisfied with the alignment, the clinician can remove the probe from the catheter system and replace it with a tool to perform the interventional step, such as a biopsy needle. In some instances, the tool may not include imaging capabilities, such that the clinician can no longer monitor the live probe image 1520. Thus, one or more of the saved snapshots can be used as a reference in the event that the live probe image is no longer available.

[0105] FIG. 15B illustrates the remote image view 1500 in reference mode. In some examples, the remote image view 1500 includes a reference selection indicator 1530 that indicates which snapshot is being used as a reference. As shown in FIGS. 15A-B, the reference selection indicator 1530 includes a series of dots. When the leftmost dot is highlighted, the remote image view 1500 is in live mode. Selecting one of the dots on the right indicates that the remote image view 1500 is in reference mode. Each dot corresponds to a different saved snapshot. In this regard, the clinician can scroll through the saved snapshots and select one as the reference. Once the reference is selected, the live probe image 1520 is replaced with a previously captured probe image 1522 from a saved snapshot. Additionally, a previously captured remote image 1512 from a saved snapshot is overlaid on the live remote view. As shown in FIG. 15B, the catheter image 1514 included in the previously captured remote image 1512 appears as a blue "shadow" against the catheter image 1516 included in the grayscale live remote view 1510. In this regard, the clinician can observe the differences between the previously captured remote image 1512 and the live remote image 1510. For example, as shown in FIG. 15B, the clinician can determine that the catheter has shifted downward relative to the reference because the "shadow" (catheter image 1514) is offset from the real-time position of the catheter (catheter image 1516). As long as the clinician wants the position of the catheter to match the position of the reference, the clinician can continue to steer the catheter upward until the live remote image 1510 matches the previously captured remote image 1512.

[0106] FIG. 15C illustrates a remote image view 1500 in a timeline mode. Instead of and / or in addition to displaying a reference selection indicator 1530 to scroll through snapshots during reference selection, the remote image view 1500 in the timeline mode may include a timeline 1540 for scrolling between consecutively and / or periodically saved snapshots. In this regard, the clinician may rewind the medical procedure to determine the desired reference. As shown in FIG. 15C, when the selected time is the current time (i.e., real time), the configuration of the remote image view 1500 is similar to the live mode shown in FIG. 15A. When the clinician scrolls back to an earlier time using the timeline 1540, the remote image view 1500 transitions to a configuration similar to the reference mode shown in FIG. 15B.

[0107] Referring again to FIG. 15A, the live remote image 1510 may correspond to a portion of the zoomed out image 1550. The clinician may select a portion of the zoomed out image 1550 to be displayed in the live remote image 1510 by drawing, resizing, and / or moving a box 1555 of the zoomed out image 1550 (representing the portion displayed in the live remote image 1510). The clinician may make the selection using a mouse click, a gesture, a multi-finger gesture, etc. These selection methods may be applied to a touch screen displaying the remote image view 1500, an external touch screen, and / or another suitable input device. Various indicators may also be displayed on the remote image view 1500. As shown in FIGS. 15A-15C, the indicators include a timer and a fluoroscopy angle indicator on the live remote image 1550, and a timestamp on the live probe image 1520 (and / or on the timeline 1540).

[0108] The clinician may place a marker 1560 on a portion of the live remote image 1510 to tag features and / or detect misalignment over time. In the example shown in FIGS. 15A and 15C, the clinician may use a probe inserted into the catheter to align the catheter to the target location. When the catheter is aligned, the clinician may place a marker 1560 on the tip of the catheter. The probe may then be removed from the catheter and replaced with a tool to perform the interventional step, such as a biopsy needle. In some examples, the tool may not include imaging capabilities, so the clinician can no longer monitor the live probe view 1510 to achieve alignment. However, assuming the fluoroscopic image does not move when exchanging the probe for the tool, the clinician may use the marker 1560 to ensure that the catheter remains in substantially the same position and thus aligned with the target location.

[0109] In some examples, a timer 1570 may be displayed to indicate the elapsed time since the clinician started the time. The timer 1570 may be used, for example, to track the duration of a breath hold, the duration of a procedure, or the like. The timer 1570 may be started and / or stopped manually by the physician and / or automatically by the system when certain events are measured (such as detection of the start and end of a breath hold). The timer 1570 may change appearance (e.g., color, font, size, texture, etc.) to alert the clinician of a risk condition, such as the elapsed time exceeding a predetermined threshold (e.g., 1 minute). In some examples, an orientation icon 1580 may be displayed to indicate the viewpoint of the live remote image 1510. In the case of a fluoroscopic image, the orientation icon 1580 may indicate the angle setting of the fluoroscopic imager.

[0110] FIG. 16 is a schematic diagram of a dual screen display 1600 for displaying a graphical user interface, such as graphical user interface 400 and / or 500, according to some embodiments. In some embodiments, dual screen display 1600 may be included in display system 110. As shown in FIG. 16, dual screen display 1600 is vertically divided into an upper screen 1610 and a lower screen 1620. The upper screen 1610 and the lower screen 1620 are approximately the same size, such that dual screen display 1600 can be folded along the vertical dividing line. The upper screen 1610 displays one or more global views 1612 and one or more compact views 1614, and the lower screen 1620 displays one or more local views 1622. These views correspond to the similarly labeled views shown in FIGS. 4-5D. In the example shown in FIG. 16, the graphical user interface is in performance mode, so the views generally correspond to the views of FIG. 5D. However, it should be understood that the dual screen display 1600 may also be used to render a graphical user interface, such as in an aligned mode as shown in Figure 5A, in a navigation mode as shown in Figures 5B-5C, etc. In one example, one or both of the screens 1610 and / or 1620 may be a touch screen that a user can manipulate to change views, including rotating views, zooming, panning, switching between views, etc. In an alternative example, the screen of Figure 16 may be provided for viewing, and a separate touch screen and / or user input device may be used to change views.

[0111] 17 is a schematic diagram of a method 1700 for monitoring a medical procedure according to some embodiments. According to some embodiments consistent with FIGS. 1-16, the method 1700 can be used to operate the graphical user interface 500 in multiple modes, including an alignment mode, a navigation mode, and a performance mode.

[0112] In process 1710, while in an alignment mode of the graphical user interface and while aligning the anatomical model to the patient's anatomy, alignment progress information is displayed via the graphical user interface. In some embodiments, the alignment mode of the graphical user interface may correspond to the alignment mode of the graphical user interface 500, as shown in FIG. 5A. In some embodiments, the alignment progress information may be displayed using a dynamic point cloud view, such as dynamic point cloud view 600. In some embodiments, the alignment progress information may be displayed using a dynamic alignment guidance view, such as dynamic alignment guidance view 700.

[0113] In process 1720, once the anatomical model is aligned to the patient's body, the graphical user interface transitions from the alignment mode to a navigation mode. In some embodiments, the graphical user interface may transition automatically in response to detecting that the alignment is complete. In some embodiments, the clinician may manually transition the graphical user interface from the alignment mode to the navigation mode. For example, the clinician may click a button that appears in the alignment mode of the graphical user interface.

[0114] In process 1730, in the navigation mode and while navigating the elongated device through the patient's anatomy to the target location, navigation progress information is displayed via the graphical user interface. In some embodiments, the navigation mode of the graphical user interface may correspond to the navigation mode of the graphical user interface 500, as shown in FIG. 5B. In some embodiments, the navigation progress information may be displayed using a global anatomical model, such as global anatomical model 1000. In some embodiments, the navigation progress information may be displayed using a reduced anatomical model, such as reduced anatomical model 1100. In some embodiments, the navigation progress information may be displayed using a live camera feed augmented with supplemental guidance information, such as live camera feed 1200. In some embodiments, the navigation progress information may be displayed using a virtual tip view augmented with supplemental guidance information, such as virtual tip view 1300.

[0115] In process 1740, the graphical user interface transitions from the navigation mode to the performance mode when the elongated device is in proximity to the target location. In some embodiments, the graphical user interface may transition automatically in response to detecting that the elongated device is within a threshold distance of the target location. In some embodiments, the clinician may manually transition the graphical user interface from the navigation mode to the performance mode. For example, the clinician may click a button that appears in the navigation mode of the graphical user interface.

[0116] In process 1750, in the performance mode and during the execution of the interventional step at the target location, performance progress information is displayed via the graphical user interface. In some embodiments, the performance mode of the graphical user interface may correspond to the performance mode of the graphical user interface 500, as shown in FIG. 5D. In some embodiments, the performance progress information may be displayed using one or more views similar to those displayed in process 1730, including a global anatomical model, a reduced anatomical model, and / or a virtual tip view. In some embodiments, the performance progress information may be displayed using a remote image view, such as the remote image view 1500. In some embodiments, one or more probes may be inserted into the elongated device after the elongated device has been navigated to the target location and prior to performing the medical procedure. One or more probes, including an endoscope, an EBUS probe, or the like, may be used to fine-tune the alignment of the elongated device relative to the target location and / or capture one or more reference images. The one or more probes may then be removed and replaced with a tool for performing the interventional step, such as a biopsy needle. When a tool for performing an interventional step is inserted into the elongated device, one or more reference images can be used to ensure that the elongated device remains aligned to the target location. The clinician can scroll through one or more reference images to select which reference image to use. Once a reference image is selected, a difference image can be displayed to highlight the difference between the live remote image and the reference image. Based on the difference, the clinician can adjust the alignment of the elongated device to match the live remote image with the reference image. If the alignment is satisfactory, the interventional step can be performed.

[0117] In some embodiments, method 1700 may end after process 1750. In some embodiments, method 1700 may return to process 1730 to navigate the elongated device to one or more additional target locations. In other embodiments, method 1700 may return to process 1710 to attempt to repeat and / or improve the alignment. At any process of method 1700, the graphical user interface may alert the clinician to a detected anomaly, in which case the clinician may correct the anomaly, terminate the medical procedure, return to a previous step, and / or continue despite the anomaly.

[0118] 18 is a schematic diagram of a method 1800 for monitoring a medical procedure using a graphical user interface, according to some embodiments. According to some embodiments consistent with FIGS. 1-16, the method 1800 can be used to generate and / or display one or more views of the graphical user interface 500, including the reduced anatomical models 522 and / or 1100.

[0119] In process 1810, a route to a target location within a patient's anatomy is received. In some embodiments, the route may be determined during the planning stage of a medical procedure. The route may then be transferred to a medical instrument system, such as medical instrument system 200, that can perform the medical procedure at the target location. In some examples, the route may define a path through an airway in the lung to a target location in the lung. In some examples, the target location may include one or more of a lesion, a nodule, a tumor, or the like.

[0120] In process 1820, one or more features of the route are extracted from or determined based on a first anatomical model. In some embodiments, the first anatomical model may include a complete model of a known passageway in a relevant portion of the patient's anatomy. For example, the first anatomical model may include a model of the lung airways that was used to determine the route during planning of the medical procedure. In some examples, the first anatomical model may include an anatomical model that is registered to the patient's anatomy and displayed in a global anatomical model view, such as global anatomical model view 1000. In some examples, the extracted features may include a simplified path of the route, such as a straight line representation of the route.

[0121] In some examples, the extracted features may include landmark features that a clinician is expected to encounter when navigating a medical instrument such as a catheter along a route. In some examples, the landmark features may include a simplified map of anatomical passageways that branch off from the route (e.g., a set of branching points encountered when traversing the route). For example, the simplified map may show the locations of anatomical passageways that branch off from the route, rather than the complete branching structure of the passageways. The locations may be shown using annotations that show the location of the branching points, such as cut-out branches, dots, hash marks, etc. (as shown in Figures 11A-11C). In some examples, the simplified map may include target icons when one or more second target locations are reachable via the corresponding branching passageway. Other landmark features may include lesions, nodules, vessels, discolorations, and / or various other notable features encountered along the route.

[0122] In some embodiments, the extracted features may include a width of a passage along the route. In some examples, the width may be represented using a hierarchical format. For example, the width may be rounded to the nearest hierarchical level and / or may be approximated based on the branching generation of the passage at a given location. In some embodiments, the extracted features may include a target location and / or a location of one or more hazards in the vicinity of the target location. In some examples, the extracted features may include an insertion trajectory (e.g., an insertion angle) from the end of the route to the target location.

[0123] In process 1830, a reduced anatomical model is generated based on the one or more extracted features of the root. In some examples, the reduced anatomical model may be significantly simplified, smaller, and / or sharper than the first anatomical model because it includes the most relevant features of the first anatomical model while excluding various less relevant details. In some examples, during generation of the reduced anatomical model, the extracted features may be flattened, straightened, clipped, simplified, etc.

[0124] In process 1840, real-time location information associated with the medical instrument during the medical procedure is received. According to some embodiments, the medical device may include a steerable catheter having a location sensor, such as a shape sensor. As a result, the real-time location information may include shape data from the shape sensor. Various other real-time information associated with the medical instrument and / or the medical procedure, such as temperature data, strain data, etc., may also be received in process 1840. Receiving the real-time location information may also include determining one or more anomalies in the real-time location information. For example, excessive bending of the catheter, overheating, excessive driving force, etc. may be determined in process 1840.

[0125] In process 1850, the real-time position information is associated (e.g., via mapping) with the reduced anatomical model. In some examples, mapping the real-time position information to the reduced anatomical model may include determining how far the medical instrument extends along the simplified route path. In some examples, in process 1850, one or more anomalies may be determined, such as a wrong turn and / or extending beyond the end of the simplified route path.

[0126] In process 1860, a scaled anatomical model with real-time position information is dynamically displayed. In an example consistent with FIG. 11, the display may correspond to scaled anatomical model 1100. The display size of the scaled anatomical model may be set in different ways, such as always adjusting to fit the full width of the available screen space and / or adjusting to a size that correlates with the physical length of the path (i.e., a long path may be displayed in a larger view than a short path). If an anomaly is detected in the real-time progress information, such as the anomaly determined in process 1840 and / or 1815, the display may include one or more indicators to alert the clinician of the anomaly. In some examples, the one or more indicators may include a turn error indicator if the anomaly causes the medical device to steer beyond the end of the route. The one or more indicators may include a reversal indicator if the anomaly causes the instrument to be driven beyond the end of the route. If the anomaly includes a tight bend radius, the one or more indicators may include an over bend indicator and / or may change the appearance (e.g., color, line style, etc.) of the medical device as shown on the reduced anatomical model. After process 1860, method 1800 may return to process 1840 to continuously receive, map, and dynamically display real-time position information regarding the reduced anatomical model.

[0127] 19 is a schematic diagram of a method 1900 for displaying a patient's anatomy using a graphical user interface, according to some embodiments. According to some embodiments consistent with FIGS. 1-16, the method 1900 may be used to generate and / or display one or more views of the graphical user interface 500 and / or portions thereof, including the reduced anatomical models 522 and / or 1100.

[0128] In process 1910, a first anatomical structure is received that includes a 3D model of a plurality of passageways within the patient's anatomical structure and a route to a target location within the plurality of passageways. In some examples, the patient's anatomical structure may correspond to a lung. Consistent with such examples, the plurality of passageways may correspond to airways of the lung. In some examples, the target location may include a lesion, a nodule, a tumor, or the like.

[0129] In process 1920, a reduced anatomical model is generated based on a subset of the passageways, the subset of passageways including path passageways that are directly connected to the route, i.e., the path passageways include a subset of passageways that a medical instrument, such as a catheter, encounters (e.g., passes through and / or enters) when passing through the route.

[0130] In process 1930, the reduced anatomical model is displayed as a linear anatomical model. In some examples, the width of the path passages may be represented using vertically spaced lines with tiered separation. For example, for path passages with higher branching generations, the tiered separation may be tiered (i.e., vertically spaced lines are moved closer together). For example, in the case of the lungs, path passages closer to the trachea may have a lower branching generation and thus be depicted using more vertically spaced lines, while path passages near the end of the root (e.g., after passing through multiple branches along the root) may have a higher branching generation and thus be tiered one or more times relative to the root passages closer to the trachea. In some examples, the location of the branches off the path passages may be represented in the linear anatomical model. Because the branched passages may not be directly connected to the root, the branches may be represented as truncated branches without representing the complete branching structure (e.g., various sub-branches).

[0131] 20 is a schematic diagram of a method 2000 for displaying targets within a patient's anatomy using a graphical user interface, according to some embodiments. According to some embodiments consistent with FIGS. 1-16, the method 2000 may be used to generate and / or display one or more views of the graphical user interface 500 and / or portions thereof, including the virtual endoscopy view 1300.

[0132] In process 2010, an anatomical model is received that includes a model of the patient's anatomy. For example, the anatomical model may be a 3D model of the patient's lungs that includes a model of the lung's airways. In process 2020, a target is determined relative to the anatomical model. For example, the target may include a predetermined size and / or location of a lesion, tumor, nodule, etc. In process 2030, an uncertainty zone surrounding the target is determined. In some examples, the uncertainty zone may be determined based on an alignment uncertainty associated with registering the anatomical model to the patient's anatomy. In some examples, the uncertainty zone may be determined based on a size of the target. In some examples, the uncertainty zone may be determined based on an expected difficulty to access the target (i.e., targets that are difficult to reach may be subject to greater uncertainty). In process 2040, the target and the uncertainty zone surrounding the target are displayed relative to the anatomical model. In some examples, such as when the target and / or the uncertainty zone are outside of a passageway of the anatomical model, the passageway of the anatomical model is rendered in a manner that is semi-transparent, such that the target and the uncertainty zone are visible through the walls of the passageway.

[0133] Some examples of control units, such as control unit 130, include non-transitory, tangible, machine-readable media that includes executable code that, when executed by one or more processors (e.g., processor 140), may cause the one or more processors to perform the processes of methods 1700-2000 and / or render graphical user interfaces 400 and / or 500. Some common forms of machine-readable media that may include instructions for the processes of methods 1700-2000 and / or rendering graphical user interfaces 400 and / or 500 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, other optical media, punch cards, paper tapes, other physical media with hole patterns, RAM, PROMs, EPROMs, FLASH-EPROMs, other memory chips or cartridges, and / or other media suitable for reading by a processor or computer.

[0134] Although exemplary embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, changes, and substitutions, and in some cases, some features of the embodiments may be used without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Thus, the scope of the present invention should be limited only by the scope of the appended claims, which claims are appropriate to be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A medical device, the medical device comprising: an elongate device including a flexible body; a tracking system disposed along at least a portion of the flexible body; one or more processors communicatively coupled to the tracking system; The one or more processors receiving a route to a target location in an anatomical structure; determining one or more characteristics of the route based on an anatomical model acquired using imaging techniques and the route within the anatomical model, said determining including identifying a plurality of hierarchical levels in the anatomical model that correlate to generations of branching of the anatomical structure along the route, and identifying a plurality of branching passages that deviate from the route in one or more of the plurality of hierarchical levels, and determining passage widths of at least the plurality of hierarchical levels included in the anatomical model; generating an elongated, linearly reduced anatomical representation based on the one or more characteristics of the route, the elongated, linearly reduced anatomical representation including the respective aisle widths of each tier of the plurality of tiers along the route, the respective aisle widths of each tier being consistent along a length of each tier; receiving real-time location information from the tracking system; Associating the real-time position information with the elongated, rectilinearly reduced anatomical representation; and dynamically displaying the elongated, linearly reduced anatomical representation together with the associated real-time position information. Medical equipment.

2. 2. The medical device of claim 1, wherein the one or more characteristics of the root include a location of a bifurcation along the root, the bifurcation including a clipped bifurcation that does not have a complete bifurcation structure of the bifurcation.

3. The medical device of claim 2 , wherein an alternative route indicator is displayed, representing an alternative route to the target location.

4. The medical device of claim 1 , wherein the one or more characteristics of the route include a hazard associated with the route.

5. 10. The medical device of claim 1, wherein dynamically displaying the real-time location information includes displaying one or more indicators when an anomaly is detected, the one or more indicators including one or more of a turn error indicator, a reversal indicator, and an over-bend indicator.

6. 1. A method for displaying an anatomical structure, the method comprising: providing a graphical user interface; receiving (i) an anatomical model including a 3D representation of a plurality of passageways within the anatomical structure, and (ii) a route to a target location within the plurality of passageways in the anatomical model, the anatomical model being acquired using imaging techniques; determining one or more characteristics of the route to the target location based on (i) the anatomical model and (ii) the route, the determining step including identifying a plurality of levels in the anatomical model that correlate to generations of branching of the anatomical structure along the route, identifying a plurality of branching paths that deviate from the route in one or more of the plurality of levels, and determining path widths of at least the plurality of levels included in the anatomical model; generating an elongated, linearly reduced anatomical representation based on the one or more characteristics of the root; and displaying the elongated, linearly reduced anatomical representation; the subset of the plurality of pathways includes branch pathways connected to the route, and the elongated, linearly reduced anatomical representation includes the pathway widths for each tier of the plurality of tiers along the route, the pathway widths for each tier being consistent along the length of each tier. method.

7. The method of claim 6 , wherein the anatomical structure corresponds to lungs and the plurality of passageways in the anatomical structure correspond to airways of the lungs.

8. The method of claim 6 , wherein the target location includes one or more of a lesion, a nodule, and a tumor.

9. The method of claim 6 , wherein the locations of the off-root branches are included in the elongated, linearly reduced anatomical representation.

10. The method of claim 6 , further comprising displaying an alternative route indicator representing an alternative route to the target location.

11. The method of claim 6 , further comprising displaying an insertion trajectory to the target location.

12. The method of claim 11 , further comprising identifying hazards associated with the route and displaying the locations of the hazards relative to the insertion trajectory.

13. receiving real-time location information associated with an implement during traversal of the route; mapping the real-time position information onto the elongated, rectilinearly reduced anatomical representation; The method of claim 6 , further comprising dynamically displaying said real-time position information along with said elongated, linearly contracted anatomical representation.

14. The method of claim 13 , wherein dynamically displaying the real-time location information comprises displaying one or more indicators when an anomaly is detected.

Images