Graphical user interface for displaying guidance information in multiple modes during image bootstrap

Abstract

The invention discloses a graphical user interface for displaying guidance information in multiple modes during an image bootstrap program. A method for displaying guidance information during a medical procedure using a graphical user interface includes, in a first mode of the graphical user interface, displaying first image data from a perspective corresponding to a distal end of an elongate device. The first image data includes a virtual roadmap. The method further includes transitioning from the first mode of the graphical user interface to a second mode of the graphical user interface. The transition is based on the occurrence of a trigger condition. The method also includes displaying second image data in a second mode of the graphical user interface from a perspective corresponding to the distal end of the elongate device. The second image data includes a target indicator corresponding to the target orientation and an alignment indicator corresponding to an expected orientation of the medical procedure at the target orientation.

Description

[0001] This application is a divisional application of Chinese patent application 2021115588459, which was filed on June 29, 2017 and is entitled “Graphical user interface for displaying guidance information in multiple modes during an image-guided procedure”, wherein the Chinese patent application 2021115588459 is a divisional application of Chinese patent application 201780039120.4 (PCT / US2017 / 040067), which was filed on June 29, 2017 and is entitled “Graphical user interface for displaying guidance information in multiple modes during an image-guided procedure”.

[0002] Related Applications

[0003] This patent application claims priority to and the benefit of the filing dates of U.S. Provisional Patent Application No. 62 / 357,258, filed on June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information in a Plurality of Modes during an Image-Guided Procedure” and U.S. Provisional Patent Application No. 62 / 486,879, filed on April 18, 2017, entitled “Graphical User Interface for Monitoring an Image-Guided Procedure”, the entire contents of which are incorporated herein by reference. This application also relates to U.S. Provisional Patent Application No. 62 / 327,217, filed on June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information During an Image-Guided Procedure”, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to systems and methods for performing image-guided procedures, and more particularly to systems and methods for displaying guidance information in multiple modes during image-guided procedures. Background Art

[0005] Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort and harmful side effects. This minimally invasive technique can be performed through the natural orifice of the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, a physician can insert a minimally invasive medical device (including surgery, diagnosis, treatment or biopsy instruments) to reach the target tissue position. A such minimally invasive technique uses a flexible and / or steerable elongated device, such as a flexible catheter that can be inserted into an anatomical passage and navigated toward the region of interest in the patient's anatomical structure. Controlling this elongated device by medical personnel involves the management of several degrees of freedom, including at least the insertion of an elongated device and the management of the steering of the device with the retraction. In addition, different operating modes can also be supported.

[0006] Therefore, it would be advantageous to provide a graphical user interface that supports intuitive control and management of flexible and / or steerable elongated devices, such as steerable catheters suitable for use in minimally invasive medical techniques. Summary of the invention

[0007] The claims appended to the description best summarize embodiments of the invention.

[0008] In some embodiments, a method for displaying guidance information during a medical procedure using a graphical user interface includes: in a first mode of the graphical user interface, displaying first image data from a perspective corresponding to a distal end of a slender device. The first image data includes a virtual roadmap. The method also includes switching from the first mode of the graphical user interface to a second mode of the graphical user interface. The switching is based on the occurrence of a trigger condition. The method also includes displaying second image data from a perspective corresponding to the distal end of the slender device in the second mode of the graphical user interface. The second image data includes a target indicator corresponding to a target orientation and an alignment indicator corresponding to an expected orientation of the medical procedure at the target orientation.

[0009] In some embodiments, a system for displaying guidance information using a graphical user interface includes a slender device including a flexible body. The system also includes one or more processors configured to display first image data from a perspective corresponding to a distal end of the slender device in a first mode of the graphical user interface. The first image data includes a virtual roadmap. The one or more processors are also configured to switch from the first mode of the graphical user interface to a second mode of the graphical user interface. The switch is based on the occurrence of a trigger condition. The one or more processors are also configured to display second image data from a perspective corresponding to the distal end of the slender device in the second mode of the graphical user interface. The second image data includes a target indicator corresponding to a target position and an alignment indicator corresponding to an expected position of a medical procedure at the target position.

[0010] It should be understood that both the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure rather than to limit the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0012] Figure 1 is a simplified schematic diagram of a teleoperated medical system according to some embodiments.

[0013] Figure 2A is a simplified schematic diagram of a medical device system according to some embodiments.

[0014] Figure 2B is a simplified schematic diagram of a medical device with an extended medical tool according to some embodiments.

[0015] Figure 3A and Figure 3B is a simplified schematic diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments.

[0016] 4A is a simplified schematic diagram of a multimodal graphical user interface in a traversal mode, according to some embodiments.

[0017] 4B is a simplified schematic diagram of a multimodal graphical user interface in alignment mode in accordance with some embodiments.

[0018] Figure 5 is a simplified schematic diagram of a window for displaying a virtual endoscopic image according to some embodiments.

[0019] Figure 6 is a simplified schematic diagram of a window for displaying a virtual global image in a fly-through mode according to some embodiments.

[0020] Figure 7 is a simplified schematic diagram of a window for displaying force indicator information in accordance with some embodiments.

[0021] Figure 8 is a simplified schematic diagram of a window for displaying actuation information.

[0022] Fig. 9 is a simplified schematic diagram of a window for displaying target guidance information in alignment mode according to some embodiments.

[0023] Fig.10is a simplified schematic diagram of a window for displaying a virtual global image in alignment mode, according to some embodiments.

[0024] Fig.11 is a simplified schematic diagram of a method of displaying a multimodal graphical user interface during an image-guided surgical procedure in accordance with some embodiments.

[0025] Fig.12 is a screenshot of the multimodal GUI in Traversal Mode.

[0026] Fig.13 is a screenshot of a multimodal graphical user interface in alignment mode according to some embodiments.

[0027] By referring to the following detailed description, examples of the present disclosure and its advantages will be best understood. It should be appreciated that the same reference numerals are used to identify the same elements illustrated in one or more of the accompanying drawings, wherein the identification is for the purpose of illustrating embodiments of the present disclosure and is not intended to limit the present disclosure. DETAILED DESCRIPTION

[0028] In the following description, the specific details of some embodiments that describe the present disclosure are set forth. A lot of specific details are set forth to provide a full understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. Those skilled in the art may realize other elements within the scope and spirit of the present disclosure although not specifically described herein. In addition, in order to avoid unnecessary repetition, one or more features shown and described in conjunction with an embodiment may be incorporated into other embodiments, unless specifically described otherwise or if one or more features will make the embodiments inoperable.

[0029] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0030] The present disclosure describes different instruments and instrument parts in terms of their states in three-dimensional space. As used herein, the term "position" refers to the orientation of an object or some parts of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an object or some parts of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or some parts of an object on at least one translational degree of freedom and the orientation of an object or some parts of an object on at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0031] Figure 1 is a simplified schematic diagram of a teleoperated medical system 100 according to some embodiments. In some embodiments, the teleoperated medical system 100 may be suitable for use in, for example, a surgical, diagnostic, therapeutic, or biopsy procedure. Figure 1 As shown, the medical system 100 generally includes a teleoperation manipulator assembly 102 for operating a medical device 104 to perform various procedures on a patient P. The teleoperation manipulator assembly 102 is mounted on or near an operating table T. A master control assembly 106 allows an operator (e.g., Figure 1 The surgeon, clinician or physician (2) is shown viewing the intervention site and controlling the teleoperated manipulator assembly 102.

[0032] The master control assembly 106 may be located at a physician console, which is typically located in the same room as the operating table T, such as on one side of a surgical table on which the patient P is located. However, it should be understood that the physician O may be located in a different room or a completely different building from the patient P. The master control assembly 106 generally includes one or more control devices for controlling the remotely operated manipulator assembly 102. The control device may include any number of various input devices, such as a joystick, a trackball, a data glove, a trigger gun, a manually operated controller, a voice recognition device, a body motion or presence sensor, and / or the like. In order to provide the physician O with a strong sense of direct control of the instrument 104, the control device may be provided with the same degrees of freedom as the associated medical device 104. In this way, the control device provides the physician O with a remote presentation or perception of the control device and the medical device 104 as one.

[0033] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical instrument 104 and still provide telepresence to the physician O. In some embodiments, the control device may optionally be a manual input device that moves with six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., closing jaws, applying potentials to electrodes, delivering drug therapy, and / or the like).

[0034] The teleoperated manipulator assembly 102 supports the medical device 104 and may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, often referred to as a set-up structure) and a teleoperated manipulator. The teleoperated manipulator assembly 102 may optionally include a plurality of actuators or motors that drive inputs on the medical device 104 in response to commands from a control system (e.g., control system 112). The actuator may optionally include a drive system that, when coupled to the medical device 104, can advance the medical device 104 to a natural or surgically created anatomical orifice. Other drive systems can move the distal end of the medical device 104 with multiple degrees of freedom, which may include three degrees of freedom for linear motion (e.g., linear motion along the X, Y, Z Cartesian coordinate axes) and three degrees of freedom for rotational motion (e.g., rotation around the X, Y, Z Cartesian coordinate axes). In addition, the actuator can be used to actuate an articulated end effector of the medical device 104 to grasp tissue in the jaws of a biopsy device and / or similar device. Actuator position sensors (such as resolvers, encoders, potentiometers, and other mechanisms) can provide sensor data describing the rotation and orientation of the motor shaft to the medical system 100. Such position sensor data can be used to determine the motion of an object manipulated by the actuator.

[0035] The teleoperated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instrument of the teleoperated manipulator assembly 102. These subsystems may include: a position / orientation sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, speed, velocity, posture, and / or shape along the distal end and / or one or more segments of the flexible body that may comprise the medical instrument 104; and / or a visualization system for acquiring images from the distal end of the medical instrument 104.

[0036] The teleoperated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical instrument 104 generated by the subsystems of the sensor system 108. The display system 110 and the master assembly 106 can be oriented so that the physician O can control the medical instrument 104 and the master assembly 106 with the perception of the telepresentation.

[0037] In some embodiments, the medical device 104 can have a visualization system (discussed in more detail below) that can include a viewing mirror assembly that records concurrent or real-time images of the surgical site and provides the images to the operator or physician O through one or more displays of the medical system 100 (such as one or more displays of the display system 110). The concurrent images can be, for example, two-dimensional or three-dimensional images acquired by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that can be integrally or removably coupled to the medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly can be used with the medical device 104 to image the surgical site. The visualization system can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is executed by one or more computer processors (which can include a processor of the control system 112).

[0038] The display system 110 may also display images of the surgical site and medical instruments captured by the visualization system. In some examples, the teleoperation medical system 100 may configure the controls of the medical instruments 104 and the master assembly 106 so that the relative positions of the medical instruments are similar to the relative positions of the eyes and hands of the physician O. In this way, the physician O is able to operate the medical instruments 104 and manual controls as if observing the workspace with a substantially real sense of presence. By realistic presentation, it is meant that the presentation of the image is a realistic perspective image that simulates the perspective of the physician physically manipulating the medical instrument 104.

[0039] In some examples, display system 110 may present an image of a surgical site recorded preoperatively or intraoperatively using image data from an imaging technique such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermography, impedance imaging, laser imaging, nanotube x-ray imaging, etc. The preoperative or intraoperative image data may be presented as a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or velocity-based information) image, and / or as an image from a model generated from a preoperative or intraoperative image data set.

[0040] In some embodiments, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered (i.e., dynamically referenced) with a preoperative or concurrent image / model, typically for the purpose of an imaging-guided surgical procedure. This may be done to present a virtual image of the internal surgical site to the physician O from the perspective of the medical device 104. In some examples, the perspective may be from the tip of the medical device 104. An image of the tip of the medical device 104 and / or other graphical or alphanumeric indicators may be superimposed on the virtual image to assist the physician O in controlling the medical device 104. In some examples, the medical device 104 may not be visible in the virtual image.

[0041] In some embodiments, the display system 110 can display a virtual navigation image in which the actual position of the medical device 104 is registered with the preoperative image or concurrent image to present a virtual image of the medical device 104 within the surgical site to the physician O from an external perspective. An image of a portion of the medical device 104 or other graphical or alphanumeric indicator can be superimposed on the virtual image to help the physician O control the medical device 104. As described herein, a visual representation of the data points can be rendered to the display system 110. For example, measured data points, moved data points, registered data points, and other data points described herein can be displayed on the display system 110 in a visual representation. The data points can be visually represented on the user interface by a plurality of points or dots on the display system 110 or as a rendered model (such as a grid or wire model created based on a set of data points). In some examples, the data points can be color-coded according to the data they represent. In some embodiments, the visual representation can be refreshed in the display system 110 after each processing operation has been implemented to change the data points.

[0042] The remotely operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control assembly 106, the sensor system 108, and the display system 110. The control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing these instructions) to implement some or all of the methods described according to the various aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112 is Figure 1is shown as a single box in the simplified schematic diagram of , but the system may include two or more data processing circuits, wherein a portion of the processing is optionally performed on or adjacent to the remote operation manipulator assembly 102, another portion of the processing is performed at the main control assembly 106, and / or the like. The processor of the control system 112 can execute some instructions, including instructions corresponding to the processing disclosed in this article and described in more detail below. Any of a variety of centralized or distributed data processing architectures can be utilized. Similarly, the programmed instructions can be implemented as several independent programs or subroutines, or they can be integrated into several other aspects of the remote operation system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE802.11, DECT, and wireless telemetry.

[0043] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to the feedback, the control system 112 may transmit a signal to the master assembly 106. In some examples, the control system 112 may transmit a command to instruct one or more actuators of the teleoperated manipulator assembly 102 to move the medical device 104. The medical device 104 may extend to an internal surgical site within the body of the patient P via an opening in the body of the patient P. Any suitable conventional and / or specialized actuator may be used. In some examples, one or more actuators may be separate from the teleoperated manipulator assembly 102, or integrated with the teleoperated manipulator assembly 102. In some embodiments, one or more actuators and the teleoperated manipulator assembly 102 are provided as part of a teleoperated cart positioned adjacent to the patient P and the operating table T.

[0044] The control system 112 may optionally further include a virtual visualization system to provide navigation assistance to the physician O when controlling the medical device 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on a reference to a preoperative or intraoperative data set of an anatomical pathway that is collected. The virtual visualization system processes images of surgical sites imaged using imaging techniques (such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc.). Software that can be used in conjunction with manual input is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of local or global anatomical organs or anatomical regions. The image data set is associated with the composite representation. The composite representation and the image data set describe various orientations and shapes of the pathway and their connectivity. The images used to generate the composite representation can be recorded preoperatively or intraoperatively during a clinical procedure. In some embodiments, the virtual visualization system can use standard representations (i.e., not patient-specific) or a mixture of standard representations and patient-specific data. The composite representation and any virtual images generated from the composite representation may represent the static pose of the deformable anatomical region during one or more phases of motion (eg, during the inspiration / expiration cycle of the lungs).

[0045] During the virtual navigation procedure, the sensor system 108 can be used to calculate the approximate position of the medical device 104 relative to the anatomical structure of the patient P. The position can be used to generate a macro-level (external) tracking image of the anatomical structure of the patient P and a virtual internal image of the anatomical structure of the patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors and / or other sensors to align and display surgical images of medical practice and preoperative records, such as those from virtual visualization systems. For example, U.S. Patent Application No. 13 / 107,562 (filed on May 13, 2011) (publishing "Medical System Providing Dynamic Registration of a Model of an Anatomical Structure for Image-Guided Surgery") discloses such a system, which is incorporated herein by reference in its entirety. The remotely operated medical system 100 may further include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems and / or suction systems. In some embodiments, the remotely operated medical system 100 may include more than one remotely operated manipulator assembly and / or more than one master assembly. The exact number of teleoperated manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room, among other factors. The master control assemblies 106 may be collocated, or they may be positioned in separate locations. Multiple master control assemblies allow more than one operator to control one or more teleoperated manipulator assemblies in various combinations.

[0046] Figure 2A is a simplified schematic diagram of a medical device system 200 according to some embodiments. In some embodiments, the medical device system 200 can be used as a medical device 104 in an image-guided medical procedure performed using a teleoperated medical system 100. In some examples, the medical device system 200 can be used for non-teleoperated exploratory procedures or in procedures involving traditional manually operated medical devices (such as endoscopes). Optionally, the medical device system 200 can be used to collect (i.e., measure) a set of data points corresponding to positions within an anatomical pathway of a patient (such as patient P).

[0047] Medical device system 200 includes an elongated device 202 coupled to a drive unit 204. Elongated device 202 includes a flexible body 216 having a proximal end 217 and a distal end 218 (also referred to as a "tip portion 218"). In some embodiments, flexible body 216 has an outer diameter of approximately 3 mm. Other flexible body outer diameters may be larger or smaller.

[0048] The medical device system 200 further includes a tracking system 230 for determining the position, orientation, speed, velocity, posture and / or shape of the flexible body 216 at the distal end 218 and / or one or more segments 224 along the flexible body 216 using one or more sensors and / or imaging devices as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into a plurality of segments 224. If the medical device system 200 is consistent with the medical device 104 of the teleoperated medical system 100, the tracking system 230. The tracking system 230 can be optionally implemented as hardware, firmware, software, or a combination thereof that interacts with or is executed by one or more computer processors, the one or more computer processors may include Figure 1 A processor of the control system 112 in the system.

[0049] The tracking system 230 may optionally use a shape sensor 222 to track the distal end 218 and / or one or more segments 224. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 222 forms a fiber optic bend sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (disclosing "Fiberoptic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fiber Bend Sensor"), which are incorporated herein by reference in their entireties. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the flexible body 216 may be determined using other techniques. For example, a history of the posture of the distal end of the flexible body 216 can be used to reconstruct the shape of the flexible body 216 over the period of time. In some embodiments, the tracking system 230 may alternatively and / or additionally use the position sensor system 220 to track the distal end 218. The position sensor system 220 may include or may be a component of an EM sensor system that includes one or more conductive coils that can be subjected to an externally generated electromagnetic field. Each coil of such an EM sensor system that implements the position sensor system 220 then generates an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the position sensor system 220 may be configured and positioned to measure six degrees of freedom (e.g., three position coordinates X, Y, Z and three orientation angles of pitch, yaw, and roll indicating a base point) or five degrees of freedom (e.g., three position coordinates X, Y, Z and two orientation angles of pitch and yaw indicating a base point).Further description of the position sensor system is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999) (disclosing "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0050] In some embodiments, tracking system 230 may alternatively and / or additionally rely on historical posture, position, or orientation data stored for known points of the instrument system along cycles of alternating motion (such as breathing). This stored data may be used to develop shape information about flexible body 216. In some examples, a series of position sensors (not shown) (such as electromagnetic (EM) sensors similar to the sensors in position sensor system 220) may be positioned along flexible body 216 and then used for shape sensing. In some examples, a history of data from one or more of these sensors acquired during a procedure may be used to represent the shape of elongated device 202, particularly if the anatomical pathway is substantially static.

[0051] Flexible body 216 includes a channel 221 sized and shaped to receive a medical device 226 . Figure 2B2 is a simplified schematic diagram of the flexible body 216 with the medical device 226 extended according to some embodiments. In some embodiments, the medical device 226 can be used for procedures such as surgery, biopsy, ablation, irradiation, perfusion or suction. The medical device 226 can be deployed through the channel 221 of the flexible body 216 and used at the target position within the anatomical structure. The medical device 226 may include, for example, an image acquisition probe (probe), a biopsy instrument, a laser ablation fiber and / or other surgical, diagnostic or therapeutic tools. The medical tool may include an end effector with a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, etc. Other end effectors may include, for example, forceps, graspers, scissors, clip applicators, etc. Other end effectors may further include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc. In various embodiments, the medical device 226 is a biopsy instrument, which can be used to remove a sample of sample tissue or cells from a target structure position. The medical device 226 can be used with an image acquisition probe also in the flexible body 216. In various embodiments, the medical device 226 can be an image acquisition probe, which includes a distal portion having a stereo camera or a monoscopic camera at or near the distal end 218 of the flexible body 216 for acquiring images (including video images), which are processed by the visualization system 231 for display and / or provided to the tracking system 230 to support tracking of one or more of the distal end 218 and / or the segments 224. The image acquisition probe can include a cable coupled to the camera for transmitting the acquired image data. In some examples, the image acquisition instrument can be a fiber optic bundle coupled to the visualization system 231, such as a fiberscope. The image acquisition instrument can be single-spectrum or multi-spectrum, for example, acquiring image data in one or more of the visible, infrared and / or ultraviolet spectra. Alternatively, the medical device 226 itself can be an image acquisition probe. The medical device 226 can be advanced from the opening of the channel 221 to perform a procedure, and then retracted into the channel when the procedure is completed. The medical device 226 may be removed from the proximal end 217 of the flexible body 216 or from another optional instrument port (not shown) along the flexible body 216 .

[0052] The medical device 226 may additionally house a cable, linkage, or other actuation control device (not shown) extending between its proximal and distal ends to controllably bend the distal end of the medical device 226. Steerable instruments are described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005) (disclosing "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (disclosing "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0053] The flexible body 216 may also house cables, linkages, or other steering controls (not shown) extending between the drive unit 204 and the distal end 218 to controllably bend the distal end 218 as shown, for example, by the dashed line depiction 219 of the distal end 218. In some examples, at least four cables are used to provide independent "up and down" steering to control the pitch of the distal end 218 and "left and right" steering to control the yaw of the distal end 281. Steerable catheters are described in detail in U.S. Patent Application No. 13 / 274,208 (filed on October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. In embodiments where the medical device system 200 is actuated by a remotely operated assembly, the drive unit 204 may include a drive input device that is removably coupled to a drive element (such as an actuator) of the remotely operated assembly and receives power from the drive element. In some embodiments, the medical device system 200 may include a clamping feature, a manual actuator, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator-controlled bending of the distal end 218. In some examples, one or more lumens are defined in the wall of the flexible body 216 through which a medical device can be deployed and used at a target surgical location.

[0054] In some embodiments, the medical device system 200 may include a flexible bronchial device, such as a bronchoscope or bronchial catheter for use in inspection, diagnosis, biopsy, or treatment of the lungs. The medical device system 200 is also suitable for navigating and treating other tissues via natural or surgically established connecting channels in any of a variety of anatomical systems, including the colon, intestines, kidneys and calyces, brain, heart, circulatory system including vascular system, etc.

[0055] Information from tracking system 230 may be sent to navigation system 232, where it is combined with information from visualization system 231 and / or preoperatively acquired models to provide real-time position information to a physician, clinician, or surgeon or other operator. In some examples, the real-time position information may be displayed on a Figure 1 on the display system 110 for controlling the medical device system 200. In some examples, Figure 1 The control system 116 can use the position information as feedback to position the medical instrument system 200. Various systems for utilizing fiber optic sensors to align and display surgical instruments with surgical images are provided in U.S. patent application Ser. No. 13 / 107,562, filed on May 13, 2011, and disclosed as “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery,” which is incorporated herein by reference in its entirety.

[0056] In some examples, the medical device system 200 may be Figure 1 In some embodiments, Figure 1 The teleoperated manipulator assembly 102 may be replaced by direct operator controls. In some examples, the direct operator controls may include various handles and operator interfaces for handheld operation of the instrument.

[0057] Figure 3A and Figure 3B is a simplified schematic diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments. Figure 3A and Figure 3BAs shown, the surgical environment 300 includes a patient P positioned on a platform 302. The patient P can be stationary in the surgical environment in the sense that significant patient movement is limited by sedation, restraints, and / or other means. Periodic anatomical motion (including breathing and cardiac motion of the patient P) can continue unless the patient is told to hold his or her breath to temporarily suspend respiratory motion. Therefore, in some embodiments, data can be collected at a specific phase in breathing and marked and identified using the phase. In some embodiments, the phase during which data is collected can be inferred based on physiological information collected from the patient P. In the surgical environment 300, a point collection instrument 304 is coupled to an instrument bracket 306. In some embodiments, the point collection instrument 304 can use an EM sensor, a shape sensor, and / or other sensor modalities. The instrument bracket 306 is mounted to an insertion table 308 fixed in the surgical environment 300. Alternatively, the insertion table 308 can be movable, but has a known position in the surgical environment 300 (e.g., known via a tracking sensor or other tracking device). The instrument bracket 306 can be a component of a teleoperated manipulator assembly (e.g., teleoperated manipulator assembly 102) that is coupled to the point collection instrument 304 to control the insertion motion (i.e., motion along the A axis) and optionally control the motion of the distal end 318 of the elongated device 310 in multiple directions (including yaw, pitch, and roll). The instrument bracket 306 or insertion stage 308 may include an actuator (not shown), such as a servo motor, that controls the motion of the instrument bracket 306 along the insertion stage 308.

[0058] The elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled and fixed relative to the instrument bracket 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 can be movable along with the instrument body 312, but the position of the proximal point 316 can be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as a distal end 318 of the elongated device 310). The point collection instrument 304 can be substantially similar to the medical instrument system 200.

[0059] As the instrument body 312 moves along the insertion axis A on the insertion stage 308, the position measurement device 320 provides information about the position of the instrument body 312. The position measurement device 320 may include a resolver, an encoder, a potentiometer, and / or other sensor that determines the rotation and / or orientation of the actuator that controls the movement of the instrument bracket 306 and, therefore, the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be arcuate, or have a combination of arcuate and linear segments.

[0060] Figure 3A The instrument body 312 and instrument bracket 306 are shown in a retracted position along the insertion table 308. In this retracted position, the proximal point 316 is at position L0 on the axis A. In this position along the insertion table 308, the A component of the orientation of the proximal point 316 can be set to zero and / or another reference value to provide a basic reference describing the position of the instrument bracket 306 and therefore the proximal point 316 on the insertion table 308. With this retracted position of the instrument body 312 and instrument bracket 306, the distal end 318 of the slender device 310 can be positioned just inside the access orifice of the patient P. Also in this position, the position measurement device 320 can be set to zero and / or another reference value (e.g., I=0). Figure 3B 308, the instrument body 312 and the instrument bracket 306 have been advanced along the linear track of the insertion table 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advanced position, the proximal point 316 is at a position L1 on the axis A. In some examples, encoders and / or other position data from one or more actuators that control the movement of the instrument bracket 306 along the insertion table 308 and / or one or more position sensors associated with the instrument bracket 306 and / or the insertion table 308 are used to determine the position Lx of the proximal point 316 relative to the position L0. In some examples, the position Lx can be further used as an indicator of the distance or insertion depth of the distal end 318 of the elongated device 310 inserted into the passage of the anatomical structure of the patient P.

[0061] 4A and 4B are simplified schematic diagrams of a multimodal graphical user interface 400 that may be displayed on display system 110 according to some embodiments. Figure 1 3 , a multimodal graphical user interface 400 can be used to assist an operator (e.g., a physician, clinician, or physician O) during operation and / or control of a medical device system (e.g., a teleoperated medical system 100 and / or medical system 200). For example, a medical device system can include an elongated device (e.g., elongated device 202) and one or more medical devices (e.g., medical device 226) inserted into a flexible body of the elongated device.

[0062] The multimodal graphical user interface 400 displays information in one or more windows 410-480 that the operator can watch. Although five or six windows that can be watched at the same time on a single screen are depicted in Figures 4A and 4B, respectively, it should be understood that the graphical user interface 400 can display any suitable number of windows on any suitable number of screens. In some examples, the number of windows that can be watched at the same time can be changed by opening and closing windows, minimizing and maximizing windows, moving windows between the foreground and background of the multimodal graphical user interface 400, switching between screens, and / or otherwise completely or partially blocking windows to avoid watching. Similarly, the arrangement of windows 410-480 (including their size, shape, orientation, sorting (in the case of overlapping windows) and / or similar arrangements) can change and / or can be user-configurable.

[0063] According to some embodiments, windows 410-480 may display image data, sensor data, indicators, control modes, and / or any combination thereof. In some examples, the image data may include preoperative or intraoperative image data. The image data may be presented as a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or speed-based information) live image, and / or presented as an image of a computational model created based on a preoperative or intraoperative image data set. In some examples, the image of the computational model may be derived from the sensor data and may include a model of an instrument introduced into an anatomical structure. In some examples, the computational model may be created based on empirical data (in addition to or in lieu of image data) and / or may be based on a predetermined geometry of an instrument and / or human anatomical structure. In some examples, the indicator may include a graphical indicator and / or an alphanumeric indicator. In some examples, the control may include a button, a text input device, a navigation panel, a task bar, an icon, an alert, and / or a similar control. According to some embodiments, the multimodal graphical user interface 400 may include a settings window 440 that displays a list of settings associated with an available control mode, a current control mode, and / or a medical device.

[0064] 4A and 4B depict multimodal graphical user interface 400 in traversal mode and alignment mode, respectively. According to some embodiments, traversal mode may be suitable for display when an elongated device is navigated through a patient's body over a considerable distance (e.g., using posture adjustments combined with substantial insertion changes to navigate through an anatomical pathway toward a target location). According to some embodiments, the target location may correspond to a target surgical location and / or a target anatomical location, as described above with respect to Figure 2A and Figure 2BAs discussed. The alignment mode may be adapted for display during adjustment of the posture of the elongated device and / or minor insertion changes (eg, while collecting a biopsy sample and / or performing laser ablation at a target location).

[0065] In some examples, the traversal mode is used when the distal end of the elongated device is away from the target location and substantial movement of the elongated device (e.g., insertion, retraction, articulation, and / or the like) is expected. For example, the traversal mode can be used to explore an anatomical pathway and / or when tracking a predetermined route to a target location. In contrast, the alignment mode is used when the distal end of the elongated device is close to the target location and fine adjustments to the posture of the elongated device are expected (e.g., when the distal end of the elongated device is manually and / or automatically oriented at an optimal angle and / or distance relative to the target location for successful delivery of the needle). For example, the alignment mode can be used when the elongated device is within two or three centimeters of the target location, and / or when there is a line of sight between the elongated device and the target location (i.e., there are no obstacles and / or bends in the anatomical pathway that separate the target location from the elongated device). In some examples, the alignment mode can be used when a live camera feed from the distal end of the elongated device is not available (i.e., when a "blind" transmission is in progress), such as after an endoscope used to assist in navigation during a fly-through mode is removed from the elongated device in preparation for a surgical procedure at a target location.

[0066] In some examples, the multimodal graphical user interface 400 can be manually and / or automatically switched between the traversal mode and the alignment mode. The user-initiated conversion can be performed by an operator, for example, by clicking a button, manually toggling a switch, pressing a pedal using a touchpad, announcing a voice command, and / or any other suitable switching input from the user. In some examples, the automatic conversion can be performed based on one or more triggering events occurring during the course of the medical procedure. For example, the trigger condition can include removing a camera probe from the elongated device, which provides image data displayed via the multimodal graphical user interface 400 in the traversal mode. In some examples, the removal of the camera probe from the elongated device can be detected based on a change in the shape of the elongated device as determined using a tracking system (e.g., a tracking system 230 including a shape sensor). In some examples, the removal of the camera probe from the elongated device can be detected based on a change in the image data provided by the camera probe (as determined using an image processor). In some examples, the change in image data can be provided by a real imaging technology (e.g., computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging). The change in image data can include new images acquired using the imaging technology at different angles relative to the target or at different times in the procedure. The trigger condition can include the attachment or disconnection of the imaging technology to the system. In some examples, the trigger condition can include the insertion of a medical device into the elongated device. Similar to removing a camera probe from an elongated device, the insertion of a medical device into the elongated device can be determined based on a change in the shape of the elongated device. In some examples, the trigger condition can include detecting that the distal end of the elongated device is within a predetermined distance of the target location, such as two centimeters or three centimeters. In some examples, the trigger condition can include determining that there is a line of sight between the distal end of the elongated device and the target location (i.e., there are no obstacles and / or bends in the anatomical pathway). In some examples, the trigger condition may include detecting that there are no remaining branches in the anatomical pathway separating the distal end of the elongated device from the target location, meaning there is no remaining risk of "losing" by navigating to the wrong branch. In some examples, the trigger condition may include reaching the end of the predetermined route to the target location. In some examples, the trigger condition may include detecting that the operator has stopped inserting the elongated device (i.e., when the elongated device has been "parked"). In some examples, the operator may have the option of overriding the automatic mode selection by manually changing the automatic mode selection.

[0067] In some examples, any suitable combination of one or more trigger conditions may be used to automatically switch between the traversal mode and the alignment mode. For example, the trigger condition may include detecting that both of the following conditions are met: (1) the elongated device is within a predetermined distance of the target location, and (2) the operator has stopped inserting the elongated device.

[0068] In the fly-through mode, as depicted in FIG. 4A , the multimodal graphical user interface 400 may display a camera view window 410, a virtual endoscope view window 420, a virtual front view window 430, a status window 435, a settings window 440, a force indicator window 445, and an actuation information window 450. The camera view window 410 displays camera data captured by a visualization system (e.g., a visualization system of the medical device 104). For example, the camera data may include a live video feed captured by an endoscope and / or a stereo camera or a monoscopic camera at or near the distal end of the elongated device. One or more indicators and / or controls may be superimposed on and / or displayed with the image data to assist the operator in controlling the elongated device.

[0069] The virtual endoscope view window 420 displays virtual image data from a viewpoint corresponding to the distal end of the elongated device. In this regard, the virtual endoscope view window 420 simulates the field of view of an endoscope. According to some embodiments, the virtual image data may be generated by a virtual visualization system (e.g., a virtual visualization system of the control system 112) using, for example, a preoperative image data set. According to some embodiments, the virtual image data displayed by the virtual endoscope view window 420 may be enhanced to display supplemental guidance information to the operator. Figure 5 Embodiments of the virtual endoscope view window 420 are discussed in greater detail.

[0070] The virtual global view window 430 displays virtual image data from a viewpoint corresponding to a zoomed-out view of the patient P. In this manner, the virtual global view window 430 simulates the field of view of an observer (e.g., physician O). In some examples, the view orientation may be selected manually and / or automatically. As with the virtual endoscope view window 420, the virtual image data may be generated by a virtual visualization system (e.g., a virtual visualization system of the control system 112). In some examples, the virtual image data may display the real-time position of the elongated device within the patient's anatomy. Figure 6 Embodiments of the virtual global view window 430 are discussed in more detail.

[0071] When windows 410-430 are displayed simultaneously, the images displayed in windows 410-430 advantageously allow the operator to simultaneously monitor and / or visualize (via camera view window 410 and / or virtual endoscope view window 420) an area near the distal end of the elongated device and (via virtual global view window 430) a broader posture of the elongated device relative to the patient's anatomy.

[0072] The status window 435 displays the system status during system setup and / or during system use. In some examples, the status may include the presence of a probe, the presence of a medical tool, the engagement of a sterile adapter or ISA, an error or FRL status, the correct placement of an elongated device, and the incorrect placement of an elongated device when the elongated device is displaced. The status window may also provide instructions, including when to install or remove an elongated device, how to recover from a system failure, or other workflow instructions. The information may be colored or of different sizes based on attributes such as the relative urgency, type, or intended audience of the information.

[0073] The force indicator window 445 displays information about the force applied by the actuator or motor during advancement of the elongated device. The force indicator can provide information about the state of the system. In some examples, a high force indicator over an extended period of time can indicate that a component of the system (e.g., a drape, a medical device, etc.) is caught or stuck during insertion. In some examples, the force displayed can be the instantaneous force at that time. In other examples, the force displayed can indicate some recent history of the force, such as a graph of the force over the last 30 seconds to 120 seconds. See below for more information. Figure 7 Embodiments of force indicator window 455 are discussed in greater detail.

[0074] The actuation information window 450 displays information about the actuation of the steerable portion of the elongated device. In some examples, the actuation information can be used to warn the operator of problems and / or potential problems that occur during the steering of the elongated device. The actuation information can additionally provide assistance to the operator to correct, avoid, and / or mitigate detected problems. Figure 8 Embodiments of actuating the information window 450 are discussed in greater detail.

[0075] In alignment mode, as depicted in FIG. 4B , multimodal graphical user interface 400 may display a target guidance view window 460, virtual global view windows 470 and 480, camera view window 410, settings window 440, force indicator window 445, and actuation information window 450. Target guidance view window 460 displays the target position from a viewpoint corresponding to the distal end of the elongated device. In some examples, when multimodal graphical user interface 400 is switched from traverse mode to alignment mode, target guidance view window 460 may replace virtual endoscope view window 420. According to some embodiments, target guidance view window 460 may display guidance information designed to assist an operator in steering an elongated device from a close range to a target position. Fig. 9 Embodiments of the target-guided view window 460 are discussed in greater detail.

[0076] Virtual global view windows 470 and 480 display virtual image data similar to virtual global view window 430. According to some embodiments, the perspectives of virtual global view windows 470 and 480 can be rotated relative to virtual global view window 430 and / or relative to each other. In some examples, the perspectives of virtual global view windows 470 and 480 can be manually and / or automatically selected. In some examples, the perspectives have a fixed offset (e.g., a 90 degree offset to maintain orthogonality) so that rotating one of the perspectives automatically rotates the other perspective by a corresponding amount. In some examples, one of the perspectives can be automatically selected so that a needle extending from an elongated device appears within a plane. In some examples, one of the perspectives can be automatically selected to match the view of a fluoroscopic imaging device used to observe the procedure. According to some embodiments, the virtual image data displayed in virtual global view windows 470 and 480 can be simplified relative to virtual global view window 430 so as to reduce the amount of processing used to generate the virtual image data. Reference is made below Fig.10 Embodiments of virtual global view windows 470 and 480 are discussed in more detail.

[0077] In the alignment mode, the camera view window 410 is optionally included in the multimodal graphical user interface 400. However, in some embodiments, when the multimodal graphical user interface 400 is in the alignment mode, the camera (e.g., endoscope) may be disabled and / or retracted. Therefore, the multimodal graphical user interface 400 may not include a camera view window in the second mode. Alternatively or additionally, the camera view window 410 may display an indicator (e.g., text) to convey to the operator that image data from the camera is not available.

[0078] In some examples, when transitioning from the through mode to the alignment mode, the multimodal graphical user interface 400 can provide one or more additional visual indicators. In some examples, the screen border of the graphical user interface 400 can transition from a first color, texture, shading, and / or transparency in the through mode to a second color, texture, shading, and / or transparency in the alignment mode. For example, the screen border can change from yellow in the through mode to red in the alignment mode. In some examples, the multimodal graphical user interface 400 can include icons that change color in a similar manner.

[0079] Figure 5 An exemplary window 500 for displaying a virtual endoscopic image according to some examples is shown. Figure 1 - In some embodiments of FIG. 4 , window 500 may correspond to virtual endoscope view window 420. Figure 5 As depicted, window 500 displays anatomical features 510 from a front-view perspective of the distal end of an elongated device inserted into a patient. In some examples, anatomical features 510 may include anatomical features of interest, such as anatomical pathways, blood vessels, organs, and / or the like. Anatomical features 510 may be displayed using virtual renderings and / or images generated from image data sets acquired preoperatively or intraoperatively. In some examples, anatomical features 510 may be displayed as semi-transparent features and / or outlines, such that features within and / or behind a given anatomical feature may be displayed simultaneously in window 500.

[0080] A virtual roadmap 520 displayed in window 500 indicates a predetermined route to a target location. Virtual roadmap 520 serves as a guide for the operator when steering the elongated device through the patient's anatomy. For example, virtual roadmap 520 may indicate which branches in the anatomical pathways steer the elongated device toward and / or away from the target location. Figure 5As depicted, the virtual roadmap 520 is displayed as a line that winds through the anatomical passage along a predetermined route. In some examples, the color, shading, texture, and / or transparency of the line can vary to indicate proximity to the target position, deviation from the predetermined route, the presence of bends, branches, and / or constrictions in the anatomical passage (which can be pre-calculated to warn the operator of an upcoming complex maneuver), the diameter of the anatomical passage, and / or other supplementary information. In some examples, various alternative or additional mapping indicators can be included as part of the virtual roadmap 520. For example, the virtual roadmap 520 can include one or more arrows indicating the direction of the turn. In some examples, the virtual roadmap 520 can include one or more alphanumeric indicators that indicate the remaining distance to the target position, the remaining time to reach the target position, the identity of the anatomical passage (e.g., trachea, RMB, B6, and / or similar identities), and / or the like. In some examples, the virtual roadmap 520 can include scale markings, such as distance-target scale markings. In some examples, navigating the elongated device in an incorrect direction that deviates from the intended route can cause one or more walls of the anatomical pathway to be rendered transparently so that the virtual roadmap 520 can be seen through the obstruction. In some examples, the virtual roadmap 520 can include audio instructions (e.g., "turn right") and / or tactile feedback (e.g., vibration of a surgical control when the elongated device deviates from the intended route).

[0081] Roadblock indicators 530 displayed in window 500 indicate undesirable anatomical features or anatomical pathways and / or directions in which the elongated device should not or cannot be steered. Figure 5As depicted, the roadblock indicator 530 includes an icon placed above the anatomical pathway. In some examples, the icon may include a "no entry" sign, a stop sign, a warning symbol, a red or yellow light, and / or the like. In some examples, the roadblock indicator 530 may be located on and / or near an anatomical pathway to which the elongated device has previously been diverted. In further examples of such embodiments, the roadblock indicator 530 may display clinical information associated with previous tests and / or procedures performed in the anatomical pathway (e.g., "biopsy result is negative"). In some examples, the roadblock indicator 530 may be located on and / or near an anatomical pathway to which it would be dangerous and / or impossible to divert the elongated device. For example, the roadblock indicator 530 may mark an anatomical pathway that is too narrow to fit the elongated device or a bend in the anatomical pathway may be too tight (e.g., a bend radius may be too tight to allow the elongated device to traverse). In some examples, the roadblock indicator 530 may be displayed by changing the color, shading, texture, and / or transparency of the anatomical pathway to be avoided. In some examples, the roadblock indicator 530 can include audio feedback (e.g., "warning" or "stop") and / or tactile feedback (e.g., vibration of the surgical control when the elongated device is turned toward a prohibited path). In some examples, the roadblock indicator 530 can appear when the operator attempts to turn the elongated device in an incorrect direction, and can then disappear when the operator corrects the course of the elongated device (e.g., by retracting the elongated device and / or pivoting the elongated device toward the desired path).

[0082] Figure 6 An exemplary window 600 for displaying a virtual global image in a fly-through mode is shown according to some examples. Figure 1 In some embodiments of FIG. 4 , window 600 may correspond to virtual global view window 420. Figure 6 As depicted, window 600 displays anatomical features 610 and elongated device 620 from a “global” perspective (e.g., the perspective of a physician O caring for a patient P). In some examples, anatomical features 610 may include anatomical features of interest, such as anatomical pathways, blood vessels, organs, and / or the like. Anatomical features 610 may be displayed using, for example, virtual renderings and / or images generated from medical imaging data sets acquired preoperatively or intraoperatively. In some examples, anatomical features 610 may be displayed as semi-transparent features and / or outlines, such that features within and / or behind a given anatomical feature may be displayed simultaneously in window 600.

[0083] In some examples, orientation indicator 615 is used to indicate the viewing angle of window 600. Figure 6As depicted, orientation indicator 615 may include a virtual figurine positioned at an angle that represents the orientation of the global view relative to the patient's body. According to some embodiments, the viewing angle of window 600 may change over time, in which case orientation indicator 650 may be continuously updated to reflect the changed viewing angle. In some examples, orientation indicator 615 may include an icon, such as an arrow, indicating a reference axis or direction (e.g., the direction of gravity). In some embodiments, orientation indicator 615 may include an alphanumeric indicator, such as a text label (e.g., "front view" or "side view") and / or a coordinate label (e.g., polar coordinates representing the viewing angle). In some examples, orientation indicator 615 may utilize a text label based on an x-ray marking scheme (e.g., "AP" for anterior and posterior views and "PA" for posterior and anterior views). Orientation indicator 615 may be located at any suitable location within and / or near window 600. In some examples, orientation indicator 615 may be located outside window 600 so as not to obscure the virtual image.

[0084] In some examples, a field of view indicator 625 is displayed at the distal end of the elongated device 620. Figure 4a In some embodiments, the field of view indicator 625 provides a representation of the field of view seen in the camera view window 410 and / or the virtual endoscope view window 420. In this way, the operator can intuitively grasp the correspondence between the images seen in each window. In some examples, the field of view indicator 625 can be displayed as a prism and / or cone shape. In some examples, a miniature version of the camera view window 410 and / or the virtual endoscope view window 420 can be projected onto the flat surface of the field of view indicator 625 to further enhance the correspondence between each of the various windows. In some examples, the field of view indicator 625 can be represented as a ray emitted from the distal end of the slender device 620.

[0085] The current target position 630 is displayed in window 600. Figure 1-Consistent with the embodiments discussed in FIG. 4 , the current target position 630 can correspond to a target surgical position and / or a target anatomical position. In some examples, the current target position 630 can be depicted as a circle, a sphere, a target (e.g., concentric circles), a crosshair, and / or any other suitable shape / design. In some examples, the shape of the current target position 630 can be determined using preoperative voxel data such as medical imaging data. For example, the current target position 630 can realistically depict the shape of a tumor. In some examples, the color, texture, shading, and / or transparency of the current target position 630 can be variable. In some examples, the current target position 630 can flash and / or otherwise have a time-varying aspect of its appearance to attract the operator's attention. A virtual roadmap 635 can display a predetermined route to the current target position 630. In some examples, the virtual roadmap 635 can have a shape that is consistent with previous descriptions of the route. Figure 5 The virtual roadmap 520 discussed herein has substantially similar properties and / or functionality.

[0086] In some examples, one or more previous target locations and / or future target locations 640 can also be displayed in window 600. Target location 640 is usually used for information purposes to help the directional operator and provide background / context for the current stage of the program. In some examples, target location 640 can be selectable, in which case the selected target location becomes the current target location. In some examples, the size, color, texture, shadow and / or transparency of target location 640 can be different from the properties of the current target location 630 so that they can be distinguished. Similarly, historical path 645 can indicate the path and / or part of the path that the elongated device 620 has previously traveled during the program. For example, historical path 645 can represent the path taken during calibration, during navigation to the previous target location and / or during previous attempts to navigate to the current target location 630. In some examples, various portions of the history path 645 may be displayed using different colors, textures, shading, and / or transparencies to indicate the purpose and / or time of a particular portion of the path (e.g., calibration, previous attempts to reach the current target location 630, and / or previous attempts to reach a different target location). In some examples, the history path 645 may be temporarily displayed and subsequently hidden from view. For example, the history path 645 may be temporarily displayed to verify the registration of the anatomical feature 610 and the history path 645. That is, when the registration is correct, the history path 645 is expected to be aligned with the anatomical feature 610, and when the registration is incorrect, the history path 645 is expected to be misaligned with the anatomical feature 610. When the registration is incorrect, the operator may resolve the problem by, for example, retracting the elongated device 620 and / or recalibrating appropriately. When the registration is correct, the operator may disable / hide the history path 645 and continue to navigate the elongated device 620 to the target location 630.

[0087] Figure 7 is a simplified schematic diagram of a window 700 for displaying force indicator information according to some embodiments. Figure 1- In some embodiments of FIG. 4 , window 700 may correspond to force indicator window 445. In some examples, the information provides a representation of the insertion force applied by the motor or actuator used to propel the elongated device. The information may be displayed in the form of a scrolling bar graph to provide a history of the insertion force over time, with the leftmost vertical bar on the graph representing the oldest force measurement and the rightmost vertical bar representing the most recent force measurement. In some examples, the time difference between the oldest force measurement and the most recent force measurement may be 90 seconds, but may represent any larger or smaller time period. The size of the vertical bar may indicate the magnitude of the force measurement. In some examples, the vertical bar may be displayed in different colors, shades, textures, and / or transparencies representing the magnitude of the force measurement. In some examples, red may be used to indicate a force measurement above a preset threshold, yellow may be used to indicate a force measurement within a preset intermediate threshold range, and green may be used to indicate a force measurement below a preset threshold.

[0088] In some embodiments, force measurements may be obtained using various sensors and calculated using various algorithms. For example, force from a force sensor at the distal end of the elongated device, or force measured by an insertion motor. The measured force may then be used to calculate the force of interest, possibly minus modeled gravity or friction forces.

[0089] Figure 8 An exemplary window 800 for displaying actuation information according to some embodiments is shown. Figure 1 4, window 800 may correspond to actuation information window 450. In some examples, window 800 is used to display actuation information associated with the bending radius of the flexible body of the elongated device, which is measured using a shape sensing probe (e.g., a fiber optic bend sensor) disposed along the length of the elongated device. According to some embodiments, window 800 may display one or more graphical indicators including an actuation information icon 810.

[0090] In general, the actuation information icon 810 provides actuation information that helps an operator detect and / or correct problems encountered when controlling an elongated device, such as an overbend of the elongated device. For example, an overbend condition may be problematic when a medical device used during a procedure cannot fit through the elongated device due to the overbend condition. In some examples, the actuation information icon 810 may include an alphanumeric indicator 812 that displays the minimum bend radius of the elongated device to alert the operator to the overbend condition. In some examples, the alphanumeric indicator may display a numerical value that continuously updates as the tightest bend radius changes, but switches to an alpha value (e.g., "yes" or "pass") when the tightest bend radius equals a value that has been predetermined to allow safe passage of the instrument. In some examples, the value may be an alpha value that displays "pass" or "fail," "yes" or "no," and / or may be another binary indicator of a sufficiently large bend radius that allows passage of the instrument at the length of the elongated device.

[0091] In some examples, the actuation information icon 810 may include a direction indicator 814 (e.g., an arrow) indicating which direction the operator should turn the elongated device to alleviate the overbending condition. In some examples, the direction indicator 814 may appear when the overbending condition is detected and may disappear when the overbending condition is alleviated. In some examples, the color, size, texture, and / or other attributes of the actuation information icon 810, the alphanumeric indicator 812, and / or the direction indicator 814 may be dynamic so as to convey supplementary guidance information to the operator. For example, different colors may correspond to different ranges of bending radius (e.g., red corresponds to a bending radius of 1-10, green corresponds to a bending radius of more than 50, and yellow and / or a gradually shifted shade of color from red to orange to yellow to green corresponds to a bending radius of 11-49). The color scheme may be applied to the actuation information icon 810 and / or portions thereof, such as the direction indicator 814 and / or the digital indicator 812.

[0092] exist Figure 8 In the illustrative example depicted in , as depicted by alphanumeric indicator 812, the minimum bend radius of the elongated device (i.e., the minimum bend radius along the length of the elongated device) is 11 mm. The smaller the number, the tighter the bend radius. To increase the minimum bend radius, as depicted by the arrow of direction indicator 814, the operator is instructed to navigate a control device, such as a joystick, trackball, and / or the like, downward and leftward. For example, in one or more embodiments, actuation information icon 810 may depict a top view of a trackball used by the operator to control the bend of the elongated device. In further examples of such embodiments, direction indicator 814 may indicate the direction in which the trackball should be rolled to straighten the elongated device.

[0093] Fig. 9 A window 900 is shown for displaying target guidance information in alignment mode according to some embodiments. Figure 1 - In some embodiments of FIG. 4 , window 900 may correspond to target guidance window 460. Window 460 displays a target indicator at target location 910 from a viewpoint corresponding to looking forward from the distal end of the elongated device. According to some embodiments, window 900 may display guidance information designed to help an operator align the elongated device with the target location 910 from a close distance. Unlike the virtual endoscopic view (e.g., virtual endoscopic view window 420 and / or window 500), window 900 may not depict any anatomical features. The absence of anatomical features in window 900 has the advantage of drawing the operator's attention to target location 910. The absence of anatomical features in window 900 has the further advantage of reducing the computational complexity involved in rendering window 900, which reduces time lag (log). In some examples, it is possible to display window 900 without any anatomical features in alignment mode because the elongated device has been navigated to within a close distance of target location 910 (e.g., within two or three centimeters). For example, there may be no other branches, bends, and / or obstructions in the patient's anatomy between the distal end of the elongated device and the target location 910. However, in some embodiments, the window 900 may include a representation of anatomical features, a subset of anatomical features, and / or a simplified representation of anatomical features for purposes such as safety, avoiding proximity of anatomical structures (e.g., an affected part) to the target location 910, and / or other functional considerations.

[0094] In some examples, the target position 910 can be represented as a circle and / or a set of concentric circles. In some examples, the shape of the target position 910 can be determined using preoperative medical imaging data. The color, size, texture, shading, and / or transparency of the target position 910 can vary depending on the distance between the distal end of the elongated device and the target position 910. According to some embodiments, an alignment indicator 920 is displayed in the window 900 to identify where the medical device is aligned relative to the target position 910. For example, the alignment indicator 920 can include a crosshair marking the center of the window and / or a position within the window 900 where a medical procedure (e.g., biopsy and / or laser ablation) will be performed, which can be off-center and / or at multiple locations. The crosshair can correspond to the radial center of the elongated object, which is related to the central axis of the instrument (e.g., a probe or biopsy needle) delivered through the elongated object. According to some embodiments, the color, texture, shading, and / or transparency of the alignment indicator 920 can vary based on the quality of the alignment between the alignment indicator 920 and the target position 910. For example, the alignment indicator 920 may turn green when the desired alignment is achieved, turn yellow when the alignment is close but not completely aligned, and turn red when it is not aligned. In some examples, the properties of the target orientation 910 that change in response to the distance between the distal end of the slender device and the target orientation 910 may be different from the properties of the alignment indicator 920 that change in response to the alignment. In this way, confusing situations are avoided, where, for example, the colors of the target orientation 910 and the alignment indicator 920 change simultaneously in different ways. The quality of the alignment between the alignment indicator 920 and the target orientation 910 and / or the distance between the distal end of the slender device and the target orientation 910 can be determined based on the position and / or orientation of the slender device sensed by a tracking system (e.g., tracking system 230). In some examples, the window 900 may additionally or alternatively display a side view between the distal end of the elongated device and the target orientation 910. The side view may help to visualize the distance between the distal end of the elongated device and the target orientation 910 because the distance is originally different from the distance between the distal end of the elongated device and the target orientation 910. Fig. 9 The perspectives depicted in are orthographic.

[0095] In some examples, one or more indicators 930 may be displayed in window 900 to indicate the location of previous procedures performed at target orientation 910. For example, when biopsy tissue has been previously collected from various angles of entry, the operator may find it useful to visualize different previous methods when queuing for the current biopsy. Indicator 930 may be displayed based on stored needle trajectory information. In some examples, needle trajectory information is generated and stored when the operator manually indicates that a biopsy is being performed, at which time the needle trajectory is calculated based on the current position and orientation of the elongated device. In some examples, the start of a biopsy is automatically detected based on information from a shape sensor at the distal end of the elongated device, which generates a signal when the needle leaves the distal end of the elongated device. In some examples, the needle may be robotically controlled and / or may include a position sensor, in which case the time and depth of needle insertion may be directly captured and stored. In some examples, fluoroscopic imaging may be used to detect the time and position of the needle extending from the elongated device. In some examples, indicator 930 may be numbered, time-stamped, and / or supplemented with clinical information about previous attempts.

[0096] In some examples, a range of motion limit indicator 940 may be displayed in window 900 to indicate that the elongated device cannot or should not be further steered in a particular direction. In some examples, the range of motion limit indicator 940 may appear when the pressure sensor detects excessive pressure on the elongated device from a given direction, such as may occur when the elongated device is steered into the wall of an anatomical passage and / or when the range of motion limit of the steerable portion of the elongated device is approached and / or exceeded. In some examples, the range of motion limit may be determined based on a physical constraint (e.g., a steering mechanism cannot bend the elongated device further) and / or a practical constraint (e.g., a minimum bend radius beyond which a medical device cannot be safely inserted through the elongated device). In this way, the range of motion limit indicator 940 warns the operator of a violation of one or more built-in limits and / or external limits that prohibit the elongated device from further steering in one or more directions. In response to observing the range of motion limit indicator 940, the operator may steer the elongated device in the opposite direction until the range of motion limit indicator 940 disappears and / or otherwise changes state to indicate that the condition has been resolved. According to some embodiments, the range of motion limit indicator 940 may be displayed as an arc. Various properties of the arc (e.g., radius, angle, length, width, color, texture, shading, transparency, and / or the like) may vary to indicate the severity of the limit violation (e.g., yellow indicates that the range of motion limit is being approached, red indicates that the limit has been reached and / or exceeded).

[0097] Fig.10A window 1000 for displaying a virtual global image in alignment mode is shown according to some examples. Figure 1 - In some embodiments of FIG. 4 , window 1000 may correspond to virtual global view windows 470 - 480 . Figure 6 10, window 1000 displays anatomical features 1010 from a global perspective and an elongated device 1020 with a field of view indicator 1025. Likewise, orientation indicator 1015 and target position 1030 generally correspond to orientation indicator 615 and target position 1030, respectively. In some examples, a centerline 1027 from the distal end of elongated device 1020 is displayed to help an operator visualize the insertion of a medical instrument (e.g., a biopsy needle) into target position 1030.

[0098] In some examples, window 1000 can provide a simplified view of anatomical features 1010 relative to anatomical features 610 of window 600. For example, anatomical features 1010 do not depict anatomical features with a high degree of detail and / or realism, but rather can use basic two-dimensional and / or three-dimensional shapes (e.g., rectangles, cubes, cylinders, spheres) to depict anatomical pathways. As with the absence of anatomical features in window 900, the use of basic shapes to depict anatomical features 1010 has the advantage of reducing the computational complexity involved in rendering window 1000, which can reduce lag (log) and thereby improve the ability to make fine alignment adjustments. Due to the fact that elongated device 1020 is within close range of target position 1030, additional simplifications may be possible and / or desirable in alignment mode, such as the absence of a virtual roadmap (e.g., virtual roadmap 635).

[0099] In some examples, a medical device may be inserted into the elongated device 1020 in an aligned mode. Therefore, it may be desirable to know whether the medical device to be inserted can safely pass through the elongated device 1020. For example, it may be desirable to know whether any portion of the elongated device 1020 is over-bent and / or otherwise damaged (e.g., chemically, thermally, mechanically, and / or biologically damaged). To address this challenge, a color scheme 1040 of the elongated device 1020 indicates the bend radius (and / or temperature, strain, and / or similar parameters) of the elongated device 1020 at different locations along its length. The color scheme 1040 may be used to display the measured bend radius by changing the color, texture, pattern, transparency, shading, and / or another visual characteristic of the elongated device 1020 depending on the location. Using the color scheme 1040, different colors and / or shading may be assigned to a range of bend radius values ​​(e.g., green may be assigned to a range that is considered straight, and red may be assigned to a range that is considered curved, while yellow may be assigned to an intermediate range). As Fig.10As depicted, a color scheme is employed in which darker portions of the elongated device 1020 correspond to smaller bend radii. Such a color scheme may alert an operator to possible portions of the elongated device 1020 that are excessively bent. For example, when the degree of bend exceeds a threshold, an area of ​​the elongated device 1020 may turn red. According to some embodiments, the threshold may correspond to a certain bend radius at which a device, such as a needle, can no longer freely pass through the elongated device 1020. In some examples, the threshold may correspond to a minimum bend radius of the elongated device 1020, such as a radius at which the elongated device 1020 becomes susceptible to forming a kink. In some embodiments, there may be multiple thresholds, each of which triggers a different change in the color scheme 1040, such as a transition to a darker shade of red to indicate that a more extreme threshold has been exceeded. Although the transition between colors is depicted as abrupt, it should be understood that in some examples the color scheme 1040 may gradually transition between colors so that properties of the color scheme 1040 (e.g., hue, brightness, and / or the like) are calculated as a function of the bend radius. In some examples, the color scheme 1040 can be applied along the entire length of the elongated device 1020. In some examples, the color scheme 1040 can be limited to the distal portion of the elongated device 1020 because the proximal portion of the elongated device 1020 may not be as susceptible to becoming overly bent as the distal portion.

[0100] In general, any of windows 500-900 may include features from any other window as appropriate. For example, window 500 may include an orientation indicator, such as orientation indicator 615 of window 600. In some examples, elongated device 620 may include a color scheme, such as color scheme 1040 of elongated device 1020. In some examples, window 1000 may include a virtual roadmap, such as virtual roadmap 635 of window 600. In some examples, window 1000 may include an indicator of the orientation of a previous procedure performed at a target orientation 1030, such as indicator 930 of window 900. In some examples, detailed anatomical features 610 of window 600 may be simplified to be similar to anatomical features 1010 in order to improve rendering efficiency. Conversely, simplified anatomical features 1010 of window 1000 may be enhanced to be similar to anatomical features 610 in order to improve the aesthetics of window 1000.

[0101] In some examples, it may be desirable to use a shape sensor to measure the position and / or shape of the elongated device when the elongated device is "parked" (i.e., when the operator is not inserting, retracting, and / or articulating the elongated device). For example, the elongated device may be displaced when a medical device (e.g., a needle) is inserted into and / or removed from the elongated device. To depict this movement in the multimodal graphical user interface 400, the original position of the elongated device (before the displacement) may be depicted as a transparent and / or semi-transparent "ghost" view in one or more of the virtual global view windows 430, 470, and / or 480. In some examples, it may be desirable to warn the operator when the patient's body moves. For example, sensors in a patient pad and / or the like may be used to detect patient movement. When patient movement is detected, one or more attributes of the multimodal graphical user interface 400 may change, such as the background and / or border color changing to red. In some examples, image data (such as representations of anatomical features 610 and / or 1010) may change color, shading, texture, and / or transparency. In some examples, these changes may be triggered when patient movement exceeds one or more predetermined thresholds.

[0102] Fig.11 1 is a flow chart illustrating a method 1100 for displaying a multimodal graphical user interface (e.g., multimodal graphical user interface 400) during an image-guided surgical procedure according to some embodiments. Method 1100 is illustrated as a set of operations or processes 1110-1140. Not all of the illustrated processes 1110-1140 may be performed in all embodiments of method 1100. Additionally, processes 1110-1140 may be included before, after, between, or as part of them. Fig.11 In some embodiments, one or more of the processes 1110-1140 of method 1100 may be implemented at least in part in the form of executable code stored on a non-transitory tangible computer-readable medium, which, when executed by one or more processors (e.g., a processor of control system 112), may cause the one or more processors to perform one or more of the processes 1110-1140.

[0103] At process 1110, a multimodal graphical user interface is displayed in a fly-through mode. Figure 1-10 In some embodiments, the through mode may include a live camera view window, a virtual endoscope view window, a virtual global view window, a settings window, and / or an actuation information window. In some embodiments, the through mode is used when driving an elongated device through a patient's body, where substantial motion associated with insertion, retraction, and / or articulation of the elongated device is expected. In some examples, a camera probe is inserted into the elongated device and positioned at a distal end of the elongated device in the through mode to provide live image data.

[0104] At process 1120, a first trigger event is detected, which indicates that the multimodal graphical user interface will switch from the traversal mode to the alignment mode. The first trigger event may include a user-initiated trigger and / or an automatically detected trigger condition. The manual trigger may include a physical input from an operator, for example, by clicking a button, toggling a switch, using a touch pad, and / or any other suitable manual input. In some examples, the automatically detected trigger condition may include removing the camera probe from the elongated device. In some examples, the removal of the camera probe from the elongated device may be detected based on a change in the shape of the elongated device as determined using a shape sensor (such as shape sensor 222). In some examples, the removal of the camera probe from the elongated device may be detected based on a change in image data provided by the camera probe as determined using an image processor. In some examples, changes in image data may be provided by real imaging techniques (e.g., computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube x-ray imaging). Changes in image data may include new images acquired using imaging techniques at different angles relative to the target or at different times in the procedure. The trigger condition may include the attachment or disconnection of the imaging technology to the system. In some examples, the trigger condition may include the insertion of a medical device into the elongated device. Similar to the removal of the camera probe from the elongated device, the insertion of the medical device into the elongated device may be determined based on a change in the shape of the elongated device. In some examples, the trigger condition may include detecting that the distal end of the elongated device is within a predetermined distance (e.g., two centimeters or three centimeters) of the target location. In some examples, the trigger condition may include determining that there is a line of sight between the distal end of the elongated device and the target location (i.e., there are no obstacles and / or bends in the anatomical pathway). In some examples, the trigger condition may include detecting that there are no remaining branches in the anatomical pathway that separates the distal end of the elongated device from the target location, which means that there is no remaining risk of "losing" by navigating to the wrong branch. In some examples, the trigger condition may include reaching the end of the predetermined route to the target location. In some examples, the trigger condition may include detecting that the operator has stopped inserting the elongated device (i.e., when the elongated device has been "parked"). In some examples, the trigger condition may include a change. In some examples, the operator may choose to override the automatic trigger by providing a manual trigger. In some examples, any suitable combination of one or more trigger conditions can be used to automatically switch between the traversal mode and the alignment mode. For example, the trigger condition can include detecting that both of the following conditions are met: (1) the elongated device is within a predetermined distance of the target orientation, and (2) the operator has stopped inserting the elongated device. When the first trigger event is detected, the method 1100 can proceed to process 1130 to display the multimodal graphical user interface in the alignment mode.

[0105] At process 1130, the multimodal graphical user interface is displayed in an aligned mode. Figure 1-10 In some embodiments, the alignment mode may include a target guidance window, one or more virtual global view windows, a settings window, and / or an actuation information window. In some embodiments, the alignment mode is used when optimizing the position and / or orientation of the elongated device relative to a target location in preparation for performing a medical procedure (e.g., a biopsy, laser ablation, imaging, and / or the like). In some examples, the camera probe is not inserted into the elongated device during the alignment mode, so that the elongated device is steered "blind".

[0106] At process 1140, a second triggering event is detected, which indicates that the multimodal graphical user interface is to be switched from the alignment mode to the traversal mode. In some examples, the second triggering event can be substantially similar to the first triggering event but in reverse order. For example, rather than detecting the removal of the camera probe from the elongated device, the reinsertion of the camera probe into the elongated device can be used as an example of a second triggering event. When the second triggering event is detected, method 1100 can proceed to process 1110 to display the multimodal graphical user interface in the traversal mode.

[0107] Fig.12 1 is a screenshot 1200 of a multimodal graphical user interface (e.g., multimodal graphical user interface 400) in a walkthrough mode according to some embodiments. Similar to multimodal graphical user interface 400 in walkthrough mode, screenshot 1200 depicts five simultaneously viewable frames or windows 1210-1250. Windows 1210-1250 generally correspond to the following: Figure 4a Depicted windows 410-450. In some examples, window 1220 for displaying a virtual endoscopic view includes a virtual roadmap 1225 as a green line leading to a target location. In some examples, window 1230 for displaying a virtual global view includes a distance indicator 1231 to indicate the distance between the distal end of the elongated device and the target location.

[0108] Fig.13 1 is a screenshot 1300 of a multimodal graphical user interface (e.g., multimodal graphical user interface 400) in alignment mode according to some embodiments. Similar to multimodal graphical user interface 400 in alignment mode, screenshot 1300 depicts six simultaneously viewable frames or windows 1310 and 1340-1380. Windows 1310 and 1340-1380 generally correspond to the following: Figure 4b Windows 410 and 440-480 are depicted.

[0109] Some examples of control units (such as control unit 130) may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 140), may cause the one or more processors to provide graphical user interface 400 or perform the processes of method 1100. Some common forms of machine-readable media that may provide graphical user interface 400 or include the processes of method 1100 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium suitable for being read by a processor or computer.

[0110] Additional Examples

[0111] A1. A method for displaying guidance information during a medical procedure using a graphical user interface, the method comprising: displaying first image data from a perspective corresponding to a distal end of a slender device, wherein the first image data includes anatomical features of a patient's anatomical structure; displaying a virtual roadmap to a target location within the patient's anatomical structure, wherein the virtual roadmap indicates a predetermined route to the target location; and changing one or more of the color, shading, texture, and transparency of the virtual roadmap based on a deviation between the distal end of the slender device and the predetermined route.

[0112] A2. A method according to example A1, wherein the virtual roadmap is displayed as lines superimposed on anatomical features included in the first image data.

[0113] A3. A method according to example A1 or A2, wherein the virtual roadmap changes one or more of color, shading, texture, and transparency based on one or more attributes of the anatomical feature, and the one or more attributes are selected from the group consisting of: diameter of an anatomical pathway, curvature of an anatomical pathway, contraction of an anatomical pathway, and branching of an anatomical pathway.

[0114] A4. A method according to any of Examples A1-A3, wherein the virtual roadmap changes one or more of color, shading, texture, and transparency based on the distance between the distal end of the slender device and the target location.

[0115] A5. A method according to any of Examples A1-A4, wherein displaying the first image data also includes displaying a roadblock indicator.

[0116] A6. The method of example A5, wherein the anatomical feature comprises an undesirable anatomical feature through which the elongated device should not be guided, and the roadblock indicator corresponds to the undesirable anatomical feature.

[0117] A7. A method according to example A5 or A6, wherein the roadblock indicator is determined based on one or more attributes of the anatomical feature, the one or more attributes including a diameter of the anatomical feature or a curvature of the anatomical feature.

[0118] A8. A method according to any of Examples A1-A7, wherein the anatomical feature changes one or more of color, shading, texture, and transparency based on the distance between the distal end of the slender device and the target location.

[0119] A9. The method according to any one of Examples A1-A8, further comprising displaying second image data from a perspective corresponding to the distal end of the slender device, wherein: the second image data includes a target indicator corresponding to the target orientation and an alignment indicator corresponding to the expected orientation of the medical procedure; and the target indicator changes one or more of color, shadow, texture, and transparency based on the alignment between the expected orientation of the medical procedure and the target orientation.

[0120] A10. A method according to example A9, wherein the target indicator is modeled based on pre-operative voxel data.

[0121] A11. A method according to example A9 or A10, wherein the target indicator changes one or more of color, shade, texture, and transparency based on the distance between the distal end of the slender device and the target location.

[0122] A12. A method according to any of Examples A9-A11, wherein the target indicator changes one or more of color, shading, texture, and transparency based on a deviation between the alignment indicator and the target indicator.

[0123] A13. A method according to any of Examples A9-A12, wherein the alignment indicator changes one or more of color, shading, texture, and transparency based on a deviation between the alignment indicator and the target indicator.

[0124] A14. A method for displaying guidance information during a medical procedure using a graphical user interface, the method comprising: in an alignment mode of the graphical user interface and during alignment of the distal end of a slender device to perform a medical procedure at a target orientation, displaying first image data from a perspective corresponding to the distal end of the slender device, wherein the first image data includes a target indicator corresponding to the target orientation and an alignment indicator corresponding to an expected orientation of the medical procedure; delivering an instrument to the target orientation, wherein delivering the instrument to the target orientation includes storing a plurality of instrument trajectories; and displaying a plurality of history indicators corresponding to the plurality of instrument trajectories.

[0125] A15. A method according to example A14, wherein the instrument is a needle.

[0126] A16. The method according to example A14 or A15 also includes: in a traversal mode of the graphical user interface and during the passage of the slender device through the anatomical pathway, displaying second image data from a perspective corresponding to the distal end of the slender device, wherein the second image data includes a virtual roadmap and a historical path of the slender device.

[0127] A17. The method according to example A16 also includes hiding the historical path based on manual input.

[0128] A18. The method of any one of Examples A14-A17, further comprising hiding the history indicator based on manual input.

[0129] A19. A method according to any of Examples A14-A18, wherein the medical procedure is one or more of a biopsy and an ablation.

[0130] A20. The method of any of Examples A14-A19 further comprising displaying a force indicator, wherein the force indicator comprises a representation of an insertion force applied by a motor or actuator used to advance the elongated device.

[0131] A21. A method according to example A20, wherein the force indicator is displayed as a scrolling bar graph.

[0132] Although illustrative embodiments have been shown and described, extensive modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and the claims should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: displaying first image data from a perspective corresponding to a distal end of the elongated device, wherein the first image data includes anatomical features of an anatomical structure of the patient; displaying a virtual roadmap to a target location within the patient's anatomy, wherein the virtual roadmap indicates a predetermined route to the target location; as well as One or more of a color, a shading, a texture, or a transparency of the virtual roadmap is changed based on a deviation between the distal end of the elongated device and the predetermined route. 2 . The non-transitory computer-readable medium of claim 1 , wherein the virtual roadmap is displayed as lines superimposed on the anatomical features contained in the first image data.

3. The non-transitory computer-readable medium of claim 1 , wherein the virtual roadmap changes one or more of color, shading, texture, or transparency based on one or more attributes of the anatomical feature, the one or more attributes comprising one or more of: The diameter of the anatomical passage, The curvature of the anatomical pathway, constriction of the anatomical pathway, or Branches of the anatomical pathway.

4. The non-transitory computer-readable medium of claim 1, wherein the virtual roadmap changes one or more of color, shading, texture, or transparency based on a distance between the distal end of the elongated device and the target location. The non-transitory computer-readable medium of claim 1 , wherein the first image data further comprises a roadblock indicator.

6. The non-transitory computer-readable medium of claim 5, wherein the anatomical feature comprises an undesirable anatomical feature in which the elongated device should not be guided, and the roadblock indicator corresponds to the undesirable anatomical feature.

7. The non-transitory computer-readable medium of claim 6, wherein the roadblock indicator is determined based on one or more attributes of the undesirable anatomical feature, the one or more attributes comprising a diameter of the undesirable anatomical feature or a curvature radius of the undesirable anatomical feature.

8. The non-transitory computer readable medium of any one of claims 1-7, wherein the anatomical feature changes one or more of color, shading, texture, or transparency based on a distance between the distal end of the elongated device and the target location.

9. The non-transitory computer readable medium of any one of claims 1-7, wherein at least a portion of the anatomical feature is rendered transparent based on a deviation of the elongated device from the predetermined path.

10. The non-transitory computer-readable medium of any one of claims 1-7, storing instructions that, when executed by the one or more processors, cause the one or more processors to: Second image data is displayed from a perspective corresponding to the distal end of the slender device, wherein the second image data includes a target indicator corresponding to the target orientation and an alignment indicator corresponding to the expected orientation of the medical procedure, and wherein the target indicator changes one or more of color, shading, texture, or transparency based on the alignment between the expected orientation of the medical procedure and the target orientation.

Images