Systems and methods for registering an elongate device to a three-dimensional image in an image-guided procedure

Abstract

Methods for using registered real-time images and prior anatomical images during image-guided procedures are provided herein. An exemplary method includes obtaining a three-dimensional image of a patient's anatomy and a portion of a medical instrument disposed therein. The three-dimensional image includes image information characterizing a shape of the portion of the medical instrument. A processing device segments the portion of the medical instrument from the three-dimensional image. Shape data is obtained from the portion of the medical instrument while the portion of the medical instrument is located within the patient's anatomy, and the processing device registers the segmented shape of the portion of the medical instrument with the shape data from the portion of the medical instrument.

Description

[0001] This application is a divisional application of Chinese Patent Application 201880005172.4 (PCT / US2018 / 012969), filed on January 9, 2018, entitled "System and method for registering an elongated device to a three-dimensional image in an image-guided procedure".

[0002] Cross-reference to related applications

[0003] This patent application claims priority and the benefit of the filing date of U.S. Provisional Patent Application US62 / 444,018, filed January 9, 2017, entitled "Systems and Methods for Registering Elongate Devices to Three-Dimensional Images in Image-Guided Procedures", the entire contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure relates to systems and methods for performing image-guided procedures, and more specifically to systems and methods for using registered real-time images and previously timed anatomical images during image-guided procedures. Background Technology

[0005] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, physicians can insert minimally invasive medical instruments (including surgical instruments, diagnostic instruments, therapeutic instruments, or biopsy instruments) to reach the target tissue location. To aid in reaching the target tissue location, the orientation and movement of the medical instruments can be correlated with preoperative images of the patient's anatomy. By correlating image-guided instruments with images, the images can be used to help the instruments navigate through pathways created naturally or surgically within the anatomical system, such as the lungs, colon, intestines, kidneys, heart, circulatory system, or the like. While preoperative images are helpful, they are typically static representations taken before the procedure.

[0006] Therefore, it is advantageous to provide registration of the instrument with the imaging modality to provide enhanced navigation information for performing image-guided procedures, wherein the imaging modality acquires images of the patient's anatomy during the procedure. Summary of the Invention

[0007] The claims accompanying the specification summarize embodiments of the invention.

[0008] In one embodiment, a method is performed by a computing system. The method includes obtaining a three-dimensional image of a patient anatomy and a portion of a medical device disposed therein. The three-dimensional image includes image information characterizing the shape of the portion of the medical device. A processing device segments the portion of the medical device from the three-dimensional image. When the portion of the medical device is located within the patient anatomy, shape data is obtained from the portion, and the processing device registers the segmented shape of the portion of the medical device with the shape data from the portion of the medical device. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more non-transitory computer storage media, each computer program configured to perform the actions of the method.

[0009] In another embodiment, the medical system includes a medical device comprising a shape sensor, a tracking subsystem configured to receive shape data from the shape sensor, and a processor coupled to the medical device and the tracking subsystem. The processor is configured to receive image data of a patient anatomy having the medical device disposed therein, such that the image data is in an image reference frame. The processor is further configured to: segment a portion of the image data corresponding to the medical device; obtain shape data from the medical device in the device reference frame; and register the device reference frame to the image reference frame by comparing the shape data with the portion of the image data corresponding to the medical device.

[0010] In another embodiment, a method is performed by a computing system. The method includes acquiring imaging data of a patient's anatomy when a medical device is placed within the patient's anatomy. The imaging data includes multiple images and is in an image reference frame. When the medical device is placed within the patient's anatomy, shape data is acquired from the medical device. The shape data includes multiple subsets and is in an device reference frame. Subsets of the multiple subsets of the shape data are matched to images in the multiple images, and the device reference frame is registered to the image reference frame based on comparing the matched subsets of the multiple subsets of the shape data to images in the multiple images. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more non-transitory computer storage media, each computer program configured to perform the actions of the method.

[0011] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the disclosure rather than to limit its scope. In this regard, additional aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0012] Figure 1 This is a simplified schematic diagram of a remotely operated medical system according to some embodiments.

[0013] Figure 2A A medical device system utilizing various aspects of this disclosure is shown.

[0014] Figure 2B An embodiment of the present disclosure is shown with extended medical tools. Figure 2A The remote end of the medical device system.

[0015] Figure 3A and Figure 3B This is a simplified schematic diagram of a side view of a medical device mounted on an insert assembly, according to some embodiments, in patient coordinate space.

[0016] Figure 4A , Figure 4B , Figure 4C and Figure 4D This shows the process during insertion into a human lung. Figure 2A , Figure 3A and Figure 3B The remote end of the medical robot system.

[0017] Figure 5 A flowchart of an image-guided medical procedure according to an embodiment of the present disclosure is shown.

[0018] Figure 6A Exemplary filtered three-dimensional images according to various aspects of this disclosure are shown, including image information associated with bronchial access and medical devices.

[0019] Figure 6B An embodiment according to this disclosure is shown. Figure 6A A segmentation model for medical devices.

[0020] Figure 7A This is another view of the surgical coordinate space according to various aspects of this disclosure, which includes, for example... Figure 3A and Figure 3B The medical devices and 3D imaging systems seen in the document.

[0021] Figure 7B It shows Figure 7A A detailed view of the distal portion of the medical device shown.

[0022] Figure 7C Uses of various aspects according to this disclosure are shown. Figure 7A and Figure 7B The shape data collected by the catheter medical device shown is shown.

[0023] Figure 8 An internal perspective view of a three-dimensional image registered with a medical device in a common reference frame, according to various aspects of this disclosure, is shown.

[0024] Figure 9A flowchart is shown of a method for performing image-guided surgery according to some embodiments of the present disclosure.

[0025] Figure 10A and Figure 10B This is a side view of a medical device according to some embodiments of the present disclosure.

[0026] Figure 11 A composite image showing a representation of an updated preoperative model, according to an example of this disclosure, is shown.

[0027] When with attachment Figure 1 As you read this, the embodiments of this disclosure will be best understood from the following detailed description. It should be appreciated that the same reference numerals are used to identify the same elements illustrated in one or more of the figures, which are shown for the purpose of illustrating embodiments of this disclosure and are not intended to limit the scope of this disclosure. Detailed Implementation

[0028] In the following description, specific details of some embodiments consistent with this disclosure are set forth. Numerous specific details are set forth in order to provide a full understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Other elements that are not specifically described herein but are within the scope and spirit of this disclosure can be implemented by those skilled in the art. Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0029] In some instances, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure multiple aspects of the embodiments.

[0030] This disclosure describes various instruments and instrument parts from the perspective of their states in three-dimensional space. As used herein, the term "position" refers to the orientation of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "attitude" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of the object or some parts of the object in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of attitudes, positions, or orientations measured along the object.

[0031] Figure 1This is a simplified schematic diagram of a remotely operated medical system 100 according to some embodiments. In some embodiments, the remotely operated medical system 100 may be adapted for use in, for example, surgical procedures, diagnostic procedures, treatment procedures, or biopsy procedures. Figure 1 As shown, the medical system 100 generally includes a manipulator assembly 102 (which may be a remotely operated manipulator assembly or a partially remotely operated manipulator assembly) for operating the medical device 104 to perform various procedures on the patient P. The manipulator assembly 102 is mounted on or near the operating console T. A master control assembly 106 allows an operator O (e.g., a surgeon, clinician, or physician) to view the intervention site and control the manipulator assembly 102.

[0032] The master control assembly 106 may be located at a physician control console, which is typically situated in the same room as the operating table T, such as on one side of the surgical table on which the patient P is located. However, it should be understood that the operator O may be located in a different room or in a completely different building from the patient P. The master control assembly 106 generally includes one or more control devices for controlling the manipulator assembly 102. The control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual operation controllers, voice recognition devices, body motion or presence sensors and / or similar devices. To provide the operator O with a strong sense of direct control of the instrument 104, the control devices may be provided with the same degrees of freedom as the associated medical instrument 104. In this way, the control devices provide the operator O with a telepresence or perception of the control devices and the medical instrument 104 as one entity.

[0033] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104, and still provide remote presentation to the operator O. In some embodiments, the control device may optionally be a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating instruments (e.g., for closing a gripping clamp, applying a potential to an electrode, delivering medication, and / or the like).

[0034] Manipulator assembly 102 supports medical device 104 and may include the kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, often referred to as a set-up structure) and a remotely operated manipulator. Manipulator assembly 102 may optionally include a plurality of actuators or motors that drive input devices on medical device 104 in response to commands from a control system (e.g., control system 112). Actuators may optionally include a drive system that, when coupled to medical device 104, can advance medical device 104 into natural or surgically created anatomical openings. Other drive systems may move the distal end of medical device 104 with multiple degrees of freedom, including three linear degrees of freedom (e.g., linear motion along the X, Y, Z Cartesian coordinate axes) and three rotational degrees of freedom (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Furthermore, the actuator can be used to actuate the articulated end effector of medical device 104 to grip tissue in the jaw of a biopsy device and / or similar device. Actuator position sensors (such as decoders, encoders, potentiometers, and other mechanisms) can provide the medical system 100 with sensor data describing the rotation and orientation of the motor shaft. This position sensor data can be used to determine the motion of the object manipulated by the actuator.

[0035] The remotely operated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instruments of the manipulator assembly 102. These subsystems may include: a position / positioning sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, velocity, rate, attitude, and / or shape along the distal end and / or one or more segments that may constitute the flexible body of the medical device 104; and / or a visualization system for acquiring images from the distal end of the medical device 104.

[0036] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical device 104 generated by a subsystem of the sensor system 108. The display system 110 and the main control assembly 106 can be oriented such that an operator O can control the medical device 104 and the main control assembly 106 by means of telepresence / presence perception.

[0037] In some embodiments, medical device 104 may have a visualization system (discussed in more detail below) that may include an endoscope assembly that records concurrent or real-time images of a surgical site and provides the images to operator O via one or more displays of medical system 100, such as one or more displays of display system 110. The concurrent images may be, for example, two-dimensional or three-dimensional images acquired by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used together with medical device 104 to image the surgical site. In some examples, the endoscope may include one or more mechanisms for cleaning one or more lenses of the endoscope when one or more lenses are partially and / or completely blocked by fluids and / or other materials encountered by the endoscope. In some examples, the one or more cleaning mechanisms may optionally include an air and / or other gas delivery system that can be used to fire a stream of air and / or other gas to clean one or more lenses. Examples of one or more cleaning mechanisms are discussed in more detail in International Publication WO / 2016 / 025465 (filed August 11, 2016) (publication “Systems and Methods for Cleaning an Endoscopic Instrument”), which is incorporated herein by reference in its entirety. A visualization system can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is executed by one or more computer processors (which may include the processor of control system 112).

[0038] Display system 110 can also display images of surgical sites and medical instruments acquired through a visualization system. In some examples, remote-operated medical system 100 can configure controls for medical instrument 104 and main control assembly 106 such that the relative positions of the medical instruments resemble the relative positions of the operator O's eyes and hands. In this way, operator O can operate medical instrument 104 and hand controls as if observing the workspace in a substantially realistic presence. In terms of realistic presence, this means that the image presentation is a realistic perspective image simulating the viewpoint of a physician physically manipulating medical instrument 104.

[0039] In some examples, the display system 110 can use image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or similar technologies to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data can be presented as two-dimensional (2D), three-dimensional (3D), or four-dimensional (2D) images (including, for example, time-based or velocity-based information), and / or as images from models created from preoperative or intraoperative image datasets.

[0040] In some embodiments, typically for the purpose of image-guided medical procedures, the display system 110 may display a virtual navigation image in which the actual orientation of the medical device 104 is registered with preoperative or real-time images / models (i.e., dynamic reference). This presents a virtual image of the internal surgical site to the operator O from the perspective of the medical device 104. In some examples, the perspective may be from the tip of the medical device 104. Images of the tip of the medical device 104 and / or other graphic or alphanumeric indicators may be overlaid on the virtual image to aid the operator O in controlling the medical device 104. In some examples, the medical device 104 may be invisible in the virtual image.

[0041] In some embodiments, display system 110 may display a virtual navigation image in which the actual orientation of medical device 104 is registered with a preoperative or live image to present a virtual image of medical device 104 within the surgical site to operator O from an external perspective. An image of a portion of medical device 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist operator O in controlling medical device 104. As described herein, a visual representation of data points may be rendered onto display system 110. For example, measured data points, moving data points, registered data points, and other data points described herein may be displayed on display system 110 as a visual representation. Data points may be visually represented on the user interface as multiple points or dots on display system 110 or as a rendering model, such as a grid or line model created based on a set of data points. In some examples, data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed in display system 110 after each processing operation has been performed to change the data points.

[0042] The remote-operated medical system 100 may further include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control assembly 106, the sensor system 108, and the display system 110. The control system 112 also includes some or all of the programmed instructions (e.g., a non-transitory machine-readable medium storing these instructions) for implementing some or all of the methods described according to the aspects disclosed herein, including instructions for providing information to the display system 110. While the control system 112... Figure 1 The simplified schematic is shown as a single block, but the system may include two or more data processing circuits, with some processing optionally performed on or near the manipulator assembly 102, and other processing performed at the master assembly 106 and / or the like. The processor of the control system 112 may execute instructions, including instructions corresponding to the processing disclosed herein and described in more detail below. Any variety of centralized or distributed data processing architectures can be utilized. Similarly, programmable instructions may be implemented as several independent programs or subroutines, or they may be integrated into several other aspects of the remote operating system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0043] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to this feedback, the control system 112 may transmit a signal to the master assembly 106. In some examples, the control system 112 may transmit a signal instructing one or more actuators of the manipulator assembly 102 to move the medical device 104. The medical device 104 may extend to an internal surgical site within the body of the patient P via an opening in the patient P's body. Any suitable conventional and / or specialized actuators may be used. In some examples, one or more actuators may be separate from or integrated with the manipulator assembly 102. In some embodiments, one or more actuators and the manipulator assembly 102 are provided as part of a remotely operated trolley positioned adjacent to the patient P and the operating table T.

[0044] The control system 112 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 104 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of acquired anatomical pathways. The virtual visualization system processes images of surgical sites imaged using imaging techniques such as CT, MRI, fluorescence microscopy, thermography, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques. Software, which can be used in conjunction with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of local or global anatomical organs or regions. The image dataset is associated with the composite representation. The composite representation and the image dataset describe various orientations and shapes of pathways and their connectivity. Images used to generate the composite representation may be recorded preoperatively or intraoperatively during a clinical procedure. In some embodiments, the virtual visualization system may use standard representations (i.e., not patient-specific) or a mixture of standard representations and patient-specific data. The composite representation and any virtual image generated from it can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0045] During the virtual navigation procedure, sensor system 108 can be used to calculate the approximate orientation of medical device 104 relative to the anatomy of patient P. This orientation can be used to generate macroscopic (external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display medical devices and preoperatively recorded surgical images, such as those from virtual visualization systems. For example, U.S. Patent Application US13 / 107,562 (filed May 13, 2011) (publishing “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”) discloses such a system, which is incorporated herein by reference in its entirety. The remote-operated medical system 100 may further include optional operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some embodiments, the remote-operated medical system 100 may include more than one remote-operated manipulator assembly and / or more than one master control assembly. The exact number of remote-controlled manipulator assemblies will depend on the medical procedure, space constraints within the operating room, and other factors. Master control assemblies 106 can be collocated, or they can be positioned in independent locations. Multiple master control assemblies allow more than one operator to control one or more remote-controlled manipulator assemblies in various combinations.

[0046] Figure 2A This is a simplified schematic diagram of a medical device system 200 according to some embodiments. In some embodiments, the medical device system 200 may be used as a medical device 104 in an image-guided medical procedure performed using a remotely operated medical system 100. In some examples, the medical device system 200 may be used in non-remotely operated exploratory procedures or in procedures involving conventionally manually operated medical devices, such as endoscopes. Optionally, the medical device system 200 may be used to collect (i.e., measure) a set of data points corresponding to orientation within an anatomical pathway of a patient (such as patient P).

[0047] Medical device system 200 includes an elongated device 202 coupled to a drive unit 204. The elongated device 202 includes a flexible body 216 having a proximal end 217 and a distal end 218. In some embodiments, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters. Examples of the elongated device 202 include endoscopes, bronchoscopes, catheters, and other medical devices.

[0048] The medical device system 200 further includes a tracking system 230 for determining, using one or more sensors and / or imaging devices, the position, orientation, velocity, rate, attitude, and / or shape of the catheter tip at the distal end 218 of the flexible body 216 and / or one or more segments 224 along the flexible body 216, as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into multiple segments 224. The tracking system 230 can optionally be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include... Figure 1 The processor of the control system 112.

[0049] The tracking system 230 may optionally use a shape sensor 222 to track the distal end 218 and / or one or more segments 224. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or externally mounted). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, this size may be larger or smaller. The optical fiber of the shape sensor 222 forms an optical fiber bending sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one or more dimensions of the structure. Various systems and methods for monitoring the shape and relative orientation of optical fibers in three dimensions are described in the following documents: U.S. Patent Application US11 / 180,389 (filed July 13, 2005) (publishing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application US12 / 047,056 (filed July 16, 2004) (publishing "Fiber-optic shape and relative positionsensing"); and U.S. Patent US6,389,187 (filed June 17, 1998) (publishing "Optical Fiber BendSensor"), all of which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the conduit may be determined using other techniques. For example, the history of the distal attitude of the flexible body 216 can be used to reconstruct the shape of the flexible body 216 over time intervals. In some embodiments, the tracking system 230 may optionally and / or additionally use the position sensor system 220 to track the distal end 218. The position sensor system 220 may be a component of an EM sensor system with position sensors, the EM sensor system including one or more conductive coils capable of withstanding an externally generated electromagnetic field. Each coil of the sensor system 220 then generates an induced electrical signal having characteristics dependent on the position and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the position sensor system 220 may be configured and positioned to measure six degrees of freedom (e.g., three position coordinates X, Y, Z and three orientation angles for pitch, yaw, and roll of an indicator base point) or five degrees of freedom (e.g., three position coordinates X, Y, Z and two orientation angles for pitch and yaw of an indicator base point).A further description of the position sensor system is provided in U.S. Patent 6,380,732 (filed August 11, 1999) (published "Six-Degree of FreedomTracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0050] In some embodiments, the tracking system 230 may alternatively and / or additionally rely on historical posture, position, or orientation data stored for known points of the instrument system along cycles of alternating motion (such as breathing). This stored data can be used to develop shape information about the flexible body 216. In some examples, a series of position sensors (not shown) (such as electromagnetic (EM) sensors similar to those in position sensor 220) may be positioned along the flexible body 216 and then used for shape sensing. In some examples, a history of data acquired during a procedure from one or more of these sensors may be used to represent the shape of the elongated device 202, particularly if the anatomical pathway is substantially static.

[0051] The flexible body 216 includes a channel 221 whose size and shape are designed to receive a medical instrument 226. Figure 2BThis is a simplified schematic diagram of a flexible body 216 with an extended medical tool 226 according to some embodiments. In some embodiments, the medical tool 226 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical tool 226 can be deployed through channels 221 of the flexible body 216 and used at a target location within an anatomy. The medical tool 226 may include, for example, an image acquisition probe, a biopsy instrument, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical tool may include an end effector with a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end effectors may include, for example, forceps, graspers, scissors, clamp applicators, and / or the like. Other end effectors may further include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, and / or the like. In various embodiments, the medical tool 226 is a biopsy instrument that can be used to remove sample tissue or cell samples from a target anatomical location. The medical tool 226 can be used in conjunction with an image acquisition probe also within the flexible body 216. In various embodiments, the medical tool 226 may be an image acquisition probe comprising a distal portion of a stereo camera or single-field-of-view camera at or near the distal end 218 of the flexible body 216 for acquiring images (including video images), which are processed by the visualization system 231 for display and / or provided to the tracking system 230 to support tracking of the distal end 218 and / or one or more segments 224. The image acquisition probe may include a cable coupled to the camera for transmitting the acquired image data. In some examples, the image acquisition probe may be a bundle of optical fibers, such as a fiber optic endoscope, coupled to the visualization system 231. The image acquisition probe may be monospectral or multispectral, for example, acquiring image data in one or more of the visible, infrared, and / or ultraviolet spectra. Alternatively, the medical tool 226 itself may be an image acquisition probe. The medical tool 226 may be advanced from an opening in the channel 221 to perform a procedure and then retracted into the channel when the procedure is complete. The medical instrument 226 can be removed from the proximal end 217 of the flexible body 216 or from another optional instrument port (not shown) along the flexible body 216.

[0052] Medical instrument 226 may additionally accommodate a cable, linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of medical instrument 226. Steering instruments are described in detail in U.S. Patent 7,316,681 (filed October 4, 2005) (published "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application 12 / 286,644 (filed September 30, 2008) (published "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0053] The flexible body 216 may also be accommodated between the drive unit 204 and the distal end 218 to controllably bend the cable, linkage, or other steering control (not shown), as illustrated, for example, by the dashed line depiction 219 of the distal end 218. In some examples, at least four cables are used to provide independent control of the pitch "up and down" steering of the distal end 218 and the yaw "left and right" steering of the distal end 218. Steering catheters are described in detail in U.S. Patent Application US13 / 274,208 (filed October 14, 2011) (published "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. In embodiments where the medical device system 200 is actuated by a remotely operated assembly, the drive unit 204 may include a drive input device removably coupled to and receiving power from a drive element (such as an actuator) of the remotely operated assembly. In some embodiments, the medical device system 200 may include clamping features, manual actuators, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator control of bending of the distal end 218. In some examples, one or more cavities are defined within the walls of the flexible body 216, through which the medical device can be deployed and used at a target surgical location.

[0054] In some embodiments, the medical device system 200 may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes for use in the examination, diagnosis, biopsy, or treatment of the lungs. The medical device system 200 is also suitable for navigating and treating other tissues via natural or surgically established pathways within any of the various anatomical systems, including the colon, intestine, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and / or the like.

[0055] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or a preoperatively acquired model to provide real-time location information to operator O or another operator. In some examples, the real-time location information can be displayed... Figure 1 The display system 110 is used for control of the medical device system 200. In some examples, Figure 1 The control system 116 can use position information as feedback to position the medical device system 200. Various systems for registering and displaying surgical instruments and surgical images using fiber optic sensors are disclosed in U.S. Patent Application US13 / 107,562, filed May 13, 2011, entitled "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery," which is incorporated herein by reference in its entirety.

[0056] In some examples, the medical device system 200 can Figure 1 The medical system is operated remotely within 100. In some embodiments, Figure 1 The manipulator assembly 102 can be replaced by a direct operator control. In some examples, the direct operator control may include various handles and operator interfaces for handheld operation of the instrument.

[0057] Figure 3A and Figure 3B This is a simplified schematic diagram of a side view of a medical device mounted on an insert assembly, according to some embodiments, in patient coordinate space. Figure 3A and Figure 3BAs shown, the surgical environment 300 has a surgical environment reference frame (Xs, Ys, Zs), in which patient P is positioned on the operating table T. Patient P may be stationary within the surgical environment in the sense of limiting overall patient movement through sedation, restraint, and / or other means. Periodic anatomical movements (including respiratory and cardiac movements of patient P) may continue unless the patient is instructed to hold their breath to temporarily cease respiratory movements. Therefore, in some embodiments, data may be collected at specific phases of respiration and used for labeling and identification. In some embodiments, the phase in which data is collected may be inferred based on physiological information collected from patient P. Within the surgical environment 300, a medical device 304 is coupled to an instrument holder 306. In some embodiments, the medical device 304 may use an EM sensor, a shape sensor, and / or other sensor modalities. The instrument holder 306 is mounted to an insertion stage 308 fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable, but has a known orientation within the surgical environment 300 (e.g., known via a tracking sensor or other tracking device). The instrument holder 306 may be a component of a remotely operated manipulator assembly (e.g., manipulator assembly 102) coupled to the medical device 304 to control insertion motion (i.e., movement along the A-axis or in the Xs direction) and optionally control movement of the distal end 318 of the elongated device 310 in multiple directions (including yaw, pitch, and roll). The instrument holder 306 or the insertion stage 308 may include actuators (not shown), such as servo motors, to control movement of the instrument holder 306 along the insertion stage 308.

[0058] An elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled to and secured relative to an instrument holder 306. In some embodiments, a fiber optic shape sensor 314 is secured at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may be movable together with the instrument body 312, but the orientation of the proximal point 316 may be known (e.g., known via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as the distal end 318 of the elongated device 310). The medical device 304 may be substantially similar to the medical device system 200.

[0059] A three-dimensional imaging system 330 is positioned near the patient P to acquire three-dimensional images of the patient as the elongated device 310 extends within the patient. The three-dimensional imaging system 330 may use imaging techniques such as CT, fluoroscopy, thermography, ultrasound, OCT, thermography, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques to provide real-time or near-real-time images of the patient P. In some embodiments, the imaging system 330 includes a moving rotating imaging element, such as the moving rotating imaging element of a moving C-arm cone-beam CT imaging system for acquiring intraoperative 3D images and / or fluoroscopic 2D images. For example, the system 330 may be the DynaCT imaging system from Siemens Corporation, Washington, D.C., the AiroCT system from Brainlab AG, Munich, Germany, or other suitable imaging systems. Thus, the imaging system 330 can acquire a set of images from one or more angles by rotating the imaging element around the patient P; this set of images may be still images, a series of still images, or video. When the elongated device 310 is located within the patient P, the image can be acquired by the imaging system 330, and the image can include image information characterizing the elongated device 310 in relation to the anatomical structure of the patient P. Although referred to as a "three-dimensional" imaging system, the system 330 can acquire two-dimensional, three-dimensional, or four-dimensional images.

[0060] As the instrument body 312 moves along the insertion axis A on the insertion stage 308, the position measuring device 320 provides information about the position of the instrument body 312. The position measuring device 320 may include a decoder, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of the actuator that controls the movement of the instrument carrier 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear, while in other embodiments, the insertion stage 308 may be arcuate, or have a combination of arcuate and linear segments.

[0061] Figure 3A The device body 312 and device holder 306 are shown in their retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is located at position L0 on axis A. In this position along the insertion stage 308, the orientation component of the proximal point 316 along axis A can be set to zero and / or another reference value to provide a basic reference describing the position of the device holder 306 and therefore the proximal point 316 on the insertion stage 308. With the device body 312 and device holder 306 in this retracted position, the distal end 318 of the elongated device 310 can be positioned precisely inside the access port of the patient P. Also in this position, the position measuring device 320 can be set to zero or another reference value. Figure 3BIn this position, the instrument body 312 and instrument holder 306 have been advanced along the linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advanced position, the proximal point 316 is located at position L1 on axis A.

[0062] Figure 4A , Figure 4B , Figure 4C and Figure 4D The diagram shows... Figure 3A and Figure 3B The slender device 310 through Figure 1 as well as Figure 3A and Figure 3B The patient P's lung 400 is advanced through anatomical pathways 402. These pathways 402 include the tracheal and bronchial airways. As the bracket 306 moves along the insertion stage 308, the elongated device 310 is advanced, and the operator O can steer the distal end 318 of the elongated device 310 to navigate through the anatomical pathways 402. During navigation through the anatomical pathways 402, the elongated device 310 presents a shape that can be measured by a shape sensor 314 extending within the elongated device 310. As described above, when the elongated device 310 is located within the lung 400 (or any other body structure or cavity), when... Figure 3A and Figure 3B When the imaging system 330 acquires images of patient P, the elongated device 310 can appear in these images.

[0063] Figure 5 This is a flowchart illustrating a method 500 for performing image-guided surgery in a surgical environment 300. The methods of this specification, including method 500, are described in... Figure 5 The methods are illustrated as a set of blocks, steps, operations, or procedures. Not all illustrated, enumerated operations can be performed in all embodiments of method 500. Additionally, some additional operations not explicitly shown in the method may be included before, after, between, or as part of the enumeration process. Some embodiments of the methods in this specification include computer-readable instructions corresponding to the procedures of the method stored in non-transitory memory. These instructions may be executed or coordinated by one or more hardware processors, such as the processor of control system 112.

[0064] Therefore, some embodiments of method 500 may begin at operation 502, wherein intraoperative three-dimensional image data of the patient's anatomy is acquired from an imaging system. The imaging system may be used during image-guided medical procedures. Imaging techniques such as CT, MRI, thermography, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging may be used to acquire the image data. In some embodiments, the image data may include two-dimensional images from multiple perspectives, which may be combined to form a pseudo-three-dimensional image. Thus, intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information). In alternative embodiments, the image data may be acquired preoperatively, providing preoperative image data corresponding to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information).

[0065] For example, the three-dimensional image data may include data representing at least a portion of the medical device 304 located within the anatomy of the patient P, such as... Figures 3A-3B and Figures 4A-4D As shown in the diagram. The data may include a representation of the shape of the elongated device 310 of the medical device 304, and a three-dimensional model of the shape may be generated as part of the segmentation process. For example, the three-dimensional image data may be real-time or pseudo-real-time image data obtained from an embodiment of the imaging system 330, such as a cone-beam CT system.

[0066] At operation 504, as part of the registration process, pixels or voxels in the image data corresponding to the medical device obtained in operation 502 can be identified. For this purpose, in some embodiments, computer software, alone or in combination with manual input, is used to convert intraoperative three-dimensional image data into one or more two-dimensional or three-dimensional composite representations or intraoperative models of the structures included in the image data. For example, the image data can be converted into partial or complete anatomical organs or anatomical regions (such as those including…). Figures 4A-4D Intraoperative models (400 mm of lungs and trunk) are used to describe the various orientations and shapes of anatomical pathways and their connectivity.

[0067] Because image data can also include images of medical devices, such as Figures 4A-4DThe elongated device 310 shown, operation 504 may include a segmentation process to delineate a set of pixels or voxels representing a specific structure, such as the elongated device 310 or another medical device or several medical devices. Operation 504 may then construct one or more models of the medical device(s). During the segmentation process, pixels or voxels may be divided into segments or elements or labeled to indicate that they share certain characteristics or computed properties, such as color, density, intensity, and texture. The system performing this operation may then apply a function such as a marching cube function to generate a 3D surface surrounding the voxels. The model may be formed by generating meshes, volumes, or voxel maps. Alternatively or additionally, surface-based models may include or may be used to generate a centerline model comprising a set of interconnected line segments or points extending through the center of the modeled device. In cases where the intraoperative model includes a centerline model (which comprises a set of interconnected line segments), these line segments may be converted into a cloud or set of points. By converting line segments, you can manually or automatically select the desired number of points corresponding to the interconnecting line segments.

[0068] In some embodiments, the segmentation process may utilize shape information obtained from the elongated device 310 (obtained from the tracking system 230 at operation 506 or similar). For example, a machine learning algorithm performing image data processing may search the image data for a set of voxels having shapes corresponding to those described by the shape information obtained from the elongated device 310, as described herein. Voxels corresponding to the shape described by the shape information, or a portion thereof, may be used to identify other voxels corresponding to a medical device having that shape. In some embodiments, the shape information may be used to identify a search region or search area including voxels corresponding to the medical device. By using shape information, the segmentation process for segmenting the medical device from the remainder of the three-dimensional image data can be simplified and the segmentation process can be performed more quickly. In this way, some embodiments of operation 504 may include using shape data to identify regions of the three-dimensional image to be segmented and segmenting the shape of portions of the medical device from the regions of the three-dimensional image.

[0069] For example, when a medical device (such as medical device 200 or 304) is in place, the medical device or a portion thereof is included in the image when a CT image is acquired. The medical device can be segmented or filtered from the image, and a model can be formed from the medical device including a representation of its shape. Similarly, anatomical pathways can be identified within the image data and used for general surface- or line-based models.

[0070] For example, by segmenting or filtering using CT values ​​or radiodensity values ​​such as Hounsfield values ​​associated with medical device 304 or its components, medical device 304 can be identified as a medical device in the image data. Data associated with medical device 304 can be isolated from other portions of the image data associated with patient P or a specific tissue type. A three-dimensional mesh can be formed around the isolated data and / or a centerline representing the centerline of the medical device can be determined.

[0071] In some embodiments, a 3D mesh can be rendered as a model on the display. This model can be rendered as opaque or semi-transparent. One or more other models of the anatomy can also be generated during the segmentation process. Figure 6A A rendered 3D image 600 is depicted, generated from a 3D CT scan of a human lung. For example, image 600 can be rendered onto a display, such as... Figure 1 The display 110. Image 600 is a filtered image showing the bronchial passage 602 of the lung and a flexible, elongated medical device 604, which in some embodiments may be... Figure 3A and Figure 3B The elongated device 310 of medical device 304. For example... Figure 6A The depicted image 600 has been segmented and filtered to make the soft tissue of the lungs invisible. (As shown...) Figure 6B As seen in the image, image 600 can be further segmented so that only medical device 604 is displayed as a rendering model. Image data associated with medical device 604 can be isolated from other types of image data and processed and manipulated separately. Information in image 600 corresponding to medical device 604 can be segmented and processed to generate a three-dimensional model of the shape of medical device 604. Figure 6B Describe an isolated segmented shape model of the medical device 604 corresponding to the shape information in image 600.

[0072] As shown, the rendered image 600 also includes a target 606. For example, target 606 could be a tumor or lesion present in the lung of patient P. The three-dimensional image data could include data characterizing and defining the location and shape of the tumor, which could be segmented or filtered by the control system 112 to determine the location and shape of the tumor, allowing the tumor to be selectively displayed. For example, the tumor could be rendered as an opaque object in image 600, while other tissues could be rendered as translucent, allowing the tumor to be seen through the other tissues. The control system 112 could calculate the position and orientation of the distal tip of instrument 604 and determine a vector 608 extending from the distal tip of instrument 604 to target 606. Vector 608 could be referred to as a trajectory vector and could indicate a direction in which the operator O should steer the medical instrument to enter target 606 for biopsy or treatment.

[0073] At operation 506, when a portion of the medical device is located within the patient's anatomy, shape data or information can be obtained from the medical device. In some embodiments, operations 504 and 506 can be performed simultaneously or nearly simultaneously, such that the information contained in the intraoperative three-dimensional image data and shape data has a high temporal correspondence. Shape data can be obtained from or through tracking system 230, which queries or receives data from one or more sensors. For example, tracking system 230 can receive position / shape information from fiber optic shape sensors or other sensor systems, such as multiple electromagnetic position sensors positioned along the elongated device 310. One or more sensors can be positioned at known or trackable locations relating to physical features of the elongated device 310, such as the outer wall, which are visible in images or a set of images acquired including the medical device positioned within the patient P. For example, Figure 3A and Figure 3B The medical device 304 includes an elongated device 310 having an optical fiber shape sensor 314 extending therein. As described herein, the material of the elongated device 310 can be imaged by an imaging system 330 and included in three-dimensional image data characterizing a patient's anatomy. The optical fiber shape sensor 314 can be used to obtain a set of measurement points relative to either an instrument reference frame or a shape sensor reference frame (X). I Y I Z I The shape of the elongated device 310 is described. Therefore, the initially acquired shape data can be represented in the shape sensor reference frame, while the initially acquired three-dimensional image is represented in the image reference frame or the anatomical model reference frame (X). CT Y CT Z CT )middle.

[0074] At operation 508, the segmented shape of the medical device can be registered with the shape data obtained from the medical device. In this way, the shape sensor reference frame or the device reference frame (X...) I Y I Z I ) is registered to the image reference frame or the anatomical model reference frame (X). CT Y CT Z CTConversely, the registration can be performed by rotating, translating, or otherwise manipulating rigid or non-rigid transformation points associated with the segmented shape and points associated with the sensed shape data. Such registration between the intraoperative model reference frame and the instrument reference frame can be achieved, for example, by using a point-based Iterative Closest Point (ICP) technique (as described in U.S. Provisional Patent Applications US62 / 205,440 and US62 / 205,433, which are incorporated by reference) or another point cloud registration technique. Alternatively, registration can be performed by matching and registering feature points in the instrument and image point clouds, where point correspondences are determined based on shape similarity in some feature space. In some embodiments, the segmented shape of the medical device is registered to shape data in the shape sensor reference frame, and the associated transformation (a vector applied to each point in the segmented shape to align with the shape data in the shape sensor reference frame) can then be applied to the entire 3D image and / or subsequently acquired during the medical procedure. The transformation can be a six-degree-of-freedom (6DOF) transformation, which allows shape data to be translated or rotated in X, Y, and Z, as well as in any one or all of pitch, roll, and yaw.

[0075] In some embodiments, the shape sensor reference frame can be registered to a surgical environment reference frame (X) before the 3D image is registered to it. S Y S Z S Alternatively, another patient reference frame can be used. In this way, shape data and intraoperative 3D images or models can be registered to a common reference frame, which can be the surgical environment reference frame. Since the medical device can be coupled to an instrument holder and / or insertion stage at a known location within the surgical environment, the spatial relationship between the shape sensor reference frame and the surgical environment reference frame can be easily determined.

[0076] In some embodiments, a first transformation may be applied to the 3D image to bring it into a shape sensor reference frame, and a second transformation may then be applied to the 3D image to bring it into a surgical environment reference frame. In some embodiments, these two transformations may be combined into a single transformation. After the 3D image and shape data are co-registered, additional operations may be performed as part of operation 510, in which medical procedures (such as surgery, biopsy, ablation, illumination, irrigation, or aspiration) are performed using the registration information by displaying the registration information on a monitor to provide image-based guidance to the operator O.

[0077] Some embodiments of method 500 may include operations to update the registration. For example, control system 112 may detect or determine that a medical device located within the patient's anatomy has been moved. Subsequently, additional three-dimensional images of the patient's anatomy may be acquired. Registration may be updated based on the additional three-dimensional images. Furthermore, some embodiments of method 500 include obtaining two-dimensional images of the patient's anatomy when a portion of the medical device is positioned within the patient's anatomy. This two-dimensional image may be obtained from the same viewpoint as the three-dimensional image, such that both images are obtained from a common imaging frame. Additionally, in some embodiments, the two-dimensional image may be obtained from a different imaging modality used to obtain the three-dimensional image. For example, the three-dimensional image may be a CT image, while the two-dimensional image may be a fluorescence imaging image. In some embodiments, shape information may be used to register the three-dimensional and two-dimensional images to a common reference frame.

[0078] Now for reference Figure 7A The combination shown above is as described above. Figure 3A and Figure 3B A side view of the exemplary surgical environment 300 discussed. Figure 3A and Figure 3B In contrast, the distal tip 318 of the elongated device 310 is inserted more deeply into the patient's anatomy, such that the distal portion of the flexible elongated device 310 (shown in range 700) is at least partially dependent on the shape of the anatomy into which the elongated device 310 is inserted. The shape of the distal portion in range 700 is... Figure 7B It is shown in more detail in the middle. Figure 7B Also shown is an optical fiber shape sensor 314 extending to the distal tip 318 within the elongated device 310. The shape sensor 314 can be queried to obtain... Figure 7C The shape data 710 is visualized in the image. This shape data corresponds to the shape of the elongated device 310 positioned within the patient's anatomy. The shape data 710 is... Figure 7C Visually illustrated, shape data 710 can be represented by numerical values ​​stored in memory, such as coordinates in a shape sensor reference frame. The shape data 710 and the three-dimensional image acquired by imaging system 330 can include timestamps to ensure that the information reflects the position and orientation of the medical device within the patient's anatomy over approximately the same time period. The shape data 710 and the segmented shape of the model of medical device 604 can be registered to bring the image data shown in image 600 into a common reference frame with the elongated device 310. This common reference frame can be a sensor or device reference frame (X-frame). I Y I Z I ), or it could be a reference frame for the surgical environment (X). S Y S Z S Both instrument 304 and imaging system 330 can be included. Figure 1 In an embodiment of the medical system 100.

[0079] Figure 8 A composite image 800 is depicted, generated by registering segmented shapes of portions of a medical device with shape data from the same portions of the medical device. The composite image 800 includes a surface model 802 of a bronchial passage, which... Figure 6A An internal perspective view of the anatomical pathway 602 of the bronchial passage is shown. The surface model 802 can be generated from imaging data obtained using the imaging techniques described above. Figure 8 As shown, the internal perspective view can be provided to the operator O to facilitate image-guided medical procedures. The internal perspective view presents a view of model 802 from the distal tip of medical device 804, which can be as follows: Figure 6A and Figure 6B The same device is shown as 604.

[0080] The composite image 800 also includes an image of the target 806. For example, the target 806 may be a tumor or lesion present in the lung of patient P. The three-dimensional image data may include data characterizing and defining the location and shape of the tumor, which may be segmented or filtered to determine the orientation of the tumor and to allow for selective display of the tumor. For example, the target 806 may be rendered as an opaque object, while other tissues are rendered as translucent, allowing the tumor to be seen through the other tissues. For example, when the target 806 is not co-located with the wall of model 802, model 802 may be rendered as translucent to allow a perspective view of the target 806. The control system 112 may calculate the position and orientation of the distal tip of the instrument model 804 and may determine a vector extending from the distal tip of the instrument model 804 to the target 806. This vector may be referred to as the trajectory vector 808 and may indicate a direction in which the operator O should steer the actual medical instrument to approach the actual target for biopsy or treatment. The trajectory vector 808 may also be included in the external perspective view, such as... Figure 6A As shown. Figure 8 This is shown on the display (such as) Figure 1 A composite image 800 is rendered in the display 110. The control system 112 can display other user interface elements in the display 110. For example, the display 110 also displays in a window 810. Figure 6A Image 600. As operator O navigates within the patient, the perspective views shown in images 600 and 800 can be updated in real time and rendered onto display 110. Other information and elements, such as physiological information or control elements, can be presented to operator O on display 110.

[0081] Operator O can navigate within the patient's anatomy, orienting the medical device based on image 600, which may include filtered CT image data. Since the medical device is registered to image 600, movement of the medical device relative to the patient P can be visualized through the display of the corresponding movement of the displayed medical device 604 within the patient's anatomy as shown in image 600.

[0082] In some embodiments, after the control system 112 determines and displays the trajectory vector 808, the control system 112 may impose a constraint that overrides the movement command received from the operator O via the master assembly 106. For example, in some embodiments of method 500, the movement command may be received from a user control device (such as the master assembly 106) indicating a commanded movement of the distal tip of the medical device. The control system 112 or another component may override the received movement command to constrain the movement of the distal tip of the medical device such that movement along the trajectory vector 808 is permitted, while movement away from the trajectory vector 808 is not permitted. For example, the movement command may be received as a vector indicating the direction in which the operator O wishes to move the distal tip of the medical device. These movement command vectors may have components parallel to the trajectory vector 808 and components not parallel to the trajectory vector 808. The control system 112 may filter out the portion of the movement command vector that is not parallel to the trajectory vector 808 and may cause the portion of the movement command vector that is parallel to the trajectory vector 808 to be executed by a servo motor.

[0083] In some embodiments, trajectory vector 808 may be generated in response to a request from operator O or based on a calculated distance from target 806. Additionally, operator O or another operator may be provided with user interface elements on display 110, the selection of which allows operator O to move the medical device away from trajectory vector 808.

[0084] Reference Figure 9 Further examples of surgical methods are described. At this point, Figure 9 This is a flowchart illustrating a method 900 for performing image-guided surgery in a surgical environment 300 according to some embodiments of the present disclosure. The methods of this specification, including method 900, are described in... Figure 9The methods are illustrated as a set of blocks, steps, operations, or procedures. Not all illustrated, enumerated operations can be performed in all embodiments of method 900. Additionally, some additional operations not explicitly shown in the method may be included before, after, between, or as part of the enumeration process. Some embodiments of the methods in this specification include computer-readable instructions corresponding to the procedures of the method stored in non-transitory memory. These instructions may be executed or coordinated by one or more hardware processors, such as the processor of control system 112.

[0085] Referring to operation 902, preoperative image data of the patient's anatomy is obtained from the imaging system. In various examples, the image data includes CT data, MRI data, temperature recording data, ultrasound data, OCT data, thermal imaging data, impedance data, laser imaging data, nanotube X-ray imaging data, and / or other suitable data representing the patient's anatomy. In some embodiments, the image data may include two-dimensional images from multiple perspectives, which may be combined into a pseudo-three-dimensional image. Thus, the preoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or velocity-based information) images. The preoperative image data is in a first reference frame (i.e., the preoperative image reference frame).

[0086] Referring to operation 904, a preoperative model of the patient's anatomy is generated from preoperative image data. Since the model is generated from preoperative image data, it can be placed within a preoperative image reference frame. In some examples, distinguishing features (such as radiopaque linearity, MRI response, density, and / or color) are used to differentiate different tissues, structures, and spaces within the patient's anatomy. The model may contain portions or entire anatomical organs or regions, such as including... Figures 4A-4D The lungs are represented by a 400-degree trunk. In one such example simulating the lungs, the model can describe the various orientations and shapes of anatomical pathways within the lungs and their connectivity. In some examples, the model includes anatomical structures such as blood vessels, pleura, bullae, and / or similar features.

[0087] Referring to optional operation 906, the operator can generate a navigation path through the anatomical pathways in the preoperative model to guide the medical device to the target. In this regard, the preoperative model may include multiple targets to be accessed during the medical procedure, such as tumors, lesions, or other areas of tissue. Such medical procedures may include biopsy, ablation, illumination, irrigation, aspiration, etc., and may be performed by guiding the medical device (such as...) Figures 3A-3B and Figures 4A-4D The medical device 304 is propelled to a target via an anatomical pathway. Therefore, a planning step can be performed where the target is identified by the operator within a preoperative model, and a navigation path via the anatomical pathway of the preoperative model is determined to guide the medical device to the target.

[0088] Referring to operation 908, the preoperative image reference frame is registered to the instrument reference frame. The medical instrument is advanced into the patient P, specifically into the patient anatomy imaged in operation 902 and included in the preoperative model in operation 904. This can be performed as part of a registration process or as part of an image-guided medical procedure as described above. During advancement, shape data in the instrument reference frame can be acquired from sensors in the medical instrument. Since the medical instrument typically conforms to the anatomical pathways of the patient anatomy, the shape data obtained from the medical instrument corresponds to the orientation of the medical instrument within the patient. Therefore, the computational system can determine the registration of the preoperative image reference frame of the preoperative model with the instrument reference frame of the medical instrument by comparing the shape data of the medical instrument with the shape of the pathways in the preoperative model.

[0089] Matching techniques such as ICP or Singular Value Decomposition (SVD) can be used to match the shape of a medical device with the shape of a channel in a preoperative model. The relationship between the medical device and the preoperative model is thus determined. An example of this registration process is described in PCT Publication WO 2016 / 191298 (published December 1, 2016) (disclosing “Systems and Methods of Registration for Image Guided Surgery”), which is incorporated herein by reference in its entirety.

[0090] In some examples, the medical device can be further advanced into the patient's anatomy. In some examples, operator O remotely advances the medical device toward the target along the navigation path of operation 906. In alternative examples, the medical device can be advanced manually. In some examples, the medical device is advanced using navigation guidance from a planned path generated from operation 906.

[0091] In some examples, navigation guidance can be displayed in the preoperative model. Composite images (such as...) Figure 8 The composite image shown can be provided to display the orientation of the instrument relative to the target and anatomical pathway. (See reference...) Figure 8 As described, the composite image 800 includes a surface model 802 of the bronchial passage, which is... Figure 6A An internal perspective view of the anatomical pathway 602 of the bronchial passage is shown. The surface model 802 can be generated from imaging data obtained using the imaging techniques described above. Figure 8 As shown, the internal perspective view can be provided to the operator O to facilitate image-guided medical procedures. The internal perspective view shows a view of model 802 from the distal tip of medical device 804, which can be... Figure 6A and Figure 6B The same device is shown as 604.

[0092] The composite image 800 also includes an image of target 806. For example, target 806 could be a tumor or lesion present in the lung of patient P. The three-dimensional image data could include data characterizing and defining the location and shape of the tumor, which could be segmented or filtered to determine the tumor's orientation and allow for selective display. For example, target 806 could be rendered as an opaque object, while other tissues are rendered semi-transparent, allowing the tumor to be seen through the other tissues. For example, model 802 could be rendered semi-transparent to allow a perspective view of target 806 when target 806 is not co-located with the walls of model 802. The control system 112 could display other user interface elements on display 110. For example, display 110 also shows in window 810. Figure 6A Image 600. As operator O navigates within the patient, the perspective views shown in images 600 and 800 can be updated in real time and rendered onto display 110. Other information and elements, such as physiological information or control elements, can be presented to operator O on display 110.

[0093] Operator O can navigate within the patient's anatomy, steer the medical device based on image 600, which may include filtered CT image data. Since the medical device is registered to images 600 and 800, movement of the medical device relative to the patient P can be visualized by displaying the corresponding movements of medical devices 604 and 804 within the patient's anatomy, respectively, as shown in images 600 and 800. In some examples (not shown), the medical device is positioned at a distance from a target within the anatomical pathway, such that the target is not displayed in the model, and indicators providing directional guidance toward the target are displayed.

[0094] Referring to operation 910, intraoperative image data of the patient's anatomy is acquired while the medical instrument is within the patient's anatomy. In some examples, the medical instrument is close to the target. Therefore, intraoperative image data can be acquired using some combination of the patient's anatomy, the medical instrument, and the target. Some aspects of operation 910 can be performed substantially as described in operation 502. Imaging systems such as the three-dimensional imaging system 330 described above can use any suitable imaging technique (such as CT, fluorescence microscopy, temperature recording, ultrasound, OCT, thermography, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques) to acquire intraoperative images. By rotating the imaging element around the patient P, the imaging system can acquire a set of intraoperative images from one or more angles; the intraoperative images can be still images, a series of still images, or videos. Therefore, each intraoperative image can have a corresponding recording time and orientation (e.g., rotation angle) relative to other images.

[0095] Similarly, intraoperative image data can have a corresponding orientation relative to the medical device. As the medical device moves in three dimensions, portions of the device may be blurred or otherwise poorly imaged in some or all of the image data. For example, the length of the device extending perpendicular to the imaging plane may not be correctly acquired in the intraoperative image data. Therefore, in some examples, the system (e.g., control system 112) guides the placement, shape, or orientation of the medical device prior to imaging in operation 910 to improve imaging of the medical device. In one such example, the system guides placement to increase the amount of length of the medical device along the imaging plane. In another such example, the system guides the device to bend or hook along the imaging plane to create different features in the intraoperative image data. In some examples, the system (e.g., control system 112) guides the placement or orientation of the imaging system prior to imaging in operation 910 to improve imaging of the medical device, for example, by increasing the amount of length of the medical device along the imaging plane. The intraoperative image data as a whole can be in a second reference frame (i.e., an intraoperative image reference frame) different from the preoperative image reference frame.

[0096] Referring to operation 912, the computing system registers the instrument reference frame to the intraoperative image reference frame. Some aspects of operation 912 can be performed substantially as described in operations 504-508. In some such examples, the computing system obtains shape data from the medical device relating to the position, orientation, and / or attitude of the medical device. The tracking system of the computing system can receive this position, orientation, and / or attitude information from fiber optic shape sensors or other sensor systems, such as multiple electromagnetic position sensors positioned along an elongated portion of the medical device. One or more sensors can be positioned at known or trackable locations relating to the physical characteristics of the medical device (e.g., the outer wall of the tube distinguishable in the intraoperative images). The sensors can be used to obtain a set of measurement points that describe the shape of the medical device within the instrument reference frame. This shape data can be obtained at regular intervals or in response to triggers such as interruptions or changes in position of the medical device, and the shape data can include timestamps or other indicators at the time each subset of the shape data is obtained. In one example, a subset of the shape data is obtained in response to an interruption that occurs each time the imaging system acquires images of intraoperative image data in operation 910.

[0097] Shape data can be used to help locate a representation of a medical device in image data. Therefore, the computational system can perform image data segmentation to identify which parts correspond to the medical device in the intraoperative image and which correspond to the patient's anatomy. By using shape information, the segmentation process for segmenting the medical device from the remainder of the three-dimensional image data can be simplified and performed more quickly. Some aspects of the segmentation can be performed substantially as described in operation 504. In some embodiments, the computational system converts the intraoperative image data into a two-dimensional or three-dimensional model of the patient's anatomy (such as including...). Figures 4A-4D The medical device is a two-dimensional or three-dimensional model of the lungs (400 units of the torso) and the medical device. In various examples, the pixels or voxels corresponding to the medical device are identified in part based on distinguishing characteristics visible in intraoperative images, such as radiopaqueness, MRI response, density, and / or color. For this purpose, the medical device may include structures that are detectable or visible within the intraoperative images, which serve as benchmark features.

[0098] For example, Figure 10A and Figure 10B This is a simplified schematic diagram of a side view of a medical device 1000 according to some embodiments of the present disclosure. The medical device 1000 includes an elongated device 1002, which may be substantially similar to elongated device 202 and / or elongated device 310. See also... Figure 10A The distal portion 1004 of the elongated device 1002 includes an axial support structure 1006 configured to bend in response to an actuating force. Thus, when unequal actuating forces are applied to the axial support structure 1006, the distal portion 1004 bends. Further examples of axial support structures are provided in U.S. Patent Application US15 / 685,979 (filed August 24, 2017) (disclosing “Axial Support Structure for a Flexible Elongate Device”) and U.S. Provisional Patent Application US62 / 535,673 (filed July 21, 2017) (disclosing “Flexible Elongate Device Systems and Methods”), each of which is incorporated herein by reference in its entirety. The axial support structure 1006 may have properties (e.g., radiopaqueness, MRI response, density, and / or color) that distinguish it from the surrounding patient anatomy and / or the remainder of the elongated device 1002 in intraoperative images. In a further example, the distal tip of the elongated device 1002 includes a control ring 1008, a tip ring 1010, or other structures at the distal tip that have distinguishable characteristics visible in intraoperative images.

[0099] In some examples, the elongated device 1002 includes a tool lumen 1011. The tool lumen 1011 allows for the deployment of a medical tool 1012 via the elongated device 1002. The medical tool 1012 may be substantially similar to the medical tool 226 described above and may be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. Suitable medical tools 1012 include biopsy instruments (such as needles), laser ablation fibers, and / or other surgical, diagnostic, or therapeutic instruments, and therefore the medical tool 1012 may have an end effector, such as a scalpel, blunt blade, forceps, grasper, scissors, applicator, and / or electrode. Parts of the tool lumen 1011 and the medical tool 1012 (such as the end effector) may have distinguishable characteristics in intraoperative images and thus may serve as reference features.

[0100] See Figure 10B The elongated device 1002 may further include a shape sensor cavity 1014 through which a shape sensor 1016 extends. The shape sensor 1016 may be substantially similar to the shape sensor 222 described above and may include an optical fiber extending through the shape sensor cavity 1014. To reduce discrepancies between data obtained via the shape sensor 1016 and imaging data, the shape sensor cavity 1014 and / or the shape sensor 1016 may have distinguishable characteristics in intraoperative images for use as reference features. Because the shape sensor cavity 1014 can be in close proximity to the shape sensor 1016 disposed therein, the imaging data corresponding to the shape sensor cavity 1014 can closely track the shape data obtained from the shape sensor 1016.

[0101] Return to reference Figure 9In operation 912, the computational system performing segmentation identifies these reference features (e.g., axial support structure 1006, control ring 1008, tip ring 1010, tool cavity 1011, medical instrument 1012, shape sensor cavity 1014, shape sensor 1016) in the intraoperative image data to identify one or more parts of the medical device, and registers the device reference frame to the intraoperative image reference frame using the position, orientation, and / or pose of the medical device determined from the shape data obtained above. Registration can be manipulated by rotating, translating, or otherwise manipulating rigid or non-rigid transformation points associated with the segmented shape and points associated with the sensed shape data. For example, registration between the device reference frame and the intraoperative image reference frame can be achieved using ICP technology or another point cloud registration technique. Alternatively, registration can be performed by matching and registering feature points within the device and image point clouds, where point correspondences are determined based on shape similarity in some feature space. In some embodiments, the segmented shape of the medical device is registered to shape data in a shape sensor reference frame, and the associated transformation (a vector applied to each point in the segmented shape to align with the shape data in the device reference frame) can then be applied to the entire 3D image and / or a 3D image subsequently acquired during a medical procedure. This transformation can be a 6DOF transformation, allowing the shape data to be translated or rotated in the X, Y, and Z axes, as well as in any one or all of pitch, roll, and yaw.

[0102] To improve the accuracy of this process, the computational system can optionally perform temporal matching between shape data and intraoperative image data. In some examples, the shape data comprises multiple subsets, each containing data points acquired at discrete times. These subsets can be acquired at regular intervals or in response to triggers such as interruptions or changes in the position of medical instruments, and the shape data can include a timestamp or other indicator at the time each shape data unit is acquired. Similarly, the images constituting the intraoperative image data can each have a timestamp record of when the image was acquired. In these examples, the computational system performing the registration can utilize the timestamps to correlate the subsets of shape data with the images of the intraoperative image data in a timely manner. In this respect, the computational system can determine the registration of the instrument reference frame with the intraoperative image reference frame by comparing intraoperative images acquired at a certain point in time with concurrent subsets of the shape data. This can reduce errors caused by anatomical movement, instrument movement, or sensor drift over time.

[0103] Alternatively, or in an alternative, the computational system may optionally correlate the temporal sequence of shape data and images with circulatory motion. For example, the patient's anatomy may move during routine circulatory activities (e.g., cardiac activity, respiratory activity, etc.). The computational system can determine at which phase of circulatory activity (e.g., diastole, respiratory maintenance) each subset of shape data and each image of intraoperative image data was acquired. For this purpose, the timestamps or triggers associated with the data can be correlated with various other patient status monitors that monitor circulatory activity (such as cardiac monitors or respiratory monitors (e.g., patient treadmills)). Thus, the computational system can determine the registration of the instrument reference frame with the intraoperative image reference frame by comparing intraoperative images acquired at a specific phase of circulatory activity with a subset of shape data acquired at the same phase. Since the activity may be periodic, image data from one cycle of activity can be compared with a subset of shape data acquired during another cycle. Furthermore, some points during circulatory activity may be more favorable for imaging. Therefore, in some examples, intraoperative images taken during the optimal point of circulatory activity are identified and compared with a subset of shape data acquired simultaneously.

[0104] Because intraoperative imaging data may not capture the entire medical device, in some examples, the computing system may optionally divide the medical device into parts and determine which parts are captured by each image of the intraoperative imaging data. For each imaging part, the computing system can correlate the shape data of that part with the images in the intraoperative imaging data. This can include temporal and spatial correspondences. For example, if the first image of the intraoperative imaging data captures the farthest part of the medical device in the first second of the scan, the shape data corresponding to the farthest part in the first second of the scan can be compared with the first image. If the second image of the intraoperative imaging data captures the second farthest part of the medical device in the second second of the scan, the shape data corresponding to the second farthest part in the second second of the scan can be compared with the second image.

[0105] To further improve accuracy, the computational system can perform multiple comparisons and weight each comparison individually. As mentioned above, intraoperative image data can include a set of images taken at different orientations. For example, intraoperative image data obtained by a moving rotation imaging element can include a set of images, each with a different time, orientation, and rotational orientation relative to the patient P. The computational system can perform registration using some or all of the images in the dataset. In this regard, each image can be weighted based on its time point (absolute time or time relative to cyclic activity), orientation, and / or other factors. For example, an image may be acquired during a time point with significant patient movement and may be weighted less than other images in the set. In another example, an image taken at a less visible orientation of the medical device, target, or another anatomical structure may be weighted less. In yet another example, an image may be of low quality and may be weighted less. When determining the registration from the device reference frame to the intraoperative image reference frame, the computational system can consider the weighting factor associated with each unit of each image or shape data. In some embodiments, suboptimal images can be identified and re-captured through repeated operation 910. The re-captured intraoperative images can then be used to perform registration.

[0106] Next, referring to operation 914, the preoperative image reference frame of the preoperative model is registered to the intraoperative image reference frame. As described above, the computing system can register the preoperative image reference frame to the instrument reference frame in operation 908, and the instrument reference frame to the intraoperative image reference frame in operation 912. Therefore, a common reference frame (i.e., the instrument reference frame) can be used to perform the registration of the preoperative image reference frame and the intraoperative image reference frame.

[0107] Alternatively, feature-based registration can be used to register a preoperative reference frame to an intraoperative reference frame to compare the corresponding models and / or images, and to compute the system's localization of the corresponding baseline features in the intraoperative model / image and the preoperative model / image. Baseline features include both artificial features and anatomical features that possess discriminative properties (e.g., radiolucency, MRI response, density, and / or color) in the intraoperative image data, preoperative image data, and / or preoperative model. Discriminative properties can differ across datasets; for example, MRI response in preoperative MRI data and radiolucency in intraoperative image data. Suitable baseline features include artificial features of bones, tissue boundaries, other organs and organ systems, voids, and both internal and external anatomical structures. In some examples, a set of external baseline features, such as a patient slab or stereotactic frame, is placed on the patient during preoperative and intraoperative imaging. Once the corresponding baseline features are identified in some combination of intraoperative and preoperative image data and / or preoperative models, the location and orientation of the baseline features can be used to determine the relationship between the preoperative and intraoperative image reference frames.

[0108] As a complement or alternative to feature-based registration, computational systems can perform image-based matching to match individual images from preoperative image data or models with images from intraoperative image data. For a given preoperative image, the computational system can select a corresponding intraoperative image for comparison, and vice versa. The corresponding image can be selected based on its orientation, rotation, and / or other factors. In the example of obtaining intraoperative image data via a movable rotating imaging element, each intraoperative image can have an orientation and rotational orientation relative to the patient P. For some or all of the intraoperative images, the corresponding preoperative image can be selected based on having a similar orientation and rotational orientation relative to the patient P. If such a preoperative image is unavailable, it can be generated by simulating an image taken with that orientation and rotational orientation using a preoperative model.

[0109] The computational system may optionally perform time matching of preoperative image data or models with intraoperative image data. In some such examples, the preoperative image data or models and intraoperative image data may include timestamps indicating when each image was acquired. In some such examples, corresponding images may be selected based on these timestamps. The timestamps may be correlated with various other patient status monitors to monitor circulatory activity and determine at which stage of the circulatory activity images were taken. For intraoperative images taken at a specific stage of the circulatory activity (e.g., diastole, respiratory maintenance), corresponding preoperative images may be selected based on images taken at a similar or corresponding stage of the activity.

[0110] Once the corresponding images are selected, they can be compared to determine changes in position and / or orientation, thereby establishing the relationship between the preoperative image reference frame and the intraoperative image reference frame.

[0111] In some examples, each image, or each pair of corresponding images, may be weighted based on its time point (absolute time or time relative to the cyclic activity), orientation, and / or other factors. For example, an image may be acquired during a time point with significant patient movement and may be weighted less than other images. In another example, an image taken at an orientation where the baseline feature is less visible may be weighted less. In yet another example, an image may be of low quality and may be weighted less. When determining the registration from the preoperative image reference frame to the intraoperative image reference frame, the computational system may consider the weighting factors associated with each image. In some embodiments, suboptimal intraoperative images may be identified and re-captured through repeated operation 910.

[0112] It should be noted that shape-based registration can be combined with feature-based registration and / or image-based registration by iterating the two registration processes once or multiple times, wherein each iteration improves the accuracy of the registration between the preoperative image reference frame and the intraoperative image reference frame.

[0113] With two image reference frames registered, the computational system can update the preoperative model based on the intraoperative image data shown in Operation 916. (Return to Reference) Figure 8 As an example, a preoperative model includes a representation of the patient's anatomy, targets such as the tumor or other areas of tissue to be treated, and / or medical instruments. After intraoperative images are acquired, the real-time orientation of the tumor relative to other anatomical structures and / or instruments may have shifted from its position represented in the preoperative model, such as... Figure 8 As shown. The computational system can update the size, orientation, and / or other attributes of the target based on intraoperative image data. In one such example, the computational system updates the target's orientation based on a measured distance between the medical device and the corresponding tissue, determined according to intraoperative image data and / or shape data obtained from the medical device. If the preoperative model changes more than a threshold relative to the intraoperative image data, some or all of the preoperative model can be replaced to reflect the tumor's real-time orientation relative to the medical device. In an alternative example, when the preoperative model changes more than a threshold and the intraoperative model is used to replace some or all of the preoperative model, the computational system generates a new model from the intraoperative image data, essentially as described in operation 904. In some examples, other anatomical structures (such as pathways, blood vessels, and the like) are provided in the preoperative model generated in operation 904. In these examples, the size, shape, and / or relative orientation of other anatomical structures can also be updated in the preoperative model in a similar manner. In this way, anatomical structures that should be avoided can be displayed as further navigation aids during the medical procedure.

[0114] Referring to operation 918, optionally, the computing system can display a representation of the preoperative model updated in operation 916. Figure 11 A composite image 1100 is depicted, which includes, according to, a display 110 (such as...) Figure 1 The composite image 1100 is a representation of the preoperative model of an example of this disclosure displayed on a monitor. In many respects, image 1100 may be substantially similar to image 800. The composite image 1100 includes a surface model 1102 of the pathway (e.g., bronchial passage) where the medical device is located, which is generated from the preoperative model. An internal perspective view may be provided to the operator O to facilitate image-guided medical procedures. The internal perspective view presents a view of model 1102 from the perspective of the distal tip of the medical device. For reference, a rendering 1104 of the medical device may be displayed. If a medical instrument has been extended through the medical device, a rendering 1106 of the medical instrument may be displayed.

[0115] The composite image 1100 also includes an image of the target 1108. The target 1108 can be rendered as an opaque object, while other tissues are rendered translucent, allowing the tumor to be seen through the other tissues. For example, when the target 1108 is not co-located with the wall of the model 1102, the model 1102 can be rendered translucent to allow a perspective view of the target 1108. The computing system can calculate the position and orientation of the distal tip of the instrument model 1104 and can display the trajectory vector 1110 extending from the distal tip of the instrument model 1104 to the target 1108. Other information and elements, such as physiological information or control elements, can be presented to the operator O in the display 110. For example, the display 110 can include in the window 1112 Figure 6A Image 600.

[0116] Optionally, the operator O and / or the remote operating system can use an updated preoperative model to perform image-guided medical procedures. For example, the operator O or the remote operating system can navigate a medical device within the patient's anatomy by turning the medical device based on image 1100. Since the medical device is registered to image 1100, the movement of the medical device relative to the patient P can be visualized by displaying the corresponding movement of the displayed medical device 1104 within the patient's anatomy as shown in image 1100. Once the medical device is positioned near the target, the operator O can use medical instruments (such as...) Figure 2A and Figure 2B The medical tool 226 is propelled through the medical device. The operator O can use the medical tool to perform procedures on a target (such as surgery, biopsy, ablation, illumination, irrigation, or aspiration), and can use its rendering 1106 to visualize the movement and operation of the medical tool during the procedure.

[0117] Referring to operation 920, optionally, a second set of intraoperative images may be acquired. This can be performed substantially as described in operation 910. The second set of intraoperative images can be used to evaluate the effectiveness of the procedure. In one example, operator O performs an ablation procedure on the target, and the second set of intraoperative images acquires the target and surrounding anatomy to assess the impact of the ablation procedure on the target. In some examples, patient movement may cause displacement of the anatomy, target, and / or instrument orientation. Therefore, the described registration technique can be used to register the second set of intraoperative images to the first set of intraoperative images taken prior to the ablation procedure, or to register the second set of intraoperative images to a preoperative model.

[0118] In some examples, obtaining a second set of intraoperative images involves registration from the instrument reference frame of Operation 912 to the intraoperative reference frame to determine the local region to be imaged. Since the positions of the medical instruments and targets in the intraoperative reference frame are known, the region to be imaged can be determined more precisely. Obtaining a second set of intraoperative images using a relatively small region can reduce patient radiation exposure. In some examples, a second set of intraoperative images may be acquired to supplement low-quality images in the first set of intraoperative images. In some examples, multiple intraoperative images may be acquired, each of which can be registered to a preoperative model or any previous intraoperative image.

[0119] Some embodiments of these embodiments of the present disclosure can facilitate the registration of three-dimensional images, or a set or series of three-dimensional images, with the shape of a medical device. For example, one or more images from a cone-beam CT scanner acquired when the medical device is in place can also be registered to the medical device by acquiring shape data from the medical device at or around the time of acquiring one or more images. A transformation relating the shape of the medical device in the image to shape data from a fiber optic shape sensor can be applied to the entire image or a portion thereof to bring the image into a common reference frame with the medical device. Furthermore, images such as preoperative images can also be registered to a common reference frame for image-guided medical procedures. Additionally, the orientation of a tumor or other target in the image can be used to constrain or otherwise adjust movement commands received from the operator, causing the medical device to follow a defined trajectory to reach the tumor for biopsy or treatment.

[0120] One or more elements in embodiments of the present invention can be implemented in software to execute on a processing device of a computer system (such as control system 112). When implemented in software, the elements in embodiments of the present invention are essentially code segments that perform necessary tasks. The program or code segment can be stored in a non-transitory machine-readable storage medium or device, including any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include: electronic circuits; semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment can be downloaded via a computer network such as the Internet or an intranet.

[0121] It should be noted that the presented processes and displays may not inherently relate to any particular computer or other device. The necessary structures for various such systems will appear as elements in the claims. Furthermore, embodiments of the invention are described without reference to any specific programming language. It should be understood that the teachings of the invention as described herein can be implemented using various programming languages. Additionally, features described in more detail with respect to specific embodiments may be combined with features of other embodiments.

[0122] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the invention and not limiting, and that embodiments of the invention are not limited to the specific constructions and arrangements shown and described, as various other modifications will be apparent to those skilled in the art.

Claims

1. A non-transitory machine-readable medium storing instructions, said instructions causing said one or more processors, when executed, to: When a medical device is placed inside a patient's anatomy, imaging data of the patient's anatomy is obtained, wherein the imaging data includes multiple images, and wherein the imaging data is in an image reference frame; When the medical device is placed within the patient's anatomy, shape data is obtained from the medical device, wherein the shape data comprises multiple subsets, and wherein the shape data is in a device reference frame; A subset of the plurality of subsets of shape data is matched with an image in the plurality of images, wherein the matching of the subset with the image includes matching at least one of the following: matching the subset with the image based on a phase of a cyclic activity in which the subset of shape data was obtained and a phase of a cyclic activity in which the image was obtained; and matching the subset with the image based on a portion of the medical device acquired in the image; and The instrument reference frame is registered to the image reference frame by comparing a matching subset of the plurality of subsets of shape data with the images in the plurality of images.

2. The non-transient machine-readable medium of claim 1, wherein the registration of the instrument reference frame to the image reference frame includes assigning weights to each of the plurality of images.

3. The non-transient machine-readable medium of claim 2, wherein assigning the weight to each of the plurality of images comprises assigning the weight to the corresponding image based on at least one of the orientation of the image, patient movement during imaging, and the quality of the image.

4. The non-transient machine-readable medium of claim 1, further comprising guiding at least one of the placement or orientation of the imaging system before obtaining the imaging data based on the relationship between the medical device and the imaging plane of the imaging data.

5. The non-transient machine-readable medium of claim 1, further comprising determining the region for imaging based on registration from the instrument reference frame to the image reference frame.

6. A non-transitory machine-readable medium storing instructions, which, when executed by one or more processors, cause the one or more processors to: When a medical device is placed inside a patient's anatomy, imaging data of the patient's anatomy is obtained, wherein the imaging data includes multiple images, and wherein the imaging data is in an image reference frame; When the medical device is placed within the patient's anatomy, shape data is obtained from the medical device, wherein the shape data comprises multiple subsets, and wherein the shape data is in a device reference frame; Matching a subset of the multiple subsets of shape data with images from the multiple images, wherein the matching of the subset with the images includes: Based on the time when the subset of shape data was obtained and the time when the image was obtained, the subset is matched with the image; as well as The instrument reference frame is registered to the image reference frame by comparing a matching subset of the plurality of subsets of shape data with the images in the plurality of images.

7. The non-transitory machine-readable medium of claim 6, wherein matching the subset with the image comprises matching at least one of: matching the subset with the image based on a timestamp of the subset of the shape data and a timestamp of the image, and matching the subset with the image based on obtaining the subset in response to an interrupt generated in response to obtaining the image.

8. The non-transient machine-readable medium of claim 6, wherein the registration of the instrument reference frame to the image reference frame includes assigning weights to each of the plurality of images.

9. The non-transient machine-readable medium of claim 8, wherein assigning the weight to each of the plurality of images comprises assigning the weight to the corresponding image based on at least one of the orientation of the image, patient movement during imaging, and the quality of the image.

10. The non-transient machine-readable medium of claim 6, wherein the instructions further cause the one or more processors to guide at least one of the placement or orientation of the imaging system before obtaining the imaging data based on the relationship between the medical device and the imaging plane of the imaging data.

11. The non-transient machine-readable medium of claim 6, wherein the instructions further cause the one or more processors to determine the region for imaging based on the registration of the instrument reference frame to the image reference frame.

12. A medical system comprising: Medical devices; and The control system includes: processor; and The memory includes machine-readable instructions that, when executed by the processor, cause the control system to: When the medical device is placed inside a patient's anatomy, imaging data of the patient's anatomy is obtained, wherein the imaging data includes multiple images, and wherein the imaging data is in an image reference frame; When the medical device is placed within the patient's anatomy, shape data is obtained from the medical device, wherein the shape data comprises multiple subsets, and wherein the shape data is in a device reference frame; Matching a subset of the plurality of subsets of shape data with images from the plurality of images, wherein the matching of the subset with the image includes: matching the subset with the image based on the time when the subset of shape data was obtained and the time when the image was obtained; and The instrument reference frame is registered to the image reference frame by comparing a matching subset of the plurality of subsets of shape data with the images in the plurality of images.

13. The medical system of claim 12, wherein matching the subset with the image comprises matching at least one of the following: matching the subset with the image based on the timestamp of the subset of the shape data and the timestamp of the image, and matching the subset with the image based on obtaining the subset in response to an interrupt generated in response to obtaining the image.

14. The medical system of claim 12, wherein the registration of the device reference frame to the image reference frame includes assigning weights to each of the plurality of images.

15. The medical system of claim 14, wherein assigning the weight to each of the plurality of images comprises assigning the weight to the respective image based on at least one of the orientation of the image, patient movement during imaging, and the quality of the image.

16. The medical system of claim 12, wherein the instructions further cause the control system to guide at least one of the placement or orientation of the imaging system before obtaining the imaging data based on the relationship between the medical device and the imaging plane of the imaging data.

17. The medical system of claim 12, wherein the instructions further cause the control system to determine the region for imaging based on the registration of the instrument reference frame to the image reference frame.

Images