Systems and methods for using registered fluoroscopic images in image guided surgery

Abstract

The invention relates to a system and method for using registered fluoroscopic images in image guided surgery. A method performed by a computing system includes receiving a fluoroscopic image of a patient anatomy when a portion of a medical instrument is positioned within the patient anatomy. The fluoroscopic image has a fluoroscopic reference frame. The portion has a sensed location in an anatomical model reference frame. The method further includes identifying the portion in the fluoroscopic image, and identifying an extracted location of the portion in the fluoroscopic reference frame using the identified portion in the fluoroscopic image. The method further includes registering a fluoroscopic reference frame to an anatomical model reference frame based on the sensed location of the portion and the extracted location of the portion.

Description

[0001] This application is a divisional application of Chinese patent application 2021110606770, filed on February 10, 2017, entitled “Systems and methods for using registered fluoroscopic images in image-guided surgery”. Divisional application 2021110606770 is a divisional application of Chinese patent application 201780003111X (PCT / US2017 / 017433), filed on February 10, 2017, entitled “Systems and methods for using registered fluoroscopic images in image-guided surgery”.

[0002] Related Applications

[0003] This patent application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 294,870, filed on February 12, 2016, entitled “SYSTEMS AND METHODS FOR USING FLUOROSCOPY TO ASSIST INSTRUMENT NAVIGATION IN IMAGE-GUIDED SURGERY,” and U.S. Provisional Patent Application No. 62 / 294,879, filed on February 12, 2016, entitled “SYSTEMS AND METHODS FOR USING REGISTERED FLUOROSCOPIC IMAGES IN IMAGE-GUIDED SURGERY,” both of which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates to systems and methods for performing image-guided surgery, and more particularly to systems and methods for using registered real-time fluoroscopic images and prior time anatomical images during image-guided surgery. Background Art

[0005] Minimally invasive medical technology is intended to reduce the amount of tissue damaged during medical surgery, thereby reducing the patient's recovery time, discomfort and harmful side effects. This minimally invasive technology can be performed through natural holes in the patient's anatomical structure or through one or more surgical incisions. Through these natural holes or incisions, clinicians can insert minimally invasive medical instruments (including: surgical, diagnostic, therapeutic or biopsy instruments) to reach the target tissue location. In order to help reach the target tissue location, the position and movement of the medical instrument may be related to the static image of the patient's anatomical structure before or during the operation. Using image-related image-guided instruments, these instruments can navigate natural orifices or surgically created orifices in the anatomical system, such as the lungs, colon, intestines, kidneys, heart, circulatory system, etc. Traditionally, preoperative or intraoperative images of patient anatomical structures are usually detailed, often three-dimensional images. However, they are static, previous time representations of the patient's anatomical structure. Fluoroscopy is an imaging method that provides real-time images of the patient's anatomical structure, including any radiopaque instruments used during medical surgery on the patient's anatomical structure. However, fluoroscopic images may not capture high-quality images of certain types of tissues. Systems and methods are needed to register real-time fluoroscopic images with previous-time static images to provide enhanced navigation information for performing image-guided surgery. Summary of the invention

[0006] Embodiments of the invention are summarized by the claims that accompany the description.

[0007] In one embodiment, a method performed by a computing system includes: receiving a fluoroscopic image of a patient's anatomy when a portion of a medical instrument is located in the patient's anatomy. The fluoroscopic image has a fluoroscopic reference frame. The portion may be referred to as an indicator portion, and the portion may include: any length of the medical instrument, including, by way of non-limiting example, a proximal portion, a middle portion, a distal portion, and / or a distal end of the medical instrument. The portion has a sensed or otherwise known position in an anatomical model reference frame. The method further includes: identifying the position of the portion in the fluoroscopic image, and using the position of the portion in the fluoroscopic image, identifying an extracted position of the portion in the fluoroscopic reference frame. The method further includes: aligning the fluoroscopic reference frame to the anatomical model reference frame based on the sensed position of the portion and the extracted position of the portion.

[0008] In another embodiment, a method performed by a computing system includes: identifying a set of locations of a plurality of anatomical landmarks rendered in an anatomical model of a bore of a patient's anatomical structure in a model reference frame, and receiving fluoroscopic image data of the patient's anatomical structure as a portion of a medical instrument passes through the plurality of anatomical landmarks in the bore of the patient's anatomical structure. The fluoroscopic image data has a fluoroscopic reference frame. The method further includes: identifying a set of partial locations at the plurality of anatomical landmarks in the fluoroscopic reference frame, and registering the set of locations of the plurality of anatomical landmarks in the model reference frame and the set of partial locations in the fluoroscopic reference frame to a common reference frame.

[0009] In another embodiment, a method performed by a computing system includes: receiving a set of model points of an anatomical model of an orifice of a patient's anatomy in a model reference frame, and receiving fluoroscopic image data of the patient's anatomy as an indicator portion of a medical instrument passes through the orifice of the patient's anatomy. The fluoroscopic image data has a fluoroscopic reference frame. The method further includes: identifying a set of indicator portion locator points in the fluoroscopic reference frame from the fluoroscopic image data, and matching each indicator portion locator point to a model point in the set of model points to generate a set of matches. The method further includes: registering the model reference frame to the fluoroscopic reference frame based on the set of matches.

[0010] In another embodiment, a computer-assisted medical system includes: a fluoroscopic imager having an azimuth plane in a surgical coordinate space, and one or more processors. The one or more processors are configured to perform a method, the method comprising: receiving a fluoroscopic image of a patient's anatomy when a portion of a medical instrument is positioned at a location within the patient's anatomy, the fluoroscopic image having an azimuth plane. The method further includes: receiving a command to drive movement of the portion of the medical instrument, constraining actuation of the portion so that the actuated movement of the portion is constrained to the azimuth plane of the fluoroscopic image, and driving movement of the portion with the constrained actuation. On the one hand, the system further includes: receiving sensor information from the medical instrument when driving movement of the portion with the constrained actuation, identifying movement of the portion outside of the azimuth plane of the fluoroscopic image from the received sensor information, and receiving a signal to adjust the portion to the azimuth plane of the fluoroscopic image.

[0011] In another embodiment, a method performed by a computing system includes receiving an anatomical model of a patient's anatomy, wherein a region of interest is identified in the anatomical model. The method further includes receiving shape sensor data from an instrument shape sensor of an instrument positioned within the patient's anatomy and registered to the anatomical model, and determining a fluoroscopic image plane for display based on the received shape sensor data and the region of interest.

[0012] It should be understood that both the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this respect, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various aspects of the present disclosure are best understood when the following detailed description is read in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased to facilitate clear discussion. In addition, the present disclosure may repeatedly reference numbers and / or letters in various examples. This repetition is for simplicity and clarity and does not in itself dictate the relationship between the various embodiments and / or configurations discussed.

[0014] Figure 1 A remotely operated medical system according to an embodiment of the present disclosure.

[0015] Figure 2A A medical instrument system utilizing aspects of the present disclosure is described.

[0016] Figure 2B A method of treating a patient with an expanded medical tool according to an embodiment of the present disclosure is described. Figure 2A The remote end of the medical instrument system.

[0017] Figure 3 Description of the location in the human lungs Figure 2A The remote end of the medical instrument system.

[0018] Figure 4 is a flow chart illustrating a method for providing guidance in image-guided surgery according to one embodiment of the present disclosure.

[0019] Figure 5 is a side view of a surgical coordinate space according to an embodiment of the present disclosure, the surgical coordinate space including a medical instrument and a fluoroscopic imaging system.

[0020] Figure 6 A flow chart of an image-guided surgical procedure according to one embodiment of the present disclosure is illustrated.

[0021] Figure 7 A flow chart of an image-guided surgical procedure according to another embodiment of the present disclosure is illustrated.

[0022] Figure 8 A flow chart illustrating a portion of an image-guided surgical procedure according to another embodiment of the present disclosure.

[0023] Fig.9A , Fig. 9B and Fig. 9C Illustration of a fluoroscopic image used to determine positioning of instrument portions in a fluoroscopic reference frame.

[0024] Fig.10 is a display device according to one embodiment of the present disclosure on which a registered side-by-side fluoroscopic model image and an anatomical model image are displayed.

[0025] Fig.11 A display device according to an embodiment of the present disclosure is provided on which an anatomical model image that is registered and overlaid on a fluoroscopic image is displayed.

[0026] Fig.12 A correlation table is illustrated that describes the correlation of anatomical landmarks between the fluoroscopic reference frame and the anatomical reference frame.

[0027] Fig.13A and Fig. 13B Segmentation of fluoroscopic images is described.

[0028] Fig.14 A flow chart illustrating a portion of an image-guided surgical procedure according to another embodiment of the present disclosure.

[0029] Fig.15 A method of determining a preferred fluoroscopic plane of view is described.

[0030] Fig.16 A method for driving a medical instrument under two-dimensional fluoroscopic guidance is described. DETAILED DESCRIPTION

[0031] In the following detailed description of various aspects of the present invention, many specific details are proposed to provide a comprehensive understanding of the disclosed embodiments. However, it is apparent to those skilled in the art that embodiments of the present invention can be implemented without these specific details. In other instances, well-known methods, processes, components and circuits are not described in detail to avoid unnecessary obscuration of various aspects of the embodiments of the present invention. And, in order to avoid unnecessary descriptive repetition, one or more components described according to an illustrative embodiment may be used or omitted from other illustrative embodiments, where applicable.

[0032] The following embodiments will describe various instruments and parts of instruments according to their state in three-dimensional space. As used herein, the term "positioning" refers to the positioning of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the positioning of an object or a portion of an object in at least one translational degree of freedom and the orientation of the object or a portion of an object in at least one rotational degree of freedom (up to a total of six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0033] With reference to the attached drawings Figure 1 For example, a teleoperated medical system for use in a surgical, diagnostic, therapeutic, or biopsy procedure is generally referred to as teleoperated medical system 100. Figure 1 As shown, the teleoperation system 100 generally includes: a teleoperation manipulator assembly 102 for operating a medical instrument system 104 when performing various operations on a patient P. The teleoperation manipulator assembly 102 may also be referred to as a teleoperation assembly 102 or a manipulator assembly 102. The medical instrument system 104 may also be referred to as a medical instrument 104. The manipulator assembly 102 is mounted on or near an operating table O. An operator input system 106 (also referred to as a "main assembly 106") allows a clinician or surgeon S to view the intervention site and control the manipulator assembly 102.

[0034] The operator input system 106 can be located at a surgeon's console, which is typically located in the same room as the operating table O. However, it should be understood that the surgeon S can be located in a different room from the patient P or in a completely different building. The operator input assembly 106 typically includes one or more devices for controlling one or more manipulator assemblies 102. The control device can include any number of various input devices, such as a joystick, a trackball, a data glove, a trigger gun, a manual controller, a voice recognition device, a body motion or presence sensor, and the like. In some embodiments, the control device will be provided with the same degrees of freedom as the associated medical instrument 104 to provide the surgeon with a remote presence or a sense that the control device is integrated with the instrument 104, so that the surgeon has a strong sense of directly controlling the instrument 104. In other embodiments, the control device can have more or fewer degrees of freedom than the associated medical instrument 104 and still provide the surgeon with a remote presence. In some embodiments, the control device is a manual input device that moves with six degrees of freedom, and it can also include: an actuation handle for actuating the instrument (e.g., for closing the grasping jaws, applying an electric potential to the electrode, delivering a drug treatment, etc.).

[0035] The teleoperation assembly 102 supports the medical instrument system 104, and the teleoperation assembly 102 may include a kinematic structure of one or more non-servo-controlled linkages (e.g., one or more linkages that can be manually positioned and locked in place, often referred to as set structures) and a teleoperation manipulator. The teleoperation assembly 102 includes a plurality of actuators or motors that drive inputs on the medical instrument system 104 in response to commands from a control system (e.g., control system 112). The motors include: a drive system that, when coupled to the medical instrument system 104, can advance the medical instrument into a naturally or surgically created anatomical hole. Other motorized drive systems can move the distal end of the medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). In addition, the motor can be used to actuate an articulated end effector of the instrument for grasping tissue in the jaws of a biopsy device, etc. Motor position sensors, such as resolvers, encoders, potentiometers, and other mechanical mechanisms, can provide sensor data describing the rotation and orientation of the motor shaft to the remote operation assembly. This position sensor data can be used to determine the motion of an object operated by the motor.

[0036] The teleoperated medical system 100 also includes a sensor system 108 having one or more subsystems for receiving information about instruments of the teleoperated assembly. Such subsystems may include: a positioning / position sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, speed, velocity, posture, and / or shape of the catheter tip and / or one or more segments along the flexible body of the medical instrument system 104; and / or a visualization system for capturing images from the distal end of the catheter system.

[0037] Visualization systems (e.g. Figure 2A The visualization system 231) may include a viewing scope component that records concurrent or real-time images of the surgical site and provides the images to a clinician or surgeon S. For example, the concurrent images may be two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In this embodiment, the visualization system includes an endoscope assembly that can be integrally or removably coupled to the medical instrument 104. However, in alternative embodiments, a separate endoscope attached to a separate manipulator assembly can be used with the medical instrument to image the surgical site. The visualization system can be implemented as hardware, firmware, software, or a combination thereof, which interacts with or is otherwise executed by one or more computer processors, which may include a processor of the control system 112 (described below). The processor of the control system 112 can execute instructions that include instructions corresponding to the processing disclosed herein.

[0038] The teleoperated medical system 100 also includes a display system 110 (also referred to as "display 110") for displaying images or representations of the surgical site and the medical instrument system(s) 104 generated by the subsystems of the sensor system 108. The display system 110 and the operator input system 106 may be oriented so that the operator can control the medical instrument system 104 and the operator input system 106 with a sense of telepresence.

[0039] The display system 110 may also display images of the surgical site and medical instruments captured by the visualization system. The display system 110 and control device may be oriented so that the relative positioning of the imaging device and the medical instrument in the scope assembly is similar to the relative positioning of the surgeon's eyes and hands, so that the operator can manipulate the medical instrument 104 and the manual controls as if viewing the workspace in a substantially real-life manner. By real-life, it is meant that the presentation of the image is a realistic perspective image that simulates the viewpoint of the operator who is physically operating the instrument 104.

[0040] Alternatively or additionally, the display system 110 can present an image of the surgical site recorded preoperatively or intraoperatively using image data from an imaging technique such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, infrared thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube x-ray imaging. The preoperative or intraoperative image data can be presented as a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or rate-based information) image or as an image of a model created from the preoperative or intraoperative image data set.

[0041] In some embodiments, often for the purpose of imaging-guided surgery, the display system 110 can display a virtual navigation image in which the actual position of the medical instrument 104 is registered (i.e., dynamically referenced) with a preoperative or concurrent image / modality to present the clinician or surgeon S with a virtual image of the internal surgical site from the viewpoint of the position of the tip of the instrument 104. An image of the tip of the instrument 104 or an image of other graphical or alphanumeric indicators can be superimposed on the virtual image to assist the surgeon in controlling the medical instrument. Alternatively, the instrument 104 may not be visible in the virtual image.

[0042] In other embodiments, the display system 110 may display a virtual navigation image in which the actual position of the medical instrument is registered with the preoperative or concurrent image to present a virtual image of the medical instrument within the surgical site from an external viewpoint to the clinician or surgeon S. An image of a portion of the medical instrument or an image of other graphical or alphanumeric indicators may be superimposed on the virtual image to assist the surgeon in controlling the instrument 104.

[0043] The teleoperated medical system 100 also includes a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown), and typically multiple processors, for causing control between the medical instrument system 104, the operator input system 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., a computer readable medium storing instructions) to implement some or all of the methods described according to aspects disclosed herein, including instructions for providing the registered image to the display system 110. Although the control system 112 is Figure 1A simplified schematic diagram of the system 112 is shown as a single block, and the system may include two or more data processing circuits, wherein a portion of the processing is optionally performed on or adjacent to the remote operation component 102, another portion of the processing is performed at the operator input system 106, and so on. Any of a variety of centralized or distributed data processing architectures can be used. Similarly, the programming instructions can be implemented as several separate programs or subroutines, or they can be integrated into several other aspects of the remote operation system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, infrared data organization IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0044] In some embodiments, the control system 112 may include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 104. In response to the feedback, the servo controller sends a signal to the operator input system 106. The (one or more) servo controllers may also send signals instructing the remote operation assembly 102 to move the (one or more) medical instrument system 104, which (one or more) medical instrument system 104 extends through the opening in the body to the internal surgical site in the patient's body. Any suitable conventional or dedicated servo controller can be used. The servo controller can be separate from the remote operation assembly 102 or integrated with the remote operation assembly 102. In some embodiments, the servo controller and the remote operation assembly are provided as part of a cart positioned as a remote operation arm adjacent to the patient's body.

[0045] The control system 112 may further include a virtual visualization system to provide navigation assistance to the (one or more) medical instrument systems 104 when used in image-guided surgery. Virtual navigation using the virtual visualization system is based on reference to a preoperative or intraoperative data set of an acquired anatomical orifice. More specifically, the virtual visualization system processes images of an imaged surgical site, which are imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, infrared thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and the like. Software, alone or in combination with manual input, is used to convert the recorded images into a two-dimensional or three-dimensional composite representation of a segmentation of a portion or entire anatomical organ or anatomical region. The image data set is associated with the composite representation. The composite representation and the image data set describe different locations and shapes of the orifice and its connections. The images used to generate the composite representation may be recorded at a previous time preoperatively or intraoperatively during a clinical procedure. In an alternative embodiment, the virtual visualization system may use a standard representation (i.e., non-patient-specific) or a mixture of a standard representation and patient-specific data. The composite representation and any virtual images generated by the composite representation may represent the static pose of the deformable anatomical region during one or more phases of motion (eg, during the inspiration / expiration cycle of the lungs).

[0046] During virtual navigation surgery, the sensor system 108 can be used to calculate the approximate position of the instrument relative to the patient's anatomy. The position can be used to generate both a macro-level (external) tracking image of the patient's anatomy and a virtual internal image of the patient's anatomy. Various systems for using electromagnetic (EM) sensors, fiber optic sensors, or other sensors to register and display medical instruments together with preoperatively recorded surgical images (such as images from a virtual visualization system) are well known. For example, U.S. Patent Application No. 13 / 107,562 (filed on May 13, 2011) (disclosing "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery (Providing a Medical System for Image-Guided Surgery that provides dynamic registration of a model of an anatomical structure)" (which is incorporated herein by reference in its entirety) discloses such a system.

[0047] The remotely operated medical system 100 may further include: optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems and / or suction systems. In alternative embodiments, the remote operation system may include more than one remote operation component and / or more than one operator input system. The exact number of manipulator components will depend on the space limitations within the surgical and operating rooms, as well as other factors. The operator input systems may be juxtaposed, or they may be located in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator components in a variety of combinations.

[0048] Figure 2A A medical instrument system 200 system (also referred to as "medical instrument 200" or "instrument system 200") is illustrated that can be used as the medical instrument system 104 in an image-guided medical procedure performed with a teleoperated medical system 100. Alternatively, the medical instrument system 200 can be used in a non-teleoperated exploratory procedure or in a procedure involving a traditional manually operated medical instrument (such as an endoscope). Additionally or alternatively, the medical instrument system 200 can be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical orifice of a patient.

[0049] The instrument system 200 includes a catheter system 202 (also referred to as "catheter 202") coupled to an instrument body 204, which may be referred to as a "housing 204" when used to house components. The catheter system 202 includes an elongated flexible catheter body 216 having a proximal end 217 and a distal end 218 (which may be referred to as a "tip portion 218" when it is the tip portion of the catheter body 216). In one embodiment, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible body outer diameters may be larger or smaller. The catheter system 202 may optionally include a shape sensor system 222 (also referred to as a "shape sensor 222") for determining the location, orientation, speed, velocity, posture, and / or shape of the catheter tip at the distal end 218 and / or one or more segments 224 along the catheter body 216. The entire length of the catheter body 216 between the distal end 218 and the proximal end 217 may be effectively divided into segments 224. If the instrument system 200 is a medical instrument 104 of a teleoperated medical system 100, the shape sensor system 222 may be a component of the sensor system 108. If the instrument system 200 is manually operated or otherwise used for a non-teleoperated procedure, the shape sensor system 222 may be coupled to a tracking system 230 that interrogates the shape sensors and processes the received shape data.

[0050] The shape sensor system 222 may include an optical fiber aligned with the flexible catheter body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor system 222 forms a fiber optic bend sensor for determining the shape of the catheter system 202. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurement in a structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in U.S. Patent Application No. 11 / 180,389 (filed on July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed on July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"); and U.S. Patent No. 6,389,187 (filed on June 17, 1998) (disclosing "Optical Fiber Bend Sensor"), all of which are incorporated herein by reference in their entirety. Sensors in alternative embodiments may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In other alternative embodiments, other techniques may be used to determine the shape of the catheter. For example, a history of the distal tip pose of the catheter may be used to reconstruct the shape of the device over time intervals. As another example, historical pose, positioning or orientation data can be stored for known points of the instrument system along a cycle of alternating motion (such as breathing). The stored data can be used to develop shape information about the catheter. Alternatively, a series of positioning sensors (such as electromagnetic (EM) sensors) positioned along the catheter can be used for shape sensing. Alternatively, the data history from positioning sensors (such as EM sensors) on the instrument system during surgery can be used to represent the shape of the instrument, especially if the anatomical orifice is generally static. Alternatively, a wireless device with positioning or orientation controlled by an external magnetic field can be used for shape sensing. The history of the positioning of the wireless device can be used to determine the shape of the orifice for navigation.

[0051] The medical instrument system may optionally include a positioning sensor system 220. The positioning sensor system 220 may be a component of an EM sensor system having a sensor system 220, the EM sensor system including one or more conductive coils that may be subjected to an externally generated electromagnetic field. In such an embodiment, each coil of the EM sensor system constitutes a positioning sensor system 220 that generates an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. In one embodiment, the EM sensor system may be configured and positioned to measure six degrees of freedom, for example, three positioning coordinates X, Y, Z and three azimuth angles that indicate pitch, yaw, and roll of a base point or five degrees of freedom, for example, three positioning coordinates X, Y, Z and two azimuth angles that indicate pitch and yaw of a base point. A further description of the EM sensor system is provided in U.S. Pat. No. 6,380,732 (filed Aug. 11, 1999) (disclosing "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety. In some embodiments, the shape sensor can also serve as a positioning sensor because the shape of the sensor, together with information about the position of the base of the shape sensor (in the patient's fixed coordinate system), allows the position of various points along the shape sensor (including the distal tip) to be calculated.

[0052] Tracking system 230 may include position sensor system 220 and shape sensor system 222 for determining the position, orientation, velocity, pose, and / or shape of distal end 218 and one or more segments 224 along instrument system 200. Tracking system 230 may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include a processor of control system 116.

[0053] The flexible catheter body 216 includes a channel 221 sized and shaped to receive a medical instrument 226 (see Figure 2B ). Medical instruments may include, for example, image capture probes, biopsy instruments, laser ablation fibers, or other surgical, diagnostic, or therapeutic tools. Medical tools may include end effectors having a single working part, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clamps. Examples of electrically activated end effectors include electrosurgical electrodes, transducers, sensors, and the like.

[0054] In various embodiments, the (one or more) medical instruments 226 can be an image capture probe, which includes a remote portion having a stereo or planar camera at or near the distal end 218 of the flexible catheter body 216 for capturing images (including video images), which are processed by the visualization system 231 for display. The image capture probe can include a cable that is connected to the camera for transmitting captured image data. Alternatively, the image capture instrument can be a fiber optic bundle, such as a fiber endoscope, which is connected to the visualization system. The image capture instrument can be single or multi-spectral, for example, capturing image data in one or more of the visible, infrared or ultraviolet spectrum.

[0055] In various embodiments, the medical instrument 226 is a biopsy instrument for removing a sample of tissue or cells from a target anatomical location. The instrument 226 can be advanced from the opening of the channel 221 to perform the procedure and then retracted into the channel when the procedure is completed. The medical instrument 226 can be removed from the proximal end 217 of the catheter flexible body or from another optional instrument port (not shown) along the flexible body.

[0056] The medical instrument 226 may house cables, linkages, or other actuation controls (not shown) extending between the proximal and distal ends of the instrument to controllably bend the distal end of the instrument. Steerable instruments are described in detail in U.S. Patent No. 7,316,681 (filed on October 4, 2005) (disclosing "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application No. 12 / 286,644 (filed on September 30, 2008) (disclosing "Passive Preload and Capstan Drive for Surgical Instruments"), all of which are incorporated herein by reference.

[0057] The flexible catheter body 216 may also house a cable, linkage or other steering control (not shown) extending between the housing 204 and the distal end 218 to controllably bend the distal end 218 (e.g., as shown by the dashed line depiction 219 of the distal end). Steerable catheters are described in detail in U.S. Patent Application No. 13 / 274,208 (filed on October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. In embodiments where the instrument system 200 is actuated by a remotely operated assembly, the housing 204 may include a drive input that is removably coupled to the remotely operated assembly and receives power from a motorized drive element of the remotely operated assembly. In embodiments where the instrument system 200 is manually operated, the housing 204 may include a gripping feature, a manual actuator or other component for manually controlling the movement of the instrument system. The catheter system may be steerable or alternatively, the system may be non-steerable, wherein there is no integrated mechanism for operator control of instrument bending. Additionally or alternatively, one or more lumens are defined in the wall of the flexible body 216 through which medical instruments may be deployed and used at the target surgical site.

[0058] In various embodiments, the medical instrument system 200 may include a flexible bronchial instrument (such as a bronchoscope or bronchial catheter) for inspection, diagnosis, biopsy, or treatment of the lungs. The instrument system 200 is also suitable for navigating and treating other tissues through natural or surgically created connected orifices in any of a variety of anatomical systems (including the colon, intestines, kidneys, brain, heart, circulatory system, etc.).

[0059] Information from the tracking system 230 may be sent to the navigation system 232 where it is combined with information from the visualization system 231 and / or preoperatively acquired models to provide real-time positioning information about the display system 110 to the surgeon or other operator for control of the instrument system 200. The control system 116 may utilize the positioning information as feedback for positioning the instrument system 200. Various systems for using fiber optic sensors to register and display surgical instruments using surgical images are provided in U.S. Patent Application No. 13 / 107,562 filed May 13, 2011, which discloses “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery,” which is incorporated herein by reference in its entirety.

[0060] exist Figure 2A In the embodiment of the present invention, the instrument system 200 is remotely operated within the teleoperation medical system 100. In an alternative embodiment, the remote operation component 102 can be replaced by direct operator control. In the alternative mode of direct operation, various processing and operator interfaces can be included for handheld operation of the instrument.

[0061] In alternative embodiments, the teleoperation system may include more than one slave manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies will depend on the medical procedure and the space constraints within the operating room, among other factors. The master assemblies may be juxtaposed, or they may be positioned in separate locations. Multiple master assemblies allow more than one operator to control one or more slave manipulator assemblies in various combinations.

[0062] Figure 3 The catheter system 202 is illustrated positioned within an anatomical orifice of a patient's anatomy. In this embodiment, the anatomical orifice is an airway of a human lung 201. In alternative embodiments, the catheter system 202 may be used in orifices of other anatomical structures.

[0063] Figure 4 is a flow chart illustrating a general method 300 for use in image-guided surgery. Although various provided examples describe surgical applications performed within an anatomical structure, in alternative embodiments, the apparatus and methods of the present disclosure need not be used within an anatomical structure, but may also be used outside of a patient's anatomy. At process 302, prior image data, including preoperative or intraoperative image data, is obtained from an imaging technique such as computed tomography (CT), magnetic resonance imaging (MRI), thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube x-ray imaging. The preoperative or intraoperative image data may correspond to a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or rate-based information) image. For example, the image data may represent Figure 3 of the human lung201.

[0064] At process 304, computer software, alone or in combination with manual input, is used to convert the recorded image into a two-dimensional or three-dimensional composite representation of the segmentation or a model of a part or the entire anatomical organ or anatomical region. The composite representation and image data set describe the various positions and shapes of the duct and its connection. More specifically, during the segmentation process, the image is divided into segments or elements (e.g., pixels or voxels) that share certain characteristics or computational properties, such as color, density, intensity, and texture. This segmentation process results in a two-dimensional or three-dimensional reconstruction that forms a model and an anatomical model of the target anatomical structure based on the image obtained. In order to represent the model, the segmentation process can outline multiple groups of voxels representing the target anatomical structure, and then apply (such as marching cube function) functions to generate a 3D surface surrounding the voxels. The model can be made by generating a grid, volume, or voxel map. Additionally or alternatively, the model may include a centerline model that includes a group of interconnected line segments or points extending through the center of the modeled duct. When the model includes a centerline model that includes a group of interconnected line segments, those line segments can be converted into a group or a group of points. By converting the line segments, a desired number of points corresponding to the interconnected line segments may be selected manually or automatically.

[0065] At process 306, before and / or during the process of performing image-guided surgery on a patient, an anatomical model, a medical instrument (e.g., instrument system 200) used to perform a medical operation, and a patient anatomical structure are co-registered in a common reference frame. The common reference frame can be, for example, a surgical environment reference frame or a patient reference frame. Process 306 includes positioning the medical instrument with respect to the patient. Process 306 also includes registering the anatomical model with respect to the patient. In general, registration involves matching the measured points to the points of the model using rigid and / or non-rigid transformations. The measured points can be generated using landmarks in the anatomical structure, electromagnetic coils or shape sensor systems scanned and tracked during surgery. The measured points can be generated for use in an iterative closest point (ICP) technique. ICP and other registration techniques are described in U.S. Provisional Patent Application No. 62 / 205,440 and U.S. Provisional Patent Application No. 62 / 205,433, both of which are filed on August 14, 2015, and are incorporated herein by reference in their entirety. At process 308 , the medical procedure may be performed using the anatomical model data to guide movement of the medical instrument.

[0066] Anatomical models created using previous image data (e.g., CT scans (i.e., computerized tomography)) are used for image-guided surgery to provide fine anatomical details suitable for many surgeries. However, models based on previous image data suffer from registration errors and do not account for the real-time configuration of the anatomical structure, including any deformation caused by periodic or non-periodic anatomical movement, the presence of medical instruments and tissue deformation caused by medical instruments, or other changes in the patient's anatomical structure that may have occurred since the previous image data was obtained. Fluoroscopy is a fluoroscopic imaging method that uses X-rays to obtain real-time moving images of the patient's anatomical structure. Traditional radiographic images are X-ray images obtained by placing a portion of the patient in front of an X-ray detector and then irradiating it with short X-ray pulses. In a similar manner, fluoroscopy uses X-rays to obtain real-time moving images of the patient's interior, including radiopaque medical instruments, radiopaque dyes, and / or radiopaque fiducial markers within the surgical environment. Fluoroscopy systems may include C-arm systems that provide positioning flexibility through manual or automatic control and are capable of orbital, horizontal, and / or vertical movement. Non-C-arm systems are stationary and provide poor flexibility in movement. Fluoroscopy systems typically use image intensifiers or flat panel detectors to generate two-dimensional real-time images of the patient's anatomical structure. Dual-plane fluoroscopy systems capture two fluoroscopic images simultaneously, each captured from a different (usually orthogonal) viewpoint. The quality and practicality of X-ray images may vary depending on the type of imaging tissue. Denser materials (such as bones and metals) are usually more visible in X-ray images than the soft tissue filled with gas in the lungs. For surgery in the lungs, CT models provide anatomical details of airways and tumors that may be difficult to discern on fluoroscopic images, but fluoroscopic images provide real-time visualization of medical instruments and dense anatomical tissues. Therefore, fluoroscopic images registered with anatomical models may be useful for clinicians to navigate certain anatomical parts (such as the lungs).

[0067] Figure 5An example surgical environment 350 is illustrated according to some embodiments, which has a surgical coordinate system Xs, Ys, Zs, wherein a patient P is positioned on a platform 352. The patient P may be stationary in the surgical environment in the sense that total patient movement is limited by sedation, restraint, or other means. Unless the patient temporarily stops respiratory movement, periodic anatomical movement including respiratory and cardiac movement of the patient P may continue. Within the surgical environment 350, a medical instrument 354 is coupled to an instrument bracket 356. The instrument bracket 356 is mounted to an insertion table 358 that is fixed or movable within the surgical environment 350. The instrument bracket 356 may be a component of a remotely operated manipulator assembly (e.g., manipulator assembly 102) that is coupled to the instrument 354 to control insertion movement (i.e., movement in the Xs direction), and optionally, movement of the distal end of the instrument in multiple directions including yaw, pitch, and roll. The instrument bracket 356 or insertion table 358 may include a servo motor (not shown) that controls movement of the instrument bracket along the insertion table. The medical instrument 354 may include a flexible catheter 360 coupled to a proximal rigid instrument body 362. The rigid instrument body 362 is coupled and fixed relative to the instrument bracket 356. A fiber optic shape sensor 364 extends along the instrument 354 and is operable to measure the shape from a fixed or known point 366 to another point (such as a portion 368 of the catheter 360). In the depicted embodiment, the portion 368 is shown as a distal portion. In other embodiments, the portion 368 may be positioned elsewhere along the length of the catheter 360, including at the middle of the catheter. The portion 368 is shaped and constructed so that it can be used as an indicator portion of the catheter 360 (e.g., a portion 368 that can be identified in the imaging data). The medical instrument 354 may be substantially similar to the medical instrument system 200. A fluoroscopic imaging system 370 (also referred to as a fluoroscopic system 370) is arranged near the patient P to obtain a fluoroscopic image of the patient when the catheter 360 is extended into the patient's body. The system 370 may be, for example, a movable C-arm fluoroscopic imaging system. In some embodiments, system 370 may be a multi-axis Artis Zeego fluoroscopic imaging system from Siemens Corporation of Washington.

[0068] Figure 6 A flow chart of a method 450 for performing image-guided surgery in a surgical environment 350 is illustrated. The method described herein, including the method 450, is described in Figure 6450 as a set of blocks, steps, operations or processes. Not all of the operations described and listed can be performed in all embodiments of method 450. In addition, some additional operations not explicitly described in these methods may be included before, after, between or as part of the listed processes. Some embodiments of this described method include instructions corresponding to the processes of the method stored in a memory. These instructions may be executed by a processor similar to the processor of control system 112.

[0069] Thus, some embodiments of method 450 may begin at process 452, where prior image data, including preoperative or intraoperative image data, is obtained from an imaging technique such as CT, MRI, thermography, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The prior image data may correspond to a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or rate-based information) image. As described above, an anatomical model is created in an anatomical model reference frame based on the prior image data. At process 454, the instrument sensor model reference frame (X-ray) is used to generate an anatomical model. I , Y I , Z I ), which may be associated with a shape sensor 222 included in the medical instrument 200, is registered to the anatomical model reference frame (X M , Y M , Z M ). Such registration between the model reference frame and the instrument reference frame can be achieved, for example, using the point-based ICP techniques described in U.S. Provisional Patent Application Nos. 62 / 205,440 and 62 / 205,433, which are incorporated herein by reference. Alternatively, the model reference frame can be registered to the sensor reference frame or the model reference frame and the sensor reference frame can be registered to another common reference frame. The common reference frame can be, for example, a surgical environment reference frame (Xs, Ys, Zs) or a patient reference frame.

[0070] At process 456 and with reference to Fig. 9B, the sensor data (compared with the shape information determined from the imaging) is used to calculate the pose information. For example, in various instances, the sensed position and orientation of a portion of the medical instrument at a position L within, near, adjacent to, or outside the patient's anatomical structure is used to determine the pose information of the medical instrument. The sensed position and orientation can be obtained, measured, or calculated from the sensor data. In this embodiment, the portion 368 of the medical instrument can be referred to as the indicator portion and can include a radiopaque distal portion. In other embodiments, other radiopaque portions of the medical instrument (visible on the fluoroscopic image) can be the indicator portion. The indicator portion can include any length of the medical instrument that is visible on the fluoroscopic image. The sensor information is received from the fiber optic shape sensor and is used to calculate the sensed position and orientation of the indicator portion 368 in the sensor reference frame. Based on the registration of the instrument sensor reference frame to the anatomical model reference frame (or other common reference frame) at process 454, the position and / or orientation (i.e., the sensed pose) of the indicator portion in the sensor frame is converted into the anatomical model reference frame at process 458.

[0071] At process 460, the fluoroscopic system 370 captures fluoroscopic image data of the patient P and the catheter 360 extended into the patient. When the indicator portion 368 is positioned at or in contact with the anatomical location L, one or more fluoroscopic images rendered according to the fluoroscopic image data are obtained. When the indicator portion 368 is at location L, the images obtained may be from a single viewpoint or may be a multi-plane image set (including a set of bi-plane images) that displays the indicator portion 368 from multiple viewpoints at the same time and in the same location. In some examples, the catheter 360 may be positioned outside of the patient so that the indicator portion 368 is visible adjacent to the anatomical location L within one or more fluoroscopic images.

[0072] At process 462, the indicator portion 368 is identified in the fluoroscopic image data, and thus, the location of the position L in the fluoroscopic reference frame is also identified. In various embodiments, the fluoroscopic system 370 is capable of generating multi-planar images, thus providing a three-dimensional fluoroscopic reference frame (X F , Y F , Z F ). Using multiple images also allows for determination of the orientation of the indicator portion 368, thus providing pose information for the indicator portion. In other embodiments, the fluoroscopic system may provide a two-dimensional fluoroscopic reference frame.

[0073] To extract the location of the indicator portion 368 in the fluoroscopic image and thus the indicated position L in the fluoroscopic reference frame, various techniques may be used. Fig.9A and Fig. 9B , the indicator portion 368 in the image data can be identified by comparing the image 600 in which the indicator portion 368 is not present and the image 610 in which the indicator portion 368 is present. For example, the two images are analyzed by a computer of the control system 112 to determine the graphical differences between the two images. Because the patient anatomical structure portion of the image is the same in the two images, the catheter 360 including the indicator portion 368 can be identified as a structure unique to the image 610. By identifying the graphical components (e.g., pixels, voxels) associated with the identified catheter 360, the presence and position of the indicator portion 368 in the fluoroscopic frame can be calculated. Because the indicator portion 368 is located at position L, the location of position L is also determined by extracting the graphical analysis of the indicator portion.

[0074] In another embodiment, reference Fig. 9B and Fig. 9C , by comparing an image 620 in which the indicator portion 368 is located at position K and an image 610 in which the indicator portion 368 is located at position L, the indicator portion 368 in the image data can be identified. For example, the two images are analyzed by a computer of the control system 112 to determine the graphical differences between the two images. Because the patient anatomical structure portion of the image is the same in the two images, the catheter 360 including the indicator portion 368 can be identified as a unique structure in each image 610, 620. By identifying the graphical component associated with the distal end of the identified catheter 360, the presence and location of the indicator portion 368 in the fluoroscopic frame of each image can be calculated. Because the indicator portion 368 is located at positions L and K, the location of positions L and K is also determined by extracting the graphical analysis of the indicator portion.

[0075] In another embodiment, reference Fig.13A , the indicator portion 368 can be identified in the image data by a semi-automatic extraction technique in which a fluoroscopic image 1000 including an instrument 1002 is displayed to a user. The user uses an input device, such as a touch screen, a mouse, a tracking ball, or eye gaze to create an outline 1004 of the instrument 1002 or a portion of the instrument (such as a distal tip). Based on the outline 1004 input received from the user, a segmentation algorithm identifies features (e.g., pixel shading, size) of the instrument 1002 in the image 1000 and proceeds to divide the image into segments (e.g., pixels or voxels) corresponding to the instrument.

[0076] In another embodiment, referring to Fig. 13B, the indicator portion 368 can be identified in the image data by a semi-automatic extraction technique in which a fluoroscopic image 1010 including the instrument 1022 is displayed to the user. A search area 1014 is identified by the user, or can be pre-defined (e.g., the lower left quadrant of the image) to search for an entry point 1016 of the medical instrument into the image. The segmentation algorithm can search for a predetermined shape associated with an expected shape of the medical instrument (e.g., expected continuity or saturation of shading, expected width, expected length, expected curvature). After the segmentation algorithm identifies the medical instrument in the image 1010, the algorithm proceeds to divide the image into segments (e.g., pixels or voxels) corresponding to the instrument.

[0077] In another embodiment, the desired shape is defined by shape information received from a shape sensor, and the desired shape is used to identify an initial search area for use by a semi-automatic extraction technique. The segmentation algorithm can further search for the desired shape in the initial search area to reduce time and computing resources. After the segmentation algorithm identifies the desired shape, the algorithm proceeds to divide the image into segments (e.g., pixels or voxels) corresponding to the instrument.

[0078] In another embodiment, the indicator portions 368 can be identified by graphically analyzing and identifying the captured fluoroscopic images for predetermined (i.e., expected or previously known) radiopaque shapes (such as shapes measured by a shape sensor or by a predetermined (i.e., expected or previously known) shape or marker at the end of the catheter). For example, the known shape of a needle extending from a catheter or a fixture at the end of the catheter can be found and extracted from the image. Graphical identification of these indicator portions in the fluoroscopic images provides localization of the location L of the contact in the fluoroscopic frame.

[0079] Refer again Figure 6 , at process 464, the fluoroscopic image reference frame is registered to the anatomical model reference frame. Alternatively, both the fluoroscopic image reference frame and the anatomical model reference frame are registered to a common reference frame, such as a surgical reference frame. Because process 462 provides the location and orientation of the indicator portion 368 at position L in the fluoroscopic image reference frame, and process 458 provides the location and orientation of the indicator portion 368 at position L in the model reference frame, the corresponding frame locations of the indicator portion 368 are associated to register the frames together. Processes 456, 458, 460, 462 can be repeated for multiple locations of the indicator portion at multiple locations in the patient's anatomy. By associating these multiple locations in the corresponding frames and by performing rigid or non-rigid transformations of points corresponding to the multiple locations, the registration at process 464 can be performed or enhanced.

[0080] At process 466 and with reference to Fig.10 and Fig.11 , a registered reference frame is displayed as the catheter traverses the patient's anatomy, allowing the clinician viewing (one or more) display images to take advantage of the benefits of real-time instrument tracking in a fluoroscopic image with anatomical detail from a prior time image (e.g., a CT image). Fig.10 A display 700 is illustrated which displays a fluoroscopic reference frame (X F , Y F , Z F ) and a fluoroscopic image 710 with a model reference frame (X M , Y M , Z M ) of a prior time model image 720 (such as, by way of non-limiting example, a CT image captured by CT technology or an image captured by any other suitable imaging technology). The fluoroscopic reference frame and the model reference frame have been registered and the registered images are displayed side by side. Using the registered reference frames, structures from one image can be optionally superimposed or overlaid on the other image to assist the clinician in performing a medical procedure. For example, a catheter 360 visible in the fluoroscopic image 710 can be extracted and overlaid on the image 720. Additionally or alternatively, a target tissue 730 (e.g., a tumor) visible in the image 720 can be extracted and overlaid on the fluoroscopic image 720.

[0081] Fig.11 A display 800 is illustrated showing a single image 810 in which prior image data 820 has been overlaid on fluoroscopic image data 830. In some embodiments (not shown), a path to the target tissue may be planned within the prior time model image data and displayed within the prior time model image (e.g., a CT image). The path may then be extracted and overlaid on the fluoroscopic image data.

[0082] Optionally, after the fluoroscopic image data is registered to the anatomical model, key features can be viewed and analyzed from a two-dimensional or three-dimensional image generated from the fluoroscopic image data, the anatomical model, or a combination of both. For example, a shadow in the fluoroscopic image data can indicate a tumor or can indicate where to look for a tumor in the segmented CT data used to generate the anatomical model.

[0083] Optionally, after the fluoroscopic image data is registered to the anatomical model, a virtual fluoroscopic view can be generated. Because the registration indicates the fluoroscopic image plane relative to the model, a virtual view of the patient's anatomy can be generated from another plane of view. For example, a virtual view orthogonal to the plane of the actual fluoroscopic image can be generated based on the anatomical model to provide the clinician with an experience similar to a bi-plane fluoroscopic image.

[0084] Additionally or alternatively, with the fluoroscopic view registered to the anatomical model, anatomical features (such as target tissue, tumor, or other landmarks) may be identified in the fluoroscopic image and may be used to locate corresponding features in the anatomical model. For example, a tumor may be visible in the fluoroscopic view to help identify the corresponding tumor in the anatomical model. Once identified in the anatomical model, the tumor may be marked as a target and a navigation path to the target may be planned in the anatomical model.

[0085] Figure 7 Flowchart illustrating another method 500 for performing image-guided surgery in surgical environment 350. The described method including method 500 is described as a set of blocks, steps, operations or processes in Figure 7 500. Not all of the illustrated enumerated operations may be performed in all embodiments of method 500. In addition, some additional operations not explicitly described in the method may be included before, after, between, or as part of the enumerated processes. Some embodiments of the described method include instructions corresponding to the processes of the method stored in a memory. These instructions may be executed by a processor similar to the processor of control system 112.

[0086] Method 500 includes the previously described process 452 of obtaining prior temporal imaging data of the patient's anatomical structure (such as a preoperative CT scan), and the process 454 of registering the instrument sensor reference frame to the anatomical model reference frame. A series of processes 502 describe sub-processes for collecting and matching anatomical landmark points and registering the fluoroscopic reference frame and the model reference frame. At process 508, the fluoroscopic system 370 captures fluoroscopic image data of the patient P and the catheter 360 when the indicator portion 368 contacts or is positioned adjacent to an anatomical landmark visible in the fluoroscopic image generated from the image data. The anatomical landmark can be any unique structure visible in fluoroscopy, such as a visible portion of the spine, ribs, sternum, clavicle, diaphragm, blood vessels, or bronchial tree. At process 510, the indicator portion 368 is identified in the fluoroscopic image data, and the location of the indicated anatomical landmark in the fluoroscopic reference frame is also identified. In various embodiments, the fluoroscopic system 370 is capable of generating multi-planar images, thereby providing a three-dimensional fluoroscopic reference frame (X-ray image). F , YF , Z F ). In other embodiments, the fluoroscopy system may provide a two-dimensional fluoroscopy reference frame. Methods for extracting an indicator portion from fluoroscopy image data are described above. At process 512, the same anatomical landmark is selected in the anatomical model and correlated with the anatomical landmark in the fluoroscopy image data. For example, when the indicator portion of the catheter contacts the anatomical landmark (as seen by the clinician in the real-time fluoroscopy image), the user can touch the touch screen (displaying the anatomical model) at the location of the anatomical landmark. Other marking methods for locating anatomical landmarks in the model may also be used. Thus, the coordinates of the anatomical landmark in the fluoroscopy reference frame and the coordinates of the anatomical landmark in the model reference frame can become correlated. Fig.12 At 900, a correlation table 900 may be compiled that references the location 920 of each landmark 910 into the fluoroscopic reference frame and the location 930 into the model reference frame. Process 502 may be repeated for each of a plurality of anatomical landmarks. For example, three or more anatomical landmarks may be selected.

[0087] At process 514, the fluoroscopic image reference frame is registered to the anatomical model reference frame. Anatomical landmark point registration is described in U.S. Provisional Patent Application Nos. 62 / 205,440 and 62 / 205,433, which are incorporated by reference. For example, a set of fluoroscopic image anatomical reference points are matched to anatomical landmark points in the model. One or both sets of matched points are then rotated, translated, or otherwise manipulated via rigid or non-rigid transformations to register the fluoroscopic reference frame and the model reference frame. At process 516 and with reference to Fig.10 and Fig.11 , a registered reference frame is displayed as the catheter passes through the patient anatomy, allowing the clinician viewing the display image(s) to take advantage of the benefit of real-time tracking of the instrument in the fluoroscopic image with the anatomical detail of a prior time image (e.g., a CT image).

[0088] Figure 8 1 illustrates a flow chart of a subprocess 550 that can be used to replace process 502 in method 500. At process 552, the fluoroscopic system 370 captures fluoroscopic image data of the patient P and the catheter 360 as the indicator portion 368 passes through the patient's anatomical passage. At process 554, multiple location points of the indicator portion 368 are identified in the fluoroscopic image data, and thus the location of the location points in the fluoroscopic reference frame is also identified. In this embodiment, the fluoroscopic system 370 is capable of generating a multi-plane image therefrom, thereby providing a three-dimensional fluoroscopic reference frame (X-ray). F , Y F , Z F). In other embodiments, the fluoroscopic system can provide a two-dimensional fluoroscopic reference frame. The method for extracting the indicator portion from the fluoroscopic image data is described above. At process 556, the fluoroscopic image reference frame is aligned to the anatomical model reference frame. Point cloud registration is described in U.S. Provisional Patent Application Nos. 62 / 205,440 and 62 / 205,433, which are incorporated by reference. For example, a set of fluoroscopic image anatomical reference points are rotated, translated, or otherwise transformed by rigid or non-rigid body to match model points in the anatomical model (e.g., model points received from a previous or concurrent CT scan of the bore). The fluoroscopic reference frame and the model reference frame are aligned by ICP or other point registration techniques.

[0089] Fig.14 A flow chart of a method 1100 for performing image-guided surgery in a surgical environment 350 is illustrated. The method 1100 begins at process 1102, where prior image data, including preoperative or intraoperative image data, is obtained from an imaging technique such as CT, MRI, thermography, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The prior image data may correspond to a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or rate-based information) image. As described above, an anatomical model is created from the prior image data. At process 1104, shape sensor information is received from the medical instrument when an indicator portion of the medical instrument is positioned at a position L within the patient's anatomy.

[0090] At process 1106, the fluoroscopic system 370 captures fluoroscopic image data of the patient P and the catheter 360 extending within the patient. When the indicator portion 368 is positioned at or in contact with the anatomical location L, one or more fluoroscopic images rendered from the fluoroscopic image data are obtained. When the indicator portion 368 is at location L, the images obtained may be from a single viewpoint, or may be a set of multi-plane images (including a set of bi-plane images) that display the indicator portion 368 from multiple viewpoints at the same time and in the same location.

[0091] At process 1108, the shape of catheter 360 is identified in the fluoroscopic image data, and the location of position L (at the distal end of the catheter) in the fluoroscopic reference frame is also identified. In various embodiments, fluoroscopic system 370 is capable of generating multi-planar images, thereby providing a three-dimensional fluoroscopic reference frame (X F , Y F , Z F). In other embodiments, the fluoroscopic system may provide a two-dimensional fluoroscopic reference frame. Various embodiments for extracting the shape of a catheter and / or the position of an indicator portion are described above. At process 1110, the shape sensor information from process 1110 is compared with the shape information determined from the fluoroscopic image data. If the shape sensor becomes twisted or sharply bent, the shape information determined from the fiber optic shape sensor may contain inaccuracies. In addition, errors accumulate over the length of the fiber optic shape sensor, so the shape and distal tip positioning of longer instruments may be more inaccurate. By fusing instrument shape information received from multiple sources, the accuracy of the sensor information can be improved. At process 1110, the shape sensor information of the medical instrument is modified based on the fluoroscopic image shape information. For example, the position of the distal tip in the shape sensor information can be adjusted to the positioning of the distal tip in the fluoroscopic image shape information. In another example, the shape information can be averaged or modified only along the portion where the inaccuracy is expected (e.g., the distal tip portion).

[0092] At process 1112, the modified instrument sensor information is registered to the anatomical model using any of the previously described registration methods. At process 1114, an image-guided medical procedure is performed using the medical instrument in which the modified sensor information provides more accurate positioning of the instrument distal tip. Based on the positioned instrument distal tip, a virtual image of the patient's anatomy from the perspective (position and orientation) of the distal tip can be generated from the anatomical model. In addition, based on the positioned instrument distal tip, a virtual image of the medical instrument overlaid on the image from the anatomical model can be used to guide the movement of the medical instrument.

[0093] Fig.15A method 1200 for determining a preferred fluoroscopic view plane is described. At process 1202, prior image data is obtained from an imaging technique such as CT, MRI, thermography, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging, including preoperative or intraoperative image data. The prior image data may correspond to a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or rate-based information) image. As described above, an anatomical model is created from the prior image data. At process 1204, the instrument sensor 222 of the medical instrument 200 is registered to the anatomical model, as described above at process 454 for method 450. At process 1206, a preferred fluoroscopic view plane is determined to allow a clinician to envision movement of a distal tip of a catheter, movement of a biopsy needle or other tool emerging from the distal tip of a catheter, and / or an area of ​​tissue to which the tool is to be engaged (e.g., a tumor) or an area of ​​tissue to which the tool is to be avoided (e.g., the lung pleura to avoid perforation). More specifically, the preferred fluoroscopic plane can be selected by determining the pose of the distal tip portion of the instrument from the shape sensor information. The preferred fluoroscopic plane can be generally parallel to the pose of the distal tip portion of the instrument (e.g., Fig. 13B Additionally or alternatively, a preferred fluoroscopic plane can be determined by measuring the distance between the distal tip portion of the instrument and the tissue region of interest and determining the fluoroscopic view plane that provides the greatest distance. The fluoroscopic view plane can be generally parallel to the trajectory between the distal tip portion of the instrument and the tissue region, thereby providing the clinician with the most accurate information as to whether the biopsy needle has intercepted the target tissue region or has avoided an area that could cause patient harm.

[0094] Fig.16A method 1300 for driving a medical instrument under two-dimensional fluoroscopic guidance is described. At process 1302, fluoroscopic image data of a patient's anatomical structure in a surgical reference frame is obtained from a fluoroscopic system 370. The orientation of the fluoroscopic system and thus the orientation of the imaging plane can be determined by various methods including the use of an external tracking system, an imaging fiducial, or the use of shape sensing data as described above for method 450. A description of the determination of the fluoroscopic imaging plane is found in U.S. Provisional Application No. 62 / 216,494 filed on September 10, 2015, which covers Systems and Methods of Pose Estimation and Calibration of Perspective Imaging System in Image Guided Surgery, which is incorporated herein by reference in its entirety. In some instances, a graphical rendering of the fluoroscopic system can be generated to facilitate obtaining a fluoroscopic image in a desired fluoroscopic image plane. In particular, such rendering can assist the user in positioning elements of a fluoroscopic system (e.g., a fluorescent arm or an X-ray imager) to properly capture a fluoroscopic image in a desired or predetermined fluoroscopic plane. In some instances, the system or user may receive a set of joint configurations for the fluoroscopic system to better implement a system configuration for obtaining a fluoroscopic image in a desired or predetermined fluoroscopic image plane. At process 1304, a two-dimensional fluoroscopic image is generated for display based on the fluoroscopic image data. The displayed image has an azimuth plane based on the azimuth plane of the imaging plane. At process 1306, the control system 112 receives a command to drive movement of the catheter system 202. At process 1308, the movement of the catheter system 202 or a portion of the catheter is constrained to the plane of the azimuth plane of the displayed fluoroscopic image. In one embodiment, only the movement of the indicator portion of the catheter may be constrained to the azimuth plane of the fluoroscopic image. The movement of the indicator portion of the catheter may be constrained by a restricted movement controlled by an operator in the teleoperation system so that the operator is prevented from moving the control in a direction that would move a portion of the catheter outside the constraint plane. Additionally or alternatively, the movement of the indicator portion of the catheter can be constrained by limiting the movement of at least one actuator or actuation cable in the teleoperated manipulator assembly. Despite the constraints of the operator control or the instrument's actuator mechanism in the open-loop constraint system described above, the indicator portion of the catheter may still experience movement outside the plane of the constraint due to anatomical forces from adjacent tissue. In a closed-loop system, sensor information received from shape, positioning or other sensors of the medical instrument can be received and analyzed by the control system to identify movement of the indicator portion outside the azimuth plane of the fluoroscopic image.In response to the identified out-of-plane movement, the control system may provide a signal or command to adjust the indicator portion back to the azimuth plane. In some instances, the signal and / or command for adjusting the indicator portion may be rejected or ignored. In other instances, the signal and / or command for adjusting the indicator portion may automatically adjust the indicator portion back to the azimuth plane. In various embodiments, one or more additional fluoroscopic images may be obtained from the orthogonal or other non-parallel planes of the constraint plane, thereby allowing the clinician to observe any movement of the indicator portion outside the constraint plane. In various embodiments, the constrained movement of the indicator portion to the azimuth plane of the displayed fluoroscopic image may be initiated by actuation of an operator-controlled switch including a manual switch, a voice-activated switch, a foot-activated switch, or other operator-controlled mechanism. Additionally or alternatively, in response to the display of the fluoroscopic image or the identification that the operator is viewing the fluoroscopic image, the constrained movement of the indicator portion to the azimuth plane of the displayed fluoroscopic image may be initiated by the control system 112.

[0095] Although the present disclosure describes various systems and methods for use in teleoperation systems, they are also contemplated for use in non-teleoperation systems in which manipulator components and instruments are directly controlled. Although various provided examples describe surgical applications performed within an anatomical structure, in alternative embodiments, the devices and methods of the present disclosure need not be used within an anatomical structure, but may also be used outside of a patient's anatomy.

[0096] One or more elements in the embodiments of the present invention may be implemented in software to be executed on a processor of a computer system (such as control system 112). When implemented in software, the elements of the embodiments of the present invention are essentially code segments to perform the necessary tasks. The program or code segment may be stored in a non-temporary processor-readable storage medium or device, which includes any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices, and the code segments may be downloaded via computer networks such as the Internet, intranets, etc.

[0097] Note that the process and display presented may not be inherently linked to any particular computer or other device. The structure required for various of these systems will appear as an element in the claims. In addition, embodiments of the present invention are not described with reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0098] While certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the broad invention and not restrictive, and that the embodiments of the present invention are not limited to the specific construction and arrangements shown and described, since various other modifications may occur to those skilled in the art.

Claims

1. A computer readable medium storing instructions, which when executed by one or more processors are adapted to cause the one or more processors to perform a method comprising: identifying a set of locations of a plurality of anatomical landmarks rendered in an anatomical model of an orifice of a patient's anatomy in a model reference frame; receiving fluoroscopic image data of the patient's anatomy as a portion of a medical instrument passes through the plurality of anatomical landmarks in the bore of the patient's anatomy, the fluoroscopic image data having a fluoroscopic reference frame; identifying, in the fluoroscopic reference frame, a set of locations of the portion of the medical instrument at the plurality of anatomical landmarks; as well as The set of locations of the plurality of anatomical landmarks in the model reference frame and the set of locations of the portion of the medical instrument at the plurality of anatomical landmarks in the fluoroscopy reference frame are registered to a common reference frame.

2. The computer-readable medium of claim 1 , wherein identifying the set of locations of the portion of the medical instrument in the fluoroscopic reference frame comprises: A predetermined shape of the portion of the medical instrument is identified in the fluoroscopic reference frame.

3. The computer-readable medium of claim 1 , wherein identifying the set of locations of the portion of the medical instrument in the fluoroscopic reference frame comprises: The fluoroscopic image data of the patient anatomy when the portion of the medical instrument is positioned within at least one of the plurality of landmarks is compared to fluoroscopic image data of the patient anatomy recorded when the portion of the medical instrument is not positioned within the patient anatomy.

4. The computer-readable medium of claim 1 , wherein identifying the set of locations in the fluoroscopic reference frame comprises: The fluoroscopic image data of the patient anatomy when the portion of the medical instrument is positioned at at least one of the plurality of landmarks is compared to an image of the patient anatomy recorded when the portion of the medical instrument was positioned at another location within the patient anatomy.

5. The computer-readable medium of claim 1, wherein the instructions are further adapted to cause the one or more processors to perform the following operations: display a co-registered image of a portion of the fluoroscopic image, a portion of the anatomical model, and the portion of the medical instrument.

6. The computer readable medium of claim 1, wherein the common reference frame is the model reference frame.

7. The computer readable medium of claim 1, wherein the common reference frame is the fluoroscopic reference frame.

8. A computer readable medium storing instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising: receiving a set of model points of an anatomical model of an orifice for an anatomy of a patient in a model reference frame; receiving fluoroscopic image data of the patient's anatomy as an indicator portion of a medical instrument passes through the aperture of the patient's anatomy, the fluoroscopic image data having a fluoroscopic reference frame; identifying from the fluoroscopic image data a set of indicator portion location points in the fluoroscopic reference frame; matching each indicator portion anchor point to a model point in the set of model points to generate a set of matches; as well as Based on the set of matches, the model reference frame is registered to the fluoroscopy reference frame.

9. The computer readable medium of claim 8, wherein identifying the set of indicator portion location points in the fluoroscopic reference frame comprises: A desired shape of the indicator portion in the fluoroscopic reference frame is identified.

10. The computer-readable medium of claim 8, wherein identifying the set of indicator portion positioning points in the fluoroscopic reference frame comprises comparing the fluoroscopic image data of the patient anatomy when the indicator portion of the medical instrument is positioned with a recording of the patient anatomy when the indicator portion of the medical instrument is not positioned within the patient anatomy.

Images