Systems and methods for pose estimation and calibration of fluoroscopic imaging systems in image-guided surgery

Abstract

This application discloses systems and methods for pose estimation and calibration of fluoroscopic imaging systems in image-guided surgery. A method performed by a computing system includes receiving first fluorescein image data of a first reference marker in a surgical coordinate space from a fluorescein imager having a first set of parameters. The method includes receiving a configuration of the first reference marker in the surgical coordinate space. The method includes determining a second set of parameters of the fluorescein imager in the surgical coordinate space based on the first fluorescein image data and the configuration of the first reference marker. In some embodiments, determining the second set of parameters includes developing a calibration model of the reference marker in the surgical coordinate space from the first fluorescein image data and the configuration of the first reference marker.

Description

[0001] This application is a divisional application of Chinese patent application 201780019575.X, filed on February 10, 2017, entitled "System and method for pose estimation and calibration of fluoroscopic imaging system in image-guided surgery".

[0002] Related applications

[0003] This patent application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 294,845, filed February 12, 2016, entitled "Systems and methods of pose estimation and calibration of PERSPECTIVE IMAGING SYSTEM IN IMAGE GUIDED SURGERY", and U.S. Provisional Patent Application No. 62 / 294,857, also filed February 12, 2016, entitled "Systems and methods of pose estimation and calibration of PERSPECTIVE IMAGING SYSTEM IN IMAGE GUIDED SURGERY", the entire contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure relates to systems and methods for performing image-guided procedures, and more specifically to systems and methods for pose estimation, calibration of fluoroscopic imaging systems, and real-time tomography synthesis to improve the accuracy of tool navigation during image-guided procedures. Background Technology

[0005] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, clinicians can insert minimally invasive medical devices (including surgical, diagnostic, therapeutic, or biopsy instruments) to reach target tissue locations. To aid in reaching the target tissue location, the position and movement of the medical device can be registered with preoperative or intraoperative images of the patient's anatomy. In the case of image-guided device registration to an image, the device can navigate natural or surgically generated pathways in anatomical systems such as the lungs, colon, intestines, kidneys, heart, circulatory system, etc. Some image-guided devices may include fiber optic shape sensors that provide information about the shape of an elongated, flexible device and the orientation of its distal end. In some embodiments, fluoroscopic imaging systems are used during surgery to assist in locating the target and / or device during the procedure. For imaging data to assist in the correct positioning of the medical device, the imaging system must be accurately registered to the coordinate system of the surgical environment. Systems and techniques are needed to minimize errors associated with pose estimation and calibrating imaging systems to the surgical environment in order to more reliably locate and navigate medical devices within the surgical environment. Summary of the Invention

[0006] Embodiments of the invention are summarized by the claims appended to the specification.

[0007] In one embodiment, the method performed by a computing system includes receiving first fluorescence microscopy image data of a first reference marker in a surgical coordinate space from a fluorescence microscopy imager having a first set of parameters. The method also includes receiving a configuration of the first reference marker in the surgical coordinate space. The method further includes determining a second set of parameters of the fluorescence microscopy imager in the surgical coordinate space based on the first fluorescence image data of the first reference marker and the configuration. In one aspect, determining the second set of parameters includes developing a calibration model of the reference marker in the surgical coordinate space from the first fluorescence image data of the first reference marker and the configuration.

[0008] In another embodiment, the computer-assisted medical system includes one or more processors and a first reference marker positioned in a surgical coordinate space in a known configuration. The first reference marker includes a shape sensor. The one or more processors perform a method including receiving first fluorescein image data of the first reference marker positioned in a known configuration in the surgical coordinate space from a fluorescein imager and receiving shape information from the shape sensor. The method also includes determining a known configuration of the first reference marker in the surgical coordinate space from the shape information, and determining a calibration model of the reference marker in the surgical coordinate space from the first fluorescein image data. The method also includes determining the orientation of the fluorescein imager in the surgical coordinate space based on the first fluorescein image data.

[0009] In another embodiment, the computer-assisted medical system includes a fluorescence microscopy imager with a full scan range in surgical coordinate space, and one or more processors configured to perform a method. In one aspect, the method includes receiving a first set of fluorescence microscopic image data of a patient's anatomy from the fluorescence microscopy imager, and receiving at least one additional set of fluorescence microscopic image data of the patient's anatomy from the fluorescence microscopy imager operating within a constrained range substantially smaller than the full scan range. The method also includes constructing a first planar tomographic image from the first set of fluorescence microscopic image data and the at least one additional set of fluorescence microscopic image data.

[0010] In another embodiment, the method of local tomographic synthesis includes receiving a first set of fluorescein image data of a patient's anatomy, the first set of fluorescein image data being obtained from the full scan range of a fluorescein imager. The method further includes receiving at least one additional set of fluorescein image data of the patient's anatomy from a fluorescein imager operating within a constrained region substantially smaller than the full scan range. The method further includes constructing a first planar tomographic image from the first set of fluorescein image data and at least one additional set of fluorescein image data. In one aspect, the method further includes operating the fluorescein imager within a second constrained region substantially smaller than the full scan range, based on the first planar tomographic image. In one aspect, the method further includes constructing a second planar tomographic image from a set of fluorescein image data received while the fluorescein imager is within the second constrained region.

[0011] In another embodiment, the method of local tomographic synthesis includes moving a fluorescence microscope imager to a first location within a constrained area, wherein the fluorescence microscope imager has a full scan range, and wherein the constrained area is substantially smaller than the full scan range. The method also includes obtaining a first fluorescence microscope image of a patient's anatomy from the first location, moving the fluorescence microscope imager to a plurality of additional locations within the constrained area, obtaining fluorescence microscope images from each of the plurality of additional locations, and constructing a first planar tomographic image from the first fluorescence microscope image and the fluorescence microscope images obtained from each of the plurality of additional locations.

[0012] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0013] When read in conjunction with the accompanying drawings, aspects of this disclosure can be best understood from the following detailed description. It should be emphasized that, in accordance with standard industry practice, the various features are not drawn to scale. In fact, the dimensions of the various features may be increased or decreased arbitrarily for clarity of discussion. Furthermore, reference numerals and / or letters may be repeated in various examples throughout this disclosure. This repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0014] Figure 1 This is a remote-operated medical system according to embodiments of the present disclosure.

[0015] Figure 2A A medical device system utilizing aspects of this disclosure is shown.

[0016] Figure 2B The illustration shows an extended medical tool according to an embodiment of the present disclosure. Figure 2A The distal end of a medical device system.

[0017] Figure 3 It shows the location inside the human lung. Figure 2A The distal end of a medical device system.

[0018] Figure 4 This is a flowchart illustrating a method for providing guidance in an image-guided surgical procedure according to an embodiment of the present disclosure.

[0019] Figure 5 This is a side view of a patient coordinate space according to an embodiment of the present disclosure, the patient coordinate space including a medical device and an optical tracking system mounted on an insertion assembly.

[0020] Figure 6A This is an embodiment of the present invention. Figure 5 The patient coordinate space shown includes a C-arm of a fluorescence imaging system and an external tracking system relative to the patient in a first position.

[0021] Figure 6B According to embodiments of this disclosure Figure 6A The patient coordinate space shown is a side view of the patient coordinate space, which includes the C-arm of the fluorescence imaging system and the external tracking system relative to the patient in a second position.

[0022] Figures 7A to 7C Various exemplary reference markers according to various embodiments of this disclosure are shown. Figure 7A An exemplary benchmark marker is shown, comprising a set of unique marker elements. Figure 7B A reference marker including a grid pattern is shown. Figure 7C The reference markers, including a chessboard pattern, are shown.

[0023] Figures 8A to 8E Several exemplary identifiable reference markers are shown according to various embodiments of the present disclosure.

[0024] Figure 9A and Figure 9B A reference sign is shown according to an embodiment of the present disclosure, having sign elements arranged in a non-uniform linear pattern.

[0025] Figure 10 This is a diagram illustrating the principle of cross ratio and projection invariance.

[0026] Figure 11 This is a diagram illustrating the principle of projection invariance in image-guided surgical procedures, where a fluoroscopic imager images a stationary reference marker from two different positions.

[0027] Figure 12 This is a diagram illustrating the principle of projection invariance in image-guided surgical procedures, where a static fluoroscopic imager images reference markers placed in two different locations.

[0028] Figure 13 A flowchart illustrating a portion of an image-guided medical procedure according to an embodiment of the present disclosure is shown.

[0029] Figure 14 The patient's anatomy and tomographic arrangement are shown from a planar view of the surgical environment.

[0030] Figure 15 This shows the view from the orthogonal plane. Figure 14 The patient's anatomical structure and tomographic arrangement.

[0031] Figure 16 A flowchart of an image-guided medical procedure according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] In the following detailed description of aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that embodiments of this disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention. Furthermore, to avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted, as applicable to other illustrative embodiments.

[0033] The following embodiments describe various instruments and parts thereof based on their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six degrees of freedom in total). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0034] Refer to the attached diagram. Figure 1 A remotely operated medical system used in procedures such as surgery, diagnosis, treatment, or biopsy is typically designated as a remotely operated medical system 100. For example... Figure 1 As shown, the remote operating system 100 typically includes a remote operation manipulator component 102 for operating the medical device system 104 to perform various procedures on the patient P. The remote operation manipulator component 102 is also referred to as "remote operation component 102" or "manipulator component 102". The medical device system 104 is also referred to as "medical device 104". Component 102 is mounted on or near the operating table O. An operator input system 106 (also referred to as "master component 106") allows the clinician or surgeon S to observe the intervention site and control the manipulator component 102.

[0035] Operator input system 106 may be located at the surgeon's console, which is typically in the same room as the operating table O. However, it should be understood that the surgeon S may be in a different room or a completely different building from the patient P. Operator input component 106 typically includes one or more control devices for controlling manipulator component 102. Control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controllers, voice recognition devices, body motion or presence sensors, etc. In some embodiments, the control device will have the same degrees of freedom as the associated medical device 104 to provide telepresence to the surgeon, or provide a perception that the control device is integrated with the device 104, giving the surgeon a strong sense of direct control over the device 104. In other embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104, and still provide telepresence to the surgeon. In some embodiments, the control device is a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating the device (e.g., for closing gripping jaws, applying a potential to electrodes, delivering medication, etc.).

[0036] The teleoperation component 102 supports the medical device system 104 and may include the motion structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, often referred to as an assembly structure) and a teleoperation manipulator. The teleoperation component 102 includes multiple actuators or motors that drive inputs on the medical device system 104 in response to commands from a control system (e.g., control system 112). The motors include drive systems that, when coupled to the medical device system 104, can advance the medical device into natural or surgically created anatomical openings. Other motorized drive systems can move the distal end of the medical device with multiple degrees of freedom, including three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate articulated end effectors of the device for grasping tissue in the jaws of biopsy equipment, etc. Motor position sensors, such as resolvers, encoders, potentiometers, and other mechanisms, can provide remotely operated components with sensor data describing the rotation and orientation of the motor shaft. This position sensor data can be used to determine the motion of an object manipulated by the motor.

[0037] The remote-operated medical system 100 also includes a sensor system 108 having one or more subsystems for receiving information about the instrument of the remote-operated component. Such subsystems may include: a position / positioning sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, speed, velocity, posture, and / or shape of the catheter tip and / or one or more segments along the flexible body of the instrument system 104; and / or a visualization system for capturing images from the distal end of the catheter system.

[0038] Visualization systems (e.g.) Figure 2A The visualization system 231 may include an observation range component that records concurrent or real-time images of the surgical site and provides the images to a clinician or surgeon S. The concurrent images may be two-dimensional or three-dimensional images, for example, captured by a fluoroscopic imaging system located outside the surgical site. In some embodiments, the fluoroscopic imaging system includes an imager that produces two-dimensional images, such as, as a non-limiting example, a fluoroscope or X-ray imager or an optical camera. Additionally or alternatively, the concurrent images may be two-dimensional or three-dimensional images, for example, captured by an endoscope located within the surgical site. In this embodiment, the visualization system includes an endoscope component that is integrally or removably coupled to the medical device 104. However, in alternative embodiments, a separate endoscope attached to a separate manipulator component may be used with the medical device to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof, which interacts with or is otherwise executed by one or more computer processors, which may include a processor of the control system 112 (described below). The processor of the control system 112 is executable instructions, including instructions corresponding to the processes disclosed herein.

[0039] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and (one or more) medical device systems 104 generated by a subsystem of the sensor system 108. The display system 110 and the operator input system 106 may be configured to enable an operator to control the medical device system 104 and the operator input system 106 using the remotely presented perception.

[0040] The display system 110 can also display images of the surgical site and medical instruments captured by the visualization system. The display system 110 and control device can be arranged such that the relative positions of the imaging device and medical instruments in the range assembly resemble the relative positions of a surgeon's eye and hand, thus enabling the operator to manipulate and manually control the medical instrument 104 as if viewing the workspace in a substantially realistic presence. This realistic presence means that the image presentation is a true perspective image simulating the viewpoint of the operator physically manipulating the instrument 104.

[0041] Alternatively or additionally, the display system 110 may use imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, thermal infrared imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data may be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information) or as images of models derived from the preoperative or intraoperative image data set.

[0042] In some embodiments, typically for the purpose of image-guided surgical procedures, the display system 110 may display a virtual navigation image, wherein the actual position of the medical device 104 is registered (i.e., dynamically referenced) with preoperative or concurrent images / models to present a virtual image of the internal surgical site to the clinician or surgeon S from a viewpoint of the tip of the device 104. An image of the tip of the device 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist the surgeon in controlling the medical device. Alternatively, the device 104 may not be visible in the virtual image.

[0043] In other embodiments, the display system 110 may display a virtual navigation image, wherein the actual position of the medical device is registered with preoperative or concurrent images to present a virtual image of the medical device within the surgical site to the clinician or surgeon S from an external viewpoint. An image of a portion of the medical device or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist the surgeon in controlling the device 104.

[0044] The remote-operated medical system 100 also includes a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown), and typically includes multiple processors for achieving control between the medical device system 104, the operator input system 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described according to the aspects disclosed herein, including instructions for providing pathological information to the display system 110. While the control system 112... Figure 1 The simplified schematic is shown as a single box, but the system may include two or more data processing circuits, with some processing optionally performed on or near the remote operation component 102, others at the operator input system 106, and so on. A wide variety of centralized or distributed data processing architectures can be employed. Similarly, programming instructions may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the remote operating system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0045] In some embodiments, the control system 112 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 104. In response to the feedback, the servo controllers transmit signals to the operator input system 106. The servo controllers may also send signals instructing the teleoperation component 102 to move the medical device system 104, which extends through an opening in the body to an internal surgical site within the patient's body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate from or integrated with the teleoperation component 102. In some embodiments, the servo controller and the teleoperation component are configured as part of a teleoperation arm trolley positioned adjacent to the patient's body.

[0046] The control system 112 may also include a virtual visualization system to provide navigation assistance to one or more medical device systems 104 during image-guided surgical procedures. Virtual navigation using the virtual visualization system is based on a reference to a set of preoperative or intraoperative data of the acquired anatomical pathway. More specifically, the virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, thermal infrared imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging, etc. Software alone, or software combined with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional synthetic representations of parts or entire anatomical organs or regions. The set of image data is associated with the synthetic representation. The synthetic representation and the set of image data describe the various locations and shapes of the pathway and their connectivity. Images used to generate the synthetic representation may be recorded preoperatively or intraoperatively during the clinical procedure. In alternative embodiments, the virtual visualization system may use a standard representation (i.e., not patient-specific) or a mixture of standard representation and patient-specific data. Synthetic representations and any virtual images generated from synthetic representations can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0047] During virtual navigation procedures, sensor system 108 can be used to calculate the approximate position of the instrument relative to the patient's anatomy. This position can be used to generate macroscopic (external) tracking images of the patient's anatomy and virtual internal images of the patient's anatomy. Various systems are known for registering and displaying medical instruments and preoperatively recorded surgical images, such as surgical images from virtual visualization systems, using electromagnetic (EM) sensors, fiber optic sensors, or other sensors. For example, U.S. Patent Application No. 13 / 107,562 (filed May 13, 2011) (disclosing "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery") discloses such a system, the entire contents of which are incorporated herein by reference.

[0048] The remote-operated medical system 100 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems, and / or suction systems. In alternative embodiments, the remote operating system may include more than one remote operating component and / or more than one operator input system. Among other factors, the exact number of manipulator components will depend on the surgical procedure and the space constraints within the operating room. Operator input systems may be juxtaposed, or they may be located in different locations. Multiple operator input systems allow more than one operator to control one or more manipulator components in various combinations.

[0049] Figure 2A A medical device system 200 is shown, which can be used as medical device system 104 in image-guided medical procedures performed using a remotely operated medical system 100. Alternatively, the medical device system 200 can be used in non-remotely operated exploratory procedures or in medical device procedures involving conventional manual operation, such as endoscopy. Additionally or alternatively, the medical device system 200 can be used to collect (i.e., measure) a set of data points corresponding to locations within the patient's anatomical pathway.

[0050] Device system 200 includes catheter system 202 coupled to device housing 204. Catheter system 202 includes an elongated, flexible catheter body 216 having a proximal end 217 and a distal end 218 (also referred to as “tip portion 218”). In one embodiment, flexible body 216 has an outer diameter of approximately 3 mm. Other flexible bodies may have a larger or smaller outer diameter. Catheter system 202 may optionally include a shape sensor system 222 (also referred to as “shape sensor 222”) for determining the position, orientation, speed, velocity, posture, and / or shape of the catheter tip at the distal end 218 and / or along one or more segments 224 of body 216. The entire length of body 216 between distal end 218 and proximal end 217 may be effectively divided into segments 224. If device system 200 is a medical device system 104 of remotely operated medical system 100, shape sensor 222 may be a component of sensor system 108. If the instrument system 200 is manually operated or otherwise used for non-remote operating procedures, the shape sensor 222 may be coupled to the tracking system 230, which queries the shape sensor and processes the received shape data.

[0051] The shape sensor 222 may include an optical fiber aligned with the flexible conduit body 216 (e.g., disposed within an internal channel (not shown) or mounted externally thereon). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor system 222 forms an optical fiber bend sensor for determining the shape of the conduit system 202. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements of the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of optical fibers in three-dimensional space are described in U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) ("Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) ("Fiber-optic shape and relative positionsensing"); and U.S. Patent No. 6,389,187 (filed June 17, 1998) ("Optical Fiber Bend Sensor"). The entire contents of these applications are incorporated herein by reference. In alternative embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In other alternative embodiments, other techniques may be used to determine the shape of the catheter. For example, the history of the distal tip posture of the catheter can be used to reconstruct the shape of the device over time intervals. As another example, historical posture, position, or orientation data can be stored for known points on an instrument system used for alternating movements such as breathing. This stored data can be used to develop shape information about the catheter. Alternatively, a series of position sensors (such as electromagnetic (EM) sensors) positioned along the catheter can be used for shape sensing. Alternatively, the shape of the instrument can be represented using the history of data from position sensors (such as EM sensors) on the instrument system during the procedure, especially if the anatomical pathway is generally static. Alternatively, a wireless device with position or orientation controlled by an external magnetic field can be used for shape sensing. The history of the wireless device's position can be used to determine the shape of the navigation pathway.

[0052] Optionally, the medical device system may include a position sensor system 220. The position sensor system 220 may be a component of an EM sensor system, wherein the sensor system 220 includes one or more conductive coils that can withstand an externally generated electromagnetic field. In this embodiment, each coil of the EM sensor system including the position sensor system 220 then generates an induced electrical signal having characteristics dependent on the position and orientation of the coil relative to the externally generated electromagnetic field. In one embodiment, the EM sensor system may be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, Z and three orientation angles, indicating pitch, yaw, and roll of a base point, or five degrees of freedom, such as three position coordinates X, Y, Z and two orientation angles, indicating pitch and yaw of a base point. Further description of the EM sensor system is disclosed in U.S. Patent No. 6,380,732 (filed August 11, 1999) (disclosing "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), the entire contents of which are incorporated herein by reference. In some embodiments, the shape sensor may also function as a position sensor because the shape of the sensor, along with information about the position of the base of the shape sensor (in a patient's fixed coordinate system), allows for the calculation of the localization of individual points along the shape sensor, including the distal tip.

[0053] Tracking system 230 may include position sensor system 220 and shape sensor system 222 for determining the position, orientation, velocity, attitude, and / or shape of distal end 218 and one or more segments 224 along instrument system 200. Tracking system 230 may be implemented as hardware, firmware, software, or a combination thereof, and may interact with or be otherwise executed by one or more computer processors, including the processor of control system 116.

[0054] The flexible catheter body 216 includes a channel 221, which is sized and shaped to receive a medical device 226. The medical device may include, for example, an image capture probe, a biopsy instrument, a laser ablation fiber, or other surgical, diagnostic, or therapeutic tool. The medical tool may include an end effector with a single working component, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, grippers, scissors, or clamp applicators. Examples of electrically activated end effectors include electrosurgical electrodes, transducers, sensors, etc. In various embodiments, the medical device 226 may be an image capture probe including a distal portion having a stereo or single-field-of-view camera at or near the distal end 218 of the flexible catheter body 216 for capturing images (including video images) processed by the visualization system 231 for display. The image capture probe may include a cable coupled to the camera for transmitting the captured image data. Alternatively, the image capture device may be a bundle of optical fibers coupled to the visualization system, such as a fiber endoscope. Image capture devices can be single-spectral or multi-spectral, for example, capturing image data from one or more of the visible, infrared, or ultraviolet spectra.

[0055] Medical device 226 may accommodate cables, linkages, or other actuation controllers (not shown) extending between the proximal and distal ends of the device to controllably flex the distal end of the device. Steering devices are described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005) (disclosing “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (disclosing “Passive Preload and Capstan Drive for Surgical Instruments”), the entire contents of which are incorporated herein by reference.

[0056] The flexible catheter body 216 may also accommodate a cable, linkage, or other steering controller (not shown) extending between the housing 204 and the distal end 218 to controllably bend the distal end 218, as illustrated, for example, by the dashed line 219 depicting the distal end. Steering catheters are described in detail in U.S. Patent Application No. 13 / 274,208 (filed October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), the entire contents of which are incorporated herein by reference. In embodiments where the instrument system 200 is actuated by a remotely operated component, the housing 204 may include a drive input removably coupled to and receiving power from a motorized drive element of the remotely operated component. In embodiments where the instrument system 200 is manually operated, the housing 204 may include clamping features, a manual actuator, or other components for manually controlling the movement of the instrument system. The catheter system may be steerable or, alternatively, the system may be non-steerable and without an integrated mechanism for operator-controlled bending of the instrument. Alternatively, one or more lumens may be defined in the wall of the flexible body 216, through which a medical device can be deployed and used at a target surgical site.

[0057] In various embodiments, the medical device system 200 may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes, for use in the examination, diagnosis, biopsy, or treatment of the lungs. The system 200 is also suitable for navigating and treating other tissues in any of a variety of anatomical systems, including the colon, intestines, kidneys, brain, heart, circulatory system, etc., via natural or surgically created access pathways.

[0058] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or a preoperatively acquired model to provide surgeons or other operators with real-time location information about display system 110 for use in the control of instrument system 200. Control system 116 can utilize the location information as feedback for positioning instrument system 200. Various systems for registering and displaying surgical instruments with surgical images using fiber optic sensors are provided in U.S. Patent Application No. 13 / 107,562, filed May 13, 2011, entitled "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery," the entire contents of which are incorporated herein by reference.

[0059] Fiber optic shape sensors are particularly useful as positioning sensors because they provide data on the overall shape of the instrument, including the orientation of the distal tip, without being sensitive to metallic objects in the area or requiring obstructive imaging equipment. However, when using fiber optic shape sensors, small position and orientation errors at the proximal end of the fiber can lead to large cumulative position and orientation errors at the distal end due to the sensor's length (e.g., approximately one meter). Systems and methods for reducing these errors are described below, and these can be used to produce more accurate fiber optic registration during procedures and thus more accurate registration of medical instruments to surgical coordinate systems and to anatomical models.

[0060] exist Figure 2A In one embodiment, the device system 200 is remotely operated within the remotely operated medical system 100. In an alternative embodiment, the remote operation component 102 may be controlled directly by an operator. In direct operation alternatives, various handles and operator interfaces may be included for handheld operation of the device.

[0061] In alternative embodiments, the remote operating system may include more than one slave manipulator component and / or more than one master component. The exact number of manipulator components will depend, among other factors, on the medical surgical procedure and space constraints within the operating room. Master components may be juxtaposed, or they may be located in different locations. Multiple master components allow more than one operator to control one or more slave manipulator components in various combinations.

[0062] like Figure 2B As shown in more detail, one or more medical instruments 228 for such procedures, such as surgery, biopsy, ablation, illumination, irrigation, or aspiration, can be deployed through channels 221 of the flexible body 216 and used at target locations within the anatomical structure. For example, if the instrument 228 is a biopsy instrument, it can be used to remove sample tissue or cells from a target anatomical location. The medical instrument 228 can be used in conjunction with an image-capturing probe also within the flexible body 216. Alternatively, the instrument 228 itself can be an image-capturing probe. The instrument 228 can advance from the opening of the channel 221 to perform the procedure and then retract into the channel when the procedure is complete. The medical instrument 228 can be removed from the proximal end 217 of the catheter flexible body or from another optional instrument port (not shown) along the flexible body.

[0063] Figure 3 A catheter system 202 is shown positioned within an anatomical pathway of a patient's anatomy. In this embodiment, the anatomical pathway is the airway of a human lung 201. In alternative embodiments, the catheter system 202 may be used in other pathways within the anatomical structure.

[0064] Figure 4This is a flowchart illustrating a general method 300 used in an image-guided surgical procedure. In procedure 302, prior image data, including preoperative or intraoperative image data, is obtained from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, thermal infrared imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The preoperative or intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information). For example, the image data may represent... Figure 3 201 human lungs.

[0065] In process 304, computer software, or computer software in combination with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional synthetic representations or models of parts or entire anatomical organs or regions. The synthetic representation and image data sets describe the various locations and shapes of pathways and their connectivity. More specifically, during the segmentation process, the image is divided into segments or elements (e.g., pixels or voxels) that share certain characteristics or computational properties such as color, density, intensity, and texture. This segmentation process results in a two-dimensional or three-dimensional reconstruction of a model of the target anatomical structure based on the acquired images. To represent the model, the segmentation process may depict a group of voxels representing the target anatomical structure and then apply a function such as a traveling cube function to generate a 3D surface surrounding the voxels. The model can be created by generating meshes, volumes, or voxel maps. Additionally or alternatively, the model may include a centerline model comprising a set of interconnected line segments or points extending through the center of the modeled pathway. In cases where the model includes a centerline model comprising a set of interconnected line segments, those line segments may be converted into a cluster or a set of points. By converting line segments, the desired number of points corresponding to the interconnecting line segments can be selected manually or automatically.

[0066] In process 306, the anatomical model data, the medical device used to perform the medical procedure (e.g., device system 200), and the patient's anatomy are jointly registered in a common reference frame before and / or during an image-guided surgical procedure on the patient. The common reference frame may be, for example, a surgical environment reference frame or a patient reference frame. Process 306 includes positioning the medical device relative to the patient. Process 306 also includes registering the anatomical model relative to the patient. Typically, registration involves matching measurement points to points in the model using rigid and / or non-rigid transformations. Measurement points can be generated using markers in the anatomy, electromagnetic coils scanned and tracked during the procedure, or shape sensor systems. Measurement points can be generated for use with Iterative Closest Point (ICP) techniques. ICP and other registration techniques are described in U.S. Provisional Patent Application Nos. 62 / 205,440 and 62 / 205,433, both filed August 14, 2015, the entire contents of which are incorporated herein by reference. In process 308, anatomical model data can be used to execute medical procedures to guide the movement of medical devices.

[0067] For example, prior image data acquired using CT and used in image-guided surgical procedures typically provides fine anatomical details suitable for many procedures. However, prior image data is affected by registration errors and does not show the real-time configuration of anatomical structures, including any deformations due to periodic or non-periodic anatomical movements, tissue deformations due to medical devices, or other changes to the patient's anatomy that may have occurred since the prior image data was acquired.

[0068] Traditional registration methods used in conjunction with image-guided surgery typically involve the use of electromagnetic or impedance-based sensing techniques. In some cases, metallic objects or certain electronic devices used in the surgical environment can introduce interference that impairs the quality of the sensed data. Alternatively or additionally, fluoroscopic imaging systems such as fluorescence imaging systems and / or optical tracking systems can be used to perform registration. In some embodiments described herein, fluorescence imaging systems and / or optical tracking systems can be used to determine the position and orientation of medical devices and patient anatomy within a coordinate system of the surgical environment. While the various examples provided describe the use of procedures performed within the anatomical structure, in alternative embodiments, the devices and methods of this disclosure do not need to be used within the anatomical structure but can also be used outside the patient's anatomy.

[0069] Fluorescence microscopy is an imaging modality that uses X-rays to obtain real-time moving images of patient anatomy, medical instruments, and any radiopaque reference markers within the imaging field. In this discussion, reference markers may also be referred to as reference elements or references. Conventional radiographs are X-ray images obtained by placing a portion of the patient in front of an X-ray detector and then irradiating it with short pulses of X-rays. Similarly, fluorescence microscopy uses X-rays to obtain real-time moving images of the patient's body, including radiopaque medical instruments, radiopaque dyes, and / or radiopaque reference markers within the surgical environment. Fluorescence microscopy systems may include C-arm systems, which offer positional flexibility and are capable of orbital, horizontal, and / or vertical movement via manual or automatic control. Non-C-arm systems are stationary and offer less flexibility in motion. Fluorescence microscopy systems typically use image intensifiers or flat panel detectors to generate two-dimensional, real-time images of the patient's anatomy. Dual-plane fluorescence microscopy systems simultaneously capture two fluorescence microscopic images, each from different (typically orthogonal) viewpoints. The quality and utility of the X-ray images can vary depending on the type of tissue being imaged. In X-ray images, denser materials such as bone and metal are generally more visible than the soft tissue of the air-filled lungs. For lung procedures, previous CT images provided anatomical details of airways and tumors that would be difficult to discern on fluorescence microscopy images, but fluorescence microscopy images provide real-time visualization of medical devices and dense anatomical tissues.

[0070] Optical tracking systems use position sensors to detect infrared emitting or reflecting markers attached to teleoperated components, medical devices, and / or patients. The position sensors calculate the position and orientation of the teleoperated components, medical devices, and / or patients based on information received from those markers. More specifically, the optical tracking system uses data captured from image sensors to triangulate the three-dimensional position of the teleoperated components, medical devices, and / or patients between calibrated cameras to provide an overlapping projection.

[0071] Therefore, registered fluorescein, previous image data, and / or optical tracking data can be used alone or in combination with kinematic and shape sensor data to assist clinicians in navigating anatomical structures such as parts of the lungs by providing more accurate posture estimation and positioning of medical devices. Specifically, by acquiring fluorescein data of reference markers with known positions and / or shape characteristics within the surgical coordinate system, and by combining the fluorescein data with the position or shape data of the reference markers, the remote operating system can more accurately register the fluorescein imaging system to the surgical coordinate system, thereby improving the reliability of device navigation during surgical procedures.

[0072] Figure 5 An exemplary surgical environment 350 according to some embodiments is shown, which has a surgical coordinate system X.S Y S Z S Patient P is positioned on platform 352. Patient P can be stationary in the surgical environment, which is in the sense that the patient's overall movement is restricted by sedation, restraint, or other means. Circulatory anatomical movements, including the patient P's breathing and cardiac movements, can continue unless the patient temporarily stops respiratory movements. Within the surgical environment 350, medical device 354 is coupled to instrument holder 356. Instrument holder 356 is mounted to an insertion stage 358 that is fixed or movable within the surgical environment 350. Instrument holder 356 may be a component of a remotely operated manipulator assembly (e.g., assembly 102) coupled to instrument 354 to control insertion movements (i.e., in X...). S Movement in the direction of the instrument, and optionally, movement of the distal end of the instrument in multiple directions (including yaw, pitch, and roll). The instrument holder 356 or the insertion stage 358 may include a servo motor (not shown) that controls the movement of the instrument holder along the insertion stage.

[0073] Medical device 354 may include a flexible catheter 360 coupled to a proximal rigid device body 362. The rigid device body 362 is coupled and fixed relative to a device support 356. A fiber optic shape sensor 364 extends along device 354 and is operable to measure the shape from a fixed or known point 366 to another point (such as the distal end portion 368 of catheter 360). Medical device 354 may be substantially similar to medical device system 200.

[0074] A fluorescence imaging system 370 is positioned near the patient P to acquire fluorescence images of the patient as the catheter 360 extends within the patient's body. For example, as... Figure 6A and Figure 6B As shown, system 370 can be a mobile C-arm fluorescence imaging system. In some embodiments, system 370 can be a multi-axis Artis Zeego fluorescence imaging system from Siemens Corporation in Washington, D.C.

[0075] Figure 6A and Figure 6B It shows Figure 5 Another view of the exemplary surgical environment 350 shown. Figure 6AA fluorescence imaging system 370 including a movable C-arm 380 is shown, positioned relative to a patient P and an external tracking system 382. In the illustrated embodiment, the C-arm 380 includes an X-ray source 384 and an X-ray detector 386 (also referred to as "X-ray imager 386"). The X-ray detector 386 generates an image representing the intensity of the received X-rays. Typically, the X-ray detector 386 includes an image intensifier that converts the X-rays into visible light and a charge-coupled device (CCD) camera that converts the visible light into a digital image. In operation, the patient P is positioned between the X-ray source 384 and the X-ray detector 386. In response to a command input from an operator or remote operating system, X-rays emitted from the X-ray source 384 pass through the patient P and enter the X-ray detector 386, producing a two-dimensional image of the patient.

[0076] The raw image generated by X-ray detector 386 is susceptible to undesirable distortions (e.g., "barrel distortion" and / or "S-shaped distortion") caused by a number of factors, including inherent image distortions from the image intensifier and external electromagnetic fields. Internal calibration (the process of correcting image distortions in the received image and learning the projection geometry of the imager) involves placing a reference marker 400 (also called a calibration marker) in the path of the X-rays, where the reference marker is an object opaque to X-rays. The reference marker 400 is rigidly arranged in a predetermined pattern in one or more planes in the X-ray path and is visible in the recorded image. Because the true relative position of the reference marker 400 in the surgical environment is known (i.e., the undistorted position), the control system 112 is able to calculate the amount of distortion at each pixel in the synthesized image (where a pixel is a single point in the image). Therefore, the control system 112 is able to digitally compensate for distortions in the initial image and generate a distortion-free or at least distortion-reduced digital image. Improving the accuracy of the true relative position of the reference marker 400 allows for enhanced posture estimation of the fluorescence imaging system 370, thereby increasing the accuracy of instrument posture determination and navigation during surgical procedures.

[0077] Although each individual image captured by the fluorescence imaging system 370 is a two-dimensional image, multiple two-dimensional images captured from different perspectives can be used to infer the projection of anatomical structures or the three-dimensional position of medical instruments within the surgical field of view. To alter the image perspective, the C-arm 380 can, for example... Figure 6B The C-arm 380 is pivotable to capture X-ray images of the patient P and reference marker 400 within the field of view of the fluorescence imaging system 370 from different angles. Figure 6B It shows Figure 6AThe surgical environment 350 shown has the C-arm 380 of the fluorescence imaging system 370 positioned in a second position relative to the patient P and the external tracking system 382. Three-dimensional tomographic images can be generated from two-dimensional projection images produced from different projection angles or viewpoints. By capturing multiple two-dimensional images of the patient P and the reference marker 400 from different perspectives, the three-dimensional positions of individual points within the field of view (e.g., individual points on the reference marker 400) can be determined.

[0078] As described above, the reference marker 400 comprises a radiopaque object placed in the field of view of the fluorescence imaging system 370, having a known configuration in a surgical setting. The reference marker 400 is presented in the fluorescence image to serve as a reference point or measurement point. In the illustrated embodiment, the reference marker 400 comprises a three-dimensional object. In other embodiments, the reference marker may comprise a substantially one-dimensional or two-dimensional object. The reference marker 400 may be placed in a fixed, predetermined position in the X-ray imaging path to achieve an image transformation (which removes distortion from an initial image generated by the X-ray detector 386) or to learn the projection geometry of the imager (i.e., to discern how pixels in the image are projected into three-dimensional space or to map a two-dimensional image of the reference marker to a known three-dimensional model). The reference marker 400 may be a three-dimensional shape presented as a two-dimensional object in the image, but the reference marker 400 may also be constructed using a thin film that is substantially two-dimensional in nature. Many possible shapes, such as, as non-limiting examples, circles, squares, triangles, rectangles, ellipses, cubes, spheres, and cylindrical rods, can be used to design the reference marker. Spheres appear as circles in two-dimensional images, and cylindrical rods appear as lines. Various types of Fiducial Marker designs are described in U.S. Patent Application No. 2010 / 0168562, filed April 23, 2009, entitled "Fiducial Marker Design and Detection for Locating Surgical Instrument in Images," the entire contents of which are incorporated herein by reference.

[0079] Figures 7A to 9B Various exemplary benchmark markers are shown. In some embodiments, such as Figure 7A As shown, the reference marker 400 may include a set of unique marker elements 401, such that each marker element 401a to 401f can be distinguished from another element and from background features. In the illustrated embodiment, each marker element 401a to 401f has a different shape. Alternatively or additionally, such as Figure 7B and Figure 7CAs shown, the reference identifier 400 may include a uniquely identifiable pattern of similar or dissimilar identifier elements. For example, in Figure 7B In this context, the reference marker 402 comprises a grid pattern of identical reference elements 404, each reference element 404 being a white square surrounded by a dark border. Although each reference element cannot be distinguished from another, the grid layout of the reference marker 402 allows the marker to be uniquely identified within the surgical coordinate system. In contrast, in... Figure 7C In this design, the reference marker 410 comprises a checkerboard grid pattern composed of two distinct reference elements: alternating dark squares 412 and light squares 414. Although each reference element is not unique, the checkerboard layout of the reference marker 410 allows the marker to be uniquely identified and located within the surgical coordinate system. Within the grid and checkerboard pattern, each square has a fixed length and width within the surgical environment.

[0080] In some embodiments, the reference identifier 400 may include identifiable identifiers, which include text and / or one or more symbols. Figures 8A to 8E Some exemplary identifiable reference markers are shown, wherein each reference marker 422, 426, 432, 438, and 444 comprises a different set of reference elements, and each reference element comprises a different alphabetic character. For example, reference markers 422 and 444 are formed by a set of different letters or reference elements (e.g., reference element 424 is composed of the letter "E") spelled "INTUITIVE SURGICAL". Similarly, reference marker 426 is formed by reference elements 428 and 430, reference marker 432 is formed by reference elements 434 and 436, and reference marker 438 is formed by reference elements 440 and 442. Reference markers 434, 436, and 444 are each composed of different design patterns to allow for more detailed positioning. Reference markers 422, 426, 432, 438, and 444 each employ the above-mentioned... Figure 7C The reference marker 410 shown is a variation of the checkerboard marker design. Although each reference element may not be unique, the overall shape and checkerboard layout of the reference markers 422, 426, 432, 438, and 444 allow them to be uniquely identified and located within the surgical coordinate system.

[0081] In some embodiments, the reference marker 400 may include a series of similar reference elements arranged in a non-uniform linear pattern. For example, in Figure 9A and Figure 9BIn this embodiment, reference markers 450 and 460 each comprise a series of identical reference markers 452 and 462, arranged in a straight line with non-uniform intervals. In these embodiments, the non-uniform intervals of the reference markers 452 and 462 form uniquely identifiable codes when the reference markers 452 and 462 are imaged. In some embodiments, such codes are similar to conventional (one-dimensional) or matrix (two-dimensional) barcodes.

[0082] In the illustrated embodiment, the reference marker 400 is shown located on the patient P. In other embodiments, the reference marker 400 may be positioned above, beside, or below the patient P (e.g., on platform 352). In one embodiment, for example, Figure 7C The checkerboard reference marker 410 shown may be positioned between patient P and platform 352. In yet another embodiment, reference marker 400 may be found or positioned within patient P. For example, in some embodiments, reference marker 400 may include catheter 360 or shape sensor 364 itself. In such embodiments, catheter 360 and / or shape sensor 364 are at least partially radiopaque. In some embodiments, catheter 360 and / or shape sensor 364 include one or more radiopaque reference elements as described above. In other embodiments, catheter 360 and / or shape sensor 364 are at least continuously radiopaque along their distal portions (e.g., a distal length that can be imaged by fluorescein imaging system 370). Additionally or alternatively, reference marker 400 may include anatomical landmarks of patient P (e.g., one or more specific ribs). In other cases, reference marker 400 may be defined by radiopaque contrast ingested or injected into patient P. Multiple identifiable locations on fluorescein images and the known length of the flexible device containing the shape sensor may also be used as reference marker points. Additionally, images of multiple devices with a common ground point can be combined to generate more reference marker points covering more space in three dimensions, reducing blurring and / or improving the accuracy of calibration and / or attitude estimation. In different embodiments, the entire device may be transmissive, rather than a specific identifiable location on the device. Attitude estimation can be performed by minimizing the back projection error of the three-dimensional curve of the device shape onto the two-dimensional curve observed in the fluorescence microscope image. In the same manner, multiple images of multiple different shapes can be used to reduce blurring and / or improve the accuracy of calibration and / or attitude estimation.

[0083] As mentioned above, obtaining multiple two-dimensional images of an object with known three-dimensional properties (e.g., reference marker 400) from different viewpoints allows the fluorescence imaging system 370 to be more accurately registered to the surgical coordinate space (i.e., by more accurately estimating the pose of the C-arm 380, and thus more accurately determining the perspective of the resulting two-dimensional image). The identification of the reference marker 400 in different imaging viewpoints and the internal calibration of the fluorescence imaging system 370 can be accomplished by using the cross-ratio as a projection invariance. Reference Figure 10 The cross ratio is a number associated with a list of four collinear points, specifically points on the projection line. Given four points A, B, C, and D on the line, their cross ratio is defined as:

[0084]

[0085] The orientation of the lines determines the sign of each distance, and the measured distances are projected onto Euclidean space. Figure 10 The principle of cross ratio is explained, and it is proved that points A, B, C, D and A', B', C', D' are related through projection transformation, such that their cross ratios (A, B; C, D) and (A', B'; C', D') are equal.

[0086] Figure 11 and Figure 12 Two different methods for internal calibration of the fluorescence microscope imaging system 370 using the cross-ratio principle are shown. Figure 11 In the process, the fluorescence imaging system 370 initially projects a two-dimensional fluorescence image (planar projection) using the X-ray imager 386 at a first position P1 to obtain the positions of reference markers A and B at A' and B'. Subsequently, the fluorescence imaging system 370 projects another two-dimensional fluorescence image (planar projection) using the X-ray imager 386 at a second position P2 to obtain the positions of reference markers A and B at A” and B”. If the reference marker 400 is a planar calibration image, then Figure 11 The method shown is particularly useful, in which the imager 386 examines the reference marker 400 from different angles.

[0087] exist Figure 12 In the process, the X-ray imager 386 of the fluorescence imaging system 370 remains stationary and projects a first two-dimensional fluorescence image (planar projection) to obtain the positions of reference markers A and B at A' and B'. Subsequently, with reference markers A and B at a second position P4, the fluorescence imaging system 370 projects another two-dimensional fluorescence image (planar projection) to obtain the positions of reference markers A and B at A” and B”. The four points A', B', A”, and B” obtained in both cases can be used for internal calibration of the fluorescence imaging system 370.

[0088] Figure 13 This is a flowchart illustrating a method 490 for performing image-guided surgery in a surgical setting 350, which involves substantially simultaneously locating / tracking a medical device (e.g., a remotely operated instrument system 200) and calibrating a fluorescence microscopy imaging system 370 in the surgical setting. The method described herein, including method 490, in... Figure 13 The methods are shown as a set of boxes, steps, operations, or procedures. Not all shown, enumerated operations are performable in all embodiments of method 490. Additionally, some additional operations not explicitly shown in the method may be included before, after, between, or as part of the enumerated procedures. Some embodiments of the methods in this specification include instructions corresponding to the procedures of the method stored in memory. These instructions may be executed by a processor, such as a processor of the control system 112.

[0089] As described above, the orientation of the fluorescein imaging system 370 in the surgical coordinate space (e.g., the position of the C-arm 380) is used to accurately determine the perspective of the two-dimensional fluorescein image. Accurate interpretation of the two-dimensional fluorescein image may also include identifying and correcting image distortions. Process 500 describes an exemplary method for calibrating the fluorescein imaging system 370 to correct distortions while determining the orientation of the fluorescein imaging system by imaging a reference marker having known three-dimensional position, shape, and size characteristics in the surgical environment. The calibration process and the generation of a calibration model can be performed prior to reconstructing the instrument orientation. The marker can be imaged by the fluorescein imaging system 370 from multiple different viewpoints. In process 502, the fluorescein imaging system 370 captures fluorescein image data of a reference marker 400 from multiple viewpoints or in multiple orientations, the reference marker 400 having a known configuration (e.g., size, shape, position) in the surgical environment (as described above regarding...). Figure 11 and Figure 12 As will be further described below, the reference marker may include a shape sensor that provides shape and orientation information for a flexible reference in a surgical environment.

[0090] In process 504, control system 112 uses the fluoroscope image data to estimate a set of internal and external parameters of the fluoroscope imaging system 370. For example, the internal parameters of the imaging system 370 include imager calibration parameters such as focal length, projection center, and pixel ratio. The external parameters of the imaging system 370 include imager position parameters such as rotation and translation, and rigidity transformation between the fluoroscope imaging system 370 and the calibration pattern or reference marker 400. In process 506, control system 112 solves an algorithm (e.g., using...) Figures 10 to 12 The cross ratio and projection invariance shown in the figure are used to generate an estimated 3D model of the fluorescence imaging system 370 associated with the surgical environment 350.

[0091] After iteratively imaging reference marker 400 (and other reference markers, if available) from different perspectives (e.g., by iterating through processes 502, 504, and 506), in process 508, the estimated model is refined into a final model of marker(s) in the surgical environment. The calibration model is based on an optimal set of intrinsic and extrinsic parameters of the fluorescence microscope imaging system 370, which corrects for or at least minimizes distortions (including S-shaped distortion) and provides an estimated pose for the imaging system in the surgical environment. Process 500 combines the two-dimensional positions of the references in multiple fluorescence microscope images with known three-dimensional position estimates of the references to reconstruct both the intrinsic and extrinsic parameters. Computer vision methods such as triangulation, beamforming, or simultaneous localization and mapping (SLAM) can be used for reconstruction. In process 510, the calibration model registers the fluorescence microscope image reference frame to the surgical coordinate system, allowing the pose of the marker imaged in the fluorescence microscope reference frame to be determined in the surgical reference frame from the calibration model. Registration of the fluorescence microscope reference frame with the surgical reference frame and the anatomical model reference frame is described in U.S. Provisional Patent Applications Nos. 62 / 205,440 and 62 / 205,433, which are incorporated herein by reference. Optionally, calibration process 500 is supplemented by one or more other various processes. For example, process 512 includes externally tracking the position and orientation of the fluorescence microscope imaging system 370 to supplement the estimation of the orientation of the imaging system 370 in the surgical setting. Reference Figure 6A and Figure 6B The surgical environment 350 also includes an external tracker system 382. Typically, the external tracker system 382 includes external means for estimating the orientation of the fluorescence imaging system 370 and / or the reference marker 400 relative to a surgical coordinate system. Although the external tracker system 382 is depicted as spaced apart from the fluorescence imaging system 370, in some embodiments, the external tracker system 382 may be disposed on the fluorescence imaging system 370. For example, in various embodiments, the external tracker system 382 may include one or more encoders on the joints of the C-arm 380, such as… Figure 6A and Figure 6B As shown by the dashed line 470. Alternatively or additionally, the external tracker system 382 may include one or more inclinometers (e.g., one inclinometer for each rotational degree of freedom of interest) mounted on the C-arm 380, such as Figure 6A and Figure 6B As shown by the dashed line 472.

[0092] Alternatively or additionally, the external tracker system 382 may include an optical tracking system. In the illustrated embodiment, the optical tracking system 382 includes a sensor 480 comprising a pair of cameras capable of emitting and receiving infrared rays reflected by at least one optical marker, the optical marker including a special reflector located on a reference array. In some embodiments, the sensor 480 may track the position of the optical marker disposed on a C-arm 380. Additionally or alternatively, the sensor 480 may track the position of the optical marker disposed on a patient P and / or a medical device (e.g., catheter 360). In some embodiments, the reference marker 400 includes an optical marker and a non-transmissive marker. In some embodiments, the optical marker may be a passive marker comprising a spherical retroreflective marker that reflects infrared light emitted by an illuminator on the sensor 480. In some embodiments, the reference marker may be an active infrared emitting marker activated by an electrical signal. For example, further descriptions of optical tracking systems are provided in U.S. Patent No. 6,288,783, filed October 28, 1999, disclosing "System for determining spatial position and / or orientation of one or more objects," and in U.S. Patent Application No. 62 / 216,494, filed September 10, 2015, disclosing "Systems and Methods For Using Optical Tracking in Image-Guided Surgery," the entire contents of which are incorporated herein by reference. In some embodiments, such as Figure 6A As shown, when the device body moves the flexible catheter 360 into or out of the patient's anatomy, the optical tracking sensor 480 tracks a set of optical reference markers attached to the reference marker 400 and / or the medical device. In other embodiments, the external tracker system 382 can be used to estimate the posture of the fluorescence imaging system 370, independent of the internal calibration process described above with respect to process 500 (which relies on the two-dimensional fluorescence image obtained from the reference marker 400).

[0093] After calibrating the fluorescein imaging system and positioning the markers in the surgical setting, as described above, new fluorescein images can be registered to a surgical reference system. In process 514, during medical procedures, fluorescein images including the reference marker 400 and catheter 360 are captured in the fluorescein reference system. In process 516, based on the fluorescein images, the position and orientation of one or more portions of the catheter (e.g., the distal tip portion) in the surgical reference system are determined by the registration from the surgical reference system to the fluorescein reference system. More specifically, when receiving fluorescein images including the reference marker and the distal tip of the catheter in the same reference system, a calibration model (with optional supplemental pose tracking of the fluorescein system) corrects for distortion and provides an estimated pose of the fluorescein imager. Based on the calibration model, the pose of the distal tip of the catheter in the surgical reference system can be determined.

[0094] As mentioned above Figure 4 The method 300 for use in image-guided surgical procedures includes positioning a medical device relative to a patient in a surgical setting. Therefore, some embodiments of method 490 may include process 492, which includes obtaining prior image data of preoperative or intraoperative image data from imaging techniques such as CT, MRI, thermal infrared imaging, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The prior image data may correspond to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information). As described above, an anatomical model is generated from the prior image data. In process 494, a surgical reference frame (X-ray) is established. S Y S Z S Registered to the instrument sensor reference frame (X) I Y I Z I Such registration between the model and instrument reference frame can be achieved, for example, using point-based ICP techniques, as described in U.S. Provisional Patent Applications Nos. 62 / 205,440 and 62 / 205,433, which are incorporated herein by reference. Alternatively or additionally, the model and sensor reference frame can be registered to another common reference frame. This common reference frame can be, for example, a surgical environment reference frame (X). S Y S Z S Or a patient reference system. For example, the reference portion of an instrument sensor may be fixed, known, or tracked within the surgical environment.

[0095] In procedure 518, the user or remote operating system may use the 3D model and / or fluorescence microscopy image data for guidance to perform a medical procedure with the medical device, since both the 3D model and the fluorescence microscopy image data are registered to a surgical reference frame or another common reference frame. Procedures 514 to 516 may be repeated throughout the medical procedure as the medical device is inserted or otherwise moved within the patient's anatomy to provide information on the current positioning of the device relative to the anatomy and the anatomical model.

[0096] Optionally, shape sensor data from shape sensor 364 can be used to supplement the fluorescence microscope data and internal calibration process described in process 500. The shape sensor data describes the shape of the sensor extending between a known reference point 366 on the rigid instrument body and the distal end of the flexible catheter 360. In some embodiments, as described above, shape sensor 364 itself can serve as a reference marker 400 because the shape sensor data provides shape characteristics of the sensor (and its extended catheter), including the posture of the distal tip and a known reference frame relative to the surgical reference frame. Obtaining two-dimensional imaging data about a reference marker 400, such as shape sensor 364 having a known three-dimensional shape, allows control system 112 to more accurately position the reference marker 400 within the surgical environment 350. When using a shape sensor as a reference marker, it can be used as a single reference marker or can be included among multiple one-dimensional, two-dimensional, and three-dimensional reference markers present in the surgical environment. In some embodiments, the shape of the medical device or catheter 360 itself is at least partially transmissive and serves as a reference marker 400. More specifically, shape information combined with the position of reference point 366 in the surgical environment (e.g., via kinematic and / or optical tracking sensor 480) provides the position and orientation of the distal tip of the flexible catheter 360 and other points along the catheter in the surgical environment. If the patient is also tracked by optical tracking sensor 480, the position and orientation of the patient P are also known in the surgical environment. Therefore, the position and orientation of the flexible catheter 360 relative to the patient's anatomy in the surgical environment reference frame are known.

[0097] Fluoroscopy image data from multiple viewpoints within a surgical environment (i.e., a fluoroscopy imager that moves between multiple locations) can be compiled to generate two-dimensional or three-dimensional tomographic images. When using a fluoroscopy imager system including a digital detector (e.g., a flat panel detector), the generated and compiled fluoroscopy image data allows for the cutting of planar images in a parallel plane according to tomographic synthesis imaging techniques. Conventional fluoroscopy imager motion has been restricted to planar motion (e.g., lines or curves). For example, the fluoroscopy imager may be constrained to linear movement along a linear track or to track movement defined by a C-arm. According to embodiments of this disclosure, two-dimensional or three-dimensional motion of the fluoroscopy imager in a surgical environment produces fluoroscopy image data that provides more detail about regions of interest in the patient's anatomy compared to image data generated by a fluoroscopy imager constrained to planar motion. Utilizing the posture of the fluoroscopy imager tracked according to any of the foregoing methods, the fluoroscopy imager can move around in a three-dimensional surgical space to capture image data combined with tomographic imaging to generate optimal images of regions of interest in the patient's anatomy.

[0098] Due to constraints within the surgical environment (e.g., space, time, and cost), tomographic imaging techniques using synthetic images from a range substantially smaller than that of a fluorescein imager can be used to prepare tomographic images. For example, if the full scan range of motion of a C-arm fluorescein imager is approximately 200° for automated scanning, the spatial constraints within the surgical environment may limit the imager's travel within that 200° full scan range due to the presence of a remote operating system. However, useful tomographic images can be constructed by manually swinging the imager within a constrained range (which defines the constrained scan area), a constrained range substantially smaller than 200°. In one embodiment using the constrained range, the imager is swung or otherwise moved to one or more locations within a range of approximately 30°, including locations within the in-plane trajectory of the C-arm and, optionally, locations outside the usual plane of the C-arm trajectory.

[0099] Figure 14 The X-plane of the surgical environment is shown. S Y S The patient's anatomical structure and tomographic arrangement are shown in Figure 550. Figure 15 The orthogonal plane Y in the surgical environment is shown. S Z S In Figure 14 The patient's anatomical structures and tomographic arrangements were analyzed. The X-ray imager had a full scanning range (R) around the patient's motion. Figure 14As shown, patient P is imaged, with the X-ray source moved within a constraint range to positions C1, C2, C3, C4, and C5, which are substantially smaller than the full scan range R. In this embodiment, the constraint range (within the X-ray range) is... S Y S (In the plane) approximately 30° or less than 20% of the full scan range. Patient P is imaged onto image plane I1 at position C1. The X-ray source is in X... S Y S The X-ray source is rotated, swung, or otherwise manually moved to position C2 in a plane (e.g., a first motion plane) to image the patient P onto the image plane I2. The X-ray source is further... S Y S The patient P is manually moved, rotated, swung, or otherwise moved in the plane to position C3 to image the patient P onto image plane I3. Images I1, I2, and I3 can be mapped and accumulated onto a single tomographic plane T1 by using the images themselves or after filtering using a planar projection transformation (i.e., homography) uniquely defined by the estimated pose of the X-ray source and calibrated internal parameters. The image reconstruction algorithm used to generate slice images at plane T1 blurs or otherwise minimizes features outside plane T1. The tomographic plane T1 can be selected based on the location of the anatomical region of interest or the location of the medical device. The same image data of images I1, I2, and I3 can also be mapped, or alternatively, to generate slice images in multiple tomographic planes such as plane T2 or plane T3. In this embodiment, the movement of the imager can be performed manually, but in other embodiments, the movement can be programmed or otherwise computer-controlled. Figure 15 As shown more clearly in the diagram, X-ray sources can also be derived from X... S Y S The plane is moved out into another motion plane within the constraint range. As the X-ray source moves to position C4, the patient is imaged onto image plane I4. As the X-ray source moves to position C5, the patient is imaged onto image plane I5. Image data from images I1, I2, I3, I4, and I5 can be mapped to a single tomographic plane T4 or a single tomographic plane T5. Planes T4 and T5 can be tilted relative to each other and plane T1. In an alternative embodiment, the X-ray imager at positions C4 and C5 can be rotated to obtain multiple images across an angular range for calculating a tomographic composite image. Images can be obtained at discrete locations, but optionally, frames from continuous fluorescence mirror video can be used for reconstruction.

[0100] Figure 16A method 580 is illustrated for image-guided medical intervention by reconstructing a tomographic composite image using images obtained from multiple three-dimensional viewpoints of a fluorescein system in a surgical setting. As previously described, the posture and position of the fluorescein system in the surgical setting can be tracked using fluorescein markers in the fluorescein images and / or external trackers (such as optical tracking systems or joint trackers for joints of the fluorescein system, e.g., encoders)). Therefore, the current or desired posture of the fluorescein imager in the surgical setting can be known or specified. The methods of this specification, including method 580, are described in... Figure 16 The methods are shown as a set of boxes, steps, operations, or procedures. Not all of the listed operations shown can be performed in all embodiments of method 580. Additionally, some additional operations not explicitly shown in the method may be included before, after, between, or as part of the listed procedures. Some embodiments of the methods in this specification include instructions corresponding to the procedures of the method stored in memory. These instructions may be executed by a processor, such as a processor of the control system 112.

[0101] In process 582, fluorescence microscopy image data of the patient's anatomy is received from a fluorescence microscopy imager (e.g., imaging system 370) located at positions C1, C2, C3 within the surgical coordinate space. In process 584, a tomographic image T1 is constructed from the fluorescence microscopy image data. The image can be displayed to the clinician in real time during the medical procedure for the patient P. If the image lacks sufficient detail of the region of interest, supplementary tomographic composite planar images may be useful. The clinician can identify the region of interest (e.g., the area where the tumor is located) by selecting a region of the image, for example, using an operator input device such as a touchscreen, mouse, or trackball. Alternatively, the region of interest for generating additional planar tomographic composite images can be selected by identifying the region of interest in a preoperative CT scan registered to the surgical environment. Alternatively, the region of interest for generating additional planar tomographic composite images can be selected by determining the location of a portion of a medical instrument (e.g., the distal tip) in the registered surgical environment and the trajectory of an instrument extending from it (e.g., a biopsy needle).

[0102] In process 586, based on the identified region of interest, instructions can be generated to move the fluorescein imager to another pose, pose group, or imaging angle range in the three-dimensional surgical coordinate space. After or during movement, the pose of the fluorescein imager can be tracked using fluorescein markers in the fluorescein image and / or external trackers (such as optical tracking systems or joint trackers for the joints of the fluorescein system, e.g., encoders) as previously described. The fluorescein imager is not limited to movement in a single plane. Instead, the movement of the fluorescein imager is unconstrained, allowing the imager to maintain a pose in a configuration that provides optimal imaging data. The instructions can be control signals that instruct the fluorescein imager's control system to move the fluorescein imager to a specified pose in the surgical coordinate system. Alternatively, instructions can be displayed to or otherwise communicated to the operator to manually move the fluorescein imager to a specified pose in the surgical coordinate system. The fluorescein imager can move freely about the surgical coordinate system and is not constrained to linear or orbital movement in a single plane. Alternatively, the movement instructions may include instructions for repositioning the imaging system 370 and obtaining multiple fluorescein images in a single plane from the repositioned location. For example, the C-arm may be moved to a new location based on the instructions, and the C-arm may be rotated to generate multiple images from the plane of rotation.

[0103] In procedure 588, additional fluorescein image data of the patient's anatomy can be received from a fluorescein imager located in a specified pose or group of poses within the surgical coordinate space. For example, the region of interest may be in plane T4, so the imager can be moved to position C4 to generate additional fluorescein image data. In procedure 590, a second-planar tomographic image can be constructed from the second fluorescein image data. The second fluorescein image data may be generated from a single image or from multiple images. In various embodiments, the second-planar tomographic image may be constructed from second fluorescein image data combined with the first fluorescein image data. Reconstruction of the desired structure can be achieved through repeated and sequential processing, in which new images are added and accumulated. Thus, procedure steps similar to 586, 588, and 590 can be repeated until the resulting tomographic composite image includes the desired structure, such as a tumor, duct, or pleural boundary. The pose history of the fluorescein imager can be tracked via a display or other user interface and provided to the user, allowing the user to know which poses and / or angular ranges in the surgical environment have been covered.

[0104] Although the systems and methods of this disclosure have been described for use in bronchial pathways connecting to the lungs, they are also suitable for navigating and treating other tissues in any of the various anatomical systems, including the colon, intestine, kidney, brain, heart, circulatory system, etc., via naturally or surgically created pathways.

[0105] Furthermore, while this disclosure describes various systems and methods for use in remote operating systems, they are also contemplated for use in non-remote operating systems in which manipulator components and instruments are directly controlled. Although the various examples provided describe the use of procedures performed within an anatomical structure, in alternative embodiments, the devices and methods of this disclosure need not be used within an anatomical structure but may also be used outside the patient's anatomy.

[0106] One or more elements in embodiments of the present invention can be implemented in software to execute on a processor of a computer system, such as control system 112. When implemented in software, the elements of embodiments of the present invention are essentially code segments to perform necessary tasks. The program or code segment can be stored in a processor-readable storage medium or device that can be downloaded via computer data signals embodied on a carrier wave over a transmission medium or communication link. A processor-readable storage device can include any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment can be downloaded via a computer network such as the Internet, intranet, etc.

[0107] Note that the processes and displays presented are not inherently associated with any particular computer or other device. The various structures required for these systems will appear as elements in the claims. Furthermore, embodiments of the invention are described without reference to any particular programming language. It should be understood that the teachings of the invention as described herein can be implemented using various programming languages.

[0108] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that such embodiments are merely illustrative and not limiting, and that embodiments of the invention are not limited to the specific structures and arrangements shown and described, as various other modifications will be apparent to those skilled in the art.

Claims

1. A method performed by a computing system, comprising: Receive first fluorescence mirror image data of reference markers in the surgical coordinate space from a fluorescence mirror imager with a first set of parameters; Receive the configuration of the reference markers within the surgical coordinate space; Based on the configuration of the first fluorescence mirror image data and the reference markers, a second set of parameters for the fluorescence mirror imager in the surgical coordinate space is determined. as well as The pose of the reference marker in the surgical coordinate space is determined based on the second set of parameters of the fluorescence microscope imager.

2. The method of claim 1, wherein determining the second set of parameters comprises developing a calibration model of the reference marker in the surgical coordinate space from the first fluorescence microscopy image data of the reference marker and the configuration.

3. The method of claim 2, wherein developing the calibration model includes determining the internal and external parameters of the fluorescence mirror imager.

4. The method according to claim 2, further comprising: Receive second fluorescence mirror image data of the reference marker and a portion of the medical device from the fluorescence mirror imager; The calibration model is used to correct distortions in the second fluorescence mirror image data; and Based on the second set of parameters of the fluorescence microscope imager, the portion of the medical device is located in the surgical coordinate space relative to the surgical coordinate space.

5. The method according to claim 2, further comprising: Shape information is received from a shape sensor included in the reference marker to determine the configuration of the reference marker within the surgical coordinate space.

6. The method according to claim 5, wherein the shape sensor is an optical fiber sensor.

7. The method of claim 5, wherein the reference marker comprises a flexible conduit, and the shape sensor extends within the flexible conduit.

8. The method according to claim 1, further comprising registering the reference frame of the fluorescence microscope imager to the surgical coordinate reference frame of the surgical coordinate space.

9. The method of claim 8, further comprising determining the orientation of the fluorescence microscope imager in the surgical coordinate reference system based on the registration.

10. The method of claim 1, wherein the first fluorescein image data comprises a plurality of fluorescein images received from the fluorescein imager.

11. The method of claim 1, wherein the first fluorescence microscopy image data comprises images of a plurality of reference markers, the plurality of reference markers comprising the reference markers, the plurality of reference markers having a known configuration in the surgical coordinate space.

12. The method of claim 11, wherein the plurality of reference markers includes barcodes.

13. The method of claim 11, wherein the plurality of reference markers comprises a grid pattern.

14. The method of claim 1, wherein the reference marker is included in a medical device, the medical device comprising an elongated flexible body.

15. The method of claim 1, wherein determining the second set of parameters of the fluorescence mirror imager includes receiving external tracking data of the fluorescence mirror imager.

16. The method of claim 15, wherein the fluorescence mirror imager includes a movable C-arm.

17. The method of claim 16, wherein the external tracking data is received from an encoder located on the movable C-arm.

18. The method of claim 16, wherein the external tracking data is received from a tiltmeter located on the movable C-arm.

19. The method of claim 15, wherein the external tracking data is received from an optical tracking sensor configured to track the position of an optical reference located on the fluorescent mirror imager.

Images