System and method for local three-dimensional volume reconstruction using standard fluoroscope

Abstract

The invention is entitled "System and method for local three-dimensional volume reconstruction using standard fluoroscope." A system for constructing fluoroscope-based three-dimensional volume data of a target region within a patient from two-dimensional fluoroscope images, including a marker structure, a fluoroscope imaging device configured to acquire a sequence of images of the target region and the marker structure, and a computing device. The computing device is configured to estimate a pose of the fluoroscope imaging device for at least a plurality of images of the sequence of images based on detection of a most likely and maximum probability of a projection of the marker structure as a whole on each of the plurality of images. The computing device is further configured to construct fluoroscope-based three-dimensional volume data of the target region based on the estimated pose of the fluoroscope imaging device.

Description

[0001] This application is a continuation-in-part of U.S. Patent Application No. 16 / 885, 1 10, filed February 2, 2020, entitled "Systems and Methods for Localized Three-Dimensional Volume Reconstruction Using Standard Fluoroscopes." TECHNICAL FIELD

[0002] The present disclosure relates to systems, devices, and methods for navigation, location confirmation, and location correction of surgical procedures. More particularly, the present disclosure relates to systems and methods for constructing a fluoroscope-based three-dimensional volume from two-dimensional fluoroscope images captured using standard fluoroscope imaging equipment and utilizing the fluoroscope-based three-dimensional volume to determine in real-time the location of a medical device relative to a target within a patient. BACKGROUND

[0003] There are several commonly used medical procedures, such as endoscopic procedures or minimally invasive procedures, for treating various diseases affecting organs, including the liver, brain, heart, lung, gallbladder, kidney, and skeleton. Typically, a clinician employs one or more imaging modalities, such as magnetic resonance imaging, ultrasound imaging, computed tomography (CT), and others, to identify a region of interest within a patient and ultimately identify a treatment target.

[0004] For example, endoscopic methods have proven useful for navigating to regions of interest within a patient, and in particular, to regions within a body luminal network, such as the lungs. To implement endoscopic (and more particularly, bronchoscopic) methods in the lungs, bronchoscopic navigation systems have been developed that use previously acquired MRI data or CT image data to generate a three-dimensional rendering or volume of a particular body portion, such as the lungs. In particular, a three-dimensional or volume rendering of the patient is generated with previously acquired images from a MRI scan or CT scan of the patient.

[0005] The resulting volume generated from the MRI scan or CT scan is then utilized to create a navigation plan to facilitate navigation of a catheter (or other suitable device) through the bronchoscope and patient bronchial branches (e.g., manually or via a console when performing robotic surgery) to a region of interest. Electromagnetic tracking can be used in conjunction with the CT data to facilitate navigation of the catheter through the bronchial branches to guide to a region of interest. In some cases, the navigation catheter can be positioned within one of the airways of the branched luminal network near or within the region of interest to provide access for one or more medical instruments.

[0006] As another example, minimally invasive procedures, such as laparoscopic procedures, including robotic-assisted surgery, can employ intraoperative fluoroscopy to increase visualization (e.g., for guidance and lesion localization) or to prevent injury and complications.

[0007] To generate a navigation plan or to even generate a three-dimensional or volume rendering of a patient's anatomy, such as a lung, a clinician needs to acquire the necessary image data for a three-dimensional volume construction with an MRI system or a CT system. MRI systems or CT-based imaging systems are extremely expensive and in many cases not available in the same location as where the navigation plan is generated or the navigation procedure is performed.

[0008] Fluoroscopic imaging equipment is typically located in the operating room during a navigation procedure. A clinician can use the standard fluoroscopic imaging equipment to visualize and confirm the placement of a tool or medical device after it has been navigated to a desired location. However, while standard fluoroscopic images show highly dense objects, such as metal tools and bones, as well as large soft tissue objects, such as the heart, the fluoroscopic images have difficulty resolving small soft tissue objects of interest, such as lesions. Furthermore, the fluoroscopic images are only two-dimensional projections. To be able to see small soft tissue objects in three-dimensional space, X-ray volume reconstruction is needed. There are several solutions that provide three-dimensional volume reconstruction of soft tissue, such as CT and cone-beam CT, which are widely used in the medical field. These machines algorithmically combine multiple X-ray projections from a known calibrated X-ray source position into a three-dimensional volume where soft tissue is visible.

[0009] To navigate a tool to a distal soft tissue target for biopsy or treatment, both the tool and the target should be visible in some three-dimensional guidance system. Most of these systems use some kind of X-ray equipment to see through the body. For example, a CT machine can be used during a procedure with iterative scans to provide guidance through the body until the tool reaches the target. This is a lengthy procedure because it requires several full CT scans, a dedicated CT room, and instrument navigation between scans. In addition, the staff needs to leave the CT room for each scan. Another option is a cone-beam CT machine, which is available in some operating rooms and is easier to operate, but is expensive and, like CT, only provides instrument navigation between scans, requires multiple iterations of navigation, and requires the staff to leave the CT room.

[0010] Accordingly, there is a need for a system that can achieve the benefits of CT and cone-beam CT three-dimensional image guidance without the potential costs, preparation requirements, and radiation side effects associated with these systems. Furthermore, there is a need for systems and methods that facilitate relatively accurate navigation of medical devices relative to targets and, in particular, effective treatment of targets. SUMMARY

[0011] The present disclosure relates to systems and methods that construct three-dimensional volume data of small soft tissue objects visible from video streams (or multiple images) captured by standard fluoroscopic imaging equipment available in most procedure rooms. The fluoroscopy-based constructed three-dimensional volume data can be used for guidance, navigation planning, improved navigation accuracy, navigation confirmation, and treatment confirmation.

[0012] Soft tissue objects are not visible in standard fluoroscopic images and videos because they are obscured by dense objects such as dense tissue. The present disclosure relates to systems and methods for creating three-dimensional pseudo-volumes where obscuring objects are filtered based on three-dimensional position and then projected back into two dimensions. In one aspect, multiple fluoroscopic images are utilized (each fluoroscopic image captured at a different angle) to construct three-dimensional pseudo-volume data. The present disclosure describes systems and methods that are capable of constructing three-dimensional pseudo-volume data with fluoroscopic images captured from a short range of angles relative to a patient or a target region of a patient. Additionally, the present disclosure also relates to systems and methods that utilize two-dimensional images or videos captured with standard fluoroscopic imaging equipment to improve previously created three-dimensional renderings.

[0013] As described in greater detail below, one aspect of the present disclosure is to determine the three-dimensional position of features in a fluoroscopic video, such as three-dimensional catheter position, three-dimensional target tissue (e.g., lesion) position, etc. To accomplish this, the pose of the fluoroscopic imaging equipment for each frame must be determined or known. If an external angle measurement device is coupled to the fluoroscopic imaging equipment, the angle and pose of the imaging equipment for each frame can be known. However, when an external measurement device is not utilized, other techniques are employed to determine the pose of the fluoroscopic imaging equipment for each frame, as described in greater detail below. For example, previously acquired CT scan data can be registered to the fluoroscopic video in order to algorithmically find the pose of the fluoroscope for each frame of the captured video. Alternatively, by tracking some visible markers (two-dimensional visible features) in the fluoroscopic video, the pose and three-dimensional position can be solved together by using some sort of motion recovery structure technique. In some aspects, a structure of markers placed outside of the patient can be utilized. The pose of the fluoroscope can then be estimated based on detection of the most likely and maximum probability projection of the structure of markers as a whole on each of the plurality of images. In some aspects, a simpler technique of knowing the pose (through an angle measurement device) can be utilized and only the position of the three-dimensional features need to be solved. To correct for movement, at least one marker (or surgical equipment, such as a catheter tip) can be utilized.

[0014] A number of fluoroscopic two-dimensional images can be algorithmically processed to create a pseudo three-dimensional volume data, similar to cone-beam CT, but with a different arbitrary angular range and fluoroscope poses, where the fluoroscope is manually rotated. The angular range can be very small (about 30°), which can result in poor three-dimensional reconstruction quality. The algorithm used is an iterative back-projection / forward-projection method, which is different from analytical algorithms (e.g., Radon transform, FDK) and does not assume any predetermined angular range or angular rotation rate. To overcome the poor three-dimensional reconstruction, instead of displaying the raw three-dimensional reconstruction data to the user, the three-dimensional reconstruction can be cropped around a region of interest (also referred to herein as a "FluoroCT Blob"). The cropped three-dimensional data can then be re-projected into two-dimensional virtual fluoroscopic images, where in particular local soft tissue features are visible. The re-projected two-dimensional images are especially of good quality (compared to the poor three-dimensional reconstruction) if projected from the same fluoroscope pose as seen in the video.

[0015] To reconstruct three-dimensional data, the pose of the fluoroscope imaging device must be determined for each two-dimensional fluoroscopic frame in the video relative to some fixed coordinate system. The pose of the fluoroscope imaging device for each frame captured can be determined using any of the methods described below. For example, an external measuring device coupled to the fluoroscope imaging device or by utilizing a structure of markers arranged outside the patient's body and relative to the target can be used to determine the pose.

[0016] Additionally or alternatively, the pose can be determined using an external measuring device and a single marker or catheter tip. In particular, in some cases the fluoroscope (or C-arm) can be jittery during rotation, in which case some stabilization is required. Unfortunately, due to filtering, the angle measuring device can still report a smooth angle throughout the video, ignoring the high frequencies present in the actual camera pose. In this case, a single two-dimensional marker can be tracked throughout the video and used to stabilize the camera pose or increase their precision at the marker's region. Since the marker will typically be located at the region of interest, this increases the camera pose precision in that region, thus improving the three-dimensional reconstruction quality. This method can also be used to compensate for patient body movement during the video, such as breathing. Instead of using the camera pose as reported by the angle measuring device, the compensated camera pose is calculated using the tracked two-dimensional marker, such that all camera poses are correct relative to the marker. The single marker used can be the tip of a tool or catheter currently inserted into the patient.

[0017] Additionally or alternatively, the pose can be determined via registration of the fluoroscopy video with previously acquired CT data. In particular, a previously acquired CT of the patient can be available. In this case, each frame of the fluoroscopy video can be registered to a virtual fluoroscopy frame of the CT (searching for the camera pose in the CT space until the virtual fluoroscopy image corresponding to the camera pose matches the camera pose seen in the video). In this way, camera pose is achieved using image-based feature matching.

[0018] Additionally or alternatively, the pose can be determined via structure of a marker. In particular, the structure of the marker is positioned outside the patient, e.g. under the patient, when the images are captured. The structure of the marker is further positioned such that each image includes a projection of at least a portion of the structure. A probability map can then be generated for each image, indicating for each pixel in the image the probability that it is a marker of the structure of the marker. A plurality of virtual candidates of the projection of the structure of the marker on the image can be generated by virtually positioning the fluoroscope at different possible positions, including possible orientations. A candidate with the highest probability of being the projection of the structure of the marker on the image can then be identified based on the probability map. The virtual pose of the fluoroscope associated with the identified candidate can then be determined as the estimated pose of the fluoroscope at the time the image was captured. Optionally, the process of identifying the candidate can be refined. A locally deformed version of the candidate can be generated based on the probability map, so as to maximize the probability that it is the projection of the structure of the marker on the image. The new virtual candidate can then be fitted to the locally deformed version of the identified candidate. The virtual pose of the fluoroscope generating the new improved candidate is computed and determined as the estimated pose of the fluoroscope at the time the image was captured.

[0019] Additionally or alternatively, the camera pose can be determined using motion reconstruction structure techniques. In particular, if a plurality of two-dimensional features can be tracked throughout the two-dimensional video frames from start to end, these two-dimensional features can be used to achieve the camera pose of each frame (as well as the three-dimensional feature positions). These features can be artificial markers introduced to the patient during the procedure.

[0020] Filtering of obscuring tissue from the tissue of interest can be done by cropping at a certain distance from the center of the generated three-dimensional pseudo-volume data, cropping at a certain distance from the tool / catheter, or registering to previously obtained three-dimensional volume data (CT) and using it to know which objects to filter.

[0021] Once the soft tissue, such as the target lesion, is visible in the three-dimensional reconstruction data (or in the two-dimensional enhanced Fluoro CT images), all three-dimensional information in the fluoroscope coordinates is available. When using the original three-dimensional reconstruction data, the three-dimensional information is available directly from this data. Alternatively, when using the two-dimensional enhanced Fluoro CT images, a 2- angle marker is needed to achieve three-dimensional position (triangulation). In some embodiments, the three-dimensional reconstruction data is divided into slices that are thin enough that three-dimensional position data is available without triangulation.

[0022] The three-dimensional data obtained can be used to determine or confirm in real time the three-dimensional relationship of the tool to the soft tissue target. For example, this data can be used to determine if the tool has reached the target, the orientation of the tool relative to the target, the distance between the tool and the target, or if the target is within the ablation zone of the tool.

[0023] Additionally or alternatively, this three-dimensional data can be used for correction of navigation. For example, these three-dimensional positions can be transformed from fluoroscope coordinates to navigation system coordinates to improve the navigation system accuracy at the region of interest. In one aspect, when used in an EMN system, the fluoroscope coordinates can be transformed to antenna coordinates by assuming that the C-arm is perfectly perpendicular to the antenna and matching the catheter tip seen in the fluoroscope video to the catheter position in the antenna at the time the video was taken. Registration of the fluoroscope to / from the antenna can also be achieved by using the earth's magnetic field to calculate the angle between the C-arm and the antenna, attaching an EMN sensor to the fluoroscope imaging device, or aligning known two-dimensional features of the antenna.

[0024] Aspects of the disclosure are described in detail in the drawings and specification set out below, in which like reference characters refer to like or similar elements. As used herein, the term "distal" refers to the portion of the description that is further from the user, while the term "proximal" refers to the portion of the description that is closer to the user.

[0025] According to one aspect of the present disclosure, a system for constructing fluoroscopy-based three-dimensional volume data from two-dimensional fluoroscopic images is provided. The system includes a computing device configured to facilitate navigation of a medical device to a target region within a patient; and a fluoroscopic imaging device configured to acquire fluoroscopic videos of the target region at a plurality of angles relative to the target region. The computing device is configured to determine a pose of the fluoroscopic imaging device for each frame of the fluoroscopic videos and construct fluoroscopy-based three-dimensional volume data of the target region using a fast iterative three-dimensional construction algorithm, where particularly soft tissue objects are visible. In aspects, the system is configured to determine the pose of the fluoroscopic imaging device by learning an angular range of movement of the fluoroscopic imaging device and calculating a relative rotational velocity of the device along the range. In some aspects, the computing device is configured to determine the pose of the fluoroscopic imaging device for at least a plurality of frames of the fluoroscopic videos via a structure of at least partially radiopaque markers. In some aspects, the computing device is configured to construct the fluoroscopy-based three-dimensional volume data of the target region using a three-dimensional construction algorithm as known in the art.

[0026] The medical device or tool can be a catheter assembly including an extended working channel configured to be positioned within a luminal network of a patient, or the medical device can be a radiopaque marker configured to be placed within a target region. The radiopaque marker is at least partially visible in the acquired fluoroscopic videos. The medical device can be a surgical instrument for use in minimally invasive surgery, including robotic minimally invasive surgery.

[0027] The computing device can be further configured to create a virtual fluoroscopic image of the patient from previously acquired CT volume data and register the generated virtual fluoroscopic image with the acquired fluoroscopic videos. The pose of the fluoroscopic imaging device for each frame of the fluoroscopic videos can be determined based on the registration between the fluoroscopic videos and the virtual fluoroscopic image.

[0028] The computing device can be further configured to detect missing frames in the fluoroscopic videos and supplement the detected missing frames with corresponding virtual fluoroscopic images. The fluoroscopy-based three-dimensional volume data can be constructed based on the fluoroscopic videos and the corresponding virtual fluoroscopic images. In one aspect, the fluoroscopy-based three-dimensional volume data can be registered with previously acquired CT data using image-based techniques such as "mutual information." The fluoroscopy-based three-dimensional volume data can be registered globally to the previously acquired CT data or locally near a target region of interest. Deep learning-based methods can be utilized where the computing device "learns" many examples of suitable and unsuitable registrations and learns how to register the two different modalities.

[0029] Additionally, the computing device can be further configured to track a two-dimensional position or orientation of the medical device navigated to the target region throughout the fluoroscopic video. The computing device can be further configured to reconstruct the position of the medical device using a motion reconstruction structure technique throughout the fluoroscopic video. The pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined based on the reconstructed position. Additionally or alternatively, the pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined based on an external angle measurement device. The external angle measurement device can include an accelerometer, a gyroscope, or a magnetic field sensor coupled to the fluoroscopic imaging device. Additionally or alternatively, the pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined via a structure of markers positioned outside of the patient's body at the time the image is captured. Additionally, in aspects, the computing device can be configured to synchronize the captured frames of the target region and compensate for shifts of the fluoroscopic imaging device or patient movement to correct the construction of the fluoroscopic-based three-dimensional volume data. Additionally or alternatively, the computing device can be configured to crop a region of interest from the fluoroscopic-based three-dimensional volume data, project the cropped region of interest onto the captured frames, and sharpen or enhance at least one of the region of interest or the captured frames to identify soft tissue objects or any other objects of interest.

[0030] In yet another aspect of the disclosure, a method of constructing fluoroscopic-based three-dimensional volume data from two-dimensional fluoroscopic images is provided. The method includes navigating a medical device to a target region within a patient, acquiring a fluoroscopic video of the target region using a fluoroscopic imaging device around a plurality of angles relative to the target region, determining a pose of the fluoroscopic imaging device for each frame of the fluoroscopic video, and constructing fluoroscopic-based three-dimensional volume data of the target region visible to soft tissue objects using a fast iterative three-dimensional construction algorithm. The medical device can be a catheter assembly including an extended working channel configured to be positioned within a luminal network of the patient, or the medical device can be a radiopaque marker configured to be placed within the target region. The radiopaque marker is at least partially visible in the acquired fluoroscopic video.

[0031] The method can further include creating a virtual fluoroscopic image of the patient from previously acquired CT volume data, and registering the fluoroscopic video with the virtual fluoroscopic image, wherein determining the pose of the fluoroscopic imaging device for each frame of the fluoroscopic video is based on the registration between the fluoroscopic video and the virtual fluoroscopic image. The method can further include detecting missing frames in the fluoroscopic video, and supplementing the detected missing frames with corresponding virtual fluoroscopic images. Additionally, in aspects of the disclosure, the method can further include tracking a two-dimensional position or orientation of the medical device navigated to the target region throughout the fluoroscopic video.

[0032] The position of the medical device throughout the fluoroscopic video can be reconstructed using motion reconstruction techniques. The pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined based on the reconstructed position. Additionally or alternatively, the pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined based on an external angle measurement device. The external angle measurement device can include an accelerometer, a gyroscope, or a magnetic field sensor coupled to the fluoroscopic imaging device. Additionally or alternatively, the pose of the fluoroscopic imaging device for each frame of the fluoroscopic video can be determined via a structure of at least partially radiopaque markers positioned outside of the patient's body relative to the target region at the time the fluoroscopic images are captured. Additionally, in aspects, the method can include synchronizing the captured frames of the target region and compensating for shifts of the fluoroscopic imaging device or patient movement to correct for construction of the fluoroscopic-based three-dimensional volume data. Additionally or alternatively, the method can include cropping a region of interest from the fluoroscopic-based three-dimensional volume data, projecting the cropped region of interest on the captured frames, and sharpening or enhancing at least one of the region of interest or the captured frames to identify soft tissue objects. BRIEF DESCRIPTION OF DRAWINGS

[0033] Various aspects and embodiments of the present disclosure are described below in connection with the drawings, in which:

[0034] Figure 1 is a perspective view of one exemplary embodiment of an electromagnetic navigation (EMN) system incorporating a fluoroscopic imaging device in accordance with the present disclosure;

[0035] Figure 2 shows a model of a fluoroscopic imaging device;

[0036] Figure 3A is a flowchart of a method of constructing a three-dimensional volume using a plurality of radiopaque markers;

[0037] Figure 3B is an illustration of an example of a frame of a fluoroscopic video captured by a fluoroscopic imaging device in accordance with the present disclosure, showing markers and an extended working channel of a catheter assembly positioned within a target region of a patient;

[0038] Figure 4 is a flowchart of a method of constructing a three-dimensional volume using a single radiopaque marker or tip of an extended working channel of a catheter assembly;

[0039] Figure 5 is a flowchart of a method of constructing a three-dimensional volume using a single radiopaque marker or tip of an extended working channel of a catheter assembly in conjunction with an angle measurement device;

[0040] Figure 6 is a flowchart of a method for three-dimensional model construction in accordance with the present disclosure;

[0041] Figure 7 It is an image of the initial video captured by the fluorescence imaging device, the frame after ramp filtering, and a diagram of the resulting three-dimensional volume.

[0042] Figure 8 It is a 3D constructed illustration generated based on a given angular range of fluorescence microscope images / videos;

[0043] Figure 9A This is a flowchart of another method for constructing the three-dimensional volume of a fluorescent mirror via the structure of markers according to this disclosure; and

[0044] Figure 9B It is based on Figure 9A A schematic diagram of the two-dimensional grid structure of the spherical markers using the method. Detailed Implementation

[0045] This disclosure relates to systems and methods for constructing local three-dimensional volumetric data of small soft tissue objects from video streams captured by standard fluorescence microscopy imaging equipment available in most procedural laboratories. This disclosure also relates to systems and methods for determining the position of a medical device relative to a soft tissue target within a patient based on the three-dimensional volumetric data. The constructed fluorescence microscopy-based local three-dimensional volumetric data or the position of the medical device relative to the soft tissue target can be used for guidance, navigation planning, improving navigation accuracy, navigation confirmation, and treatment confirmation.

[0046] The terms “tools,” “surgical instruments,” “surgical equipment,” “energy equipment,” and “medical equipment” are used interchangeably here.

[0047] Figure 1 An electromagnetic navigation (EMN) system 100 is described, configured to review CT image data to identify one or more targets, plan a path to the identified targets (planning phase), navigate the extension working channel (EWC) 12 of a catheter assembly to the targets via a user interface (navigation phase), and confirm the placement of the EWC 12 relative to the targets. One such EMN system is the Electromagnetic Navigation Bronchoscopy currently sold by Medtronic PLC. ® (ENB) system. The target can be a tissue of interest identified during the planning phase by reviewing CT image data. Medical devices (such as biopsy tools or other instruments) can be inserted into the EWC 12 following the navigation to obtain tissue samples from tissue located at or near the target site.

[0048] like Figure 1As shown, EWC 12 is part of a catheter guidance assembly 40. In practice, EWC 12 is inserted into bronchoscope 30 for access to the luminal network of patient "P." Specifically, EWC 12 of catheter guidance assembly 40 can be inserted into the working channel of bronchoscope 30 for navigation through the luminal network of the patient. A locatable guide (LG) 32, including a sensor 44, is inserted into EWC 12 and locked into place such that sensor 44 extends a desired distance beyond the distal tip of EWC 12. The position and orientation of sensor 44, and thus the distal portion of EWC 12, within the electromagnetic field relative to the frame of reference can be derived. Catheter guidance assembly 40 is currently marketed by Medtronic PLC under the brand name SUPERDIMENSION® ® EDGE® Procedure Kit TM The EDGE® procedure kit or EDGE® is marketed and sold, and can be contemplated for use with the present disclosure. For a more detailed description of catheter guidance assembly 40, reference is made to commonly owned U.S. Patent Publication No. 2014 / 0046315 to Ladtkow et al., filed March 15, 2013, U.S. Patent No. 7,233,820, and U.S. Patent No. 9,044,254, the entire contents of each of which are hereby incorporated by reference.

[0049] EMN system 100 generally includes an operating table 20 configured to support a patient "P"; a bronchoscope 30 configured for insertion through the mouth of patient "P" into the airway of patient "P"; a monitoring device 120 (e.g., a video display for displaying video images received from a video imaging system of bronchoscope 30) coupled to bronchoscope 30; a tracking system 50 including a tracking module 52, a plurality of reference sensors 54, and an emitter mat 56; and a computing device 125 including software and / or hardware for facilitating identification of a target, path planning to the target, navigation of a medical instrument to the target, and confirmation of placement of EWC 12 or a suitable device therethrough relative to the target.

[0050] Also included in this particular aspect of the system 100 is a fluoroscopic imaging device 110 capable of acquiring fluoroscopic or x-ray images or videos of the patient "P". The images, series of images, or videos captured by the fluoroscopic imaging device 110 can be stored within the fluoroscopic imaging device 110 or transmitted to the computing device 125 for storage, processing, and display. Additionally, the fluoroscopic imaging device 110 can be moved relative to the patient "P" such that images can be acquired from different angles or perspectives relative to the patient "P" to create a fluoroscopic video. In one aspect of the present disclosure, the fluoroscopic imaging device 110 includes an angle measurement device 111 configured to measure the angle of the fluoroscopic imaging device 110 relative to the patient "P". The angle measurement device 111 can be an accelerometer. The fluoroscopic imaging device 110 can include a single imaging device or more than one imaging device. In embodiments including multiple imaging devices, each imaging device can be a different type of imaging device or be the same type. Further details regarding the imaging device 110 are described in U.S. Patent No. 8,565,858, which is incorporated by reference herein in its entirety.

[0051] The computing device 125 can be any suitable computing device including a processor and a storage medium, where the processor is capable of executing instructions stored on the storage medium. The computing device 125 can also include a database configured to store patient data, CT data sets (including CT images), fluoroscopic data sets (including fluoroscopic images and videos), navigation plans, and any other such data. Although not explicitly shown, the computing device 125 can include an input, or can otherwise be configured to receive CT data sets, fluoroscopic images / videos, and other data described herein. Additionally, the computing device 125 includes a display configured to display a graphical user interface. The computing device 125 can be connected to one or more networks through which one or more databases can be accessed.

[0052] With respect to the planning phase, the computing device 125 utilizes previously acquired CT image data to generate and view a three-dimensional model of the airway of the patient "P", enable (automatically, semi-automatically, or manually) identification of targets on the three-dimensional model, and allow determination of a path through the airway of the patient "P" to tissue located at and around the targets. More specifically, CT images acquired from a prior CT scan are processed and assembled into a three-dimensional CT volume, which is then used to generate a three-dimensional model of the airway of the patient "P". The three-dimensional model can be displayed on a display associated with the computing device 125 or in any other suitable manner. Various views of the three-dimensional model or enhanced two-dimensional images generated from the three-dimensional model are presented using the computing device 125. The enhanced two-dimensional images can have some three-dimensional capabilities in that they are generated from three-dimensional data. The three-dimensional model can be manipulated to facilitate identification of targets on the three-dimensional model or two-dimensional images, and a suitable path through the airway of the patient "P" into tissue located at the targets can be selected. Once selected, the path plan, three-dimensional model, and images derived therefrom can be saved and exported to a navigation system for use during one or more navigation phases. One such planning software is ILOGIC® currently sold by Medtronic PLC. ® planning suite.

[0053] With respect to the navigation phase, a six degree of freedom electromagnetic tracking system 50 (similar to those disclosed in U.S. Patent Nos. 8,467,589, 6,188,355, and published PCT Application Nos. WO 00 / 10456 and WO 01 / 67035, the entire contents of each of which are incorporated herein by reference) or other suitable position measurement system is used to perform image registration and navigate the path, although other configurations are also contemplated. The tracking system 50 includes a tracking module 52, a plurality of reference sensors 54, and an emitter pad 56. The tracking system 50 is configured for use with the positionable guide 32, and in particular the sensor 44. As described above, the positionable guide 32 and sensor 44 are configured for insertion through the EWC 12 into the airway of the patient "P" (with or without the aid of the bronchoscope 30), and can be selectively locked relative to one another via a locking mechanism.

[0054] The transmitter pad 56 is positioned beneath the patient "P". The transmitter pad 56 generates an electromagnetic field around at least a portion of the patient "P" within which the positions of the plurality of reference sensors 54 and sensor elements 44 can be determined using the tracking module 52. One or more of the reference sensors 54 are attached to the chest of the patient "P". The six degree of freedom coordinates of the reference sensors 54 are sent to the computing device 125 (which includes appropriate software) where they are used to compute a patient reference coordinate system. Registration as detailed below is generally performed to coordinate the positions of the three-dimensional model from the planning phase and the two-dimensional images with the airway of the patient "P" as viewed through the bronchoscope 30 and to allow the navigation phase to begin with exact knowledge of the position of the sensors 44, even in portions of the airway that the bronchoscope 30 cannot reach. More details of this registration technique and its implementation in luminal navigation can be found in U.S. Patent Application Publication No. 2011 / 0085720 (the entire contents of which are incorporated herein by reference), although other suitable techniques are also contemplated.

[0055] Registration of the position of the patient "P" on the transmitter pad 56 can be performed by moving the LG 32 through the airway of the patient "P". More specifically, data relating to the position of the sensors 44 as the positionable guide 32 is moved through the airway is recorded using the transmitter pad 56, the reference sensors 54, and the tracking module 52. The shape resulting from this position data is compared to the internal geometry of the passageway of the three-dimensional model generated in the planning phase, and a position correlation between the shape and the three-dimensional model based on the comparison is determined, for example, with software on the computing device 125. In addition, the software identifies non-tissue space in the three-dimensional model (e.g., air-filled cavities). The software aligns or registers the images representing the position of the sensors 44 with the three-dimensional model and two-dimensional images generated from the three-dimensional model based on the recorded position data and the assumption that the positionable guide 32 has been in non-tissue space in the airway of the patient "P" at all times. Alternatively, a manual registration technique can be employed by navigating the bronchoscope 30 to pre-specified locations in the lungs of the patient "P" with the sensors 44 and manually correlating the images from the bronchoscope with the model data of the three-dimensional model.

[0056] After the patient "P" is registered to the image data and the pathway is planned, a user interface is displayed in the navigation software that sets out the path that the clinician is to follow to reach the target. One such navigation software is the ILOGIC® navigation suite currently sold by Medtronic PLC. ® Navigation suite.

[0057] Once the EWC 12 has been successfully navigated to the vicinity of the target as depicted on the user interface, the positionable guide 32 can be unlocked from the EWC 12 and removed, leaving the EWC 12 in place as a guide channel for guiding medical devices, including but not limited to optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation devices), laser probes, cryoprobes, sensor probes, and aspiration needles leading to the target.

[0058] Already described Figure 1 The components of the system 100 depicted herein are described below. Figures 2 to 9B The description provides an exemplary workflow for using components of system 100 (including fluorescence microscopy imaging device 110) to construct local three-dimensional volumetric data of a desired region of interest and for using the fluorescence microscopy imaging device 110 of system 100 to determine the relationship between a medical device and a soft tissue target. The systems and methods described herein can be used to visualize specific target regions of a patient using imaging devices typically located within a surgical setting during EMN procedures, and further can be used to visualize medical devices relative to the target during medical procedures, thereby eliminating the need for subsequent MRI or CT scans.

[0059] Now go to Figure 2 A model of a fluorescence imaging device 110 is shown. The fluorescence imaging device 110 includes an X-ray source 201 and a detector 203. The detector 203 defines a plurality of two-dimensional pixels. Each two-dimensional pixel is associated with a single X-ray beam that traverses a three-dimensional space from the X-ray source 201 to the detector 203. The field of view is determined using the following formula: The size D of the detector 203 and the source-detector distance SDD.

[0060] .

[0061] The main differences between different fluoroscopes lie in the detector size and source-detector distance. In fluoroscope data, pixels are not normalized to a specific scale. Their brightness depends on the gain / exposure used by the fluoroscope and on other objects in the scene across which the X-rays traverse between the source and detector, such as operating tables, surgical equipment, etc. In CT data, each voxel is measured in Henle units. Henle units are measured relative to the observed brightness of water and air using the following formula:

[0062] .

[0063] µ is the attenuation coefficient. They range from 0 to infinity and are used to measure the difficulty of an X-ray beam traversing a three-dimensional volume of matter of 205. "Thicker" matter has a larger µ. HU measures the attenuation coefficient, with air at -1000, water at 0, and thicker matter rising to infinity.

[0064] Using the Beer-Lambert law, each two-dimensional pixel in the fluoroscope detector 203 gives the following:

[0065] .

[0066] μ(x, y, z) is the attenuation coefficient of the three-dimensional volume 205 at position (x, y, z). I0 is the X-ray source 201 energy (higher energy produces brighter images). For simplicity, assume I0 = 1. After taking the logarithm, each two-dimensional pixel is represented as follows:

[0067] .

[0068] This is so for each two-dimensional detector pixel, which provides a linear equation for the three-dimensional attenuation coefficient. If many such equations are utilized (for each two-dimensional pixel of the detector 203), the attenuation coefficient can be solved and a three-dimensional volume data of the three-dimensional volume 205 can be constructed.

[0069] Fluoroscope to CT - discretization:

[0070] The three-dimensional volume 205 can be divided into a discrete grid, where the coordinates of the voxels are (x k , y k , z k ). Thus, the equation for solving each two-dimensional pixel can then be written as:

[0071] .

[0072] The left-hand side of the above equation is the observed two-dimensional pixel of the detector 203, and the right-hand side of the above equation is a weighted sum of the attenuation coefficient (to be solved) times , which is determined by the known fluoroscope imaging device position of the X-ray source 201.

[0073] In order to be able to solve for the volume attenuation coefficient values, enough linear equations are needed, i.e. enough two-dimensional observations of different two-dimensional pixels are needed. A standard detector usually has 1024 x 1024 pixels, where each pixel is represented by a single equation. Thus, many two-dimensional observations (many two-dimensional observed pixels) need to be solved in order to reconstruct the three-dimensional volume (solve for many voxels). That is, many two-dimensional pixels are needed in order to solve for many voxels. In order to know the weights in these equations, the fluoroscope imaging device X-ray source 201 configuration (position, orientation, field of view) must be known.

[0074] In use, the three-dimensional volume 205 is a portion of a patient's body. A fluoroscopic video composed of multiple fluoroscopic images (as frames of a video) is taken from many different positions relative to the patient (e.g., a 180° rotation around the patient) to capture multiple observations of two-dimensional pixels at different positions relative to the patient. As will be described in greater detail below, the position of the fluoroscopic imaging device relative to the patient at a given time can be determined using a variety of techniques, including motion structure-from-motion analysis of radiopaque markers placed within the patient ( Figures 3A to 3B ), registration between the captured fluoroscopic images / video and generated virtual fluoroscopic images ( Figure 4 ), external angle measurement devices such as accelerometers, gyroscopes, or magnetic field sensors ( Figure 5 ), or external structures of markers ( Figures 9A to 9B ). In one aspect, where markers that move with the body are utilized, the system can correct for patient movement when measuring angles. Specifically, all two-dimensional fluoroscopic images can be synchronized to the same three-dimensional position based on the position of the placed markers in each image.

[0075] Turning now to Figures 3A to 3B and Figures 4 to 6 , a method of constructing a local three-dimensional volume of a target region using a standard fluoroscopic imaging device (such as the fluoroscopic imaging device 110 of Figure 1 ) in conjunction with a system (such as the system described in Figure 1 ) will now be described in particular detail. While the methods shown and described herein are shown and described as being performed in a particular order and requiring particular steps, any of the methods can include some or all of these steps and can be implemented in any order not specifically described.

[0076] A fluoroscopic-based three-dimensional volume generated by any of the methods described below can be incorporated into the system 100 for a variety of purposes. For example, the fluoroscopic-based three-dimensional volume can be registered with previously generated three-dimensional volume data for medical instrument navigation. The system 100 can utilize the registration between the fluoroscopic-based three-dimensional volume and previously acquired three-dimensional volume data to update sensors 44 placed in the patient's body ( Figure 1The calculation location of the lungs. More specifically, a three-dimensional model of the patient's lungs generated from previously acquired CT scans may not provide sufficient basis for accurately guiding the medical device to the target during electromagnetic navigation procedures. In some cases, deformation of the patient's lungs relative to the lungs during the procedure at the time of acquisition of previously acquired CT data can cause inaccuracies. This deformation (CT versus body divergence) can be caused by many different factors, such as: sedation versus non-sedation, bronchoscopy altering the patient's posture and also pushing tissues, different lung volumes due to CT being performed during inspiration while navigation is performed during expiration, different beds, days, etc. Therefore, an alternative imaging modality is needed to visualize the target and / or terminal bronchial branches and to enhance electromagnetic navigation procedures by: correcting navigation during the procedure, visualizing the target, and confirming the placement of surgical devices during the procedure. To this end, the system described herein processes and converts the image data captured by the fluorescence imaging device 110, as will be described in detail below. The fluorescence microscopy image data can be used to identify such targets and terminal bronchial branches, or combined with data from CT scans, and used to update data from CT scans to provide more accurate results / corrections for electromagnetic navigation procedures.

[0077] Additionally, users can visually confirm that the placement of the guided medical device is correctly positioned relative to the target tissue within the target area. Furthermore, fluorescence-based 3D volume can be used to visualize the target area in three dimensions after the procedure has been performed. For example, fluorescence-based 3D volume can be used to visualize the target area after markers have been placed within it, after a biopsy, or after treatment of a target.

[0078] Special reference Figures 3A to 3B A method for constructing a three-dimensional volume using multiple transmissive markers placed near the target will now be described, and this method will be referred to as method 300. Method 300 begins with step 301, in which an electromagnetic navigation system (such as the EMN system 100 described above) is utilized. Figure 1 The marker placement device is navigated to the target area. Navigation to the target area can be achieved using a previously created navigation plan (which includes routes created during the planning phase). In step 303, radiopaque markers are placed within the target area. In one example, four radiopaque markers are used. However, fewer or more than four radiopaque markers may be used.

[0079] In step 305, with the radiopaque markers placed in the target region, the fluoroscopic imaging device is positioned such that all of the radiopaque markers placed in step 303 are visible. That is, step 305 includes aligning the fluoroscopic imaging device such that it can be rotated 30° around the markers with all of the markers visible. In step 307, the fluoroscopic imaging device is used to capture a video of about a 30° rotation (from -15° to +15° rotation) of the imaging device 110 around the patient and thus around the markers. By rotating up to 15° from the centered angle (on each side), it can be ensured that the markers will remain in the image / frames throughout the rotated video and that the imaging device will not hit the patient or bed. Figure 3B Six frames f1-f6 of the captured video are shown. Each of the frames f1-f6 is an image of the fluoroscopic video showing different positions and orientations of each of the radiopaque markers m1-m4 at different points in time of the video, with the fluoroscopic imaging device positioned at different angles relative to the patient at each given time.

[0080] In step 309, the two-dimensional positions of each of the radiopaque markers are tracked throughout the video. In step 311, the marker positions are constructed in three dimensions using a motion recovery structure technique, and the pose of the fluoroscopic imaging device for each video frame is obtained. Motion recovery structure is a method of reconstructing the three-dimensional positions of points and the fluoroscopic imaging device position (camera pose) by tracking the points in a two-dimensional sequential video. In step 311, by introducing markers into the patient, the positions and orientations of these markers can be tracked along a sequential fluoroscopic rotation video, their three-dimensional positions in space can be reconstructed, and the corresponding fluoroscopic imaging device positions can be determined. With the fluoroscopic imaging device positions determined through video analysis, the fluoroscopic imaging device positions can be used to solve for the three-dimensional data.

[0081] In step 313, a local three-dimensional volume is constructed. In particular, a fast iterative reconstruction algorithm (OCTRecon) Figure 6 ) is used to reconstruct a local three-dimensional volume at the region of the markers corresponding to the local anatomy, and where soft tissue is visible. Step 313 can include reconstructing a global three-dimensional volume from the acquired two-dimensional fluoroscopic data and cropping the global three-dimensional volume at the region of the target to create a “FluoroCT Blob” volume. This cropped volume can be displayed to the user in the form of the original three-dimensional data or as two-dimensional re-projection images. In this cropped volume, all of the anatomy in the target region, including the target tissue, will be visible. The re-projection images, which do not include distant dense obscuring objects, can be enhanced by stretching the darker values to black and the brighter values to white to increase the contrast and also sharpening in the three-dimensional before projection or in the projected images so that the soft tissue is visually identified.

[0082] As described above, the fluoroscopy-based three-dimensional volume can be incorporated into the system 100 for a variety of purposes. For example, the fluoroscopy-based three-dimensional volume can be registered with previously generated three-dimensional volume data for medical instrument navigation. The system 100 can utilize the registration between the fluoroscopy-based three-dimensional volume and the previously acquired three-dimensional volume data to update the calculated position of the sensor 44 Figure 1 . In addition, the user can visually confirm that the placement of the navigated medical instrument is positioned in the desired location relative to the target tissue within the target region.

[0083] Turning now to Figure 4 A method for constructing a three-dimensional volume using a single radiopaque marker placed near a target or a distal portion of a navigated tool, such as the tip of an extended working channel of a catheter assembly positioned near a target, will now be described and will be referred to as method 400. While method 400 is described as utilizing the tip of an extended working channel of a catheter assembly, the method can utilize any tool to perform this function. For example, the tip of a navigated catheter, the tip of a biopsy tool, or the tip of a therapy tool can be utilized. In one aspect, the tool is bronchoscopically navigated to the target. In other aspects, the tool can be a tool inserted percutaneously into the patient, such as a transthoracic navigation of a therapy device, such as an ablation device.

[0084] Method 400 begins with step 401 in which an extended working channel is navigated to a target region using an electromagnetic navigation system, such as the EMN system 100 described above Figure 1 The navigation of the EWC to the target region can be accomplished using a previously created navigation plan, which includes a route created during the planning phase. Method 400 can optionally include the additional step of navigating a marker placement device to the target region via the EWC to place a single radiopaque marker within the region of the target (step 403). In one aspect, step 401 includes percutaneously inserting the tool to the target region.

[0085] After the EWC or tool is in place, or after the radiopaque marker is placed, the fluoroscopic imaging device is positioned such that the tip of the navigation of the EWC or tool (and / or the placed radiopaque marker) is visible within the field of view of the fluoroscopic imaging device. That is, step 405 includes aligning the fluoroscopic imaging device such that it can be rotated 30° about the marker to make the marker visible, and / or 30° about the tip of the EWC or tool to make the tip of the EWC or tool visible. In step 407, the fluoroscopic imaging device is used to capture a video of about 30° rotation (from -15° to +15° rotation) of the imaging device 110 about the patient and thus about the marker and / or the tip of the EWC or tool. By rotating up to 15° from the centered angle (on each side), it can be ensured that the marker and / or the tip of the EWC or tool will remain in the image / frames throughout the rotated video and that the imaging device will not hit the patient or bed. Step 407 can include capturing a video of about 30° rotation about the distal portion of the EWC (and the radiopaque marker if placed). If a 30° rotation video is captured, one angle (in the middle of the range) is sufficient. That is, two projections 30° between the two are sufficient to confirm or correct the three-dimensional relationship of the tool to the soft tissue.

[0086] In step 409, the two-dimensional position of the distal portion of the EWC or tool (and / or the radiopaque marker if placed) is tracked throughout the captured video.

[0087] In step 411, virtual fluoroscopic images are created from the previously acquired CT data. The previously acquired CT data is typically the CT data used during the planning phase to plan a navigation path to the target. In step 411, the CT data is manipulated to create a computer model of a fluoroscopic image of the patient. The location of the target in the virtual fluoroscopic images corresponds to the location of the target identified by the clinician during the planning phase. The virtual fluoroscopic images generated by the system based on the previously acquired CT data depict the field of view that would be captured by the fluoroscopic imaging device. Additionally, each of the virtual fluoroscopic images has a virtual fluoroscopic imaging device pose.

[0088] In step 413, each video frame of the captured fluoroscopic video in step 407 is registered to the previously acquired CT data by matching each of the fluoroscopic video frames to a virtual fluoroscopic image. In step 415, the fluoroscopic imaging device pose for each video frame of the captured fluoroscopic video is determined based on the registration of step 413. That is, once a fluoroscopic frame is matched to a virtual fluoroscopic image, the virtual fluoroscopic imaging device pose of the virtual fluoroscopic image can be associated with the corresponding fluoroscopic frame.

[0089] In step 417, the source of the fluoroscopic imaging device pose determined in step 415 is corrected by using the tracked position of the distal portion of the EWC or tool (and / or the radiopaque marker if placed) which is used to compensate for patient movement (such as movement caused by respiration). In step 419, a local three-dimensional volume is constructed. In particular, a fast iterative reconstruction algorithm (OCTRecon®) Figure 6 ) is used to reconstruct a local three-dimensional volume at the region of the target lesion corresponding to the local anatomy, and where soft tissue is visible. Step 319 can include reconstructing a global three-dimensional volume from the acquired two-dimensional fluoroscopic data and cropping the global three-dimensional volume at the region of the target to create a "FluoroCT Blob" volume. This cropped volume can be displayed to the user in the form of the original three-dimensional data or as two-dimensional re-projection images. In this cropped volume, all of the anatomy in the target region (including the target tissue) will be visible. The re-projection images (which do not include distantly dense obscuring objects) can be enhanced by stretching the darker values to black and the brighter values to white to increase the contrast and also to sharpen in the three-dimensional before projection or in the projected images so that the soft tissue is visually identified.

[0090] Method 400 can also include an additional step (step 421) of completing the captured fluoroscopic video in step 407 to include virtual fluoroscopic images generated by the system, which represent fluoroscopic imaging device poses outside the range of fluoroscopic imaging device poses captured in the fluoroscopic video. In particular, in aspects, the system 100 can also generate virtual fluoroscopic images of the patient using previously generated CT volume data of the patient used to create the navigation plan. The generated virtual fluoroscopic images are fluoroscopy-like images that display to the user a view of how the fluoroscopic images of the patient should look when captured by the fluoroscopic imaging device at a given angle. In step 421, the virtual fluoroscopic images can be used to fill in any gaps in the captured fluoroscopic video (captured in step 407). This can include, for example, replacing images (such as frames) in the captured video that appear distorted or corrupted. Additionally or alternatively, this can include supplementing the captured fluoroscopic video (captured in step 407) with virtual fluoroscopic images that represent fluoroscopic images outside the range of angles included in the fluoroscopic video. For example, if the fluoroscopic video includes a scan of about 30° around the patient, virtual fluoroscopic images outside the 30° range can be incorporated into the video to generate a fluoroscopic video with a range greater than 30°.

[0091] Method 300 (FIG. 3) and method 400 (FIG. 4) can be performed in any order or simultaneously. Figure 4Both are used to construct three-dimensional CT volume data using fluoroscopic video without knowing the fluoroscopic imaging device pose of each frame of the fluoroscopic video. To this end, each of the methods 300 and 400 require a step of determining the fluoroscopic imaging device pose of each frame of the fluoroscopic video using image-based techniques. In contrast, and as described in more detail below, the method 500 Figure 5 is a method for constructing three-dimensional CT volume data, wherein the fluoroscopic imaging device pose of each frame of the acquired fluoroscopic video is determined using a pose / angle measurement device, which can include an accelerometer, a gyroscope, or a magnetic field detector to detect the position / pose of the fluoroscopic imaging device relative to the patient.

[0092] The method 500 is a method of constructing a three-dimensional volume using a single radiopaque marker placed near the target or the tip of an extended working channel of a catheter assembly positioned near the target in conjunction with a fluoroscopic angle measurement device. The method 500 begins with step 501 in which a fluoroscopic calibration is performed using a fluoroscopic calibration jig to calculate typical fluoroscopic projection parameters and geometry. This calibration is performed once by a technician on each fluoroscopic device in the installation phase. The calibration jig is used to determine the projection parameters (field of view angle) of the fluoroscope as well as the geometry of the C-arm: position relative to the rotation axis in an automated process. These parameters are sometimes given in the technical drawings of each fluoroscopic device, but can also be derived using our calibration jig. In step 503, previously acquired CT volume data is imported into the system along with the previously generated navigation plan.

[0093] In step 505, a system such as the EMN system 100 described above ( Figure 1 ) is used to navigate the extended working channel to the target region using electromagnetic navigation techniques. Navigation of the EWC to the target region can be achieved using the previously created navigation plan, which includes the route created during the planning phase. The method 500 can optionally include an additional step of navigating a marker placement device to the target region via the EWC to place a single radiopaque marker within the region of the target (step 507).

[0094] After the EWC or tool is in place, or after a radiopaque marker has been placed, method 500 proceeds to step 509, which includes aligning the fluorescein imaging device such that it can be rotated 30° around the marker to make the marker visible, and / or rotated 30° around the tip of the EWC or tool to make the tip of the EWC or tool visible. In step 511, the fluorescein imaging device is used to capture video of the imaging device 110 rotating approximately 30° (from -15° to +15°) around the patient and thus around the marker or the tip of the EWC or tool. By rotating 15° from a central angle (on each side), it is ensured that the marker or the tip of the EWC or tool will remain in the image / frame of the entire rotated video and that the imaging device will not hit the patient or the bed. If a 30° rotated video is captured, one angle (in the middle of that range) is sufficient. That is, two projections at 30° between them are sufficient to confirm or correct the three-dimensional relationship between the tool and the soft tissue. In step 513, the two-dimensional position and orientation of the distal end of the EWC (and / or a non-transmissive marker, if placed) are tracked throughout the captured video.

[0095] In step 515, the calibration data from step 501 is combined with measurements from an external angle measuring device to calculate the position of the fluorescein imaging device in world coordinates (less source). Specifically, as described above, the calibration fixture is used to automatically derive the projection parameters and C-arm geometry of a specific fluorescein device using an optimization method. Once these parameters are known, the angles obtained from the angle measuring device (i.e., the angles of the fluorescein's detector) determine the unique three-dimensional orientation of the detector. It will not be located anywhere else in space except where it was previously explained by the given angles and mounting parameters.

[0096] In step 517, a local 3D volume is constructed. Specifically, a fast iterative reconstruction algorithm (…) Figure 6 This is used to reconstruct a local three-dimensional volume at a region of target tissue corresponding to a local anatomical structure, where soft tissue is visible. Step 517 may include reconstructing a global three-dimensional volume from the acquired two-dimensional fluorescence microscopy data and cropping the global three-dimensional volume at the target region to create a "FluoroCT Blob" volume. This cropped volume may be displayed to the user as raw three-dimensional data or as a two-dimensional reprojection image. In this cropped volume, all anatomical structures (including target tissue) in the target region will be visible. The reprojection image can be enhanced by stretching darker values ​​to black and brighter values ​​to white (excluding distant dense occlusions) to increase contrast and sharpen the image in the unprojected three-dimensional region or the projected image, making soft tissue visually identifiable.

[0097] A CT reconstruction method will now be described. CT reconstruction methods can be divided into analytical methods (Radon, FDK...) and algebraic methods (ART, SART...). Analytical methods assume very specific configurations (such as a full 180° rotation) and reconstruct the CT volume in a single iteration (using some exact formula). Algebraic methods are more flexible but slower and treat the problem as a large system of equations solved iteratively (using some kind of gradient descent method). All methods use projection (3D to 2D) and back-projection (2D to 3D).

[0098] In terms of projection, since each detector pixel is essentially a weighted sum of 3D voxels along a ray, the detector image can be seen as a 2D projection of a 3D volume from certain fluoroscope device positions. If 3D volume data is already available, then a fluoroscope image can be reproduced by projecting it from known fluoroscope device positions. A 3D reconstruction is considered good if its 2D projections resemble the observed fluoroscope images that created the 3D reconstruction.

[0099] In terms of back-projection, at each voxel, back-projection determines which rays cross a particular voxel and adds them together. To make this determination, the fluoroscope device positions must be known. If the back-projection operator is applied to the captured video's fluoroscope images, a 3D volume can be constructed, but the constructed 3D volume will be very blurry and inaccurate because while summing many times the true central voxel, many unrelated voxels surrounding the true central voxel are also summed many times. After reconstructing a 3D volume using one method or another, the quality of the reconstruction is assessed by taking the reconstructed 3D data, projecting it into 2D frames (which can be virtual fluoroscope frames), and comparing these 2D frames to the original fluoroscope 2D frames that created the 3D data. One goal of a volume reconstruction algorithm is to arrive at 3D data that explains the 2D observations so that if the 3D data is projected back into 2D frames, these frames will look like the original, true fluoroscope frames. When the product is blurry, this means that the projection is blurry and does not match the true images. To address this problem, the method provides a ramp filter and a correction iteration.

[0100] Turning now to Figure 6A method for reconstructing local 3D volumes using a fast iterative algorithm, referred to as Method 600, will now be described. The 3D volume reconstruction algorithm (e.g., Method 600) consists of multiple iterations of projection-backprojection. The goal of this algorithm is to derive 3D data that interprets 2D observations from a fluorescein imaging device. If it successfully derives such data, the resulting 3D data is assumed to represent the patient's 3D anatomy. In each iteration, the resulting current 3D data is projected into 2D (which should resemble the initial fluorescein video), and then the 2D error is backprojected back into the 3D data, updating the 3D data in such a manner. This process is repeated several times until the 3D data converges and the process stops. To accelerate this process, the 2D projected images are filtered before being backprojected back into 3D. A ramp filter is merely an exemplary filter proven effective in accelerating convergence. This filter is applied when reconstructing standard CT using the classical Radon transform. Using this classical method, the process is performed in a single iteration: filtering (e.g., a ramp filter), backprojection. In this method, this process is iteratively repeated in several steps.

[0101] Method 600 begins with step 601, in which the equation starts with an initial volume V (which may simply be zero). In step 603, the volume V is projected from the known position of the fluorescence microscope imaging device onto the image Q. i In the middle. For example, the fluoroscope image filtered by the ramp function can be projected without the fluoroscope image. The iteration with the following residuals can undergo ramp filtering before backprojection. In step 605, the residual R is calculated. i = P i - Q i , where P i This is the projection observed from the captured fluorescence mirror video. In step 606, a ramp filter is used to convolve R as in Radon. i In step 607, R is determined. i Is it below a predetermined threshold? If R i If the value is below a predetermined threshold (yes in step 607), then method 600 completes. If R i If the value is not lower than a predetermined threshold (no in step 607), then method 600 proceeds to step 609. In step 609, R... i The image is back-projected into E (corrected volume). In step 611, the volume V is set to V + E (V = V + E), and method 600 returns to step 603, where the volume V (now V + E) is projected from the known position of the fluorescence microscope imaging device onto the image Q. i middle.

[0102] Now go to Figure 7 and Figure 8. Figure 7 A frame 700 of an initial video captured by a fluoroscopic imaging device, an image 703 of the frame after ramp filtering, and an image 705 of the resulting three-dimensional volume are shown. Figure 8 is an illustration of a three-dimensional build at angles 801-88, where 801 is 30 degrees, 803 is 60 degrees, 805 is 90 degrees, 807 is 120 degrees, 809 is 150 degrees, and 811 is 180 degrees.

[0103] Reference is now made to Figure 9A is a flowchart of another method of constructing a fluoroscopic three-dimensional volume via a structure of markers in accordance with the present disclosure. A method of constructing fluoroscopy-based three-dimensional volume data of a target region within a patient from two-dimensional fluoroscopic images is disclosed herein. In step 900, a sequence of images of a structure of markers and a target region are acquired via a fluoroscopic imaging device. The structure of markers can include a plurality of at least partially radiopaque markers arranged in a particular pattern. The structure of markers is positioned outside of the patient, for example, positioned below the patient, while the images are captured. The structure of markers is further positioned such that each image includes a projection of at least a portion of the structure of markers. The structure of markers can be positioned between the patient and the fluoroscopic imaging device. In some embodiments, the structure of markers can be in a two-dimensional pattern. In some embodiments, the structure of markers can be in a periodic pattern, such as a grid. In some embodiments, the target is a soft tissue target. In some embodiments, the target region can include at least a portion of, for example, a lung, and the structure of markers is positioned relative to the target region such that the structure of markers is visible in the two-dimensional fluoroscopic images. Figure 1 System instantiation of

[0104] In step 910, the pose of the fluoroscopic imaging device for at least a plurality of images of the image sequence can be estimated. The pose estimation can be performed based on detection of the most likely and maximum probability projection of the structure of the marker on each of the plurality of images. A probability map can then be generated for each image, indicating the probability of each pixel in the image being a marker of the structure of the marker. A plurality of virtual candidates of the projection of the structure of the marker on the image can be generated by virtually positioning the fluoroscope in different possible positions, including different orientations, i.e., different poses. The candidate with the highest probability of being the projection of the structure of the marker on the image can then be identified based on the probability map. The virtual pose of the fluoroscope associated with the identified candidate can then be determined as the estimated pose of the fluoroscope at the time of capturing the image. Optionally, the process of identifying the candidate can be refined. A locally deformed version of the candidate can be generated based on the probability map, so as to maximize the probability of it being the projection of the structure of the marker on the image. The new virtual candidate can then be fitted to the locally deformed version of the identified candidate. The virtual pose of the fluoroscope that generates the new improved candidate is calculated and determined as the estimated pose of the fluoroscope at the time of capturing the image. Further details regarding the disclosed pose estimation can be found in commonly owned U.S. Patent Application No. 62628017, entitled “SYSTEM AND METHOD FOR POSE ESTIMATION OF AN IMAGING DEVICE AND FOR DETERMINING THE LOCATION OF A MEDICAL DEVICE WITH RESPECT TO A TARGET,” filed February 8, 2018, by Barak et al., the entire contents of which are hereby incorporated by reference.

[0105] In step 920, a fluoroscopic-based three-dimensional volume data of the target region is constructed based on the estimated pose of the fluoroscopic imaging device, in accordance with the disclosed systems and methods. However, other systems and methods based on imaging device pose information, as known in the art, can be used.

[0106] In optional step 930, the medical device can be positioned in the target region prior to acquisition of the image sequence. Thus, in addition to the target, the image sequence and thus the fluoroscopy-based three-dimensional volume data can include a projection of the medical device. The offset between the medical device and the target (i.e., Ax, Ay, and Az) can then be determined from the fluoroscopy-based three-dimensional volume data. The target can be visible or better revealed in the generated three-dimensional volume data. Thus, the target can be detected automatically or manually by a user in the three-dimensional volume data. The medical device can be detected automatically or manually by a user in the captured image sequence or in the generated three-dimensional volume data. Automatic detection of the target and / or the medical device can be performed based on systems and methods as known in the art and such as, for example, described in commonly owned U.S. Patent Application No. 62 / 627,911, entitled “SYSTEM AND METHOD FOR CATHETER DETECTION IN FLUOROSCOPIC IMAGES AND UPDATING DISPLAYED POSITION OF CATHETER,” by Birenbaum et al., filed February 8, 2018. Manual detection can be performed by displaying the three-dimensional volume data and / or the captured images to the user and requesting their input. Once the target and the medical device are detected in the three-dimensional volume data and / or the captured images, their positions in the fluoroscopy reference coordinate system can be obtained and the offset between them can be determined.

[0107] The offset between the target and the medical device can be used for various medical purposes, including facilitating medical device access to the target region and treatment. Navigation of the medical device to the target region can be facilitated via a tracking or positioning system and a display, such as the tracking system 50 and the monitor device 120 of Figure 1 . The positioning system localizes or tracks movement of the medical device through the patient’s body. The display can show the medical device position to the user relative to the surrounding environment of the medical device and the target within the patient’s body. The positioning system can be, for example, an electromagnetic or optical positioning system or any other such system as known in the art. When, for example, the target region includes a portion of the lung, the medical device can be navigated to the target region through the airway lumen network and as described with respect to Figure 1 .

[0108] In optional step 940, a display of the position of the medical device relative to the target can be corrected based on the determined offset between the medical device and the target. In some embodiments, a 3D rendering of the target region can be displayed on the display. The 3D rendering of the target region can be generated based on CT volume data of the target region acquired previously (i.e., prior to the current procedure or operation) (i.e., pre-operative CT). In some embodiments, the positioning system can be registered to the 3D rendering of the target, such as for example, relative to the Figure 1 rendering. The correction of the offset between the medical device and the target can then be performed by updating the registration of the positioning system to the 3D rendering. In general, to perform such an update, the transformation between the reference frame of the fluoroscopy image and the reference frame of the positioning system should be known. The geometric positioning of the structure of markers relative to the positioning system can determine such a transformation. In some embodiments, the structure of markers and the positioning system can be positioned such that the same reference frame will apply to both, or such that one will only be a translated version of the other. For example, and with reference to Figure 1 , the emitter pad 56 can be incorporated with the structure of markers.

[0109] In some embodiments, the update of the registration of the positioning system to the 3D rendering (e.g., based on CT) can be performed in a local manner and / or in a gradual manner. For example, the registration can be updated only in the surrounding of the target, e.g., only within a certain distance from the target. This is due to the fact that the update can not be as accurate when performed away from the target. In some embodiments, the update can be performed in a gradual manner, e.g., by applying a weight according to the distance from the target. In addition to the accuracy considerations, such a gradual update can make it more convenient or easier for the user to view, handle and make necessary changes during the procedure, as compared to a sudden change in the position of the medical device on the display.

[0110] In some embodiments, the patient can be instructed to hold his breath during the capture of the images in order to prevent the target region from moving due to breathing. In other embodiments, a method for compensating for breathing movement during image capture can be performed. For example, the estimated pose of the fluoroscopy device can be corrected according to the movement of a fiducial marker placed in the target region. Such a fiducial can be a medical device placed in the target region, e.g., a catheter. The movement of the catheter, for example, can be determined based on the positioning system. In some embodiments, the breathing pattern of the patient can be determined according to the movement of a fiducial marker located in the target region, such as a catheter. These movements can be determined via the positioning system. Based on the pattern, only images of inhalation or exhalation can be considered when determining the pose of the imaging device.

[0111] Reference is now made to Figure 9B , which is a diagram according to Figure 9AA schematic diagram of a two-dimensional (2D) mesh structure of a spherical marker 950 is provided for the method. The 2D mesh structure of the spherical marker 950 includes multiple spherical markers, such as spherical markers 960a and 960b, arranged in a two-dimensional mesh pattern. Unlike 3D patterns, using 2D patterns facilitates the pose estimation process. Furthermore, 2D patterns are more convenient for patients when, for example, a patient needs to lie on the marker structure to estimate the pose of the fluorescein while scanning the patient. In some embodiments, the shape of the markers may be symmetrical, such that the projection of the markers onto the image will be the same regardless of the orientation of the imaging device. This configuration simplifies and enhances the pose estimation process and / or makes it more efficient. For example, when the imaging device rotates around the marker structure, markers such as spheres are preferred. The size of the marker structure and / or the number of markers in the structure may be determined depending on the specific use of the disclosed system and method. For example, since pose estimation is used to construct the 3D volume of the region of interest within the patient, the size of the marker structure may be similar to or larger than the size of the region of interest. In some embodiments, the pattern of the marker structure, depending on the disclosed method and system, can be periodic, such as a grid. Figure 9B As shown. Using a periodic pattern structure of markers can further enhance and facilitate the attitude estimation process, making it more efficient. The structure of the markers, serving as a reference, should be positioned in a fixed manner during the capture of the fluorescence microscope image. An exemplary 2D mesh structure of the spherical markers (which can be compared with...) Figure 1 In the combined use of the system, the diameter of the spherical markers can be 2±0.2mm and the distance between the spheres can be about 15±0.15mm, and they are isotropic.

[0112] Figure 1 The system can be configured to execute Figure 9A The method. Figure 9A The method or part thereof may be provided by a computing device (such as Figure 1 The instructions are in the form of commands executed by a computing device 125. The computing device may include one or more hardware processors, one or more memory or storage devices, and a display. The one or more hardware processors may be configured to perform the steps of the method. The one or more memory or storage devices may be configured to store these instructions and / or fluorescein image data. The instructions may be displayed on a display or on a separate monitor (such as...). Figure 1 The monitoring device 10) displays medical equipment and / or targets.

[0113] Based on the foregoing and with reference to the accompanying drawings, those skilled in the art will understand that certain modifications can be made to this disclosure without departing from its scope. For example, while the systems and methods described herein are intended for use with EMN systems for navigation through luminal networks (such as the lungs), the systems and methods described herein can be used with systems utilizing other navigation and treatment devices (such as percutaneous devices). Furthermore, although the aforementioned systems and methods are described for use within a patient's luminal network, it should be understood that the systems and methods can be used for other target areas, such as the liver, kidneys, or heart. Additionally, the aforementioned systems and methods can also be used in transthoracic needle aspiration procedures. Furthermore, although the aforementioned systems and methods are described primarily for use relative to soft tissue targets, it should be understood that the systems and methods can be used relative to non-soft tissue targets (such as bone). Moreover, the aforementioned systems and methods can also be used with robotic surgical systems (such as those currently developed by Intuitive Surgical). ® Vinci for sale ® Used together with surgical systems. For example, and refer to Figure 1 System 100 may be such a robotic surgical system. A surgical console 20, used to position the patient during surgery, may be integrated into a cart comprising one or more robotic arms. The one or more robotic arms may be coupled to one or more specialized medical devices. Monitoring equipment 120 and / or computing equipment 125 may be integrated into a controller (e.g., a computerized console), allowing the surgeon to manipulate the one or more robotic arms while viewing the patient's body. Detailed embodiments of this disclosure are disclosed herein. However, these disclosed embodiments are merely examples of this disclosure, which may be embodied in various forms and aspects. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as the basis of the claims and as a representative basis for teaching those skilled in the art to adopt this disclosure differently with virtually any suitable detailed structure.

[0114] As can be appreciated, medical instruments such as biopsy tools or energy devices such as microwave ablation catheters that can be positioned through one or more branch luminal networks of a patient for treatment of tissue can prove useful in surgical regions, and the present disclosure relates to systems and methods that can be used with these instruments and tools. Navigation techniques can be used to access the luminal network percutaneously or through a natural orifice. Additionally, image guidance can be used to achieve navigation through the luminal network. These image guidance systems can be separate or integrated with the energy device or separate access tools, and can include MRI, CT, fluoroscopy, ultrasound, electrical impedance tomography, optical, and / or device tracking systems. Methods for positioning access tools include EM, IR, echolocation, optical, and others. Tracking systems can be integrated to the imaging device, where tracking is performed in virtual space or fused with preoperative or real-time images. In some cases, the treatment target can be accessed directly from within the lumen, such as for treatment of bronchial walls for COPD, asthma, lung cancer, etc. In other cases, the energy device and / or additional access tools can be required to puncture the lumen and extend into other tissue to reach the target, such as for treatment of disease within thin-walled tissue. Final positioning and confirmation of energy device or tool placement can be performed by imaging and / or navigation guidance using standard fluoroscopy imaging devices in conjunction with the methods and systems described above.

[0115] While several embodiments of the present disclosure have been shown in the drawings, the present disclosure is not intended to be limited to the embodiments shown in the drawings, as the present disclosure is intended to be as broad as allowed by the scope of the art, and is intended to be read similarly broadly. Accordingly, the above description should not be interpreted as limiting, but merely as exemplification of the specific embodiments. Those skilled in the art will be able to devise other modifications within the scope and spirit of the claims appended hereto.

Claims

1. A system for constructing a three-dimensional volume, comprising: a computing device and a computer-readable storage medium having instructions stored thereon, the computing device comprising a processor and a display configured to display a graphical user interface, the instructions, when executed by the processor: receiving a sequence of fluoroscopic images from a fluoroscopic imaging device, each image comprising at least a portion of a medical device; for a plurality of images in the sequence of fluoroscopic images, estimating a pose of the fluoroscopic imaging device; constructing a three-dimensional volume of a target region based on the estimated poses of the fluoroscopic imaging device, the three-dimensional volume comprising the medical device and a target; determining a location of the target in the three-dimensional volume; determining a location of a distal portion of the medical device in a plurality of the received fluoroscopic images; determining an offset between the locations of the medical device and the target; and correcting a displayed location of the medical device relative to a displayed location of the target on the display based on the determined offset between the locations of the medical device and the target. the instructions, when executed by the processor, determine the pose of the fluoroscopic imaging device when each of the plurality of images is acquired using an angle measurement device.

2. The system of claim 1, wherein, the instructions, when executed by the processor, determine the pose of the fluoroscopic imaging device when each of the plurality of images is acquired by registering a plurality of virtual fluoroscopic images from a pre-procedure image dataset with the received sequence of fluoroscopic images.

3. The system of claim 1, wherein, the instructions, when executed by the processor, present the three-dimensional volume constructed from the sequence of fluoroscopic images in the graphical user interface on the display.

4. The system of claim 1, wherein, the instructions, when executed by the processor, present a two-dimensional re-projection image of the three-dimensional volume in the graphical user interface on the display.

5. The system of claim 1, wherein, the received sequence of fluoroscopic images is obtained from a fluoroscopic scan of at least 120 degrees.

6. The system of claim 1, wherein, 7. A system for constructing a three-dimensional volume, comprising: a computing device and a computer-readable storage medium having instructions stored thereon, the computing device comprising a processor and a display configured to display a graphical user interface, the instructions, when executed by the processor: receiving a sequence of fluoroscopic images from a fluoroscopic imaging device, each image comprising at least a portion of a structure of a marker; for a plurality of images in the sequence of fluoroscopic images, estimating a pose of the fluoroscopic imaging device; constructing a three-dimensional volume of a target region based on the estimated poses of the fluoroscopic imaging device, the three-dimensional volume comprising a medical device and a target; determining a location of the target in the three-dimensional volume; determining a location of a distal portion of the medical device in a plurality of the received fluoroscopic images; determining an offset between the locations of the medical device and the target; and correcting a displayed location of the medical device relative to a displayed location of the target on the display based on the determined offset between the locations of the medical device and the target. ​ ​ 8. The system of claim 7, wherein, The instructions, when executed by the processor, determine the pose of the fluoroscopic imaging device when each of the plurality of images was acquired using an angle measurement device.

9. The system of claim 7, wherein, The instructions, when executed by the processor, determine the pose of the fluoroscopic imaging device when each of the plurality of images was acquired by registering a plurality of virtual fluoroscopic images from a pre-procedure image dataset with the received sequence of fluoroscopic images.

10. The system of claim 7, wherein, The instructions, when executed by the processor, present in the graphical user interface on the display the three-dimensional volume constructed from the sequence of fluoroscopic images.

11. The system of claim 7, wherein, The instructions, when executed by the processor, present in the graphical user interface on the display a two-dimensional re-projection image of the three-dimensional volume.

Images