System and method for fluorescence-ct imaging for initial registration

Abstract

This disclosure relates to systems and methods for fluorescence-CT imaging for initial registration. Systems and methods for registering a preoperative image dataset (e.g., CT data) or a 3D model derived from such a dataset with a patient's cavitary structures (e.g., airways in the lungs) using intraoperative fluorescence microscopy imaging techniques.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 888,905, filed August 19, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to surgical imaging systems, and more particularly to systems and methods for assisting clinicians in performing surgical procedures by registering preoperative images with intraoperative images to enable tools to navigate through cavity networks. Background Technology

[0004] Several commonly used medical methods, such as endoscopic or minimally invasive surgery, are employed to treat a variety of diseases affecting organs including the liver, brain, heart, lungs, gallbladder, kidneys, and bones. Clinicians typically use one or more imaging modalities, such as magnetic resonance imaging (MRI), ultrasound, computed tomography (CT), fluorescence microscopy, and others, to identify and navigate to regions of interest within the patient and ultimately to the target for treatment.

[0005] For example, endoscopic approaches have proven useful in navigating to regions of interest within a patient's interior, and particularly for regions within the body's luminal network, such as the lungs. To enable endoscopic approaches, and more specifically, bronchoscopic approaches within the lungs, endobronchial navigation systems have been developed that use preoperative or pre-acquired MRI or CT image data to generate a three-dimensional (3D) rendering or model of a specific body part. A navigation plan is then created using the resulting 3D model or rendering generated from the MRI or CT scan to facilitate the advancement of the navigation catheter (or other suitable medical device) through the bronchoscopy and luminal network, such as the patient's lung airways, to the identified target or region of interest.

[0006] However, to effectively navigate to a target or region of interest within the patient's lungs, the 3D model or rendering of the lungs derived from preoperative images must be registered to the patient's lungs. This is to ensure that the bronchoscope and other instruments inserted into the patient follow the preoperative plan, and their position within the patient must be aligned with the preoperative plan.

[0007] While current registration techniques are effective, improvements are always desirable, particularly improvements that could reduce the clinical hardware required to implement registration. Summary of the Invention

[0008] The present disclosure is a system and method of registering a fluoroscopic image and tissue and medical devices found therein with pre-operative CT image data. In addition, the present disclosure relates to a system and method of registering sensor positioning and location data to a fluoroscopic image. In addition, the present disclosure relates to using fluoroscopic imaging to register sensor positioning and location data with pre-operative CT image data.

[0009] One aspect of the present disclosure is a method of registering two image data sets, the method comprising performing a fluoroscopic glance of a desired portion of a patient and generating a 3D reconstruction from data received from the fluoroscopic glance. The method further comprises receiving an indication of a point in the 3D reconstruction that occurs in pre-operative CT image data, registering the 3D reconstruction to the pre-operative CT image data, displaying the 3D reconstruction, and displaying a portion of a navigation plan associated with the pre-operative CT image data on the 3D reconstruction. Other implementations of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0010] Implementations can include one or more of the following features. The received indication of a point can be a location of a major protuberance in the 3D reconstruction. The method can further include receiving an indication of two additional points in the 3D reconstruction. The indicated three points can be matched to points in the pre-operative CT image data. The method can further include solving two additional angles of orientation of the 3D reconstruction so that the 3D reconstruction matches the pre-operative CT image data. The method, wherein the 3D reconstruction matches a 3D model derived from the pre-operative CT image data. The method can further include searching the 3D reconstruction and the pre-operative CT image data to identify points of relevance. The method can further include the step of receiving an indication of a point in the 3D reconstruction that occurs in the pre-operative CT image data is a confirmation of a point selected from the search. The method can further include solving three orientation angles so that an orientation of the 3D reconstruction matches the pre-operative CT image data. Displaying the portion of the navigation plan depicts a location of a target identified in the pre-operative CT image data on the 3D reconstruction. Displaying the portion of the navigation plan depicts a path through a network of lumens to the target. Implementations of the described technology can include hardware, a method or process, or computer software on a computer-accessible medium.

[0011] One general aspect includes a system for registering fluoroscopic image data with pre-operative CT image data, the system including a computing device having a processor and a memory, the memory storing therein an application that, when executed by the processor, causes the processor to perform the steps of: generating a 3D reconstruction from data received from a fluoroscopic sweep; receiving an indication of a point in the 3D reconstruction that occurs in pre-operative CT image data; registering the 3D reconstruction to the pre-operative CT image data; and displaying the 3D reconstruction. The system further includes a display for displaying a portion of a navigation plan associated with the pre-operative CT image data on the 3D reconstruction based on the registration.

[0012] Further aspects relate to a method for registering images to a patient, the method including receiving positioning data of a sensor associated with a catheter, performing a fluoroscopic sweep. The method further includes generating a 3D reconstruction from data received from the fluoroscopic sweep and generating a 2D slice image from the 3D reconstruction. The method further includes receiving an indication of a positioning of the catheter in the 2D slice image and registering the 3D reconstruction to the positioning data of the sensor.

[0013] The method can further include receiving a second indication of a positioning of the catheter in a second 2D slice image. In addition, the method can include performing image processing to determine a positioning of the catheter in additional 2D slice images. The indication of the positioning of the catheter in the 2D slice image can be generated by image processing techniques. The method can further include receiving an indication of a point in the 3D reconstruction that occurs in pre-operative CT image data, registering the 3D reconstruction to the pre-operative CT image data, displaying the 3D reconstruction, and displaying a portion of a navigation plan associated with the pre-operative CT image data on the 3D reconstruction. The method can further include displaying a location of a sensor associated with the catheter in the 3D reconstruction based on the received positioning data. The method further includes updating the location of the sensor associated with the catheter as the catheter is navigated through the luminal network and new positioning data is received. BRIEF DESCRIPTION OF DRAWINGS

[0014] Various aspects and features of the disclosure are described below with reference to the accompanying drawings, in which:

[0015] Figure 1 depicts an imaging and navigation system in accordance with the present disclosure;

[0016] Figure 2A is a partial flowchart of an imaging and navigation procedure in accordance with the present disclosure;

[0017] Figure 2B is a partial flowchart of an imaging and navigation procedure in accordance with the present disclosure;

[0018] Figure 3A is a partial flowchart of an imaging and navigation procedure in accordance with the present disclosure;

[0019] Figure 3B is a partial flowchart of an imaging and navigation procedure according to the present disclosure;

[0020] Figure 4 depicts a user interface for marking structures in a fluoroscopic image according to the present disclosure;

[0021] Figure 4A depicts a user interface for marking a catheter in a fluoroscopic image according to the present disclosure;

[0022] Figure 5 depicts a user interface for marking a target in a fluoroscopic image according to the present disclosure;

[0023] Figure 6 depicts a user interface for navigating to a target according to the present disclosure;

[0024] Figure 7 depicts a mat with markers to be placed under a patient according to the present disclosure;

[0025] Figure 8 depicts features and components of a computing device according to the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure relates to systems and methods that can use intraoperative fluoroscopic imaging techniques to register a preoperative image dataset (e.g., CT data) or a 3D model derived from the preoperative image dataset with a patient's luminal structure (e.g., airways in the lungs).

[0027] Registration can be implemented using a variety of techniques. For example, a robotic system can be deployed to navigate an endoscope to points within the lung. By bringing these points into contact with the endoscope and correlating their locations within the patient's lung with locations within the 3D model, the 3D model can be registered with the patient's lung and with the coordinate system of the robot. In this way, the robot can then determine where a region of interest is located within the patient's lung and follow a navigation plan to the region of interest or develop a path through the lung to the region of interest.

[0028] Similarly, a flexible sensor can be employed to achieve registration. As the robot or clinician navigates an endoscope inside the patient, the shape (formed on or in the endoscope or other tool) of the flexible sensor as it advances and bends through the airways can cause it to sense a shape that matches the airways in the 3D model or rendering. This shape matching results in registration of the location of the endoscope in the patient with a location in the luminal network within the 3D model that has the same shape.

[0029] Another registration method employs electromagnetic (EM) sensors and EM navigation. An endoscope or another tool can include an EM sensor. An EM field generator generates an EM field, and when the EM sensor is placed in the EM field, an electric current is produced. This electric current is fed to a computer, which can determine the X, Y, Z, pitch, yaw, and roll coordinates (six degrees of freedom) of the EM sensor within the magnetic field. In practice, registration can be implemented in at least two different ways. In one, similar to the robotic system described above, the EM sensor can be placed in predefined positions inside the patient that can be viewed with a bronchoscope. Typically this is between 4 and 10 points. Matching of these points to the same points in the 3D model or rendering results in registration of the 3D model to the patient. In a second method, as the EM sensor is navigated through the airway network, the coordinates of the EM sensor are collected. When hundreds or thousands of these coordinates are collected, a point cloud of coordinates is created. Assuming that the point cloud taken from within the airway network has a 3D dimensional shape, this 3D dimensional shape can then be matched to the 3D shape inside the airway network. Once matched, the airway network in the 3D model and the airway network of the patient are registered. Once registered, the detected location of the EM sensor can be used to follow a navigation plan in the 3D model to a region of interest within the airway network of the patient.

[0030] Figure 1 is a perspective view of an exemplary system for navigation of a medical device, such as a biopsy or treatment tool, through the airways of the lung to a target. One aspect of the system 100 is a software application for viewing computed tomography (CT) image data that has been separately acquired from the system 100. Viewing the CT image data allows a user to identify one or more targets and plan a path to the identified target. This is often referred to as the planning phase. Another aspect of the software application is the navigation phase, which allows the user to navigate a catheter or other tool to the target using a user interface (the navigation phase) and confirm placement of the catheter or tool relative to the target. The target is often a tissue of interest for biopsy or treatment that is identified by viewing the CT image data in the planning phase. After navigation, a medical device such as a biopsy tool or a treatment tool can be inserted into the catheter to obtain a tissue sample from tissue located at the target or tissue proximate to the target or to treat such tissue. The treatment tool can be selected to effect microwave ablation, radiofrequency ablation, cryoablation, chemical ablation, or other treatment mechanism of the target preferred by the clinician.

[0031] Figure 1One aspect is a catheter system 102 including a sensor 104 at a distal end. The catheter system 102 includes a catheter 106. In practice, the catheter 106 is inserted into a bronchoscope 108 to access the luminal network of a patient P. Specifically, the catheter 106 of the catheter guide assembly 106 can be inserted into the working channel of the bronchoscope 108 for navigation through the luminal network of the patient. If configured for EMN (as described below), a locatable guide (LG) 110 including a sensor 104 such as an EM sensor can be inserted into the catheter 106 and locked into place such that the sensor 104 extends a desired distance beyond the distal tip of the catheter 106. It should be noted, however, that the sensor 104 can be incorporated into one or more of the bronchoscope 108, the catheter 106, or a biopsy or therapy tool without departing from the scope of the present disclosure.

[0032] If the catheter 106 is inserted into the bronchoscope 108, then both the distal end of the EWC 102 and the LG 110 extend beyond the distal end of the bronchoscope 108. The location or position and orientation of the distal portion of the sensor 104 and thus the LG 110 within the electromagnetic field can be derived based on positioning data in the form of electrical currents generated by an EM sensor present in the magnetic field or by other means described herein. Although the use of EM sensors and EMN is not required as part of the present disclosure, their use can further enhance the utility of the present disclosure in intraluminal navigation, such as navigation of the lungs. As the bronchoscope 108, catheter 106, LG 110, or other tools can be used interchangeably or in combination herein, the term catheter will be used here to refer to one or more of these elements. Further, as an alternative to the use of EM sensors, flexible sensors such as fiber Bragg sensors, ultrasonic sensors, accelerometers, and other sensors can be used in conjunction with the present disclosure to provide output to the tracking system 114 to determine the location of the catheter, including but not limited to the bronchoscope 108, catheter 106, LG 110, or a biopsy or therapy tool without departing from the scope of the present disclosure.

[0033] The system 100 generally includes an operating table 112 configured to support the patient P, a bronchoscope 108 configured for insertion into the airway of the patient P through the mouth of the patient P, a monitoring equipment 114 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 108) coupled to the bronchoscope 108. If configured for EMN, the system 100 can include a positioning or tracking system 114 and a positioning module 116, a plurality of reference EM sensors 118, and an emitter mat 120 including a plurality of integrated markers Figure 7 ). Although in Figure 7The markers are shown as repeating patterns, but other patterns— including three-dimensional markers that are different relative depths in the emitter pad 120— or non-repeating patterns can be employed without departing from the scope of the present disclosure. Also included is a computing device 122 that includes software and / or hardware to facilitate identification of the target, path planning to the target, navigation of the medical device to the target, and / or confirmation and / or determination of placement of the catheter 106 or suitable device therethrough relative to the target. The computing device 122 can be similar to the workstation 1001 of Figure 8 and can be configured to perform the methods of the present disclosure, including the methods of Figure 2A ,2B and Figure 3A ,3B

[0034] The computing device 122 can be any suitable computing device including a processor and a storage medium, where the processor is capable of executing instructions stored on the storage medium as one or more applications. The computing device 122 can also include a database configured to store patient data, CT data sets including CT images, fluoroscopic data sets including fluoroscopic images and videos, fluoroscopic 3D reconstructions, navigation plans, and any other such data. Although not explicitly illustrated, the computing device 122 can include inputs, or can be otherwise configured to receive CT data sets, fluoroscopic images / videos, and other data described herein. Additionally, the computing device 122 includes a display configured to display a graphical user interface. The computing device 122 can be connected to one or more networks through which one or more databases can be accessed. Further details of the computing device are described below in connection with Figure 8 .

[0035] With respect to the planning phase, the computing device 122 utilizes pre-acquired CT image data to generate and view a three-dimensional model or rendering of the airway of the patient P such that a target on the three-dimensional model can be identified (automatically, semi-automatically, or manually) and a determination of a path through the airway of the patient P to tissue at the target and tissue surrounding the target is allowed. More specifically, CT images and CT image data sets acquired from CT scans are processed and assembled into a three-dimensional CT volume, which is then used to generate a three-dimensional model of the airway of the patient P. The three-dimensional model can be displayed on a display associated with the computing device 122, or in any other suitable manner. An example of such a user interface can be found in Figure 6Various views of the three-dimensional model or augmented two-dimensional images generated from the three-dimensional model are presented using the computing device 122. The augmented two-dimensional images can have certain three-dimensional functionality because they are generated from three-dimensional data. The three-dimensional model can be manipulated to facilitate identification of a target on the three-dimensional model or the two-dimensional images, and a suitable path through the airway of the patient P into tissue located at the target can be selected. Once selected, the path plan, the three-dimensional model, and images derived therefrom can be saved and exported to a navigation system for use during a navigation phase.

[0036] As shown above, also included in the system 100 is a fluoroscopic imaging device 124 capable of acquiring fluoroscopic or X-ray images or videos (fluoroscopic image datasets) of the patient P. Images, sequences of images, or videos captured by the fluoroscopic imaging device 124 can be stored within the fluoroscopic imaging device 124, or transmitted to the computing device 122 for storage, processing, and display. Additionally, the fluoroscopic imaging device 124 can be moved relative to the patient P so that images can be acquired from different angles or perspectives relative to the patient P to create a sequence of fluoroscopic images such as a fluoroscopic video. The pose of the fluoroscopic imaging device 124 relative to the patient P and at the time of capturing images can be estimated using the markers 121 and various pose estimation and image processing techniques. The markers 121 can be incorporated into the emitter mat 120, incorporated into the operating table 112, or otherwise incorporated into another appliance placed on or near the operating table 112 so that they can be seen in the fluoroscopic images. The markers 121 are typically positioned below the patient P and between the patient P and a radiation source or sensing unit of the fluoroscopic imaging device 124. The fluoroscopic imaging device 124 can include a single imaging device or more than one imaging device.

[0037] With respect to Figure 2A and Figure 2B One method 200 employing the fluoroscopic imaging device 124 in the system 100 is described. As an initial step 202, the clinician wishes to view the navigation plan generated from the preoperative CT images, the navigation plan can be loaded and / or displayed on a display such as a display associated with the computer 122. After viewing the navigation plan, the clinician can insert one or more of the bronchoscope 108, the catheter 106, the LG 110 into the luminal network (e.g., airway) of the patient.

[0038] When the bronchoscope 108 captures images that can be viewed by the clinician as the bronchoscope 108 is advanced into the luminal network, the clinician cannot be confident that they are following the navigation plan derived from the pre-procedure CT image data. To ensure that the bronchoscope 108 is following the navigation plan, a fluoroscopic sweep of the patient can be performed at step 204. That is, as the fluoroscopic imaging device 124 is rotated around the patient, a series of fluoroscopic images can be acquired. The sweep can be between about 20 and 180 degrees around the patient, in some embodiments between 25 and 150 degrees, between 30 and 120 degrees, between 40 and 100 degrees, between 50 and 80 degrees, between 60 and 70 degrees, and any whole number integer between these ranges of angles. In particular embodiments, the sweep is 30 degrees, 40 degrees, or 50 degrees, but other angular sweeps can be taken without departing from the scope of the present disclosure.

[0039] Once a sufficient number of images have been acquired, a 3D reconstruction can be generated at step 206 at step 206. The 3D reconstruction of the fluoroscopic images results in a 3D volume of the region imaged during the fluoroscopic sweep. This 3D volume can be processed using a variety of techniques to provide real-time information to the clinician. In a first technique, the 3D reconstruction can be processed at step 208 to produce a series of 2D slice images. These 2D slice images are virtual fluoroscopic images in that they are generated from the 3D reconstruction but are not necessarily one of the fluoroscopic images acquired to render the 3D reconstruction. The 3D reconstruction can be sliced to produce 2D slice images along any axis that the clinician can desire, but for orientation purposes, the 2D images can be displayed in one or more of a standard axial view, a coronal view, or a sagittal view. These slice images can be presented in a user interface in a manner that the user can scroll through the slice images. Figure 4 A user interface 400 is depicted in which the user can scroll through a series of 2D slice images 402 using tabs 404 and bars 406, the latter of which represent the full set of 2D slice images generated from the 3D reconstruction.

[0040] By scrolling through the 2D slice images, the clinician can identify an indication of the location of the carina or another known anatomical feature, and at step 210, the application can receive an indication of the location of the carina or another known anatomical feature. The carina is a hard cartilaginous tissue that is the first branching point of the airways in the lung and marks the end of the trachea. Additionally, the carina is readily observable in the fluoroscopic images and 2D slice images from the fluoroscopic 3D reconstruction. However, other anatomical features are readily observable in the fluoroscopic images and 2D slice images from the fluoroscopic 3D reconstruction.

[0041] Depending on the application being executed by the processor in the computing device 122, the method can proceed to step 212, where the system 100 receives an indication of two more points in the 3D reconstruction. These points can be hila, blood vessels, ribs, fissures, or other features in the 2D slices of the 3D reconstruction. The only limitation is that the points need to be observable in both the 3D reconstruction and the preoperative CT image data. As one example, the three points can be the main hilum and the hila of the second bifurcation of the left and right lung lobes. All three of these points should be readily visible in the 3D reconstruction and specifically in the 2D slice images of the 3D reconstruction. Similarly, these points should be readily visible in the preoperative CT image data and the 3D model generated from the preoperative CT image data. These registration points can have been identified in the CT image data when the 3D model and navigation plan were constructed. Alternatively, these points can be identified after the 3D reconstruction is generated and three points therein are identified. In either case, the three points identified in both the 3D reconstruction and the 3D model from the CT image data must be matched to one another at step 214.

[0042] This matching of the three points in each of the 3D model and the 3D reconstruction allows for the registration of the 3D model with the 3D reconstruction at step 216. The registration ensures that all features in the 3D model (not just the three points identified) are aligned with the 3D reconstruction.

[0043] As an alternative to receiving an indication of two additional points in the 3D reconstruction at step 212, the application can instead solve the two additional degrees of freedom mathematically. That is, the identification of the main hilum or another known anatomical feature provides a single point for matching and ensures a degree of freedom for comparison and registration with respect to the 3D reconstruction and the 3D model from the CT image data. Specifically, by identifying the main hilum or another known anatomical feature in the 3D reconstruction, the application registers only a single point to the 3D model. But, with that single point ensured, the 3D model only needs to rotate the 3D reconstruction about three axes (e.g., X, Y, and Z) to seek a match of the orientation of the 3D model with the 3D reconstruction along those axes. Thus, at step 218, the application solves for at least two orientation angles such that the 3D reconstruction matches the 3D model. Again, the result of this matching is the registration of the preoperative 3D model with the 3D reconstruction at step 216. According to one aspect of the disclosure, the application solves for the two orientation angles by rotating the 3D reconstruction until the 3D reconstruction matches the 3D model as determined from a comparison of the gray scale or brightness or other common information of certain features appearing in both the 3D model and the 3D reconstruction.

[0044] A third option for registering the 3D reconstruction to the 3D model can be performed without receiving an indication of any points in the 3D reconstruction for matching with points in the 3D model (e.g., even without manually identifying the principal carina). According to this method, a search is performed on the 3D reconstruction and the CT image data (or 3D model) at step 220. The search finds points between the 3D reconstruction and the CT image data that have a correlation by analyzing the gray scale and common information of both image data sets. At step 222, the points having a gray scale match or common information are identified by the application. Once a sufficient number of points are identified, the application can select one or more of the points having a correlation or common information. One of the points is likely to be the principal carina, and the application can be optimized according to its size or general location or other parameters to solve for the principal carina.

[0045] At step 224, the application can select at least one of the points having a correlation or common information. Once selected, there are two options. In one aspect, at step 226, the application can present a request on the user interface for a confirmation that the point having a correlation or common information is correct and receive the confirmation. Once received, the method proceeds to step 218 and solves for at least two orientation angles so that the other points in the 3D reconstruction and 3D model match, as described above.

[0046] Alternatively, instead of selecting only a single point at step 224 and receiving a confirmation at step 226, the application can select multiple points at step 224, and the application can proceed to step 228 where, in the case that multiple points having a correlation and common information are identified, the application can solve for all three orientation angles. Once the three angles are solved, the 3D reconstruction can be registered to the 3D model at step 316.

[0047] The process of the application selecting points at step 224 and solving for a confirmation and common information at step 228 can be implemented by the computing device 122, which stores a learning algorithm in a memory therein. For each procedure, whether implemented manually or automatically by the application, the results can be analyzed by the learning algorithm to refine the attributes and parameters of the points to be selected according to these methods. For each procedure, the attributes and parameters (e.g., brightness in the CT image, proximity to other points, etc.) are identified and added to the experiential aspect of the learning algorithm to further refine the algorithm for future procedures.

[0048] With respect to any of the processes described above, the computing device 122 can utilize the locations of the markers 121 in the fluoroscopic images. This technique relies on the markers 121 being positioned in a non-repeating pattern. However, this non-repeating pattern is known, and the relative positions of any individual marker 121 to the antennas of the emitter pad 120 are also known. In essence, the locations of the markers 121 are registered to one another during the manufacture of the emitter pad 120 compared to the antennas of the emitter pad 120. This known relative position of the markers 121 to the antennas of the emitter pad 120 can be used by the computing device 122 and the fluoroscopic imaging device to identify the specific marker 121 of the markers 121 that appears in the fluoroscopic image. Once the marker 121 is identified in the fluoroscopic image, the computing device can use the known relative position of the marker 121 to the antennas of the emitter pad 121 to register the coordinates of the fluoroscopic image to the coordinates of the antennas of the emitter pad 121. In this way, the catheter position in the 3D reconstruction can be compared to the catheter position detected by the EM, allowing the 3D reconstruction to be registered to the 3D model

[0049] Once the 3D reconstruction and the 3D model from the navigation plan are registered to one another at step 216, at step 230 the application can cause the 3D reconstruction to be displayed on a display associated with the computing device 122. By displaying the 3D reconstruction, features from the navigation plan can be imported into and displayed on the 3D reconstruction since the preoperative 3D model is registered to the 3D reconstruction. This can be as an overlay on the 3D reconstruction at step 232. Alternatively, the imported features from the 3D model can be fused together with the 3D reconstruction. Other techniques for incorporating features from the 3D model and the navigation plan with the 3D reconstruction can also be used without departing from the scope of the present disclosure. These features can be selectively applied to the 3D reconstruction. For example, indications of the path plan and / or target positioning can be shown in the 3D reconstruction.

[0050] Once these features are imported into the displayed 3D reconstruction, the navigation plan can be followed until the catheter (e.g., bronchoscope 108, catheter 106) reaches the target. Optionally, at step 234 the application can determine when the bronchoscope or tool following the navigation plan is within a threshold distance from the target and provide an indication on the user interface. This can be done by comparing the bronchoscope images generated by the bronchoscope to the virtual bronchoscope images generated by the 3D reconstruction. In the event that the bronchoscope has become wedged and is no longer able to navigate through the airways, the proximity determination of the catheter 106 or other tool can require a new fluoroscopic sweep (i.e., return to step 204) or other traditional fluoroscopic imaging techniques.

[0051] In any event, once the bronchoscope or tool is proximate to the target, a second fluoroscopic sweep is performed at step 236. This second fluoroscopic sweep is to determine the location of the target with improved accuracy, and importantly, to determine the relative position of the bronchoscope or tool with respect to the target. After the sweep is performed as described above, the user interface can present the user with the fluoroscopic images and request that the user identify the target in the fluoroscopic images at step 238. An example of the user interface 500 that can be presented to the user is shown in Figure 5 where a scrollable fluoroscopic image 502 is presented to the user. Once identified in one fluoroscopic image 502, the user interface allows the user to scroll using the scroll bar 504 to identify the second fluoroscopic image in which the target is identified. Alternatively, the application can search the fluoroscopic images and automatically identify the target. Similarly, the user interface can present the user with a user interface in which the user identifies the end of the catheter (e.g., bronchoscope 108 or catheter 106). The application receives this indication at step 240.

[0052] Once the target and catheter are identified in the fluoroscopic images, a second 3D reconstruction can be generated at step 242 and displayed at step 244. This display of the 3D reconstruction includes a clear delineation of the target as marked in the fluoroscopic images of the fluoroscopic sweep at step 240. This provides an accurate indication of the location of the target and the relative location of the catheter (e.g., bronchoscope 108 or catheter 106), and can make a determination as to whether the catheter is aligned with the target and the distance from the end of the catheter to the target. The relative position data can be displayed on the user interface, or the clinician can simply determine the alignment based on the observation of the 3D reconstruction. If the target is aligned with the bronchoscope or tool at step 246, the method can proceed to step 248 where a biopsy sample or treatment is performed.

[0053] If it is determined that the tool and target are not aligned, the method proceeds to step 250 where the catheter (e.g., bronchoscope 108 or catheter 106) or tool is repositioned. After repositioning, the method returns to step 236 to perform another fluoroscopic sweep. This procedure can be repeated as necessary until alignment is achieved at step 246, and a biopsy or treatment can be performed at step 248.

[0054] As an alternative, the fluoroscopic sweep 236 can return the process to the fluoroscopic sweep 204, where a new 3D reconstruction is generated at step 206. The process can then continue as described in steps 206-216, and all of the registration arrangements described above (e.g., steps 210-228) and navigation plan data can be applied to the new 3D reconstruction and displayed therewith.

[0055] This fast generation of a 3D reconstruction of the region of interest can provide real-time 3D imaging of the target. Real-time imaging of the target and medical devices positioned in the region of the target can be beneficial for many interventional procedures, such as biopsy and ablation procedures in different organs, vascular interventions, and orthopedic surgeries. For example, when considering navigational bronchoscopy, the goal can be to receive accurate information about the location of the catheter with respect to the target to ensure accurate treatment or biopsy.

[0056] As another example, minimally invasive procedures including robotic assisted surgical procedures such as laparoscopic procedures can employ intraoperative fluoroscopy to increase visualization, for example, to guide and lesion localization and to prevent unnecessary damage and complications. Employing the above mentioned systems and methods for real-time reconstruction of fluoroscopic 3D imaging of a target region and for navigation based on the reconstruction can also be beneficial for such procedures.

[0057] As described above, the system 100 can be configured for electromagnetic navigation (EMN). When performing EMN, the system 100 employs a six degrees of freedom electromagnetic positioning or tracking system 114 or other suitable system to determine positioning data for the sensor 104, such as an EM sensor. The tracking system 114 is configured for use with the positionable guide 110 and, in particular, the sensor 104. As described above, the positionable guide 110 and the sensor 104 are configured for insertion through the catheter 106 into the airway of the patient P (with or without the bronchoscope 108) and can be selectively locked with respect to one another via a locking mechanism.

[0058] The transmitter pad 120 is positioned beneath the patient P. The transmitter pad 120 generates an electromagnetic field around at least a portion of the patient P within which the position of the plurality of reference sensors 118 and the sensor 104 can be determined using the tracking module 116. A second electromagnetic sensor 104 can also be incorporated into the end of the catheter 106. Additionally or alternatively, the second electromagnetic sensor 104 can be incorporated into a biopsy tool or a treatment tool for use in the procedure.

[0059] The second electromagnetic sensor 104 can be a five degrees of freedom sensor or a six degrees of freedom sensor. One or more of the reference sensors 118 are attached to the chest of the patient P. The six degrees of freedom coordinates of the reference sensors 118 are sent to the computing device 122 (which includes appropriate software) where they are used to compute a patient reference coordinate system.

[0060] When the system 100 is configured for EMN, registration is needed to transform the detected EM coordinates of the sensor 104 into CT image data coordinates so that the detected positioning or location of the sensor 104 can be displayed in the CT image data (e.g., in the 3D model or navigation plan) and updated as the sensor 104 is navigated through the luminal network. As described above, for the EMN-enabled system 100, this registration can be performed (among other methods) by inserting the sensor 104 into the airway and generating a point cloud of the detected location of the sensor 104. The matching of the point cloud to the airway of the 3D model registers the actual airway of the patient to the 3D model. Further, this process defines the conversion from EMN coordinates (the detected location of the sensor in the EM field) to CT image data coordinates. In this manner, the navigation plan can be followed and the detected positioning of the sensor 104 can be presented as the sensor 104 in the 3D model and, then, the catheter (e.g., bronchoscope 108 or catheter 106) is traversed through the luminal network.

[0061] However, when performing the initial registration of the navigation plan to the luminal network of the patient using the above-described fluoroscopic imaging techniques, there is no bridge from the EM coordinates to the CT image data coordinates and, thus, the progress in the navigation plan cannot be updated as the catheter is navigated through the luminal network. Although repeated fluoroscopic imaging can update the position of the catheter (e.g., bronchoscope 108 or catheter 106) in the navigation plan, this results in additional radiation to the patient and clinical staff. Instead, a bridge between the EM coordinates and the CT coordinates can be achieved by using the fluoroscopic imaging techniques. Specifically, the registration of the fluoroscopic image data from the fluoroscopic sweeps to the detected location of the sensor 104 in combination with the registration of the fluoroscopic image data to the pre-operative CT image data and the navigation plan results in an empirical transformation that allows for the registration of the EM coordinate system to the pre-operative CT image data coordinate system.

[0062] Figure 3A and Figure 3B A method of implementing fluoroscopic image data registration to the detected EMN coordinates of the sensor 104 is depicted. The method 300 begins with the application on the computing device 122 loading the navigation plan developed from the pre-operative CT image data at step 302. Once loaded, the catheter (e.g., bronchoscope 108 or catheter 106) including the sensor 104 can be inserted into the EM field generated by the transmitter mat 120. As in the above-described method, the navigation plan is displayed on the display 124 and the detected location of the sensor 104 is displayed in the 3D model of the luminal network. Figure 1As shown in FIG. 2, the transmitter pad 120 is placed directly under the patient P and the EM field will be generated around the patient. In a scenario where lung navigation is desired, placing the sensor 104 in the EM field would include placing a catheter (e.g., bronchoscope 108 or catheter 106) with the sensor 104 into the patient's airways, e.g., to a point near the main carina. The exact positioning of the catheter and sensor 104 is not critical, so long as it is at a position that can be imaged by the fluoroscopic imaging device 124. Once within the EM field, the sensor 104 will generate a current that can be analyzed by the localization module 116 in the tracking system 114 to determine the location of the sensor 104 in the EM field at step 304. That is, step 304 identifies the EM coordinates (localization data) of the sensor 104.

[0063] At this point, the fluoroscopic imaging device 124 can take a fluoroscopic sweep at step 306. A 3D reconstruction can be formed from the images taken by the fluoroscopic imaging device 124 at step 308, and 2D slice images of the 3D reconstruction are generated at step 310. Steps 306-310 can be the same steps as 204-208 of Figure 2A and need not be repeated.

[0064] Once the 2D slice images are generated, the application can present one of the slices to the user on the user interface at step 312 and request that the user identify the location of the catheter tip in the image as depicted in Figure 4 or Figure 4A The location of the distal tip of the catheter (e.g., bronchoscope 108, catheter 106, LG 110, or biopsy or treatment tool) serves as an identification of the location of the sensor 104 in the 2D slice image. The location of the sensor 104 relative to the tip of the catheter can be known to the application, e.g., saved in the memory of the computing device 122. At step 314, the user interface presents a second 2D slice image from the 3D reconstruction and requests identification of the catheter tip in the second 2D slice image. As shown in Figure 4A , both images can be presented at the same time. If the two images are from widely spread portions of the fluoroscopic sweep (i.e., at a wide angle from each other), the application can accurately determine the location of the catheter tip and, thus, the location of the sensor 104 in the 3D reconstruction.

[0065] Because the location of the sensor 104 in the EM field is known from the localization module 116 and has been determined in the 3D reconstruction, the EM N-coordinate system and the coordinate system of the fluoroscopic imaging device 124 can be registered to each other at step 318.

[0066] Instead of receiving an indication of the catheter tip location in both 2D slice images, the application can implement an image processing step at step 316 that identifies the catheter. This can optionally be aided by the presence of fiducial markers formed at intervals along the length of the catheter. Even without fiducial markers, the shape of the catheter (e.g., bronchoscope 108, catheter 106, LG 110, or biopsy or treatment tool) should be readily identifiable in the 2D slices of the fluoroscope 3D reconstruction. By identifying the catheter in each of the 2D slice images, the application can determine the location of the tip and, in turn, the location of the sensor 104 in the 3D reconstruction.

[0067] In addition to receiving manual identification of the location of the catheter tip or an automatic image processing procedure, the present application contemplates a combination of the two. In this scenario, the application receives an indication of the location of the catheter tip in both images and performs image processing on all or a substantial portion of the remaining 2D slice images. After this combined procedure, the coordinates of the fluoroscope imaging device 124 and the conversion of image data derived therefrom to EMN coordinates are derived, and the 3D reconstruction is registered to the location of the sensor 104, 128 detected in the EM field.

[0068] As described above, at step 320, registration of the 3D reconstruction to preoperative CT image data can be performed. Any method of registering the 3D reconstruction to the preoperative CT image data can be employed. Once both registration procedures have been performed, all three coordinate systems are registered to one another. The fluoroscope coordinate system to the preoperative CT imaging coordinate system, and the fluoroscope coordinate system to the EMN coordinate system. As a result, a transformation for registering the EMN coordinates to the preoperative CT imaging coordinate system is established.

[0069] With the multiple registrations, the application can either proceed by simply using the registration of the sensor 104 to the preoperative CT image data to update the location of the detected EM sensor in the navigation plan developed from the preoperative CT image data and display the navigation plan at step 322. Using the navigation plan, the location of the detected sensor 104, and following the path defined in the navigation plan, the sensor 104 can be navigated to the target in the navigation plan.

[0070] Optionally, at step 324, the application can determine when the catheter (e.g., bronchoscope 108 or WC 1060 is within a threshold distance from the target and provide an indication on the user interface. Regardless, once the catheter is proximate to the target, a second fluoroscopic sweep is performed at step 326. This second fluoroscopic sweep is to determine the location of the target with improved accuracy, and importantly, to determine the relative position of the bronchoscope 108 or another tool with respect to the target. After implementing the sweep as described above, the user interface can present the fluoroscopic image to the user and request that the user identify the target in the fluoroscopic image, which the application receives at step 328. Once identified, the user interface can present the second fluoroscopic image to the user in which to identify the target, as shown in Figure 5 . Alternatively, the application can search the fluoroscopic image and automatically identify the target.

[0071] Once the target is identified in the fluoroscopic image, the user interface can present the fluoroscopic image to the user in which to identify the catheter tip, which the application receives at step 330, as shown in Figure 4 and Figure 4A . A second 3D reconstruction can be generated at step 332, and the relative position of the catheter tip to the target can be updated in the navigation plan derived from the preoperative CT image data. This updated relative position in the navigation plan can be displayed on the user interface 602 at step 334, as seen in Figure 6 . This provides an accurate indication of the location of the catheter tip with respect to the target, and it can be determined whether the sensor 104 is aligned with the target, and the distance from the sensor 104 and then from the end of the bronchoscope 108 or other tool to the target. This data can be displayed on the user interface, or the clinician can simply determine the alignment based on observation of the 3D reconstruction. If the target is aligned with the bronchoscope or tool at step 336, the method can proceed to step 338, where a biopsy sample or treatment is performed.

[0072] If it is determined that the sensor 104 and target are not aligned, the method proceeds to step 340, where the bronchoscope 108 or another tool is repositioned. After repositioning, the method returns to step 326 to implement another fluoroscopic sweep. This procedure can be repeated as necessary until alignment is achieved at step 338, and a biopsy or treatment can be performed at step 338.

[0073] Reference is now made to Figure 8 , which is configured for use with a system comprising Figure 2A , 2B, and Figure 3AFIG. 1 is a schematic diagram of a system 1000 for use with the methods of the present disclosure, including the methods of FIGS. 2B and 3B. The system 1000 can include a workstation 1001 and optionally a fluoroscopic imaging device or fluoroscope 1015. In some embodiments, the workstation 1001 can be coupled directly or indirectly with the fluoroscope 1015, for example, through wireless communication. The workstation 1001 can include a memory 1002, a processor 1004, a display 1006, and an input device 1010. The processor or hardware processor 1004 can include one or more hardware processors. The workstation 1001 can optionally include an output module 1012 and a network interface 1008. The memory 1002 can store applications 1018 and image data 1014. The applications 1018 can include instructions that can be executed by the processor 1004 to perform the methods including Figure 2A ,2B and Figure 3A ,3B of the present disclosure.

[0074] The applications 1018 can also include a user interface 1016. The image data 1014 can include CT scans, generated fluoroscopic 3D reconstructions of a target region, and / or any other fluoroscopic image data and / or generated one or more virtual fluoroscopy images. The processor 1004 can be coupled with the memory 1002, the display 1006, the input device 1010, the output module 1012, the network interface 1008, and the fluoroscope 1015. The workstation 1001 can be a fixed computing device such as a personal computer or a portable computing device such as a tablet computer. The workstation 1001 can be embedded in a plurality of computer devices.

[0075] The memory 1002 can include any non-transitory computer-readable storage media for storing data and / or software, including instructions that can be executed by the processor 1004 and which control the operation of the workstation 1001 and, in some embodiments, also the fluoroscope 1015. In accordance with the present disclosure, the fluoroscope 1015 can be used to capture a sequence of fluoroscopic images and to capture real-time 2D fluoroscopic views, a fluoroscopic 3D reconstruction being generated based on the sequence of fluoroscopic images. In embodiments, the memory 1002 can include one or more storage devices, such as solid state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid state storage devices, the memory 1002 can include one or more mass storage devices connected to the processor 1004 by a mass storage controller (not shown) and a communication bus (not shown).

[0076] Although the description of the computer-readable medium included herein refers to a solid state storage, those skilled in the art will appreciate that the computer-readable storage medium can be any available medium that can be accessed by the processor 1004. That is, the computer-readable storage medium can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, surgical modules, or other data. For example, the computer-readable storage medium can include RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology; CD-ROM, DVD, Blu-Ray, or other optical storage; magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to store the desired information and that can be accessed by the workstation 1001.

[0077] The application 1018, when executed by the processor 1004, can cause the display 1006 to present a user interface 1016. The user interface 1016 can be configured to present a single screen to a user that includes: a three-dimensional (3D) view of a 3D model of a target from a tip perspective of a medical device; a real-time two-dimensional (2D) fluoroscopy view showing the medical device; and a target marker corresponding to the 3D model of the target overlaid on the real-time 2D fluoroscopy view. The user interface 1016 can also be configured to display the target marker in different colors depending on whether the medical device tip is in three-dimensional alignment with the target.

[0078] The network interface 1008 can be configured to connect to a network such as a local area network (LAN) consisting of wired and / or wireless networks, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. The network interface 1008 can be used to make a connection between the workstation 1001 and the fluoroscope 1015. The network interface 1008 can also be used to receive the image data 1014. The input device 1010 can be any device through which a user can interact with the workstation 1001 such as, for example, a mouse, a keyboard, a foot pedal, a touch screen, and / or a voice interface. The output module 1012 can include any connectivity port or bus such as, for example, a parallel port, a serial port, a universal serial bus (USB), or any other similar connectivity port known to those skilled in the art.

[0079] While several aspects of the present disclosure have been described, it should be apparent that various modifications, adaptations, and variations of those aspects can be developed by those skilled in the art in light of the foregoing description and the appended claims. Accordingly, the particular aspects discussed above are provided for illustration only and not for limitation.

Claims

1. A method of registering two image data sets, the method comprising: performing a fluoroscopic sweep of a desired portion of a patient to generate a fluoroscopic image data set; generating a 3D reconstruction from data received from the fluoroscopic sweep; performing a search of the 3D reconstruction and pre-operative CT image data to identify at least one point of gray scale correlation or common information; registering the 3D reconstruction to the pre-operative CT image data, wherein the registration includes solving for two rotation angles by rotating the 3D reconstruction about three axes such that the 3D reconstruction matches the pre-operative CT image data along the three axes; displaying the 3D reconstruction; and displaying portions of a navigation plan associated with the pre-operative CT image data on the 3D reconstruction based on the registration.

2. The method of claim 1, wherein, the 3D reconstruction matches a 3D model derived from the pre-operative CT image data.

3. The method of claim 1, wherein, displaying portions of a navigation plan depict the location of a target identified in the pre-operative CT image data on the 3D reconstruction.

4. The method of claim 3, wherein, displaying portions of a navigation plan depict a path through a network of lumens to the target.

5. The method of claim 1, further comprising performing a second fluoroscopic sweep.

6. The method of claim 5, further comprising identifying a target in two fluoroscopic images from the second fluoroscopic sweep.

7. The method of claim 6, further comprising identifying portions of a catheter in the two fluoroscopic images from the second fluoroscopic sweep.

8. The method of claim 6, further comprising generating a partial 3D reconstruction.

9. The method of claim 8, further comprising confirming that the catheter is aligned with the target in the 3D reconstruction.

10. A system for registering fluoroscopic image data with pre-operative CT image data, comprising: a computing device including a processor and a memory, the memory having an application stored therein, which when executed by the processor, causes the processor to perform the following steps: generating a 3D reconstruction from data received from a fluoroscopic sweep; performing a search of the 3D reconstruction and pre-operative CT image data to identify at least one point of gray scale correlation or common information; registering the 3D reconstruction to the pre-operative CT image data, wherein the registration includes solving for two rotation angles by rotating the 3D reconstruction about three axes such that the 3D reconstruction matches the pre-operative CT image data along the three axes; displaying the 3D reconstruction; and a display for displaying portions of a navigation plan associated with the pre-operative CT image data on the 3D reconstruction based on the registration.

11. The system of claim 10, wherein the application, when executed, further causes the processor to perform the step of performing a second fluoroscopic sweep.

12. The system of claim 11, wherein the application, when executed, further causes the processor to perform the step of identifying a target in two fluoroscopic images from the second fluoroscopic sweep. ​ 13. The system of claim 12, wherein the application, when executed, further causes the processor to perform the step of: identifying a portion of the catheter in two fluoroscopic images from a second fluoroscopic sweep.

14. The system of claim 13, wherein the application, when executed, further causes the processor to perform the step of: generating a local 3D reconstruction.

15. The system of claim 14, wherein the application, when executed, further causes the processor to perform the step of: confirming that the catheter is aligned with the target in the 3D reconstruction.

Images