Method and system for clinical tool tracking and visualization for motion stabilization

Abstract

Various methods and systems for x-ray imaging are provided. In one embodiment, a method includes: acquiring a plurality of fluoroscopic images depicting an interventional tool positioned relative to an anatomical structure of interest of a patient; segmenting the interventional tool in the plurality of fluoroscopic images; measuring motion of the patient in the plurality of fluoroscopic images; correcting the plurality of fluoroscopic images to remove the motion of the patient; registering the segmented interventional tool to the anatomical structure of interest in the corrected plurality of fluoroscopic images; and displaying an image with the segmented interventional tool registered to the anatomical structure of interest. In this way, a physician can view the position and movement of an interventional tool located within a patient's body relative to a static image of the anatomical structure without motion artifacts or errors caused by patient motion such as respiratory motion or cardiac motion.

Description

TECHNICAL FIELD

[0001] Embodiments of the subject matter disclosed herein relate to x-ray imaging. BACKGROUND

[0002] Imaging techniques, such as x-ray imaging, allow for the non-invasive acquisition of images of internal structures or features of a subject or object. Digital x-ray imaging systems produce digital data that can be reconstructed into radiographic images. In a digital x-ray imaging system, radiation from a source is directed at a subject in a medical application, a package or luggage in a security screening application, or a manufactured part in an industrial quality control inspection application. A portion of the radiation passes through the subject / object and impacts a detector. The detector includes an array of discrete picture elements or detector pixels, and generates an output signal based on the amount or intensity of radiation that impacts each pixel area. The output signal is subsequently processed to generate an image that can be displayed for viewing. These images are used to identify and / or inspect internal structures and organs within a patient's body, objects within a package or container, or defects such as cracks within a manufactured part. SUMMARY

[0003] In one embodiment, a method of creating a motion-adjusted image of a patient to guide an interventional medical procedure includes acquiring a plurality of fluoroscopic images of an anatomical structure of interest of a patient, the plurality of fluoroscopic images depicting an interventional tool positioned relative to the anatomical structure of interest; segmenting the interventional tool in the plurality of fluoroscopic images; measuring motion of the patient in the plurality of fluoroscopic images; correcting the plurality of fluoroscopic images based on the motion of the patient; registering the segmented interventional tool to the anatomical structure of interest in the corrected plurality of fluoroscopic images; and displaying an image with the segmented interventional tool registered to the anatomical structure of interest. In this way, a physician can view the position and movement of an interventional tool located within a patient's body relative to a static image of the anatomical structure without motion artifacts or errors caused by patient motion such as respiratory motion or cardiac motion.

[0004] It should be appreciated that the above Brief Description of the Invention is provided merely for purposes of summarizing some select concepts of the detailed description that follows. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined solely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the above Background or any part of this disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0005] The present application will be better understood with a reading of the following description of non-limiting embodiments, the said description being produced with reference to the attached drawings in which:

[0006] Figure 1 An exemplary x-ray imaging system according to one embodiment is shown;

[0007] Figure 2 A block diagram is shown illustrating an exemplary system for interventional tool tracking and guidance during fluoroscopy imaging, according to one embodiment;

[0008] Figure 3 A high-level flow diagram is shown illustrating an exemplary method for interventional tool tracking and guidance, according to one embodiment;

[0009] Figure 4 A block diagram is shown illustrating an exemplary graphical user interface for displaying fluoroscopy images and interventional tool guidance, according to one embodiment;

[0010] Figure 5 A high-level flow diagram is shown illustrating an exemplary method for stabilized tool rendering, according to one embodiment;

[0011] Figure 6 A set of images is shown illustrating unstable positions of an interventional tool in fluoroscopy imaging due to patient motion, according to one embodiment; and

[0012] Figure 7 A block diagram is shown illustrating an exemplary graphical user interface for displaying stabilized tool rendering, according to one embodiment. DETAILED DESCRIPTION

[0013] The following description relates to various embodiments of x-ray imaging. In particular, systems and methods for motion-stabilized clinical tool tracking and visualization are provided. An x-ray imaging system, such as a C-arm, is configured to create a stabilized rendering for tool navigation in fluoroscopy. Figure 1 The depicted x-ray imaging system includes an x-ray detector positionable relative to an x-ray source. The x-ray imaging system can include a clinical tool tracker, such as a C-arm, configured to create a stabilized rendering for tool navigation in fluoroscopy. Figure 2 As shown, the clinical tool tracker provides an accurate fusion between a fluoroscopy sequence that moves with respiration and a static rendering of the anatomy. A method for motion-stabilized clinical tool tracking, as shown, includes tool segmentation, respiration motion estimation, positioning the segmented tool relative to the anatomy of interest, and specialized rendering of tool motion. As shown, the images generated by the clinical tool tracker provide a view in which tool motion corresponds only to operator pose and not respiration motion, which allows the operator to simply assess their position relative to the anatomy of interest. As shown, Figure 3 Figure 4 Figure 5 ​​As shown, another method for motion stabilized clinical tool tracking includes decomposing motion between images to determine patient motion such as respiratory motion or cardiac motion, and displaying images of a model of the anatomy of interest, with segmented images of the interventional tool overlaid thereon. In the case of an unstable interventional tool, the position of the interventional tool drifts due to motion of the patient in the displayed fluoroscopic images, as shown. By stabilizing the position of the imaged interventional tool relative to the model of the anatomy of interest, as shown, navigation of the interventional tool relative to the anatomy of interest is improved. Figure 6 Figure 7

[0014] Turning now to Figure 1 , a block diagram of an x-ray imaging system 100 is shown in accordance with one embodiment of the present disclosure. The x-ray imaging system 100 includes an image acquisition unit 102 and an operating console 142. The operating console 142 includes a processor 181, a memory 182, an x-ray controller 187, an x-ray data acquisition unit 191, an image processor 192, and a clinical tool tracker 194 for tracking and visualizing motion of a clinical tool 130 positioned within a subject 112. The operating console 142 is communicatively coupled to a user interface 183 and a display device 195, as depicted, although it should be appreciated that in some examples, the operating console 142 can also include one or more of the user interface 183 and the display device 195. In some examples, the x-ray imaging system 100 includes a mobile x-ray imaging system such that the image acquisition unit 102 and the operating console 142 are portable or mobile.

[0015] The image acquisition unit 102 includes a radiation source, such as an x-ray source 104. The x-ray source 104 is configured to emit a beam of radiation, such as an x-ray beam 106 having a field of view toward an object 110. In Figure 1 examples, the object 110 is an anatomical region or region of interest in a subject, such as a patient 112. In another example, the object 110 can correspond to a package or luggage in a security screening application. In yet another example, the object 110 can be a manufactured part in an industrial application. In some examples, the image acquisition unit 102 can also include a C-arm (not shown), with the x-ray source 104 and the x-ray detector 108 mounted to opposite ends of the C-arm.

[0016] ​​In some examples, the x-ray imaging system 100 further includes a patient table (not shown) configured to support the patient 112. The x-ray beam 106 can be differentially attenuated by portions of the anatomical region 110 when impinging thereon. An x-ray detector 108 disposed in a field of view of the x-ray beam 106 acquires the attenuated x-ray beam. By way of non-limiting example, the x-ray detector 108 can include an x-ray exposure monitor, an electrical substrate, or the like. The x-ray detector 108 can be moved by an operator of the mobile x-ray imaging system 100 to be positioned manually with respect to the x-ray beam 106.

[0017] The operator console 142 includes a processor 181, a memory 182, an x-ray controller 187, an x-ray data acquisition unit 191, an image processor 192, and a grid artifact correction unit 193. X-ray image data acquired by the x-ray detector 108 is transmitted from the x-ray detector 108 and received by the x-ray data acquisition unit 191. The collected x-ray image data is image processed by the image processor 192. A display device 195 communicatively coupled to the operator console 142 displays the image processed x-ray image thereon. The x-ray controller 187 supplies power of suitable voltage current to the x-ray source 104 for powering the x-ray source 104.

[0018] The image acquisition unit 102 is further configured to generate an x-ray image corresponding to the subject 110 based on the detected x-ray beam. In Figure 1 In examples, the x-ray image is a projection of the anatomical region 110 of the subject 112 in a detector plane of the x-ray detector 108.

[0019] The image processor 192 is communicatively coupled to the x-ray data acquisition unit 191 and is configured to receive the x-ray image from the x-ray data acquisition unit 191. In some examples, the image processor 192 is configured to identify a medical condition of the anatomical region 110 of the subject 112 based on the x-ray image. In one embodiment, the image processor 192 is configured to display the x-ray image, the identified medical condition, or a combination thereof on the display device 195. To this end, the image processor 192 processes the x-ray image with one or more image processing techniques including, but not limited to, segmentation techniques, deep learning techniques, or the like.

[0020] In some examples, the display device 195 can be integrated with the user interface 183. For example, the display device 195 can include a touch-sensitive display device or a touch screen such that the display device 195 can display a graphical user interface and detect inputs of an operator.

[0021] Further, the processor 181 is communicatively coupled to the memory unit 182, the image processor 192, and the clinical tool tracker 194 via the communication bus 190 and is configured to provide computational and control functions. The processor 181 includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor, and a controller. In other embodiments, the processor 181 includes custom processor elements such as, but not limited to, application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). The processor 181 can be further configured to receive commands and / or parameters from an operator via the user interface 183. In some embodiments, the processor 181 can perform one or more functions of at least one of the image processor 192 and the clinical tool tracker 194. The processor 181 can include more than one processor that work in coordination with each other to perform the functions described herein. The processor 181 can also be configured to store content to and retrieve content from the memory 182. In one example, the processor 181 is configured to initiate and control the functions of at least one of the image acquisition unit 102 and the clinical tool tracker 194.

[0022] In one embodiment, the memory 182 includes random access memory (RAM), read only memory (ROM), flash memory, or any other type of computer readable memory accessible by one or more of the image acquisition unit 102, the clinical tool tracker 194, the image processor 192, and the processor 181. Additionally, in some examples, the memory 182 includes a non-transitory computer readable medium encoded with a program having a plurality of instructions to instruct at least one of the image acquisition unit 102, the clinical tool tracker 194, the image processor 192, and the processor 181 to perform a sequence of steps to generate an x-ray image. The program can also instruct the display device 195 to display the x-ray image to an operator to evaluate the x-ray image.

[0023] As an illustrative example, Figure 2 A block diagram is shown that illustrates an example system 200 for interventional tool tracking and guidance, according to one embodiment. In particular, the method 200 involves tracking an interventional tool with a clinical tool tracker 210. For example, the clinical tool tracker 210 can include the clinical tool tracker 194 of the x-ray imaging system 100.

[0024] The current image 205 and the prior image 207 are provided to a clinical tool tracker 210. The current image 205 includes a fluoroscopic image acquired via the x-ray image acquisition unit 102. The clinical tool tracker 210 processes the current image 205 to locate an interventional tool, such as the interventional tool 130, within the current image 205 and track the motion of the interventional tool over time. The clinical tool tracker 210 also generates a guidance image by superimposing a segmented image of the interventional tool on the prior image 207. The prior image 207 can include a two- or three-dimensional image of patient anatomy acquired prior to the fluoroscopy, in which the current image 205 is acquired, and the prior image can be static or dynamic. The prior image 207 can include a 3D segmented object projection rendering, a previously recorded image such as a road map of vessels, a digital subtraction angiogram, a contrast injection sequence, an image manual annotation, a pre-operative volume projection, etc. The prior image 207 can be acquired via the fluoroscopic imaging modality or one or more imaging modalities of the x-ray imaging system 100 in addition to the fluoroscopic imaging modality.

[0025] The clinical tool tracker 210 includes a segmentation module 212, a tool model module 214, a motion estimation module 216, a registration module 218, and a rendering module 220. The segmentation module 212 is configured to segment a clinical tool or interventional tool in the current image 205. The tool model module 214 models the orientation, position, and motion of the interventional tool in the current image 205 over time. The motion estimation module 216 estimates motion in the current image 205 caused by patient motion, such as respiratory motion or other motion. The motion estimation module 216 also estimates the motion of the interventional tool in the current image 205. The segmentation of the interventional tool by the segmentation module 212 can be used to estimate the motion relative to a static segmented anatomy of the patient. The tool model module 214 can update or correct the tool model based on the patient motion estimation and the tool motion estimation. The registration module 218 registers the segmented interventional tool with the prior image 207 of the patient anatomy. The rendering module 220 renders or generates an image depicting the movement of the interventional tool relative to the prior image 207 without the changes caused by patient respiratory motion. The image output generated by the rendering module 220 is as a tool guidance image 225. The tool guidance image 225 depicts to a physician or operator an image with the merged position of the segmented interventional tool and an image of the patient anatomy previously acquired with the image changes caused by patient respiratory motion removed.

[0026] The region of interest can be in close proximity to the patient anatomy for interventional tool insertion. As illustrative and non-limiting examples, the region of interest can include a tumor or biopsy target, an embolization target, etc. As illustrative and non-limiting examples, the patient anatomy for interventional tool insertion can include vasculature or bronchial tubes.

[0027] Thus, as further discussed herein, a method of creating a motion-adjusted image of a patient to guide an interventional medical procedure includes acquiring a plurality of fluoroscopic images of a region of interest of a patient, the plurality of images having an interventional tool positioned near the region of interest; processing the plurality of fluoroscopic images to segment the interventional tool; further processing the plurality of fluoroscopic images to determine motion of the patient; estimating motion relative to a static segmented anatomy of the patient based on the segmentation of the interventional tool; registering the segmented interventional tool with a previously acquired image of the patient anatomy, and displaying to a physician an image with the merged location of the segmented interventional tool and the previously acquired static image of the patient anatomy with the image changes caused by patient respiratory motion removed. In one example, the registration occurs only after the motion is determined and corrected, where the motion is determined and corrected only after the interventional tool is segmented. In this way, for a given processing power, the image processing can be performed more quickly, as the motion after segmentation can be measured and corrected more efficiently, even with additional operations such as registration.

[0028] Figure 3 An advanced flow diagram is shown that illustrates an exemplary method 300 for interventional tool tracking and guidance according to one embodiment. In particular, the method 300 involves automatically tracking an interventional tool in fluoroscopic images and displaying the location of the interventional tool for use in guiding an operator during fluoroscopic imaging. The method 300 is described with reference to the systems and components of Figure 1 and Figure 2 but it should be understood that the method can be implemented with other systems and components without departing from the scope of the present disclosure. The method 300 can be implemented as executable instructions in a non-transitory memory such as the memory 182 and can be executed by a processor of an x-ray imaging system such as one or more of the processors 181, the image processor 192, and the clinical tool tracker 194 of the x-ray imaging system 100.

[0029] The method 300 begins at 305. At 305, the method 300 initiates a fluoroscopy of a patient. For example, during the fluoroscopy, the method 300 controls the x-ray source 104 and the x-ray detector 108, e.g., via the x-ray controller 187 and the x-ray data acquisition unit 191, to acquire a series of fluoroscopic images of the subject or patient 112. During the fluoroscopy, a clinical or interventional tool 130 is inserted or applied to an anatomical structure of interest or region of interest, such as the subject 110, which can include a medical needle, a guidewire, a catheter, a biopsy instrument, a pedicle screw, a probe, or another surgical tool or instrument. Fluoroscopic imaging of the interventional tool 130 can thus navigate under fluoroscopic guidance in which the anatomical structure of interest is not visible, e.g., because the anatomical structure of interest is within the patient. Thus, to initiate the fluoroscopy of the patient, the method 300 can initiate control of the x-ray source 104 and the x-ray detector 108. In some examples, initiating the fluoroscopy of the patient can include receiving one or more of an indication of the patient, a desired setting for the fluoroscopic imaging, an indication of the anatomical structure of interest, a selection of one or more prior images of the anatomical structure of interest, etc. Further, initiating the fluoroscopy of the patient can include acquiring at least one fluoroscopic image of the patient.

[0030] At 310, the method 300 retrieves one or more prior images of the patient anatomy. For example, the method 300 can retrieve one or more prior images of the anatomical structure of interest of the patient from the non-transitory memory 182, e.g., based on an indication of the patient and / or the anatomical structure of interest received during initialization of the fluoroscopy. The one or more prior images can include a static rendering of the anatomical structure of interest, and thus can include a projection rendering of a three-dimensional object (e.g., the anatomical structure of interest), a roadmapping acquisition, or another image of the anatomical structure of interest acquired by a suitable imaging modality (e.g., fluoroscopic imaging, non- perspective x-ray imaging, CT imaging, etc.). As discussed further herein, an image of the interventional tool can be superimposed on the one or more prior images to provide guidance to an operator of the imaging system.

[0031] At 315, the method 300 acquires a fluoroscopic image of the region of interest of the patient with the interventional tool positioned proximate the region of interest. For example, the method 300 controls the x-ray source 104 and the x-ray detector 108, e.g., via the x-ray controller 187 and the x-ray data acquisition unit 191, to acquire at least one fluoroscopic image of the subject or patient 112. At 320, the method 300 displays the fluoroscopic image, e.g., via a display device such as the display device 195.

[0032] At 325, the method 300 segments the interventional tool in the fluoroscopic image. For example, the method 300 processes the fluoroscopic image acquired at 315, e.g., with the image processor 192 and / or the clinical tool tracker 194, to segment the interventional tool in the fluoroscopic image. The method 300 can use any suitable segmentation method, such as thresholding, clustering, motion-based segmentation, histogram-based thresholding, edge detection, machine learning-based segmentation, etc., or combinations thereof. The method 300 thus obtains at least one segmented image of the interventional tool in the fluoroscopic image. The at least one segmented image can include a selection of pixels in the fluoroscopic image that correspond to the interventional tool, or alternatively can include a different image that includes pixels in the fluoroscopic image that correspond to the interventional tool.

[0033] At 330, the method 300 determines motion of the patient in the fluoroscopic image. As an illustrative and non-limiting example, the method 300 can determine motion of the patient in the fluoroscopic image by measuring optical flow between two or more of the fluoroscopic images, where the optical flow indicates relative motion of objects (such as tissue or other anatomical structures) depicted in the fluoroscopic images and the interventional tool in the fluoroscopic images relative to one or more previous fluoroscopic images. For example, the method 300 can determine motion of the patient in the fluoroscopic image relative to an initial fluoroscopic image acquired during initialization of the fluoroscopy at 305 during a first iteration, and can determine motion of the patient in the fluoroscopic image relative to a fluoroscopic image acquired at 315 during a previous iteration. To determine motion of the patient in the fluoroscopic image and thereby distinguish motion of the patient from motion of the interventional tool, the method 300 can decompose or filter the determined motion to identify low-frequency motion, which can correspond to respiratory motion (e.g., motion of tissue or other anatomical structures in the patient caused by respiration of the patient).

[0034] At 335, the method 300 estimates the position of the interventional tool based on the segment of the tool in the fluoroscopic image. At 340, the method 300 corrects the estimated position of the interventional tool and the segment of the interventional tool based on the motion of the patient. For example, the method 300 can correct the estimated position of the interventional tool by removing the motion of the patient, in particular the respiratory motion identified at 330, thereby correcting the segment of the interventional tool in the fluoroscopic image. At 345, the method 300 registers the corrected segment of the tool with a prior image of the patient anatomy. In particular, the method 300 registers the corrected segment of the interventional tool to the anatomical structure of interest in the prior image in which the interventional tool is positioned. At 350, the method 300 generates one or more images with the corrected segment of the tool superimposed on the prior image from the corrected position of the interventional tool. By correcting the segment of the interventional tool and registering the corrected segment of the interventional tool to the anatomical structure of interest, the images generated with the corrected segment superimposed on the prior image from the corrected position of the interventional tool include a stable view of the interventional tool relative to the anatomical structure of interest. At 355, the method 300 displays the one or more images of the segment of the tool superimposed on the prior image, for example via the display device 195. In some examples, the one or more images can be positioned adjacent to the corresponding fluoroscopic image in a graphical user interface displayed via the display device 195, such that a user can view the fluoroscopic image of the interventional tool and the stable view simultaneously. In this way, as the user moves the interventional tool relative to the anatomical structure of interest, the user can discern the position of the interventional tool relative to the anatomical structure of interest while also discerning the effect of patient motion depicted in the fluoroscopic image on the position of the interventional tool and the anatomical structure of interest, thereby improving navigation of the interventional tool relative to the anatomical structure of interest.

[0035] At 360, the method 300 determines whether a command to end the fluoroscopy is received. For example, a command to end the fluoroscopy can be received via the user interface 183. If a command to end the fluoroscopy is received (“yes”), the method 300 proceeds to 365, where the method 300 ends the fluoroscopy. The method 300 ends the fluoroscopy by stopping the acquisition of fluoroscopic images by the control of the x-ray source 104 and the x-ray detector 108. The method 300 then returns.

[0036] However, with reference again to 360, if no command to end the fluoroscopy is received ("No"), the method 300 proceeds to 370. At 370, the method 300 acquires a fluoroscopic image of the region of interest of the patient. At 375, the method 300 displays the fluoroscopic image via the display device 195. At 380, the method 300 evaluates the fluoroscopic image. The method 300 can evaluate the fluoroscopic image acquired at 370 relative to the fluoroscopic image acquired at 315, for example, or relative to a fluoroscopic image previously acquired at 370 during a previous iteration, to determine motion of the interventional tool. The method 300 can determine motion of the interventional tool from any apparent motion of the interventional tool between the fluoroscopic images, or can decompose the motion between the fluoroscopic images to determine non-respiratory motion corresponding to motion of the interventional tool.

[0037] At 385, the method 300 determines whether the fluoroscopic image acquired at 370 indicates motion of the interventional tool. If there is no motion of the interventional tool ("No"), the method 300 continues to 390. At 390, the method 300 continues to display the previously generated image of the segment of the tool overlaid on the previous image. The method 300 then continues to 360, where the method 300 determines whether there is a command to end the fluoroscopy. Thus, when the interventional tool is not moving, the method 300 continues to acquire and display fluoroscopic images without updating the stable image of the interventional tool.

[0038] With reference again to 385, if the fluoroscopic image acquired at 370 indicates motion of the interventional tool ("Yes"), the method 300 returns to 325 to segment the interventional tool in the fluoroscopic image acquired at 370. Thus, the method 300 continues to determine motion of the patient in the fluoroscopic image at 330, estimate the position of the interventional tool at 335, and so on, to generate an updated image with a segment of the interventional tool overlaid on a previous image, with corrections applied to account for patient motion, such as respiratory motion.

[0039] As an illustrative example, Figure 4 A block diagram is shown that illustrates an example graphical user interface 400 including an image display region 402 that displays a fluoroscopic image 410 and an image of an interventional tool guide 420, according to one embodiment. As discussed above with respect to Figure 3 A stable image of the interventional tool can be generated during a fluoroscopy to provide guidance to a user navigating the interventional tool relative to an anatomical structure of interest within a patient, as discussed above with respect to

[0040] As another illustrative and non-limiting example of how to stabilize imaging of an interventional tool during an examination to improve guidance and navigation, Figure 5 An advanced flowchart is shown that illustrates an example method 500 for stabilizing tool rendering according to one embodiment. In particular, the method 500 involves automatically tracking an interventional tool in a fluoroscopic image and displaying a stabilized position of the interventional tool relative to an anatomical structure of interest for guiding an operator during fluoroscopic imaging. Reference is made to Figure 1 and Figure 2 The system and components of

[0041] The method 500 begins at 505. At 505, the method 500 initiates a fluoroscopic examination of a patient. For example, as described above, during a fluoroscopic examination, the method 500 controls the x-ray source 104 and the x-ray detector 108, e.g., via the x-ray controller 187 and the x-ray data acquisition unit 191, to acquire a series of fluoroscopic images of a subject or patient 112. During the fluoroscopic examination, a clinical tool or interventional tool 130 is inserted or applied to an anatomical structure of interest or region of interest, such as the subject 110, which can include a medical needle, a guidewire, a catheter, a biopsy instrument, a pedicle screw, a probe, or another surgical tool or instrument. Fluoroscopic imaging of the interventional tool 130 is thus able to navigate under fluoroscopic guidance in which the anatomical structure of interest is not visible, e.g., because the anatomical structure of interest is within the patient. Thus, to initiate the fluoroscopic examination of the patient, the method 500 can initiate control of the x-ray source 104 and the x-ray detector 108. In some examples, initiating the fluoroscopic examination of the patient can include receiving one or more of an indication of the patient, a desired setting for the fluoroscopic imaging, an indication of the anatomical structure of interest, a selection of one or more prior images of the anatomical structure of interest, etc. Further, initiating the fluoroscopic examination of the patient can include acquiring at least one fluoroscopic image of the patient.

[0042] At 510, the method 500 acquires an image of the patient’s anatomical structure of interest. For example, the method 500 controls the x-ray source 104 and the x-ray detector 108, e.g., via the x-ray controller 187 and the x-ray data acquisition unit 191, to acquire an image, such as a fluoroscopic image of the anatomical structure of interest. As an illustrative and non-limiting example, the anatomical structure of interest can include a tumor, a blood vessel, or another anatomical structure that can be a target of the interventional tool.

[0043] At 515, the method 500 registers a model of the anatomical structure of interest to the anatomical structure of interest in the image. The model can include a projection of a three-dimensional model of the anatomical structure of interest in the plane of the image acquired at 510, as an illustrative and non-limiting example, where the model can be generated or adapted to the anatomical structure of interest in the image acquired at 510. The model can alternatively include a projection of a three-dimensional segmented image volume of the anatomical structure of interest acquired, for example, via a three-dimensional imaging modality such as CT. As yet another example, the model can include another static rendering of the anatomical structure of interest obtained from a road map acquisition or another type of prior image of the anatomical structure of interest acquired prior to the fluoroscopy. The method 500 registers the model of the anatomical structure of interest to the corresponding anatomical structure of interest in the image such that the position and orientation of the model of the anatomical structure of interest corresponds to the position and orientation of the anatomical structure of interest in the image.

[0044] At 520, the method 500 segments the interventional tool in the image acquired at 510. For example, the method 500 processes the fluoroscopic image acquired at 315, for example, with the image processor 192 and / or the clinical tool tracker 194, to segment the interventional tool in the fluoroscopic image. The method 500 can use any suitable segmentation method, such as thresholding, clustering, motion-based segmentation, histogram-based thresholding, edge detection, machine learning-based segmentation, or the like or combinations thereof. The method 500 thus obtains at least one segmented image of the interventional tool in the fluoroscopic image. The at least one segmented image can include a selection of pixels in the fluoroscopic image corresponding to the interventional tool, or alternatively can include a different image containing pixels in the fluoroscopic image corresponding to the interventional tool.

[0045] At 525, the method 500 registers the segmented tool to the registered model of the anatomical structure of interest. The method 500 can register the segmented tool to the registered model of the anatomical structure of interest, for example, by imposing a boundary on the segmented tool such that the segmented tool is positioned within the anatomical structure of interest, for example, if the anatomical structure of interest is a blood vessel through which the interventional tool is guided. Similarly, for other anatomical structures of interest, the position of the interventional tool can be registered to the position of the registered model such that the relative position does not move when respiratory motion or other patient motion is considered, as discussed further herein.

[0046] At 530, the method 500 displays an image, e.g., via the display device 195, with the registered model of the anatomical structure of interest and the registered segmentation tool overlaid thereon. The image can include, e.g., the fluoroscopic image acquired at 510, or can include a prior image of the anatomical structure of interest acquired via the x-ray imaging system 100 or another imaging modality. Alternatively, in some examples, the image can include only the registered model of the anatomical structure of interest and the registered segmentation tool overlaid thereon, such that the image does not include other anatomical structures or tissue.

[0047] At 535, the method 500 acquires a second image of the anatomical structure of interest. For example, the method 500 controls the x-ray source 104 and the x-ray detector 108, e.g., via the x-ray controller 187 and the x-ray data acquisition unit 191, to acquire the second image. At 540, the method 500 decomposes the motion between the image and the second image to determine the respiratory motion between the images. For example, as an illustrative and non-limiting example, the method 500 can determine the motion of the patient between the image and the second image by measuring the optical flow between the two images, where the optical flow indicates the relative motion of objects (such as tissue or other anatomical structures) and interventional tools depicted in the second image relative to the image. The method 500 can then decompose the determined motion and remove the determined low frequency components that correspond to patient motion, such as respiratory motion or cardiac motion. Continuing at 545, the method 500 corrects the second image based on the patient motion, such as respiratory motion or cardiac motion. In one example, the correction removes all or a portion of the measured motion. In some examples, the second image can be corrected by an image registration technique, such as an optical flow algorithm. In other examples, the second image can be corrected by registering the segmented tool itself, decomposing the motion into respiratory, cardiac, and / or navigation components, and removing the respiratory and cardiac components of the motion to stabilize the objects. The method 500 then updates the registration of the model of the anatomical structure of interest to the anatomical structure of interest in the corrected second image at 550.

[0048] At 555, the method 500 segments the interventional tool in the corrected second image using a suitable segmentation method as described above. Then, at 560, the method 500 updates the registration of the segmented tool to the updated registered model of the anatomical structure of interest. At 565, the method 500 displays the second image with the updated registered model of the anatomical structure of interest and the updated registered segmented tool overlaid thereon, where the second image can include the second image acquired at 535 or another static image as described above.

[0049] At 570, the method 500 determines whether to end the fluoroscopy. If the examination is not ended ("No"), the method 500 returns to 535 to acquire additional images of the anatomical structure of interest, such as a third image. At 540, the method 500 then decomposes the motion between the third image and the second image to determine the respiratory motion between the third image and the second image. Thus, the method 500 continues to correct for respiratory motion between images and update the registration of the model and the interventional tool accordingly during the fluoroscopy until the method 500 determines to end the examination at 570. Once the method 500 determines to end the examination at 570 ("Yes"), the method 500 continues to 575, where the method 500 ends the fluoroscopy. The method 500 then returns.

[0050] As an illustrative example of no stable interventional tool guidance, Figure 6 An image set 600 is shown that illustrates unstable positions of an interventional tool due to patient motion in fluoroscopic imaging. In a first image 605 of the image set 600, a model 615 of an anatomical structure of interest (e.g., a blood vessel) is displayed on a fluoroscopic image, and an interventional tool 620 (e.g., a guidewire) is depicted as being within the anatomical structure of interest shown via the model 615. Displaying the model 615 enhances the visualization of the anatomical structure of interest, which can not be visible through fluoroscopic imaging, which can be calibrated to image the interventional tool itself. Thus, superimposing the model 615 onto the image 605 enables a user to visualize the position of the interventional tool 620 relative to the anatomical structure of interest depicted by the model 615. However, due to respiratory motion, the position of tissue 610 depicted in the image 605 moves to a different position in a subsequent image 625. The respiratory motion further causes the anatomical structure of interest to move, and thus the position of the interventional tool 620 in the subsequent image 625 also moves. If the relative position of the interventional tool 620 and the model 615 is not stable, the position of the interventional tool 620 moves with patient motion in the acquired fluoroscopic images. By segmenting and registering the interventional tool 620 to the model 615 as described above with respect to Figure 3 and Figure 5 the unstable view of the interventional tool relative to the anatomical structure of interest can be provided to the user.

[0051] For example, Figure 7A block diagram is shown illustrating an exemplary graphical user interface 700 for displaying a stable tool presentation according to one embodiment. For example, the graphical user interface 700 may be displayed via a display device 195, and this graphical user interface depicts an image 702 of a model 715 of the anatomical structure of interest, overlaid with a segmented image 720 of the interventional tool. Optionally, image 702 may include a fluorescence microscopic image or still image depicting tissue 710 or other anatomical structures, with the model 715 and the segmented interventional tool 720 overlaid thereon. Alternatively, as described above regarding... Figure 4 The fluorescent microscopy image can be drawn in the graphical user interface 700 at a position adjacent to the image 702, allowing the user to view the acquired fluorescent microscopy image and update the image 702 according to the movement of the interventional tool to visualize the movement of the interventional tool relative to the anatomical structure of interest depicted by the model 715.

[0052] The technical advantages of this disclosure include displaying a motion-stabilized clinical tool inserted into a patient's anatomical structure of interest. Another technical advantage of this disclosure is motion correction of images to remove respiratory and / or cardiac motion. Yet another technical advantage of this disclosure is the acquisition of fluorescein images, the motion correction of such fluorescein images for the patient, and the display of such corrected fluorescein images using a stable view of the interventional tool depicted in the corrected fluorescein images.

[0053] In one embodiment, a method includes: acquiring multiple fluorescence microscopic images of a patient’s anatomy of interest, the multiple fluorescence microscopic images depicting an interventional tool positioned relative to the anatomy of interest; segmenting the interventional tool in the multiple fluorescence microscopic images; measuring the patient’s motion in the multiple fluorescence microscopic images; correcting the multiple fluorescence microscopic images to remove the patient’s motion; registering the segmented interventional tool to the anatomy of interest in the corrected multiple fluorescence microscopic images; and displaying an image of the segmented interventional tool registered to the anatomy of interest.

[0054] In a first example of the method, measuring the motion of the patient in the plurality of fluoroscopic images includes measuring low frequency motion between a first fluoroscopic image and a second fluoroscopic image of the plurality of fluoroscopic images. In a second example of the method, which optionally includes the first example, correcting the plurality of fluoroscopic images to remove the motion of the patient includes subtracting the low frequency motion from the second fluoroscopic image. In a third example of the method, which optionally includes one or more of the first and second examples, the method further includes registering a model of the anatomical structure of interest to the anatomical structure of interest in the plurality of fluoroscopic images. In a fourth example of the method, which optionally includes one or more of the first through third examples, the method further includes retrieving a prior image of the anatomical structure of interest, and determining the model of the anatomical structure of interest from the prior image of the anatomical structure of interest. In a fifth example of the method, which optionally includes one or more of the first through fourth examples, the prior image includes a three-dimensional image volume, and wherein the model of the anatomical structure of interest includes a projection of a three-dimensional model of the anatomical structure of interest in a plane of the plurality of fluoroscopic images, the three-dimensional model of the anatomical structure of interest being determined from the three-dimensional image volume. In a sixth example of the method, which optionally includes one or more of the first through fifth examples, registering the segmented interventional tool to the anatomical structure of interest in the corrected plurality of fluoroscopic images includes registering the segmented interventional tool to the model of the anatomical structure of interest. In a seventh example of the method, which optionally includes one or more of the first through sixth examples, displaying the image with the segmented interventional tool registered to the anatomical structure of interest includes displaying a motion stabilized view of the segmented interventional tool superimposed on the model of the anatomical structure of interest. In an eighth example of the method, which optionally includes one or more of the first through seventh examples, the anatomical structure of interest includes a tumor or a blood vessel, and wherein the motion of the patient includes one or more of respiratory motion and cardiac motion.

[0055] In another embodiment, a method includes acquiring a series of fluoroscopic images of an anatomical structure of interest of a patient, the series of fluoroscopic images depicting an interventional tool positioned relative to the anatomical structure of interest, and while acquiring the series of fluoroscopic images: segmenting the interventional tool in the series of fluoroscopic images; determining a motion between a first fluoroscopic image and a second fluoroscopic image of the series of fluoroscopic images; decomposing the motion into a motion of the patient and a motion of the interventional tool; correcting the second fluoroscopic image to remove the motion of the patient; registering the segmented interventional tool to the anatomical structure of interest in the corrected second fluoroscopic image; and displaying an image with the registered segmented interventional tool superimposed on the anatomical structure of interest.

[0056] In a first example of the method, the method further includes determining a model of the anatomical structure of interest and registering the model of the anatomical structure of interest to the anatomical structure of interest in the corrected second fluoroscopic image. In a second example of the method optionally including the first example, registering the segmented interventional tool to the anatomical structure of interest in the corrected second fluoroscopic image includes registering the segmented interventional tool to the registered model of the anatomical structure of interest. In a third example of the method optionally including one or more of the first and second examples, displaying the image with the registered segmented interventional tool superimposed on the anatomical structure of interest includes displaying the image with the registered segmented interventional tool superimposed on the registered model of the anatomical structure of interest. In a fourth example of the method optionally including one or more of the first through third examples, the method further includes updating the image with the registered segmented interventional tool superimposed on the anatomical structure of interest in accordance with motion of the interventional tool in fluoroscopic images acquired after the second fluoroscopic image, and displaying the updated image.

[0057] In yet another embodiment, a system includes an x-ray source configured to generate x-rays, an x-ray detector configured to detect x-rays, a display device, and a controller communicatively coupled to the x-ray source, the x-ray detector, and the display device, the controller configured to: control the x-ray source and the x-ray detector to acquire a plurality of fluoroscopic images of an anatomical structure of interest of a patient, the plurality of fluoroscopic images depicting an interventional tool positioned relative to the anatomical structure of interest; segment the interventional tool in the plurality of fluoroscopic images; measure motion of the patient in the plurality of fluoroscopic images; correct the plurality of fluoroscopic images to remove the motion of the patient; register the segmented interventional tool to the anatomical structure of interest in the corrected plurality of fluoroscopic images; and display, via the display device, an image with the segmented interventional tool registered to the anatomical structure of interest.

[0058] In a first example of the system, the controller is further configured to measure the motion of the patient in the plurality of fluoroscopy images by measuring low frequency motion between a first fluoroscopy image and a second fluoroscopy image of the plurality of fluoroscopy images. In a second example of the system, optionally including the first example, the controller is configured to correct the plurality of fluoroscopy images to remove the motion of the patient by subtracting the low frequency motion from the second fluoroscopy image. In a third example of the system, optionally including one or more of the first example and the second example, the controller is further configured to register a model of the anatomical structure of interest to the anatomical structure of interest in the plurality of fluoroscopy images. In a fourth example of the system, optionally including one or more of the first example through the third example, the controller is further configured to register the segmented interventional tool to the anatomical structure of interest in the corrected plurality of fluoroscopy images by registering the segmented interventional tool to the model of the anatomical structure of interest. In a fifth example of the system, optionally including one or more of the first example through the fourth example, the controller is further configured to display an image with the segmented interventional tool registered to the anatomical structure of interest by displaying a motion stabilized view of the segmented interventional tool overlaid on the model of the anatomical structure of interest.

[0059] As used herein, an element or step recited in the singular and preceded with the word "a" or "an" should be understood as not excluding plural of said elements or steps, unless explicitly stated that such exclusion applies. Also, aspects of the disclosure can include "one embodiment," or "an embodiment," or "some embodiments," and the phrases "comprises," "has," "has," and the like can refer to a singular or plurality unless explicitly stated otherwise. Additionally, the terms "first," "second," and "third," etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0060] This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and can include other examples that occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.

Claims

1. A method of clinical tool tracking and visualization, the method comprising: acquiring a plurality of fluoroscopic images of a patient's anatomy of interest, the plurality of fluoroscopic images depicting an interventional tool positioned relative to the anatomy of interest; segmenting the interventional tool in the plurality of fluoroscopic images; measuring motion of the patient in the plurality of fluoroscopic images; correcting the plurality of fluoroscopic images based on the motion of the patient; registering the segmented interventional tool to the anatomy of interest in the corrected plurality of fluoroscopic images; displaying an image with the segmented interventional tool registered to the anatomy of interest; based on a further fluoroscopic image of the patient's anatomy of interest acquired, determining non-respiratory motion of the interventional tool in the further fluoroscopic image, wherein the further fluoroscopic image depicts the interventional tool positioned relative to the anatomy of interest; and based on the absence of non-respiratory motion of the interventional tool in the further fluoroscopic image, continuing to display the image with the segmented interventional tool registered to the anatomy of interest.

2. The method of claim 1, wherein measuring the motion of the patient in the plurality of fluoroscopic images comprises measuring low frequency motion between a first fluoroscopic image and a second fluoroscopic image in the plurality of fluoroscopic images.

3. The method of claim 2, wherein correcting the plurality of fluoroscopic images to remove the motion of the patient comprises subtracting the low frequency motion from the second fluoroscopic image.

4. The method of claim 1, further comprising registering a model of the anatomy of interest to the anatomy of interest in the plurality of fluoroscopic images.

5. The method of claim 4, further comprising retrieving a prior image of the anatomy of interest, and determining the model of the anatomy of interest from the prior image of the anatomy of interest.

6. The method of claim 5, wherein the prior image comprises a three-dimensional image volume, and wherein the model of the anatomy of interest comprises a projection of a three-dimensional model of the anatomy of interest in a plane of the plurality of fluoroscopic images, the three-dimensional model of the anatomy of interest determined from the three-dimensional image volume.

7. The method of claim 4, wherein registering the segmented interventional tool to the anatomy of interest in the corrected plurality of fluoroscopic images comprises registering the segmented interventional tool to the model of the anatomy of interest.

8. The method of claim 7, wherein displaying the image with the segmented interventional tool registered to the anatomy of interest comprises displaying a motion stabilized view of the segmented interventional tool superimposed on the model of the anatomy of interest.

9. The method of claim 1, wherein the anatomy of interest comprises a tumor or a blood vessel, and wherein the motion of the patient comprises one or more of respiratory motion and cardiac motion. ​ 10. A method of clinical tool tracking and visualization, the method comprising: acquiring a series of fluoroscopic images of a patient's anatomy of interest, the series of fluoroscopic images depicting an interventional tool positioned relative to the anatomy of interest; and while acquiring the series of fluoroscopic images: segmenting the interventional tool in the series of fluoroscopic images; determining motion between a first fluoroscopic image and a second fluoroscopic image in the series of fluoroscopic images; decomposing the motion into motion of the patient and non-respiratory motion of the interventional tool; correcting the second fluoroscopic image to remove the motion of the patient; registering the segmented interventional tool to the anatomy of interest in the corrected second fluoroscopic image; and displaying an image with the registered segmented interventional tool superimposed on the anatomy of interest, based on a further fluoroscopic image of the patient's anatomy of interest taken, determining non-respiratory motion of the interventional tool in the further fluoroscopic image, wherein the further fluoroscopic image depicts the interventional tool positioned relative to the anatomy of interest; and based on an absence of non-respiratory motion of the interventional tool in the further fluoroscopic image, continuing to display the image with the registered segmented interventional tool superimposed on the anatomy of interest.

11. The method of claim 10, further comprising determining a model of the anatomy of interest and registering the model of the anatomy of interest to the anatomy of interest in the corrected second fluoroscopic image.

12. The method of claim 11, wherein registering the segmented interventional tool to the anatomy of interest in the corrected second fluoroscopic image comprises registering the segmented interventional tool to the registered model of the anatomy of interest.

13. The method of claim 12, wherein displaying the image with the registered segmented interventional tool superimposed on the anatomy of interest comprises displaying the image with the registered segmented interventional tool superimposed on the registered model of the anatomy of interest.

14. The method of claim 10, further comprising updating the image with the registered segmented interventional tool superimposed on the anatomy of interest according to non-respiratory motion of the interventional tool in a fluoroscopic image acquired after the second fluoroscopic image, and displaying the updated image.

15. A system of clinical tool tracking and visualization, the system comprising: an x-ray source configured to generate x-rays; an x-ray detector configured to detect the x-rays; a display device; and a controller communicatively coupled to the x-ray source, the x-ray detector, and the display device, the controller configured to perform the method of any of claims 1-9. ​ ​

Images