Systems and methods for motion-modulated device guidance using vascular roadmaps

Abstract

A system and method for creating motion-adjusted or motion-compensated images of a patient to guide an interventional medical procedure is provided. The method includes displaying a static roadmap and a plurality of dynamic images to show an interventional medical device aligned on the static roadmap using a motion transform. The alignment of the interventional medical device on the static roadmap is based on a user selection of one of: user-selected motion compensation of the interventional medical device relative to the static roadmap to produce a plurality of images that do not show patient motion, or motion adjustment of the static roadmap relative to the interventional medical device to produce a plurality of images that show patient motion.

Description

[0001] Cross-references to related applications

[0002] This application is based on, claims the benefit of, and claims priority to, U.S. Patent Application No. 16 / 455,331, filed on June 27, 2019, which is hereby incorporated by reference in its entirety.

[0003] A note about federally funded research

[0004] This invention was made with government support under EB024553 awarded by the National Institutes of Health. The government has certain rights in this invention. Background Art

[0005] The present invention relates to interventional radiology systems and methods. More particularly, the present invention relates to systems and methods for guiding the deployment of interventional devices in the presence of patient motion.

[0006] Interventional radiology is a key component of modern healthcare, reducing risk and speeding up recovery time for patients. However, these procedures can be costly and, more importantly, require complete coordination between many complex systems in order to be successful. Typically, the interventional radiologists who schedule these procedures must overcome many different issues. For example, during the procedure, the radiologist must not only be able to maneuver themselves effectively, but they must also be able to maneuver their instruments within relatively confined spaces. This can be particularly difficult because typical imaging systems only allow access to tight clearances of the patient (e.g., bores), and unfortunately, the instruments required during the procedure are often cumbersome to navigate within these tight clearances.

[0007] Difficulties also arise from deviations from the ideal resulting from the patient, which may include artifacts from patient movement (e.g., breathing), anatomical structures of varying sizes and shapes, and so forth. Further, other difficulties arise from accurately acquiring the images needed to complete the procedure within the time allowed. For example, some anatomical structures, instruments, and so forth, may not appear sharp or bright enough in the image. Other examples arise from imaging systems in which images must be acquired, processed, and displayed within a specific amount of time. If these problems occur simultaneously, such as acquiring images of anatomical areas that are difficult to view while the patient is moving, they can become particularly difficult. Therefore, the field of interventional radiology relies heavily on the skill and knowledge of the interventional radiologist to overcome these problems and make decisions to effectively complete the procedure.

[0008] An example of a typical and widely used interventional radiology procedure is fluoroscopic image guidance for minimally invasive surgery. This procedure, especially transarterial embolization, currently plays a key role in the treatment of patients with primary and metastatic liver tumors. In liver embolization, a catheter is guided to a specific branch of a visceral artery or hepatic artery via fluoroscopy, where particles or microspheres are delivered directly to the blood supply artery of the tumor (e.g., to prevent further tumor growth, reduce the blood supply of the tumor, provide local treatment, etc.). Due to the targeted area, accurate and rapid placement of the catheter is essential to reduce surgical time and achieve good tumor results. In order to effectively place instruments (such as catheters, guidewires, etc.), images of the area and instruments need to be displayed. However, some anatomical structures, including vascular systems, are almost impossible to distinguish from fluoroscopic images. Therefore, usually, before placing or operating an instrument, a reference to the vascular system is created by injecting contrast agent into the vascular system area, obtaining a perspective image of the contrast-enhanced vascular system, and displaying a static 2D digital subtraction angiography ("DSA") on the contrast-enhanced vascular system.

[0009] DSA vascular images can be overlaid or displayed side-by-side with real-time fluoroscopic images while the instrument is manipulated until the instrument shape conforms to the path of the desired vascular branch. Unfortunately, this approach can be difficult and time consuming because the instruments often need to be repositioned. More importantly, this approach fails to take into account the properties of this particular anatomical region, specifically, the true shape of the vasculature continuously moves and changes shape due to patient movement (e.g., respiratory motion, cardiac motion, etc.). Therefore, even when the patient is instructed to hold his breath during the acquisition of the vascular images to be digitally subtracted, subtraction artifacts are inherently introduced. Further, inaccurate image guidance can result from the failure to account for patient movement because the DSA images only display a static image of the vasculature at a specific point in time.

[0010] Accordingly, improved systems and methods for interventional device guidance are desired. Summary of the invention

[0011] The present invention provides systems and methods that overcome the above-mentioned shortcomings by providing a method for creating motion-adjusted or motion-compensated images of a patient to guide an interventional medical procedure. The method includes displaying a static roadmap and a plurality of dynamic images to show an interventional medical device aligned on the static roadmap using motion compensation. The alignment of the interventional medical device on the static roadmap is based on a user selection of one of the following: motion compensation of the interventional medical device relative to the static roadmap to produce a plurality of images that do not show patient motion, or motion adjustment of the static roadmap relative to the interventional medical device to produce a plurality of images that show patient motion.

[0012] According to another non-limiting example of the present disclosure, a method for creating a motion-adjusted image of a patient is provided to guide an interventional medical procedure. The method includes acquiring a first plurality of images of a patient having a non-contrast enhanced vasculature, acquiring a second plurality of images of a patient having a contrast enhanced vasculature, and generating a static roadmap of the patient's vasculature using the first plurality of images and the second plurality of images. The method also includes generating a motion model of the patient using the first plurality of images and the second plurality of images, acquiring a third plurality of images of the patient using an interventional medical device deployed in the patient, and generating motion tracking data of one of the patient or the interventional medical device using the third plurality of images. The method also includes generating a motion transform using the motion tracking data and the motion model, and displaying the static roadmap and the third plurality of images to display the interventional medical device aligned on the static roadmap using the motion transform. The alignment of the interventional medical device on the static roadmap is based on a user-selected one of the following: motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion, or motion adjustment of the static roadmap relative to the interventional medical device to produce multiple images that show patient motion.

[0013] According to another non-limiting example of the present disclosure, a fluoroscopic imaging system is provided, including an X-ray source assembly coupled at one end, and an X-ray detector array assembly coupled at the other end, and a computer system. The computer system is configured to control the X-ray source assembly and the X-ray detector array assembly to acquire a first plurality of images of a patient having a non-contrast enhanced vasculature, and to control the X-ray source assembly and the X-ray detector array assembly to acquire a second plurality of images of a patient having a contrast enhanced vasculature. The computer system is further programmed to generate a static roadmap of the patient's vasculature using the first plurality of images and the second plurality of images, to generate a motion model of the patient using the first plurality of images and the second plurality of images, and to control the X-ray source assembly and the X-ray detector array assembly to acquire a third plurality of images of the patient using an interventional medical device deployed in the patient. The computer system is also configured to generate motion tracking data of one of the patient or the interventional medical device using the third plurality of images, to generate a motion transformation using the motion tracking data and the motion model, and to display the static roadmap and the third plurality of images to display the interventional medical device aligned on the static roadmap using the motion transformation. Alignment of the interventional medical device on the static roadmap is based on a user selection of one of: motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion, or adjustment of the static roadmap relative to motion of the interventional medical device to produce multiple images that show patient motion.

[0014] The foregoing and other aspects and advantages of the present invention will become apparent from the following description. In this specification, reference is made to the accompanying drawings forming a part thereof, in which are shown, as illustrations, preferred embodiments of the present invention. However, such embodiments do not necessarily represent the full scope of the present invention, and reference is made to the claims and this document for interpretation of the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an exemplary “C-arm” X-ray imaging system according to some non-limiting examples of the present disclosure.

[0016] Figure 2 is an example of a schematic diagram of a processing architecture according to some non-limiting examples of the present disclosure.

[0017] Figure 3 is a specific process of one non-limiting example of a system according to the present disclosure.

[0018] Figure 4 is a flow chart illustrating one non-limiting example of a process for creating a vasculature model in accordance with the present disclosure.

[0019] Figure 5 is an image of a vasculature using superimposed spline grids and image patches in generating a dynamic vasculature model according to the present disclosure.

[0020] Figure 6 is a graph of Euler numbers and corresponding binarized images used in generating a dynamic vasculature model according to the present disclosure.

[0021] Figure 7 is a flow chart illustrating one non-limiting example of generating a respiratory model within a dynamic vasculature model in accordance with the present disclosure.

[0022] Figure 8 is a fluoroscopic image with superimposed and colored edges for use in generating a dynamic vasculature model in accordance with the present disclosure.

[0023] Fig. 9 is a flow chart illustrating one non-limiting example of generating a real-time tracking system within a dynamic vasculature model in accordance with the present disclosure.

[0024] Fig.10 is a set of correlation graphs showing quantitative evaluation of a dynamic vascular model according to the present disclosure.

[0025] Fig.11 is a set of related images showing a comparison between a ground truth image and an estimated vasculature image generated from a dynamic vasculature model.

[0026] Fig.12is a schematic diagram of the architecture of one non-limiting example of a neural network for automatically segmenting medical devices within fluoroscopic images according to the present disclosure.

[0027] Fig.13 Four examples are provided to explain typical DSA procedures, Fig.12 A set of images with visual representations of the results compared between the output of the neural network and the ground truth images.

[0028] Fig.14 is a block diagram illustrating an example of implementing a two-dimensional ("2D") fluoroscopy guidance system according to the present disclosure.

[0029] Fig.15 is another block diagram illustrating an example of a line of sight 2D fluoroscopy guidance system according to the present disclosure.

[0030] Fig.16 is an image showing simplified movement of the vasculature, for example limited to whole body motion caused by breathing.

[0031] Fig.17 is a set of two fluoroscopic images generated according to the present disclosure.

[0032] Fig.18 is a set of two additional fluoroscopic images generated according to the present disclosure.

[0033] Fig.19 is an example of a display showing a user selecting between motion compensation and motion adjustment.

[0034] Fig. 20 is a block diagram illustrating one non-limiting example of implementing a three-dimensional ("3D") fluoroscopy guidance system in accordance with the present disclosure.

[0035] Fig.21 is a diagram showing the relationship between the respiratory state and the relative motion of the vascular system in the Z dimension according to the present disclosure. DETAILED DESCRIPTION

[0036] Before explaining the non-limiting examples of the present invention in detail, it should be understood that the present invention is not limited to the construction details and component configurations set forth in the following specification or shown in the accompanying drawings in this application. The present invention can be implemented with other non-limiting examples and can be practiced or executed in various ways. In addition, it is to be understood that the use of the words and terms used herein is for the purpose of description and should not be considered as limiting. In addition, the "right", "left", "front", "back", "high", "low", "up", "down", "top", or "bottom" and their variants used herein are for the purpose of description and should not be considered as limiting. In this article, the use of "including", "comprising" or "having" and its variants is meant to cover the items listed later and their equivalents and additional items. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variants are widely used and cover direct and indirect installation, connection, support and coupling. In addition, "connect" and "couple" are not limited to physical or mechanical connections or couplings.

[0037] Unless otherwise specified or limited, phrases like "at least one of A, B, and C," "one or more of A, B, and C," etc. mean A, B, or C, or any combination of A, B, and / or C, including combinations with multiple or single instances of A, B, and / or C.

[0038] As used herein, the term "controller" includes the following devices: any device capable of executing a computer program, or any device including logic gates configured to perform the functions described. For example, this may include a processor, a microcontroller, a current programmable gate array, a programmable logic controller, etc.

[0039] As described above, current fluoroscopic guidance systems do not account for patient motion (e.g., motion due to breathing, heart movement, etc.). These issues include artifacts introduced from the constant movement of the vasculature during the DSA procedure, and therefore, the DSA image is only a snapshot of the vasculature at a specific moment in time. This can result in inaccurate guidance because the target area is static, even though the vasculature and instruments (e.g., due to moving vasculature) are continually moving. Additionally, static DSA images without motion compensation may require the interventional radiologist to reposition, remove, or reinsert instruments to complete the procedure. Increased procedure time increases the patient's radiation exposure and may require additional doses of contrast agent to effectively see the instrument.

[0040] Previous attempts have been made to account for and compensate for patient motion. For example, external sensors (e.g., electrocardiogram ("ECG") electrodes, ultrasonic sensors) have been used in individuals to select and display a specific fluoroscopic image from a series of previously captured images based on sensor data corresponding to a specific time point in the respiratory cycle. These sensor-based attempts rely heavily on these external sensors, which are often not effectively associated with a specific vasculature representation and can hinder the use of surgical images. Other previous imaging-based systems for motion compensation either require the use of external sensors within the field of view or make inaccurate assumptions about the application (e.g., assuming that surrounding soft tissue corresponds to the vasculature, when in fact, high contrast objects visible in fluoroscopic images move at a different speed or direction than the vasculature).

[0041] The present invention overcomes these challenges by providing systems and methods for displaying a static roadmap and multiple images showing an interventional medical device aligned on the static roadmap using a motion transform. The alignment of the interventional medical device on the static roadmap is based on a user selection of one of: (i) motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion, or (ii) adjustment of the static roadmap relative to motion of the interventional medical device to produce multiple images that show patient motion.

[0042] In one non-limiting example, the system and method can track respiratory motion by extracting curvilinear features in a fluoroscopic image sequence, rather than relying on an additional imaging modality or external sensor to track respiratory motion. Additionally, the method does not require the presence of specific structures in the field of view to determine respiratory state, and is therefore applicable to a wider variety of surgeries. Further, the method allows tracking of local deformation of the vasculature based on respiratory motion by utilizing contrast-enhanced image sequences under free breathing conditions.

[0043] In some non-limiting examples of the present disclosure, a motion compensation system for fluoroscopic device guidance overcomes the shortcomings associated with previous systems while providing significant advantages. These advantages include reducing surgical time, improving surgical accuracy, reducing the amount of contrast required, and reducing radiation exposure time (e.g., via faster surgery). In some non-limiting examples, the present disclosure includes techniques for estimating vascular system deformation in respiratory motion tracking and native fluoroscopic images. Motion tracking and estimation can simplify the workflow of minimally invasive surgery (such as liver embolism), thereby allowing for improved surgical accuracy and speed (e.g., reducing the repositioning of instruments). Specifically, the system and method can create a motion model using contrast-enhanced vascular system fluoroscopic images and extract curved background features from native (e.g., non-contrast) fluoroscopic image sequences. Importantly, these fluoroscopic images do not require the patient to hold his breath, so these fluoroscopic images can be obtained under free breathing conditions. The two models used to achieve motion adjustment establish a relationship between the respiratory state (e.g., inferred from the curved background features) and the vascular morphology under the same respiratory state. Therefore, after acquiring the vessel morphology as described above, and in a real-time imaging procedure, curve feature detection is applied to the real-time fluoroscopic image to determine the vessel mask to be displayed. This will create a dynamic motion-adjusted vessel mask that can be superimposed on the real-time fluoroscopic image including the interventional medical device.

[0044] Now go to Figure 1 , shows an example of an imaging system that can be used with the systems and methods of the present disclosure. In this non-limiting example, a so-called "C-arm" X-ray imaging system 100 is shown. However, this is just one example, and fixed-position, single-source, dual-plane, and other architectures can also be easily used with the systems and methods of the present disclosure.

[0045] exist Figure 1 In a non-limiting example, a C-arm X-ray imaging system 100 includes a gantry 102 having a C-arm, an X-ray source assembly 104 coupled to one end of the C-arm, and an X-ray detector array assembly 106 coupled to the other end of the C-arm. The gantry 102 enables the X-ray source assembly 104 and the detector array assembly 106 to be oriented at different positions and angles around a subject 108, such as a medical patient or an object under examination positioned on a table 110. When the subject 108 is a medical patient, this configuration enables a physician to approach the subject 108.

[0046] The X-ray source assembly 104 includes at least one X-ray source that projects an X-ray beam, which may be a fan beam or a cone beam of X-rays, to an X-ray detector array assembly 106 on the opposite side of the gantry 102. The X-ray detector array assembly 106 includes at least one X-ray detector that includes a plurality of X-ray detector elements. Examples of X-ray detectors that may be included in the X-ray detector array assembly 106 include flat panel detectors, such as so-called "small flat panel" detectors, where the detector array panel may be approximately centimeters in size. Such a detector panel allows a field of view of approximately twelve centimeters to be covered.

[0047] The X-ray detector elements in one or more X-ray detectors housed in the X-ray detector array assembly 106 together sense projected X-rays passing through the subject 108. Each X-ray detector element generates an electrical signal that can be representative of the intensity of the impinging X-ray beam, and therefore the attenuation of the X-ray beam as it passes through the subject 108. In some configurations, each X-ray detector element is capable of counting the number of X-ray photons that impinge on the detector. During a scan to acquire X-ray projection data, the gantry 102 and components mounted thereon rotate about the isocenter of the C-arm X-ray imaging system 100.

[0048] The gantry 102 includes a support base 112. A support arm 114 is rotatably secured to the support base 112 for rotation about a horizontal pivot 116. The pivot 116 is aligned with the centerline of the table 110, and the support arm 114 extends radially outward from the pivot 116 to support a C-arm drive assembly 118 on its outer end. The C-arm gantry 102 is slidably secured to the drive assembly 118 and is coupled to a drive motor (not shown) that slides the C-arm gantry 102 to rotate it about the C-axis, as indicated by arrow 120. The pivot 116 and the C-axis are orthogonal and intersect each other at the isocenter of the C-arm X-ray imaging system 100, the intersection being indicated by a black circle and located above the table 110.

[0049] The X-ray source assembly 104 and the X-ray detector array assembly 106 extend radially inward to the pivot 116 so that the central ray of the X-ray beam passes through the isocenter of the system. Therefore, during the acquisition of X-ray attenuation data from the subject 108 placed on the table 110, the central ray of the X-ray beam can be rotated about the pivot 116, the C-axis, or both about the system isocenter. During scanning, the X-ray source and the detector array are rotated about the system isocenter to acquire X-ray attenuation projection data from different angles. As an example, the detector array is capable of acquiring 30 projections or views per second.

[0050] The C-arm X-ray imaging system 100 also includes an operator workstation 122, which will typically include a display 124, one or more input devices 126 (such as a keyboard and mouse), and a computer processor 128. The computer processor 128 may include a commercially available programmable machine running a commercially available operating system. The operator workstation 122 provides an operator interface so that scan control parameters can be input into the C-arm X-ray imaging system 100. Typically, the operator workstation 122 communicates with a data storage server 130 and an image reconstruction system 132. For example, the operator workstation 122, the data storage server 130, and the image reconstruction system 132 can be connected via a communication system 134, which can include any suitable network connection, whether wired, wireless, or a combination of both. As an example, the communication system 134 can include a proprietary or dedicated network, as well as an open network, such as the Internet.

[0051] The operator workstation 122 is also in communication with a control system 136 that controls the operation of the C-arm X-ray imaging system 100. The control system 136 generally includes a C-axis controller 138, a pivot controller 140, an X-ray controller 142, a data acquisition system ("DAS") 144, and a table controller 146. The X-ray controller 142 provides power and timing signals to the X-ray source assembly 104, and the table controller 146 is operable to move the table 110 to different positions and orientations within the C-arm X-ray imaging system 100.

[0052] The rotation of the gantry 102 to which the X-ray source assembly 104 and the X-ray detector array assembly 106 are coupled is controlled by a C-axis controller 138 and a pivot controller 140, which respectively control the rotation of the gantry 102 about the C-axis and about the pivot axis 116. In response to motion commands from the operator workstation 122, the C-axis controller 138 and the pivot controller 140 provide power to motors within the C-arm X-ray imaging system 100, which respectively generate rotation about the C-axis and the pivot axis 116. For example, a program executed by the operator workstation 122 generates motion commands to the C-axis controller 138 and the pivot controller 140 to move the gantry 102, thereby moving the X-ray source assembly 104 and the X-ray detector array assembly 106, in a prescribed scanning path.

[0053] The DAS 144 samples data from one or more X-ray detectors in the X-ray detector array assembly 106 and converts the data into digital signals for subsequent processing. For example, the digitized X-ray data is transmitted from the DAS 144 to the data storage server 130. The image reconstruction system 132 then retrieves the X-ray data from the data storage server 130 and reconstructs an image therefrom. The image reconstruction system 132 may include a commercially available computer processor, or may be a highly parallel computer architecture, such as a system including a multi-core processor and a massively parallel, high-density computing device. Optionally, the image reconstruction may also be performed on the processor 128 in the operator workstation 122. The reconstructed image may then be transmitted back to the data storage server 130 for storage or to the operator workstation 122 for display to an operator or clinician.

[0054] The C-arm X-ray imaging system 100 may also include one or more networked workstations 148. As an example, the networked workstation 148 may include a display 150, one or more input devices 152 (such as a keyboard and a mouse), and a processor 154. The networked workstation 148 may be located in the same facility as the operator workstation 122, or in a different facility, such as a different medical institution or clinic.

[0055] The networked workstation 148, whether in the same facility as the operator workstation 122 or in a different facility, can obtain remote access to the data storage server 130, the image reconstruction system 132, or both via the communication system 134. Thus, multiple networked workstations 148 can access the data storage server 130, the image reconstruction system 132, or both. In this manner, x-ray data, reconstructed images, or other data can be exchanged between the data storage server 130, the image reconstruction system 132, and the networked workstations 148 so that the data or images can be processed remotely by the networked workstations 148. This data can be exchanged in any suitable format, such as according to the Transmission Control Protocol ("TCP"), the Internet Protocol ("IP"), or other known or suitable protocols.

[0056] Although the present disclosure will be described below with reference to the use of a dual-plane fluoroscopic imaging system (e.g., a "C-arm" X-ray imaging system 100), in other non-limiting examples, other imaging systems (e.g., a single-plane fluoroscopic imaging system) may be used.

[0057] Figure 2is a schematic diagram of a process flow 200 for creating a motion-adjusted image of a patient to guide an interventional medical procedure. The process 200 includes acquiring (e.g., via the X-ray imaging system 100) a plurality of images 202 of a vasculature having non-contrast enhancement from a patient having vasculature having contrast enhancement, and acquiring a second plurality of images 204. Both of these plurality of images may be acquired throughout a respiratory cycle (e.g., from the beginning of inspiration to the end of expiration). The process 200 includes generating a static roadmap 206 of the patient's vasculature and generating a motion model 208 of the patient, each using the first plurality of images 202 and the second plurality of images 204. The process 200 further includes acquiring a third plurality of images 210, wherein an interventional medical device is deployed within the patient (e.g., inserted during surgery).

[0058] The process 200 may also include generating motion tracking data 212 for the patient or the interventional medical device using the third plurality of images. The process 200 may further include generating a motion transform 214 using the motion model 208 and the motion tracking data 212. The process 200 then includes displaying a static roadmap 216 and a third plurality of images 210 showing the interventional medical device aligned on the static roadmap using the motion transform 214. The alignment of the interventional medical device is based on user selections of: (1) motion compensation of the interventional medical device relative to the static roadmap to produce a plurality of images that do not show patient motion (e.g., vascular motion due to respiratory motion), and (2) motion adjustment of the static roadmap relative to the interventional medical device to produce a plurality of images that show patient motion.

[0059] Figure 3A more detailed process 220 is shown, which is a more specific implementation of the process 200 for creating a motion-adjusted image of a patient. Specifically, the process 220 includes first acquiring a non-contrast image 222 (e.g., a mask image) and a contrast image 224 of the patient during the entire respiratory cycle (e.g., from the start of inspiration to the end of exhalation). In some non-limiting examples, the contrast image 224 can be acquired using a specific procedure that ensures that the contrast image 224 is acquired only during the arterial phase, so that the vasculature of interest is fully contrasted and clearly visible in all captured frames. Specifically, the process 220 includes the steps of automatically detecting the arrival of contrast agent in the vasculature to determine the arterial, tissue, and venous phases of the injection. Then, only the arterial phase contrast enhancement image can be used. For example, the arrival of contrast agent is estimated by calculating the average intensity of each frame. Additionally, the average intensity of past frames including the current frame is also calculated. When the current average intensity is higher than the average intensity of two frames in a row, the algorithm will send a signal that the contrast agent has arrived. Then, the phase can be determined by calculating the center of mass in the Y dimension. Since contrast is usually injected at the bottom of the image (e.g., liver surgery), the center of mass moves downward and reaches a turning point at the end of the arterial phase, which can be used to determine and therefore only use arterial phase contrast images (e.g., images between the arrival of contrast and the event at the end of the arterial phase). Subsequently, the center of mass moves upward in the perfusion phase and moves downward again in the venous phase.

[0060] Once the non-contrast images 222 and contrast images 224 are acquired and selected (e.g., utilizing only contrast images during the arterial phase), the process 220 generates subtracted images formed by subtracting the contrast images 224 from the non-contrast images 222. These subtracted images may be used to generate a static roadmap 228 of the patient's vasculature. For example, the static roadmap 228 may be generated by averaging the subtracted images at consistent breathing states, such as at the end of exhalation (e.g., λ t =0) averages some or all of the subtracted images. Non-contrast images 222 and contrast images 224 are also used to generate a vascular system motion model 230, which is a specific implementation of motion model 208. Process 220 also includes tracking features 232 from non-contrast images. As discussed in more detail below, tracking features 232 may include identification and tracking of curvilinear features. Additionally or alternatively, tracking features 232 may include identification and tracking of a center of mass within a non-contrast image, as discussed below (e.g., with respect to a 3D guidance system).

[0061] Process 220 further includes acquiring a plurality of live images 234 that include the invasive medical device in the image and are used to track a respiratory state 236, which is a specific form of motion tracking data (e.g., with respect to process 200). The plurality of live images 234 are also used to extract the invasive medical device from the plurality of live images 234. Once extracted, the image of the medical device may be enhanced (e.g., compressed, stretched, changed in color, etc.) as indicated at step 240.

[0062] As will be further described, the process 220 also includes generating a transformed roadmap 238 using the tracking of the respiratory state 236 and the vasculature motion model 230. In one instance, the transformed roadmap 238 can be a motion compensation of the medical device image relative to the static roadmap 238. Alternatively, in another instance, the transformed roadmap 238 can be a motion adjustment of the static roadmap relative to an image of the medical device (e.g., extracted from the real-time image 234). The process 220 can generate a real-time display 242 of the current image 234, and the transformed roadmap 238 in either of at least two configurations. In a first configuration, the current image 234 and the static roadmap 228 are displayed, and the transformed roadmap 238 is motion compensation of the medical device image overlaid on the roadmap 238. In a second display, the current image 234 and the image of the medical device are displayed, and the transformed roadmap 238 is motion compensation applied to the static roadmap 228 to show patient movement. The interventional radiologist can switch between the first display and the second display to change which image is motion compensated (eg, the medical device or a static roadmap) based on the interventional radiologist user selection.

[0063] Figure 4 An example of a specific process for generating a vasculature motion model 230 is shown. The vasculature motion model 230 utilizes a subtracted image generated by subtracting a contrast image 224 from a non-contrast image 222. However, prior to subtraction, the non-contrast image 222 and the contrast image 224 are matched and registered at step 252, which ensures that each image within the non-contrast image 222 effectively corresponds in time to each image within the contrast image 224. At step 252, the sequence of non-contrast images may be defined as And the contrast image sequence can be defined as In both image sequence identifiers, t represents the acquisition time of a given image. Then, each contrast image is matched with a non-contrast image such that the mean square error between the images is minimized. For example, image represents the non-contrast frame corresponding to the contrast frame acquired at time t and can be determined by Equation 1. For example, Equation 1 determines the time point t of the corresponding non-contrast image of a given contrast image acquired at time tm Thus, Equation 1 returns the value of the index of the non-contrast frame corresponding to a given contrast frame. In some non-limiting examples, if more than one non-contrast image (e.g., 5) corresponds to a given contrast image, the non-contrast images can be averaged to generate an average non-contrast image to be subtracted from the corresponding contrast image, as discussed below.

[0064]

[0065] Although the image pairs may correspond to each other in time, there is no guarantee that the respiratory state of the two images is accurate. For example, slight differences in the position of organs or other anatomical structures may cause subtraction artifacts. To avoid this problem, block matching registration techniques can be used to register the non-contrast image to the corresponding contrast image. Therefore, Can be divided into different blocks B i , and the translation vector Δ can be estimated for each block by minimizing the gray value difference according to Equation 2 i As shown in Equation 2, n represents the number of pixels in each block, and E[X] represents the expected value of x.

[0066]

[0067] The final image transform is then calculated using cubic spline interpolation. Figure 5 An example of the output from this procedure is shown in FIG. The dashed line 250 represents the parsed block, while the solid line 251 represents the calculated cubic spline curve superimposed on the image. Figure 5 As shown, the light grey area represents the difference between the fixed image and the registered moving image.

[0068] After the corresponding frames are matched and registered (e.g., step 252), the images are subtracted and denoised at step 254 to generate a subtracted image. For example, the registered image pair is subtracted to obtain a difference image Since subtraction increases the noise variance, a Wiener filter is applied to reduce the noise and simplify the segmentation.

[0069] Once the corresponding images are subtracted and the noise is mitigated at step 254, a global thresholding algorithm can be applied to segment the vasculature, represented by a binarization step 256. This threshold is based on the Euler number determined by the number of connected components in the binary image minus the number of holes in these objects. For example, given any image, the Euler number for a threshold less than the minimum gray value in the image is 1, because all pixels are considered to be part of the same object.

[0070] When the threshold increases (for example, Figure 6 ), holes start to appear inside the objects, which causes the Euler number to become negative. Further increase in the threshold causes the segmentation to split into multiple objects, which again increases the Euler number (e.g., just after Figure 6 After reaching the second positive peak (e.g., Figure 6 After the second positive peak (e.g., at Figure 6 The local minimum value after (for example, Figure 6 In other words, the thresholds used are those that are higher than the threshold at the second positive peak (e.g., Figure 6 denoted as “3” in the figure). Figure 6 An example of the output of an image after the binarization process 256 is shown in , where four output images are represented, and each image has a specific threshold and corresponding Euler number. Typically, the threshold used is the value that occurs after the second maximum (e.g., X>20 threshold).

[0071] Although the process of generating the vascular motion model 230 involves a binarization step 256, in some non-limiting examples, the binarization step 256 may be omitted or bypassed. In this manner, for example, the subtracted image is denoised at step 254 and used directly to estimate the motion of the vasculature at step 258, as described below. In some cases, avoiding the binarization step 256 may be desirable for radiologists who are not accustomed to viewing binarized images.

[0072] The output image from the binarization process 256 represents the segmented vasculature for each frame and is represented as After the binarization process 256 is completed to produce a segmented image, the motion of the vasculature is estimated at step 258. For example, the deformation of the vasculature due to respiratory motion can be estimated by tracking the pixel motion between adjacent frames using a diffeomorphicdemons method. This method estimates a translation vector for each pixel, ensuring that both the image deformation and its inverse deformation are smooth. The transformation can be described according to Equation 3.

[0073]

[0074] Once the transformation is calculated, the respiratory motion of each individual vessel pixel can be accounted for using a single parameter λ t Parameterization, the parameter λ t represents the respiratory state at time t (e.g., at step 230). Therefore, according to Equation 4, the position p of each vessel pixel given the respiratory state is iIt can be approximated by a linear function (or other parameterization).

[0075] p i (λ i )=p i (0)+λ t ·(p i (1)-p i (0)) (4)

[0076] In equation 4, p i (0) represents the most exhaled breathing state in the initial contrast-enhanced sequence (i.e., λ t =0). On the contrary, p i (1) indicates the state of maximum inhalation (i.e., λ t =1). In some non-limiting examples, if the patient's respiratory motion exceeds the motion of the initial angiographic image sequence, then during the current tracking period λ t Values ​​outside the range [0, 1] may be assumed; however, due to the parameterization (eg, a linear function according to Eq. 4), the motion of the vasculature may be extrapolated to values ​​outside the range [0, 1].

[0077] In an alternative example of the motion estimation step 258, Equation 5 (below) is a cost function that can be used to generate a translation vector for each pixel that tracks motion between adjacent mask frames, but the translation vector is based on multiple respiratory variables for a given respiratory state, rather than a single respiratory variable for a given respiratory state (e.g., similar to Equation 3).

[0078]

[0079] For example, regarding Equation 5, t 0 is a constant offset, t c is the motion vector associated with chest breathing, t d is the motion vector associated with diaphragmatic breathing. c Set equal to 0, this cost function can also be used to generate a single respiratory variable for a given respiratory state. The translation vector t c and t d The corresponding breathing parameters (variables) r can be used respectively c and r d4) to generate a corresponding parameterization (e.g., a linear function) for each translation vector and a corresponding respiratory parameter (e.g., similar to Equation 4). Therefore, as will be discussed below, if (multiple) respiratory variables are known, the vascular system motion model 230 can generate (multiple) corresponding translation vectors. In the case of a multiple translation vector configuration, if two corresponding translation vectors are extracted using two extracted respiratory parameters (variables), the two corresponding translation vectors can be added to generate a combined translation vector.

[0080] In some non-limiting examples, the motion of the vasculature may be estimated at step 258 by using a two-dimensional (“2D”) affine image registration using a regular step gradient descent optimization method, or using a differential homeomorphic deformation registration method. This generates a transformation matrix that describes the deformation of the vasculature, and uses a single parameter λ t Alternatively, a multi-parameter method is used to parameterize multiple respiratory variables representing the respiratory state at time t, where the parameter λ t represents the respiratory state at time t (using step 230). Specifically, the parameterization is done for each matrix element, rather than for each pixel in the differential homeomorphic deformation registration method. This will generate a parameterized matrix that can be used for each respiratory state λ t calculate.

[0081] Once the motion of the vasculature is estimated at step 258 , the vasculature motion model 230 is created and may be stored, for example, at the operator workstation 122 or a networked workstation 148 .

[0082] Figure 7 An example process for generating a respiratory motion model 260 is shown, which is configured to extract (multiple) respiratory states from an image, input the (multiple) respiratory states into a vascular system motion model 230 to generate respiratory motion tracking data (e.g., (multiple) translation vectors) from a patient (e.g., a particular form of motion tracking data).

[0083] As discussed above, the respiratory motion model 260 generally determines the patient's current respiratory state by utilizing only native fluoroscopic images (e.g., a particular implementation of 236), and therefore does not require external sensors to determine the respiratory state that has plagued previous systems. The generation of the respiratory motion model 260 first generates a 3D image of the patient's current respiratory state in all non-contrast images (e.g., the images previously defined as ). This average image blurs moving edges but preserves static edges (e.g., edges near ribs). The respiratory motion model 260 then continues at step 262 for each non-contrast fluoroscopic image (e.g., in the image previously defined as In some non-limiting examples, as implemented, detecting edges 262 may include using a Canny edge detector. For example, applying a Canny edge detector to the first native image frame (i.e., ) to extract contours along anatomical structures. In other non-limiting examples, rather than utilizing a Canny edge detector, edges may be detected at step 262 by using convolution and derivatives of a Gaussian kernel.

[0084] Once the edges are detected, the average image output from the detect edges step 262 (e.g., the edge-filtered average image) is subtracted from each non-contrast image output from the detect edges step 262 (e.g., the edge-filtered non-contrast image) to generate an edge-filtered subtracted image. In some non-limiting examples, using the average image subtracted from the non-contrast image can eliminate non-moving edges, which can cause problems when tracking features. At step 264, the edge-filtered subtracted image is used to extract the centerline by using a topology-preserving thinning algorithm, which reduces all contours to a one-pixel-thin centerline. Then, by tracking each contour pixel from endpoint to endpoint, a set of curved features can be extracted from the centerline as a list of connected 2D coordinates. An endpoint is defined as a contour pixel that is connected to only one other contour pixel or a branch point that is connected to more than two contour pixels. As indicated in step 266, each feature f from the first non-contrast image frame is tracked on the remaining non-contrast image frames. i to generate tracking data in step 268. For example, this can be done by creating a cost image To achieve this, the cost image is formed by first applying a Canny edge detector to a given non-contrast image, then applying a Gaussian smoothing filter (or in some cases, convolution and the derivative of a Gaussian kernel), and subtracting that image from the previously determined mean image that has been output from the detect edges step 262. The cost function may then be optimized using the Nelder-Mead algorithm. Figure 8 An example of a cost image with highlighted curve features superimposed on the cost image is shown in .

[0085] The translation vector d that determines how the feature moves in space i (t) can be estimated by maximizing the average grayscale value along the curve feature using Equation 6.

[0086]

[0087] The characteristic motion can then be approximated by a linear function based on the breathing state(s). The characteristic f at time t iEvery point f ij The coordinates of can be described by Equation 7. In other words, the movement of the selected feature(s) can be used to generate a linear function that relates the movement of the feature to the respiratory state. (7)

[0089] In the implementation of multiple breathing parameters for a given breathing state (e.g., using r c and r d ), a separate linear function (similar to Equation 7) can be generated to relate the movement of the chest (e.g., chest characteristics) to the breathing parameter r c and the motion of the diaphragm (eg, diaphragmatic characteristics) and the respiratory parameter r d Using this information, at step 269 , parameterization may be completed to form the breathing model 260 .

[0090] For a linear function that relates features to a single respiratory state (e.g., Equation 7), some features are not suitable for tracking respiratory motion and are therefore not used in Equation 7. Therefore, as indicated at step 268, a subset is determined that contains only reliable tracking features that contribute to respiratory state estimation (e.g., diaphragm and chest). This subset of features can be obtained by first generating a vector by |f i |For a given feature f i Then, exclude all |f from the subset i |<25. In other words, initially captured features with less than 25 points are omitted. Similarly, only features with ||d i (1) Features with || ≥ 2.5 mm are used. This means that only features with a movement length greater than 2.5 mm are used throughout the breathing cycle. Therefore, the remaining features with generally smaller movement lengths are omitted. Specifying a specific required movement length (e.g., translation length) for a given feature helps avoid static features or those with only a small movement. Finally, to remove features that were not successfully tracked in all frames, the criterion c is calculated and used. a According to Equation 8, the criterion c is calculated based on the average gray value of each feature in the cost image over time. a .

[0091]

[0092] Regarding Equation 8, the variable n t represents the number of time frames in the contrast-enhanced image sequence. a (f i )<τ a The features of are excluded from the feature set. In some non-limiting examples, τ a can be any number, although according to this embodiment, τ is used a=0.3, which is determined empirically. The remaining features that pass the above three criteria are used to determine the respiratory state (eg, for calculation of Equation 7).

[0093] Once the vascular system motion model 230 and the respiratory motion model 260 are generated, a real-time system 280 (e.g., a specific implementation of steps 234, 236, 238, 240, 242) may be used, such as Fig. 9 As shown. Real-time system 280 begins by acquiring a current image 282 (e.g., similar to current image 234). Then, as indicated at step 284, a given current image within current image 282 is optimized within respiratory motion model 260 to analyze and extract curve features to determine the respiratory state. For example, for a given fluoroscopic image frame derived from a current fluoroscopic image acquisition, To estimate the respiratory state. As described above, the respiratory state is determined based on the extracted curve feature set. Therefore, for a given fluoroscopic image frame Compute cost image This is similar to calculation Once the calculation is complete, a linear search is performed over the entire range of respiratory states using Equation 9 (e.g., using the respiratory motion model 260) to find the current respiratory state λ t .

[0094]

[0095] As indicated in step 286, after finding the current breathing state λ t After that, the current state λ t It can be the input of the previously generated vascular system motion model 230. t A corresponding translation vector(s) is generated for each vessel pixel. Then, depending on a display selection (e.g., implemented via user selection), the translation vector(s) for each pixel may be applied to a static roadmap of the vasculature to generate a dynamic vessel mask 288 to motion compensate the vasculature. Alternatively, the translation vector(s) may be inverted and then applied to the image of the medical device to motion compensate the medical device. Any of these implementations may be displayed as indicated by step 290. Once displayed, the real-time system 280 repeats these steps by first utilizing another live perspective image (e.g., within live image 282). Thus, the real-time system 280 performs a real-time display of an accurate representation of the vasculature that affects the respiratory state of the individual at any point in time.

[0096] As described above, the vascular system motion model 230 is evaluated to determine how well the model is performed. A quantitative evaluation of the model is performed using a digital 4D CT phantom (XCAT, Duke University, Durham, North Carolina, USA), which provides realistic human anatomy, including complex respiratory and cardiac motion models. Respiratory motion can be changed by changing the respiratory cycle length, the maximum diaphragmatic motion, and the maximum chest expansion in the anterior-posterior direction. Each parameter can be changed individually. For each time frame, a CT volume is generated with an isotropic spatial resolution of 0.25 mm. The volume is then projected in the anterior-posterior direction using an orthogonal projection of a 28.95x 21.00 cm field of view. Contrast-enhanced images are simulated by setting the attenuation of the vessels in the hepatic arterial system to 120 Hounsfield units ("HU"). Additional Poisson noise is added to the image to simulate real image quality. A total of four test data sets were generated. Each data set contains 4 sub-datasets, each of which contains a contrast-enhanced image sequence and a native image sequence within a respiratory cycle, as well as a tracking sequence within four respiratory cycles. The temporal resolution of all sequences is 5 frames per second. The respiratory motion parameters were modified in each image sequence to simulate irregular breathing, as shown in Table 1 below. Specifically, Table 1 shows four sets of data sets, each with modified respiratory parameters including: cycle length, diaphragm motion, chest motion, and contrast-to-noise ratio ("CNR").

[0097]

[0098] Table 1

[0099] To evaluate the accuracy of respiratory motion tracking itself, the estimated respiratory state was calculated according to Equation 10 and the true state λ t The absolute error between λ .

[0100]

[0101] In addition to the absolute error calculation, the estimated dynamic vessel mask for each frame is compared with the ground-truth vessel position and shape. The accuracy is measured by the coefficient and the 99th percentile of the perimeter error. Given the ground truth and the estimated vessel segmentation, the perimeter error is calculated for each point along the contour of the estimated vessel segmentation as the distance to the nearest contour point in the ground truth segmentation.

[0102] The quantitative evaluation results of the above four groups of respiratory state tracking are as follows Fig.10 shown. Fig.10The histogram in represents the distribution of absolute errors in percentages, with line 270 representing the 50th percentile and bar 272 representing the 90th percentile. Fig.10 The bottom right graph shows the ground truth (ie, represented by a solid line 274) and the estimated breathing state (ie, represented by a dashed line with dots 276) for all sequences of Group 4. The black vertical dashed lines show the boundaries between different sequences.

[0103] For group 1, only the respiratory cycle length was varied in each image sequence, with an absolute error of 1.03 ± 1.22%. For group 2, only the maximum diaphragmatic motion was varied in each sequence, and an error of 1.03 ± 1.37% was observed. For group 3, the error of varying the maximum anterior-posterior motion was 1.03 ± 1.22%. For group 4, all parameters were modified dynamically, with an absolute error of 1.01 ± 1.21% at a CNR of 15 and 1.09 ± 1.25% at a CNR of 50. The accuracy of the estimated dynamic vessel mask produced similar Dice values ​​for all sequences, ranging between 0.94 ± 0.01 and 0.96 ± 0.01. The 99th percentile errors of the estimated vessel mask outlines were 0.92 ± 0.21 mm, 0.86 ± 0.29 mm, and 0.64 ± 0.09 mm for groups 1 to 3, respectively. For Group 4, the error was 0.96 ± 0.19 mm using a CNR of 15 and 0.86 ± 0.23 mm using a CNR of 50. An overview of all results is given in Table 2 below.

[0104]

[0105] Table 2

[0106] Fig.11 Overlays of ground truth and estimated vessel masks under different respiratory states are shown. Specifically, Fig.11 The first row of images in are native image frames used to estimate the respiratory state of three different time frames. The second row shows the dynamic roadmap (red) overlapped with the native image. The third row shows the overlap of the estimated dynamic vessels and the ground truth dynamic vessels. The consistent areas are shown in white, while the ground truth mask itself is shown in green. The estimated mask is shown alone in magenta in the last row.

[0107] As discussed above, a vascular roadmap (e.g., a vasculature motion model 230) can be created to provide an accurate representation of the current state of the vasculature, thereby allowing accurate reference to instruments during interventional radiology procedures. However, during fluoroscopic procedures, instruments, particularly guidewires, can be difficult to discern from surrounding tissue. Therefore, in some non-limiting examples, it is contemplated to provide a clear representation of the instrument by utilizing a neural network to segment the instrument from the fluoroscopic image and then superimpose the instrument onto the vascular roadmap (e.g., a vasculature motion model 230). In this case, the clear representation of the instrument can be displayed on a separate screen alongside an image of the dynamic vascular mask 288 or the static roadmap 228. Alternatively, rather than displaying the dynamic vascular mask 288 or the static roadmap 228 together, a continuous and connected representation of the medical instrument can be overlaid or superimposed with the static or dynamic image.

[0108] Previous systems have attempted to provide a clear representation of instruments (e.g., guidewires), but have not been successful. For example, one previous method for guidewire segmentation is to subtract a previously acquired non-contrast image (e.g., a mask image) and then perform a global thresholding. However, this technique is not suitable for applications with respiratory motion because subtraction artifacts can make it difficult to reliably segment the guidewire. Other techniques for guidewire segmentation in fluoroscopic images include, for example, the use of line enhancement filters based on Hessian filters or steerable filters. Unfortunately, these attempts do not ensure that the instrument is represented as a single connected path. Line enhancement filters attempt to address this issue, including incorporating the use of path search methods such as Dijkstra's algorithm or automatic driving vectors. Similarly, a different approach proposed by Vincent et al. and Birith et al. uses a local minimum path search based on an intensity-weighted distance transform to detect curved structures. The exponential complexity of the brute force approach can be reduced by evaluating the minimum path of increasing path length. Conversely, optimization-based techniques have been proposed that use non-rigid registration between adjacent frames or use phase-consistency optimized splines to detect curved features.

[0109] Finally, machine learning based methods were proposed that used classification methods to identify small segments, which were then combined using linear programming or hierarchical shape models based on principal component analysis. Many of the proposed methods require computationally expensive iterative methods, and the robustness of all methods depends on manually defined line detection features.

[0110] The deep learning method proposed in the present disclosure overcomes the problems associated with previous attempts. Due to the curved nature, inconsistent shapes, and unbalanced grade frequencies of instruments including guidewires, previous systems have difficulty extracting continuous and connected shapes. Instead, previous attempts extracted discontinuous and discontinuous shapes of instruments. Similarly, with previous attempts including subtraction, patient motion (e.g., respiratory motion) may create unwanted artifacts. The system and method according to the present disclosure overcome the above-mentioned defects of previous systems, for example, by segmenting the entire instrument from a given fluoroscopic image, so that a continuous and connected image of a medical instrument is extracted from the fluoroscopic image. Specifically, the extracted continuous and connected image of the medical instrument does not require the use of an algorithm or a hierarchical shape model. For example, previous machine learning methods input a fluoroscopic image containing a medical instrument and output a discontinuous and discontinuous image of the medical instrument. Therefore, these previous methods require other algorithms or hierarchical shape models to connect the discontinuous segments or parts of the medical instrument to create a continuous representation of the medical instrument. Notably, the system and method according to the present disclosure can output a continuous and connected image of a medical instrument for a given fluoroscopic image containing a medical instrument.

[0111] Specifically, Fig.12 As shown, the neural network 300 is based on the SegNet architecture, using a VGG-16 network with pre-trained weights from the "ImageNet Large Scale Visual Recognition Challenge" dataset. The neural network 300 includes a total of 91 layers, which are grouped in an encoder stage 302 or a decoder stage 304. The encoder stage 302 is generally used to reduce information and is composed of specific layers. The encoder stage 302 includes a first input layer 306 configured to receive a given image. The encoder 302 is typically followed by thirteen layers 308, wherein each layer 308 includes a convolutional layer, a batch normalization layer, and a rectifier linear unit ("ReLU"). The layer 308 is in Fig.12 The encoder 302 also includes five max pooling layers 310 that reduce the size of the input image from 1024 x 1024 pixels to 32 x 32 pixels. The max pooling layers 310 are Fig.12 The decoder 304 generally has the same stages as the encoder 302, but in reverse order. Additionally, the decoder 304 has replaced the pooling layer 310 with the unpooling layer 312, which is Fig.12310 is designated as a dashed box in FIG. 311 . The non-pooled layer 312 upsamples the reduced information and locates the object. The solid line connecting a given pooling layer 310 to the corresponding non-pooling layer 312 indicates the forwarding of the index of the maximum value within the pooling window. The decoder 304 further includes a layer 314, which is Fig.12 The softmax (soft maximum transfer function) designated as the dashed line in FIG. 3 and used to normalize the output. The decoder 304 and the final layer of the neural network 300 are classification layers (not shown) that use a weighted cross entropy loss function for two classes (eg, guidewire and background).

[0112] To train the neural network 300, the image data was randomly divided into training (60%), validation (20%), and test (20%) datasets. Training was performed using a single graphics processing unit ("GPU") (e.g., NVIDIA Geforce GTX 1080Ti, Santa Clara, California, USA). Training was performed using a stochastic gradient descent technique with an initial learning rate of 0.001, a momentum of 0.9, an L2 regularization of 0.0005, and a batch size set to 1 image. Since the prior probabilities of the two classes are very different (approximately 99.6% of all pixels are background pixels), the class weights were adjusted to avoid the network being biased towards the background class. Therefore, the relative frequency of each class was determined based on the training data. The class weights were then set to the inverse of the relative frequency. The neural network 300 was trained for 56,768 iterations (1 epoch) and received a validation accuracy of 99.91%.

[0113] To improve image quality, the neural network 300 may be trained by providing the current image and the past two frames as inputs to the neural network 300. Advantageously, this allows the neural network 300 to learn to track moving parts in the image, which are typically easier to detect than static objects due to noise. In other non-limiting examples, the neural network 300 may be trained by providing the current image and the past four frames.

[0114] After training on the above dataset, the neural network 300 is evaluated by comparing the segmentation results on the test dataset with the mask subtraction based segmentation algorithm. A corresponding mask image is created for each test image containing only the anatomical background, assuming that no motion occurred between the acquisition of the mask and the test image. The same amount of Poisson noise used for the guidewire image is added to the mask image to simulate an actual clinical acquisition. The mask image is then subtracted from the test image, filtered using an orientation line enhancement filter, and globally thresholded to extract the guidewire pixels. An intensity threshold is selected based on the training dataset. The false positive rate and false negative rate, Hausdorff distance, and The Dice coefficient is used to measure the segmentation accuracy of the two algorithms. These results are shown in Table 3, which compares the performance of deep learning and mask subtraction-based segmentation. coefficient (“SDC”), false positive rate (“FPR”), false negative rate (“FNR”), and Hausdorff distance (“HSDF”).

[0115]

[0116] Table 3

[0117] Fig.13 The comparison results between different methods are visually represented. For example, the first column of images shows the starting image, the second column of images is the resulting image output from the neural network 300, the third column of images represents the results from the mask subtraction segmentation algorithm, and the last column is the ground truth column (e.g., an image of an instrument example superimposed on a fluoroscopic image).

[0118] In some non-limiting examples, deep learning methods for segmenting instruments (such as guidewires) may be more advantageous than previous attempts in situations where the acquired fluoroscopic images have large intensity variations, include image noise, and have small device signals relative to the entire image (e.g., in chest / abdominal surgery). For example, although previous subtraction-based methods eliminate large variations in background intensity and allow the use of threshold-based segmentation, accurate segmentation in the presence of noise remains challenging. In fact, subtraction-based segmentation is often discontinuous or incomplete, despite the use of optimized thresholds derived from a training dataset. Additionally, subtraction-based results are obtained without simulating the respiratory motion that typically occurs between the mask and guidewire images. Therefore, the subtraction artifacts that already exist will be made worse if acquired during respiratory motion, as this introduces additional, typically worse, subtraction artifacts. Therefore, deep learning methods allow for accurate segmentation of instruments with curved shapes (e.g., guidewires).

[0119] In some non-limiting examples, components of the vasculature motion model 230 (and corresponding systems or processes) and the instrument deep learning segmentation method can be combined. For example, components within each system can be formed to create the guidance systems 320, 340.

[0120] Fig.14A block diagram of a guidance system 320 is shown. As shown, the guidance system 320 has similar components that have been introduced. Therefore, the previous description of these components also applies to the components within the guidance system 320. The guidance system 320 includes the acquisition of a non-contrast fluoroscopic image set 322 and a contrast contrast fluoroscopic image set 324. Each of these image sets is similar to the image sets previously discussed and, as described above, is used to generate the vasculature motion model 230 and the respiratory motion model 260. Each of the image sets 316, 318 can be obtained prior to an interventional procedure, and the vasculature motion model 230 and the respiratory motion model 260 can be generated prior to the interventional procedure.

[0121] The neural network 300 may be trained while the vasculature motion model 230 and the respiratory motion model 260 are being generated, or in some cases, prior to acquiring the image sets 316, 318. The neural network 300 may be trained using training images 316, which may be, for example, previously acquired fluoroscopic images that include instruments to be used in the surgery. Although the discussion below will be described with reference to guidewires, other instruments may be used. For example, other curvilinear instruments such as catheters may be used. However, the neural network 300 needs to be trained for the specific instrument to be used. Alternatively, in some cases, if the neural network 300 has been previously trained to extract a continuous and connected guidewire, then in a subsequent surgery, the neutral network 300 need not be trained again prior to the surgery.

[0122] Once the neural network 300 is prepared, a real-time fluoroscopic image 326 is acquired (e.g., via the x-ray imaging system 100) while a guidewire is inserted into the patient (i.e., the procedure has begun). In some cases, as shown, if the interventional radiologist desires, the real-time fluoroscopic image 322 can be displayed on a display 331 (e.g., similar to the display 150, the display 124, etc.). Accordingly, a given image within the real-time fluoroscopic image 326 is directed to both the neural network 300 and the respiratory motion model 260. As previously discussed, a given image within the real-time fluoroscopic image 326 input to the neural network 300 will output a continuous and connected image of the guidewire 328, which can be displayed on the display 330 (e.g., similar to the display 150, the display 124, etc.). Additionally, a given image in the real-time fluoroscopic image 326 is directed to the respiratory motion model 260, which analyzes and subsequently extracts curve features from the given image for use in the respiratory motion model 260 to determine the respiratory state 332 (e.g., λ t). Respiration state 332 is input into vasculature motion model 230 to determine corresponding translation vector(s) for each pixel of respiratory state 332. The translation vector(s) for each pixel are applied to a previously generated static roadmap within vasculature motion model 230 to generate dynamic vascular mask 334. Dynamic vascular mask 334 may be displayed on display 336 (e.g., similar to display 150, display 124, etc.).

[0123] The displays 331, 330, 336 within the guidance system 320 allow the interventional radiologist to view key features individually. For example, during the procedure, the display 336 shows a motion-compensated representation of the vascular system, which works in compression and changes the spatial position of the vascular system during the respiratory cycle. This is advantageous because the vascular system is difficult to view in native, unprocessed fluoroscopic images, and previous segmentation techniques for the vascular system do not show an accurate moving representation of the vascular system, but rather a static representation of the vascular system. During the procedure, the interventional radiologist can also view a clear representation of the guide wire (or other instrument) on the display 330. In some non-limiting examples, the continuous and connected images of the medical device 328 can be adjusted (e.g., increase size, change color, etc.) before being displayed on the display 330. This is advantageous because, similar to the vascular system, some instruments, including the guide wire, are difficult, if not impossible, to view in native, unprocessed fluoroscopic images. Finally, the interventional radiologist can view the real-time fluoroscopic image 326 on the display 331. Therefore, the interventional radiologist can view whichever screen he desires to effectively guide the guidewire to the target.

[0124] In some cases, it may be desirable to combine the images prior to display so that the interventional radiologist can view a single display rather than constantly shifting focus between displays (e.g., displays 331, 330, 336). Therefore, in some non-limiting examples, a guidance system 340 is provided that allows for a single display so that the interventional radiologist only needs to view a single display to effectively complete the interventional procedure.

[0125] Fig.15 A block diagram of a guidance system 340 is shown. The guidance system 340 has similar components already introduced, particularly with respect to the guidance system 320. Therefore, the previous description of these components also applies to the components within the guidance system 340. For example, a given image within the real-time fluoroscopic image 326 is directed to the respiratory motion model 260, which analyzes and then extracts curve features from the given image to compare the curve features within the respiratory motion model 260 to determine the respiratory state 332 (e.g., λ t). In addition, a given image within the real-time fluoroscopic image 326 input to the neural network 300 will output a continuous and connected image of the guidewire 328. However, in the align instrument step 339, the continuous and connected images of the instrument 328 are combined in a particular manner with the translation vector(s) for each pixel of the static vascular roadmap and respiratory state 332. The manner in which these are combined, e.g., how motion compensation is applied, depends on the user selection 338 (e.g., an actuation button, a user interface, etc.).

[0126] For example, user selection 338 can select a first state, applying the inverse of the translation vector (s) of each pixel of the respiratory state 332 to a continuous and connected image of the instrument 328. The motion-compensated image of the medical instrument is overlapped with a static vascular roadmap (e.g., also output from the vascular system motion model 230). The output from the first state is displayed on a display 346 (e.g., similar to display 150, display 124, etc.). The first state applies motion compensation to the guidewire while keeping the vascular system image (e.g., static vascular system roadmap) stationary. Therefore, the first state is advantageous for interventional radiologists because the target (e.g., part of the vascular system) is stationary. However, the guidewire is moved (e.g., translated, compressed) according to a specific respiratory state to compensate for respiratory motion. Therefore, the position of the guidewire can be corrected so that the guidewire is aligned with the static vascular roadmap. This allows the interventional radiologist to guide the guidewire to a target position that is stationary (e.g., a static roadmap) and manipulate the guidewire so that the apparent movement of the guidewire (e.g., on the display) can only be caused by the interventional radiologist.

[0127] The user selection 338 may also select a second state of the align instrument step 339. As discussed above, the align instrument step 339 receives the continuous and connected images of the instrument 328, the static vascular system roadmap, and the (multiple) translation vectors for each pixel of the respiratory state 332. However, in the second state, the (multiple) translation vectors for each pixel are applied to the static vascular roadmap to generate a dynamic vascular mask (e.g., dynamic vascular mask 334), and the continuous and connected images of the dynamic vascular mask instrument 328 are overlapped with the dynamic vascular mask. This second state applies motion compensation to the image of the vascular system to generate a "video" of the moving vascular system together with subsequent images. This enables the interventional radiologist to clearly view the guidewire relative to the accurate representation of the target vascular system. For example, the desired target point of the guidewire as part of the vascular system moves and deforms throughout the respiratory cycle and is ideally coincident with the guidewire (e.g., the guidewire is aligned with the dynamic vascular mask 334). Therefore, specifically, the vascular system forms a real-time video so that the interventional radiologist can accurately guide the guidewire to the target location.

[0128] In some non-limiting examples, the guidance systems 320, 340 may be implemented on the X-ray imaging system 100. In some cases, the X-ray imaging system 100 may be modified to efficiently run the neural network 300 (eg, including a GPU, parallel processors, etc.).

[0129] Fig.16 A diagram is shown for displaying a static roadmap and a dynamically "moving" vasculature to show an interventional medical device aligned on the static roadmap using a motion transform. That is, as will be described, the alignment of the interventional medical device on the static roadmap can be based on a user selection. The motion adjustment enables alignment of at least two different display options for the user. For example, the motion adjustment can include motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion. Alternatively, the motion adjustment can be relative to the static roadmap of the interventional medical device to produce multiple images that show patient motion.

[0130] Fig.16 The vasculature is shown in two different breathing states. In the example shown, the same vasculature is shown at a first location in a first breathing state 350 and at a second location in a second breathing state 352. As can be seen when only the two locations of the same vasculature are overlapped during the two breathing states 350, 352, the failure to adjust for the patient's motion obscures otherwise clear information. To add further complexity, the systems and methods described above not only Figure 3 In addition to compensating or adjusting for patient motion during acquisition of the current image 234, the moving current image 234 must also be compensated or adjusted relative to the static roadmap 228. Furthermore, the vasculature is not a rigid structure that simply moves during patient motion. Rather, the vasculature translates, rotates, compresses, stretches, etc. Even in addition to all of these biological structures, interventional medical devices must track and adjust based on changes in the anatomy. Therefore, as Fig.16 As shown, the blurred information caused by vessel movement in the current image between two respiratory states is only one layer of complexity described in the present disclosure.

[0131] Likewise, the systems and methods described above are capable of performing motion compensation or motion adjustment throughout the patient's entire range of motion, including compression, expansion, deformation, translation, rotation, and the like. Fig.17 A fluoroscopic image output from a first implementation of the guidance system 320 is shown (eg, without a static vascular roadmap). Fig.17 The top fluoroscopic image in shows the vessel in a more compressed / contracted state (e.g., during exhalation), while Fig.17 The bottom fluoroscopic image in shows the vessel in a decompressed / dilated state (eg, during inspiration).

[0132] Fig.18 A fluoroscopic image (e.g., with a static vascular image) output from a second implementation of the guidance system 340 is shown. As shown, the vascular system does not move from the top fluoroscopic image to the bottom fluoroscopic image, which are acquired at different time points. Instead, each fluoroscopic image shows only the guidewire moving.

[0133] like Fig.19 As shown, the display design according to the present disclosure is designed to allow the user to select between (1) motion compensation and (2) motion adjustment. That is, the user can use the systems and methods described above to display images using motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion. Alternatively, the user can use the systems and methods described above to display images using motion adjustment of the static roadmap relative to the interventional medical device to produce multiple images that show patient motion.

[0134] Specifically, refer to Fig.19 , a display 380 according to the present disclosure. Display 380 shows a static vascular roadmap 382 (shown in solid lines). Display 380 also shows an interventional medical device 384. Display 380 is used for illustrative purposes to show that a user can select between (1) compensation for motion of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion and (2) adjustment of the static roadmap relative to motion of the interventional medical device to produce multiple images that show patient motion.

[0135] If motion compensation is selected, a static vessel roadmap 382 will be shown in the permanent display and Figure 3 The display of the interventional medical device 384 moving in the current image 234 of the image is presented as being aligned consistently with the static vascular roadmap 382. Thus, this is referred to as "motion compensation" because compensation is applied to remove the appearance of the user's motion. That is, from the fixed external perspective of the user when viewing the display 380 of the image, the static vascular roadmap 382 and the interventional medical device 384 do not move with patient motion (such as respiratory motion). Instead, the static vascular roadmap 382 is displayed as a stationary structure and is aligned therein as the interventional medical device 384 is advanced through the static vascular roadmap 382.

[0136] Alternatively, if motion adjustment is selected, the static vascular roadmap 382 is adjusted to track Figure 3234 of the current image 234. In this manner, the static vascular roadmap 382 is kinematically adjusted between a static position shown in solid lines and a trajectory that moves with the movement of the interventional medical device 384 as it moves to the second position 386. That is, as the static vascular roadmap 382 moves to the second position 386, it also moves to the second position 388 to maintain alignment with the interventional medical device 384. In this manner, patient motion, such as respiratory motion, is displayed in the display 380. That is, from a fixed external perspective of the user while viewing the display 380 of the image, the static vascular roadmap 382 and the interventional medical device 384 do move with patient motion, such as respiratory motion, such that the static vascular roadmap 382 appears to move and align with the interventional medical device 384 as the interventional medical device 384 is advanced through the static vascular roadmap 382 and moves to the second position 386.

[0137] Although the guidance systems 320, 340 have been described for two-dimensional implementations of fluoroscopic guidance, in some non-limiting examples, it is desirable to have a guidance system that can implement three-dimensional guidance. The three-dimensional guidance system 400 utilizes some previously disclosed components. Therefore, the components discussed above also relate to the three-dimensional guidance system 400.

[0138] Fig. 20 4 is a block diagram of a three-dimensional guidance system 400 that can be implemented on the X-ray imaging system 100 (or the modified variants discussed above) and can include the acquisition of a contrast 2D fluoroscopic image set 402, a non-contrast 2D fluoroscopic image set 404, and a 3D contrast X-ray image set 406. The contrast 2D fluoroscopic image set 402 and the non-contrast 2D fluoroscopic image set 404 can be the same as the image sets 324, 322, respectively (e.g., both image sets contain fluoroscopic images of the entire respiratory cycle). Alternatively, the image sets 402, 404 can be acquired by using the following process.

[0139] In some non-limiting examples, the desired fluoroscopic imaging vasculature can be the portal vein system and can be obtained by performing superior mesenteric artery angiography. In order to inject contrast agent, an angiographic catheter (e.g., 5Fr size) can be placed in the superior mesenteric artery under fluoroscopic image guidance, and then a contrast agent (e.g., iodine) is injected, followed by injection of saline. When the injected contrast agent drains from the superior mesenteric artery to the portal vein, a fluoroscopic image is acquired (e.g., by a single mode or dual flat panel mode of the imaging system 100). For example, a contrast 2D fluoroscopic image set 402 can be obtained at a specific sampling rate (e.g., 15 frames per second) and includes fluoroscopic images throughout the patient's entire respiratory cycle. After the contrast agent has been discharged (e.g., the residue of the contrast agent does not remain in the vascular system), or conversely, before the contrast agent is injected, a non-contrast 2D fluoroscopic image set 404 can be acquired so that the image set includes a perspective image throughout the patient's entire respiratory cycle.

[0140] Once the image sets 402, 404 are acquired, or before both are acquired, a 3D contrast X-ray image set 406 can be acquired. The 3D image set can be acquired by first using an intra-arterial injection (e.g., in the superior mesenteric artery) and then using a saline injection. After the injection, the patient performs a breath hold during the acquisition of the image set 406. Specifically, the image set 406 includes X-ray images acquired at two different spatial locations of the X-ray detector array assembly 106. For example, within the 3D contrast X-ray image set 406, multiple X-ray images are acquired at different pivot angles of the X-ray source assembly 104 and the X-ray detector array assembly 106 around the horizontal pivot axis 116. Therefore, the images within the image set 406 are acquired on multiple planes. In some non-limiting examples, the pivot angle can be between 0° and 360°, with any step size therebetween. In other non-limiting examples, only two perspective images within the image set 406 need to be from separate planes to extrapolate the 3D volume of the vascular system. In yet another non-limiting example, to avoid additional radiation and contrast injection from the acquisition of two contrast image sets 402, 406, a 3D contrast fluoroscopic image set 406 may be acquired during the acquisition of the contrast 2D fluoroscopic image set 402. For example, a particular target location may be selected and the 3D location may be determined by the intersection of the back-projections of the 2D points in the 3D volume space if fluoroscopic images from two different planes are acquired, as will be discussed in more detail below.

[0141] Once the 3D contrast X-ray image set 406 is acquired, a 3D volume can be generated. Then, a 3D volume of the vascular system 408 (e.g., the portal venous system) can be extracted from the 3D volume, or removed from the 3D volume (e.g., from other anatomical parts in the 3D volume). For example, this can be achieved by first applying a Gaussian filter (e.g., standard deviation 1.5) to reduce noise. Then, a global threshold method (e.g., threshold = -250) can also be used to extract only the 3D volume of the vascular system 408, similar to the process used to generate the vascular system motion model 230. Additionally, a connected component analysis can be performed to find all connected regions in the binarized volume. In some cases, only the largest region can be used to represent the vascular system. For example, smaller regions can be manually parsed and deleted because they are usually noise and defects. The output from these produces a 3D volume of the vascular system 408.

[0142] Although respiratory motion tracking utilizes a different process to track respiratory state, extraction of curve features may be substituted for determining respiratory state (e.g., replacing respiratory motion calculation 414). Guidance system 400 includes generating a respiratory motion model 414 that utilizes brightness differences between regions within a fluoroscopic image, where the regions may include locations above and below the diaphragm. Specifically, to generate respiratory motion model 414, an average brightness in the x-dimension may be calculated for a given fluoroscopic image within image set 404. For example, calculating the average brightness in the x-dimension helps reduce information, which is represented in Equation 11 as Essentially, the output using the average or sum in the x dimension is the same because the mean normalization cancels out in equation 11. In some non-limiting examples, in addition to calculating the average brightness in the x dimension, the sum, maximum, median, minimum, etc. can also be used to reduce information. The x-dimensional calculation can be used to determine the center of mass in the y dimension, which is used as an indicator of respiratory state. Specifically, the center of mass in the y dimension is related to the point (in the y direction), where the sum of all pixel values ​​above the center of mass is the same as the sum of all pixel values ​​below the center of mass. The center of mass calculation in the y direction is the same as the r(t) calculation. Once the center of mass in the y dimension is calculated for a given image, the respiratory state r(t) is related to the center of mass in the y direction according to equation 11. This process is used to calculate the respiratory state of all given images in the non-contrast 2D fluoroscopic image set 404. With respect to equation 11, the variable t represents time, which is the fluoroscopic image acquired at time t. This method can be used for single-plane or dual-plane acquisition. For dual-plane data, the respiratory state of each image is averaged.

[0143]

[0144] Fig.21A plot of the output of Equation 11 (eg, respiratory state) versus the estimated motion of the vasculature in the Z dimension is shown. As shown, the respiratory state is closely related to the relative motion of the vasculature in the Z dimension.

[0145] Reference again Fig. 20 , in order to create a vascular motion model 416, the contrast image set 402 and the non-contrast image set 404 are subtracted. More specifically, for each frame in the contrast image set 402, there are many more corresponding frames in the native 2D image set 404. Therefore, for each frame in the contrast image set 402 at an acquisition time, five of the best corresponding frames in the non-contrast image set 404 corresponding to the acquisition time are selected. For example, in order to identify image pairs, the mean square error can be calculated and minimized to generate contrast and non-contrast frame pairs. This is similar to the process of equation 1. However, typically, as described above, there are more non-contrast images than contrast images. Therefore, for each contrast image, there are many matching non-contrast images (e.g., 5, 10, etc.). The best match can be selected from the number of matching non-images, such as 5 best matches, or a single best match can be used instead. If more than one best match is used, the images will be averaged. The corresponding contrast image is then subtracted from the average image based on the best match of the non-contrast image, or from the best matching non-contrast image. This subtraction is done for all contrast images. Subtraction removes any anatomical background.

[0146] Then, to create the vascular motion model 416, a cost function CF is defined that projects all points in the 3D volume of the vascular system 408 into the 2D image space. Once the 3D volume 408 is projected into the 2D image space, the average squared brightness is calculated for all projected points in the subtracted image, where the output is defined as the cost.

[0147] All subtracted image frames are then calculated using the above cost function. Once evaluated, the subtracted 2D frame with the lowest cost is identified and selected as the starting frame. The subtracted 2D frame is used to perform 3D to 2D registration by minimizing the cost function of the estimated affine transformation matrix. The minimization is performed using a regular step gradient descent algorithm. The algorithm is applied to blurred versions of the subtracted image with different blur kernels, starting with a large kernel to allow for larger translations. In the first step, only rigid body motion is considered, and then affine parameters are considered. A regularization term can be added to the cost function to avoid large deformations of the vascular system. Afterwards, 3D to 2D registration will be applied to the remaining subtracted image frames. In some non-limiting examples, the results of the previous frame or the next frame are used as initialization.

[0148] After generating a transformation matrix for each subtracted 2D frame, a linear function, or in some cases a parameterization as discussed above, is estimated for each parameter of the transformation matrix, which maps or relates the respiratory state to the respective parameters. Thus, the respiratory state of each subtracted 2D frame is used as x and the parameters of each subtracted 2D frame are used as y. A robust regression method using a bi-square cost function is then used for estimation and a linear relationship is generated between the respiratory state and the parameters within the transformation matrix.

[0149] In some non-limiting examples, as discussed above, the 3D volume of the vasculature 408 is generated by only two different planes of the fluoroscopic images. Therefore, in some cases, the contrast fluoroscopic image set 402 can be used to generate the 3D volume of the vasculature 408, and the generation of the vasculature motion model 416 will be slightly different. For example, instead of using the acquired 3D DSA, this alternative method defines the target in two frames of the 2D projection image. The target position is tracked over time using the surrounding neighborhood of the target position. For all subsequent frames, a cost function is minimized, which calculates the mean square error between the patch around the target in the original image and the patch around the transformed target in the current image. After tracking the target in the two image sequences, the corresponding 3D position can be calculated for each 2D subtracted frame by determining the projection rays to the two planes of the 3D volume and calculating the intersection (or if there is no exact intersection, the closest point to the two lines) to generate the 3D volume 408. Then, a 3-element translation vector is used to estimate the linear relationship between the 3D target position and the respiratory state to create the vasculature motion model 416, rather than using the affine transformation matrix discussed above.

[0150] Once the respiratory motion model 414 and the vasculature motion model 416 are generated, a 3D volume of the instrument can be generated at 420. First, a real-time fluoroscopic image 418 is acquired via the X-ray imaging system 100, and the real-time fluoroscopic image 418 includes the instrument acquired from different planes. These images within the real-time fluoroscopic image 418 are then processed to segment the instrument and create a 3D volume of the instrument (e.g., at 420). To segment the instrument, the mean square error of the current image relative to each non-contrast image frame is calculated, and the best non-contrast image frame (e.g., the image frame with the smallest error) is subtracted from the given / current image. A line detection filter is then applied, followed by a dynamic threshold that varies with the average intensity of each image line to binarize the image. In some non-limiting examples, rather than in this process, instrument segmentation can be performed using a neural network (e.g., as previously described neural network 300).

[0151] After the segmented 2D image is binarized, the segmented binary 2D image is thinned using topology-preserving thinning, and all possible curve segments are extracted. Then, a 2D path search is applied, which finds a single connected path representing the centerline of the instrument. The curve segment set represents a directed graph, in which each segment represents a node, and the connection weight is defined by the Euclidean distance between the endpoints of the two segments and the angle between the segments. Additionally, the number of points of all unused segments will be added to the final cost of each path. The Dijkstra method is used to find the path that minimizes the total cost through the segment. After segmenting (or outputting from the neural network 300) the 2D device path for two real-time images obtained from two different planes, the corresponding point pairs from the two planes are identified, and the corresponding point pairs represent the same 3D points. This can be achieved by constructing a 2D image, in which each pixel represents a potential point pair, all pixels from left to right represent points on the device path from the first plane (starting from the tip), and pixels from top to bottom represent points on the second plane (starting from the tip). The value of each pixel is the distance between the projected 3D ray from the focus to the corresponding point on the 2D plane. Then, a monotonic function identifies point correspondences, which can be determined by finding a path that minimizes the cost of running through the image from the upper left pixel to the lower boundary or the right boundary. Each pixel on the path represents a point correspondence. Then, the 3D device path is calculated by reconstructing each pair of corresponding points separately, where the points are reversely projected into the volume space, and the intersection (or the point closest to the two lines) represents the 3D position. The process generates a 3D centerline of the instrument at step 420. Then, if the 3D centerline is determined, the 3D volume of the instrument can also be generated at step 420. For example, the cross section (or any desired cross section) and diameter (or any desired diameter) of the instrument are used to extrude along the 3D centerline to generate the 3D volume of the instrument. Alternatively, the 3D centerline method can be used only for initialization. For example, for all subsequent frames (e.g., in the real-time fluoroscopic image 418), the instrument can be tracked using 3D to 2D registration. This allows reconstruction without first segmenting the device and may be more robust in some cases. Therefore, the point cloud representing the 3D device centerline from the previous frame is reprojected into the 2D image space, and a cost function is calculated based on the average squared brightness. Then, an affine transformation is estimated by minimizing the cost function (e.g., using the Nelder-Mead Simplex approach) to determine the change in the device position and orientation compared to the previous time frame. If P represents the set of points representing the device centerline from the previous frame, the transformation can be determined based on the current image I according to Equation 12.

[0152]

[0153] The new device position can be described by Equation 13, where the point set is P n .

[0154] P n ={T·x 0 , T·x 1 , ...T·x n} (13)

[0155] Once the respiratory motion model 414, vasculature motion model 416, and 3D volume of the instrument are generated, an interventional radiology procedure may be performed. As discussed above, the real-time fluoroscopic image 418 may be directed to the respiratory motion model 414 to determine the respiratory state r(t) indicated by reference numeral 422. After the respiratory state r(t) 422 has been determined, the respiratory state is input to the vasculature motion model 416 to generate a corresponding transformation matrix (or 3-element translation vector) based on a previously calculated linear relationship (e.g., an estimated linear relationship between the transformation matrix / 3-element translation vector and the respiratory state).

[0156] The align instrument step 428 may be implemented on the operator workstation 122, the networked workstation 148, etc., and may apply motion compensation to the 3D volume of the instrument or the 3D volume of the vasculature 408. In a first implementation, the align instrument step 428 receives the transformation matrix / 3-element translation vector corresponding to the respiratory state 422, inverses the transformation matrix / 3-element translation vector, and applies it to the 3D volume of the instrument, thereby spatially manipulating the instrument. However, alternatively, in some non-limiting examples, the inverse of the transformation matrix / 3-element translation vector may be applied to the 3D centerline of the instrument, and then, the spatially manipulated 3D centerline may be extruded to generate the spatially manipulated 3D volume of the instrument. The 3D volume of the vasculature 408 may then be superimposed with the motion compensated 3D volume of the instrument and may be displayed on a display 430 (e.g., similar to the display 150, the display 124, etc.). This allows a relatively static representation of the vasculature (e.g., the 3D volume of the vasculature) to be aligned with the motion compensated 3D volume of the instrument, so that apparent motion of the instrument can only be caused by the interventional radiologist's manipulation.

[0157] The calibrate instrument step 428 may also apply motion compensation to the 3D volume of the vasculature 408. For example, the calibrate instrument step 428 receives a transformation matrix / 3-element translation vector corresponding to the respiratory state 422 (e.g., from the vasculature motion model 416) and applies it to the 3D volume of the vasculature 408. Then, at step 420, the spatially manipulated 3D volume of the vasculature may be overlaid with the 3D volume of the instrument to display it on the display 430. This allows the medical instrument to be aligned on the spatially moved 3D representation of the vasculature. In some non-limiting examples, an interventional radiologist may easily convert / switch between either implementation (e.g., the calibrate instrument step 428).

[0158] Movement data associated with the spatial manipulation of different portions of the vasculature from one respiratory state to another (e.g., the translation matrix / 3-element translation vector above) can be used to generate a 3D motion compensated representation of the vasculature, or alternatively, a 3D motion compensated representation of a medical device. For example, if the respiratory state is known, movement data indicating how the vasculature should move in that respiratory state is also known. This movement data can then be applied to a 3D volume of the vasculature (e.g., at r(t) in a fully inhaled or exhaled state) to manipulate the 3D volume of the vasculature in the observation plane so that the 3D volume represents the true orientation of the vasculature in that respiratory state. Alternatively, this movement data can also be applied to the medical device to move the medical device based on movement of a specific location of the 3D volume of the vasculature. For example, if the medical device 460 is located in a portion of the vasculature that translates, rotates, compresses, extends, etc., then the inverse of that movement of the vasculature will be applied to the medical device. For example, this may be advantageous because the target (e.g., vasculature) may be a static 3D volume, while the instrument may move (e.g., compress, extend, rotate, translate) based on the movement of the vasculature in that particular respiratory state. As mentioned above, the interventional radiologist may switch between a first state of the guidance system 400 based on user selections, in which the movement data is applied to the 3D volume of the vasculature to move the 3D volume of the vessel within the observation plane (e.g., to compensate for respiratory motion of the vasculature), and a second state in which the movement data is applied to the 3D volume of the medical instrument to move the 3D volume of the medical instrument within the observation plane (e.g., to compensate for respiratory motion of the vasculature).

[0159] The systems described above may be configured or otherwise used to perform surgery according to the present disclosure. In particular, as will be further described in detail, the present invention has been described according to one or more preferred non-limiting examples, and it should be understood that many equivalents, alternatives, variations and modifications are possible, except for those explicitly stated, and are within the scope of the present invention.

Claims

1. A method for creating a motion-adjusted image of a patient to guide an interventional medical procedure, the method include: acquiring a first plurality of images of a patient having non-contrast enhanced vasculature; acquiring a second plurality of images of the patient having contrast-enhanced vasculature; generating a static roadmap of the patient's vasculature using the first plurality of images and the second plurality of images; generating a motion model of the patient using the first plurality of images and the second plurality of images; acquiring a third plurality of images of the patient using an interventional medical device deployed in the patient; generating motion tracking data of one of the patient or the invasive medical device using the third plurality of images; generating a motion transform using the motion tracking data and the motion model; displaying the static roadmap and the third plurality of images to show the interventional medical device aligned on the static roadmap using the motion transform, wherein the alignment of the interventional medical device on the static roadmap is based on a user selection of one of: motion compensation of the invasive medical device relative to the static roadmap to produce a plurality of images that do not show patient motion; and The static roadmap is adjusted relative to the motion of the invasive medical device to produce a plurality of images showing patient motion.

2. The method of claim 1, further comprising extracting the invasive medical device from the third plurality of images using a neural network. 3 . The method of claim 1 , further comprising generating the motion model by determining curvilinear features using the first plurality of images. 4 . The method of claim 1 , further comprising generating the motion model by determining curvilinear features within a third plurality of images.

5. The method of claim 1, further comprising determining a respiratory state of the patient using the third plurality of images, and determining a motion adjustment reflected in the motion transform using the motion model and the respiratory state.

6. The method of claim 5, wherein determining the respiratory state of the patient comprises calculating a one-dimensional average brightness using the third plurality of images.

7. The method according to claim 6, in, The respiratory state of the patient is determined using a transform configured to generate a translation vector for one or more pixels in the third plurality of images.

8. The method of claim 1, wherein at least one of the first plurality of images, the second plurality of images, and the third plurality of images comprises a three-dimensional image.

9. The method of claim 1, wherein motion compensation of the interventional medical device relative to the static roadmap that produces multiple images that do not show patient motion does not present a static display relative to the user, the static display only showing movement of the interventional medical device due to advancement by the user.

10. The method of claim 1, wherein the static roadmap that generates multiple images showing patient motion is adjusted relative to motion of the interventional medical device to present a dynamic display relative to a user, the dynamic display showing movement of the patient and the interventional medical device due to physiological motion and propulsion of the interventional medical device by the user.

11. A fluoroscopic imaging system, the fluoroscopic imaging system include: an X-ray source assembly coupled at one end and an X-ray detector array assembly coupled at another end; a computer system configured to control the x-ray source assembly and the x-ray detector array assembly to acquire a first plurality of images of a patient having non-contrast enhanced vasculature; controlling the x-ray source assembly and the x-ray detector array assembly to acquire a second plurality of images of the patient having contrast enhanced vasculature; generating a static roadmap of the patient's vasculature using the first plurality of images and the second plurality of images; generating a motion model of the patient using the first plurality of images and the second plurality of images; controlling the X-ray source assembly and the X-ray detector array assembly to acquire a third plurality of images of the patient using an invasive medical device deployed in the patient's body; picture; generating motion tracking data of one of the patient or the invasive medical device using the third plurality of images; generating a motion transform using the motion tracking data and the motion model; displaying the static roadmap and the third plurality of images to show the interventional medical device aligned on the static roadmap using the motion transform, wherein the alignment of the interventional medical device on the static roadmap is based on one of the following user selections: indivual: The motion compensation of the interventional medical device relative to the static roadmap to produce multiple images that do not show patient motion; and The static roadmap is adjusted relative to the motion of the interventional medical device. Adjust to produce multiple images showing patient motion.

12. The system of claim 11, wherein the computer system is further configured to extract the interventional medical device from the third plurality of images using a neural network.

13. The system of claim 11, wherein the computer system is further configured to generate the motion model by determining curvilinear features using the first plurality of images.

14. The system of claim 11, wherein the computer system is further configured to generate the motion transform by determining curvilinear features within a third plurality of images.

15. The system of claim 11, wherein the computer system is further configured to determine a respiratory state of the patient using the third plurality of images, and to determine a motion adjustment reflected in the motion transform using the motion model and the respiratory state.

16. The system of claim 15, wherein the computer system is further configured to calculate a one-dimensional average brightness using the third plurality of images to determine the respiratory state of the patient.

17. The system of claim 16, wherein the computer system is further configured to use a transform configured to generate a translation vector for one or more pixels within the third plurality of images to determine the respiratory state of the patient.

18. The system of claim 11, wherein at least one of the first plurality of images, the second plurality of images, and the third plurality of images comprises a three-dimensional image.

19. The system of claim 11, wherein motion compensation of the interventional medical device relative to the static roadmap that produces multiple images that do not show patient motion does not present a static display relative to the user, the static presentation only showing movement of the interventional medical device due to advancement by the user.

20. The system of claim 11, wherein the static roadmap that generates multiple images showing patient motion is adjusted relative to motion of the interventional medical device to present a dynamic display relative to a user, the dynamic display showing movement of the patient and the interventional medical device due to physiological motion and propulsion of the interventional medical device by the user.

21. A method of creating a motion-adjusted image of a patient to guide an interventional medical procedure, the method include: acquiring a plurality of images of a patient having contrast-enhanced vasculature; generating a static roadmap of the patient's vasculature using the plurality of images; generating a motion model of the patient using the plurality of images; acquiring another plurality of images of the patient using an interventional medical device deployed within the patient; generating motion tracking data of one of the patient or the invasive medical device using the another plurality of images; generating a motion transform using the motion tracking data and the motion model; displaying the static roadmap and the another plurality of images to show the interventional medical device aligned on the static roadmap using the motion transform, wherein the alignment of the interventional medical device on the static roadmap is based on a user selection of one of: motion compensation of the invasive medical device relative to the static roadmap to produce a plurality of images that do not show patient motion; or The static roadmap is adjusted relative to the motion of the invasive medical device to produce a plurality of images showing patient motion.

Images