Methods and systems for dynamic coronary roadmapping
The method generates dynamic coronary roadmaps using synchronized ECG and deep learning to compensate for cardiac and respiratory motions, addressing accuracy and safety issues in PCI procedures by reducing contrast agent use and radiation exposure.
Patent Information
- Application Number
- JP2024135170
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-01-11
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Current image-guided navigation methods in minimally invasive procedures like PCI suffer from limited accuracy due to subjective operator evaluation and reduced image quality, leading to challenges in visualizing coronary arteries without excessive contrast agent use, which can harm patients and expose them to radiation.
A computer-implemented method for generating dynamic coronary roadmaps by creating multiple roadmaps of coronary arteries across cardiac cycles, aligning them with real-time fluoroscopic images using ECG synchronization and deep learning-based tracking to compensate for cardiac and respiratory motions, allowing continuous visualization without contrast agents.
Enhances the accuracy and safety of PCI procedures by providing real-time, motion-compensated coronary artery visualization, reducing patient exposure to contrast agents and radiation, and improving procedural guidance.
Smart Images

Figure 2024161465000001_ABST
Abstract
Description
[Technical field]
[0001] The present application relates to the technical field of medical imaging, in particular percutaneous interventions, but may also find application in any field in which it is necessary to provide a dynamic roadmap in image sequences in which structures of interest are not well visible.
[0002] The trend towards minimally invasive has increased the importance of imaging in clinical interventions. Because the incisions made for interventions are small, clinicians can no longer use direct visual inspection to manipulate their tools and must instead rely on intraprocedural images generated from real-time imaging modalities such as x-ray angiography and fluoroscopy, ultrasound echo, and intraprocedural magnetic resonance imaging. Currently, x-ray angiography is the primary imaging modality used during the treatment of cardiovascular disease with minimally invasive procedures, also known as percutaneous coronary intervention (PCI). X-ray angiographic images are used to guide interventional procedures, for example, to treat atherosclerosis. Atherosclerosis is the buildup of cholesterol and fatty deposits (called plaque) on the inner walls of arteries. These plaques can restrict blood flow to the heart muscle by physically clogging the arteries or by causing abnormalities in arterial tone and function. Without an adequate blood supply, the heart becomes starved of oxygen and vital nutrients it needs to function properly. This can result in chest pain called angina. If the blood supply to a portion of the heart muscle is completely blocked, or if the heart's energy demands become much greater than the blood supply, a heart attack (damage to the heart muscle) or stroke can occur. Atherosclerosis is treated using a variety of methods in the arteries of the heart, head, neck, and peripheral parts of the body. The most common methods, such as angioplasty, bare metal stenting, drug-eluting stenting (permanently implanted and biodegradable), various types of energy delivery, and rotational atherectomy, all treat the artery evenly around the targeted length of the arterial lumen.
[0003] During PCI, the (interventional) cardiologist passes a deflated balloon or other device on a catheter through the blood vessels from the inguinofemoral or radial artery until it reaches the site of the artery blockage. PCI usually involves inflating the balloon to open the artery, with the goal of restoring unobstructed blood flow. A stent or scaffold can be placed at the site of the blockage to keep the artery open. During these procedures, medical devices inserted by a guiding catheter are advanced to treat, for example, a coronary artery stenosis. The guiding catheter is first placed at the mouth of the coronary artery. The guiding catheter introduces a balloon catheter carrying a stent over a guidewire to the location of the stenosis. PCI is usually performed under image guidance using x-ray angiography. The coronary arteries are visualized by x-ray opaque contrast material. Because the opacity of the coronary arteries only lasts for a short period of time, the interventional cardiologist may repeatedly inject contrast material to visualize the blood vessels during the procedure. Periprocedural contrast usage correlates with operator experience, procedural complexity, renal function, and imaging settings. Furthermore, the risk of contrast-induced nephropathy is also related to contrast volume. However, the manipulation of guidewires and materials is usually performed without continuous injection of contrast. In this situation, device navigation is guided by “vessel-free” fluoroscopic images. The cardiologist must imaginatively reconstruct the location of the vessels and stenosis based on the previous angiogram.
[0004] For example, in minimally invasive interventions such as PCI, image guidance and visualization can cause a bottleneck in the physician-instrument-patient feedback loop. Image-guided navigation, which is performed by visual inspection of intraprocedural images, inherently suffers from subjective operator assessment, limiting accuracy and operator bias. This problem is even more pronounced when navigating moving regions such as the thorax. Furthermore, intraprocedural images generated by real-time imaging modalities usually have poor image quality and do not always represent the anatomical structures relevant to the clinical procedure. Contrast agents can be used to visualize anatomical structures. However, to prevent harm to the patient, intraprocedural administration of contrast agents, including additional x-ray radiation, needs to be limited. For example, it is desirable to reduce the time and amount of x-ray exposure to the patient as well as the amount of potentially toxic contrast agents injected into the patient's bloodstream. Most of these contrast agents are used to highlight the aorta and coronary arteries in fluoroscopic images to visually guide the physician. For example, if contrast is injected during PCI, the coronary tree will be visible, and if contrast is not injected, the coronary tree will not be visible. Dynamic coronary road mapping is a promising solution to improve visual feedback and reduce contrast use during PCI. To develop a dynamic coronary road mapping system, it is important to accurately superimpose the coronary road map onto the fluoroscopic image, but this is challenging because there is limited vascular information in the target fluoroscopic image to infer the compensation of vascular motion due to the patient's breathing and heartbeat. Proposed methods for motion compensation in dynamic coronary road mapping can generally be divided into two categories: direct road mapping methods and model-based approaches.
[0005] Direct roadmapping methods use information from X-ray images and ECG signals to directly correct for motions due to breathing and heartbeat. For example, US Pat. No. 6,299,433 uses digital subtraction of contrast and mask sequences to create a full cardiac cycle of a coronary artery roadmap. This roadmap is stored and later synchronized to a live fluoroscopy sequence by registering it with the R-wave of the corresponding ECG signal. This system corrects for cardiac motion of the vessels, but does not correct for respiratory motion and patient motion during the intervention. Furthermore, this must be performed with a cardiac-gated frame. US Pat. No. 6,299,433 discloses an X-ray diagnostic device for subtraction angiography. It utilizes a single mask frame and integrates it over a full cycle. By integrating the mask over the entire cardiac cycle, position displacement errors are simply distributed and not minimized.
[0006] Unlike direct roadmapping methods, model-based approaches construct a model to predict the motion in the fluoroscopy frame. This motion model is often one or more functions that relate the roadmap motion to a surrogate signal derived from an image or an ECG, so that once the surrogate of the fluoroscopy frame is acquired, the motion can be calculated by the motion model. For cardiac interventions, including PCI, organ motion is primarily affected by respiration and cardiac motion. Previous techniques, such as those taught by "ECG-derived cardiac signals and respiratory signals obtained from diaphragm tracking," have constructed motion models parameterized by ECG-derived cardiac signals and diaphragm tracking. A limitation of model-based approaches is that the motion model is patient-specific, which requires training the model for each new subject. Furthermore, when the surrogate values being inferred fall outside the surrogate range that constitutes the model (e.g., when the motion is abnormal), extrapolation is required, which prevents accurate motion compensation.
[0007] Therefore, it is desirable to track the motion of the coronary artery tree in fluoroscopic images without contrast enhancement to reduce the aforementioned drawbacks. The use of this method could better assist cardiologists during PCI and reduce patient exposure to contrast and x-ray radiation. [Prior art documents] [Patent documents]
[0008] [Patent Document 1] U.S. Pat. No. 4,878,115 [Patent Document 2] European Patent No. 0,193,712 [Patent Document 3] U.S. Pat. No. 9,256,936 [Patent Document 4] U.S. Patent Application Publication No. 16 / 379,248 [Patent Document 5] European Patent Application Publication No. 3206183 [Patent Document 6] U.S. Patent No. 10,229,516 [Patent Document 7] U.S. Patent Application Publication No. 16 / 438,955 [Patent Document 8] U.S. Patent Application Publication No. 16 / 256,793 [Patent Document 9] U.S. Patent No. 9,576,360 [Patent Document 10] U.S. Patent No. 10,192,352 [Non-patent literature]
[0009] [Non-Patent Document 1] Shechter et al., "Prospective Motion Correction of X-Ray Images for Coronary Interventions," IEEE Transactions on Medical Imaging 2005:24, 441-450. [Non-Patent Document 2] Frangi et al., "Multiscale vessel enhancement filtering," Medical Image Computing and Computer-Assisted Intervention - MICCAI 1998 Lecture Notes in Computer Science 1496 / 1998:130. [Non-Patent Document 3] Ma et al., "Fast forward detection of contrast inflow in X-ray angiograms using convolutional and recurrent neural networks," International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 453-461 [Non-Patent Document 4] Li et al., "Gray-scale image skeletonization from incomplete boundaries," Proceedings of the International Conference on Image Processing, ICIP 2008, October 12-15. [Non-Patent Document 5] Maintz et al., "Overview of Medical Image Registration Methods," Symposium of the Belgian Association of Hospital Physicists, 1996. [Non-Patent Document 6] Hao et al., "Vessel Layer Segmentation in X-Ray Angiograms Using Fully Convolutional Networks," in Proc. SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling. [Non-Patent Document 7] Ma et al., "Layer Separation for Vessel Enhancement in Interventional X-Ray Angiography Using Morphological Filtering and Robust PCA," Workshop on Augmented Environments for Computer-Assisted Interventions 2017, Springer pp. 104-113 [Non-Patent Document 8] Ma et al., "Automatic online layer separation for vessel enhancement in x-ray angiograms for percutaneous coronary intervention," Med Image Anal. 2017 Jul;39:145-161 [Non-Patent Document 9] Zang et al., "Catheter Attention Machine for Occlusion Landmark Detection in 2D X-ray Angiograms," Jan 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) [Non-Patent Document 10] Ambrosini et al., "Fully Automatic and Real-Time Catheter Segmentation in Fluoroscopy," International Conference on Medical Image Computing and Computer-Assisted Intervention 2017, Springer. pp. 577-585 [Non-Patent Document 11] Nasr-Esfahani et al., "Vessel Extraction in X-Ray Angiograms Using Deep Learning," Conf Proc IEEE Eng Med BiolSoc 2016 Aug;2016:643-646 [Non-Patent Document 12] Girasis et al., "Advances in two-dimensional quantitative coronary angiographic assessment of bifurcation lesions: improved small lumen diameter detection and automated reference vessel diameter derivation," EuroIntervention 2012 Mar; 7(11):1326-35 [Non-Patent Document 13] Wang et al., "Vessel Segmentation in Coronary X-Ray Angiography," Conf Proc IEEE Eng Med Biol Soc. 2005; 2: 1584-1587. [Non-Patent Document 14] Kim et al., "Real-time fluoroscopic to angiographic registration for image-guided percutaneous coronary intervention," International journal of computer assisted radiology and surgery 2018:13, 203-213 [Non-Patent Document 15] Arulampalam et al., "A Tutorial on Particle Filters for Online Nonlinear / Non-Gaussian Bayesian Tracking," IEEE Transactions on signal processing 2002:50, 174-188. [Non-Patent Document 16] Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical image computing and computer-assisted intervention 2015, Springer. pp. 234-241 [Non-Patent Document 17] Milletari et al., "V-net: A fully convolutional neural network for volumetric medical image segmentation," 2016 Fourth International Conference on 3D Vision (3DV), IEEE. pp. 565-571. [Non-Patent Document 18] He et al., "Deep residual learning for image recognition," 2016 Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. [Non-Patent Document 19] Laina et al., "Simultaneous segmentation and localization for tracking of procedural instruments," 2017 International conference on medical image computing and computer-assisted intervention, Springer. pp. 664-672 [Non-Patent Document 20] Farneback et al., "Two-frame motion estimation based on polynomial expansions," Scandinavian conference on Image analysis 2003, Springer. pp. 363-370. [Non-Patent Document 21] Kingsma et al., "Adam: A Probabilistic Optimization Method," International Conference on Learning Representations, ICLR 2015. [Non-Patent Document 22] Lesage et al., "Bayesian Maximal Path for Coronary Artery Segmentation from 3D CT Angiograms," MICCAI 2009, Part 1, LNCS 5761, pp 222-229 [Non-Patent Document 23] Kirissli et al., "A standardized evaluation framework for evaluating coronary stenosis detection, stenosis quantification, and lumen segmentation algorithms in computed tomography angiography," Medical Image Analysis, vol. 17, no. 8, pp. 859-876, 2013. [Non-Patent Document 24] Dey et al., "Dimensional Quantification of Noncalcified and Calcified Coronary Artery Plaque from Coronary CT Angiograms," Cardiovasc Comput Tomogr. 2009;3(6):372-382 [Non-Patent Document 25] Gerretsen et al., "Visualization of coronary artery wall atherosclerosis in asymptomatic subjects and patients with coronary artery disease using magnetic resonance imaging," PLoS One. 2010 Sep 29;5(9) [Non-Patent Document 26] Adame et al., "Automated Segmentation and Plaque Characterization in Atherosclerotic Carotid Artery MR Images," Magnetic Resonance Materials in Physics, Biology and Medicine 2004; 16(5):227-234. [Non-Patent Document 27] Wentzel et al., "Geometry-guided data averaging enables interpretation of shear stress-related plaque development in human coronary arteries," Journal of Biomechanics 2005, pp. 1551-1555. [Non-Patent Document 28] Lay, "Linear Algebra and Its Applications", 2012, 4th edition, p142-143, Addison-Wesley Longman [Non-Patent Document 29] Baka et al., "3D+t / 2D+t CTA-XA Registration Using Population-Based Motion Estimation," in Demirci, Lee, Radeva, Unal (eds): MICCAI-STENT 2012, pp 64-71 [Non-Patent Document 30] Chen et al., "Kinematic and deformation analysis of 4D coronary artery trees reconstructed from cine angiograms," IEEE Transactions on medical imaging, Vol. 22, No. 6, June 2003, pp 710-721. [Non-Patent Document 31] Zheng et al, "Sequential reconstruction of vascular skeleton from X-ray coronary angiography sequences," Computerized Medical Imaging and Graphics 34(2010)333-345Zheng et al, "Sequential reconstruction of vascular skeleton from X-ray coronary angiography sequences," Computerized Medical Imaging and Graphics 34(2010)333-345 [Non-Patent Document 32] Baka et al., "Statistical Coronary Artery Motion Model for 2D+t / 3D Registration of X-ray Coronary Angiography and CTA," Medical Image Analysis, Volume 17, Issue 6, August 2013, Pages 698-709 [Non-Patent Document 33] Dibildox et al., "3D / 3D Registration of Coronary CTA and Biplane XA Reconstructions for Improved Image Guidance," MedPhys 2014 September; 41(9) [Non-Patent Document 34] Baka et al., "Oriented Gaussian Mixture Models for Nonrigid 2D / 3D Coronary Artery Registration," Med Phys. 2014 Sep;41(9) [Non-Patent Document 35] Girasis et al., "Advanced 3D quantitative coronary angiographic assessment of bifurcation lesions: methodology and phantom validation," IEEE Trans Med Imaging. 2017 Mar;36(3):757-768 [Non-Patent Document 36] Ambrosini et al., "Hidden Markov Model for 3D Catheter Tip Tracking with 2D X-ray Catheter Sequences and 3D Rotational Angiograms," IEEE Trans Med Imaging. 2017 Mar;36(3):757-768 [Non-Patent Document 37] Ma et al., "PCA-derived respiratory motion surrogate from x-ray angiograms for percutaneous coronary intervention," Int J Comput Assist Radiol Surg. 2015 Jun;10(6):695-705 [Non-Patent Document 38] Rohr et al., "Landmark-based elastic registration using approximate thin-plate splines," IEEE Trans Med Imaging. 2001 Jun;20(6):526-34 [Non-Patent Document 39] Metz et al., "Coronary Artery Models from ECG-Gated CTA Data for Intraoperative Dynamic Registration of X-Ray Images and CTA," Med Image Comput Comput Assist Interv. 2009;12(Pt 1):369-76
[0010] It is therefore an object of embodiments herein to provide a computer-implemented method for dynamically visualizing information from first image data of a target object together with second image data of the target object, the method comprising the steps of: i) using the first image data to generate a plurality of roadmaps of the object of interest; ii) determining a plurality of reference locations of a device, such as a guide catheter, guidewire or other intraluminal device or instrument, within the first image data, the plurality of reference locations corresponding to the plurality of roadmaps of the object of interest of i); iii) selecting a roadmap from the plurality of roadmaps; iV) determining a location of the device within the second image data; V) using the reference position of the device corresponding to the one roadmap selected in iii) and the position of the device determined in iV) to transform the one roadmap selected in iii) to generate a dynamic roadmap of the object of interest; and vi) overlaying the visual representation of the dynamic roadmap of the object of interest generated in V) onto the second image data for display.
[0011] Related systems and program storage devices are also described and claimed.
[0012] In an embodiment, the first image data covers at least one cardiac cycle of the patient, and the generated multiple roadmaps of the object of interest cover different phases of the patient's cardiac cycle.
[0013] In an embodiment, a phase of the patient's cardiac cycle is offset in time relative to a predefined portion of the patient's cardiac cycle.
[0014] In embodiments, the methods may further include, for example, processing an ECG signal synchronous with the second image data, acquired along with the second image data, to determine a phase of the patient's cardiac cycle corresponding to the second image data. The roadmap may be selected iii) by matching a phase of the patient's cardiac cycle for the second image data to a phase of the patient's cardiac cycle for the selected roadmap.
[0015] In embodiments, the methods may further include processing the first image data to determine a phase of the patient's cardiac cycle for the image frame and associating the phase of the cardiac cycle with a roadmap corresponding to the image frame.
[0016] In an embodiment, the multiple roadmaps of the target object can include multiple two-dimensional roadmaps or multiple three-dimensional roadmaps. For example, the multiple three-dimensional roadmaps can be derived from a three-dimensional model of the target object. In another example, the multiple three-dimensional roadmaps can be derived from two X-ray angiographic image sequences of the target object acquired with a contrast agent. In yet another example, the multiple three-dimensional roadmaps can be derived from a three-dimensional model of the target object and a X-ray angiographic image sequence of the target object acquired with a contrast agent.
[0017] In an embodiment, multiple roadmaps may include information characterizing properties of a target object (such as a centerline, a contour, and / or an image mask).
[0018] In embodiments, the multiple roadmaps can include at least one measurement for the object of interest, for example, the at least one measurement can be selected from the group consisting of location and extent of vascular occlusion, diameter and area, pressure, blood flow velocity, fractional flow, wall shear stress, vascular curvature, anterior foreshortening, location and extent, type of coronary plaque, location and extent of total coronary occlusion, or location and extent of coronary occlusion of the object of interest.
[0019] In embodiments, the location of the device in the second image data can be determined using a Bayesian filtering method. In one example, the Bayesian filtering method can use a deep learning network to estimate the likelihood of the location of the device in the second image data. In another example, the Bayesian filtering method can be configured to set the location of the device in the second image data equal to a weighted arithmetic average of multiple locations and their associated weights. In another example, the Bayesian filtering method can be configured to resample points around locations with high weight values.
[0020] In an embodiment, operation V) applies a transformation to the selected roadmap to compensate for motion between the first image data and the second image data, for example motion can include respiratory motion and / or cardiac motion and / or patient motion and / or table motion.
[0021] In an embodiment, the transformation applied to the selected roadmap can be a rigid or non-rigid transformation based on the displacement obtained from the reference position of the device corresponding to the selected roadmap and the position of the device in the second image data.
[0022] In an embodiment, the visual representation of the dynamic roadmap can be generated by projecting a superposition of the dynamic roadmap onto the second image data using a transparent mode and / or by dilating the dynamic roadmap and projecting the boundaries of the resulting dynamic roadmap such that the visual representation of the dynamic roadmap is configured not to obscure equipment used to process the target object.
[0023] In embodiments, the reference points may be stored as part of roadmaps for the target object.
[0024] In embodiments, the selected roadmap can be a three-dimensional roadmap that is transformed to generate at least one dynamic roadmap for overlaying onto the second image data. For example, the three-dimensional roadmap can be transformed according to a viewpoint (e.g., viewing angle) used to collect the second image data, thereby allowing the second image data to be collected from a different viewpoint than the first image data.
[0025] In other embodiments, the particular roadmap can be a two-dimensional roadmap that is transformed to generate at least one two-dimensional dynamic roadmap for overlaying onto second image data, where the first image data and the second image data can be collected from a common viewpoint.
[0026] In an embodiment, the first image data may be derived by subtraction of a baseline image, and the second image data may be obtained by subtraction of a baseline image.
[0027] In an embodiment, the first image data is angiography image data collected using an X-ray imaging modality with a contrast agent, and the second image data is fluorescence image data collected using an X-ray imaging modality without a contrast agent.
[0028] In an embodiment, operations iii) to vi) may be repeated for successive frames of a live image sequence acquired without the use of contrast agent.
[0029] In an embodiment, for dynamic guidance, a roadmap can be selected for each image frame in successive image frames of the second image data for transformation and registration within the image frames of the second image data, while in another embodiment, for static guidance, one roadmap (the same roadmap) can be selected for successive image frames of the second image data for transformation and registration within the image frames of the second image data.
[0030] In embodiments, the device is selected from the group consisting of a guiding catheter, a guidewire, or other intraluminal device or instrument.
[0031] In an embodiment, this location of the device corresponds to the end or tip of the device.
[0032] In an embodiment, the object of interest is a heart, a portion of the coronary artery tree, a blood vessel, or other part of the vascular system.
[0033] In embodiments, the methods may further include displaying a superimposition of a visual representation of a dynamic roadmap of the object of interest onto the image data of iV).
[0034] Embodiments herein also relate to a method for visualizing information in image data of an object of interest of a patient, the method comprising the steps of: i) acquiring first image data of the object of interest, the first image data being acquired with a contrast agent and an interventional device being present in the first image data; ii) using the first image data to generate a plurality of roadmaps of the object of interest; iii) determining a plurality of reference locations of the device in the first image data, the plurality of reference locations corresponding to the plurality of roadmaps of the target object; iV) acquiring second image data of the object of interest, the second image data being collected without a contrast agent and the device being present in the second image data; V) selecting a roadmap from the plurality of roadmaps for the target object; vi) determining a location of the device within the second image data; vii) using the reference position of the device corresponding to the one roadmap selected in V) and the position of the device determined in vi) to transform the one roadmap selected in V) to generate a dynamic roadmap of the target object; and viii) overlaying a visual representation of the dynamic roadmap of the object of interest generated in vii) onto the second image data for display.
[0035] Embodiments herein also relate to a system for generating an image of a target object of a patient, the system including at least one processor configured to perform operations of a method according to embodiments herein when executing program instructions stored in a memory.
[0036] In one embodiment, the at least one processor performs the following steps: i) acquiring first image data of the object of interest, the first image data being acquired with a contrast agent and an interventional device being present in the first image data; ii) using the first image data to generate a plurality of roadmaps of the object of interest; iii) determining a plurality of reference locations of the device in the first image data, the plurality of reference locations corresponding to the plurality of roadmaps of the target object; iV) acquiring second image data of the object of interest, the second image data being collected without a contrast agent and the device being present in the second image data; V) selecting a roadmap from the plurality of roadmaps for the target object; vi) determining a location of the device within the second image data; vii) using the reference position of the device corresponding to the one roadmap selected in V) and the position of the device determined in vi) to transform the one roadmap selected in V) to generate a dynamic roadmap of the target object; and viii) overlaying a visual representation of the dynamic roadmap of the object of interest generated in vii) onto the second image data for display; is configured to run
[0037] In an embodiment, the system further comprises an imaging acquisition subsystem, typically using an X-ray imaging modality, configured to acquire the first image data and the second image data.
[0038] In an embodiment, the system further comprises a display subsystem configured to display a superposition of the visual representation of the dynamic roadmap of the object of interest generated in viii) onto the second image data.
[0039] Further improvements are the object of the dependent claims. [Brief description of the drawings]
[0040] [Figure 1] 1 illustrates a flowchart of a method for determining a dynamic coronary roadmap according to embodiments herein. [Figure 2a] FIG. 1 illustrates a functional block diagram of an exemplary single plane angiography system according to embodiments herein. [Figure 2b] 1 shows a functional block diagram of an X-ray system connected to a system including an embodiment of the present disclosure. [Diagram 3] 1 shows a diagram of the dynamic roadmap process. [Figure 4] 4 shows the read out angiographic image data. [Diagram 5] 1 shows a flowchart of a method for acquiring a device position within an X-ray angiography image sequence. [Figure 6] 1 illustrates an example of a method for detecting the start of a contrast injection frame in an X-ray image sequence. [Figure 7] Several approaches for extracting the cardiac cycle in successive X-ray image frames are presented. [Figure 8] 1 shows an example of layer separation in an X-ray angiography frame. [Figure 9] 1 shows an example of several fluoroscopic image frames and their acquisition times relative to the cardiac cycle within a fluoroscopic image sequence. [Figure 10a] 1 shows an example of dynamic load mapping. [Figure 10b] 13 illustrates another example of dynamic load mapping. [Figure 11] Here are some examples of labels used to train a deep learning network: [Figure 12] 1 shows a joint segmentation and detection network for catheter tip detection. [Figure 13] This paper summarizes an algorithm for tracking the catheter tip using a deep learning-based Bayesian filtering method. [Figure 14] Four examples of deep learning-based Bayesian filtering catheter tip tracking methods are presented. [Figure 15] 1 shows a flowchart of a method for determining a dynamic coronary roadmap in which image enhancement is used to improve identification and processing of objects of interest. [Figure 16] 1 shows an example of image enhancement using differential processing. [Figure 17] FIG. 2 illustrates an example of an X-ray cine-fluoroscopy unit block diagram according to embodiments herein. [Figure 18] 1 shows a flowchart of a method for determining a dynamic 3D coronary artery roadmap. [Figure 19] 13 shows some examples of generating 3D models based on 3D angiography image data. [Figure 20] An example of a coronary vessel with a WSS superimposed on the luminal surface is shown. [Figure 21]1 shows an example of a coronary vessel with pressure and fractional flow reserve superimposed on the luminal surface. [Figure 22a] 1 shows guidance maps of surrounding vessel foreshortening, overlap, and in-plane coronary artery motion for one cardiac phase. [Figure 22b] 13 shows guidance maps for in-plane coronary artery motion including near-far foreshortening, surrounding vessel overlap, and multiple cardiac phases. [Diagram 23] 1 shows a guidance map for selecting an optimal second projection relative to a first projection. [Figure 24] 1 shows some examples of single 3D roadmap frames. [Diagram 25] Two examples of 3D segmentation of the coronary artery lumen are shown, including segmentation of coronary plaque. [Figure 26] 13 shows a flow chart of an alternative method for determining a dynamic coronary roadmap. [Figure 27] 1 shows an illustration of the realignment step. [Figure 28] 1 shows a flow chart of a method for determining the location and amount of calcified plaque. [Figure 29] 1 shows a typical example of an X-ray angiography image sequence in which contrast agent is administered during the sequence. [Diagram 30] A high level method for determining the location and amount of calcified plaque is shown. [Diagram 31] Shows that the scaffold is resolved into separate vascular branches. [Diagram 32] An example is shown in which calcified plaque is highlighted. [Diagram 33] 1 illustrates a setup in which operations are performed according to embodiments herein by a processor unit of a system connected to an X-ray system. [Figure 34a] An example of a roadmap overlaid on X-ray fluoroscopic image data is shown. [Figure 34b] An example of the roadmap overlaid on X-ray angiography image data is shown. [Diagram 35]Shows integration of device location into roadmap. [Diagram 36] 13 shows an example of tracking results of the alternative proposed method on an example image frame. [Figure 37] A proposed tracking method is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0041] This application describes a method and system to provide a real-time dynamic overlay or dynamic coronary roadmap that can be superimposed on a live fluoroscopy or angiography image stream / sequence, thereby supporting clinicians to improve patient care. This approach compensates for changes in vessel geometry and cardiac motion by selecting roadmaps of the same cardiac phase with time registration, and corrects for respiratory induced motion and patient motion by tracking the device.
[0042] In the following method and system description, X-ray angiography will be used to mean the acquisition of X-rays of an object of interest (e.g., a portion of the vasculature or the coronary vascular tree) after administration of a contrast agent resulting in visualization of the object of interest and other objects that are radiopaque. Fluoroscopy means acquisition of X-ray images without the use of a contrast agent, and thus does not include enhancement of the visualization of the object of interest. The terms image or image frame mean a single image, and the terms image sequence or image stream mean multiple images acquired over time. An X-ray angiography image sequence may include administration of a contrast agent, and image frames preceding the administration of the contrast agent do not include contrast agent and therefore do not include enhancement of the vasculature. An X-ray angiography image sequence may comprise multiple frames covering one or more phases of the cardiac cycle. Throughout this patent application, a cardiac cycle is defined as a period that is specific to a patient and covers one heartbeat of the patient. A cardiac cycle can be defined as the period between the peaks of successive R waves in the patient's ECG signal. A phase means an instant (or period) within the patient's cardiac cycle. The phase can be measured as the offset from the apex of the R wave in the patient's ECG signal, as shown in FIG. 9 and the corresponding description.
[0043] Within this application, a method is proposed for generating and displaying a dynamic overlay or dynamic roadmap as an overlay onto a live X-ray image stream, which consists of the following elements: Construct a model of the target organ from angiographic image sequences of the target organ (such as a coronary artery tree). · Use the patient's electrocardiogram (ECG) measurements to select portions (or frames) of the model that are aligned to the patient's cardiac phase. Update or transform selected parts of a model to compensate for respiratory motion in a patient's live fluoroscopic or angiographic image stream using guiding catheter tracking. Rendering portions of the resulting model and integrating the rendered view of the model as an overlay with the patient's live fluoroscopy or angiography image stream. The described dynamic coronary roadmapping method is performed in real time using a graphics processing unit (GPU) and can therefore be used during PCI in a real clinical setting.
[0044] 1 shows a flow chart illustrating operations according to one embodiment of the present application using an imaging system capable of acquiring and processing a two-dimensional image sequence of a vascular organ (or a portion thereof) or other object of interest. Single-plane or double-plane angiography systems such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD) can be used.
[0045] FIG. 2a is a functional block diagram of an exemplary single plane angiography system. The system includes an angiography imager 212 that operates under commands from a user interface module 216 and will provide data to a data processing module 210. The single plane angiography imager 212 captures two-dimensional X-ray image sequences of a vascular organ of interest, for example, in an occipital-frontal direction. The single plane angiography imager 212 typically includes an X-ray source and detector pair mounted on the arms of a supporting gantry. The gantry allows the X-ray source and detector arms to be positioned at various angles relative to a patient supported on a table between the X-ray source and detector. The data processing module 210 can be realized by a personal computer, a workstation, or other computer processing system. The data processing module 214 processes the two-dimensional image sequences captured by the single plane angiography imager 212 to generate data as described herein. The user interface module 216 interacts with a user and communicates with the data processing module 210. The user interface module 216 may include a display screen for visual output, a touch screen for touch input, a mouse pointer or other pointing device for input, a microphone for audio input, a speaker for audio output, a keyboard and / or keypad for input, etc. The data processing module 210 and the user interface module 216 cooperate to perform the operations described in Figure 1, Figure 15, Figure 18, Figure 26, or Figure 28 below.
[0046] The operations of Figure 1, 15, 18, 26 or 28 may also be performed by software code embodied in a computer product (e.g., an optical disk or other form of persistent memory, a USB drive or a network server). The software code may be directly loaded into the memory of a data processing system to perform the operations of Figure 1, 15, 18, 26 or 28. Such a data processing system may also be physically separate from the angiography system used to acquire images using any type of data communication as input to acquire such images.
[0047] In this example, it is assumed that the imaging system has already acquired and stored at least one two-dimensional image sequence of the object of interest. Any imaging device capable of providing two-dimensional angiographic image sequences can be used for this purpose. For example, biplane or single-plane angiographic systems such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD) can be used.
[0048] 2b is an exemplary alternative functional block diagram according to an embodiment herein including an X-ray system 202 operating under command from a user interface module and providing data to a data analysis module 203. The X-ray system 202 collects X-ray image data of a region of interest, such as a heart. The data analysis module 203 can be implemented by a personal computer, a workstation, or other computer processing system. The data analysis module 203 processes the collected X-ray image data of the X-ray system 202 to generate, for example, a coronary roadmap or a quantification of a coronary analysis. The user interface module 201 interacts with a user and communicates with the data analysis module 203. The user interface module 201 can include a display screen for visual output, a touch screen for touch input, a mouse pointer or other pointing device for input, a microphone for voice input, a speaker for voice output, a keyboard and / or keypad for input, etc. Module 204 represents a dynamic coronary roadmap representing a coronary artery tree calculated from the collected X-ray image data as described herein. The dynamic coronary artery roadmap is superimposed in real time onto the x-ray image sequence acquired by the x-ray system module 202. The module 204 can be implemented by a personal computer, workstation, or other computer processing system and interacts with the x-ray system 202.
[0049] An embodiment will now be disclosed with reference to FIG. 1. The operations shown therein can obviously be performed in any logical order and can be partially omitted. As it is the object of this application to provide a workflow that can be used during an intervention, reference will also be made to steps of an example workflow. In a preferred embodiment, the method of this application envisages a scenario in which dynamic coronary road mapping is performed to guide a PCI procedure. As shown in FIG. 1, the workflow comprises several steps representing an offline phase and an online phase. The offline phase, represented by block 100 in FIG. 1, creates a set of coronary road maps using X-ray angiography sequences acquired with injection of contrast over multiple cardiac phases. The online phase, represented by block 110 in FIG. 1, creates a dynamic road map based on tracking of the device's position using the set of road maps created in the offline phase in combination with X-ray angiography sequences acquired without injection of contrast. A visual representation of the dynamic road map is overlaid on a live fluoroscopic image stream of X-ray fluoroscopic image sequences to provide visual guidance to the physician in support of the placement of one or more medical devices inserted by a guiding catheter. For example, the medical instrument may be a guidewire, a dilatation balloon, a stent, or other suitable intravascular device or instrument.
[0050] Before describing each step in FIG. 1 in detail, a brief high-level description of the sub-steps of the offline and online phases will be provided.
[0051] Offline phase: An offline phase, represented by block 100 in FIG. 1, is performed before the actual road mapping is performed. In the offline phase, a roadmap of the coronary arteries over multiple phases of the cardiac cycle is created from an X-ray angiography image sequence covering multiple phases of the cardiac cycle. Typically, at least one cardiac cycle is considered. Along with the X-ray angiography sequence, cardiac cycle time information (e.g., ECG signal) is acquired and stored. The roadmap includes information characterizing the properties of the coronary arteries in a particular phase of the cardiac cycle. Such information can include centerlines of the coronary arteries, contours (lumen boundaries) of the coronary arteries over their respective lengths, and / or mask images (e.g., images covering the space occupied by the coronary arteries) representing the coronary arteries. In an embodiment, the roadmap can include vascular models in the form of centerlines, contours, and / or mask images representing the coronary arteries in a given phase of the cardiac cycle. The roadmap(s) can also include information of clinical interest, such as the location and percentage of vascular occlusion, diameter and area, pressure, blood flow velocity, shunt reserve, wall shear stress, vessel tortuosity, amount of distal foreshortening, location, amount and type of coronary plaque (e.g., calcification, soft plaque, mixed plaque), location and extent of total coronary occlusion, or location and extent of coronary occlusion. Creation of a roadmap will be described in detail in step 103 of FIG. 1.
[0052] The device position (e.g., catheter tip position) within the x-ray angiography sequence is obtained and this position is associated with the roadmap to serve as a reference point for roadmap transformation (as described in more detail by step 107 of FIG. 1). This reference point (device position) can be integrated into the roadmap as described in more detail by step 102 of FIG. 1.
[0053] Online Phase: During the online phase, represented by block 110 in FIG. 1, the dynamic road mapping is actually performed. In the online phase, a sequence of fluoroscopic image data is acquired (see step 104 in FIG. 1) without the use of contrast agents and preferably at the same viewing angles (C-arm angulation and C-arm rotation) as when the road map was created during the offline phase. At the same time, an ECG signal is acquired together with the fluoroscopic image data and compared to the stored ECG to select the road map (step 105 in FIG. 1) that best matches the one created during the offline phase. This is to compensate for changes in vessel shape and position between frames due to cardiac motion. At the same time, the device position (e.g., catheter tip position) in the acquired fluoroscopic image data is tracked using image processing techniques and, in a preferred embodiment, using the proposed deep learning-based Bayesian filtering method described in step 106 in FIG. 1. The displacement of the device location (e.g., catheter tip) between the current frame of fluoroscopic image data and the reference point (device location) associated with the selected roadmap is obtained and applied to transform the selected roadmap to generate a dynamic roadmap as described in step 107 of Figure 1. Finally, a visual representation of the dynamic roadmap is rendered and overlaid on the current frame of fluoroscopic image data to guide the PCI procedure (step 108 of Figure 1).
[0054] The dynamic roadmap process is illustrated in Fig. 3. In Fig. 3, the offline phase is represented by pictures 301 and 302, where 301 shows an X-ray angiography image sequence of the right coronary artery, and 302 shows the roadmap generated from the X-ray angiography sequence (301). During the online phase, the generated roadmap (302) is superimposed on the X-ray fluoroscopy image stream (303), as shown by pictures 303 and 304, resulting in a dynamic coronary roadmap (304). For illustrative purposes, only one frame (representing one instant in the cardiac cycle) is shown in Fig. 3.
[0055] In the following sections, the steps depicted in FIG. 1 are explained in more detail.
[0056] Step 101: Read angiographic image data In the first step, step 101, angiographic image data is read out. In a preferred embodiment, the angiographic image data represents the acquisition of an object of interest by X-ray angiography, which results in an X-ray angiographic image sequence. Single-plane or double-plane angiographic systems can be used, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD). The object of interest can be, for example, the heart, a part of the coronary tree, or a blood vessel, and the object of interest is visualized by contrast during the acquisition. The X-ray angiographic image sequence is acquired such that the object of interest is clearly visible. The C-arm can therefore be rotated and angulated by the clinician so that the best projection is obtained. Preferably, an electrocardiogram (ECG) is part of the angiographic image data and is simultaneously recorded during the acquisition of the X-ray angiogram. The ECG allows for the matching of cardiac phases between different X-ray angiographic frames or X-ray angiographic acquisitions. By cardiac phase matching, displacement of the object of interest due to cardiac motion between image frames within the same cardiac phase is minimized.
[0057] FIG. 4 provides an illustration of retrieved angiographic image data consisting of an X-ray angiographic image sequence including an ECG signal.
number
number
number
number
[0058] Step 102: Obtaining the device location within the angiographic image data There are many factors that can cause the object of interest or the roadmap to be displaced between successive X-ray image frames in an X-ray angiography image sequence or between different X-ray angiography image sequences. These factors are cardiac motion due to cardiac contraction, respiratory motion due to patient breathing, and patient motion due to patient movement. Cardiac motion can be compensated for by, for example, matching cardiac phases using the read-out ECG signal. Respiratory motion, including possible patient motion, can be corrected by identifying reference points in the images. The reference points (represented by devices visible in the image sequence) will be used for the translation of the object of interest or the roadmap during the online phase. The reference points could be, for example, the tip of a catheter, a pacemaker or an anatomical landmark, or any other object whose motion can be correlated to respiratory motion and possible patient motion. Such reference points can be acquired in all frames of the X-ray angiography image sequence. The identification of the reference points in the frames may be done manually. Instead of manually identifying reference points, image processing techniques may be applied to detect objects or landmarks within the medical image data, for example as described in step 503 of FIG.
[0059] In a preferred embodiment, the steps for obtaining the device position within an X-ray angiography image sequence are illustrated by the flow chart of Figure 5. In step 501 of Figure 5, the frames (406 of Figure 4) within the X-ray angiography image sequence where contrast fluid enters the coronary artery tree are detected. Figure 6 illustrates a method for detecting the start of a contrast injection frame. For each frame in the X-ray angiography image sequence, the average pixel intensity is calculated and visualized by graph 601. For example, the average pixel intensity of the first frame (602) is represented by 603, and the average pixel intensity of the second frame (604) is represented by 605.
number
number
number
number
number
number
[0060] Another approach to detect frames in an X-ray angiography image sequence where contrast agent enters the coronary artery tree is the method taught by [3]. In this work, [3] describes two different approaches to detect frames in an image sequence where contrast agent first appears. The first approach trains a convolutional neural network (CNN) to distinguish whether a frame has contrast agent or not. The second approach extracts first contrast agent features from an image with enhanced vascular structures, and detects contrast agent frames based on changes in the feature curves using the long short-term memory established by a recurrent neural network architecture.
[0061] Next, in step 502, cardiac cycle information is extracted from the X-ray angiography image sequence. If the ECG signal is part of the angiography image data (101 in FIG. 1), this step is omitted. As mentioned above, coronary artery motion includes cardiac motion due to heart contraction, respiratory motion due to patient breathing, and patient motion due to patient movement. To extract cardiac cycle information, only the motion of the coronary arteries and / or devices such as guiding catheters should be considered. Thus, a proxy extraction of cardiac motion by analyzing the motion patterns of objects representing cardiac motion such as the coronary artery tree and catheters provides information about the cardiac cycle in the X-ray angiography image sequence. In FIG. 7, 701 represents several consecutive X-ray image frames in the X-ray angiography image sequence. The motion of the diaphragm 704 is visible in the consecutive X-ray image frames 701.
[0062] One approach to extracting cardiac cycle information is shown in FIG. 7 at 702. Photograph 702 shows the coronary arteries and catheter extracted from the image frame of 701. This can be achieved by applying a Frangi vascularity filter followed by non-maximum suppression (e.g., as taught by [4]). Next, the extracted coronary arteries between successive frames are registered with each other (e.g., by 2D registration techniques taught by [5]) and this information is then
number
[0063] Photograph 703 in FIG. 7 shows another approach to extract cardiac cycle information. In this situation, the images are processed to eliminate structures that do not represent cardiac motion and to emphasize blood vessels. In this situation, cardiac motion is the main cause of intensity variations in the image sequence and can be further analyzed with methods that have source decomposition capabilities such as principal component analysis. First, the image sequence 701 is processed (703) to emphasize only blood vessels and devices such as catheters. This can be done using the layer separation method taught by [6], [7] or [8]. Principal component analysis is usually used for dimensionality reduction. It transforms a multivariate data set into a new Cartesian coordinate system such that most of the variance of this data set can be represented by a few coordinates. Thus, this dimensionality reduction is usually achieved without losing much information by retaining only a few coordinates in the new coordinate system. Principal component analysis is used on the post-processed images (703) to obtain the principal components of the image sequence. The frames of the sequence are divided into
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0064] The post-processed sequence (703) is then
number
number
number
number
number
number
number
number
[0065] Referring back to FIG. 5, the next step is to detect the device in the X-ray angiography image sequence.
number
number
number
[0066] Frame
number
number
[0067] Another method of detecting the device (e.g., catheter tip) can be performed by model-based detection or by using convolutional neural networks. Catheter tip detection may be improved by incorporating temporal information in pre- or post-processing. A fully automatic catheter segmentation technique including temporal information based on convolutional neural networks is taught in Non-Patent Document 10.
[0068] Another method for detecting a device (e.g., a catheter tip) is to first measure the amount of consecutive frames representing one cardiac cycle in an angiography image sequence (
number
number
[0069] Returning to Figure 1, the next step (103) is to create a roadmap. Within step 103, the location of the device is integrated into the created roadmap to serve as a reference point for the roadmap transformation (described in more detail by step 107 of Figure 1).
[0070] Step 103: Create a Roadmap In step 103, the X-ray angiography image sequence is processed to generate a roadmap of the coronary arteries for multiple phases of the cardiac cycle following the frame where the contrast agent entered the coronary arteries. Typically, the roadmap is generated for a certain amount of frames covering at least one cardiac cycle.
[0071] For example, a coronary roadmap can include a vascular model, which can represent a vessel centerline, a vessel boundary (contour), or a mask image representing the vessel. Additionally, the vascular model can also include clinically relevant information, such as, for example, the location and percentage of vessel occlusion, and / or the vessel diameter, area, length, or curvature, and / or the location and amount of calcified plaque.
[0072] In an embodiment, the coronary roadmap can be generated for all frames within one cardiac cycle after contrast injection. For example, a complete cardiac cycle (resulting from step 501 of FIG. 5) (and
number
number
[0073] FIG. 8 shows an example of layer separation on an X-ray angiography frame. It can be seen that the coronary arteries are significantly enhanced in the vessel layer (804, FIG. 8) compared to the original X-ray angiography image frame (801, FIG. 8). Such layer separation methods are taught, for example, by Non-Patent Document 6, Non-Patent Document 7, or Non-Patent Document 8. In FIG. 3, 302 provides an example of creating a roadmap of an X-ray image 302 by layer separation.
[0074] Another approach to creating a roadmap is to apply image processing skeleton techniques as taught, for example, by Non-Patent Document 4. Another approach to creating a roadmap is to utilize deep learning techniques as taught, for example, by Non-Patent Document 11 or Non-Patent Document 13.
[0075]
number
number
[0076] Optionally, quantitative image analysis can be performed based on the generated roadmap to extract clinically relevant information, such as the location and percentage of vascular occlusion, diameter and area, vessel length or vessel curvature, as taught by Fischer et al., 2003, Physiology and Radiology (2003) 143:1311-1354, 2003. Optionally, the location and amount of calcified plaque can also be extracted from X-ray angiographic image sequences, as disclosed in detail further in this application, for example by the description of the flow chart in FIG.
[0077] Finally, in step 103, the device positions obtained as a result of step 102 are integrated into the created roadmap. Fig. 35 illustrates how to integrate the device positions into the roadmap using examples. The first example represents the creation of the roadmap by layer separation (3501), as described above. In 3501, the device positions (3502) as a result of step 102 are integrated into the vascular layer (3501). The final roadmap showing the vascular structure (3503) also includes the device positions (3502). The second example represents the creation of the roadmap by skeletonization (3506, and for illustration purposes superimposed on the X-ray angiography frame by 3505), as described above. In 3505, the device positions (3504) as a result of step 102 are shown.
[0078] Step 104: Acquire fluoroscopic image data The first step of the online phase (110) is the readout of fluoroscopic image data, represented by step 104 in FIG. 1. In a preferred embodiment, the fluoroscopic image sequence is collected in real time as a live stream. At the same time, the patient's ECG signal can be acquired in real time, synchronized to the fluoroscopic image sequence. The fluoroscopic images are collected continuously at the same projection (viewing angle) as used during the offline phase (100) by step 101. Using the same projection ensures that the object of interest and the roadmap have the same orientation as the fluoroscopic image sequence (101) used to generate the roadmap (103). FIG. 9 shows three fluoroscopic image frames (901, 902, and 904) from a fluoroscopic image sequence. The catheter is visible in the fluoroscopic image frame (904) and is highlighted for illustration purposes. Each fluoroscopic frame is acquired within any instant of the cardiac cycle, as indicated by 906, 907, and 908 in the electrocardiogram (ECG) signal representing the cardiac cycle (909). The start of the cardiac cycle is, for example, the apex of the R wave (910), and all acquired X-ray frames (906, 907, and 908) represent a phase within the cardiac cycle (909) that is offset relative to the start of the cardiac cycle (910) for acquiring the respective X-ray frame. The ECG signal is preferably acquired simultaneously with the acquisition of the X-ray frames. During the online phase (110), the ECG signal is used to match the acquired frames with the corresponding roadmap extracted by step 103 to represent the same cardiac phase in the cardiac cycle. This is to compensate for changes in the shape and position of the blood vessels between the fluoroscopic image frames due to cardiac motion, and is further described in step 105 of FIG. 1. Simultaneously, the device (e.g., the position of the catheter tip) within the collected fluoroscopic image stream is tracked, for example, by using a deep learning-based Bayesian filtering method, as further described by step 106.Next, the displacement of the catheter tip between the current fluoroscopic image frame and the selected roadmap is obtained and this is applied to transform the roadmap, which will be further described by step 107. Finally, the transformed roadmap is superimposed on the current fluoroscopic image frame to guide the procedure, which will be described in step 108.
[0079] Step 105: Select a Roadmap After reading out the fluoroscopic images containing the ECG signals described by step 104, a selected roadmap from the roadmaps (created in step 103) is selected for every single fluoroscopic image.
[0080] The selection of the roadmap, represented by step 104 in Fig. 1, can be achieved by comparing the ECG signals associated with the online fluoroscopic images with the ECGs of the offline angiography sequence, such that the best candidate roadmap is selected for which the best match of the ECG signals is found. The selected roadmap has the same (or very similar) cardiac phase as the online fluoroscopic images, which compensates for differences in vessel shape and orientation caused by cardiac motion.
[0081] To select the roadmap and images based on ECG, we first need to build a temporal mapping between the X-ray image and the ECG signal points. It is assumed that the ECG signal and the X-ray image are well synchronized during acquisition.
[0082] In the offline phase (block 100 in FIG. 1), the start and end of the image sequence are aligned with the start and end of the ECG signal points. The X-ray angiography frames between them are evenly distributed on the ECG timeline. In this way, a mapping can be established between the stored sequence images and their ECG signals. For each image, the ECG signal point closest to the image's position on the timeline can be found; for each ECG point, the image closest to this point on the timeline can be similarly located. Once the mapping is available, all images with good angiographic contrast injection and the ECG points associated with these images are selected from the X-ray angiography sequence in the pool of the roadmap. In this process, at least one heartbeat of frames must be collected.
[0083] In addition, applying the method of step 501, images taken before the contrast agent reaches the blood vessels can be discarded, which will speed up the mapping process of the ECG data and the image sequence.
[0084] In the online phase (block 110 in FIG. 1), for each image acquisition, a block of NECG latest ECG signal points is constantly saved. These NECGECG signal points are considered as the ECG signal corresponding to the online fluoroscopy frame.
[0085] To compare the ECG signals associated with the offline angiography sequence and the online fluoroscopic images, a temporal registration of the two signals is applied, for example using cross-correlation. This is taught in non-patent document 14. The two ECG signals are first cross-correlated for all possible positions on the signals, resulting in a one-dimensional (1D) vector of correlation scores. Then, the candidate frames for the dynamic superposition and roadmap are selected as the frames associated with the points on the ECG of the offline angiography sequence corresponding to the highest correlation score.
[0086] Step 106: Tracking the Device In the online phase (block 110 in FIG. 1), a transformation of the object of interest or the roadmap is necessary to correct for respiratory and patient motion. Thus, in step 102, the position of the device is acquired in the angiographic image data. At the same time, due to the roadmap selection in step 105 in FIG. 1, the same device placed in step 102 is tracked in the retrieved fluoroscopic image in step 106. The device can be, for example, a catheter tip, a pacemaker or an anatomical landmark or other object whose motion can be correlated to respiratory and possible patient motion.
[0087] Next, a method for tracking a device is presented, and as an example, the tracked object is the tip of a catheter.
[0088] Exemplary Catheter Tip Tracking The global tracking of the catheter tip uses a deep learning based Bayesian filtering method and is summarized in Figure 13. The method uses a convolutional neural network to model the likelihood term of Bayesian filtering and integrates it with particle filtering in a comprehensive way, which results in a robust tracking. In summary, for every position in the image, a new position (in the new image) of the catheter tip (predicting the catheter motion) can be predicted using an optical flow method (see line 6 of Figure 13) and adding noise (see line 5 of Figure 13). Furthermore, the weights are updated (see line 8 of Figure 13) by checking the likelihood of the position using the deep learning network. Then, all the weights are normalized (see line 10 of Figure 13). The actual catheter tip position is equal to the weighted arithmetic mean of all the positions and their weights. Finally, a resampling of the points is performed around the positions with high weight values (see line 12 of Figure 13). Next, the tracking method is described in more detail.
[0089] Bayesian Filtering Bayesian filtering, as described for example in non-patent document 15, is a state-space approach that aims to estimate the true state of a time-varying system from a sequence of noisy measurements made on the system.
[0090] Bayesian filtering typically involves the following components: a hidden system state, a state transition model, an observation, and an observation model.
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0091] These definitions and
number
number
number
number
number
number
number
[0092] Initial Probability
number
number
number
number
number
[0093] Deep Learning-Based Likelihood Likelihood
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0094] To obtain the training labels,
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0095] The network used follows an encoder-decoder with skip connections similar to the U-net described, for example, in
[16] . In addition, similar to the work of
[17] , residual blocks are employed at each resolution level of the encoder and decoder, which facilitates gradient propagation in deep networks. Residual blocks are essentially a special case of highway networks where the skip connections have no gates. Essentially, residual blocks allow memory (or information) flow from the first layer to the last layer, as described, for example, in
[18] .
[0096] The encoder consists of four down blocks, where a residual block followed by stride-2 convolutions is used to extract and downscale the feature maps. The number of feature maps is doubled at each downsampling step. The decoder has four up blocks, where a stride-2 transposed convolution is used to upsample the input feature maps. Dropout is used in the residual units of the up blocks for network regularization. Another residual block is used between the encoder and the decoder to process the feature maps extracted by the encoder. The detailed network architecture is shown in Figure 12. Figure 12 shows a joint segmentation and detection network for catheter tip detection, and shows an example of a network with four levels of depth (number of down or up blocks). The meaning of the abbreviations in Figure 12 is as follows:
number
number
number
number
number
number
number
number
number
[0097] Since the tip of the guiding catheter and the corners of background structures such as vertebrae, lung tissue, stitches, and guidewires have similar appearances, there may be ambiguity if the network is expected to output only one blob in the probability map. To mitigate this issue, we adopt a strategy similar to that taught by
[19] to jointly train the network using the catheter mask (1102, Fig. 11) as an additional label to output both the catheter segmentation heat map and the likelihood probability map. The segmentation heat map is obtained by applying a 1x1 convolution with rectified linear units (Relu) activation to the feature map of the last up block. To compute the likelihood probability map, first the residual block is applied to the feature map of the last up block. Then the output feature map is concatenated to the segmentation heat map as one additional channel, followed by a 1x1 convolution. Finally, to ensure that the network detection output fits the definition of the probability map of image locations, a 1x1 convolution is followed by a spatial softmax layer, as shown in Equation 5.
number
number
number
number
number
number
number
[0098] The training loss is defined as a combination of the segmentation loss and the detection loss.
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0099] Approximating Posterior Probabilities with Particle Filters When a deep neural network is trained, its weights are determined by the posterior probability of new data.
number
number
number
[14] .
[0100] The particle filter method uses the associated weights
number
number
number
number
number
number
number
number
number
number
number
number
[0101] The sample is
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[20] .
number
number
number
number
[0102] Once the samples are drawn and their weights are updated, a so-called "resampling" of the samples must be performed to prevent the problem of degeneracy. In this case, the weights of all samples except one will be negligible after a few iterations. The resampling step involves resampling the existing samples according to the updated weights, and then resampling the weights of all samples.
number
number
number
[0103] Frame
number
number
number
[0104] Figure 14 presents four examples (sequences 1 to 4) of the catheter tracking method described here. In the first column online fluoroscopic images are shown, in the second column the position of the catheter tip obtained from the aforementioned tracking method is shown in points. In the last column on the right hand side the ground truth of the catheter tip is shown.
[0105] The final catheter tip position obtained from the tracking method is used in the next step 107 to determine the transformation between the device position in the online fluoroscopic images and the corresponding offline angiographic images of the selected roadmap.
[0106] In an alternative embodiment, step 106 can be accomplished by the teachings of US Pat. No. 6,399,366 entitled "Method and Apparatus for Tracking Objects in a Target Region of a Moving Organ," which uses the properties of synchronized periodic motion to establish dynamic geometric relationships between the positions of tracked features. This is achieved by learning the periodic motion of individual features, such as other visible devices that are typically present in the image view during a percutaneous coronary intervention procedure. If one of the features is occluded, knowledge of the pattern of periodic motion of both features, the position of one of the features, and the phase of the periodic motion can be used to derive the position of the occluded feature. If the first feature is the target location of the device itself (e.g., the tip of a catheter), such a position is immediately determined. If the first feature is not the target location, but a feature that has the same motion as the target location (e.g., another visible device), the position of the target location of the device can be derived from the position of the first feature using the fact that the first feature has the same motion as the target region. This requires knowledge of the geometric relationship between the first feature and the target region.
[0107] Step 107: Convert the selected roadmap to generate a dynamic roadmap As described herein, the reference points can be used to compensate for motion between the roadmap acquired from the X-ray angiography image sequence (offline phase) and the X-ray fluoroscopic image stream (online phase). The reference points that can be extracted from the device position can be any object whose motion can be correlated with respiratory and patient motion, and the device can be, for example, a catheter tip, a pacemaker, or an anatomical landmark. In a preferred embodiment, the device is a guiding catheter, and the device position is the guide catheter tip. In step 107, the device (e.g., catheter tip) position in the current fluoroscopic frame as a result of step 106 and the device position (e.g., catheter tip) from the selected roadmap frame as a result of step 105 are used to obtain a transformation function that aligns the selected roadmap to the current fluoroscopic image frame. In an embodiment, the transformation function can be a rigid transformation based on the displacement acquired between the catheter tip in the current frame and the catheter tip in the selected roadmap frame. Alternatively, the transformation function can be a non-rigid transformation.
[0108] For example, the rigid transformation of the roadmap can be performed using a rigid transformation function.
number
number
number
number
number
number
[0109] Step 108: Overlay the dynamic roadmap onto the fluoroscopic image data The dynamic roadmap resulting from the cardiac matching and transformation of the roadmap is rendered and integrated so as to be overlaid onto a corresponding fluoroscopic or angiographic image data frame of the patient's live angiographic image data stream.
[0110] Examples of dynamic roadmaps superimposed on contrast-free images are shown in Figures 10a and 10b. An example of dynamic roadmapping is shown in Figure 10a. The image on the left shows an extraction of the coronary artery tree, and the image on the right shows the coronary artery tree superimposed on a single frame in a fluoroscopic image sequence, corrected for dynamic motions such as respiratory, patient and cardiac motion.
[0111] Figure 10b shows another example: the image on the left shows a single frame in a fluoroscopic image sequence, and the image on the right shows the same fluoroscopic image superimposed with a dynamic roadmap, corrected for dynamic motion such as respiratory motion, patient motion, and cardiac motion.
[0112] The method provides real-time dynamic overlay or roadmapping where a visual representation of the dynamic roadmap is rendered and superimposed onto a live fluoroscopy or angiography image stream / sequence, thereby providing support to clinicians in improved patient care.
[0113] In an embodiment, step 108 can be accomplished by rendering the transformed roadmap (dynamic roadmap) resulting from step 103 according to the viewpoint of the current live image in the patient's live image stream / sequence. Pixels of the dynamic roadmap rendering that correspond to the target organ (e.g., coronary artery tree) can be assigned a predefined range of color values, such as a range of color values from red to white. In addition to coloring the dynamic roadmap, transparency can also be applied to the rendered model. By providing transparency to the model, a dual view is provided in which the fluoroscopic image and the dynamic roadmap are displayed. To enable visual recognition of a stent, balloon, or other device or instrument to treat the diseased artery and to enable visual recognition of the projected superimposition, the superimposition can be projected onto the fluoroscopic image data in a transparent mode, as shown by 3410 in FIG. 34a. It is clear that the degree of transparency can be adjusted. In Fig. 34a, 3411 provides another example where the superimposition is projected without obscuring the therapeutic instrument used to treat the diseased artery, and in this example the therapeutic instrument is a balloon (3412). This is achieved by displaying only the outer boundary of the roadmap image, which can be generated by first dilating the roadmap by a predefined amount and then extracting the boundary of the dilated roadmap. Of course, this can also be done in a transparency mode where the degree of transparency can be adjusted.
[0114] Mapping of the model (vascular layer image in Fig. 10a) onto the dynamic roadmap overlay can be performed by converting pixel intensities into color values (e.g., red to white, see example in Fig. 10a), where white represents the dark pixel intensity of the blood vessels (Fig. 10a, right image) and red represents the light-dark intensity of the blood vessels (Fig. 10a, right image).
[0115] The color scheme can also represent quantitative parameters (e.g., geometric parameters such as the local curvature and local diameter of the vessel). These geometric parameters can, for example, be derived from the vessel model created in step 103. Another example of a quantitative parameter is a pathology parameter such as the amount of calcified plaque in the lumen, which is derived in step 103 and is more particularly described in the flowchart of FIG. 28. After step 108, the method returns to step 104 to process the next fluoroscopic image frame, and the flowchart depicted in FIG. 1 ends when the readout of the fluoroscopic image data stops, as described in step 104.
[0116] Experimental setup for dynamic coronary road mapping In this section, an example is provided for training the device tracking method as described in step 106 of Figure 1, including training a deep neural catheter detection network as described in Figure 12 and tuning a deep learning based Bayesian filtering tracking method as described in the algorithm of Figure 13. Additionally, several experiments are described using the dynamic coronary roadmapping workflow as described in the flowchart of Figure 1. Here, the following four databases are used: 1. Training dataset for catheter tip detection. This dataset consists of a large number of X-ray fluoroscopy image sequences and
number
number
number
[0117] Because the images can be collected with different X-ray systems and different X-ray imaging protocols, the image dimensions (number of rows and columns) and pixel depth (range of pixel intensities) can vary. Therefore, all images in a dataset are gridded.
number
number
number
number
[0118] All the experiments carried out in this section are outlined as follows: First, the method of training the deep neural network to obtain optimal detection is described. Then, the training of the catheter tip tracker (tuning the particle filter) is described. Finally, the evaluation of the trained catheter tracker is described.
[0119] Training deep neural networks To train a deep neural network to provide accurate likelihood probability maps,
number
number
[0120] Data augmentation is performed to increase the number of training samples and their diversity. Augmentation includes geometric transformations such as flipping (left / right, up / down), rotations by multiples of 90 degrees, random affine transformations (translation, scaling, rotation, shear, etc.), and random elastic deformations. In addition to geometric transformations, data augmentation can also be performed by adding Gaussian noise to pixel values to make the trained model robust to noise.
[0121] During training, the two terms (in Eq. 8)
number
number
number
[21] with a learning rate typically of 0.0001. The number of training examples in a batch is typically 4, and the network is typically trained for 100 epochs to ensure convergence.
[0122] Training a deep neural network to output a reasonable likelihood probability map is performed by tuning the hyperparameters of the network. An evaluation metric is needed to select the hyperparameters and model weights in training. Since the deep network is essentially a catheter tip detector, accurate detection of the tip location is desired. For example, the average Euclidean distance between the ground truth and the predicted tip location averaged over all validation frames can be used as a validation criterion to select the optimal training epochs and network hyperparameters.
[0123] Catheter tip tracking training The catheter tip is tracked in the fluoroscopic images using the algorithm described in Figure 12, based on the network trained with the optimal hyperparameter settings from the previous section "Training the Deep Neural Network". The parameters of the optical flow method used in the algorithm described in Figure 12 and particle filtering (section "Approximating Posterior Probabilities with a Particle Filter" in the description of step 107 in Figure 1) are calculated based on the dataset.
number
[0124] When the approach of
[20] is used as the optical flow method (Equation 11), a grid search to find the optimal parameter settings is performed based on the following parameters of the Fameback method: (1) image scale for constructing the pyramid, (2) number of pyramid levels, (3) averaging window size, (4) number of iterations, (5) size of pixel neighborhood used to find polynomial expansion for each pixel, and finally (6) Gaussian distribution used to smooth the derivative used as the base for the standard deviation polynomial expansion. The above parameters are tuned independently of the deep neural network, since the optical flow method directly estimates the motion of the catheter tip between two frames. To tune the parameters, the catheter tip tracked in the fluoroscopy room was tracked starting from the ground truth tip position in the first frame using the motion field between two adjacent frames estimated by the optical flow method. The mean and median distance between the tracked tip position and the ground truth are used as evaluation criteria for tuning.
[0125] The parameter for tuning the particle filter is the number of samples.
number
number
[0126] Assessment of catheter tip tracking The tracking method proposed in the algorithm described in FIG. 13 converts the ground truth chip probability map of the first frame into an initial PDF
number
[0127] The first option uses only the detection network described in section "Deep Learning Based Likelihood" in the description of step 107 of FIG. 1, and tracks the catheter tip with the selected network architecture and trained parameters described in section "Training Catheter Tip Tracking" above. Therefore, this method does not use temporal information. This method is called "Detection(Net)".
[0128] The other two methods in this experiment use only optical flow to track the catheter tip from the ground truth tip position in the first frame. The motion fields estimated by these two methods towards the current frame were based on deformations from the previous or first frame in the sequence, respectively. These are called "Optical Flow (Previous)" and "Optical Flow (First)".
[0129] The tracking accuracy of all methods reported in this section is
number
[0130] Figure 37 shows how the tracking method proposed by step 107 of Figure 1 works on the same four frames of Figure 36. Figure 37 shows high probabilities in the detection map in light colors, and samples or particles are displayed as green dots. Blue points indicate the predicted catheter tip location; red points indicate the ground truth location (label). Figure 37 also shows that the previous hypotheses (samples) help to focus on the correct target location and result in a reliable posterior estimate, especially when detection is ambiguous due to multiple catheters or contrast agents remaining in the image.
[0131] In an alternative embodiment, the roadmap and set of dynamic roadmaps generated by this process can be three-dimensional (3D) roadmaps, as represented by the flowchart in Figure 18. The advantage of using 3D resulting in a sequence of 3D roadmaps (3D+t) would be that a) the X-ray viewing angle, magnification, and table position can be different between the offline phase (1800) and the offline phase (1810), thus giving the physician more degrees of freedom during the interventional procedure, and b) supplemental information such as plaque type obtained by non-invasive 3D imaging can be included in the 3D roadmap, and c) advanced quantitative analysis such as calculation of fractional flow reserve (cFFR) or a combination of these can be obtained from either the calculation of coronary wall shear stress (WSS) or from the non-invasive 3D imaging or from the 3D reconstruction of the target vessel from multiple X-ray angiography sequences. In the next section, the steps in Figure 18 are explained in detail.
[0132] Step 1801: Generate a 3D model from the 3D angiography image data In step 1801, which is an optional step, a 3D model of (a large portion of) the coronary artery tree is first generated using data collected from a 3D angiographic imaging modality such as computed tomography (CT), x-ray rotational angiography, 3D ultrasound, or magnetic resonance imaging (MRI). This 3D model can be in the form of, for example, a 3D centerline, a 3D surface contour representing the luminal and / or extravascular surfaces of the vessel, plaque, a 3D mask, or a combination of these.
[0133] The 3D centerline can be created manually, for example by indicating the vascular centerline in the 3D volumetric image data, or automatically, for example as taught by et al., “Ultra-vascular endoscopic imaging of vascular lesions in 3D volumetric image data,” in Proc.
[0134] Detection of coronary artery lumen, artery wall, and coronary artery plaque from CT angiography image data can be (semi-)automatically detected, for example, as taught by et al., “Detection of coronary artery lumen, artery wall, and detection of coronary artery plaque from CT angiography image data,” Journal of Cardiology, vol. 23, no. 1, 2011, which describes a method for detecting coronary artery lumen, artery wall, and detecting coronary artery plaque from CT angiography image data.
[0135] Coronary artery plaque can be detected, for example, as disclosed in U.S. Pat. No. 5,399,413 (Method and system for assessing vascular obstruction based on machine learning) or as taught, for example, by U.S. Pat. No. 5,499,636, or U.S. Pat. No. 5,523,363, or U.S. Pat. No. 5,623,636.
[0136] FIG. 19 shows some examples of the generation of 3D models based on 3D angiography image data. Picture 1901 shows the extracted coronary arteries, both the left coronary artery (LCA) and the right coronary artery (RCA), including a part of the ascending aorta based on CT. Picture 1902 is similar to 1901, except for the ascending aorta. In picture 1903, a part of the luminal vessel tree and atherosclerotic plaques (1905) detected based on MRI are displayed, and 1906 shows the detected luminal boundary (inner vessel wall; representing the vessel lumen) and the arterial boundary (outer wall) of the vessel. FIG. 25 provides two examples of 3D segmentation of the coronary artery lumen, including the segmentation of the coronary plaque. This can be extracted from 3D angiography image data as described in step 1803 of FIG. 18 or by 3D coronary artery reconstruction based on X-ray angiography, and calcified plaques can be extracted from X-ray image data as described in the flowchart of FIG. 28.
[0137] Optionally, the 3D angiographic CT image data can be used to calculate cFFR along the coronary tree, for example as disclosed by US Pat. No. 6,399,433, which discloses a method for calculating fractional flow reserve along the coronary arteries. Optionally, the 3D model extracted from either CT, rotational angiography, or MRI can be used to calculate the coronary WSS, or the time-averaged WSS, for example as taught by US Pat. No. 6,399,433. FIG. 20 shows an example of a coronary vessel with WSS superimposed on the luminal surface. FIG. 21 shows an example of a coronary vessel with pressure and fractional flow reserve superimposed on the luminal surface. The generated 3D model can include all the elements mentioned above in step 1801. It is clear that, except for rotational angiography and biplane x-ray angiography, the data cannot be collected or created during the intervention. Therefore, this data needs to be available, for example, on a server when starting the intervention. This generated 3D model will be used later.
[0138] Step 1802: Read out X-ray angiography image data In step 1802, the X-ray angiographic image data is read, which is similar to step 101 of FIG. 1. In a preferred embodiment, the angiographic image data represents an acquisition of the object of interest by X-ray angiography, which results in an X-ray angiographic image sequence. For example, single-plane or double-plane angiographic systems, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD), can be used. The object of interest is, for example, the heart, a part of the coronary tree, or a blood vessel, and the object of interest is visualized by contrast during the acquisition. Preferably, an electrocardiogram (ECG) is part of the angiographic image data and is simultaneously recorded during the acquisition of the X-ray angiogram. The X-ray angiographic image sequence is acquired such that the object of interest is clearly visible. Thus, the clinician can rotate and angulate the C-arm to obtain the best projection, also called optimal projection.
[0139] For the creation of the 3D roadmap described in step 1803, it is important that the X-ray angiographic image sequence is acquired from the correct perspective, defined as the angulation of the X-ray system (both rotation and angulation of the system) that contains as much information as possible about the target segment. In this perspective, foreshortening and overlap of the surrounding vessels are minimized. Foreshortening is an event where an object appears compressed when viewed from a certain perspective, distorting the information. The perspective in which the target object is viewed with minimum foreshortening is called the optimal perspective. In a preferred embodiment, the 3D model acquired from the 3D angiographic image data modality described in step 1801 is used to suggest to the user the optimal perspective with regard to foreshortening and overlap of the surrounding vessels. The orientation of this 3D model with respect to the X-ray system is known. The optimal perspective with regard to minimum foreshortening is determined as the perspective that is perpendicular to the orientation of the 3D model or its section. The model is viewable from a variety of angles, all of which are vertical, so that a different number of optimal perspectives are possible.
[0140] However, an optimal perspective depends not only on minimizing foreshortening, but also on the overlap of surrounding vessels, and therefore a measure of this overlap is also taken into account. The overlap of surrounding vessels may occur for one or more cardiac phases, since the surrounding vessels may overlap the segment of interest at a particular moment, either due to motion of the heart itself or due to respiratory motion.
[0141] The 3D model extracted as a result of step 1801 is backprojected onto a 2D plane representing a particular perspective, as taught by
[28] . Every point in the 3D model that lies within the target vessel is assigned a particular value. For each 3D point, that value is added to the corresponding 2D point in the backprojected image. The plane with the greatest amount of 2D points that contain a value is the most desirable perspective in terms of least overlap.
[0142] Additionally, a perspective can be shown in terms of the least amount of in-plane coronary artery motion. This perspective shows the target vessel with the least amount of in-plane coronary artery motion in the perspective. This allows the clinician to view the target vessel in a position that is as stationary as possible. A measure of the amount of in-plane coronary artery motion for each perspective can be determined by back-projecting the 3D model extracted from the CT data onto a 2D plane representing the particular perspective, for example, as taught in
[28] . The position of every centerline point of each vessel in the 3D model is known. Next, the 3D model extracted from the CT data can be deformed using the model taught in
[29] to generate 3D models at different times. This deformed 3D model is then back-projected onto the 2D plane representing the particular perspective. Again, the position of each centerline point is known, but now at a different time. For each centerline point, the in-plane coronary motion can be determined by comparing the position of each centerline point in both backprojections. The plane with the least in-plane motion for all centerline points is the most desirable perspective in terms of least in-plane coronary motion. This can be performed for one or more cardiac phases.
[0143] Next, for every combination of angulation and rotation (and therefore for each perspective), the degree to which the resulting perspective is optimal can be shown. This display can be, for example, a weighted sum of foreshortening, surrounding vessel overlap, and / or in-plane coronary artery motion for at least one cardiac phase. If multiple cardiac phases are considered, the calculations are performed frame by frame, time point by time. Next, a weighted sum of all frames in the multiple cardiac phases is created to give an overall indication of how optimal the perspective is. This can be visualized, for example, using a color map as shown in FIG. 22. FIG. 22a shows a color map for foreshortening, overlap, and in-plane coronary artery motion for one cardiac phase, while FIG. 22b shows a grayscale map for foreshortening, surrounding vessel overlap, and in-plane coronary artery motion including multiple cardiac phases. Obviously, this map can also be displayed as a color map. Using this result, the user can select a perspective from which an X-ray angiography image sequence can be acquired. In the preferred embodiment, two X-ray angiographic image sequences are acquired, however, if a 3D model is available, at least one X-ray angiographic image sequence is required.
[0144] If a 3D model is not available, two X-ray angiographic image sequences are required. The first X-ray angiographic image sequence is similar to that described in step 101 of FIG. 1, and it is assumed that the physician has selected the optimal projection to display the object of interest. The second projection can be acquired at the same time as the first projection is acquired, and this can be done using a biplane X-ray angiographic system. Otherwise, the second projection can be acquired by a single-plane X-ray system. The second projection is acquired after the first projection. The user can be assisted in selecting the optimal second projection relative to the first projection. A projection map can be generated (2303 in FIG. 23) based on the first projection (2301) and the orientation of the object of interest in the first projection (2302). This method is described in detail in US Pat. No. 5,399,363.
[0145] Step 1803: Create a 3D Roadmap Within this step, a sequence of 3D roadmaps (3D+t) is created, which is similar to step 103 of FIG. 1. This step (1803) results in the generation of a 3D roadmap of the coronary arteries covering multiple phases of at least one cardiac cycle. The 3D roadmap of the coronary arteries can be, for example, a vessel model. The vessel model may take the form of a centerline, a contour, a mask, etc. In turn, this vessel model may contain clinically relevant information, such as, for example, the location and percentage of vessel occlusion, vessel diameter and area, length, pressure, blood flow velocity, fractional coronary flow reserve, wall shear stress, vessel curvature, location, amount and type of coronary plaque, or location and amount of calcified plaque. In contrast to the 2D roadmap used in the flowchart of FIG. 1, the current workflow uses a 3D roadmap. While the 2D roadmap is defined in the coordinate system of the x-ray image used to generate the 2D roadmap, the 3D roadmap is defined in a 3D world coordinate system, thus allowing arbitrary x-ray geometries. The 3D roadmap is created from frames within one cardiac cycle after contrast injection. Thus, a complete cardiac cycle is selected in the X-ray angiography image sequence after the frame where contrast entered the coronary arteries. The frame in the X-ray angiography image sequence where contrast entered the coronary arteries can be defined by the method previously described by step 501 of FIG. 5. In step 1803, the 3D+t roadmap can be generated by three different methods, which mainly depend on the available data from steps 1801 and 1802.
[0146] If step 1801 is not performed, a first method (1803a) for creating a 3D+t roadmap is applicable, and the 3D+t roadmap is created by processing two X-ray angiographic image sequences resulting from step 1802. The 3D+t roadmap of all frames within one cardiac cycle after contrast injection can be generated, for example, by the method taught in Non-Patent Document 30 or Non-Patent Document 31. An example of a single 3D roadmap frame resulting from this step is provided by 2402 in FIG. 24. If the X-ray angiographic image sequence is acquired by a biplane system, the temporal resolution of the 3D+t roadmap can be improved by using a delay between each frame acquired by the frontal and lateral imaging sources.
[0147] When a 3D model is available as a result of step 1801 and one X-ray angiography image sequence is available as a result of step 1802, a second method (1803b) for creating a 3D+t roadmap can be applied, for example based on the method taught by
[32] . In this work,
[32] proposed a method to build a population-based average and predicted motion from a 4D-CT dataset. This is then used to perform a 3D+t / 2D+t registration based on distance minimization over one cardiac cycle. An example of a single 3D roadmap frame as a result of this step is provided by 2408 in FIG. 24.
[0148] If a 3D model is available as a result of step 1801 and two X-ray angiographic image sequences are available as a result of step 1802, a third method (1803c) for creating a 3D+t roadmap becomes available. In this method, both the 3D model resulting from step 1801 and the 3D reconstruction based on X-ray angiographic image data as a result of step 1802 are combined to create a 3D+t roadmap. This can be achieved, for example, by the method described in US Pat. No. 6,313,363, entitled "Method and Apparatus for Improved 3D + Time Reconstruction," which describes a method for performing a 3D surface reconstruction of an object over time from two or more two-dimensional X-ray images of the object. Alternatively, the 3D+t roadmap can be created as taught, for example, by Friedrich Schmidt et al., "A 3D roadmap for a 3D-based 3D reconstruction of an object over time," in US Pat. No. 6,313,363, entitled ...." Alternatively, the 3D+t roadmap can be created, for example, as taught by Friedrich Schmidt et al., "A 3D roadmap for a 3D-based 3D reconstruction of an object over time," in US Pat. No. 6,313,363, entitled "A 3D roadmap for a 3D-based 3D reconstruction of an object over time." An example of a single 3D roadmap frame as a result of this step is provided by 2410 in FIG.
[0149] Optionally, based on the generated roadmap, quantitative image analysis can be performed to extract clinically relevant information such as location and percentage, diameter and area of vascular occlusion, vessel length or vessel curvature, as taught, for example, in non-patent document 35 or non-patent document 13.
[0150] Optionally, the generated roadmap can be used to calculate cFFR along the coronary tree, as disclosed for example by US Pat. No. 6,399,433, which discloses a method for calculating fractional flow reserve along the coronary arteries based on a 3D coronary reconstruction obtained by X-ray angiography. The 3D+t model allows the calculation of cFFR at all time points and takes into account the geometric changes of the vascular tree segments during the cardiac cycle and the hemodynamic changes during the cardiac cycle. Optionally, the generated roadmap can be used to calculate the coronary WSS or the time-averaged WSS as taught for example by US Pat. No. 6,399,433, which discloses a method for calculating the ...
number
[0151] Optionally, the location and amount of calcified plaque can be extracted from an X-ray angiographic image sequence, for example as disclosed in detail further in this application by the description of the flow chart in FIG.
[0152] Step 1804: Acquire devices in X-ray angiography image data In step 1804, information is acquired to enable registration (step 1808) of the 3D+t roadmap (as a result of step 1803) during live registration (1809) as part of the online phase (1810), and step 1804 is similar to step 102 of FIG. 1. As described in step 102, respiratory motion, including possible patient motion, is compensated for by identifying reference points in the X-ray angiography image sequence. The reference points (represented by visible devices in the image sequence) will be used for translation of the object of interest or the roadmap during the online phase (1810). The reference points can be, for example, the tip of a catheter, a peacemaker or an anatomical landmark, or alternatively any other object whose motion can be correlated to respiratory motion and possible patient motion. Such reference points can be acquired in every frame of the X-ray angiography image sequence.
[0153] If step 1801 is not performed, this step (1804) is identical to steps 501, 502, and 503 described in FIG. 5 and can be applied to both X-ray angiographic image sequences as a result of step 1802. Optionally, the tip of the catheter is reconstructed in 3D, for example as taught by US Pat. No. 5,399,363. The acquired device position (e.g., catheter tip) is then integrated into the 3D+t roadmap created by method 1803a described in step 1803. Optionally, the device (e.g., catheter) is reconstructed in 3D and integrated into the 3D+t roadmap. The 3D+t roadmap ...
number
number
[0154] When a 3D+t roadmap is created by method 1803b, steps 501, 502, and 503 are applied to a single X-ray angiography image sequence, and 3D reconstruction of the device is not performed. However, the resulting device position from step 503 is integrated into the 3D+t roadmap created by method 1803b. An example of a single 3D roadmap frame resulting from this substep is provided by 2405 in FIG. 24, where 2408 represents the single 3D roadmap resulting from step 1803b, and 2407 represents the tip of the catheter (2407) integrated into the 3D roadmap model described above.
[0155] The above method is equally applicable if a 3D+t roadmap is created by method 1803c as described in step 1803 and at least two X-ray angiography image sequences are available as a result of step 1802. An example of a single 3D roadmap frame as a result of this sub-step is provided by 2409 in Figure 24, where 2410 represents the single 3D roadmap as a result of step 1803c, and 2411 shows a 3D reconstruction of the catheter and catheter tip (2412) integrated into the 3D roadmap model described above.
[0156] Step 1805: Read out the fluoroscopic image data. This step is the same as step 104 of Figure 1, except that in step 1805, any x-ray system geometry is allowed, such as any x-ray angulation and rotation, any x-ray magnification, any table position, etc., and the fluoroscopic image data can be acquired from either a single plane or a bi-plane acquisition. Optionally, the fluoroscopic image data can be acquired in a bi-plane acquisition.
[0157] Optionally, as any geometry of the X-ray system is possible within the flow chart of FIG. 18, a guidance map is visualized to assist the physician in selecting the optimal fluoroscopic projection. Similar to what is described in step 1802 and based on a method fully described in U.S. Pat. No. 5,399,633, the optimal fluoroscopic projection is defined as the projection where the X-ray viewing direction is perpendicular to the device (e.g., catheter). This device information is obtained based on the results of step 1804. Furthermore, such optimal projection may include a weighting of the amount of foreshortening of the device; the less foreshortening, the better the projection.
[0158] Step 1806: Select 3D Roadmap This step is the same as step 105 in Fig. 1 where the roadmap selection is based on ECG matching. The ECG signal is correlated with the online fluoroscopic images (see 1805 in Fig. 18) and the ECG of the offline angiography sequence (see 1802 in Fig. 18). This selects the best candidate roadmap that finds the best match with the ECG signal.
[0159] Alternatively, if fluoroscopic image data is acquired in a biplane acquisition, images from both projections can be acquired in close proximity to each other with a small time delay, resulting in higher resolution. This higher temporal resolution allows for more accurate roadmap selection.
[0160] Step 1807: Track the device This step is the same as step 106 in FIG. 1, where devices in the online fluoroscopic image data are tracked using, for example, a deep learning based Bayesian filtering method.
[0161] Alternatively, if fluoroscopic data is acquired by biplane acquisition, tracking of the device can be performed separately in both fluoroscopic image projections according to the technique described in step 106 of Figure 1. This results in two translations of the device with respect to the device position acquired in step 1804 in both biplane projections. Furthermore, the catheter tip can be reconstructed in 3D and used to convert the 3D roadmap in a next step, as taught, for example, by US Patent No. 5,399,633.
[0162] Step 1808: Convert the selected 3D roadmap to generate a dynamic 2D roadmap This step is similar to step 107 of Figure 1, where reference points are used to compensate for motion between the roadmap generated from the X-ray angiography image sequence (offline phase) and the X-ray fluoroscopy image stream (online phase). Within step 1808, the same objectives are acquired as described in step 107, except that a) the roadmap is a 3D roadmap, and b) the device can be in a 3D position or 3D in addition to a 2D position as in step 107.
[0163] In step 1808, the device position in the current fluoroscopic frame as a result of step 1807 and the device position (e.g., catheter tip) from the selected 3D roadmap frame as a result of step 1804 are used to obtain a transformation function to register the selected roadmap to the current fluoroscopic image frame. In an embodiment, this transformation function can be a rigid transformation based on the displacement obtained from the catheter tip between the current frame and the catheter tip in the selected roadmap frame. An alternative transformation function is a non-rigid transformation.
[0164] For example, the rigid transformation of the roadmap can be performed using a rigid transformation function.
number
number
number
number
number
number
number
[0165] Alternatively, if fluoroscopic data is acquired in a biplane acquisition, tracking of the device can be performed in both fluoroscopic image projections, in which case the above transformations can be performed for each fluoroscopic projection and two 2D dynamic roadmaps are created.
[0166] Step 1809: Overlaying the dynamic roadmap on the fluoroscopic image data This step is similar to step 108 of Figure 1, where the transformed roadmap is rendered to generate a visual representation of the transformed roadmap that is overlaid on each live fluoroscopic image frame. If the fluoroscopic data is acquired in a bi-plane acquisition, then two 2D dynamic roadmap visual representations can be rendered and overlaid on each live fluoroscopic projection.
[0167] In an embodiment, step 1809 can be accomplished by rendering the transformed model resulting from step 1808 according to the viewpoint of the current live image in the patient's live image stream / sequence. Pixels of the model rendering that correspond to the organ of interest (e.g., the coronary artery tree) can be assigned a predefined range of color values, such as a range of color values from red to white. In addition to the model coloring, transparency of the rendered model can also be applied. Model transparency provides a dual view where the fluoroscopic image is visible and the model.
[0168] Mapping the model onto the dynamic roadmap overlay can be performed by converting pixel intensities into color values, where the color scheme represents the quantitative parameters described in step 1804.
[0169] In addition, multiple registrations are rendered and integrated as a registration on the live fluoroscopy or angiography image stream. The registration can consist of a 3D roadmap and, in addition to the roadmap, e.g., 3D volume data or quantitative parameters. The 3D volume data needs to be backprojected into the current fluoroscopy image frame based on the projection (viewing angle) in the same way as the 3D roadmap.
[0170] Obviously, steps 1808 and 1809 may be performed in a slightly different order. In step 1808, a 3D transformed roadmap is obtained based on information from steps 1808 and 1807, and this transformed 3D roadmap is backprojected onto the current fluoroscopic image frame in step 1809, which accounts for the current fluoroscopic angulation, magnification, and table motion during the backprojection.
[0171] After step 1809, the method continues to step 1805 to process the next fluoroscopic image frame, and when the readout of fluoroscopic image data stops as described in step 1805, the flowchart ends as described in FIG. 18.
[0172] In an alternative embodiment, subtraction is used as an image enhancement technique to improve the identification and processing of target objects. Image enhancement is a useful tool to improve the identification of target objects. An example of image enhancement is subtraction, where static objects and background are removed from a target image by subtracting a baseline image from this image.
[0173] Figure 15 shows a similar workflow to that of Figure 1, where differential processing is used as an image enhancement technique to improve the identification and processing of objects of interest. There is an offline phase 1500 and an online phase 1512. Only an additional baseline image data set is retrieved and image enhancement is performed, for example by differential processing.
[0174] First, the same X-ray angiography image data including ECG as 101 in Figure 1 is read out 1501. Next, fluoroscopic image data including ECG (1502) is acquired with the same projection (viewing angle) as in 1501 and the catheter is pulled back from the ostium. The fluoroscopic image data of 1502 should include one complete cardiac cycle and this image data is a baseline image sequence where only the background and stationary objects are visible.
[0175] Next, in step 1503, one entire cardiac cycle of the fluoroscopic image data 1502 is selected. For every frame of the X-ray angiographic image data 1501, a baseline image frame 1502 is selected based on ECG matching. The ECG matching can be performed as described in 105 of FIG. 1. After matching the X-ray angiographic image data and the baseline image data, the X-ray angiographic image data (see 1601 of FIG. 16) is subtracted from the baseline image data (see 1602 of FIG. 16) and the image is enhanced (see 1603 of FIG. 16). In the enhanced image, dynamic objects are clearly visible. Furthermore, image processing techniques such as edge enhancement can be applied to improve the contours of the target objects.
[0176] The enhanced image obtained from 1503 is used to obtain the device position in step 1504. Step 1504 is identical to step 102 in Figure 1. A roadmap is then created in step 1505, which is identical to step 103 in Figure 1.
[0177] In the online phase 1512, fluoroscopic image data containing the same ECG as 104 with the same projection (viewing angle) as 1501 and 1502 is read out 1506. Next, in step 1507, image enhancement is applied to the fluoroscopic image of 1506, and the same process as in step 1503 is performed, including ECG matching of the fluoroscopic image data 1506 with the baseline data 1502. Enhanced fluoroscopic image data is obtained by subtracting the fluoroscopic image data 1506 from the matched baseline data 1502. Furthermore, image processing techniques such as edge enhancement can be applied to improve the contours of the target object.
[0178] Then, in step 1508, roadmap selection is performed similar to step 105 of FIG. 1. The enhanced fluoroscopic image frame from step 1507 is used to track the device in step 1509. Then, in step 1510, the roadmap selected in step 1508 is transformed similar to step 107 of FIG. 1. Finally, in step 1511, the roadmap is presented as an overlay on the fluoroscopic image data read out in 1506. After step 1511, the method returns to step 1506 to process the next fluoroscopic image frame, and the flow chart described in FIG. 15 ends when the reading out of the fluoroscopic image data described in step 1506 stops.
[0179] In an alternative embodiment, the online phase of dynamic roadmapping can be resumed with an x-ray angiographic image stream and is represented by the flowchart of Figure 26. Such an x-ray angiographic image stream is acquired during live fluoroscopic image acquisition after the roadmap has been realigned, when the physician injects a limited amount of contrast agent.
[0180] The flowchart in Figure 26 is an alternative to either the flowchart in Figure 1 (2D roadmap) or the flowchart in Figure 18 (3D roadmap). The advantages of this alternative embodiment are that a) the table can be moved during the online phase, b) additional devices introduced during the (live) procedure can be used for realignment during the tracking phase, and c) the physician is notified if the alignment of the roadmap projected on the live fluoroscopic image is incorrect. The next section describes the steps in Figure 26 in more detail.
[0181] Step 2601: Offline Phase Within step 2601, a sequence of roadmaps is created, and if this sequence of roadmaps represents a 2D roadmap, the step is identical to the offline phase (100) represented by steps 101, 102, and 103 in Figure 1. If a 3D roadmap is created, step 2601 is identical to the offline phase (1800) represented by steps 1801, 1802, 1803, and 1804 in Figure 18.
[0182] Step 2602: Start X-ray Angiography Within this step, the roadmap is realigned to the current situation. The term current situation means the true live status during the online phase (2610). Such realignment is useful, for example, after a table movement during the online phase (2610) or in other situations where the alignment from the current situation of the roadmap may be incorrect, or when additional devices are introduced during the online phase to allow more accurate alignment. This step can be omitted if the system transitions from step 2601 to the online phase (2610) for the first time. Step 2602 can start from a signal triggered by the x-ray system or another external system after the contrast bolus injection has been started. Alternatively, the contrast bolus injection is automatically detected by analyzing the x-ray fluoroscopic image stream (2603), for example, as taught by "A New Approach to Fluoroscopic Imaging," in ...
[0183] The first substep in 2602 is the readout of the X-ray angiography image stream, which can be acquired from either a single plane or a double plane acquisition. Once a certain amount of X-ray angiography image frames are available (preferably an amount of frames covering at least one cardiac cycle), the roadmap is realigned to the current situation. Concurrently with the X-ray angiography image stream, an ECG signal is also acquired.
[0184] If the offline phase represents a 2D roadmap, the second sub-step in 2602 deals with the update (re-creation, or re-registration) of the 2D roadmap sequence and is identical to steps 102 and 103 of FIG. 1. If the offline phase represents a 3D roadmap, the second sub-step in 2602 is the re-creation of the 3D roadmap sequence and is identical to steps 1803 and 1804 of FIG. 18. FIG. 27 illustrates the re-registration step 2602 in the case of a 3D roadmap. Photograph 2701 shows the roadmap (2608) back-projected onto the X-ray angiography biplane acquired in the first sub-step of 2602. In 2701, the misregistration is visible, with extreme displacements for illustration. Photograph 2702 shows the result of extracting the 2D vasculature of the biplane image. This will be used to create the 3D model described in steps 1803 and 1804 of FIG. 18. 2703 shows the aligned backprojected roadmap.
[0185] Alternatively, if image data is read from the acquisition of multiple single-plane images with different projections or from a bi-plane acquisition in an offline phase, the position of the device can be determined for each individual image data projection according to step 102 of Figure 1 or 1804 of Figure 18. Additionally, based on the projection information of the image data, the position of the 3D device can be determined for each individual position of the device in all the individual projections.
[0186] Step 2603: Read out X-ray fluoroscopic image data In step 2601, the fluoroscopic image data is read, and this step is identical to step 104 of Fig. 1. If the offline phase represents the 3D roadmap (step 2601), any X-ray system geometry is possible, such as any X-ray angulation and rotation, any X-ray magnification, any table position, and the fluoroscopic image data can be obtained from a single-plane or bi-plane acquisition. Furthermore, as described in step 1805 of Fig. 18, a guidance map can be visualized to support the physician in selecting the optimal fluoroscopic projection.
[0187] Step 2604: Select a roadmap In step 2604, a current roadmap is selected from the sequence of roadmaps, either as a result of step 2601 or, if updated, as a result of step 206. If the offline phase represents a 2D roadmap, this step is identical to step 105 of Figure 1. If the offline phase represents a 3D roadmap, this step is identical to step 1806 of Figure 18.
[0188] Alternatively, fluoroscopic data is acquired with a biplane acquisition. In a biplane system, images from both projections are acquired one after the other with a small time delay, resulting in a high temporal resolution. This high temporal resolution allows for more accurate roadmap selection.
[0189] Step 2605: Tracking the Device In step 2605, the device is tracked in the live fluoroscopic image stream. If the offline phase represents a 2D roadmap, this step is identical to step 106 in Figure 1, and if the offline phase represents a 3D roadmap, this step is identical to step 1807 in Figure 18.
[0190] Alternatively, the fluoroscopic data can be acquired in a biplane acquisition, in which case tracking of the device can be performed separately in both fluoroscopic image projections according to the technique described in step 106 of Figure 1. This results in two translations of the device with respect to the device position acquired in step 2601 in both biplane projections.
[0191] The device position can then be determined as a 3D position based on both projections from a biplane acquisition, as taught, for example, by et al., 2011, 35. This allows 3D tracking of the device.
[0192] Step 2606: Transform the selected roadmap to generate a dynamic roadmap In step 2606, the selected roadmap is transformed to create a dynamic roadmap for the current live fluoroscopic image frame. If the offline phase represents a 2D roadmap, this step is the same as step 107 in Figure 1. If the offline phase represents a 3D roadmap, this step is the same as step 1808 in Figure 18.
[0193] Alternatively, the fluoroscopic data can be acquired in a biplane acquisition, in which case two translations are acquired in step 2605. These two translations and the 3D orientation of both image projections result in a 3D transformation of the selected roadmap.
[0194] Step 2607: Overlay the dynamic roadmap onto the fluoroscopic image data If the offline phase generates a 2D roadmap, this step is identical to step 107 in FIG. 1. If the offline phase generates a 3D roadmap, this step is identical to step 1808 in FIG. 18. Optionally, the method of visualizing the roadmap on the X-ray image data can be modified. As a result of the contrast agent administered in step 2602, the vascular structures will be highlighted as long as the contrast agent is in the vasculature (this depends on the amount and duration of the contrast agent injection, and is on average about 5 cardiac cycles). To allow the visualization of the vascular structures and the projected superposition, the superposition can be projected on the X-ray image data in a transparent mode, as shown by 3401 in FIG. 34b. The degree of transparency can obviously be adjusted. In FIG. 34b, 3402 provides another example where the superposition is projected without obscuring the vascular structures of interest. This is achieved by displaying only the outer border of the roadmap image. This can be generated by first dilating the roadmap by a predefined amount and then extracting the border of the dilated roadmap. Obviously this can also be a translucent mode where the degree of transparency can be adjusted.
[0195] Step 2608: Detect Drift During step 2608, the system detects whether a realignment of the roadmap is necessary. For example, the X-ray system triggers a signal if the table is adjusted during the online phase (2610) or after contrast administration has started. Optionally, within step 2608, a misalignment of the roadmap can be detected by image processing of the fluoroscopic image stream, for example a comparison of the current image frame with an image frame from the same cardiac cycle of the previous cardiac cycle. The comparison between both images can be performed, for example, by image registration, cross-correlation or (minimum) difference. When applying image registration, the displacement resulting from image registration is a measure of the variation between both images and a threshold value can be defined for detecting drift. In case of cross-correlation, the cross-correlation value represents the similarity between both images and a threshold value can be defined for detecting drift, for example. Alternatively, one image is shifted with respect to the other image and the cross-correlation between both images is calculated for multiple shifts. The maximum cross-correlation value may then correspond to a certain shift of the images. The magnitude of the shift is the magnitude of the drift. If there is a difference between both images, the difference value is a measure of the drift. A threshold may be defined for drift detection. Alternatively, one image is shifted relative to the other image and the difference of both images is calculated. The minimum difference can be calculated for multiple shifts and the shift corresponding to the minimum difference is a measure of the drift.
[0196] Another method for drift detection is the evaluation of the ECG signal. The ECG is a periodic signal. For example, in the case of arrhythmia, the heartbeat causes irregular and abnormal heart movements, and the ECG signal deviates from the normal cycle. Deviations in the ECG signal can interfere with the selection of the roadmap, which can cause erroneous superposition of the roadmap. Therefore, the evaluation of the ECG signal helps to detect drift.
[0197] During regular breathing, the diaphragm contracts, which is called normal breathing. For example, if a patient has hiccups, the diaphragm movement is irregular and therefore the movement of the target object may also be irregular. The diaphragm contraction may be visible in an X-ray angiographic image. Tracking the diaphragm in an X-ray image sequence helps to detect irregular breathing and the need for drift correction. Tracking of the diaphragm movement can be performed, for example, as taught by
[37] . Thus, evaluation of the diaphragm contraction (movement) helps to detect drift.
[0198] If step 2610 indicates that a misalignment exists, step 2602 is entered, otherwise the system returns to step 2603. When the reading of fluoroscopic image data stops as described in step 2603, FIG. 26 ends.
[0199] Although calcified plaque is radiopaque, its presence is not discernible in a single X-ray image frame and is barely visible when evaluating a sequence of X-ray images. FIG. 28 represents a flow chart showing an embodiment of extracting the location and amount of calcified plaque by X-ray image data. The X-ray image data can represent a fluoroscopic image sequence or an X-ray angiographic image sequence, where the X-ray angiographic image sequence includes image data before and after administration of a contrast agent. In either case, the ECG signal should be considered as part of the X-ray image data. FIG. 30 illustrates the challenges in identifying calcified plaque. In FIG. 30, a high-level method as described in the flow chart of FIG. 28 is shown. A calcified plaque is present in the dashed circle (3006), but it is barely discernible in the images (3001, 3002, 3003, and 3005). After registering the images, the calcified plaque (3010) is enhanced and made visible by adding the image information into a single image (3009). Furthermore, quantitative analysis can be performed on the enhanced calcified plaque. In the next section, the steps in Figure 28 are explained in detail.
[0200] First, in step 2801, X-ray image data is read. As mentioned before, this can be either a fluoroscopic image sequence or an X-ray angiographic image sequence including image data before and after the administration of a contrast agent. First, the steps are described when the X-ray image data represents X-ray angiographic image data, and then the method is described when the X-ray image data represents fluoroscopic image data. In all cases, it is assumed that an ECG signal is part of the X-ray image data.
[0201] Fig. 29 shows a typical example of an X-ray angiography image sequence, where 2900 shows several successive X-ray image frames in such a sequence and 2906 shows the corresponding ECG signal. In Fig. 29, the moment of administration of contrast agent is indicated by 2905, and previous frames (2901, 2902) show unenhanced X-ray image data of the vasculature next to the catheter (2907). After administration of contrast agent, the frames also show the vasculature (2903, 2904), which in the present example represents the left coronary artery. Furthermore, the moment at which each X-ray frame was acquired with respect to the ECG signal is visualized (2901, 2902, 2903, and 2904).
[0202] Alternatively, the characteristics of the x-ray beam can be altered so that calcified tissue absorbs x-rays more efficiently than the tissue surrounding it. When applied to the evaluation of vascular structures, the characteristics of the x-ray beam are defined so that the x-rays absorb an iodine-rich contrast agent optimally relative to the surrounding tissue. The two main means of defining the characteristics of the x-ray beam produced by an x-ray tube are a) changing the current (mA) and b) changing the voltage (kV). The current (measured and often called mA or milliamps) across the x-ray tube determines the number of electrons that are emitted and strike the anode. Increasing mA increases the number of electrons striking the anode, which in turn linearly increases the number of photons produced by the x-ray tube. The voltage across the x-ray tube (often measured in units called kV or kilovolts) affects the speed of the electrons as they strike the anode. This affects the energy of the photons that can be produced by the x-ray tube. Furthermore, faster electrons produce more photons, on the order of (kV)3. There are two main ways that an x-ray beam interacts with tissue. The first main effect is the photoelectric effect, where a photon uses up all its energy and ejects an electron from an atom, while the electron moves around and ionizes neighboring atoms, but there are no scattered photons. The second main effect is Compton (incoherent) scattering, where a photon hits an atom and ionizes an electron, but not all of its energy is used up. The photon then scatters in another direction with a little less energy, and the free electron does damage. The scattered photons can return to the x-ray tube, pass through the patient and hit the detector from any odd angle, or scatter again within the patient. As the x-ray beam passes through tissue, the energy is reduced as the photons are absorbed. This is known as attenuation. High energy photons pass through tissue more easily than low energy photons (which means that high energy photons are less likely to interact with matter). Many of these effects are related to the photoelectric effect. The probability of photoelectric absorption is approximately proportional to (Z / E), where Z is the atomic number of the tissue atom and E is the photon energy. As E becomes large, the likelihood of interaction decreases rapidly.Compton scatter decreases slowly with increasing energy, but is fairly constant for different energies. Therefore, instead, fluoroscopic or angiographic image sequences are acquired in which the tube characteristics (tube voltage and / or tube current) of the x-ray system are optimized so that calcified tissue absorbs x-rays more efficiently than the surrounding tissue.
[0203] Next, in step 2802, the target vessel is obtained. This can be done either by the method described in step 103 or the result of step 103 of FIG. 1, or by the method described in step 1803 or the result of step 1803 of FIG. 18, or by a user manually specifying the target vessel.
[0204] In step 2803, a frame from the x-ray angiography image sequence is identified in which contrast fluid enters a coronary artery; and
number
[0205] Next, in step 2804, the vascular skeleton is calculated for each frame in the sequence after the onset of contrast agent as a result of step 2803.
number
[0206] In step 2806, the corresponding fluoroscopic images selected as a result of step 2805 are registered with each other. Registration is performed for each decomposed branch vessel as described in step 2804. Optionally, landmarks are automatically identified from the skeleton and / or the decomposed skeleton to improve the performance of the registration. Such landmarks can be for example branch points (3104) or start / end positions (3105) or large local curvatures (3106). The 2D to 2D registration can be performed for example as taught by [5] or
[38] .
[0207] In step 2807, an image with enhanced calcified plaques is generated. Before generating the enhanced image, the registered fluoroscopic images (resulting from step 2806) are pre-processed. X-ray images usually suffer from a transparent background layer that potentially obscures the calcified plaques. To eliminate the impact of this effect, a background subtraction process is performed. A simple and effective way to remove the static transparent layer is to use the assumption that the individual layers only add mass. Taking the maximum intensity for each pixel over time (the lower the pixel intensity, the more X-rays are absorbed) would generate an image that shows the least amount of mass over time. If we assume that all pixels will not contain any contribution from a moving mass layer at any given time, the maximum intensity image will be equal to the static background layer. Often, a single common mask image is created for each frame, for example based on the maximum intensity of each pixel over all frames, as described above. However, such a single mask image suffers from artifacts due to large, slow-moving objects in the background (such as the diaphragm, ribs, lungs). Therefore, the background subtraction process performed determines a more local background mask of x consecutive frames (x is usually 5 frames) symmetrically around each frame. Alternatively, a static background layer can be generated as taught by [7] or [8].
[0208] The pre-processed frames are then combined into a single image frame. This is done by appending the pre-processed frames (3008 in FIG. 30) and, optionally, introducing a weighting factor that correlates to the likelihood of non-rigid deformation of each frame, for example due to foreshortening. An example of the result of this process is shown in FIG. 30 at 3009. Within the photograph 3009, calcified plaque is highlighted (3010).
[0209] Before proceeding to step 2808, the method of FIG. 28 is now described for the case where the X-ray image data represents a fluoroscopic image sequence (as a result of step 2801). In this case, it is assumed that a device including a radiopaque marker is present in the target vessel. This can be, for example, an undeployed stent, a measurement guidewire, or any other device that moves in synchrony with the motion of the target vessel. An example of an undeployed stent is provided in FIG. 32, 3202. Step 2803 is not applicable to fluoroscopic image sequences, and step 2804 now detects and tracks the radiopaque marker in the fluoroscopic image sequence. Since all image frames in the fluoroscopic image sequence are fluoroscopic image frames, step 2805 is not applicable. In step 2806, the tracked radiopaque marker is used to register the images with each other, and step 2807 is identical to that described above. FIG. 32 provides an illustration in which an enhanced calcified plaque image is superimposed on an X-ray angiography image frame that has been registered with the enhanced calcified plaque image by ECG. Image 3202 shows a single image frame in a fluoroscopic image sequence, and image 3203 shows the result of the enhanced calcified plaque image. Image 3201 shows an X-ray angiography image frame that has been registered with ECG to image 3203. Image 3204 shows the result of subtracting the enhanced calcified plaque image from the X-ray angiography image frame, and image 3205 shows the result of segmenting the enhanced calcified plaque with the X-ray angiography image frame. Both images 3204 and 3205 show the location of the calcified plaque and the visible volume relative to the vessel lumen.
[0210] Returning to FIG. 28, the final step 2808 involves a quantitative analysis of the enhanced calcified plaque. The area of the calcified plaque (3010, 3206) can be calculated by manual and / or (semi-)automatic detection of the calcified plaque (calcified plaque area) using the enhanced images (3009, 3204). A video densitometry analysis can also be performed. The volume and / or mass of the calcified plaque can be derived by comparing the density of the calcified plaque area with other radiopaque areas in the enhanced or X-ray image data (2801). If the geometry and the properties of the radiopaque area such as its (mass) attenuation coefficient are known, the volume and / or mass of the calcified plaque area can be calculated using the Beer-Lambert law. The radiopaque area can also be an area of a blood vessel obtained from an X-ray angiography image sequence. In this situation, the (mass) attenuation coefficient of the contrast agent needs to be known in order to be able to calculate the volume and / or mass of the calcified plaque area using the Beer-Lambert law. Alternatively, a videodensitometry analysis can be performed, for example as disclosed by US Pat. No. 5,399,633.
[0211] Typically, X-ray image data includes corresponding ECG signal records. If the X-ray image data does not include ECG information, matching of cardiac phases between contrast-enhanced and non-contrast-enhanced images based on the ECG signal is not possible. To solve this problem, a cardiac model can be used that mimics the cardiac motion. Such a cardiac model can be generated, for example, from several cardiac CT acquisitions. The cardiac motion is extracted from the image data, and a model that mimics the cardiac motion is generated.
[0212] The cardiac motion model provides cardiac motion in 3D+t. Based on an X-ray angiography acquisition, the expected cardiac motion in a particular projection can be extracted from the cardiac motion model.
[0213] The cardiac motion model is applied to both the contrast and non-contrast images and, for example, the correlation between both images is calculated to identify the best match between both images, as taught, for example, by
[39] .
[0214] Other uses The above-described embodiments are directed to providing real-time dynamic overlay or dynamic angiographic image streams / sequences that can be overlaid onto live X-ray projections or coronary artery roadmaps.
[0215] In an embodiment, the method described in the present application can also be used to provide static guidance, which means selecting only one roadmap in a sequence of roadmaps. Preferably, the roadmap with the least cardiac motion of the coronary arteries is selected. Furthermore, the 3D model created from the 3D angiography image data described in step 1801 of FIG. 18 can be used as a static roadmap to be superimposed on the X-ray image data. Optionally, the registration to the X-ray image data can be performed, for example, by techniques previously described or disclosed by the present application. For example, the model presented by FIG. 25, which also includes coronary plaque information, can also be used as static guidance.
[0216] In other embodiments, the methods described can be used as pre-procedural planning, and optionally, X-ray angiographic images can be simulated, for example as described in U.S. Pat. No. 6,399,633, to improve preparation for an actual PCI procedure.
[0217] Alternatively, a single roadmap can be selected. The image associated with the selected roadmap can be used as the reference image. The relationship between the roadmap and the reference image can be determined, for example, using the patient's ECG signal. Then, all images of the live fluoroscopy or angiography image stream / sequence are registered to the reference image. With image registration, the images of the live fluoroscopy or angiography image stream / sequence are "frozen" and aligned to the static roadmap. Image registration can be performed by methods known in the art, for example, as taught by "Analysis of Fluoroscopic Images of Live Fluoroscopic Images", vol. 13, No. 1, pp. 1111-1115, 2002.
[0218] The above-described embodiments are related to provide a real-time dynamic overlay or dynamic coronary roadmap that can be overlaid on a live fluoroscopy or angiography image stream / sequence. Alternatively, the embodiments specifically focused on the extracted roadmap described in Figures 1, 15, 18 and 26 can be used in the field of robotic assisted percutaneous coronary intervention (PCI) or peripheral vascular intervention.
[0219] Since the first introduction of PCI in the 1970s, interventional cardiology has undergone significant evolution in device technology, treatment techniques, and drug therapies, with the average treatment of the most complex lesions possible (such as complete coronary artery occlusion). Despite steady progress in almost all aspects of the field of coronary intervention, the mechanical aspects of PCI, such as the manipulation of coronary guidewires, balloons, and stents, and the occupational hazards to the operator and catheterization laboratory staff, have remained largely unchanged since its introduction in the 1970s. Interventional cardiologists manipulate intravascular devices under direct fluoroscopy guidance, and this imposes the need to wear heavy protective clothing. Over the course of an interventional cardiology career, operators are exposed to the adverse effects of cumulative radiation exposure and an increased prevalence of orthopedic injuries.
[0220] A robotic system for handling the manipulation of intravascular devices could significantly reduce the aforementioned drawbacks and risks and revolutionize percutaneous coronary intervention procedures. Such robot-assisted PCI systems, while still in their early phases, are manufactured, for example, by Corindus vascular robotics.
[0221] One of the requirements for such a robotic system is to manipulate intravascular devices at the sub-millimeter level, and requires knowledge of the 3D geometry and morphology of the vasculature. This also includes knowledge during the procedure of changes in the 3D shape of the vasculature due to, for example, cardiac motion, respiratory motion, and / or patient motion. Using the methods described herein integrated with a robotic system to handle the manipulation of intravascular devices, there is a good chance that patient outcomes can be improved, and advanced pre-procedure planning tools with robotic precision can bring new innovative standards of care to patients.
[0222] The operations can be performed by a processor unit of a stand-alone system, or by a processor unit of a semi-stand-alone system connected to an X-ray system (FIG. 2b) and described in detail with reference to FIG. 33, or directly included in an X-ray fluoroscopy system or other imaging system (FIG. 2a), for example for acquiring a sequence of 2D angiographic images. FIG. 17 shows an example of a high-level block diagram of an X-ray cine-fluoroscopy system. This block diagram shows an example of how an embodiment can be integrated into such a system.
[0223] Parts of the system (defined by the various functional blocks) may be implemented by dedicated hardware, analog and / or digital circuitry, and / or by one or more processors operating program instructions stored in memory.
[0224] The X-ray system of FIG. 17 includes an X-ray tube 1701 with a high voltage generator 1702 that generates an X-ray beam 1703. The high voltage generator 1702 controls and supplies power to the X-ray tube 1701. The voltage generator 1702 applies a high voltage to a vacuum gap between the cathode and a rotating anode of the X-ray tube 1701. The voltage applied to the X-ray tube 1701 causes an electron transfer from the cathode to the anode of the X-ray tube 1701, which causes an X-ray photon generation effect, also called Bremsstrahlung. The generated photons form an X-ray beam 1703 that is directed to an image detector 1706.
[0225] The X-ray beam 1703 comprises photons with a spectrum of energies ranging up to a maximum determined, inter alia, by the voltage and current applied to the X-ray tube 1701. The X-ray beam 1703 then passes through a patient 1704 lying on an adjustable table 1705. The X-ray photons of the X-ray beam 1703 penetrate the patient's tissue with various angulations. Different structures of the patient 1704 absorb the radiation differently, which modulates the beam intensity. The modulated X-ray beam 1703 emerging from the patient 1704 is detected by an image detector 1706 located on the opposite side of the X-ray tube. This image detector 1706 can be either an indirect or direct detection system.
[0226] For an indirect detection system, the image detector 1706 comprises a vacuum tube (X-ray image intensifier) that converts the X-ray emission beam 1703' into an amplified visible light image, which is then transmitted to a visible light image receptor, such as a digital video camera, for display and recording of the image, thereby producing a digital image signal.
[0227] In the case of a direct detection system, the image detector 1706 comprises a flat panel detector. The flat panel detector converts the x-ray emission beam 1703' directly into a digital image signal. The digital image signal obtained from the image detector 1706 is passed to a digital image processing unit 1707. The digital image processing unit 1707 converts the digital image signal from 1706 into a corrected (e.g., inverted and / or contrast enhanced) x-ray image in, for example, a DICOM standard image file format. This corrected x-ray image can then be stored on a hard drive 1708.
[0228] 17 further comprises a C-arm 1709. The C-arm holds the X-ray tube 1701 and the image detector 1706 such that the patient 1704 and adjustable table 1705 are between the X-ray tube 1701 and the image detector 1706. The C-arm can be moved (rotated and angulated) to a desired position to acquire a particular projection in a controlled manner using the C-arm control 1710. The C-arm control allows manual or automatic input to adjust the C-arm to a desired position for X-ray recording at a particular projection.
[0229] The X-ray system of Figure 17 can be either a single-plane or a bi-plane imaging system. In a bi-plane imaging system, there are multiple C-arms 1709, each consisting of an X-ray tube 1701, an image detector 1706, and a C-arm control 1710.
[0230] Additionally, the adjustable table 1705 can be moved using the table control 1711. The adjustable table 1705 can be moved along the x, y, and z axes and can be tilted about a particular point.
[0231] Furthermore, in this X-ray system there is a measurement unit 1713. This measurement unit contains information about the patient, for example ECG, aortic pressure, biomarkers and / or information about height, length etc.
[0232] There is also a general purpose unit 1712 in this X-ray system. This general purpose unit 1712 can be used to interact with a C-arm control 1710, a table control 1711, a digital image processing unit 1707 and a measurement unit 1713.
[0233] One embodiment is performed by the x-ray system of Figure 17 as follows: A clinician or other user acquires a sequence of at least two x-ray angiogram images of a patient 1704 by moving the C-arm 1709 to a desired position relative to the patient 1704 using the C-arm controls 1710. The patient 1704 lies on an adjustable table 1705 that the user has already moved to a particular position using the table controls 1711.
[0234] A sequence of X-ray images is then generated as described above using the high voltage generator 1702, the X-ray tube 1701, the image detector 1706 and the digital image processing unit 1707. These images are then stored on the hard drive 1708. Using these sequences of X-ray images, the general purpose processing unit 1712 performs the methods described in this application, e.g. as described in Figure 1, 15, 18, 26 or 28, using for example the measurement unit 1713, the digital image processing unit 1707, the C-arm control unit 1710 and the table control unit 1711.
[0235] As mentioned above, the operations can also be performed by a processor unit (FIG. 2b) of a semi-standalone system connected to an X-ray system. FIG. 33 shows such a setup. In FIG. 33, 3301 represents the X-ray system and 3302 represents the semi-standalone system in which the operations described by the present application are performed. To acquire a live X-ray image data stream, which can be either an X-ray angiography image data stream or a fluoroscopic image data stream, the semi-standalone system 3302 includes a frame grabber (3303) that allows digitization of live X-ray image data acquired from a video output from the X-ray system. Live access to the ECG signal is obtained by an analog-to-digital converter (3304) connected to the ECG signal output of the X-ray system or to another system that measures the patient's ECG. The roadmap overlay created by the method presented and described with respect to the flowcharts of FIG. 1, FIG. 15, FIG. 18, FIG. 26, or FIG. 28 is provided to the X-ray system video input (3304) and visualized by the X-ray system monitor.
[0236] Several embodiments of the method and apparatus for quantitative flow analysis have been described and illustrated herein. Although specific embodiments have been described, it is not intended to limit the invention thereto, as the invention is as broad as the art will permit, and the specification is intended to be read as such. For example, data processing operations can be performed offline on images stored in a digital storage such as a PACS commonly used in medical imaging. This is usually done in a universal (vendor independent) language such as DICOM (Digital Imaging and Communications in Medicine). The storage can be a hard disk, a PACS (picture archiving and communications system) server, a VNA (vendor neutral archive), or other image archiving and communication system commonly used in medical imaging. Thus, those skilled in the art will appreciate that still other modifications can be made to the provided invention without departing from its spirit and scope as set forth in the claims.
[0237] The embodiments described herein may include various data stores and other memory and storage media, as discussed above. These may be located in a variety of locations, such as storage media local (and / or resident) to one or more computers, or storage media remote from any or all computers on a network. In a particular set of embodiments, information may reside in a storage-area network ("SAN"), which is well known to those skilled in the art. Similarly, files necessary to perform functions specific to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computing devices, each device may include hardware elements that may be electrically coupled via a bus. The elements may include, for example, at least one central processing unit ("CPU" or "processor"), and at least one input device (e.g., mouse, keyboard, controller, touch screen, or keypad), and at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random access memory ("RAM") or read-only memory ("ROM") and removable media devices, memory cards, flash cards, etc.
[0238] Such devices may also include computer-readable storage media readers, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.) and the working memory described above. The computer-readable storage media readers may be configured to connect to or receive computer-readable storage media representing remote, local, fixed and / or removable storage devices, as well as storage media for storing, transmitting and retrieving computer-readable information on a temporary and / or more permanent basis. The systems and various devices also include several software applications, modules, services or other elements that are typically located in at least one working memory device, including operating systems and application programs such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations from those described above. For example, customized hardware may be used and / or certain elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices such as network input / output devices may also be used.
[0239] Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing description on a computer readable medium. Storage media and computer readable media for containing code or portions of code may include any suitable media known or used in the art. The suitable media include, but are not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storing and / or transmitting information, such as computer readable instructions, data structures, program modules, or other data. The storage media and communication media may include RAM, ROM, Electrically Erasable Programmable Read Only Memory ("EEPROM"), Flash memory or other memory technology, Compact Disc Read Only Memory ("CD-ROM"), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other media that may be used to store the necessary information and that may be accessed by a system device. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods for implementing the various embodiments.
[0240] Although the disclosed embodiments are described with respect to single or biplane X-ray imaging modalities, variations within these embodiments are also applicable to 3D reconstructions based on, for example, rotational angiography, computed tomography, magnetic resonance imaging, etc. Accordingly, the present specification and drawings are to be regarded as disclosed in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.
[0241] There are other variations that are within the spirit of the disclosure. Thus, the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above. However, it is not intended that the invention be limited to the particular forms disclosed. On the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.
[0242] The use of the terms "a" and "an" and "the" and similar articles in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected," when referring to a physical connection without modification, is construed as including, in whole or in part, in other words, attached or joined together, even if there is something intervening. Recitation of ranges of values herein is intended to serve merely as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein. Use of the term "set" (e.g., "set of items") or "subset" should be construed as a non-empty collection containing one or more members, unless otherwise stated or contradicted by context.
[0243] The operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more present applications) collectively executed on one or more processors. The code may be stored on a computer-readable storage medium, for example in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory.
[0244] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the invention. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventors anticipate that such variations will be utilized by those skilled in the art as appropriate, and the inventors intend that the embodiments of the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, the scope of the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is within the scope of the present disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
[0245] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.
Claims
1. 1. A method for dynamically visualizing information from first image data of an object of interest together with second image data of the object of interest, the object of interest comprising a heart, a portion of a coronary artery tree, a blood vessel, or other part of a patient's vascular system, the method comprising: the first image data is angiography image data acquired using an X-ray imaging modality with a contrast agent; the second image data is fluoroscopic image data acquired using an X-ray imaging modality without a contrast agent; i) using the first image data to generate a plurality of roadmaps of the object of interest, the first image data covering at least one cardiac cycle of the patient; ii) determining a plurality of reference locations of a tip of an interventional device, such as a guiding catheter, a guidewire or other intraluminal device or instrument, within the first image data, the plurality of reference locations corresponding to the plurality of roadmaps of the object of interest of i); iii) selecting a roadmap from the plurality of roadmaps; iv) tracking the location of the tip of the device within the second image data; V) using the reference position of the tip of the device corresponding to the one roadmap selected in iii) and the position of the tip of the device determined in iV) to transform the one roadmap selected in iii) to generate a dynamic roadmap of the target object; and vi) overlaying the visual representation of the dynamic roadmap of the object of interest generated in V) onto the second image data for display; Equipped with i) the plurality of roadmaps of the object of interest cover different phases of the cardiac cycle of the patient, the phases of the cardiac cycle of the patient being offset in time with respect to a predefined reference portion of the cardiac cycle of the patient; method.
2. processing an ECG signal synchronized with the second image data to determine a phase of the patient's cardiac cycle corresponding to the second image data; and selecting the one of the roadmaps (iii) by matching the phase of the cardiac cycle of the patient in the second image data to the phase of the cardiac cycle of the patient in the one of the roadmaps; The method of claim 1 , further comprising:
3. processing the first image data to determine a phase of the patient's cardiac cycle for an image frame, and associating the phase of the cardiac cycle with a roadmap corresponding to the image frame. The method according to any one of claims 1 to 2.
4. the plurality of roadmaps are derived from a three-dimensional model of the object of interest extracted from at least one imaging modality selected from the group consisting of computed tomography (CT), x-ray rotational angiography, 3D ultrasound, or magnetic resonance imaging (MRI); The method according to any one of claims 1 to 3.
5. the plurality of roadmaps are derived from two X-ray angiography image sequences of the object of interest acquired with a contrast agent; The method according to any one of claims 1 to 4.
6. the plurality of roadmaps are derived from a three-dimensional model of the object of interest and at least one X-ray angiography image sequence of the object of interest acquired with a contrast agent. The method according to any one of claims 1 to 5.
7. the plurality of roadmaps including information characterizing properties such as centerlines, contours, and / or image masks of the target object; The method according to any one of claims 1 to 6.
8. the plurality of roadmaps include at least one measurement of the object of interest selected from the group consisting of location and extent of vascular occlusion, diameter and area, pressure, blood flow velocity, fractional flow reserve, wall shear stress, vascular curvature, amount of distal foreshortening, location and extent and type of coronary plaque, location and extent of total coronary occlusion, or location and extent of coronary occlusion; The method according to any one of claims 1 to 7.
9. v) a reference location of the tip of the device within the second image data is determined using a Bayesian filtering method. The method according to any one of claims 1 to 8.
10. the Bayesian filtering method uses a deep learning network to estimate the likelihood of the location of the device within the second image data; and / or the Bayesian filtering method causes the position of the tip of the device to be equal to a weighted arithmetic average of a number of positions and their associated weights; and / or The Bayesian filtering method resamples points around locations with high weight values.
10. The method of claim 9.
11. V) applying a transformation to the roadmap to compensate for motion between the first and second image data, including respiratory motion and / or cardiac motion and / or patient motion and / or table motion; The transformation comprises a rigid or non-rigid transformation to the one roadmap based on a displacement obtained from a reference position of the device corresponding to the one roadmap selected in iii) and a position of the device determined in iV); The method according to any one of claims 1 to 10.
12. a visual representation of the dynamic roadmap; - projecting a superposition of said dynamic roadmap onto said second image data using a transmission mode; and / or - dilating the dynamic roadmap and projecting the boundary of the resulting dynamic roadmap onto the second image data; It is generated by whereby the visual representation of the dynamic roadmap is configured not to obscure the interventional device. The method according to any one of claims 1 to 11.
13. the plurality of reference locations are stored as part of a plurality of roadmaps of the target object, the selected roadmap comprising a three-dimensional roadmap that is transformed to generate at least one dynamic roadmap for overlaying onto the second image data. The method according to any one of claims 1 to 12.
14. The three-dimensional roadmap is transformed according to a viewpoint used to collect the second image data, or the second image data is collected from a different viewpoint than the first image data. The method of claim 13.
15. the first image data is derived by subtraction of a baseline image; and the second image data is obtained by subtraction of the baseline image; The method according to any one of claims 1 to 14.
16. A non-transitory program storage device tangibly embodying a program of machine-executable instructions for carrying out the method of any one of claims 1 to 15.
17. 1. A system for generating an image of an object of interest in a patient, comprising: When executing program instructions stored in memory, at least one processor: i) acquiring first image data of the object of interest, the first image data being acquired with a contrast agent and an interventional device being present in the first image data; ii) using the first image data to generate a plurality of roadmaps of the object of interest; iii) determining a plurality of reference locations of the tip of the device in the first image data, the plurality of reference locations corresponding to the plurality of roadmaps of the target object; iv) acquiring second image data of the object of interest, the second image data being collected without a contrast agent and the device being present in the second image data; V) selecting a roadmap from the plurality of roadmaps for the target object; vi) determining a location of the tip of the device within the second image data; vii) using the reference position of the tip of the device corresponding to the one roadmap selected in V) and the position of the tip of the device determined in vi) to transform the one roadmap selected in V) to generate a dynamic roadmap of the target object; viii) overlaying a visual representation of the dynamic roadmap of the object of interest generated in vii) onto the second image data for display; A system that is configured to run
18. 20. The system of claim 17, further comprising an imaging acquisition subsystem configured to acquire the first image data and the second image data using an X-ray imaging modality.
19. 19. The system of claim 17 or 18, further comprising a display subsystem configured to display a superposition of a visual representation of the dynamic roadmap of the object of interest generated in viii) onto the second image data.
Citation Information
Patent Citations
Medical radiographic apparatus
JP2006034355A
Image processing apparatus, x-ray diagnostic apparatus and program
JP2014004465A
Pairing live images with anatomical structure displays
JP2014533545A
Medical image processor
JP2015167790A
CLR2015
