Method and system for providing a dynamic roadmap of coronary arteries

Abstract

A method is provided for dynamically visualizing information within image data of a patient's target object, including both offline and online phases. [Solution] In the offline phase, a contrast agent is used to acquire first image data of the target object using an interventional device, multiple roadmaps covering different phases of the cardiac cycle are generated, and multiple reference positions for the corresponding devices are determined. In the online phase, live image data collected without the use of a contrast agent is acquired using devices present in the live image, a roadmap matching the phase of the cardiac cycle is selected from multiple roadmaps covering different phases of the cardiac cycle, the position of the devices in the live image data is determined, the roadmap is transformed to generate a dynamic roadmap, and a visual representation of the dynamic roadmap is superimposed on the live image data.

Description

Technical Field

[0001] This application relates to the technical field of medical imaging, particularly percutaneous intervention, but can also find use in any field where there is a need to provide a dynamic roadmap for image sequences in which the target structure is not well visible. Background Art

[0002] The trend toward minimally invasive procedures has increased the importance of imaging in clinical interventions. Because the incisions made for interventions are small, clinicians can no longer handle their tools using direct visual inspection alone and must instead rely on in-procedure images generated from real-time imaging modalities such as radiographic angiography and fluoroscopy, ultrasound, and in-procedure magnetic resonance imaging. Currently, radiographic angiography is the primary imaging modality used during the treatment of cardiovascular disease with minimally invasive procedures, also known as percutaneous coronary intervention (PCI). Radiographic angiography images are used, for example, to guide interventional procedures to treat atherosclerosis. Atherosclerosis is the accumulation of cholesterol and fatty deposits (called plaque) on the inner walls of arteries. These plaques can restrict blood flow to the heart muscle by physically blocking the arteries or by causing abnormal arterial tone and function. If the heart does not receive enough blood, it will lack oxygen and essential nutrients necessary for proper functioning. This can cause chest pain known as angina. If the blood supply to a portion of the heart muscle is completely cut off, or if the heart's energy demands are far greater than the blood supply, a heart attack (damage to the heart muscle) or stroke can occur. Atherosclerosis affects the arteries of the heart, head, neck, and surrounding parts of the body and is treated using a variety of methods. The most common methods, such as angioplasty, bare-metal stent placement, drug-eluting stent placement (permanently implantable and biodegradable), various types of energy delivery, and rotational atherectomy, all treat the arteries evenly around the target length of the arterial lumen.

[0003] During PCI, an interventional cardiologist injects a deflated balloon or other device on a catheter through the femoral or radial artery until it reaches the site of arterial occlusion. PCI is typically performed by inflating the balloon to open the artery, usually with the aim of restoring unobstructed blood flow. A stent or scaffold can be placed at the site of occlusion to keep the artery open. During these procedures, medical devices inserted via a guide catheter are advanced, for example, to treat coronary artery stenosis. The guide catheter is initially placed at the opening of the coronary artery. The guide catheter then guides a balloon catheter carrying a stent to the stenotic site via a guidewire. PCI is usually performed under image guidance using radiographic angiography. The coronary arteries are visualized with an opaque contrast agent. Because the opacity of the coronary arteries is only temporary, the interventional cardiologist may repeatedly inject the contrast agent to visualize the vessels during the procedure. The amount of contrast agent used during the perioperative period correlates with the operator's experience, the complexity of the procedure, renal function, and imaging settings. Furthermore, the risk of contrast-induced nephropathy is also related to the amount of contrast agent used. However, the manipulation of guidewires and materials is usually performed without continuous injection of contrast agent. In such situations, device navigation is guided by "vascular-free" fluoroscopic images. The cardiologist must reconstruct the positions of vessels and stenoses based on previous angiographic images.

[0004] For example, in minimally invasive interventions such as PCI, image guidance and visualization can become a bottleneck in the physician-instrument-patient feedback loop. Image-guided navigation, performed by visual inspection of intraprocedural images, inherently suffers from limitations in accuracy due to subjective evaluation by the operator and operator bias. The impact of this problem is even more pronounced when navigating moving areas such as the chest. Furthermore, intraprocedural images produced by real-time imaging modalities are usually of reduced quality and do not always represent anatomical structures relevant to the clinical procedure. Contrast agents can be used to visualize anatomical structures. However, to prevent harm to the patient, intraprocedural administration of contrast agents, including additional X-ray exposure, should be limited. For example, the duration and amount of X-ray exposure to the patient, as well as the potentially toxic contrast agent injected into the patient's bloodstream, should be reduced. Most such contrast agents are used to highlight the aorta and coronary arteries in fluoroscopic images to visually guide the physician. For example, if contrast agent is injected during PCI, the coronary artery tree will be visible, and if no contrast agent is injected, the coronary artery tree will not be visible. Dynamic coronary artery roadmapping is a promising solution to improve visual feedback and reduce the use of contrast agents during PCI. Developing a dynamic coronary artery roadmapping system requires accurately overlaying the coronary artery roadmap onto fluoroscopic images, but this presents challenges because there is limited vascular information within the target fluoroscopic image for inferring compensation for vascular motion due to the patient's respiration and heart rate. Methods proposed for motion compensation in dynamic coronary artery roadmapping can generally be classified into two categories: direct roadmapping methods and model-based approaches.

[0005] Direct roadmapping methods use information from X-ray images and electrocardiogram signals to directly correct for motion caused by respiration and heartbeat. For example, Patent Document 1 uses digital differential processing of contrast (contrast) and mask sequences to create a complete cardiac cycle of a coronary artery roadmap. This roadmap is saved and later synchronized to a live fluoroscopy sequence by aligning it with the R wave of the corresponding electrocardiogram signal. This system corrects for cardiac motion in the vessels but not for respiratory motion or patient motion during intervention. Furthermore, this must be performed on a cardiac gate frame. Patent Document 2 discloses an X-ray diagnostic device for differential processing angiography. This utilizes a single mask frame and integrates it over the full cycle. By integrating the mask over the entire cardiac cycle, positional displacement errors are simply distributed and not minimized.

[0006] Unlike direct roadmapping methods, model-based approaches construct a model for predicting motion within a fluoroscopic frame. This motion model is often one or more functions relating the motion of the roadmap to surrogate signals derived from images or electrocardiograms; therefore, once the surrogates of the fluoroscopic frame are acquired, the motion can be computed by the motion model. In cardiac interventions, including PCI, organ motion is primarily influenced by respiration and cardiac motion. Previous techniques, for example, as taught by Non-Patent Document 1, constructed motion models parameterized by cardiac signals derived from ECG and respiratory signals obtained from diaphragm tracking. A limitation of model-based approaches is that the motion model is patient-specific, requiring the model to be trained for each new subject. Furthermore, if surrogate values ​​during inference fall outside the surrogate range that constitutes the model (e.g., if the motion is abnormal), extrapolation is required, hindering accurate motion compensation.

[0007] Therefore, to mitigate the aforementioned drawbacks, it is desirable to track the motion of the coronary artery tree in fluoroscopic images without contrast enhancement. The use of this method would better support cardiologists during PCI and reduce the patient's exposure to contrast agents and X-ray radiation. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] U.S. Patent No. 4,878,115 [Patent Document 2] European Patent No. 0,193,712 [Patent Document 3] U.S. Patent No. 9,256,936 [Patent Document 4] U.S. Patent Application Publication No. 16 / 379,248 [Patent Document 5] European Patent Application Publication No. 3206183 [Patent Document 6] U.S. Patent No. 10,229,516 [Patent Document 7] U.S. Patent Application Publication No. 16 / 438,955 [Patent Document 8] U.S. Patent Application Publication No. 16 / 256,793 [Patent Document 9] U.S. Patent No. 9,576,360 [Patent Document 10] U.S. Patent No. 10,192,352 [Non-patent literature]

[0009] [Non-Patent Document 1] Shechter, "Anterior motion correction of X-ray images for coronary intervention," IEEE Transactions on Medical Imaging 2005:24, 441-450. [Non-Patent Document 2] Frangi, "Multiscale Vascular Enhancement Filtering," Medical Image Computing and Computer-Assisted Intervention - MICCAI 1998 Lecture Notes in Computer Science 1496 / 1998:130 [Non-Patent Document 3] Ma, "A fast forward detection network and recurrent neural network for contrast agent inflow in X-ray angiography using convolutional neural networks," International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 453-461. [Non-Patent Document 4] "Skeling of Grayscale Images from Imperfect Boundaries" by Li, Proceedings of the International Conference on Image Processing, ICIP 2008, October 12-15. [Non-Patent Document 5] "Outline of Medical Image Registration Methods" by Maintz, Symposium of the Belgian Hospital Physicists Association, 1996. [Non-Patent Document 6] Hao, "Vascular layer separation in X-ray angiography using a fully convolutional network," Proc. SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling. [Non-Patent Document 7] Ma, "Layer separation for vascular enhancement in radiographic angiography of interventions using morphological filtering and robust PCA," Workshop on Augmented Environments for Computer-Assisted Interventions 2017, Springer, pp. 104-113. [Non-Patent Document 8] Ma (Ma) - "Automated online layer separation for vascular enhancement in radiographic angiography for percutaneous coronary intervention," Med Image Anal. 2017 Jul;39:145-161. [Non-Patent Document 9] "Catheter Attention Machine for Occlusion Landmark Detection in 2DX Angiography," by Zang, presented at the Jan 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). [Non-Patent Document 10] Ambrosini, et al., "Fully Automated and Real-Time Catheter Segmentation in Fluoroscopy," International Conference on Medical Image Computing and Computer-Assisted Intervention 2017, Springer. pp. 577-585. [Non-Patent Document 11] Nasr-Esfahani et al., "Vascular Extraction in X-ray Angiography Using Deep Learning," Conf Proc IEEE Eng Med BiolSoc 2016 Aug;2016:643-646. [Non-Patent Document 12] "Advances in two-dimensional quantitative coronary angiography of branched lesions: Improved detection of small lumen diameter and automated reference vessel diameter derivation," by Girasis, EuroIntervention 2012 Mar; 7(11):1326-35. [Non-Patent Document 13] "Vascular Extraction in Coronary X-ray Angiography," by Wang, Conf Proc IEEE Eng Med Biol Soc. 2005; 2: 1584-1587. [Non-Patent Document 14] Kim, et al., "Registration of Angiographic Images to Real-Time Fluoroscopic Images for Image-Guided Percutaneous Coronary Intervention", International journal of computer assisted radiology and surgery 2018:13, 203-213 [Non-Patent Document 15] Arulampalam, et al., "A Tutorial on Particle Filters for Online Nonlinear / Non-Gaussian Bayesian Tracking", IEEE Transactions on signal processing 2002:50, 174-188 [Non-Patent Document 16] Ronneberger, et al., "U-net: Convolutional Networks for Biomedical Image Segmentation", International Conference on Medical image computing and computer-assisted intervention 2015, Springer. pp. 234-241 [Non-Patent Document 17] Milletari, et al., "V-net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation", 2016 Fourth International Conference on 3D Vision (3DV), IEEE. pp. 565-571 [Non-Patent Document 18] He, et al., "Deep Residual Learning for Image Recognition", 2016 Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770- 778 [Non-Patent Document 19] Laina, et al., "Simultaneous Segmentation and Localization for Tracking of Surgical Instruments", 2017 International conference on medical image computing and computer-assisted intervention, Springer. pp. 664-672

Non-Patent Document 20

Non-Patent Document 21

Non-Patent Document 22

Non-Patent Document 23

Non-Patent Document 24

Non-Patent Document 25

Non-Patent Document 26

[0010] Accordingly, an object of the embodiments herein is to provide a computer implementation method for dynamically visualizing information from first image data of a target object together with second image data of the target object. This method comprises the following steps: i) A step of using the first image data to generate multiple roadmaps for the target object; ii) A step of determining a plurality of reference positions in the first image data for a device such as a guide catheter, guide wire, or other intraluminal device or instrument, wherein the plurality of reference positions correspond to the plurality of roadmaps of the target object in i); iii) The step of selecting one roadmap from the aforementioned multiple roadmaps; i) A step of determining the position of the device within the second image data; V) the step of using the reference position of the device corresponding to the roadmap selected in iii) and the position of the device determined in iV) in order to convert the one roadmap selected in iii) to generate a dynamic roadmap of the target object; and vi) A step of overlaying the visual representation of the dynamic roadmap of the target object generated in V) onto the second image data for display.

[0011] The related systems and program storage devices are also described and billed.

[0012] In this embodiment, the first image data covers at least one cardiac cycle of the patient, and the multiple roadmaps of the generated target object cover different phases of the patient's cardiac cycle.

[0013] In this embodiment, the phases of the patient's cardiac cycle are temporally offset from a predefined portion of the patient's cardiac cycle.

[0014] In embodiments, these methods may further include, for example, processing an ECG signal synchronized with a second image data acquired together with the second image data to determine the phase of the patient's cardiac cycle corresponding to the second image data. The roadmap can be selected by matching the phase of the patient's cardiac cycle relative to the second image data with the phase of the patient's cardiac cycle relative to a selected roadmap.

[0015] In embodiments, these methods may further include processing first image data to determine the phase of the patient's cardiac cycle relative to the image frame, and associating the cardiac cycle phase with a roadmap corresponding to the image frame.

[0016] In the embodiment, the multiple roadmaps for the target object may include multiple two-dimensional roadmaps or multiple three-dimensional roadmaps. For example, multiple three-dimensional roadmaps may be derived from a three-dimensional model of the target object. In another example, multiple three-dimensional roadmaps may be derived from two sequences of X-ray angiography images of the target object acquired with a contrast agent. In yet another example, multiple three-dimensional roadmaps may be derived from a three-dimensional model of the target object and a sequence of X-ray angiography images of the target object acquired with a contrast agent.

[0017] In the embodiment, multiple roadmaps may include information that characterizes the properties of the target object (such as a center line, contour, and / or image mask).

[0018] In embodiments, multiple roadmaps may include at least one measurement for the object of interest. For example, the at least one measurement may be selected from a group consisting of the location and extent of vascular occlusion, diameter and area, pressure, blood flow velocity, fractional flow rate, wall shear stress, vascular curvature, distal shortening, location and extent, coronary plaque type, location and extent of complete coronary occlusion, or location and extent of coronary occlusion of the object of interest.

[0019] In some embodiments, the location of a device in a second image can be determined using a Bayesian filtering method. In one example, the Bayesian filtering method can use a deep learning network to estimate the likelihood of the device's location in the second image. In another example, the Bayesian filtering method can be configured to determine the device's location in the second image by equaling the weighted arithmetic mean of multiple locations and their associated weights. In yet another example, the Bayesian filtering method can be configured to resample points around locations with high weight values.

[0020] In the embodiment, operation V) applies a transformation to a selected roadmap to compensate for motion between the first image data and the second image data. For example, motion may include respiratory motion and / or cardiac motion and / or patient motion and / or table motion.

[0021] In the embodiment, the transformation applied to the selected roadmap can be a rigid or non-rigid transformation based on the displacement obtained from the reference position of the device corresponding to the selected roadmap and the position of the device in the second image data.

[0022] In this embodiment, a visual representation of a dynamic roadmap can be generated by projecting a superposition of the dynamic roadmap onto a second image data using a transparency mode, and / or by inflating the dynamic roadmap and projecting the resulting boundaries of the dynamic roadmap so that the visual representation of the dynamic roadmap does not obscure the equipment used to process the object of interest.

[0023] In this embodiment, multiple reference points can be saved as part of multiple roadmaps for the target object.

[0024] In the embodiment, the selected roadmap may be a three-dimensional roadmap that is transformed to generate at least one dynamic roadmap for superimposing on a second image data. For example, the three-dimensional roadmap may be transformed according to the viewpoint (e.g., field of view) used to collect the second image data. This allows the second image data to be collected from a viewpoint different from the first image data.

[0025] In other embodiments, a particular roadmap may be a two-dimensional roadmap that is transformed to generate at least one two-dimensional dynamic roadmap for superimposing on a second image data. In this case, the first and second image data can be collected from a common viewpoint.

[0026] In this embodiment, the first image data can be derived by subtraction processing of baseline images, and the second image data can be obtained by subtraction processing of baseline images.

[0027] In this embodiment, the first image data is angiographic image data acquired using an X-ray imaging modality with a contrast agent, and the second image data is fluorescence image data acquired using an X-ray imaging modality without a contrast agent.

[0028] In this embodiment, operations iii) to vi) can be repeated for consecutive frames of a live image sequence acquired without contrast agent.

[0029] In one embodiment, for dynamic guidance, a roadmap can be selected for each consecutive image frame of the second image data for transformation and superposition within the image frame of the second image data. In another embodiment, for static guidance, one roadmap (the same roadmap) can be selected for consecutive image frames of the second image data for transformation and superposition within the image frame of the second image data.

[0030] In the embodiment, the device is selected from the group consisting of a guide catheter, a guidewire, or other intraluminal devices or instruments.

[0031] In this embodiment, the location of this device corresponds to the end or tip of the device.

[0032] In the embodiment, the object of interest is the heart, a portion of the coronary artery tree, a blood vessel, or another part of the vascular system.

[0033] In embodiments, these methods may further include displaying a superimposed visual representation of the dynamic roadmap of the target object on the image data of iV).

[0034] Embodiments of this specification also relate to a method for visualizing information within image data of a patient's object. This method comprises the following steps: i) A step of acquiring first image data of the target object, wherein the first image data is collected by a contrast agent and an interventional device is present in the first image data; ii) A step of using the first image data to generate multiple roadmaps for the target object; iii) A step of determining a plurality of reference positions of the device in the first image data, wherein the plurality of reference positions correspond to the plurality of roadmaps of the target object; iV) A step of acquiring a second image data of the target object, wherein the second image data is collected without the use of a contrast agent, and the device is present in the second image data; V) The step of selecting one roadmap from the multiple roadmaps of the target object; vi) A step of determining the position of the device in the second image data; vii) The step of using the reference position of the device corresponding to the roadmap selected in V) and the position of the device determined in vi) in order to convert the one roadmap selected in V) to generate a dynamic roadmap of the target object; and viii) For display purposes, the visual representation of the dynamic roadmap of the target object generated in vii) is superimposed onto the second image data.

[0035] Embodiments of this specification also relate to a system for generating images of a patient's object, which includes at least one processor configured to perform the operation of the method according to the embodiments herein when executing program instructions stored in memory.

[0036] In one embodiment, at least one processor performs the following steps: i) A step of acquiring first image data of the target object, wherein the first image data is collected by a contrast agent and an interventional device is present in the first image data; ii) A step of using the first image data to generate multiple roadmaps for the target object; iii) A step of determining a plurality of reference positions of the device in the first image data, wherein the plurality of reference positions correspond to the plurality of roadmaps of the target object; iV) A step of acquiring a second image data of the target object, wherein the second image data is collected without the use of a contrast agent, and the device is present in the second image data; V) The step of selecting one roadmap from the multiple roadmaps of the target object; vi) A step of determining the position of the device in the second image data; vii) The step of using the reference position of the device corresponding to the roadmap selected in V) and the position of the device determined in vi) in order to convert the one roadmap selected in V) to generate a dynamic roadmap of the target object; and viii) For display, the visual representation of the dynamic roadmap of the target object generated in vii) is superimposed on the second image data. It is configured to perform the following:

[0037] In some embodiments, the system further comprises an imaging acquisition subsystem, typically using an X-ray imaging modality, configured to collect first image data and second image data.

[0038] In one embodiment, the system further comprises a display subsystem configured to display a superimposed visual representation of the dynamic roadmap of the target object generated in viiii) on the second image data.

[0039] Further improvement is the objective of the dependent claim. [Brief explanation of the drawing]

[0040] [Figure 1] A flowchart of a method for determining a dynamic coronary artery roadmap according to embodiments of this specification is shown. [Figure 2a] This document shows a functional block diagram of an exemplary single-plane angiography system according to the embodiments of this specification. [Figure 2b] A functional block diagram of an X-ray system connected to a system including embodiments of this specification is shown. [Figure 3] The diagram shows the dynamic roadmap process. [Figure 4] The retrieved angiography image data is shown. [Figure 5] This flowchart shows a method for obtaining the device position within an X-ray angiography image sequence. [Figure 6] This example shows a method for detecting the start of a contrast agent injection frame within an X-ray image sequence. [Figure 7] This paper presents several approaches to extracting the cardiac cycle within a series of X-ray image frames. [Figure 8] An example of layer separation in an X-ray angiography frame is shown. [Figure 9] Examples of acquisition times for cardiac cycles within several fluoroscopic image frames and sequences are shown. [Figure 10a] An example of dynamic load mapping is shown. [Figure 10b] Here's another example of dynamic roadmapping. [Figure 11] Here are some examples of labels used to train deep learning networks. [Figure 12] This shows a collaborative segmentation and detection network for catheter tip detection. [Figure 13] This is a summary of an algorithm that tracks the tip of a catheter using a deep learning-based Bayesian filtering method. [Figure 14] Four examples of tip tracking methods for Bayesian filtering catheters based on deep learning are presented. [Figure 15] This flowchart shows a method for determining a dynamic coronary artery roadmap, where image enhancement is used to improve the identification and processing of target objects. [Figure 16] An example of image enhancement using differential processing is shown. [Figure 17] An example of an X-ray cine fluoroscopy unit block diagram according to the embodiments of this specification is shown. [Figure 18] A flowchart illustrating the method for determining a dynamic 3D coronary artery roadmap is shown. [Figure 19] This section presents several examples of generating 3D models based on 3D angiography image data. [Figure 20] This shows an example of a coronary vessel in which WSS is superimposed on the luminal surface. [Figure 21]This shows an example of a coronary vessel where pressure and coronary flow reserve ratio are superimposed on the luminal surface. [Figure 22a] This shows a guidance map regarding the proximity, shortening, overlap, and intra-plane coronary artery motion of surrounding vessels for a given cardiac phase. [Figure 22b] This shows a guidance map regarding intra-plane coronary motion, including distance-to-proximity shortening, overlap of surrounding vessels, and multiple cardiac phases. [Figure 23] This provides a guidance map for selecting the optimal second projection for the first projection. [Figure 24] Here are some examples of a single 3D roadmap frame. [Figure 25] Two examples of 3D segmentation of the coronary artery lumen, including segmentation of coronary artery plaque, are shown. [Figure 26] A flowchart of alternative methods for determining a dynamic coronary artery roadmap is shown. [Figure 27] The diagram shows the repositioning step. [Figure 28] A flowchart illustrating a method for determining the location and amount of calcified plaque is shown. [Figure 29] This shows a typical example of an X-ray angiography imaging sequence in which a contrast agent is administered during the sequence. [Figure 30] This paper presents a high-level method for determining the location and amount of calcified plaque. [Figure 31] This shows that the skeleton is broken down into separate vascular branches. [Figure 32] This shows an example where calcified plaque is highlighted. [Figure 33] This shows a setup in which the operation is performed according to the embodiments herein by a system processor unit connected to an X-ray system. [Figure 34a] This shows an example of a roadmap overlaid on X-ray fluoroscopy image data. [Figure 34b] This shows an example of a roadmap overlaid on X-ray angiography image data. [Figure 35]This indicates the integration of device locations into the roadmap. [Figure 36] An example of the tracking results of the proposed alternative method for an exemplary image frame is shown. [Figure 37] The proposed tracking method is shown below. [Modes for carrying out the invention]

[0041] This application describes a method and system for providing a real-time dynamic overlay or dynamic coronary roadmap that can be superimposed on a live fluoroscopic or angiographic image stream / sequence, thereby supporting clinicians in improving patient care. This approach corrects for changes in vascular geometry and cardiac motion by selecting roadmaps of the same cardiac phase through temporal alignment, and corrects for respiration-induced motion and patient motion by tracking the device.

[0042] In the following descriptions of methods and systems, radiographic angiography will be used to mean radiographic acquisition of a target object (e.g., a portion of the vascular system or a coronary artery tree) after administration of a contrast agent, which results in visualization of the target object and other objects that are radiopaque. Radiographic fluoroscopy means acquiring radiographic images without the use of a contrast agent, and therefore does not involve highlighting the visualization of the target object. The term image or image frame means a single image, and the term image sequence or image stream means multiple images acquired over time. A radiographic angiographic image sequence may include contrast agent administration, and image frames preceding contrast agent administration do not include contrast agent and therefore do not include highlighting of the vascular system. A radiographic angiographic image sequence may comprise multiple frames covering one or more phases of a cardiac cycle. Throughout this patent application, a cardiac cycle is defined as a period that is unique to a patient and covers one heartbeat in that patient. A cardiac cycle can be defined as the period between consecutive R-wave peaks in the patient's ECG signal. A phase means an instant (or period) within the patient's cardiac cycle. The phase can be measured as an offset from the peak of the R wave in the patient's ECG signal, as shown in Figure 9 and its corresponding description.

[0043] Within this application, a method is proposed for generating and displaying a dynamic overlay or dynamic roadmap as an overlay on a live X-ray image stream, comprising the following elements: • Construct a model of the target organ from angiographic image sequences of the target organ (such as a coronary artery tree). • Use the patient's electrocardiogram (ECG) measurement to select a portion (or frame) of the model that is aligned with the patient's cardiac phase. • Update or transform selected parts of the model to correct respiratory motion in the patient's live fluoroscopy or angiography image stream using guided catheter tracking. • Render a portion of this resulting model, and then integrate the rendered view of the model as an overlay of the patient's live fluoroscopy or angiography image stream. The dynamic coronary artery roadmap method described is executed in real time using a graphics processing unit (GPU), so it can be used during PCI in actual clinical settings.

[0044] Figure 1 shows a flowchart illustrating an operation according to one embodiment of the present application. These operations utilize an imaging system capable of acquiring and processing a two-dimensional image sequence of a vascular organ (or part thereof) or other target object. Single-plane or double-plane angiography systems can be used, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD).

[0045] Figure 2a is a functional block diagram of an exemplary single-plane angiography system. This system includes an angiography imaging device 212 which will operate under commands from a user interface module 216 and provide data to a data processing module 210. The single-plane angiography imaging device 212 captures a two-dimensional X-ray image sequence of a target vascular organ, for example, in the occipital-frontal direction. The single-plane angiography imaging device 212 typically includes a pair of X-ray sources and detectors mounted on the arms of a support gantry. The gantry allows the arms of the X-ray sources and detectors to be positioned at various angles relative to a patient supported by a table between the X-ray sources and detectors. The data processing module 210 can be implemented by a personal computer, workstation, or other computer processing system. The data processing module 214 processes the two-dimensional image sequence captured by the single-plane angiography imaging device 212 to generate the data described herein. The user interface module 216 interacts with the user and communicates with the data processing module 210. The user interface module 216 may include a display screen for visual output, a touchscreen for touch input, a mouse pointer or other pointing device for input, a microphone for voice input, a speaker for voice output, a keyboard and / or keypad for input, etc. The data processing module 210 and the user interface module 216 work together to perform the operations shown in Figures 1, 15, 18, 26, or 28 below.

[0046] The operations in Figures 1, 15, 18, 26, or 28 can also be executed by software code embodied in a computer product (e.g., optical disc or other form of persistent memory, USB drive, or network server). The software code can be loaded directly into the memory of a data processing system to perform the operations in Figures 1, 15, 18, 26, or 28. Such a data processing system can also be physically separated from the angiography system used to acquire images by utilizing any type of data communication to obtain such images as input.

[0047] In this example, it is assumed that the imaging system has already collected and stored at least one two-dimensional image sequence of the target object. Any imaging device capable of providing a two-dimensional angiography image sequence can be used for this purpose. For example, a biplane or single-plane angiography system, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD), can be used.

[0048] Figure 2b is an exemplary alternative functional block diagram according to embodiments herein, including an X-ray system 202 that operates under commands from a user interface module and provides data to a data analysis module 203. The X-ray system 202 collects X-ray image data of a target area, such as the heart. The data analysis module 203 can be implemented by a personal computer, workstation, or other computer processing system. The data analysis module 203 processes the X-ray image data collected by the X-ray system 202 to generate, for example, a coronary artery roadmap or a quantification of coronary artery analysis. The user interface module 201 interacts with the user and communicates with the data analysis module 203. The user interface module 201 may include a display screen for visual output, a touchscreen for touch input, a mouse pointer or other pointing device for input, a microphone for voice input, a speaker for voice output, a keyboard and / or keypad for input, etc. Module 204 represents a dynamic coronary artery roadmap, which represents a coronary artery tree calculated from the collected X-ray image data, as described in this application. The dynamic coronary artery roadmap is superimposed in real time onto the X-ray image sequence acquired by the X-ray system module 202. Module 204 can be implemented by a personal computer, workstation, or other computer processing system and interacts with the X-ray system 202.

[0049] Herein, with reference to Figure 1, one embodiment is disclosed. The operations shown herein can obviously be performed in any logical order and can be partially omitted. Since the object of this application is to provide a workflow that can be used during an intervention, the steps of the example workflow will also be referenced. In a preferred embodiment, the method of this application assumes a scenario in which dynamic coronary roadmapping is performed to guide a PCI procedure. As shown in Figure 1, the workflow comprises several steps representing an offline phase and an online phase. The offline phase, represented by block 100 in Figure 1, uses an X-ray angiography sequence acquired from the injection of contrast agent over multiple cardiac phases to create a set of coronary roadmaps. The online phase, represented by block 110 in Figure 1, uses the set of roadmaps created in the offline phase, combined with an X-ray angiography sequence acquired without the injection of contrast agent, to create a dynamic roadmap based on tracking the position of the device. The visual representation of the dynamic roadmap is superimposed on a live fluoroscopic image stream of an X-ray fluoroscopic image sequence to support the placement of one or more medical devices inserted by a guide catheter and to provide visual guidance to the physician. For example, the medical device may be a guidewire, an inflatable balloon, a stent, or other suitable intravascular device or apparatus.

[0050] Before describing each step in Figure 1 in detail, we will briefly explain the substeps of the offline and online phases at a high level.

[0051] Offline phase: The offline phase, represented by block 100 in Figure 1, is performed before the actual roadmapping is carried out. In the offline phase, a roadmap of the coronary arteries across multiple phases of the cardiac cycle is created from a sequence of radiographic angiography images covering multiple phases of the cardiac cycle. Typically, at least one cardiac cycle is considered. Along with the radiographic angiography sequence, temporal information of the cardiac cycle (e.g., ECG signals) is acquired and stored. The roadmap contains information characterizing the properties of the coronary arteries at a particular phase of the cardiac cycle. Such information may include the midline of the coronary arteries, the contours (lumen boundaries) of the coronary arteries along their respective lengths, and / or mask images representing the coronary arteries (e.g., images covering the space occupied by the coronary arteries). In embodiments, the roadmap may include a vascular model in the form of midlines, contours, and / or mask images representing the coronary arteries at a given phase of the cardiac cycle. One or more roadmaps may also include clinically interesting information such as the location and percentage of vascular occlusion, diameter and area, pressure, blood flow velocity, flow reserve, wall shear stress, vessel curvature, amount of distal-proximal shortening, location, amount and type of coronary plaque (e.g., calcified, soft plaque, mixed plaque), location and extent of complete coronary occlusion, or location and extent of coronary occlusion. The creation of the roadmap will be described in detail in step 103 of Figure 1.

[0052] The device position (e.g., catheter tip position) within the X-ray angiography sequence is acquired, and this position is associated with the roadmap to serve as a reference point for roadmap conversion (as described in more detail by step 107 in Figure 1). This reference point (device position) can then be integrated into the roadmap, as described in detail in step 102 in Figure 1.

[0053] Online phase: Dynamic roadmapping is actually performed during the online phase, represented by block 110 in Figure 1. During the online phase, a sequence of fluoroscopic image data is collected without the use of contrast agents and preferably at the same field of view (C-arm angulation and C-arm rotation) as when the roadmap was created during the offline phase (see step 104 in Figure 1). Simultaneously, the ECG signal is acquired along with the fluoroscopic image data and compared with the stored ECG to select the roadmap that best matches the one created in the offline phase (step 105 in Figure 1). This is to compensate for changes in the shape and position of blood vessels between frames due to cardiac motion. At the same time, the device position (e.g., catheter tip position) in the collected fluoroscopic image data is tracked using image processing techniques and, in a preferred embodiment, using the proposed deep learning-based Bayesian filtering method described in step 106 in Figure 1. The displacement of the device position (e.g., catheter tip) between the current frame of the fluoroscopic image data and a reference point (device position) associated with the selected roadmap is obtained and applied to transform the selected roadmap to generate a dynamic roadmap as shown in step 107 of Figure 1. Finally, a visual representation of the dynamic roadmap is rendered and superimposed on the current frame of the fluoroscopic image data to guide the PCI procedure (step 108 of Figure 1).

[0054] The dynamic roadmap process is illustrated in Figure 3. In Figure 3, the offline phase is represented by photographs 301 and 302. Here, 301 shows a sequence of radiographic angiography images of the right coronary artery, and 302 shows the roadmap generated from the radiographic angiography sequence (301). During the online phase, as shown by photographs 303 and 304, the generated roadmap (302) is superimposed on the radiographic fluoroscopy image stream (303), resulting in the dynamic coronary artery roadmap (304). For illustrative purposes, only one frame (representing one moment in the cardiac cycle) is shown in Figure 3.

[0055] The following sections provide a more detailed explanation of the steps shown in Figure 1.

[0056] Step 101: Read the angiography image data. In the first step, step 101, angiography image data is read out. In a preferred embodiment, the angiography image data represents the acquisition of a target object by X-ray angiography (which consequently produces an X-ray angiography image sequence). Single-plane or double-plane angiography systems can be used, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD). The target object is, for example, the heart, part of the coronary artery tree, or a blood vessel, and the target object is visualized by the contrast agent during acquisition. The X-ray angiography image sequence is acquired so that the target object is clearly visible. Thus, the C-arm can be rotated and angled by the clinician to obtain the best projection. Preferably, an electrocardiogram (ECG) is part of the angiography image data and is recorded simultaneously during the acquisition of the X-ray angiography. The ECG enables matching of cardiac phases between different X-ray angiography frames or between X-ray angiography acquisitions. By matching the cardiac phase, the displacement of the target object due to cardiac motion between image frames within the same cardiac phase is minimized.

[0057] Figure 4 provides an illustration of the readout angiographic image data, consisting of an X-ray angiographic image sequence including the ECG signal. Photo 401 shows

number

number

number

number

[0058] Step 102: Obtain the device location in the angiography image data. Numerous factors can cause displacement of the target object or roadmap between consecutive X-ray image frames within an X-ray angiography image sequence, or between different X-ray angiography image sequences. These factors include cardiac motion due to cardiac contraction, respiratory motion due to patient breathing, and patient motion due to patient movement. Cardiac motion can be compensated for, for example, by matching the cardiac phase using the readout ECG signal. Respiratory motion, including possible patient motion, can be corrected by identifying reference points in the images. Reference points (represented by devices visible within the image sequence) will be used to transform the target object or roadmap during the online phase. Reference points can be, for example, the tip of a catheter, a pacemaker or anatomical landmark, or any other object whose motion can be correlated with respiratory motion and possible patient motion. Such reference points can be acquired in all frames of the X-ray angiography image sequence. Identification of reference points within frames may be performed manually. Instead of manually identifying reference points, image processing techniques may be applied to detect objects or landmarks within medical image data, for example, as described in step 503 of Figure 5.

[0059] In a preferred embodiment, the steps for obtaining the device position within the X-ray angiography image sequence are shown by the flowchart in Figure 5. In step 501 of Figure 5, the frame in the X-ray angiography image sequence in which the contrast agent enters the coronary artery tree (406 in Figure 4) is detected. Figure 6 shows how to detect the start of the contrast agent injection frame. For each frame in the X-ray angiography image sequence, the average pixel intensity is calculated and visualized by graph 601. For example, the average pixel intensity of the first frame (602) is represented in 603, and the frame

number

number

number

number

number

number

[0060] Another approach to detecting frames in an X-ray angiography image sequence in which contrast agent enters the coronary artery tree is the method taught in Non-Patent Literature 3. In this study, Non-Patent Literature 3 describes two different approaches to detecting frames in an image sequence in which contrast agent first appears. The first approach trains a convolutional neural network (CNN) to distinguish whether or not a frame contains contrast agent. The second approach extracts the first contrast agent feature from images with enhanced vascular structure and then detects the contrast agent frame based on changes in the feature curve using long short-term memory established by a recurrent neural network architecture.

[0061] Next, in step 502, cardiac cycle information is extracted from the X-ray angiography image sequence. This step is omitted if the ECG signal is part of the angiography image data (101 in Figure 1). As previously mentioned, coronary motion includes cardiac motion due to cardiac contraction, respiratory motion due to patient respiration, and patient motion due to patient movement. To extract cardiac cycle information, only the motion of the coronary arteries and / or devices such as guide catheters should be considered. Therefore, surrogate extraction of cardiac motion by analyzing the motion patterns of objects representing cardiac motion, such as coronary artery trees or catheters, provides information about the cardiac cycle within the X-ray angiography image sequence. In Figure 7, 701 represents several consecutive X-ray image frames within the X-ray angiography image sequence. The motion of the diaphragm 704 is visible within the consecutive X-ray image frames 701.

[0062] One approach to extracting cardiac cycle information is shown in Figure 7, 702. Image 702 shows coronary arteries and catheters extracted from image frames 701. This can be achieved by applying a Frangi vascular filter, and then non-maximal suppression (as taught, e.g., by Non-Patent Literature 4) (Non-Patent Literature 2). The coronary arteries extracted between each consecutive frame are then registered with each other (by a 2D registration technique as taught, e.g., by Non-Patent Literature 5), and this information is then collected.

number

[0063] Image 703 in Figure 7 illustrates another approach to extracting cardiac cycle information. In this scenario, these images are processed to remove structures that do not represent cardiac motion and to highlight blood vessels. In this scenario, cardiac motion becomes the main factor in the intensity changes of the image sequence and can be further analyzed using a method with source decomposition capabilities, such as principal component analysis. First, the image sequence 701 is processed so that only blood vessels and devices such as catheters are highlighted (703). This can be done using the stratification method taught in Non-Patent Literature 6, or Non-Patent Literature 7 or Non-Patent Literature 8. Principal component analysis is typically used for dimensionality reduction. This transforms a multivariate dataset into a new Cartesian coordinate system so that most of the variance of this dataset can be represented by a few coordinates. Thus, this dimensionality reduction is usually achieved without losing much information by retaining only a few coordinates in the new coordinate system. Principal component analysis is used on the post-processed image (703) to obtain the principal components of the image sequence. The frames of the sequence are

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0064] Next, the post-processed sequence (703) is:

number

number

number

number

number

number

number

number

[0065] Referring back to Figure 5, the next step is to detect the device in the X-ray angiography image sequence over one cardiac cycle.

number

number

number

[0066] frame

number

number

[0067] Another method for detecting devices (e.g., catheter tips) can be performed by model-based detection or by using convolutional neural networks. Catheter tip detection may be improved by incorporating temporal information into pre-processing or post-processing. A fully automated catheter segmentation technique incorporating temporal information based on convolutional neural networks is taught in Non-Patent Document 10.

[0068] Another method for detecting the device (e.g., the tip of a catheter) is to first measure the amount of consecutive frames representing one cardiac cycle prior to the frame in which the contrast agent enters the coronary artery (as shown in step 501 of Figure 5) in the angiography image sequence.

number

number

[0069] Returning to Figure 1, the next step (103) is to create a roadmap. Within step 103, the device locations are integrated into the created roadmap so that they serve as reference points for the roadmap transformation (described in more detail by step 107 in Figure 1).

[0070] Step 103: Create a roadmap In step 103, the X-ray angiography image sequence is processed to create a coronary artery roadmap for multiple phases of the cardiac cycle after the frame in which the contrast agent entered the coronary arteries. Typically, the roadmap is created for a certain number of frames that cover at least one cardiac cycle.

[0071] For example, a coronary artery roadmap may include a vascular model. The vascular model may represent the midline of the vessel, the boundaries (contours) of the vessel, or a mask image representing the vessel. Furthermore, the vascular model may also include clinically relevant information such as the location and percentage of vascular occlusion, and / or the diameter, area, length or curvature of the vessel, and / or the location and amount of calcified plaque.

[0072] In this embodiment, a coronary artery roadmap can be created for all frames within a single cardiac cycle after contrast agent injection. For example, a complete cardiac cycle (the result of step 501 in Figure 5) (and

number

number

[0073] Figure 8 shows an example of layer separation on an X-ray angiography frame. Compared to the original X-ray angiography image frame (801, Figure 8), it can be seen that the coronary arteries are significantly enhanced in the vascular layer (804, Figure 8). Such layer separation methods are taught, for example, in Non-Patent Literature 6, or Non-Patent Literature 7 or Non-Patent Literature 8. In Figure 3, 302 provides an example of creating a roadmap of the X-ray image 302 by layer separation.

[0074] Another approach to creating a roadmap is to apply the image processing framework techniques taught in Non-Patent Document 4, for example. Another approach to creating a roadmap is to utilize the deep learning techniques taught in Non-Patent Document 11 or Non-Patent Document 13, for example.

[0075]

number

number

[0076] Optionally, quantitative image analysis can be performed based on the generated roadmap, as taught in Non-Patent Document 12 or Non-Patent Document 13, to extract clinically relevant information such as the location and percentage of vascular occlusion, diameter and area, vessel length, or vessel curvature. Optionally, the location and amount of calcified plaque can also be extracted from the radiographic angiography image sequence, for example, as further disclosed in this application by the flowchart in Figure 28.

[0077] Finally, in step 103, the device location obtained as a result of step 102 is integrated into the created roadmap. Figure 35 illustrates how to integrate the device location into the roadmap using an example. The first example represents the creation of a roadmap by layer separation (3501) as described above. Within 3501, the device location (3502) as a result of step 102 is integrated into the vascular layer (3501). The final roadmap showing the vascular structure (3503) also includes the device location (3502). The second example represents the creation of a roadmap by skeletalization (3506, and superimposed on the X-ray angiography frame by 3505 for illustrative purposes) as described above. Within 3505, the device location (3504) as a result of step 102 is shown.

[0078] Step 104: Collect perspective image data. The first step of the online phase (110) is the readout of fluoroscopic image data, represented by step 104 in Figure 1. In a preferred embodiment, the fluoroscopic image sequence is acquired in real time as a live stream. Simultaneously, the patient's ECG signal can be acquired in real time and synchronized with the fluoroscopic image sequence. The fluoroscopic images are acquired sequentially by step 101 with the same projection (field of view) as used during the offline phase (100). Using the same projection ensures that the target object and roadmap have the same orientation as the fluoroscopic angiography image sequence (101) used to generate the roadmap (103). Figure 9 shows three fluoroscopic image frames (901, 902, and 904) from the fluoroscopic image sequence. A catheter can be seen within the fluoroscopic image frame (904) and is highlighted for description. Each fluoroscopic frame is acquired within any moment of the cardiac cycle, indicated by 906, 907, and 908 in the electrocardiogram (ECG) signal representing the cardiac cycle (909). The start of the cardiac cycle is, for example, the peak of the R wave (910), and all acquired X-ray frames (906, 907, and 908) represent phases within the cardiac cycle (909), with each being offset relative to the start of the cardiac cycle (910) for acquiring the respective X-ray frame. It is preferable to acquire the ECG signal simultaneously with the acquisition of the X-ray frames. During the online phase (110), the ECG signal is used to match the acquired frames with the corresponding roadmap extracted in step 103 to represent the same cardiac phase in the cardiac cycle. This is to compensate for changes in the shape and position of blood vessels between fluoroscopic image frames due to cardiac motion, which is further explained in step 105 of Figure 1. Simultaneously, the device (e.g., the position of the catheter tip) in the collected fluoroscopic image stream is tracked using a deep learning-based Bayesian filtering method, for example, as further described in step 106.Next, the displacement of the catheter tip between the current fluoroscopic image frame and the selected roadmap is obtained and applied to transform the roadmap, which will be further described in step 107. Finally, the transformed roadmap is superimposed on the current fluoroscopic image frame to guide the procedure described in step 108.

[0079] Step 105: Select a roadmap After reading the fluoroscopic images containing the ECG signals described in step 104, the roadmap selected from the roadmap (created in step 103) is selected for all single fluoroscopic images.

[0080] The selection of a roadmap, as represented by step 104 in Figure 1, can be achieved by comparing the ECG signal associated with the online fluoroscopic image with the ECG of the offline angiography sequence, so that the optimal candidate roadmap is selected, which finds the best match of the ECG signal. The selected roadmap has the same (or very similar) cardiac phase as the online fluoroscopic image, which compensates for differences in vascular shape and posture caused by cardiac motion.

[0081] To select a roadmap and images based on ECG, it is first necessary to establish a temporal mapping between X-ray images and ECG signal points. It is assumed that the electrocardiogram signals and X-ray images are well-synchronized during acquisition.

[0082] In the offline phase (block 100 in Figure 1), the start and end of the image sequence are aligned with the start and end of the ECG signal points. The X-ray angiography frames between these are evenly distributed on the ECG timeline. In this way, a mapping can be established between the saved sequence images and their ECG signals. For each image, the ECG signal point closest to the image's position on the timeline can be found; similarly, for each ECG point, the image closest to this point on the timeline can be positioned. Once the mapping is available, all images with good contrast agent injection and the ECG points associated with these images are selected from the X-ray angiography sequence in the roadmap pool. This process requires the collection of at least one heartbeat per frame.

[0083] In addition, by applying the method in step 501, images taken before the contrast agent reaches the blood vessels can be discarded. This will speed up the mapping process between ECG data and image sequences.

[0084] In the online phase (block 110 in Figure 1), N is used to collect each image. ECG The latest ECG signal point blocks are always saved. These N ECG The ECG signal point is considered to be the ECG signal corresponding to the online fluoroscopy frame.

[0085] To compare ECG signals associated with offline angiography sequences and online fluoroscopic images, temporal registration of the two signals is applied, for example, using cross-correlation. This is taught in Non-Patent Literature 14. The two ECG signals are first cross-correlated for all possible locations on the signals, resulting in a one-dimensional (1D) vector of correlation scores. Next, candidate frames for dynamic superposition and roadmap are selected as frames associated with points on the ECG of the offline angiography sequence corresponding to the highest correlation scores.

[0086] Step 106: Track the device In the online phase (block 110 in Figure 1), transformation of the target object or roadmap is required to compensate for respiratory motion and patient motion. Therefore, in step 102, the device position is acquired in the angiographic image data. Simultaneously, the roadmap selection in step 105 in Figure 1 tracks the same device positioned in step 102 in the readout fluoroscopic image in step 106. The device can be, for example, the tip of a catheter, a pacemaker, or an anatomical landmark or other object whose motion can be correlated with respiratory motion and possible patient motion.

[0087] Next, a method for tracking a device is presented, and as an example, the object to be tracked is the tip of a catheter.

[0088] Tracking the tip of an exemplary catheter The overall tracking of the catheter tip is performed using a deep learning-based Bayesian filtering method, summarized in Figure 13. This method uses a convolutional neural network to model the likelihood term of Bayesian filtering and integrates it into particle filtering in a comprehensive manner, which results in robust tracking. In summary, for every position in an image, the new position of the catheter tip (in the new image) (predicting the catheter's motion) can be predicted using the optical flow method (see line 6 in Figure 13) and added noise (see line 5 in Figure 13). Furthermore, the weights are updated by checking the likelihood of the position using a deep learning network (see line 8 in Figure 13). Next, all weights are normalized (see line 10 in Figure 13). The actual catheter tip position is equal to the weighted arithmetic mean of all positions and their weights. Finally, point resampling is performed around positions with high weight values ​​(see line 12 in Figure 13). The tracking method is then described in more detail.

[0089] Bayesian filtering Bayesian filtering is a state-space approach aimed at estimating the true state of a system as it changes over time from a sequence of noisy measurements performed on the system, as described, for example, in Non-Patent Document 15.

[0090] Bayesian filtering typically includes the following components: hidden system states, state transition models, observations, and observation models.

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

Number

Number

Number

Number

Number

[0091] These definitions and

Number

Number

Number

Number

Number

Number

Number

[0092] Initial probability

Number

Number

Number

Number

Number

[0093] Deep learning-based likelihood Likelihood

Number

Number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0094] To obtain training labels,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0095] The network used follows an encoder-decoder with skip connections similar to the U-net described in Non-Patent Document 16, for example. In addition, residual blocks are employed at each resolution level of the encoder and decoder, as in the study in Non-Patent Document 17, thereby facilitating gradient propagation in the deep network. Residual blocks are essentially a special case of a highway network without gates in the skip connections. Essentially, residual blocks allow for the flow of memory (or information) from the first layer to the last layer, as described in Non-Patent Document 18, for example.

[0096] The encoder consists of four downblocks, followed by a residual block with a stride 2 convolution, which is used for extracting and downscaling feature maps. The number of feature maps doubles with each downsampling step. The decoder has four upblocks, where a stride 2 transposed convolution is used for upsampling the input feature map. Dropout is used in the residual units of the upblocks for network regularization. Another residual block is used between the encoder and decoder to process the feature maps extracted by the encoder. A detailed network architecture is shown in Figure 12. Figure 12 shows a joint segmentation and detection network for catheter tip detection and an example of a network with four levels of depth (number of downblocks or upblocks). The meaning of the abbreviations in Figure 12 is as follows:

number

number

number

number

number

number

number

number

number

[0097] Because the tip of the guide catheter and the corners of background structures such as vertebrae, lung tissue, stitches, and guidewires have similar appearances, ambiguity can exist if the network is expected to output only one blob in the probability map. To mitigate this problem, a strategy similar to that taught in Non-Patent Literature 19 is employed to jointly train the network using a catheter mask (1102, Figure 11) as an additional label to output both a catheter segmentation heatmap and a likelihood probability map. The segmentation heatmap is obtained by applying a 1x1 convolution with normalized linear function (Relu: rectified linear units) activation to the feature map of the last up block. To compute the likelihood probability map, first, the residual block is applied to the feature map of the last up block. Next, the output feature map is concatenated to the segmentation heatmap as one additional channel, followed by a 1x1 convolution. Finally, to ensure that the network detection output fits the definition of a probability map of image locations, a spatial softmax layer is applied following a 1x1 convolution, as shown in Equation 5.

number

number

number

number

number

number

number

[0098] Training loss is defined as the combination of segmentation loss and detection loss. Segmentation loss for this task

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0099] Approximation of posterior probability using particle filters Once a deep neural network is trained, its weights are used to calculate the posterior probability of new data.

number

number

number

[0100] The particle filter method is associated with weights

number

number

number

number

number

number

number

number

number

number

number

number

[0101] The sample is,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0102] Once samples are extracted and their weights are updated, so-called "resampling" of the samples must be performed to prevent degeneration problems. In this case, the weights of all but one sample will become negligible after several iterations. The resampling step involves resampling the existing samples according to the updated weights, and then the weights of all samples are updated.

number

number

number

[0103] frame

number

number

number

[0104] Figure 14 presents four examples (sequences 1 to 4) of the catheter tracking method described herein. The first column shows online fluoroscopic images, and the second column shows the position of the catheter tip obtained from the aforementioned tracking method as a point. The last column on the right shows the ground truth of the catheter tip.

[0105] The final catheter tip position obtained from the tracking method is used in the next step 107 to determine the conversion between the device position in the online fluoroscopic image and the corresponding offline angiographic image of the selected roadmap.

[0106] In an alternative embodiment, step 106 can be achieved by teaching in Patent Document 3, “Method and Apparatus for Tracking Objects in a Target Region of a Moving Organ.” Patent Document 3 establishes dynamic geometric relationships between the positions of tracked features using the properties of synchronized periodic motion. This is achieved by learning the periodic motion of individual features, such as other visible devices commonly present in the image view during percutaneous coronary intervention procedures. If one of the features is hidden, knowledge of the periodic motion patterns of both features, the position of one of the features, and the phase of the periodic motion can be used to derive the position of the hidden feature. If the first feature is the target location of the device itself (e.g., the tip of a catheter), such a position is determined immediately. If the first feature is not the target location but a feature that moves in the same way as the target location (e.g., another visible device), the position of the target location of the device can be derived from the position of the first feature using the fact that the first feature moves in the same way as the target region. This requires knowledge of the geometric relationship between the first feature and the target region.

[0107] Step 107: Convert the selected roadmap to generate a dynamic roadmap As described herein, reference points can be used to compensate for motion between a roadmap acquired from an angiography image sequence (offline phase) and a fluoroscopic image stream (online phase). A reference point, which can be extracted from a device location, can be any object whose motion can be correlated with respiratory motion and patient motion, and this device can be, for example, a catheter tip, a pacemaker, or an anatomical landmark. In a preferred embodiment, this device is a guide catheter, and the location of this device is the tip of the guide catheter. Within step 107, the location of the device (e.g., catheter tip) in the current fluoroscopic frame as a result of step 106, and the location of the device (e.g., catheter tip) from the selected roadmap frame as a result of step 105, are used to obtain a transformation function that aligns the selected roadmap to the current fluoroscopic image frame. In embodiments, this transformation function can be a rigid transformation based on the displacement acquired between the catheter tip in the current frame and the catheter tip in the selected roadmap frame. Alternatively, the transformation function can be a non-rigid transformation.

[0108] For example, the rigid body transformation of a roadmap can be performed using a rigid body transformation function. The original roadmap (as a result of step 105) is transformed into a function.

number

number

number

number

number

number

[0109] Step 108: Overlay a dynamic roadmap onto perspective image data. The dynamic roadmap, derived from the cardiac matching and transformation results of the roadmap, is rendered and integrated so that it can be overlaid on the corresponding fluoroscopic or angiography image data frames of the patient's live angiography image data stream.

[0110] Examples of dynamic roadmaps superimposed on contrast-free images are shown in Figures 10a and 10b. Figure 10a shows an example of dynamic roadmapping. The image on the left shows the extraction of the coronary artery tree, and the image on the right shows the coronary artery tree superimposed on a single frame within a fluoroscopic image sequence, with dynamic motion such as respiratory motion, patient motion, and cardiac motion corrected.

[0111] Figure 10b shows another example. The image on the left shows a single frame within a sequence of fluoroscopic images, and the image on the right shows the same fluoroscopic image with a dynamic roadmap superimposed, corrected for dynamic motions such as respiratory motion, patient motion, and cardiac motion.

[0112] This method provides real-time dynamic overlay or roadmapping where a visual representation of the dynamic roadmap is rendered and superimposed onto a live fluoroscopy or angiography image stream / sequence, thereby supporting clinicians in improved patient care.

[0113] In the embodiment, step 108 can be achieved by rendering the transformed roadmap (dynamic roadmap) resulting from step 103 according to the viewpoint of the current live image in the patient's live image stream / sequence. Pixels in the rendering of the dynamic roadmap corresponding to the target organ (e.g., coronary artery tree) can be assigned color values ​​within a predefined range, such as a range of color values ​​from red to white. In addition to coloring the dynamic roadmap, transparency can also be applied to the rendered model. By applying transparency to the model, a dual view is provided in which the fluoroscopic image and the dynamic roadmap are displayed. To enable visual recognition of stents, balloons, or other devices or instruments for treating diseased arteries and to enable visual recognition of the projected overlay, the overlay can be projected onto the fluoroscopic image data in transparent mode, as shown by 3410 in Figure 34a. It is clear that the degree of transparency can be adjusted. In Figure 34a, 3411 provides another example where the overlay is projected without obscuring the therapeutic instrument used to treat the affected artery, and in this example, the therapeutic instrument is a balloon (3412). This is achieved by displaying only the outer boundary of the roadmap image, which can be generated by first inflating the roadmap by a predefined amount and then extracting the boundary of the inflated roadmap. Of course, this can also be done in transparency mode, where the degree of transparency can be adjusted.

[0114] The mapping of the model (vascular layer image in Figure 10a) to a dynamic roadmap superimposition can be performed by converting pixel intensity to color values ​​(e.g., red to white, see example in Figure 10a). Here, white represents the dark pixel intensity of the blood vessels (Figure 10a, right image), and red represents the lighter, darker intensity of the blood vessels (Figure 10a, right image).

[0115] The color scheme can also represent quantitative parameters (e.g., geometric parameters such as the local curvature and local diameter of the vessels). These geometric parameters can be derived, for example, from the vessel model created in step 103. Another example of quantitative parameters is pathological parameters such as the amount of calcified plaque in the lumen, which is derived in step 103 and described in more detail in the flowchart of Figure 28. After step 108, the method returns to step 104 to process the next fluoroscopic image frame, and the flowchart shown in Figure 1 terminates when the reading of fluoroscopic image data stops, as described in step 104.

[0116] Experimental setup for dynamic coronary artery roadmapping This section provides an example for training a device tracking method, as described in step 106 of Figure 1. This example includes training a deep neural catheter detection network, as described in Figure 12, and tuning a Bayesian filtering tracking method, as described in the deep learning-based algorithm in Figure 13. Furthermore, several experiments are described using the dynamic coronary artery roadmapping workflow shown in the flowchart of Figure 1. The following four databases are used here: 1. Training dataset for catheter tip detection. This dataset consists of a large number of X-ray fluoroscopy image sequences and

number

number

number

[0117] Since images can be collected using different X-ray systems and different X-ray imaging protocols, the image dimensions (number of rows and columns) and pixel depth (range of pixel intensity) can vary. Therefore, all images in the dataset are grid-based.

number

number

number

number

[0118] All experiments conducted in this section are outlined as follows: First, the method for training a deep neural network to obtain optimal detection is described. Next, the training of catheter tip tracking (tuning of particle filters) is described. Finally, the evaluation of the trained catheter tracking is described.

[0119] Training of deep neural networks To train a deep neural network to provide an accurate likelihood probability map, the dataset

number

number

[0120] Data augmentation is performed to increase the number and diversity of training samples. Augmentation includes geometric transformations such as flipping (left / right, up / down), rotations in multiples of 90 degrees, random affine transformations (translation, scaling, rotation, shear, etc.), and random elastic deformations. In addition to geometric transformations, data augmentation can also be performed by adding Gaussian noise to pixel values ​​to make the trained model robust to noise.

[0121] During training, two terms (Equation 8)

number

number

number

[0122] Training a deep neural network to produce a reasonable likelihood probability map is performed by tuning the network's hyperparameters. Evaluation metrics are needed to select hyperparameters and model weights during training. Since deep networks are essentially catheter tip detectors, accurate detection of tip position is desirable. For example, the average Euclidean distance between ground truth and the predicted tip position averaged across all validation frames can be used as a validation criterion for selecting the optimal training epoch and network hyperparameters.

[0123] Catheter tip tracking training The catheter tip is tracked in fluoroscopic images using the algorithm described in Figure 12, based on a network trained with optimal hyperparameter settings from the previous section, "Training the Deep Neural Network." The parameters of the optical flow method used in the algorithm described in Figure 12 and particle filtering (in the section "Approximation of Posterior Probability by Particle Filter" described in step 107 of Figure 1) are set to the dataset.

number

[0124] When the approach in Non-Patent Document 20 is used as the optical flow method (Equation 11), a grid search to find the optimal parameter settings is performed based on the following parameters of the Fameback method: (1) the image scale for constructing the pyramids, (2) the number of pyramid levels, (3) the average window size, (4) the number of iterations, (5) the size of the pixel neighborhood used to find the polynomial expansion of each pixel, and finally (6) the Gaussian distribution used to smooth the derivative used as the basis for the standard deviation polynomial expansion. Since the optical flow method directly estimates the motion of the catheter tip between two frames, the above parameters are tuned independently of the deep neural network. To tune the parameters, a catheter tip tracked in a fluoroscopy room was tracked using the motion field between two adjacent frames estimated by the optical flow method, starting from the ground truth tip position in the first frame. The mean and median distances between the tracked tip position and the ground truth are used as criteria for evaluating the tuning.

[0125] The parameters for tuning the particle filter are the number of samples.

number

number

[0126] Evaluation of catheter tip tracking The tracking method proposed by the algorithm shown in Figure 13 uses the ground truth chip probability map of the first frame as an initial PDF.

number

[0127] The first option uses only the detection network described in the section "Deep Learning-Based Likelihood" in step 107 of Figure 1, and tracks the catheter tip with the selected network architecture and trained parameters described in the section "Training for Catheter Tip Tracking." Therefore, this method does not use temporal information. This method is called "Detection (Net)."

[0128] Two other methods in this experiment track the catheter tip from the ground truth tip position in the first frame using only the optical flow method. The motion field toward the current frame, estimated by these two methods, was based on deformation from the previous frame or the first frame of the sequence, respectively. These are referred to as "Optical Flow (Previous)" and "Optical Flow (First)."

[0129] The tracking accuracy of all methods reported in this section is based on the dataset.

number

[0130] Figure 37 shows how the tracking method proposed by step 107 in Figure 1 works on the same four frames in Figure 36. In Figure 37, high probabilities in the detection map are shown in bright colors, and samples or particles are shown as green dots. Blue points indicate the predicted catheter tip location; red points indicate the ground truth location (label). Figure 37 also shows that the previous hypothesis (sample) helps to focus on the correct target location and provides a reliable post-hoc estimate, especially when detection is obscured by multiple catheters or contrast agents remaining in the image.

[0131] In an alternative embodiment, the set of roadmaps and dynamic roadmaps generated by this process can be a three-dimensional (3D) roadmap represented by the flowchart in Figure 18. The advantages of using 3D, which results in a sequence of 3D roadmaps (3D+t), are that a) the X-ray field of view, magnification, and table position can be made different in the offline phase (1800) and the offline phase (1810), thereby giving the physician more degrees of freedom during interventional procedures; b) supplementary information such as plaque type obtained by non-invasive 3D imaging can be included in the 3D roadmap; and c) advanced quantitative analysis such as calculation of coronary flow reserve ratio (cFFR) or a combination thereof can be obtained from the calculation of wall shear stress (WSS), or from non-invasive 3D imaging, or from 3D reconstruction of the target vessel from multiple X-ray angiography sequences. The steps in Figure 18 will be described in detail in the following section.

[0132] Step 1801: Generate a 3D model from 3D angiography image data. In the optional step 1801, a 3D model of (most of) the coronary artery tree is initially generated. This is done using data collected from 3D angiography imaging modalities such as computed tomography (CT), rotational angiography, 3D ultrasound, or magnetic resonance imaging (MRI). This 3D model can take the form of, for example, a 3D centerline, 3D surface contours representing the luminal and / or extravascular surfaces of the vessels, plaques, 3D masks, or a combination thereof.

[0133] 3D centerlines can be created manually, for example, by indicating the vascular centerlines in 3D volumetric image data, or automatically, as taught, for example, in Non-Patent Document 22.

[0134] The detection of coronary artery plaque from coronary artery lumen, arterial wall, and CT angiography image data can be (semi)automatically detected, for example, as taught in Non-Patent Document 23, which describes a method for detecting coronary artery lumen, arterial wall, and coronary artery plaque from CT angiography image data.

[0135] Coronary artery plaque can be detected, for example, as disclosed in Patent Document 4 (Method and System for Evaluating Vascular Occlusion Based on Machine Learning), or as taught, for example, by Non-Patent Documents 24, 25, or 26.

[0136] Figure 19 shows several examples of generating 3D models based on 3D angiography image data. Photo 1901 shows coronary arteries extracted from both the left coronary artery (LCA) and the right coronary artery (RCA), including a portion of the ascending aorta, based on CT. Photo 1902 is similar to 1901, except for the ascending aorta. Within Photo 1903, a portion of the luminal vascular tree and atherosclerotic plaque (1905) detected based on MRI are shown, and 1906 shows the detected luminal boundary (vascular inner wall; representing the vascular lumen) and arterial boundary (external wall) of the vessel. Figure 25 provides two examples of 3D segmentation of the coronary artery lumen, including segmentation of coronary artery plaque. This can be extracted from 3D angiographic image data as described in step 1803 of Figure 18, or by 3D coronary artery reconstruction based on X-ray angiography, and calcified plaques can be extracted from X-ray image data as described in the flowchart of Figure 28.

[0137] Optionally, cFFR along the coronary artery tree can be calculated using 3D angiographic CT image data, for example, as disclosed in Patent Document 4, which discloses a method for calculating the coronary flow reserve ratio along the coronary arteries. Optionally, coronary WSS, or time-averaged WSS as taught in Non-Patent Document 27, can be calculated using a 3D model extracted from either CT, rotational angiography, or MRI. Figure 20 shows an example of a coronary vessel with WSS superimposed on the lumen surface. Figure 21 shows an example of a coronary vessel with pressure and coronary flow reserve ratio superimposed on the lumen surface. The generated 3D model may include all the elements described above within step 1801. Except for rotational angiography and biplane X-ray angiography, it is clear that the data cannot be collected or created during the intervention. Therefore, this data must be available on the server, for example, when the intervention is initiated. This generated 3D model will be used later.

[0138] Step 1802: Read X-ray angiography image data. In step 1802, the X-ray angiography image data is read out, which is similar to step 101 in Figure 1. In a preferred embodiment, the angiography image data represents the acquisition of the target object by X-ray angiography, which consequently becomes an X-ray angiography image sequence. For example, a single-plane or double-plane angiography system can be used, such as those manufactured by Siemens (Artis zee Biplane) or Philips (Allura Xper FD). The target object is, for example, the heart, part of the coronary artery tree, or a blood vessel, and the target object is visualized by the contrast agent during acquisition. Preferably, an electrocardiogram (ECG) is part of the angiography image data and is recorded simultaneously during the acquisition of the X-ray angiography. The X-ray angiography image sequence is acquired so that the target object is clearly visible. Thus, the clinician can rotate and angulate the C-arm to obtain the best projection, also known as the optimal projection.

[0139] For the creation of the 3D roadmap described in step 1803, it is important that the X-ray angiography image sequence is obtained from the correct perspective, defined as the angulation of the X-ray system (both rotation and angulation of the system), which contains as much information as possible about the target segment. In this perspective, the shrinkage and overlap of surrounding vessels are minimized. Shrinkage is an event that distorts information, causing an object to appear compressed when viewed from a particular perspective. The perspective in which the target object is viewed with the least shrinkage is called the optimal perspective. In a preferred embodiment, the 3D model obtained from the 3D angiography image data modality described in step 1801 is used to suggest the optimal perspective to the user with respect to the shrinkage and overlap of surrounding vessels. The orientation of this 3D model relative to the X-ray system is known. The optimal fluoroscopy with respect to the least shrinkage is determined as the perspective perpendicular to the orientation of the 3D model or its section. This model is viewable from various angles where everything is vertical, allowing for a wide range of optimal perspectives.

[0140] However, the optimal perspective depends not only on minimizing near-far shortening but also on the overlap of surrounding blood vessels. Therefore, the measurement of this overlap is also considered. Since the movement of the heart itself or the respiratory movement can cause surrounding blood vessels to overlap with the target segment at a particular moment, the overlap of surrounding blood vessels can be for one or more cardiac phases.

[0141] The 3D model extracted as a result of step 1801 is back-projected onto a 2D plane representing a specific perspective, as instructed by Non-Patent Document 28. Every point within the target blood vessel of the 3D model is assigned a specific value. For each 3D point, that value is added to the corresponding 2D point in the back-projected image. The plane with the largest number of 2D points containing the value is the most desirable perspective in terms of having the least overlap.

[0142] In addition, perspectives can also be shown in terms of the minimum intra-plane coronary motion. This perspective indicates the target vessel with the least amount of intra-plane coronary motion in that perspective. This allows clinicians to visualize the target vessel in the most stationary position possible. The measured amount of intra-plane coronary motion for each perspective can be determined, for example, by back-projecting a 3D model extracted from CT data onto a 2D plane representing a specific perspective, as taught in Non-Patent Document 28. The position of all centerline points of each vessel in the 3D model is known. Next, the 3D model extracted from CT data can be deformed using the model taught in Non-Patent Document 29 to generate a 3D model at a different point in time. This deformed 3D model is then back-projected onto a 2D plane representing a specific perspective. Again, the position of each centerline point is known, but it is now a different point in time. For each centerline point, the intra-plane coronary motion can be determined by comparing the position of each centerline point in both retrospective projections. The plane in which the in-plane motion is minimized for all centerline points is the most desirable perspective in terms of minimal intra-plane coronary motion. This can be done for one or more cardiac phases.

[0143] Next, for every combination of angulation and rotation (and thus each perspective), it is possible to indicate to what extent the resulting perspective is optimal. This indication is, for example, a weighted sum of distal shortening, overlap of surrounding vessels, and / or intra-plane coronary motion for at least one cardiac phase. If multiple cardiac phases are considered, the calculation is performed for each time point per frame. Next, a weighted sum of all frames within multiple cardiac phases is created, giving an overall indication of how optimal the perspective is. This can be visualized, for example, using the color map shown in Figure 22. Figure 22a shows a color map for distal shortening, overlap of surrounding vessels, and intra-plane coronary motion for one cardiac phase, while Figure 22b shows a grayscale map for distal shortening, overlap of surrounding vessels, and intra-plane coronary motion including multiple cardiac phases. This map can obviously also be displayed as a color map. Using these results, the user can select a perspective from which to collect X-ray angiography image sequences. In a preferred embodiment, two X-ray angiography image sequences are acquired. However, if a 3D model is available, at least one X-ray angiography image sequence is required.

[0144] If a 3D model is unavailable, two X-ray angiography image sequences are required. The first X-ray angiography image sequence is similar to that described in step 101 of Figure 1, and it is assumed that the physician has selected the optimal projection to display the target object. The second projection can be acquired simultaneously with the acquisition of the first projection, and this can be done using a biplane X-ray angiography system. Otherwise, the second projection can be acquired by a single-plane X-ray system. The second projection is acquired after the first projection. The user can receive support in selecting the optimal second projection relative to the first projection. A projection map can be generated based on the first projection (2301) and the orientation of the target object in the first projection (2302) (2303 in Figure 23). This method is described in detail in Patent Document 5.

[0145] Step 1803: Create a 3D roadmap Within this step, a sequence of 3D roadmaps (3D+t) is created, which is similar to step 103 in Figure 1. This step (1803) results in the generation of a 3D roadmap of the coronary arteries covering multiple phases of at least one cardiac cycle. The 3D roadmap of the coronary arteries can be, for example, a vascular model. The vascular model may take the form of a centerline, contour, mask, etc. This vascular model may then include clinically relevant information such as the location and percentage of vascular occlusion, the diameter and area of ​​the vessel, length, pressure, blood flow velocity, coronary flow reserve ratio, wall shear stress, curvature of the vessel, location, amount and type of coronary plaque, or location and amount of calcified plaque. In contrast to the 2D roadmap used in the flowchart of Figure 1, the current workflow uses a 3D roadmap. Since the 2D roadmap is defined within the coordinate system of the X-ray images used to generate the 2D roadmap, the 3D roadmap is defined in a 3D world coordinate system, and therefore arbitrary X-ray geometry is possible. The 3D roadmap is created from frames within one cardiac cycle after contrast agent injection. Thus, the complete cardiac cycle is selected within the X-ray angiography image sequence after the frame in which the contrast agent entered the coronary artery. The frame in the X-ray angiography image sequence in which the contrast agent entered the coronary artery can be defined by the method described above by step 501 in Figure 5. Within step 1803, the 3D+t roadmap can be generated by three different methods, and these methods mainly depend on the available data from steps 1801 and 1802.

[0146] If step 1801 is not performed, a first method (1803a) for creating a 3D+t roadmap is applicable, and the 3D+t roadmap is created by processing two X-ray angiography image sequences obtained from the results of step 1802. A 3D+t roadmap for all frames within one cardiac cycle after contrast agent injection can be generated, for example, by the method taught in Non-Patent Literature 30 or Non-Patent Literature 31. An example of a single 3D roadmap frame obtained as a result of this step is provided by 2402 in Figure 24. If the X-ray angiography image sequences are acquired by a biplane system, the temporal resolution of the 3D+t roadmap can be improved by using the delay between each frame acquired by the frontal and lateral imaging sources.

[0147] A second method (1803b) for creating a 3D+t roadmap becomes applicable, for example, based on the method taught by Non-Patent Literature 32, when a 3D model is available as a result of step 1801 and a single X-ray angiography image sequence is available as a result of step 1802. In this study, Non-Patent Literature 32 proposed a method for constructing population-based mean and predicted motion from a 4D-CT dataset. This is then used to perform 3D+t / 2D+t registration based on distance minimization in a single cardiac cycle. An example of a single 3D roadmap frame as a result of this step is provided by Figure 2408.

[0148] A third method (1803c) for creating a 3D+t roadmap becomes available when a 3D model is available as a result of step 1801 and two X-ray angiography image sequences are available as a result of step 1802. In this method, both the 3D model obtained from the results of step 1801 and the 3D reconstruction based on the X-ray angiography image data as a result of step 1802 are combined to create a 3D+t roadmap. This can be achieved, for example, by the method described in Patent Document 6, “Method and Apparatus for Improving 3D+Time Reconstruction,” which describes a method for performing a time-course 3D surface reconstruction of an object from two or more 2D X-ray images of the object. Alternatively, the 3D+t roadmap can also be created, for example, as taught by Non-Patent Document 33 or Non-Patent Document 34. An example of a single 3D roadmap frame as a result of this step is provided by Figure 24-2410.

[0149] Optionally, based on the generated roadmap, quantitative image analysis can be performed to extract clinically relevant information such as the location and percentage of vascular occlusion, diameter and area, vascular length, or vascular curvature, as taught in Non-Patent Document 35 or Non-Patent Document 13, for example.

[0150] Optionally, the generated roadmap can be used to calculate cFFR along the coronary artery tree, for example, as disclosed in Patent Document 7, which discloses a method for calculating the coronary flow reserve ratio along the coronary arteries based on 3D coronary artery reconstruction obtained by radiographic angiography. The 3D+t model enables the calculation of cFFR at all time points and takes into account changes in the geometry of the vascular tree portion during the cardiac cycle and hemodynamic changes during the cardiac cycle. Optionally, the generated roadmap can be used to calculate coronary WSS, or time-averaged WSS as taught in Non-Patent Document 27, for example. Time-averaged WSS is calculated at all time points within the 3D+t model.

number

[0151] Optionally, the location and amount of calcified plaque can be extracted from the X-ray angiography image sequence, for example, as further disclosed in this application by the flowchart in Figure 28.

[0152] Step 1804: Retrieve the device in the X-ray angiography image data. Within step 1804, information is acquired to enable the alignment of the 3D+t roadmap (as a result of step 1803) during live overlay (1809) as part of the online phase (1810), and step 1804 is analogous to step 102 in Figure 1. As described in step 102, respiratory motion, including possible patient motion, is compensated for by identifying reference points in the X-ray angiography image sequence. The reference points (represented by visible devices in the image sequence) will be used during the online phase (1810) for target object or roadmap transformation. The reference points can be, for example, the tip of a catheter, a piecemaker or anatomical landmark, or alternatively, any other object whose motion can be correlated with respiratory motion and possible patient motion. Such reference points can be acquired in all frames of the X-ray angiography image sequence.

[0153] If step 1801 is not performed, this step (1804) is identical to steps 501, 502, and 503 shown in Figure 5 and can be applied to both X-ray angiography image sequences as a result of step 1802. Optionally, the catheter tip is reconstructed in 3D, for example, as taught by Patent Document 36. The acquired device position (e.g., catheter tip) is then integrated into a 3D+t roadmap created by method 1803a as described in step 1803. Optionally, the device (e.g., catheter) is reconstructed in 3D and integrated into a 3D+t roadmap. The 3D+t roadmap is created after contrast agent initiation.

number

number

[0154] When a 3D+t roadmap is created by method 1803b, steps 501, 502, and 503 are applied to a single sequence of X-ray angiography images, and no 3D reconstruction of the device is performed. However, the device position resulting from step 503 is integrated into the 3D+t roadmap created by method 1803b. An example of a single 3D roadmap frame resulting from this substep is provided by 2405 in Figure 24, where 2408 represents the single 3D roadmap resulting from step 1803b, and 2407 represents the tip of the catheter (2407) integrated into the aforementioned 3D roadmap model.

[0155] The aforementioned method is also applicable if a 3D+t roadmap is created by method 1803c as described in step 1803, and at least two X-ray angiography image sequences are available as a result of step 1802. An example of a single 3D roadmap frame as a result of this substep is provided by 2409 in Figure 24, where 2410 represents the single 3D roadmap as a result of step 1803c, and 2411 shows the 3D reconstruction of the catheter and catheter tip (2412) integrated into the aforementioned 3D roadmap model.

[0156] Step 1805: Read the perspective image data. This step is the same as step 104 in Figure 1, except that in step 1805, any X-ray system geometry is possible, such as arbitrary X-ray angulation and rotation, arbitrary X-ray magnification, and arbitrary table position, and the fluoroscopic image data can be acquired from either single-plane or biplane acquisition. Optionally, the fluoroscopic image data can be acquired from biplane acquisition.

[0157] Optionally, any geometry of the X-ray system is possible within the flowchart in Figure 18, so that a guidance map is visualized to help the physician select the optimal fluoroscopic projection. Similar to that described in step 1802 and based on the method fully described in Patent Document 8, the optimal fluoroscopic projection is defined as a projection in which the X-ray field of view direction is perpendicular to the device (e.g., catheter). This device information is obtained based on the results of step 1804. Furthermore, such an optimal projection may include weighting of the distal shortening of the device, with less distal shortening resulting in a better projection.

[0158] Step 1806: Select the 3D Roadmap This step is the same as step 105 in Figure 1, where roadmap selection is based on ECG matching. The ECG signal is associated with online fluoroscopic images (see 1805 in Figure 18) and ECGs from offline angiography sequences (see 1802 in Figure 18). This allows for the selection of the optimal candidate roadmap, which will have the best match with the ECG signal.

[0159] Alternatively, when X-ray fluoroscopy image data is acquired using biplane acquisition, images from both projections can be acquired from each other in a short time with only a slight time delay, resulting in high resolution. This high temporal resolution allows for more accurate roadmap selection.

[0160] Step 1807: Track the device This step is the same as step 106 in Figure 1, in which devices in online fluoroscopic image data are tracked using, for example, a deep learning-based Bayesian filtering method.

[0161] Alternatively, if the fluoroscopic data is acquired by biplane acquisition, device tracking can be performed separately in both fluoroscopic image projections according to the technique described in step 106 of Figure 1. This results in two translations of the device with respect to the device position acquired in step 1804 in both biplane projections. Furthermore, the tip of the catheter can be reconstructed in 3D, for example, as taught by Patent Document 36, and used to transform the 3D roadmap in the next step.

[0162] Step 1808: Convert the selected 3D roadmap to generate a dynamic 2D roadmap. This step is similar to step 107 in Figure 1, where a reference point is used to compensate for the motion between the roadmap, which is acquired and generated from the X-ray angiography image sequence (offline phase), and the fluoroscopy image stream (online phase). Within step 1808, the same objectives as described in step 107 are acquired, except that a) the roadmap is a 3D roadmap, and b) the device can be in a 3D position or 3D, in addition to a 2D position as in step 107.

[0163] In step 1808, the device position in the current fluoroscopy frame as a result of step 1807, and the device position from the selected 3D roadmap frame as a result of step 1804 (e.g., the tip of the catheter) are used to obtain a transformation function for aligning the selected roadmap to the current fluoroscopy image frame. In embodiments, this transformation function can be a rigid body transformation based on the displacement obtained from the tip of the catheter between the current frame and the tip of the catheter in the selected roadmap frame. An alternative transformation function is a non-rigid body transformation.

[0164] For example, the rigid body transformation of a roadmap can be performed using a rigid body transformation function. The original 3D roadmap (as a result of step 1806) is transformed into a function.

number

number

number

number

number

number

number

[0165] Alternatively, if fluoroscopic data is acquired via biplane acquisition, device tracking can be performed in both fluoroscopic image projections. In this case, the aforementioned transformation can be performed for each fluoroscopic projection, and two 2D dynamic roadmaps are created.

[0166] Step 1809: Overlay a dynamic roadmap onto perspective image data. This step is similar to step 108 in Figure 1, generating a visual representation of the transformed roadmap, which is then rendered and superimposed onto each live fluoroscopic image frame. If the X-ray fluoroscopy data is acquired via biplane acquisition, the visual representations of the two 2D dynamic roadmaps can be rendered and superimposed onto their respective live fluoroscopic projections.

[0167] In the embodiment, step 1809 can be achieved by rendering the transformed model obtained from step 1808 according to the viewpoint of the current live image in the patient's live image stream / sequence. Pixels in the rendering of the model corresponding to the target organ (e.g., coronary artery tree) can be assigned a predefined range of color values, such as a range of color values ​​from red to white. In addition to coloring the model, transparency can also be applied to the rendered model. Model transparency provides a dual view where the fluoroscopic image is both visible and the model.

[0168] The mapping of the model to the dynamic roadmap superposition can be performed by converting the pixel intensity to color values, where the color scheme represents the quantitative parameters described in step 1804.

[0169] In addition, multiple superpositions are rendered and integrated as superpositions on a live fluoroscopy or angiography image stream. The superpositions can exist as a 3D roadmap and, in addition to this roadmap, as well as, for example, 3D volumetric data or quantitative parameters. This 3D volumetric data needs to be back-projected onto the current fluoroscopy image frame based on projection (field of view) in the same way as the 3D roadmap.

[0170] Obviously, steps 1808 and 1809 may be performed in a slightly different order. Step 1808 yields a 3D-transformed roadmap based on information from steps 1808 and 1807, and step 1809 processes the current fluoroscopic angulation, magnification, and table movement during the backprojection, in which step 1809 backprojects this transformed 3D roadmap onto the current fluoroscopic image frame.

[0171] After step 1809, this method proceeds to step 1805, processing the next fluoroscopic image frame. If the reading of fluoroscopic image data stops as described in step 1805, the flowchart terminates as described in Figure 18.

[0172] In alternative embodiments, differential processing is used as an image enhancement technique to improve the identification and processing of target objects. Image enhancement is a useful tool for improving the identification of target objects. An example of image enhancement is differential processing, where static objects and background are removed from the target image by differential processing of the baseline image from this image.

[0173] Figure 15 shows a workflow similar to that in Figure 1, where differential processing is used as an image enhancement technique to improve the identification and processing of target objects. Offline phase 1500 and online phase 1512 exist. Only an additional baseline image dataset is read, and image enhancement is performed, for example, by differential processing.

[0174] First, X-ray angiography image data 1501, including the same ECG as 101 in Figure 1, is read out. Next, fluoroscopic image data (1502), including the ECG, is acquired, having the same projection (field of view) as in 1501, and showing the catheter being withdrawn from the small opening. The fluoroscopic image data 1502 should contain one complete cardiac cycle, and this image data is a baseline image sequence in which only the background and stationary objects are visible.

[0175] Next, in step 1503, an entire cardiac cycle from the fluoroscopic image data of 1502 is selected. For all frames of the angiography image data of 1501, baseline image frames from 1502 are selected based on ECG matching. ECG matching can be performed as shown in Figure 1, 105. After matching the angiography image data with the baseline image data, the baseline image data is subtracted from the angiography image data (see Figure 16, 1601) (see Figure 16, 1602), and the image is enhanced (see Figure 16, 1603). In this enhanced image, dynamic objects become clearly visible. Furthermore, image processing techniques such as edge enhancement can be applied to improve the contours of target objects.

[0176] The enhanced image obtained from step 1503 is used to obtain the device position in step 1504. Step 1504 is identical to step 102 in Figure 1. Next, the roadmap is created in step 1505, which is identical to step 103 in Figure 1.

[0177] In online phase 1512, fluoroscopic image data 1506 is read, which includes the same ECG as 104 and has the same projection (field of view) as 1501 and 1502. Next, in step 1507, image enhancement is applied to the fluoroscopic image of 1506, and the same process as in step 1503 is performed. This includes ECG matching of the fluoroscopic image data 1506 and the baseline data 1502. The enhanced fluoroscopic image data is obtained by subtraction processing between the fluoroscopic image data 1506 and the matched baseline data 1502. Furthermore, image processing techniques such as edge enhancement can be applied to improve the contour of the target object.

[0178] Next, in step 1508, roadmap selection is performed in the same way as in step 105 in Figure 1. The enhanced fluoroscopic image steps from step 1507 are used to track the device in step 1509. Then, in step 1510, the roadmap selected in step 1508 is transformed in the same way as in step 107 in Figure 1. Finally, in step 1511, the roadmap is presented as an overlay on the fluoroscopic image data read in 1506. After step 1511, the method returns to step 1506 to process the next fluoroscopic image frame, and when the reading of fluoroscopic image data described in step 1506 stops, the flowchart described in Figure 15 ends.

[0179] In an alternative embodiment, the online phase of dynamic roadmapping can be restarted by an X-ray angiography image stream, as shown in the flowchart of Figure 26. Such an X-ray angiography image stream is acquired during live fluoroscopy image acquisition after the roadmap has been repositioned, while the physician injects a limited amount of contrast agent.

[0180] The flowchart in Figure 26 is an alternative to either the flowchart in Figure 1 (2D roadmap) or the flowchart in Figure 18 (3D roadmap). The advantages of this alternative embodiment are that a) the table can be moved during the online phase, b) additional devices introduced during the (live) procedure can be used for repositioning during the follow-up phase, and c) if the roadmap projected onto the live fluoroscopic image is misaligned, the physician is notified. The steps in Figure 26 are described in detail in the following section.

[0181] Step 2601: Offline Phase If a sequence of roadmaps is created within step 2601, and this sequence of roadmaps represents a 2D roadmap, then the step is identical to the offline phase (100) represented by steps 101, 102, and 103 in Figure 1. If a 3D roadmap is created, then step 2601 is identical to the offline phase (1800) represented by steps 1801, 1802, 1803, and 1804 in Figure 18.

[0182] Step 2602: Start X-ray angiography. Within this step, the roadmap is realigned to the current status. The term "current status" refers to the true live status during the online phase (2610). Such realignment is useful, for example, after moving the table during the online phase (2610), or in other situations where the roadmap's alignment from the current status may be incorrect, or when additional devices are introduced during the online phase to allow for more precise alignment. This step can be omitted if the system is transitioning from step 2601 to the online phase (2610) for the first time. Step 2602 can be initiated by a signal triggered by the X-ray system or another external system after contrast agent bolus injection has started. Alternatively, contrast agent bolus injection is automatically detected by analyzing the fluoroscopic image stream (2603), for example, as taught by Non-Patent Literature 3, or as described by step 501 in Figure 5.

[0183] The first substep in 2602 is the readout of the radiographic angiography image stream, which can be acquired from either single-plane or double-plane acquisition. Once a certain amount of radiographic angiography image frames become available (preferably enough frames to cover at least one cardiac cycle), the roadmap is re-aligned to the current situation. The ECG signal is acquired simultaneously with the radiographic angiography image stream.

[0184] If the offline phase represents a 2D roadmap, the second substep in 2602 deals with updating (recreating or realigning) the 2D roadmap sequence and is identical to steps 102 and 103 in Figure 1. If the offline phase represents a 3D roadmap, the second substep in 2602 is the recreation of the 3D roadmap sequence and is identical to steps 1803 and 1804 in Figure 18. Figure 27 illustrates the realignment step 2602 in the case of a 3D roadmap. Photo 2701 shows the roadmap (2608) back-projected onto the radiographic angiography biplane acquired in the first substep of 2602. In 2701, an extremely displaced misalignment is visible for illustrative purposes. Photo 2702 shows the result of extracting the 2D vascular system from the biplane image. This will be used to create the 3D model described in steps 1803 and 1804 in Figure 18. Figure 2703 shows a re-projected roadmap with alignment.

[0185] Alternatively, if image data is read from the acquisition of multiple single-plane images with different projections or from biplane acquisition in the offline phase, the device position can be determined for each individual image data projection according to step 102 in Figure 1 or 1804 in Figure 18. In addition, the position of the 3D device can be determined for each individual position of the device in all individual projections based on the projection information of the image data.

[0186] Step 2603: Read X-ray fluoroscopy image data. In step 2601, fluoroscopic image data is read out, and this step is identical to step 104 in Figure 1. If the offline phase represents a 3D roadmap (step 2601), any X-ray system geometry is possible, such as arbitrary X-ray angulation and rotation, arbitrary X-ray magnification, and arbitrary table position, and the fluoroscopic image data can be obtained from single-plane or bi-plane acquisition. Furthermore, a guidance map can be visualized to help the physician select the optimal fluoroscopic projection, as shown in step 1805 in Figure 18.

[0187] Step 2604: Select a roadmap In step 2604, as a result of step 2601, or if updated, as a result of step 206, the current roadmap is selected from the sequence of roadmaps. If the offline phase represents a 2D roadmap, this step is identical to step 105 in Figure 1. If the offline phase represents a 3D roadmap, this step is identical to step 1806 in Figure 18.

[0188] Alternatively, X-ray fluoroscopy data is acquired using biplane acquisition. In a biplane system, images from both projections are acquired sequentially with only a slight time delay, resulting in high temporal resolution. This high temporal resolution allows for more accurate roadmap selection.

[0189] Step 2605: Track the device In step 2605, the device is tracked within the live fluoroscopic image stream. If the offline phase represents a 2D roadmap, this step is identical to step 106 in Figure 1; if the offline phase represents a 3D roadmap, this step is identical to step 1807 in Figure 18.

[0190] Alternatively, fluoroscopic data can be acquired by biplane acquisition. In this case, device tracking can be performed separately in both fluoroscopic image projections according to the technique described in step 106 of Figure 1. This results in two translations of the device with respect to the device position acquired in step 2601 in both biplane projections.

[0191] Next, the device position can be determined as a 3D position based on both projections from biplane acquisition, for example, as taught in Non-Patent Document 35. This enables 3D tracking of the device.

[0192] Step 2606: Convert the selected roadmap to generate a dynamic roadmap In step 2606, the selected roadmap is transformed to create a dynamic roadmap for the current live fluoroscopy image frame. If the offline phase represents a 2D roadmap, this step is the same as step 107 in Figure 1. If the offline phase represents a 3D roadmap, this step is the same as step 1808 in Figure 18.

[0193] Alternatively, X-ray fluoroscopy data can be acquired using biplane acquisition. In this case, two translations are acquired in step 2605. These two translations, along with the 3D orientations of both image projections, result in a 3D transformation of the selected roadmap.

[0194] Step 2607: Overlay a dynamic roadmap onto X-ray fluoroscopy image data. If the offline phase generates a 2D roadmap, this step is identical to step 107 in Figure 1. If the offline phase generates a 3D roadmap, this step is identical to step 1808 in Figure 18. Optionally, the method of visualizing the roadmap on the X-ray image data can be modified. As a result of the contrast agent being administered in step 2602, vascular structures will be highlighted as long as the contrast agent is present in the vascular system (this depends on the amount and duration of contrast agent injection, and on average about 5 cardiac cycles). To enable visualization of the vascular structures and the projected overlay, the overlay can be projected onto the X-ray image data in transparent mode, as shown by 3401 in Figure 34b. The degree of transparency can obviously be adjusted. Within Figure 34b, 3402 provides another example where the overlay is projected without obscuring the target vascular structures. This is achieved by displaying only the outer boundary of the roadmap image. This can be generated by first inflating the roadmap by a predefined amount and then extracting the boundary of the inflated roadmap. Clearly, this can also be configured as a transparency mode where the degree of transparency can be adjusted.

[0195] Step 2608: Detect drift During step 2608, the system detects whether roadmap realignment is necessary. For example, the X-ray system triggers a signal if the table is adjusted during the online phase (2610) or after contrast agent administration has started. Optionally, within step 2608, roadmap misalignment can be detected by image processing of the fluoroscopic image stream, e.g., by comparing the current image frame with an image frame from the same cardiac cycle in a previous cardiac cycle. The comparison between both images can be performed, for example, by image registration, cross-correlation, or (minimum) difference. If image registration is applied, the displacement resulting from image registration is a measure of the variation between both images, and a threshold can be defined for detecting drift. In the case of cross-correlation, the cross-correlation value represents the similarity between both images, and a threshold can be defined, for example, for detecting drift. Alternatively, one image is shifted relative to the other, and the cross-correlation between both images is calculated for multiple shifts. The maximum cross-correlation value may then correspond to a particular shift of the image. The magnitude of the shift is the magnitude of the drift. If there is a difference between both images, the value of the difference is a measure of the drift. A threshold may be defined for drift detection. Alternatively, one image is shifted relative to the other, and the difference between the two is the calculated image. The minimum difference can be calculated for multiple shifts, and the shift corresponding to the minimum difference is the measure of drift.

[0196] Another method for drift detection is evaluation of the ECG signal. The ECG is a periodic signal. For example, in the case of arrhythmia, the heartbeat is irregular and abnormal, causing the ECG signal to deviate from the normal period. Deviations in the ECG signal can interfere with roadmap selection, which can lead to incorrect superposition of roadmaps. Therefore, evaluation of the ECG signal is useful for detecting drift.

[0197] During regular breathing, the diaphragm contracts, which is called normal breathing. For example, if a patient is hiccuping, the movement of the diaphragm is irregular, and therefore the movement of the object may also be irregular. Diaphragmatic contraction may be visible in radiographic angiography. Tracking the diaphragm in a radiographic image sequence can help detect irregular breathing and the need for drift correction. Tracking diaphragmatic movement can be performed, for example, as taught by Non-Patent Document 37. Therefore, evaluating diaphragmatic contraction (movement) helps detect drift.

[0198] If step 2610 indicates that a misalignment exists, step 2602 is initiated; otherwise, the system returns to step 2603. Figure 26 ends when the reading of the fluoroscopic image data described in step 2603 stops.

[0199] Calcified plaques are radiopaque, but their presence cannot be recognized in a single X-ray image frame and is barely visible when evaluating a sequence of X-ray images. Figure 28 shows a flowchart illustrating an embodiment for extracting the location and amount of calcified plaques from X-ray image data. The X-ray image data can represent a sequence of X-ray fluoroscopic images or an X-ray angiography image sequence, including image data before and after the administration of a contrast agent. In either case, the ECG signal should be considered part of the X-ray image data. Figure 30 illustrates the challenges in identifying calcified plaques. Figure 30 shows a high-level method, such as that described in the flowchart of Figure 28. Calcified plaque is present within the dashed circle (3006), but it is barely recognizable in the images (3001, 3002, 3003, and 3005). After registering the images, the calcified plaques (3010) are highlighted and made visible by adding the image information to a single image (3009). Furthermore, quantitative analysis can be performed on the highlighted calcified plaques. The steps in Figure 28 are described in detail in the following section.

[0200] First, in step 2801, the X-ray image data is read out. As previously mentioned, this can be either a fluoroscopic image sequence or an angiography image sequence containing image data before and after contrast agent administration. First, the steps for when the X-ray image data represents an angiography image are described, and then the method for when the X-ray image data represents fluoroscopic image data is explained. In either case, it is assumed that the ECG signal is part of the X-ray image data.

[0201] Figure 29 shows a typical example of an X-ray angiography image sequence. Here, 2900 shows several consecutive X-ray image frames in such a sequence, and 2906 shows the corresponding ECG signal. In Figure 29, the moment of contrast agent administration is indicated by 2905, and the preceding frames (2901, 2902) show unenhanced X-ray image data of vascular structures lateral to the catheter (2907). After contrast agent administration, the frames also represent vascular structures (2903, 2904), and in the current example, represent the left coronary artery. Furthermore, the moment each X-ray frame was acquired with respect to the ECG signal is visualized (2901, 2902, 2903, and 2904).

[0202] Alternatively, the properties of the X-ray beam can be altered so that calcified tissue absorbs X-rays more efficiently than the surrounding tissue. When applied to the evaluation of vascular structures, the properties of the X-ray beam are defined so that the X-rays are absorbed optimally by the surrounding tissue in relation to an iodine-rich contrast agent. Two main means of defining the properties of the X-ray beam produced by an X-ray tube are a) changing the current (mA) and b) changing the voltage (kV). The current across the X-ray tube (measured and often called mA or milliamperes) determines the number of electrons emitted and colliding with the anode. Increasing the mA increases the number of electrons colliding with the anode, and consequently, the number of photons produced by the X-ray tube increases linearly. The voltage across the ends of the X-ray tube (often measured in units called kV or kilovolts) affects the velocity of the electrons when they collide with the anode. This affects the energy of the photons that can be produced by the X-ray tube. Furthermore, faster electrons (kV) 3 It generates more photons on the order of 100. There are two main ways in which an X-ray beam interacts with tissue. The first major effect is the photoelectric effect, where a photon uses up all its energy, releasing an electron from an atom, while the electron moves around and ionizes a neighboring atom, but no scattered photons are produced. The second major effect is Compton (non-coherent) scattering. In this scattering, a photon collides with an atom and ionizes an electron, but not all of its energy is consumed. Next, the photon is scattered in another direction with slightly less energy, and the free electron is damaged. The scattered photons can return to the X-ray tube, pass through the patient and hit the detector from any odd angle, or be scattered again within the patient. As the X-ray beam passes through tissue, the energy decreases as photons are absorbed. This is known as attenuation. High-energy photons pass through tissue more easily than low-energy photons (this means that high-energy photons are less likely to interact with matter). Much of this effect is related to the photoelectric effect. The probability of photoelectric absorption is (Z / E) 3It is approximately proportional to the photon energy, where Z is the atomic number of the tissue atom and E is the photon energy. As E increases, the likelihood of the interaction decreases rapidly. Compton scattering decreases slowly with increasing energy but remains nearly constant for different energies. Therefore, instead, the tube characteristics of the X-ray system (tube voltage and / or tube current) are optimized so that the calcified tissue absorbs X-rays more efficiently than the surrounding tissue, and X-ray fluoroscopic or X-ray angiographic image sequences are collected.

[0203] Next, in step 2802, the target vessel is acquired. This can be done by the method described in step 103 or the result of step 103 in Figure 1, or by the method described in step 1803 or the result of step 1803 in Figure 18, or by the method used in the user manual to identify the target vessel.

[0204] In step 2803, frames from the X-ray angiography image sequence in which the contrast agent fluid enters the coronary artery are identified, and

number

[0205] Next, in step 2804, the vascular skeleton is visualized in each frame of the sequence following the initiation of the contrast agent as a result of step 2803.

number

[0206] In step 2806, the corresponding fluoroscopic images selected as a result of step 2805 are registered with each other. Registration is performed for each decomposed vascular branch, as described in step 2804. Optionally, to improve registration performance, landmarks are automatically identified from the skeleton and / or the decomposed skeleton. Such landmarks may be, for example, bifurcation points (3104), start / end positions (3105), or large local curvatures (3106). 2D-to-2D registration can be performed, for example, as taught in Non-Patent Document 5 or Non-Patent Document 38.

[0207] In step 2807, an image with calcified plaques highlighted is generated. Before generating the highlighted image, the registered fluoroscopic image (as a result of step 2806) is preprocessed. The X-ray image typically has the problem of a transparent background layer that potentially obscures the calcified plaques. Background subtraction is performed to eliminate the effects of this. A simple and effective way to remove the static transparent layer is to use the assumption that each layer only adds mass. Obtaining the maximum intensity for each pixel over time (lower pixel intensity means more X-ray absorption) will result in an image showing the minimum amount of mass over time. Assuming that all pixels will not contain any contribution from a moving mass layer at any given time, the maximum intensity image will be equal to a static background layer. Often, a single general mask image is created frame by frame based on the maximum intensity of each pixel in all frames, for example, as described above. However, such a single mask image is affected by artifacts from large, slow-moving objects in the background (such as the diaphragm, ribs, and lungs). Therefore, the background diffing process performed determines a more local background mask of x (x is usually 5 frames) consecutive frames that are symmetrical around each frame. Alternatively, the static background layer can be generated as taught in Non-Patent Document 7 or Non-Patent Document 8.

[0208] Next, the preprocessed frames are combined into a single image frame. This is done by adding the preprocessed frames (3008 in Figure 30), and optionally, weighting coefficients are introduced that correlate with the likelihood of non-rigid deformation in each frame due to perspective shortening, for example. An example of the results of this process is shown in 3009 in Figure 3030. In photograph 3009, the calcified plaque is highlighted (3010).

[0209] Before proceeding to step 2808, the method shown in Figure 28, where the X-ray image data represents a fluoroscopic image sequence (as a result of step 2801), is described below. In this case, it is assumed that a device containing radiopaque markers is present within the target vessel. This could be, for example, an undeployed stent, a measurement guidewire, or any other device that moves in sync with the movement of the target vessel. An example of an undeployed stent is provided in Figures 32 and 3202. Step 2803 cannot be applied to the fluoroscopic image sequence, and step 2804 now detects and tracks the radiopaque markers within the fluoroscopic image sequence. Since all image frames within the fluoroscopic image sequence are fluoroscopic image frames, step 2805 cannot be applied. In step 2806, the tracked radiopaque markers are used to register the images with each other, and step 2807 is identical to that described above. Figure 32 provides an illustration of enhanced calcified plaque images superimposed on enhanced calcified plaque images and X-ray angiography image frames aligned by ECG. Image 3202 shows a single image frame within a sequence of X-ray fluoroscopic images, and Image 3203 shows the result of enhanced calcified plaque images. Image 3201 shows the X-ray angiography image frame aligned to Image 3203 by ECG. Image 3204 shows the result of subtracting the enhanced calcified plaque images from the X-ray angiography image frame, and Image 3205 shows the result of segmenting the enhanced calcified plaques by the X-ray angiography image frame. Both Images 3204 and 3205 show the visible amount of the location and vascular lumen of the calcified plaques.

[0210] Returning to Figure 28, the final step 2808 involves quantitative analysis of the enhanced calcified plaques. The area of ​​the calcified plaques (3010, 3206) can be calculated by manual and / or (semi)automatic detection of the calcified plaques (calcified plaque regions) using the enhanced images (3009, 3204). Video densitometry analysis can also be performed. The volume and / or mass of the calcified plaques can be derived by comparing the density of the calcified plaque region with another radiopaque region in the enhanced image or X-ray image data (2801). If the properties of the radiopaque region, such as its geometry and its (mass) decay coefficient, are known, the volume and / or mass of the calcified plaque region can be calculated using the Beer-Lambert law. The radiopaque region can also be a region of blood vessels obtained from an X-ray angiography image sequence. In this situation, it is necessary to know the (mass) decay coefficient of the contrast agent so that the volume and / or mass of the calcified plaque area can be calculated using Lambert-Beer's law. Alternatively, video densitometry analysis can be performed, for example, as disclosed in Patent Document 9.

[0211] Generally, X-ray image data includes a corresponding ECG signal recording. If the X-ray image data does not include ECG information, matching the cardiac phase between contrast-enhanced and non-contrast-enhanced images based on the ECG signal is impossible. To solve this problem, a cardiac model that mimics cardiac motion can be used. Such a cardiac model can be generated, for example, from CT acquisitions of several hearts. Cardiac motion is extracted from the image data, and a model that mimics cardiac motion is generated.

[0212] The cardiac motion model provides cardiac motion in 3D+t. Based on X-ray angiography acquisition, the expected cardiac motion in a particular projection can be extracted from the cardiac motion model.

[0213] The cardiac motion model is applied to both contrast-enhanced and non-contrast-enhanced images, and the correlation between the two images is calculated, for example, to identify the best match between the two images, as taught, for example, by Non-Patent Document 39.

[0214] Other uses The aforementioned embodiments are associated with providing a real-time dynamic overlay or dynamic angiography image stream / sequence that can be superimposed on live X-ray projection or coronary artery roadmap.

[0215] In embodiments, the method described in this application may also be used to provide static guidance, meaning that only one roadmap is selected within a sequence of roadmaps. Preferably, the roadmap in which coronary artery cardiac motion is minimized is selected. Furthermore, a 3D model created from 3D angiography image data as described in step 1801 of Figure 18 can be used as a static roadmap superimposed on X-ray image data. Optionally, registration to X-ray image data can be performed, for example, by previously described techniques or techniques disclosed in this application. For example, the model presented in Figure 25, which also includes coronary plaque information, can also be used as static guidance.

[0216] In other embodiments, the described method can be used as pre-procedural planning, and optionally, radiographic angiography images can be simulated, for example, as described in Patent Document 10, to improve preparation for the actual PCI procedure.

[0217] Alternatively, a single roadmap can be selected. Images associated with the selected roadmap can also be used as reference images. The relationship between the roadmap and the reference images can be determined, for example, using the patient's ECG signal. Next, all images from the live fluoroscopy or angiography image stream / sequence are registered as reference images. Image registration "freezes" the images from the live fluoroscopy or angiography image stream / sequence and aligns them to the static roadmap. Image registration can be performed, for example, by methods known in the art as taught in Non-Patent Document 5.

[0218] The embodiments described above are associated with providing a real-time dynamic superposition or dynamic coronary roadmap that can be superimposed on a live fluoroscopy or angiography image stream / sequence. Alternatively, embodiments that specifically focus on the extraction roadmaps described in Figures 1, 15, 18, and 26 can also be used in the field of robot-assisted percutaneous coronary intervention (PCI) or peripheral vascular intervention.

[0219] Since the initial introduction of PCI in the 1970s, interventional cardiology has undergone significant advancements in device technology, treatment techniques, and drug therapies, enabling it to treat the most complex possible lesions (such as complete coronary occlusion). Despite steady progress in almost every aspect of the field of coronary intervention, the mechanical aspects of PCI, such as the manipulation of coronary guidewires, balloons, and stents, and the occupational risks to operators and catheterization lab staff, have remained largely unchanged since their introduction in the 1970s. Interventional cardiologists manipulate intravascular devices under direct fluoroscopy guidance, which necessitates the wearing of heavy protective clothing. Throughout their interventional cardiology careers, operators are exposed to the adverse effects of cumulative radiation exposure and an increased prevalence of orthopedic injuries.

[0220] Robotic systems that handle the manipulation of intravascular devices have the potential to significantly reduce the aforementioned disadvantages and risks and revolutionize percutaneous coronary intervention procedures. While still in its early stages, such robot-assisted PCI systems are being manufactured, for example, by Corindus Vascular Robotics.

[0221] One requirement for such robotic systems is the ability to manipulate intravascular devices at a sub-millimeter level and requires knowledge of the 3D geometry and morphology of the vascular system. This also includes knowledge of the changes in the 3D shape of the vascular system due to, for example, cardiac motion, respiratory motion, and / or patient motion during the procedure. Using the methods described in this application, integrated into a robotic system that handles the manipulation of intravascular devices, there is a likelihood that patient outcomes can be improved, and advanced pre-procedure planning tools with robotic precision can bring new and innovative standards of treatment to patients.

[0222] The operation can be performed by a processor unit in a standalone system, or by a processor unit in a semi-standalone system that is connected to an X-ray system (Figure 2b) and described in detail with reference to Figure 33, or is directly incorporated into, for example, a fluoroscopy system or other imaging system for acquiring a sequence of two-dimensional angiographic images (Figure 2a). Figure 17 shows an example of a high-level block diagram of an X-ray cine fluoroscopy system. This block diagram illustrates an example of how embodiments can be integrated into such a system.

[0223] The parts of the system (defined by various functional blocks) can be implemented by one or more processors that operate dedicated hardware, analog and / or digital circuits, and / or program instructions stored in memory.

[0224] The X-ray system in Figure 17 includes an X-ray tube 1701 equipped with a high-voltage generator 1702 that generates an X-ray beam 1703. The high-voltage generator 1702 controls and supplies power to the X-ray tube 1701. The voltage generator 1702 applies a high voltage to the vacuum gap between the cathode and the rotating anode of the X-ray tube 1701. The voltage applied to the X-ray tube 1701 causes electron transfer from the cathode to the anode of the X-ray tube 1701, resulting in an X-ray photon generation effect, also known as Bremsstrahlung. The generated photons form the X-ray beam 1703 directed toward the image detector 1706.

[0225] The X-ray beam 1703 comprises photons having an energy spectrum ranging up to a maximum value determined by the voltage and current applied to the X-ray tube 1701. The X-ray beam 1703 then passes through the patient 1704, who lies on an adjustable table 1705. The X-ray photons of the X-ray beam 1703 penetrate the patient's tissue at various angles. Different structures of the patient 1704 result in different absorption of radiation, which modulates the beam intensity. The modulated X-ray beam 1703 emitted from the patient 1704 is detected by an image detector 1706 located on the opposite side of the X-ray tube. This image detector 1706 can be either an indirect or direct detection system.

[0226] In an indirect detection system, the image detector 1706 includes a vacuum tube (X-ray image intensifier) ​​that converts the X-ray emission beam 1703' into an amplified visible light image. This amplified visible light image is then transmitted to a visible light image receptor, such as a digital video camera, for display and recording of the image. This generates a digital image signal.

[0227] In a direct detection system, the image detector 1706 comprises a flat panel detector. The flat panel detector directly converts the X-ray emission beam 1703' into a digital image signal. The digital image signal obtained from the image detector 1706 passes through the digital image processing unit 1707. The digital image processing unit 1707 converts the digital image signal from 1706 into a corrected (e.g., inverted and / or contrast-enhanced) X-ray image in a standard image file format, such as DICOM. This corrected X-ray image can then be stored in the hard drive 1708.

[0228] Furthermore, the X-ray system in Figure 17 includes a C-arm 1709. The C-arm holds the X-ray tube 1701 and the image detector 1706 such that the patient 1704 and the adjustable table 1705 are between the X-ray tube 1701 and the image detector 1706. The C-arm can be moved (rotated and angulated) to a desired position and a specific projection can be acquired in a controlled manner using the C-arm control 1710. The C-arm control allows for manual or automatic input to adjust the C-arm to a desired position for X-ray recording of a particular projection.

[0229] The X-ray system in Figure 17 can be either a single-plane or biplane imaging system. The biplane imaging system has multiple C-arms 1709, each consisting of an X-ray tube 1701, an image detector 1706, and a C-arm control 1710.

[0230] Furthermore, the adjustable table 1705 can be moved using the table control 1711. The adjustable table 1705 can be moved along the x, y, and z axes and tilted around specific points.

[0231] Furthermore, this X-ray system includes a measurement unit 1713. This measurement unit includes patient information, such as ECG, aortic pressure, biomarkers, and / or information regarding height, length, etc.

[0232] This X-ray system also includes a general-purpose unit 1712. This general-purpose unit 1712 can be used to interact with the C-arm control 1710, the table control 1711, the digital image processing unit 1707, and the measurement unit 1713.

[0233] One embodiment is carried out by the X-ray system shown in Figure 17, as follows: A clinician or other user collects at least two sequences of X-ray angiographic images of patient 1704 by using the C-arm control 1710 to move the C-arm 1709 to a desired position relative to patient 1704. Patient 1704 lies on an adjustable table 1705, which has already been moved to a specific position by the user using the table control 1711.

[0234] Next, a sequence of X-ray images is generated using the high-voltage generator 1702, the X-ray tube 1701, the image detector 1706, and the digital image processing unit 1707, as described above. These images are then stored in the hard drive 1708. Using these sequences of X-ray images, the general-purpose processing unit 1712 performs the method described in this application, as shown in Figures 1, 15, 18, 26, or 28, using, for example, the measurement unit 1713, the digital image processing unit 1707, the C-arm control unit 1710, and the table control unit 1711.

[0235] As previously mentioned, the operation can also be performed by a processor unit of a semi-standalone system connected to the X-ray system (Figure 2b). Figure 33 shows such a setup. In Figure 33, 3301 represents the X-ray system, and 3302 represents the semi-standalone system on which the operation described in this application is performed. To acquire a live X-ray image data stream, which can be either an X-ray angiography image data stream or an X-ray fluoroscopy image data stream, the semi-standalone system 3302 is equipped with a frame grabber (3303) that enables the digitization of live X-ray image data acquired from the video output of the X-ray system. Live access to the ECG signal is acquired by an analog-to-digital converter (3304) connected to the ECG signal output of the X-ray system or another system that measures the patient's ECG. A roadmap overlay created by the methods presented and described with respect to the flowcharts in Figure 1, Figure 15, Figure 18, Figure 26, or Figure 28 is provided to the X-ray system video input (3304) and visualized by the X-ray system monitor.

[0236] This specification has described and illustrated several embodiments of methods and apparatus for quantitative flow analysis. While specific embodiments have been described, the invention is broad in scope as the art permits, and this specification is intended to be read similarly; therefore, the invention is not intended to be limited thereto. For example, data processing operations can be performed offline on images stored in digital storage such as PACS, which is commonly used in medical imaging technology. This is usually done in a vendor-independent universal language such as DICOM (Digital Imaging and Communications in Medicine). The storage is a hard disk, a PACS (picture archiving and communications system) server, a VNA (vendor neutral archive), or other image archiving and communications systems commonly used in medical imaging technology. Accordingly, those skilled in the art will understand that further modifications can be made to the inventions provided without departing from the spirit and scope set forth in the claims.

[0237] The embodiments described herein, as stated above, may include a variety of data storage and other memory and storage media. These may reside in various locations, such as local (and / or resident) storage media of one or more computers, or storage media remote to any or all computers on a network. In a particular set of embodiments, information may reside in a storage area network ("SAN"), as is well known to those skilled in the art. Similarly, files necessary to perform functions specific to a computer, server, or other network device may be stored locally and / or remotely, as necessary. Where the system includes computer devices, each device may include hardware elements that can be electrically coupled via a bus. These elements may include, for example, at least one central processing unit ("CPU" or "processor"), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random-access memory ("RAM") or read-only memory ("ROM"), as well as removable media devices, memory cards, flash cards, etc.

[0238] Such devices may also include computer-readable storage media readers, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.), and the working memory described above. Computer-readable storage media readers may be configured to connect to or receive computer-readable storage media representing remote, local, fixed, and / or removable storage devices, as well as storage media that temporarily and / or more permanently store, transmit, and retrieve computer-readable information. This system and various devices also typically include several software applications, modules, services, or other elements located within at least one working memory device, including an operating system and application programs such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations from those described above. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may also be used.

[0239] Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented on a computer-readable medium in accordance with the foregoing description. The storage medium and computer-readable medium for containing the code or part of the code may include any suitable medium known or used in the art. This suitable medium includes, but is not limited to, storage and communication media such as volatile and non-volatile, removable and non-removable media, implemented in any way or technique for storing and / or transmitting information such as computer-readable instructions, data structures, program modules, or other data. This storage and communication medium includes RAM, ROM, electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technologies, compact disk read-only memory ("CD-ROM"), digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store the necessary information and that can be accessed by system devices. Based on the disclosures and teachings provided herein, those skilled in the art will understand other methods and / or methods for carrying out various embodiments.

[0240] Although the disclosed embodiments are described with respect to single or biplane X-ray imaging modalities, variations within these embodiments are also applicable to 3D reconstruction based on, for example, rotational angiography, computed tomography, magnetic resonance imaging, etc. Accordingly, this specification and drawings should be considered as illustrative rather than restrictive. However, it will be apparent that various modifications and changes can be made to such inventions without departing from the broader spirit and scope of the invention as described in the claims.

[0241] Other modifications are possible within the spirit of this disclosure. Therefore, the disclosed techniques are open to various modifications and alternative configurations, some of which are illustrated in the drawings and have been described in detail above. However, the invention is not intended to be limited to the specific forms disclosed. Rather, the intention is to cover all modifications, alternative structures, and equivalents that fall within the spirit and scope of the invention, as defined in the claims.

[0242] In the context describing the disclosed embodiments (particularly in the context of the following claims), the use of the terms “a,” “an,” “the,” and similar articles should be interpreted as encompassing both singular and plural forms unless otherwise indicated herein, or unless it is clearly inconsistent with the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as unrestrictive unless otherwise specified (i.e., “including but not limited to…”). The term “connected” should be interpreted as including, or being attached together with, any intervening elements, if they are physical connections, in whole or in part, even if they exist; that is, attached together with or joined together with. Enumerations of value ranges herein are intended to function simply as a concise way of referring individually to each individual value that falls within that range, unless otherwise indicated herein. The use of the terms “set” (e.g., “set of items”) or “subset” should be interpreted as a non-empty set containing one or more components, unless otherwise specified or inconsistent with the context.

[0243] The operations of the processes described herein may be performed in any suitable order, unless otherwise indicated herein or unless it is clearly inconsistent with the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems comprising executable instructions and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more of these applications) that are collectively executed on one or more processors. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program containing multiple instructions executable by one or more processors. The computer-readable storage medium may be non-temporary.

[0244] Preferred embodiments of the Disclosure, including the best mode known to the inventors for carrying out the invention, are described herein. Modifications of these preferred embodiments will be apparent to those skilled in the art by reading the foregoing description. The inventors anticipate that those skilled in the art will appropriately use such modifications, and they intend that embodiments of the Disclosure will be carried out in ways other than those specifically described herein. Accordingly, the scope of the Disclosure includes all modifications and equivalents of the subject matter described in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the above elements in all possible modifications thereof is included in the scope of the Disclosure unless otherwise indicated herein or clearly rejected by the context.

[0245] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as each reference is indicated to be incorporated by reference individually and specifically, and as is contained herein in whole.

Claims

1. A method for operating a medical device to generate an image of a target object of a patient, wherein the target object comprises a part of the patient's vascular system. The method described above is i) An operation to generate a three-dimensional (3D) model of the target object from 3D image data acquired using a 3D angiography imaging modality. ii) An operation to generate at least one roadmap using the 3D model of the target object, wherein the information includes at least one of the center line, contour, and image mask of the target object, iii) An operation to determine a reference position relative to the tip of an interventional device used in a procedure to treat the target object, wherein the reference position corresponds to at least one roadmap, iv) An operation to obtain non-contrast X-ray image data of the target object, wherein the non-contrast X-ray image data is acquired using an X-ray imaging modality without a contrast agent and the interventional device is present in the non-contrast X-ray image, V) An operation to determine the position of the tip of the interventional device within the non-contrast X-ray image data, The operation of selecting or accessing a specific roadmap generated in vi) iii), vii) an operation to transform the particular roadmap using the reference position of iii) corresponding to the particular roadmap and the position of the tip of the intervention device determined in v), wherein the transformation comprises a rigid or non-rigid transformation of the particular roadmap based on a displacement obtained from the reference position corresponding to the particular roadmap and the position of the tip of the intervention device determined in v), and The operation of overlaying the visual representation of the converted roadmap from viiii)vii) onto the non-contrast X-ray image data for display. A method that includes [a certain feature].

2. The aforementioned at least one roadmap comprises at least one two-dimensional roadmap. The method according to claim 1.

3. The aforementioned at least one roadmap comprises at least one 3D roadmap. The method according to claim 1.

4. An operation to obtain contrast-enhanced X-ray image data of the target object, further comprising the operation wherein the contrast-enhanced X-ray image data is acquired using an X-ray imaging modality together with a contrast agent, the interventional device is present in the contrast-enhanced X-ray image, and the contrast-enhanced X-ray image data covers at least one cardiac cycle of the patient, The operation in ii) generates a sequence of 3D roadmaps of the target object over time, covering multiple phases of at least one cardiac cycle of the patient, using the 3D model and the contrast-enhanced X-ray image data. The operation in iii) determines the reference position corresponding to the sequence of the 3D roadmap of the target object over time, The operation in vi) selects a specific 3D roadmap from the sequence of 3D roadmaps over time, The operation in vii) transforms the particular 3D roadmap using the reference position in iii) corresponding to the particular 3D roadmap and the position of the tip of the intervention device determined in v). The method according to claim 1.

5. An operation to acquire an ECG signal while acquiring the non-contrast X-ray image data, and to process the ECG signal to determine the phase of the patient's cardiac cycle corresponding to the non-contrast X-ray image data, and By matching the phase of the patient's cardiac cycle in the non-contrast X-ray image data to the phase of the patient's cardiac cycle in the selected specific 3D roadmap, the operation of selecting the specific 3D roadmap in vi) Furthermore, prepare The method according to claim 4.

6. i) The 3D model includes a 3D vascular centerline and a 3D surface contour representing at least one of the luminal vascular surface, plaque, and 3D mask, The method according to claim 1.

7. i) The 3D angiography imaging modality is selected from the group consisting of computed tomography (CT), rotational X-ray angiography, 3D ultrasound, or magnetic resonance imaging (MRI). The method according to claim 1.

8. The operation involves processing the contrast-enhanced X-ray image data of the target object to determine the phase of the patient's cardiac cycle relative to the image frame, and associating the phase of the cardiac cycle with a 3D roadmap corresponding to the image frame. Furthermore, the method according to claim 4.

9. The sequence of the 3D roadmap over time is derived from the 3D model in i) and a sequence of X-ray angiography images of the target object acquired using a contrast agent, or The sequence of the 3D roadmap over time is derived from the 3D model in i) and two sequences of X-ray angiography images of the target object acquired using a contrast agent. The method according to claim 4.

10. The sequence of the 3D roadmap over time is defined in a 3D coordinate system. The method according to claim 4.

11. i) The 3D model includes a 3D surface contour representing the luminal vessel surface and plaque, and The sequence of the 3D roadmap over time is defined in a 3D coordinate system and includes information characterizing vascular contours and plaques. The method according to claim 4.

12. vii) The aforementioned operation applies a transformation to the specific roadmap in order to correct the movement. The method according to claim 1.

13. The aforementioned movements include breathing movements and / or heart movements and / or patient movements and / or table movements. The method according to claim 12.

14. vii) The conversion is based on the viewpoint used to acquire the non-contrast X-ray image data. The method according to claim 1.

15. The visual representation of the converted roadmap is generated by a) projecting the superposition of the converted roadmap onto non-contrast X-ray image data using transparency mode, and / or b) projecting the boundaries of the converted roadmap onto the non-contrast X-ray image data. The method according to claim 1.

16. The visual representation of the roadmap being converted is configured such that it does not obscure any instruments used to treat the object of interest. The method according to claim 1.

17. The aforementioned non-contrast X-ray image data is obtained by subtracting the baseline image. The method according to claim 1.

18. The operations described in iv) to viiii) are repeated for consecutive frames of the live image sequence acquired without contrast agent. The method according to claim 1.

19. The interventional device is selected from the group consisting of a guide catheter, a guide wire, or other intracavitary devices or instruments. The method according to claim 1.

20. The system further includes an operation to display the superimposed visual representation of the converted roadmap of interest on the non-contrast X-ray image data. The method according to claim 1.

21. The object of interest comprises the coronary artery trees, blood vessels, and / or a portion of the heart of the patient, The method according to claim 1.

22. A system for generating images of a patient's object of interest, wherein the object of interest comprises a part of the patient's vascular system, and the system The system comprises at least one processor configured to perform the method of claim 1 when executing program instructions stored in memory, system.

23. An image acquisition subsystem configured to acquire non-contrast X-ray image data, wherein the image acquisition subsystem further comprises an image acquisition subsystem that uses an X-ray imaging modality. The system according to claim 22.

24. The system further includes a display subsystem configured to display the superposition of the visual representation of the roadmap of interest being converted on the non-contrast X-ray image data. The system according to claim 22.

25. A non-temporary program storage device that tangibly embodies a program of instructions executable on a machine for performing the operation of claim 1 for generating an image of a patient's object of interest, wherein the object of interest comprises a part of the patient's vascular system.

Images