Method and system for aligning intra-object data with extra-object data

Abstract

To provide a method for correlating intraluminal data with x-ray image data.SOLUTION: A method comprises: a step of providing x-ray image data of a tubular organ; a step 101 of providing intraluminal data of the tubular organ; a step 102 of synchronizing the x-ray image data with the intraluminal data based on a cardiac cycle extracted from an available input signal and / or the x-ray image data and / or the intraluminal data to match x-ray image data with intraluminal data in a specific cardiac phase; and a step 103 of co-registering x-ray image data with intraluminal data by creating a cumulative image at the specific cardiac phase which reconstructs a path followed by an intraluminal wire used for acquiring the intraluminal data to link the intraluminal data to spatial locations of the path followed by the wire within the x-ray image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates in particular to the technical field of medical imaging in percutaneous interventions, but also to the imaging of inside and outside of objects. Mutual alignment The invention may also find application in any field where there is a need to [Background technology]

[0002] Coronary artery disease is the narrowing or blockage of the coronary arteries, usually due to atherosclerosis. Atherosclerosis is the buildup of cholesterol and fatty deposits (called plaque) on the inner walls of the arteries. These plaques can restrict blood flow to the heart muscle by physically clogging the arteries or by causing abnormal arterial tone and function. If there is not enough blood supplied, the heart is starved of oxygen and vital nutrients it needs to function properly. This can cause chest pain called angina. If the blood supply to a part of the heart muscle is completely cut off or the heart's energy demands become much greater than the blood supply, a heart attack (damage to the heart muscle) or stroke can occur. Atherosclerosis is treated using a variety of methods in the arteries of the heart, head, neck, and peripheral parts of the body. The most common methods, such as angioplasty, bare metal stenting, drug-eluting stenting (permanently implanted and bioabsorbable), various types of energy delivery, and rotational atherectomy, all treat the artery evenly around the target length of the arterial lumen.

[0003] Currently, x-ray angiography is an imaging modality used during the treatment of vascular disease with minimally invasive procedures also known as percutaneous coronary intervention (PCI). X-ray angiography imaging is used to guide interventional procedures to treat atherosclerosis. During PCI, an (interventional) cardiologist passes a deflated balloon or other device on a catheter through the blood vessels from the femoral or radial artery in the groin until it reaches the site of blockage in the artery. PCI usually involves opening the artery by inflating the balloon with the goal of restoring unobstructed blood flow. The artery can be kept open by placing a stent or scaffold at the site of the blockage.

[0004] X-ray angiography is also the standard imaging technique (known as X-ray diagnostic angiography) for the anatomical evaluation of coronary arteries and the diagnosis of vascular disease. Even when the geometric dimensions of the stenosis are quantified, for example, by two-dimensional (2D) quantitative coronary analysis (QCA) as taught by Non-Patent Document 1, or by two-dimensional (2D) QCA as contemplated by Non-Patent Document 2, it is not easy to translate the anatomical lumen reduction into the severity of hemodynamic stenosis, since the severity of functional coronary lesions depends on the hemodynamics of the coronary arteries. This weak relationship between the anatomical and functional severity of coronary stenosis has also been shown by Non-Patent Document 3 using a new promising non-invasive technique using multi-detector computed tomography coronary angiography (CCTA). For example, in the case of intermediate coronary lesions, defined as luminal reduction of 30-70%, it is not always clear whether the stenosis is a risk to the patient and whether it is advisable to take action. Overestimating the severity of the stenosis may lead to treatment that is unnecessary in retrospect. This therefore exposes the patient to unnecessary risks. However, underestimating the stenosis may create risks, as the patient will be left untreated if the stenosis is actually severe. Especially in these situations, it is advisable to perform an additional functional evaluation to support appropriate decision-making.

[0005] Therefore, additional vascular imaging modalities are usually performed during PCI to obtain additional information of the diseased vessel. For example, atherosclerotic plaques are evaluated by intravascular imaging modalities such as intravascular ultrasound (IVUS) and optical coherence tomography (OCT) during the interventional procedure. Another example of an additional imaging modality is intravascular fractional flow reserve, which is used to identify the hemodynamic significance of a lesion. Fractional flow reserve (FFR) has been increasingly used over the past 15-20 years as a way to identify and effectively target coronary lesions most likely to benefit from PCI. FFR is a technique used to measure pressure differentials across a coronary stenosis to determine the likelihood that the stenosis will impede oxygen delivery to the myocardium. This technique involves the percutaneous insertion of a pressure-transducing wire into the coronary artery, guided by x-ray coronary angiography, and measuring the pressure behind (distal) and before (proximal) the lesion. FFR is best performed at maximal coronary diastolic conditions, since at maximal coronary diastolic conditions, blood flow to the myocardium is proportional to the myocardial perfusion pressure. Thus, FFR provides a quantitative assessment of the functional severity of coronary lesions, as explained by Fischer et al., 2003, 141:1421-1436. Since measuring FFR using a pressure-transducing wire is invasive, this technique is also called invasive FFR. Another example of an intravascular imaging device for assessing the hemodynamic significance of lesions is the instantaneous wave-free ratio (iFR). This is not different from FFR, except that imaging is performed at rest and does not require maximal coronary diastolic conditions. Coronary flow reserve (CFR) is another example of intravascular imaging to assess the hemodynamic significance of a lesion, defined as the maximal increase in blood flow through a coronary artery above its normal resting volume.

[0006] Intravascular imaging such as IVUS, OCT, FFR, iFR, CFR is performed while an intravascular device is pulled back through the blood vessel and generates a large number of images showing the cross-section of the blood vessel in the case of IVUS or OCT. In the case of FFR, iFR, or CFR, the resulting measurement is a pressure value. In general, there are two approaches to intravascular imaging: a) automatic pullback, where the pullback of the intravascular imaging device is performed automatically at a predefined pullback speed, and b) manual pullback, where the (interventional) cardiologist manually pulls back the intravascular imaging device.

[0007] The challenge of these imaging modalities is to correlate specific image positions or measurements to intravascular locations in angiographic views. In practice, physicians look for anatomical landmarks, such as branching points and multiple side branches, that are visible in both x-ray angiographic images and intravascular images to cross-correlate the information of both imaging modalities. However, this is time-consuming and prone to errors, and may not result in optimal treatment of vascular disease. Furthermore, when the results of intravascular imaging devices are measurements such as FFR, iFR, CFR, such a correlation approach becomes difficult or even impossible.

[0008] The solution to this problem is to compare intravascular images with x-ray angiographic images. Mutual alignment is disclosed in US Pat. No. 6,399,633, which is performed by generating a 3D reconstruction of the blood vessels based on two x-ray angiographic projections. Mutual alignment To do this, the user must add two x-ray fluoroscopic images, identifying the tip of the intravascular transducer in each fluoroscopic image. This allows Mutual alignment This results in a 3D point in space that is used to perform the procedure. This approach assumes that the center of the 3D model corresponds to the intravascular path within the vessel during pullback. Furthermore, errors in the 2D segmentation of the vessel in the x-ray angiogram image are important because they directly affect the 3D reconstruction of the vessel and the resulting centerline. Mutual alignmentThese drawbacks were addressed by US Pat. No. 6,399,436, in which a 3D model consisting only of a 3D centerline representing the pullback path of the intravascular device is truly representative of the intravascular path, and the method does not rely on a 2D segmentation of the vessel to generate the 3D model. However, this method requires that information regarding the pullback speed is available. A further drawback of the above methods is that these 3D models require two x-ray angiographic images.

[0009] In Patent Document 3, a vascular image and an X-ray image are obtained by tracking an intravascular transducer located at the tip of a guidewire with X-ray fluoroscopy during controlled pullback. Mutual alignment A method is disclosed for performing the procedure described above. A drawback of this method is that it requires automatic pullback of the intravascular wire.

[0010] In Patent Document 4, the IVUS transducer located at the tip of the IVUS wire is tracked by x-ray fluoroscopy during the pullback, and IVUS and x-ray images are combined. Mutual alignment A method is disclosed for measuring the length of a stent using IVUS. The drawbacks of this method are the limitation to IVUS and the limited accuracy of the length assessment due to foreshortening or out-of-plane magnification errors, which are important when selecting the correct stent and / or balloon length during the treatment of vascular disease.

[0011] Therefore, in order to at least partially solve the above-mentioned shortcomings, the use of intravascular imaging modalities (intra-object imaging) and x-ray angiography imaging modalities (inter-object imaging) has been proposed. Mutual alignment The process needs to be improved. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] U.S. Patent No. 7,729,746 [Patent Document 2] U.S. Patent No. 9,811,939 [Patent Document 3] U.S. Patent No. 7,930,014 [Patent Document 4] U.S. Patent No. 8,565,859 [Patent Document 5] U.S. Patent Application Serial No. 16 / 739,718 [Non-patent literature]

[0013] [Non-Patent Document 1] Gronenschild E et al. "CAASII: A second generation system for off-line and on-line quantitative coronary angiography." Cardiovascular Diagnosis 1994; 33:61-75 [Non-Patent Document 2] Yoshinobu Ohnuma et al., "A new dedicated 3D quantitative coronary analysis methodology for bifurcation lesions," EuROIntervention 2011; 6:1-00 [Non-Patent Document 3] Sarno et al., "Inadequacy of noninvasive multidetector computed tomography coronary angiography to induce coronary revascularization: Comparison with invasive angiography," JACC CardiovascInterv 2009 Jun;2(6):550-7 [Non-Patent Document 4] Pijls et al., Measurement of fractional flow reserve to assess the functional severity of coronary artery stenosis, N Engl J Med 1996, 334:1703-1708 [Non-Patent Document 5] Pan et al., "Real-time QRS detection algorithm," IEEE Transactions on Biomedical Engineering. BME-32 (3): 230-236. [Non-Patent Document 6] Frangi et al., "Multiscale Vascular Enhancement Filtering," Medical Image Computing and Computer-Assisted Intervention - MICCAI 1998 Lecture Notes in Computer Science 1496 / 1998:130 [Non-Patent Document 7] Li et al., "Grayscale Image Skeletonization from Incomplete Boundaries," Proceedings of the International Conference on Image Processing, ICIP 2008, October 12-15. [Non-Patent Document 8] Hao et al., "Vascular Layer Separation in X-ray Angiograms Using Fully Convolutional Networks," Proc. SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling. [Non-Patent Document 9] Ma et al., "Layer Separation for Vascular Enhancement in Interventional X-Ray Angiography Using Morphological Filtering and Robust PCA," Workshop on Augmented Environments for Computer-Assisted Interventions 2017, Springer. pp. 104-113 [Non-Patent Document 10] Ma et al., "Automatic online layer separation for vascular enhancement in x-ray angiography for percutaneous coronary intervention," Med Image Anal. 2017 Jul; 39:145-161 [Non-Patent Document 11] Matsumoto, H., "Cardiac Phase Detection in Intravascular Ultrasound Images," SPIE Proceedings,Medical Imaging 2008:Ultrasonic Imaging and Signal Processing, Vol.6920, 69200D. [Non-Patent Document 12] Shalbaf et al., "Real-time algorithms for extracting heart rate in invasive blood pressure," Blood Pressure, Biomed 2008, Proceedings 21, pp. 103-106. [Non-Patent Document 13] "Coronary Circulation" by Berne RB, Levy MN (2001), Cardiovascular Physiology, 8th Edition, Mosby / Elsevier, p227-240 [Non-Patent Document 14] Lu et al., "Coronary Artery Motion During the Cardiac Cycle and Optimal ECG Triggering for Coronary Artery Imaging," Investigative Radiology 2001;36:250-256. [Non-Patent Document 15] Zang et al., "Cascaded Attention Machines for Landmark Detection in 2D X-ray Angiography," Jan 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) [Non-Patent Document 16] Wagner et al., "Deep Learning-Based Guidewire Segmentation in X-Ray Images," SPIE Proceedings, Medical Imaging 2019:Physics of Medical Imaging, 1094844 (1 March 2019) [Non-Patent Document 17] Ambrosini et al., "Fully automatic and real-time catheter segmentation in fluoroscopy," International Conference on Medical Image Computing and Computer-Assisted Intervention 2017, Springer. pp. 577-585 [Non-Patent Document 18] "Overview of medical image registration methods," symposium of the Belgian hospital physicists association: 1996 [Non-Patent Document 19] Ma et al., "Fast prospective detection of contrast inflow in x-ray angiograms using convolutional and recurrent neural networks," International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 453-461 [Non-Patent Document 20] Klein et al., "Elastix: A toolbox for intensity-based medical image registration," Med. Imag. IEEE Trans. 29(1), 196-205. [Non-Patent Document 21] Gronenschild et al., "CAASII: A second generation system for off-line and on-line quantitative coronary angiography," Cardiovascular Diagnosis 1994; 33: 61-75. Summary of the Invention [Problem to be solved by the invention]

[0014] Accordingly, it is an object of embodiments of the present application to provide a method for acquiring multiple modality information, including endoluminal data. Alignment The present invention provides an improved method and system for performing the above-mentioned steps.

[0015] According to embodiments of the present application, there are provided devices, computer program products, and computer-implemented methods for correlating endoluminal data and x-ray image data, the devices, program products, and methods being configured under the control of one or more computer systems with specific executable instructions to: providing x-ray image data of the tubular organ; providing endoluminal data of the tubular organ; synchronizing the endoluminal data and the x-ray image data based on available input signals and / or cardiac cycles extracted from the x-ray image data and / or the endoluminal data to match the endoluminal data and the x-ray image data at a specific cardiac phase; By creating cumulative images at the specific cardiac phase, the x-ray image data is converted into endoluminal data. Mutual alignment wherein this cumulative image reconstructs the path followed by an endoluminal wire used to acquire the endoluminal data in order to link the endoluminal data to a spatial location of the path followed by the wire in the x-ray image; Equipped with.

[0016] According to one embodiment, creating a cumulative image comprises extracting image frames from the x-ray image data that include the endoluminal wire at the same particular cardiac phase, and forming an image of the endoluminal wire by accumulating the extracted image frames. Prior to forming the cumulative image, the endoluminal wire may be identified or segmented.

[0017] Optionally, the start of acquisition of the endovascular data may be determined by analyzing the movement curve created by calculating the absolute distance of the detected endoluminal wire between a reference frame and the remaining image frames of the sequence.

[0018] The x-ray image data usually comprises fluoroscopic image data, acquired without contrast agent, and angiographic image data, acquired with contrast agent, as is usual in an acquisition session. Since it is not necessary to detect the vascular tree, but only the wire path, an accumulated image is advantageously created from the fluoroscopic image data. The accumulated image thus created can for example be superimposed on the angiographic image data.

[0019] Radiopaque wire sections extracted from the x-ray fluoroscopic image data are used to visualize the structure of tubular organs. Mutual alignmentIt may be advantageous to use the resulting image to calculate foreshortening along the length of the segment and / or to convert pixels to millimeters.

[0020] If the endoluminal data comprises values ​​such as pressure, such values ​​may be superimposed on the angiographic image data at corresponding spatial locations and / or may be used to measure the pressure of the tubular organ. Mutual alignment It is advantageous to be able to have the resulting image displayed as a graph along the length of the segment. Such a display can, for example, aid in determining whether a lesion is focal or diffuse.

[0021] If the endoluminal data includes an endoluminal image, one embodiment includes a step of receiving an input to select a spatial location within the angiographic image data and a step of indicating an endoluminal image already acquired from that spatial location.

[0022] For example, motion not due to the cardiac cycle, such as breathing and / or patient motion, Alignment It would be advantageous to be able to modify the process to improve it, for example by tracking the tip of the catheter in the fluoroscopic image data.

[0023] In an advantageous embodiment, the tracking is performed using a deep learning based Bayesian filtering approach.

[0024] If the temporal relationship between the x-ray image data and the endoluminal data is known, the step of synchronizing the x-ray image data with the endoluminal data can include extracting a cardiac cycle from an ECG signal, the x-ray image data, or the endoluminal data, selecting the particular cardiac phase, calculating a delay between the particular cardiac phase and the x-ray image data and the endoluminal data, and aligning the x-ray image data and the endoluminal data by correcting the calculated delay.

[0025] If the temporal relationship between the x-ray image data and the endoluminal data is unknown, the method may further comprise the steps of extracting a cardiac cycle from the x-ray image data or from an ECG signal correlated with the x-ray image data and from the x-ray image data, identifying the endoluminal data, a particular cardiac phase of the extracted cardiac cycle, calculating a delay between the particular cardiac phase, the x-ray image data and the endoluminal data, and aligning the x-ray image data and the endoluminal data by correcting for the calculated delay.

[0026] In one embodiment, the reliability of the ECG signal is assessed by matching the ECG signal with a template and determining a correlation function between the ECG signal and the template.

[0027] If the x-ray image data comprises image frames containing a fixed element associated with the cardiac motion, e.g. the tip of a catheter, an embodiment comprises determining the motion of the fixed element, e.g. by image segmentation techniques or machine learning, and extracting the cardiac cycle from said motion. It is advantageous to enhance the structures associated with the cardiac motion in the image frames, e.g. by using principal component analysis techniques.

[0028] If the endoluminal data comprises two-dimensional images, one embodiment comprises determining the motion of the luminal boundary relative to a reference point or line, such as the location of the imaging probe used to extract the images, and extracting a cardiac cycle from said motion. The motion of the luminal boundary comprises calculating a distance from a reference point or line to the luminal boundary of each or some of the images, and combining the calculated distances to reconstruct a cardiac cycle.

[0029] If the endoluminal data comprises non-image data, such as pressure data, embodiments comprise determining cyclic features in the non-image data and extracting cardiac cycles from the cyclic features.

[0030] The embodiment also provides a combination device comprising: an angiographic imaging device for acquiring an x-ray image of a tubular organ; an intravascular device for acquiring endoluminal data of the tubular organ; and an interface element for receiving the x-ray image from the angiographic imaging device and the endoluminal data from the intravascular device. Alignment The data processing module synchronises the x-ray image data with the endoluminal data based on available input signals and / or x-ray image data and / or cardiac cycles extracted from the endoluminal data by matching the x-ray image with the endoluminal data at a particular cardiac phase and creating a cumulative image at the particular cardiac phase. Mutual alignment This is configured to reconstruct the path followed by an endoluminal wire used to acquire the endoluminal data and link the endoluminal data to the spatial location of the path followed by the wire in the x-ray image in order to perform one or more steps disclosed above.

[0031] In one embodiment, the combination device is part of an angiographic imaging device, in which case the data processing module of the combination device is one of the processors of the angiographic imaging device which has access to x-ray images, and which advantageously has an input for receiving endoluminal data from an intravascular device.

[0032] The combination device may also be part of the intravascular device, in which case the data processing module of the combination device is one of the processors of the intravascular device which has access to the endoluminal data, and which advantageously has an input for receiving x-ray images from the x-ray imaging device.

[0033] In one embodiment, the system further comprises an ECG module interfaced with the x-ray machine, the intravascular device, or the combination device to enable the data processing module to process ECG signals.

[0034] Advantageously, the x-ray images, the intravascular data and the ECG signals are input to the combination device at least in part via a frame grabber, an A / D converter, a DICOM (Digital Imaging and Communications in Medicine) transfer protocol, or a combination thereof.

[0035] In one embodiment, the intravascular device comprises an intraluminal wire having a sensor at its tip for collecting endoluminal data via a console while the endoluminal wire is pulled through the tubular organ, the sensor being adapted to collect endoluminal data, such as pressure or images, selected from the group consisting of FFR, iFR, CFR, IVUS, and OCT data.

[0036] The features of the embodiments herein and advantages derived therefrom will become more apparent from the following description of non-limiting embodiments illustrated in the accompanying drawings. [Brief description of the drawings]

[0037] [Figure 1] 1 illustrates a flowchart of a method for determining a dynamic coronary roadmap according to embodiments herein. [Diagram 2] 1 illustrates a functional block diagram of an exemplary single plane angiography system. [Diagram 3] 1 shows a flow chart for reading patient-specific x-ray image data and intravascular data. [Figure 4A] 1 shows a schematic diagram of an FFR measuring device. [Figure 4B] A schematic diagram of an FFR wire inserted into the left coronary artery is shown. [Figure 5A]An example of measured pressure is shown (x-axis represents time, y-axis represents pressure). [Figure 5B] An example of one IVUS image frame and an OCT frame is shown. [Figure 6] Illustrative examples of x-ray fluoroscopy and x-ray angiography frames are shown. [Figure 7] 1 shows a sequence of x-ray fluoroscopic images covering the pullback of an intravascular imaging device. [Figure 8] A method for synchronizing endovascular data to the fluoroscopy frame is shown. [Figure 9] The real-world time relationships between the loaded image data indicate a possible synchronization approach. [Figure 10] 1 shows a flowchart for synchronizing endovascular data and fluoroscopy frames when real-world time is unknown. [Figure 11] 1 shows an example of an ECG signal with a unique phase within the cardiac cycle. [Figure 12] Several approaches for extracting the cardiac cycle from a sequence of x-ray fluoroscopic images are presented. [Figure 13] We present an approach to extract the cardiac cycle from 2D intravascular imaging data (e.g., IVUS). [Figure 14] We present an alternative approach for extracting the cardiac cycle from 2D intravascular image data (eg, IVUS). [Figure 15] 4 shows the correlation between pressure and ECG signals. [Figure 16] We show how to synchronize intravascular image data and x-ray fluoroscopy frames when real-world time is unknown. [Figure 17] FIG. 2 illustrates an example block diagram of an x-ray cinefluorographic unit according to an embodiment of the present application. [Figure 18] 1 shows a flow chart for synchronizing endovascular data to an x-ray fluoroscopy frame when real world time is known. [Figure 19] 1 shows a flowchart of a procedure for co-registering x-ray image data and intravascular data. [Figure 20] 1 illustrates an approach for generating a cumulative image. [Figure 21] 1 illustrates a method for creating a cumulative image that is superimposed on an x-ray angiography image frame. [Figure 22A] An example is shown providing co-registration results where the intravascular imaging modality represents the FFR modality (1D intravascular image data). [Figure 22B] Another example is shown providing co-registration results where the intravascular imaging modality represents the FFR modality (1D intravascular image data). [Diagram 23] An example is shown providing co-registration results where the intravascular imaging modality represents the IVUS modality (2D intravascular image data). [Figure 24] An example of combining geometric information of a target vessel with co-registration is shown. [Diagram 25] Illustrate foreshortening. [Figure 26] 13 shows an example illustrating the amount of foreshortening along the length of pullback. [Figure 27] 1 illustrates various high-level system components into which the operations of the present application may be integrated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] This application combines the intravascular modality (intra-object) and the x-ray angiography imaging modality (inter-object). Mutual alignment The present application relates to a method and system for Mutual alignmentThe term "inter-object mapping" refers to aligning intravascular data and x-ray angiographic images from the same subject to map intra-object information to inter-object space. The method is primarily disclosed using FFR as an example intravascular modality, but the method is not limited to FFR as such intravascular modalities and is applicable to IVUS, OCT, iFR, CFR, etc. An intravascular wire (see Fig. 4A, Fig. 4B and corresponding description) inserted into the target vessel consists of an intravascular imaging sensor and typically resides around the top of the intravascular wire radiopaque material. This radiopaque material (or section) is within a predefined distance of the intravascular imaging sensor and is typically only a few millimeters. Depending on the type of intravascular wire (which can also depend on the type of intravascular device), the radiopacity of the intravascular sensor can vary. Throughout this patent application, a radiopaque wire section is defined to be either a radiopaque section or an intravascular sensor. Radiopaque wire section as used in the specification of this application is used to identify the location of intravascular measurements in an x-ray image. In the description of this method, x-ray angiography is used, which refers to an x-ray acquisition of the vasculature (e.g., the coronary vascular tree, in which the vasculature and other objects that are radiopaque are visible after administration of contrast fluid). X-ray fluoroscopy, as used in this application, refers to an x-ray acquisition without the use of contrast, and the vasculature may not be visualized. In this application, the term image or image frame refers to a single image, and the term image sequence refers to multiple images acquired over time, and when used in connection with x-ray, it includes multiple frames covering one or more phases of the cardiac cycle, and when used in connection with intravascular data, it includes multiple frames acquired during pullback of an intravascular device. An x-ray angiography image sequence may include contrast administration, and image frames before contrast administration will not include contrast and therefore will not include enhancement of the vasculature. An x-ray angiography image sequence and an x-ray fluoroscopy sequence may include multiple frames covering one or more phases of one or more cardiac cycles.Throughout this application, a cardiac cycle is defined as a period of time that is specific to a patient and covers one heart beat of the patient. A cardiac cycle can be defined, for example, as the period between successive R wave apexes in the patient's ECG signal (see FIG. 7, 710) or successive peaks in the patient's pressure signal (see FIG. 15, 1504). Phase refers to a moment (or period) in the patient's cardiac cycle, for example, end systole, which defines the moment in the cardiac cycle when the blood volume in the left ventricle is minimal. Phase can be measured as an offset from the apex of the R wave in the patient's ECG signal, as shown in FIG. 7 and corresponding description, or as an offset from the peak in the patient's pressure signal, as shown in FIG. 13 and corresponding description. The term cardiac cycle information as used throughout this application refers to information such as the start of a cardiac cycle, the end of a cardiac cycle, or a particular phase in a cardiac cycle.

[0039] FIG. 1 shows a flow chart illustrating operations according to embodiments of the present disclosure. These operations use an imaging system capable of acquiring and processing a two-dimensional image sequence of a vascular organ (or a part thereof) or other object of interest, and an intravascular system for acquiring intravascular data. For example, a single-plane or biplane angiography system, such as manufactured by Siemens (Artis zee biplane) or Philips (Allura Xper FD), can be used to acquire x-ray image data. For example, an FFR system, such as manufactured by ACIST (ACIST RXi System) or Boston Scientific (Polaris system) or Philips (Core Mobile), can be used to acquire intravascular data.

[0040] FIG. 2 is a functional block diagram of an exemplary single plane angiography system, which includes an angiography imager that operates under command from a user interface module 216 and provides data to a data processing module 210. The single plane angiography imager 212 captures x-ray image sequences of a two-dimensional target vascular organ, for example in a posteroanterior direction. The single plane angiography imager 212 typically includes an x-ray source and detector pair mounted on the arms of a supporting gantry. The gantry positions the x-ray source and detector arms at various angulations relative to a patient supported on a table between the x-ray source and detector. The data processing module 214 may be implemented by a personal computer, workstation, or other computer processing system. The data processing module 214 processes the two-dimensional image sequences captured by the single plane angiography imager 212 to generate data as described herein. The user interface module 216 interacts with a user and communicates with the data processing module 214. User interface module 216 can include various types of input / output devices, such as a display screen for visual output, a touch screen for touch input, a mouse pointer or other pointing device for input, a microphone for audio input, speakers for audio output, a keyboard and / or keypad for input, etc.

[0041] The angiographic imaging device 2701 may for example be interfaced with a console 2702 of an intravascular system as shown in Fig. 4A, either by a combination device 2703 comprising a data processing module 2704 and a user interface module 2705 as shown in Fig. 27, or directly by providing an input device in which the data processing module of the angiographic imaging device is adapted to process data coming from the intravascular system console and can receive intravascular data from such an intravascular system console. Also, a part of the intravascular system, in particular its data processing module, can function as a unit for processing data coming from the angiographic system console via an input device. In any of the possible configurations, the data processing module and the user interface module of the angiographic imaging device, the combination device or the intravascular system console, or a combination thereof, cooperate to perform the operations of Fig. 1, as will be explained below.

[0042] Such operations may also be performed by software code embodied in a computer product (e.g., optical disk or other type of persistent memory such as a USB drive or network server). The software code may be directly loaded into the memory of any data processing system to perform the operations of FIG. 1. Such a data processing system may be physically separate from the angiography system used to acquire the images or the endovascular system used to acquire the endovascular data. Such a data processing system may also be included in the combination device described above, utilizing any type of data communication to acquire such angiography images and endovascular data, for example, as input to the combination device.

[0043] In the following example, it is assumed that the x-ray imaging system has already acquired and stored at least one two-dimensional image sequence of the object of interest, and that the endovascular system has already acquired and stored endovascular data such as information such as pressure, FFR / iFR / CFR data, IVUS / OCT images, etc.

[0044] An embodiment will now be disclosed with reference to Fig. 1. It will be apparent that the operations shown therein can be performed in any logical order and some can be omitted. As it is the object of this application to provide selected (e.g. optimal) workflows that can be used during an intervention, reference will also be made to steps of example workflows.

[0045] As shown in FIG. 1, the workflow is composed of several steps. First, in step 101, patient-specific image data is acquired. This patient-specific image data includes both x-ray image data and data acquired by intravascular modalities. Block 101 illustrates an example in which pressure data acquired by FFR measurement is used. The x-ray image data can be x-ray angiography image data and x-ray fluoroscopy image data. In step 102, the pressure data is synchronized with the acquired x-ray image data, and in step 103, the x-ray image data is combined with the pressure data. Thus, the x-ray angiography image data is Mutual alignment The pressure data measured at the specified location and Mutual alignment The result is then displayed or reported by step 104. In the next section, these steps are described in more detail.

[0046] Step 101: Read patient-specific x-ray image data and intravascular image data. The reading of the patient-specific x-ray image data and intravascular data (e.g., pressure data or IVUS images) is explained in detail with reference to FIG. 3. The patient-specific x-ray image data and intravascular data are read during an x-ray coronary angiogram, which is a common procedure of a PCI procedure or a diagnostic x-ray angiogram procedure. An example of an x-ray angiogram image is provided by 602 in FIG. 6 showing the left coronary artery. Here, 601 in FIG. 6 shows an example of a fluoroscopic x-ray image acquired at the same x-ray angulation of the same patient. In the drawing 601 in FIG. 6, an intravascular wire (e.g., an FFR wire) is inserted into the left coronary artery, and a radiopaque marker of the intravascular wire (e.g., an FFR wire) is visible (603). FIGS. 4A and 4B show the details of the intravascular system in more detail, and in the example of FIGS. 4A and 4B, an FFR system is shown. FIG. 4A shows a schematic diagram of an FFR measurement device. The FFR wire (401) is connected to the console (404) via an interface (405). The FFR wire is a very thin, flexible wire that is inserted through a standard diagnostic catheter (406, FIG. 4B). Typically, the tip of the FFR wire has a radiopaque material (403) that makes it visible in a fluoroscopy room. The pressure sensor (402) is placed within a known distance from the radiopaque material (403). FIG. 4B shows a schematic of the FFR wire inserted into the left coronary artery (LAD) 412 distal to a coronary occlusion (407). A diagnostic guide catheter (406), inserted percutaneously through the vasculature toward the aorta (410), is used to push the wire toward the desired location. If measurements of the LAD (412) or circumflex (411) are required, the guide catheter is placed in the left coronary ostium, and if measurements of the right coronary artery (RCA) 413 are required, the diagnostic guide catheter is placed in the right coronary ostium. Drawing 501 in FIG. 5A shows an example of measured pressures (x-axis represents time, y-axis represents pressure). In 501, two pressure curves are visualized, one pressure curve measured with the FFR wire and representing the pressure in the coronary artery (

number

number

[0047] In step 301 of Figure 3, an intravascular wire (e.g., a pressure wire) is placed at a distal location in a target vessel. An example of a fluoroscopic image acquired after the pressure wire has been placed is shown by 601 in Figure 6. The radiopaque portion of the intravascular wire (603 indicates the FFR wire) is clearly visible and no visible information of the vessel is present in 601.

[0048] In step 302 of FIG. 3, an x-ray angiogram image is acquired. This image or image sequence may include: Mutual alignment This will be explained in more detail in the discussion of step 103 in FIG. 1. An example of such an x-ray angiogram is provided by 602 in FIG. 6. This angiogram is acquired at the same angulation as image 601, and contrast is administered through the guiding catheter so that the left coronary vessels are clearly visible (604).

[0049] In step 303 of Figure 3, recording of the intravascular modality begins, and in the case of FFR, recording of pressure data begins. If the intravascular modality is FFR, maximum coronary blood flow (during coronary diastole) is induced, for example, by injecting adenosine or papaverine.

[0050] Next, in step 304, a fluoroscopic image sequence is acquired covering the pullback of the intravascular imaging device (e.g., the pullback of the FFR pressure wire). The pullback covers several cardiac cycles and depends on the pullback speed, the patient's heart rate, and the length of the pullback. Typically, the pullback covers 8-40 cardiac cycles. FIG. 7 shows three fluoroscopic frames (701, 702, and 703) of the acquired fluoroscopic image sequence. The radiopaque markers (material) of the intravascular wire are visible (704) as is the guide catheter (705) in the fluoroscopic image frames and are highlighted for illustration purposes. Each fluoroscopic frame is acquired within any instant (phase) of the cardiac cycle, as indicated by 706, 707, and 708 in the electrocardiogram (ECG) signal (709) representing the cardiac cycle (710). The start of the cardiac cycle is, for example, the apex of the R wave (711), and all acquired x-ray frames (706, 707, and 708) represent a phase within the cardiac cycle (710), which is given a time offset with respect to the start of the cardiac cycle (710) for acquiring each x-ray frame. It should be noted that the ECG signal in FIG. 7 represents the ECG period that corresponds to the image frame (to the right of the visualized ECG signal), and it is not necessary that it is from the same cardiac cycle. Typically, the ECG signal is acquired together with the image data and stored together in DICOM, which allows identifying each image frame with respect to the ECG signal. This information is used further during the description of step 102 in FIG. 1. If an ECG signal is not acquired or is unavailable, the time (instant or phase) of a particular image frame in the image sequence with respect to the cardiac cycle can be extracted from the image data itself. The method is based on motion extracted from features in an x-ray fluoroscopic image sequence, such as a catheter, and is further described by step 1002 of FIG. 10, or from motion extracted from an intravascular image sequence, and is further described by step 1003 of FIG. 10.

[0051] Finally, in step 305 of FIG. 3, recording of the intravascular modality is stopped after pullback of the intervascular device has stopped.

[0052] Step 102: Synchronize the x-ray image data and the intravascular data In step 102, intravascular data (e.g., pressure data) is synchronized with the image data captured in step 101 (more specifically, with reference to the flowchart in FIG. 3). As illustrated by the flowchart in FIG. 3, the intravascular data is captured during pullback of the intravascular imaging device, which is captured by a sequence of fluoroscopic images (304). The synchronization of the intravascular data (e.g., pressure data, IVUS / OCT images) is the time alignment of the intravascular data with the fluoroscopic frames in the fluoroscopic sequence, which is illustrated in FIG. 8 (where the data represents pressure data). FIG. 8 shows a typical example of a sequence of fluoroscopic images, where 801 indicates several consecutive x-ray image frames in such a sequence, 807 indicates the corresponding ECG signal, and 808 indicates the pressure data signal (pressure waveform). The phases within the cardiac cycle at which fluoroscopic image frames (803, 804, 805 and 806) are acquired relative to the ECG signal (807) and pressure waveform (808) are shown in FIG.

[0053] Once the data has been synchronized in step 102, the cardiac cycles are then extracted and within each cardiac cycle, a specific cardiac cycle is defined. This information will be used by step 103 to reduce cardiac motion. There are two scenarios in how to synchronize the data, with different approaches. If the real-world time relationship between the loaded image data (x-ray image data, ECG data, intravascular image data) is not known relative to each other, a synchronization algorithm is needed to align the cardiac cycles within the loaded image data. If the real-world time relationship between the loaded image data is known relative to each other, a direct synchronization can be performed. This is illustrated in FIG. 9, where the second scenario is shown. In FIG. 9, 905 represents the real-world time, which is defined as local time. The sequence of x-ray fluoroscopic images is represented by 901, with each successive frame being represented by a dot (904, which represents the first frame of the x-ray fluoroscopic sequence). The ECG signal is represented by 902, the intravascular image data is represented by 903, and in this example the intravascular image data represents a pressure signal. At real-world time (906), a sequence of x-ray fluoroscopic images is taken. Profit The real-world time 907 is then the acquisition of the ECG signal begins. The real-world start time of the pressure data is represented by 908. Since the real-world start times are known, a (one-time) calibration needs to be performed to extract possible signal delays (909, 910). The situation shown in FIG. 9 is typically the case in a system configuration where the image data to be read is physically collected by one or more systems, as shown in FIG. 27. The term "physically collected" in this context means generating image data in a memory that knows the real-world time. For example, when a system digitizes x-ray fluoroscopic image data by a frame grabber, the system knows when this data was digitized in real-world time. The same applies when either a frame grabber or an A / D converter is used to acquire the ECG signal and the intravascular image data by the A / D converter, depending on the type of intravascular image device.

[0054] If the ECG signal and the fluoroscopic image data are read by file transfer (e.g., DICOM files), there is real-world time alignment between the ECG and the respective fluoroscopic image frames, but there is no real-world time alignment with the actual intravascular image data. Furthermore, in this situation, the ECG data may not be available or the ECG data is acquired separately using an A / D converter. A typical situation for this is when the x-ray image data and the intravascular image data are received in separate files (e.g., DICOM) and the real-world times are not connected to each other, as shown in FIG. 27. This can also happen when the data are provided in memory that lacks real-world time information for each other.

[0055] The real-world time relationship between loaded image data is unclear: The synchronization of intravascular image data and fluoroscopic image data when a real-world time relationship between the loaded image data is not available is further explained with reference to the flow chart of Figure 10. Since the presence of an ECG signal is not always guaranteed, the flow chart of Figure 10 includes an alternative step 1002 for extracting the cardiac cycle based on the loaded fluoroscopic image data (step 101) instead of the ECG data.

[0056] The first step 1001 of FIG. 10 involves extracting cardiac cycles based on the ECG signal. If an ECG signal is not available, the workflow proceeds to step 1002. Cardiac cycles can be determined / identified by matching the acquired ECG signal with a generic template of the ECG signal consisting of one or more cardiac cycles, and a correlation measure can be calculated as a minimization measure of the matching algorithm. This generic template of the ECG signal can be derived, for example, from a large population of patient data. The amount of correlation calculated, for example using a cross-correlation calculation, can be a measure of periodicity and can be used to evaluate the reliability of the ECG signal. Another option to extract periodicity from the read ECG signal is to perform a Fourier analysis on the read ECG signal. Detection of certain features in the ECG signal can be used to determine the cardiac cycle, for example, the ECG QRS complex detection algorithm taught by, for example, et al., "Echocardiography: A New Approach to Cardiac Cycle Detection," vol. 10, no. 5, pp. 1111-1115. Each detected QRS complex can be used as the start of a new cardiac cycle. Optionally, the reliability of the ECG signal can be evaluated based on a cardiac cycle discrimination algorithm, such as, for example, the ECG QRS complex detection algorithm taught by, for example, et al., "Evaluation of Cardiac Cycles in Cardiac Functions," IEEE Transactions on Cardiac Functions, vol. 10, no. 10, pp. 1111-1115, 2002. A predefined threshold can be used to evaluate whether the ECG signal is reliable or not. If the ECG signal is deemed unreliable, the workflow continues to step 1002.

[0057] If an ECG signal is present and the ECG signal is defined as reliable, the workflow continues as step 1003 .

[0058] In step 1002, the cardiac cycle is extracted from the fluoroscopic image data. This step is only applicable when (reliable) ECG data is not available. In general, the coronary artery motion consists of the cardiac motion due to the contraction of the heart, the respiratory motion due to the patient's breathing, and the patient motion due to the patient's movements. If the cardiac cycle is extracted from the fluoroscopy read by step 304 of FIG. 3, the tip of the catheter visible in the image sequence can be used. The tip of the catheter is located in the ostium of the coronary artery, so it will be associated with the motion due to the contraction of the heart. The cardiac cycle can be extracted from this motion. To extract the cardiac cycle, it is necessary to consider only the coronary artery motion, which can be extracted from the tip of the (guide) catheter. Thus, by analyzing the motion pattern of an object representing the cardiac motion, such as the tip of the catheter, a surrogate extraction of the cardiac motion provides information about the cardiac cycle in the sequence of fluoroscopic images. In FIG. 12, 1201 represents several (two) consecutive x-ray image frames in the sequence of fluoroscopic images. The movement of the diaphragm 1202 is visible in the successive x-ray image frames 1201. One approach to extracting the cardiac cycle is shown by 1203 in FIG. 12. Drawing 1203 shows the catheter extracted (detected) from the image frames 1201. This detection can be performed without user interaction using various known image segmentation techniques such as thresholding, convolution, correlation techniques, or other automatic segmentation techniques. For example, the catheter can be detected by applying a Frangi vascularity filter (Non-Patent Document 6) followed by non-maximum suppression (e.g., Non-Patent Document 7). Also, machine learning approaches can be used to detect the catheter tip, as disclosed, for example, in US Pat. No. 6,399,413 (which discloses a method using a deep learning-based Bayesian filtering approach for catheter tip detection). In summary, the method of US Pat. No. 6,399,413, which is incorporated herein by reference, uses a convolutional neural network to model the likelihood term of Bayesian filtering and comprehensively integrates it with particle filtering to provide a more robust tracking.Optionally, user input can be used to segment the image and to indicate the location of the tip of the catheter. A function that includes the displacement value between each successive frame based on the catheter segmentation.

number

[0059] In FIG. 12, drawing 1204 shows another approach to extracting the cardiac cycle. In this situation, the images are processed such that structures that do not represent cardiac motion are rejected and structures with cardiac motion, such as the catheter, are enhanced. In this situation, cardiac motion is the main cause of intensity variations in the image sequence and can be further analyzed by methods with source decomposition capabilities such as principal component analysis. First, the sequence of x-ray fluoroscopic images 1201 is processed (1205) such that only structures associated with cardiac motion, such as the catheter, are enhanced. This is performed by using the layer separation method taught by [8] or [9] or

[10] . Principal component analysis is usually used for dimensionality reduction. It transforms a multivariate dataset into a new Cartesian coordinate system so that most of the variance of the multivariate dataset can be represented by a few coordinates. Thus, the dimensionality reduction is usually achieved by retaining only a small number of coordinates in the new coordinate system without losing much information. Principal component analysis is used on the post-processed images (1204) to obtain the principal components of the image sequence. Frames of a sequence

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0060] Motion proxy function

number

number

[0061] Since the ECG data and the fluoroscopic image data are combined, a cardiac cycle that can be linked to the fluoroscopic image data is extracted from step 1001 or 1002. In order to be able to synchronize the fluoroscopic image data with the endovascular data, the cardiac cycle information of the endovascular data needs to be determined, which allows alignment of the fluoroscopic data with the cardiac cycle information.

[0062] In step 1003, cardiac cycles are extracted from the intravascular image data. The intravascular image data can be intravascular image data (intravascular 2D image data) such as OCT, IVUS, etc., and the intravascular data can be pressure data (intravascular 1D image data, also referred to herein as non-image data) from FFR, iFR, etc.

[0063] Extraction of cardiac cycle from intravascular 2D image data : If the intravascular imaging performed in step 304 of FIG. 3 is represented by 2D images such as OCT, IVUS, etc., the cardiac cycle can be extracted from this intravascular image data. During intravascular imaging such as OCT, IVUS, etc., cardiac motion causes an excessive motion between the imaging probe and the lumen boundary during acquisition. This motion of the imaging probe relative to the lumen boundary can be used to extract the cardiac cycle from the intravascular images. This can be done, for example, by calculating the distance (1303, FIG. 13) from the imaging probe (1305) to the lumen boundary (1306) in each cross-sectional image frame (1301, 1302). Combining the calculated distances in each cross-sectional image frame into a graph 1304 provides information about the cardiac cycle. From this distance graph, cardiac cycle information can be calculated similarly to the technique described to determine this information in order to guide the catheter. Instead of using cross-sectional images, the same technique can also be applied to longitudinal images created from all the cross-sections of the pullback. An example of a longitudinal IVUS image is given by 1401 in FIG. 14. In the longitudinal image 1401, the sawtooth shape of the lumen-vessel wall boundary, indicated by 1402, reveals the transient motion between the imaging probe and the lumen wall. By evaluating the lumen boundary position (sawtooth) in the longitudinal image, information about the cardiac motion is provided. Other solutions have been revealed in the literature that evaluate signals generated from pixel values ​​in a region of interest in an intravascular image frame. Matsumoto et al., 2003, 14:131–135, 2003, evaluated four signals generated from the ROI, the entire ultrasound image: average intensity (AI), average intensity density (AID), cross-correlation coefficient (CC), and mutual information (MI).

[0064] Extraction of cardiac cycle from intravascular 1D image data: If the intravascular imaging performed in step 304 of FIG. 3 represents a 1D image such as FFR, iFR, CFR, the cardiac cycle can be extracted from this 1D image data, which usually represents a pressure signal. To be able to extract the cardiac cycle, a periodic signal is required, whose periodicity corresponds to the cardiac cycle. If the intravascular imaging modality represents a pressure imaging modality such as FFR, iFR, CFR, the acquired pressure signal can be used. Since the heart pumps blood throughout the body by successive contractions and relaxations of the ventricles, the blood pressure will vary with a period that is linked to the cardiac cycle. During contraction, blood is pushed into the blood vessels, increasing the blood pressure, and during relaxation, the heart chambers fill with blood, decreasing the pressure in the vessels. Therefore, the pressure data is also a periodic signal and correlates with the cardiac cycle. The pressure signal is illustrated in FIG. 15 at 1502, which also shows the correlation with the ECG signal (1501, FIG. 15). Optionally, signal pre-processing can be performed first to remove possible baseline shifts or drifts of the signal. This can be done, for example, by high-pass filtering of the pressure signal. Periodic cardiac cycles can be found by detecting periodic features in the signal, such as peaks in the pressure signal (1503), which can be detected instance by instance as considered in

[12] . The inter-peak intervals (1504, FIG. 15) can then be correlated to the cardiac cycles, for example according to

[13] .

[0065] In step 1004, the endovascular data is synchronized to the fluoroscopic image data. This step includes time alignment between the endovascular image data and the fluoroscopic image sequence to align to the same real-world time. This is done by aligning all cardiac cycles (result of step 1002) extracted from the ECG (result of step 1001) or fluoroscopic image data with the cardiac cycles determined in the endovascular data (result of step 1003). This is further explained by reference to FIG. 16. In FIG. 16, an ECG signal (1601) is used, but if, for example, the ECG is not present or is unreliable, it can be replaced by the catheter tip motion signal, determined in step 1002. In FIG. 16, the cardiac cycles extracted from the endovascular image data (result of step 1003) are indicated by 1602, and in this example, a pressure signal is shown for an FFR, iFR or CFR endovascular system. A particular cardiac phase must be detected in each cardiac cycle (1603, 1604, 1605 and 1606) in the ECG signal (1601) and the intravascular signal (1602), which must be the same phase (e.g., end-systolic phase) in both signals. The particular cardiac phase is then identified as the selected phase.

[0066] Then, one signal is shifted in time until the time difference between the detected phases of the two signals is minimized, as shown at 1607, 1608, 1609 and 1610. This principle works because there is always some variation between the duration of the cardiac cycles, i.e. the duration of each cardiac cycle changes slightly over time. This is illustrated by FIG. 16, where the duration 1612 of a cardiac cycle is smaller than the duration 1613 of the next cardiac cycle. In drawing 1614 the result of this step is visualized, where the 1D pressure signal (1602) is aligned (1611) to the 1D ECG signal (1601).

[0067] To identify the selected phase, the 1D signals (created by steps 1001, 1002 and 1003) are used. These 1D signals all represent repetitive signals with a frequency based on the cardiac cycle and can be processed and analyzed using similar techniques. Each of these 1D signals contains features that can be recognized within the signal that belong to a unique phase of the cardiac cycle. These features are, for example, the R wave apex, T wave apex, QRS complex, QT interval, etc. in the ECG signal as shown in FIG. 11. Coronary artery motion and pressure signals also contain repetitive features such as peaks within each cardiac cycle. For example, coronary artery motion throughout the cardiac cycle has been investigated by

[14] . The teachings of

[14] can be used, for example, to link motion features within each cycle to cardiac phase.

[0068] Several techniques can be used to identify these unique features that correspond to specific phases of the cardiac cycle. An initial template signal can be used to correlate with the 1D signal. The template signal, whose cardiac phase is known, can be matched to the 1D signal by shifting and stretching the template signal using template matching or correlation algorithms that use a metric such as cross-correlation as the minimization metric. These template signals can be generated, for example, from patient populations whose cardiac phases have been annotated by clinical experts or from studies such as

[14] .

[0069] Another technique is to find, for example, a peak in the signal (e.g., the apex of the R wave, see FIG. 10 or the maximum pressure in a pressure signal). If the relationship of the peak with respect to the cardiac phase is known, for example from the literature or clinical expertise, the cardiac cycle can be assigned to a cardiac phase.

[0070] When the fluoroscopic image data and the intravascular data are consistent, it is possible to select a fluoroscopic image frame from a particular cardiac phase and its corresponding intravascular data.

[0071] The real-world time relationships between acquired image data are known Returning to step 102 of Figure 2, synchronization of endovascular and fluoroscopic image data, when real-world time relationships between the read-out image data are available, is further described with reference to the flow chart of Figure 18.

[0072] If the start of acquisition of the images / signals is known relative to one another, the signals / images can be synchronized based on the time difference between the start of acquisition and the sample and / or frame rate. Figure 18 illustrates this workflow. The synchronization of the different signals 1801, 1802 and 1803 is performed in step 1804.

[0073] As mentioned before, Fig. 9 shows the case where the real-world time relationship between the loaded image data is available. 901 represents a sequence of fluoroscopic images. A dot (904, representing the first frame of the fluoroscopic sequence) represents each successive frame. 902 represents the ECG signal and 903 represents the intravascular image data (in this example, the intravascular image data represents a pressure signal). At real-world time (906) the acquisition sequence of the fluoroscopic images starts, and at real-world time 907 the acquisition of the ECG signal starts. 908 represents the real-world start time of the pressure data. Since the respective real-world start times are known, a (single) calibration needs to be performed (909, 910) to extract possible signal delays.

[0074] Determining the particular cardiac phase (selected phase) within each cycle is also part of step 102. Since all signals are synchronized, it is only necessary to determine the cardiac phase (selected phase) for every cardiac cycle for one signal (901, 902 or 903). This can be done, for example, by the methods described above in the description of step 1004.

[0075] Step 103: Co-register the x-ray image data and the intravascular data In step 103, the intravascular image data and the x-ray image data are actually Mutual alignment Figure 19 shows the X-ray image data and intravascular image data. Mutual alignment2 presents a flowchart of steps for performing an accumulation image (1904) of a single phase of the cardiac cycle from a sequence of x-ray fluoroscopic images representing a radiopaque wire section (see 2006 in FIG. 20) during pullback of the endovascular wire. This accumulation image is generated along with x-ray angiography image frames representing the same cardiac cycle used to generate the accumulation image. Alignment Since the time alignment between the intravascular image data and the x-ray image data is known (as a result of step 102), the time alignment between the intravascular image data and the x-ray image data is Mutual alignment Next, all steps from the flow chart of FIG. 19 will be explained in more detail.

[0076] In step 1901 of FIG. 19, spatial information of the radiopaque wire section is obtained from the sequence of fluoroscopic images (step 304 of FIG. 3) covering the pullback of an intravascular device (e.g., pressure wire) by detecting the radiopaque wire section of the intravascular wire visible in the fluoroscopic image frames (704 of FIG. 7). The detection of the radiopaque wire section can be performed on all frames in the sequence of fluoroscopic images, but should be performed at least on all frames having the same phase in the cardiac cycle, as determined by step 102 of FIG. 1. For example, by using frames of a specific phase in the cardiac cycle, such as the apex of the R wave, cardiac motion can be minimized to make the calculation of the cumulative image in step 1904 appropriate. The detection of the radiopaque wire section can be performed fully automatically, i.e., without user interaction, or semi-automatically using various known image segmentation techniques, such as, for example, thresholding, convolution, correlation techniques, or other automatic segmentation techniques. An example of (semi-)automatic segmentation is to manually identify the radiopaque wire section in one frame and provide this location in the remaining frames using image processing techniques as taught for example in Non-Patent Document 15. Machine learning approaches can also be used for the detection of radiopaque wire sections as taught for example in Non-Patent Document 16. Optionally, user input can be used for image segmentation or the radiopaque wire section.

[0077] In step 1902, a correction is optionally performed for respiratory motion and possible patient motion present in the sequence of fluoroscopic images. As previously mentioned, the coronary artery motion consists of cardiac motion due to cardiac contraction, respiratory motion due to patient breathing, and patient motion due to patient movement. The cardiac motion correction is performed by only selecting fluoroscopic image frames of a specific single phase of the cardiac cycle (see also steps 1901 and 1904). In step 1902, the respiratory motion and possible patient motion correction is based on the assumption that the tip of the catheter (FIG. 20, 2007) remains in the same anatomical position in the coronary artery tree, which makes it possible to map the frames to each other using this position as a landmark. The method involves the detection of the catheter tip position in all frames of the sequence of fluoroscopic images, or at least in all fluoroscopic image frames having the same phase in the cardiac cycle, as determined by step 102 of FIG. 1. This catheter tip motion can then be determined and used to eliminate the respiratory motion and / or the patient motion. For example, if the location of the catheter tip in frame A (2003) and frame B (2004) is known, the displacement of the catheter tip can be calculated and the inverse of this displacement can be applied to image B to eliminate respiratory motion between images A and B. Catheter (tip) detection can be performed using techniques such as machine learning, for example as disclosed in US Pat. No. 6,399,436, which discloses a method for catheter tip detection using a deep learning based Bayesian filtering approach. In summary, the method of US Pat. No. 6,399,436 models the likelihood term of a Bayesian filtering approach with a convolutional neural network and comprehensively integrates it with particle filtering to achieve more robust tracking. Another method for catheter (tip) detection based on convolutional neural networks is taught in Non-Patent Document 17. Catheter (tip) segmentation can also be based on more general segmentation techniques, for example as used in pressure wire detection step 1901 (FIG. 19). If necessary, image segmentation can be performed using techniques such as those taught by Non-Patent Document 18. AlignmentTechniques can also be used to eliminate small errors. For example, if the entire pressure wire is visible, the wire (tip) may be aligned with a wire in another image frame. Alignment It is possible to do so.

[0078] Optionally, in step 1903, the start of intravascular wire pullback is detected. This step Mutual alignment However, during the pullback of the intravascular wire, it becomes possible to only take into account the intravascular image data. The initiation of the pullback of the intravascular wire is achieved by examining a movement curve along the frames of the sequence of fluoroscopic images of the detected radiopaque wire section (step 1901), which has already been corrected for respiratory motion as a result of step 1902. The movement curve is created by calculating the absolute distance of the detected radiopaque wire section between the first fluoroscopic image frame (reference frame) and all remaining fluoroscopic image frames in the sequence of fluoroscopic images. The time (or fluoroscopic frame) at which the pullback of the intravascular wire starts is defined as the instant at which the movement curve crosses, for example, a predefined threshold.

[0079] In step 1904, an accumulation image is created that represents the radiopaque wire section during the pullback of the intravascular wire. This accumulation image represents a single phase (a "freeze" point) in the cardiac cycle, thereby eliminating cardiac motion. This is further explained by reference to FIG. 20. In FIG. 20, 2001 represents a sequence of fluoroscopic images, with each dot representing a successive fluoroscopic image frame in the sequence of fluoroscopic images. In step 1901, radiopaque wire sections that are visible in the fluoroscopic image frames are detected in at least all frames having the same phase in the cardiac cycle, as determined by step 102 of FIG. 1. As an example, this phase is selected as the apex (2005) of the R wave of the ECG signal (2002). Drawing 2003 shows a fluoroscopic frame at the time 2009 of the R wave apex, and drawing 2004 shows a fluoroscopic frame (2010) of the same cardiac phase (R wave apex) but in a different cardiac cycle. Detection of the radiopaque wire section (2006) is described by step 1901, and the respiratory motion and possible patient motion, already corrected by the detection of the catheter (tip), are described by step 1902. When this step is performed for all frames with the same phase in the cardiac cycle determined step by step by step 102 of FIG. 1, and in the example of FIG. 20 where the selected phase is the R wave apex, a cumulative image (2008) is created. This image will not include cardiac motion, respiratory and / or patient motion. Drawing 2011 shows another example of such a cumulative image. This image includes only segmented radiopaque wire sections from all frames of the selected phase and may be smoothed into one segmented object. Since synchronization has been established between the fluoroscopic image data and the intravascular image data in step 102 of Figure 1, this cumulative image also directly contains a link between the spatial locations in the cumulative image and the intravascular image data. This is illustrated by 2012 and 2011 in Figure 20, where 2012 represents the intravascular image data (pressure values ​​in this example). In signal 2012, the value of the apex (phase) of the R wave is

number

number

[0080] In step 1905, an accumulation image is created that is superimposed on the x-ray angiogram view. In step 1904, an image is created that provides information about the wire position during the pullback of the intravascular wire. However, since this image is created using a sequence of x-ray fluoroscopic images acquired without the administration of contrast agent (step 304 in FIG. 3), it is not possible to provide the user with visual feedback of any vascular structures. To provide this feedback, in step 1905, the accumulation image is superimposed on an x-ray angiogram frame from the x-ray angiogram image data resulting from step 302 in FIG. 3. The x-ray angiogram frame is selected from the x-ray angiogram sequence as in step 1001 or 1002, as a frame of the same phase as that used during the creation of the accumulation image (step 1901) and during a period after contrast agent has been injected into the coronary vessels of the object. The frame at which the contrast bolus arrives can be detected by analyzing the x-ray angiography image data (302), for example, as taught by et al., “Contrast Imaging with CT Images,” IEEE Transactions on Radiography, vol. 19, no. 19, pp. 111-115.

[0081] After the correct x-ray angiography image frame has been selected, e.g. Alignment Using the technique, an accumulated image is overlaid on a selected x-ray angiography frame. The accumulated image (2008 or 2011) represents the segmented radiopaque wire section for each selected phase in the successive cardiac cycles and represents the pullback of the intravascular wire. Since the intravascular wire is within the vessel, the accumulated image (2008 or 2011) must be within the vessel, so that the accumulated image can be used to image the vessel location. AlignmentIn step 1901, when image frames having the same cardiac phase are selected and used for detecting the wire position, the cardiac motion has already been reduced, so that Alignment Note that this makes it easier to Alignment can be performed using several techniques. By first segmenting the vessels in the x-ray angiography image frame, the labeled features can be aligned. Then, the centerline of the segmentation can be placed on the wire location. Alternatively, Alignment For example, one could create an "artificial" vessel around the wire position trajectory ends, and then use a non-rigid B-spline image that minimizes the mean squared error between the two images, for example. Alignment Techniques (Non-Patent Document 20) or other image (non)rigid Alignment Using the technique, the image of the artificial blood vessel is converted into a contrast image 302 (FIG. 3). Alignment FIG. 21 illustrates a method of creating a cumulative image superimposed on an x-ray angiogram image frame. In FIG. 21, 2101 represents an x-ray angiogram image frame in which a coronary vessel is visible as the endovascular wire is being pulled back. This frame is obtained from the x-ray angiogram image data as a result of step 302 of FIG. 3, and is preferably the phase used to create the detection of the radiopaque wire section (step 1901). Drawing 2102 shows the cumulative image as a result of step 1904, and image 2103 shows the cumulative image superimposed on the x-ray angiogram image frame. The segmented radiopaque wire section (2105) is added to the image as described above for image 2103. Alignment Note that the result of the transformation (2104) is

[0082] Step 104: Displaying Data and Running Advanced Analysis Methods At step 104, the intravascular modality (intra-object) and the x-ray angiography imaging modality (inter-object) are Mutual alignment The results are provided in FIG. Mutual alignment22A shows an example providing the results of a method for detecting radiopaque wire sections, in which the intravascular modality represents the FFR modality (1D intravascular image data). In FIG. 22A, 2201 represents the x-ray angiographic image data resulting from the steps from step 302 of FIG. 3, and preferably a frame representing the phase used to create the detection of the radiopaque wire section (step 1901). Drawing 2202 shows the pressure curve as a result of the pullback described in step 304 of FIG. 3. In 2202, 2203 represents the most distal position of the intravascular wire and may represent the pressure value corresponding to the start of the pullback as a result of step 1903 of FIG. 19. In 2202, 2204 represents the most proximal position of the intravascular wire. Optionally, this position may be automatically defined, similar to the technique described by step 1903, but here this position is defined such that the movement curve does not change length (a predefined threshold) within a predefined time. In drawing 2201, Mutual alignment The measured pressure values ​​may be overlaid at predefined intervals along the axis "position on FFR wire along vessel" (2202). This results in, for example, 0.70 (2201) being overlaid corresponding to position 2203, and 1.0 (2201) being overlaid corresponding to position 2204. Alternatively, the user can place the mouse over the curve (2202) or drag the marker 2204 along the curve. Mutual alignment The value (pressure) being measured is highlighted.

[0083] FIG. 22B is a Mutual alignment 22B shows another example of providing a result, and in this example the intravascular imaging modality represents the FFR, iFR, or modality. The visualization is similar to that provided in FIG. 22A, except that in FIG. 22B an x-ray angiogram frame has also visualized the segmented radiopaque wire sections as a result of step 1905. When the user hovers the mouse over the curve (2202) or drags a marker 2204 along the curve, a result 2205 is obtained in which the marker (2204) is visualized along with the corresponding pressure value (2206).

[0084] FIG. 23 shows that the intravascular imaging modality represents the IVUS modality (2D intravascular image data); Mutual alignment In FIG. 23, 2301 represents the x-ray angiography image data as a result of the steps from step 302 of FIG. 3, and is preferably a frame representing the phase used to create the detection of the radiopaque wire section (step 1901). Drawing 2303 represents a longitudinal view of the IVUS image data. A longitudinal view is an image in which the endovascular frames acquired during pullback are stacked on top of each other along a longitudinal line (e.g., 2307). Drawing 2302 shows a single cross-sectional IVUS frame. Depending on the patient application, Mutual alignment As a result, position 2306 corresponds to the most distal position of the intravascular wire, and position 2304 corresponds to the most proximal position of the intravascular wire. At position 2305, the corresponding IVUS frame is visualized by 2302. When the user hovers the mouse over the longitudinal view (2303) or drags the marker 2305 along the curve, the result 2301 is obtained where the marker (2305) is visualized with the corresponding IVUS frame (2302).

[0085] Alternatively, the intravascular wire path may be automatically segmented as taught, for example, by et al., "Integrated endovascular surgery using a vascular catheter," Journal of Cardiology, vol. 21, no. 1, 2003. This allows: Mutual alignment Also, the geometry information is displayed. Awakening Segmentation may be initiated by a segmented radiopaque wire section resulting from step 1905, or by a user identifying a vessel of interest. This segmentation is preferably performed on the x-ray angiographic image data resulting from steps from step 302 of FIG. 3, and is a frame representing the phase used to create the detection of the radiopaque wire section (step 1901). FIG. 24 shows geometry information of the vessel of interest. Mutual alignment24 provides an example of combining the pressure curve with the pressure marker 2401. The intravascular imaging modality in the example of FIG. 24 represents the FFR modality (1D intravascular image data) indicated at 2402. The segmentation is visualized (2401) in an x-ray angiography frame and the geometry result is extracted, for example, along a diameter along the vessel (2403). The user can place the mouse over the curve (2404) or drag a marker 2404 along the curve, whereby the marker (2404) is visualized (2401) together with the corresponding pressure value. It should be noted that the x-axis of the pressure curve (2402) and the diameter curve (2403) are expressed in millimeters. This can be achieved by performing a calibration on the x-ray angiography image frame. Such calibration can be performed by converting pixels to physical dimensions (e.g., millimeters) and using, for example, geometric information of the x-ray system from which the x-ray angiography images were acquired (distance from the x-ray source to the image intensifier, distance to the object, pixel size at the x-ray source and image intensifier) ​​or other techniques known in the prior art. Alternatively, the calibration (determination of pixel size) is described in the next section, " Determining pixel size and foreshortening It can also be run as described in "

[0086] Determining pixel size and foreshortening The intravascular wire includes a sensor (402, FIG. 4A, FIG. 4B) that performs the imaging, and a radiopaque wire section (403, FIG. 4A, FIG. 4B). The length of the radiopaque wire section is specified by the manufacturer that provides the intravascular wire. The radiopaque wire section is visible in the fluoroscopic image sequence.

[0087] In estimating pixel size and foreshortening factor, it can be assumed that the fluoroscopic image frame in the fluoroscopic sequence at the time (frame) where the radiopaque wire section has the longest distance is the frame where foreshortening does not occur. With this assumption, pixel size can be estimated by using the pixel length of the radiopaque wire section (step 1901) detected in this image frame (no foreshortening) and the identified length of the intravascular device radiopaque wire section.

[0088] Additionally, the amount of foreshortening present in the x-ray image data can be depicted along the pullback path. Since x-ray angiography is a 2D imaging modality that visualizes 3D structures in a 2D image, it can cause "foreshortening". Foreshortening is an event where an object (e.g., blood vessel) appears compressed (e.g., the length of the vessel is visualized in 2D as being shorter) when viewed from a certain viewpoint, and foreshortening means that the image of the vessel and other structures in the x-ray appears too short. This is due to excessive vertical angulation of the x-ray tube while taking the x-ray drawing. An example of this is shown in FIG. 25. In the front view (2501) of FIG. 25, a vessel (2504) without foreshortening is shown. The side view (2502) is an x-ray view acquired based on the x-ray beam 2503. In this view, there is foreshortening that can vary along the length of the vessel and results in the x-ray image 2505.

[0089] During pullback of the FFR wire, successive image frames are available in which the radiopaque wire section is visible, which can be automatically detected (see step 1901). The x-ray image frames in which the sensor portion length is at a maximum value can be used as a reference for the image frames in which the sensor portion length is shorter. For each frame, the sensor portion length is estimated, which allows the change in length (foreshortening) relative to the reference length to be determined. This is shown in FIG. 26, where the y-axis represents the foreshortening value and the x-axis represents the position along the pullback length (in either pixels or millimeters).

[0090] Localized / diffuse FFR measurements presented along the geometry of interest shown in FIG. 22A or FIG. 22B, in combination with quantification of the geometry derived from this geometry, can be used to provide added value to clinical decision making. This data can be used, for example, to determine whether a lesion is focal or diffuse, which informs treatment planning. For example, focal lesions can be more easily treated by dilating the stenosis using an inflatable balloon followed by placement of a stent or scaffold. Diffuse lesions are more difficult to treat and may require different treatment strategies, such as medication, that do not impose unnecessary risks and costs on the patient. This decision can be made based on the shape of the FFR curve along the geometry, combined with geometric information such as diameter / area change and length. For example, a rapid local decrease in FFR or pressure drop indicates that the lesion is more focal.

[0091] System Configuration This application relates to the interoperability of intravascular imaging modalities (within object) and x-ray angiography imaging modalities (between object). Mutual alignmentThe present application relates to methods and systems for performing a vascular imaging procedure. This section provides some examples of system configurations that utilize the operations of the present application. FIG. 27 illustrates different high level system components, including an x-ray imaging system (2701), an intravascular imaging system (2702), and a system, e.g., a computer system (2703), on which the operations of the present application are performed. In general, the system, block 2703, on which the operations of the present application are performed receives and processes data received from the x-ray imaging system (2701) and the intravascular imaging system (2702). The data received from the x-ray imaging system consists of x-ray image data and possibly ECG data, and the data received from the intravascular imaging system consists of intravascular imaging data, such as cross-sectional images in the case of an IVUS, OCT intravascular imaging system, or pressure image data in the case of an FFR, iFR, or CFR intravascular imaging system. Such data transfer from the x-ray imaging system (2701) and the intravascular imaging system (2702) to the system on which the operations of the present application are performed (2703) can be accomplished by file transfer, memory transfer, or by digitizing the imaging data.

[0092] In a first exemplary system, the method receives x-ray image data using a frame grabber or digital visual interface (DVI) and, in some cases, acquires an ECG signal using an analog-to-digital (A / D) converter. Data from an intravascular imaging system is received by either a frame grabber or DVI in the case of intravascular systems such as IVUS or OCT, or by an A / D converter in the case of FFR, iFR, or CFR intravascular imaging systems.

[0093] In a second exemplary system, data transfer is accomplished by file transfer (e.g., DICOM file standard). In this situation, the method can be performed on a stand-alone computer system that is physically separate from the imaging device. Because the data is stored in a file, a direct connection between the x-ray and intravascular imaging systems is not required, and the method can be used offline.

[0094] In any of the system configurations described herein, operation according to the embodiments herein may be integrated into the x-ray imaging system or intravascular imaging system or into a separate computer system.

[0095] If the application is part of an x-ray imaging system, the x-ray image data and possibly ECG data are transferred to the application's operation in a preferred embodiment by memory transfer. Data transfer between the x-ray imaging system and the intravascular imaging system for data required by the application is in a preferred embodiment by a frame grabber or DVI in the case of an intravascular system such as IVUS or OCT, or by an A / D converter in the case of an FFR, iFR, or CFR intravascular imaging system.

[0096] When the operation of the present application is part of an intravascular imaging system, the intravascular data is transferred to the operation of the present application by memory transfer in a preferred embodiment. Data transfer between the intravascular imaging system and the x-ray imaging system for the data required by the present application is transferred by a frame grabber or digital visual interface (DVI) in a preferred embodiment, and optionally an analog-to-digital (A / D) converter is used to acquire the ECG signal.

[0097] The operations can be performed by a processor unit of a stand-alone system, or by a processor unit of a semi-stand-alone system connected to an x-ray system (FIG. 2) and described in detail with reference to FIG. 27, or directly included in an x-ray fluoroscopy system or other imaging system (FIG. 2), for example for acquiring a sequence of 2D angiographic images. FIG. 17 shows an example of a high-level block diagram of an x-ray cine-fluoroscopy system. This block diagram shows an example of how an embodiment can be integrated into such a system.

[0098] Parts of the system (defined by the various functional blocks) may be implemented by dedicated hardware, analog and / or digital circuitry, and / or by one or more processors operating program instructions stored in memory.

[0099] The x-ray system of FIG. 17 includes an x-ray tube 1701 with a high voltage generator 1702 that generates an x-ray beam 1703. The high voltage generator 1702 controls and supplies power to the x-ray tube 1701. The voltage generator 1702 applies a high voltage to a vacuum gap between a cathode and a rotating anode of the x-ray tube 1701. The voltage applied to the x-ray tube 1701 causes an electron transfer from the cathode to the anode of the x-ray tube 1701, which causes an x-ray photon generation effect, also called Bremsstrahlung. The generated photons form an x-ray beam 1703 that is directed to an image detector 1706.

[0100] The x-ray beam 1703 comprises photons with a spectrum of energies ranging up to a maximum determined, inter alia, by the voltage and current applied to the x-ray tube 1701. The x-ray beam 1703 then passes through a patient 1704 lying on an adjustable table 1705. The x-ray photons of the x-ray beam 1703 penetrate the patient's tissue with various angulations. Different structures of the patient 1704 absorb the radiation differently, which modulates the beam intensity. The modulated x-ray beam 1703 emerging from the patient 1704 is detected by an image detector 1706 located on the opposite side of the x-ray tube. This image detector 1706 can be either an indirect or direct detection system.

[0101] For an indirect detection system, the image detector 1706 comprises a vacuum tube (x-ray image intensifier) ​​that converts the x-ray emission beam 1703' into an amplified visible light image, which is then transmitted to a visible light image receptor, such as a digital video camera, for display and recording of the image, thereby producing a digital image signal.

[0102] In the case of a direct detection system, the image detector 1706 comprises a flat panel detector. The flat panel detector converts the x-ray emission beam 1703' directly into a digital image signal. The digital image signal from the image detector 1706 is passed to a digital image processing unit 1707. The digital image processing unit 1707 converts the digital image signal from 1706 into a corrected (e.g., inverted and / or contrast enhanced) x-ray image in, for example, a DICOM standard image file format. This corrected x-ray image can then be stored on a hard drive 1708.

[0103] Additionally, the x-ray system of Fig. 17 includes a C-arm 1709. The C-arm holds the x-ray tube 1701 and the image detector 1706 such that the patient 1704 and adjustable table 1705 are between the x-ray tube 1701 and the image detector 1706. The C-arm can be moved (rotated and angulated) to a desired position to acquire a particular projection in a controlled manner using the C-arm control 1710. The C-arm control allows manual or automatic input to adjust the C-arm to a desired position for x-ray recording at a particular projection.

[0104] The x-ray system of Figure 17 can be either a single-plane or a bi-plane imaging system. In a bi-plane imaging system, there are multiple C-arms 1709, each consisting of an x-ray tube 1701, an image detector 1706, and a C-arm control 1710.

[0105] Additionally, the adjustable table 1705 can be moved using the table control 1711. The adjustable table 1705 can be moved along the x, y, and z axes and can be tilted about a particular point.

[0106] Furthermore, in this x-ray system there is a measurement unit 1713. This measurement unit contains information about the patient, for example ECG, aortic pressure, biomarkers and / or information about height, length etc.

[0107] There is also a generic unit 1712 in this x-ray system. This generic unit 1712 can be used to interact with a C-arm control 1710, a table control 1711, a digital image processing unit 1707 and a measurement unit 1713.

[0108] One embodiment is performed by the x-ray system of Figure 17 as follows: A clinician or other user acquires a sequence of at least two x-ray angiogram images of a patient 1704 by moving the C-arm 1709 to a desired position relative to the patient 1704 using the C-arm control 1710. The patient 1704 lies on an adjustable table 1705 that the user has already moved to a particular position using the table control 1711.

[0109] A sequence of x-ray images is then generated as described above using high voltage generator 1702, x-ray tube 1701, image detector 1706, and digital image processing unit 1707. These images are then stored on hard drive 1708. Using these sequences of x-ray images, general purpose processing unit 1712 performs the methods described in this application, as described in FIG. 1, using for example measurement unit 1713, digital image processing unit 1707, C-arm control unit 1710, and table control unit 1711.

[0110] Several embodiments of the method and apparatus for restoring missing information regarding the order and flow direction of velocity components have been described and illustrated herein. Although specific embodiments of the invention have been described, it is not intended to limit the invention thereto, as the invention is as broad as the art will permit, and the specification is intended to be read in the same manner. For example, data processing operations can be performed off-line on images stored in digital storage, such as PACS, commonly used in medical imaging technology. Thus, those skilled in the art will appreciate that still other modifications can be made to the provided invention without departing from its spirit and scope as set forth in the claims.

[0111] The embodiments described herein may include various data stores and other memory and storage media, as discussed above. These may be located in a variety of locations, such as storage media local (and / or resident) to one or more computers, or storage media remote from any or all computers on a network. In a particular set of embodiments, information may reside in a storage-area network ("SAN"), which is well known to those skilled in the art. Similarly, files necessary to perform functions specific to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computing devices, each device may include hardware elements that may be electrically coupled via a bus. The elements may include, for example, at least one central processing unit ("CPU" or "processor"), and at least one input device (e.g., mouse, keyboard, controller, touch screen, or keypad), and at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random access memory ("RAM") or read-only memory ("ROM") and removable media devices, memory cards, flash cards, etc.

[0112] Such devices may also include computer-readable storage media readers, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.) and the working memory described above. The computer-readable storage media readers may be configured to connect to or receive computer-readable storage media representing remote, local, fixed and / or removable storage devices, as well as storage media for storing, transmitting and retrieving computer-readable information on a temporary and / or more permanent basis. The systems and various devices also include several software applications, modules, services or other elements that are typically located in at least one working memory device, including operating systems and application programs such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations from those described above. For example, customized hardware may be used and / or certain elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices such as network input / output devices may also be used.

[0113] Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing description on a computer readable medium. Storage media and computer readable media for containing code or portions of code may include any suitable media known or used in the art. The suitable media include, but are not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storing and / or transmitting information, such as computer readable instructions, data structures, program modules, or other data. The storage media and communication media may include RAM, ROM, Electrically Erasable Programmable Read Only Memory ("EEPROM"), Flash memory or other memory technology, Compact Disc Read Only Memory ("CD-ROM"), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other media that may be used to store the necessary information and that may be accessed by a system device. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods for implementing the various embodiments.

[0114] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0115] There are other variations that are within the spirit of the disclosure. Thus, the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above. However, it is not intended that the invention be limited to the particular forms disclosed. On the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.

[0116] The use of the terms "a" and "an" and "the" and similar articles in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected," when referring to a physical connection without modification, is construed as including, in whole or in part, in other words, attached or joined together, even if there is something intervening. Recitation of ranges of values ​​herein is intended to serve merely as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein. Use of the term "set" (e.g., "set of items") or "subset" should be construed as a non-empty collection containing one or more members, unless otherwise stated or contradicted by context.

[0117] The operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more present applications) collectively executed on one or more processors. The code may be stored on a computer-readable storage medium, for example in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory.

[0118] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the invention. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventors anticipate that such variations will be utilized by those skilled in the art as appropriate, and the inventors intend that the embodiments of the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, the scope of the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is within the scope of the present disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0119] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

Claims

1. 1. A method for correlating endoluminal data and x-ray image data, the method comprising the steps of: a) providing x-ray image data of a tubular organ; b) providing endoluminal data of said tubular organ; c) synchronizing the endoluminal data and the x-ray image data based on available input signals and / or cardiac cycles extracted from the x-ray image data and / or the endoluminal data in order to match the endoluminal data and the x-ray image data at a specific cardiac phase; d) co-registering the x-ray angiography image data and the endoluminal data by creating an accumulated image at said particular cardiac phase, said accumulated image reconstructing a path followed by an endoluminal wire used to acquire said endoluminal data and linking said endoluminal data to a spatial location of said path followed by said endoluminal wire in said x-ray image, said accumulated image being created from a sequence of x-ray fluoroscopic images representing a radiopaque wire section during pullback of said endoluminal wire at a particular phase of said cardiac cycle and being registered with an x-ray angiography image frame representing the same cardiac cycle as used to generate said accumulated image; A method for providing the above.

2. 2. The method of claim 1, wherein creating an accumulated image comprises extracting image frames from the x-ray image data that include the endoluminal wire at the same said particular cardiac phase, and forming an image of the endoluminal wire by accumulating the extracted image frames.

3. The method of claim 2 , wherein the endoluminal wire is identified or segmented prior to forming the cumulative image.

4. The method of any one of claims 1 to 3, further comprising determining the start of acquisition of endovascular data by analyzing a movement curve created by calculating the absolute distance of the detected endoluminal wire between the reference frame and the remaining image frames of the sequence.

5. 5. The method of claim 1, wherein the x-ray image data comprises fluoroscopic image data obtained without a contrast agent and angiographic image data obtained with a contrast agent, and the cumulative image is created from the fluoroscopic image data.

6. The method of claim 5 further comprising overlaying the cumulative image onto angiographic image data.

7. The method according to claim 5 or 6, wherein the endoluminal data comprises pressure values, which are superimposed on the angiographic image data at corresponding spatial locations.

8. 8. The method of claim 5, wherein the endoluminal data comprises an endoluminal image, the method comprising receiving an input for selecting a spatial location within the angiographic image data and presenting an endoluminal image already acquired from that spatial location.

9. A method according to any one of claims 1 to 8, further comprising correcting for respiration and / or patient movement by tracking the tip of the catheter within the x-ray fluoroscopic image data.

10. 10. The method of claim 1, wherein the x-ray image data and the endoluminal data have a known temporal relationship, and wherein synchronizing the endoluminal data and the x-ray image data comprises: extracting an ECG signal, the x-ray image data or the endoluminal data; selecting a specific cardiac phase; calculating a delay between the specific cardiac phase, the x-ray image data and the endoluminal data; and aligning the x-ray image data and the endoluminal data by correcting the calculated delay.

11. 10. The method of claim 1, wherein the x-ray image data and the endoluminal data have an unknown temporal relationship, further comprising: extracting a cardiac cycle from the x-ray image data or from an ECG signal correlated with the x-ray image data and from the endoluminal data; identifying a specific cardiac phase of the extracted cardiac cycle; calculating a delay between the specific cardiac phase, the x-ray image data and the endoluminal data; and aligning the x-ray image data to the endoluminal data by correcting the calculated delay.

12. The method according to claim 10 or 11, wherein the reliability of the ECG signal is assessed by matching the ECG signal with a template and determining a correlation function between the ECG signal and the template.

13. 13. The method of any one of claims 1 to 12, wherein the x-ray image data comprises image frames including a fixation element coupled to the motion of the heart at the tip of the catheter, determining the motion of the fixation element by image segmentation techniques or machine learning, and extracting the cardiac cycle from said motion.

14. The method of claim 13 , wherein determining the motion of the fixation element comprises enhancing structures associated with cardiac motion in the image frames by using a principal component analysis technique.

15. 15. The method of claim 1, wherein the endoluminal data comprises a two-dimensional image, the method comprising determining the motion of the luminal boundary relative to a reference point or line, which is the position of an imaging probe used to acquire the image, and extracting the cardiac cycle from the motion.

16. 16. The method of claim 15, wherein determining the motion of the lumen boundary comprises calculating distances from the reference points or lines to the lumen boundary in each portion or part of the image, and combining the calculated distances to reconstruct a cardiac cycle.

17. 17. The method of claim 1, wherein the endoluminal data comprises non-image data of pressure data, and the method comprises determining cyclical features in the non-image data, and extracting cardiac cycles from the cyclical features.

18. a) an angiographic imaging device (2701) for acquiring x-ray images of a tubular organ; b) an intravascular device (2702) for acquiring endoluminal data of said tubular organ; c) a combination device (2703) comprising a data processing module (2704) and an interface element for receiving x-ray images from said angiographic imaging device (2701) and for receiving intraluminal data from an intravascular device (2702); A system for registering endoluminal data and x-ray images of a tubular organ, comprising: The data processing module includes: synchronizing the x-ray image data to the endoluminal data based on available input signals and / or cardiac cycles extracted from the x-ray image data and / or the endoluminal data in order to match the x-ray image data to the endoluminal data at a specific cardiac phase; and co-registering the x-ray angiography image data and the endoluminal data by creating an accumulated image at the particular cardiac phase, which accumulated image reconstructs a path followed by an endoluminal wire used to acquire the endoluminal data and links the endoluminal data to a spatial location of the path followed by the endoluminal wire in the x-ray image, the accumulated image being created from a sequence of x-ray fluoroscopic images representing a radiopaque wire section during pullback of the endoluminal wire at a particular phase of the cardiac cycle and being registered with an x-ray angiography image frame representing the same cardiac cycle as that used to generate the accumulated image.

23. A system configured to:

19. The system of claim 18, wherein the combination device (2703) is part of the angiographic imaging device (2701), the data processing module (2704) of the combination device (2703) is one of the processors of the angiographic imaging device (2701) having access to the x-ray images, and the angiographic imaging device (2701) has an input for receiving intraluminal data from the intravascular device (2702).

20. 20. The system of claim 18, wherein the combined device (2703) is part of the intravascular device (2702), the data processing module (2704) of the combined device (2703) is one of the processors of the intravascular device (2702) that has access to the intraluminal data, and the intravascular device (2702) has an input for receiving x-ray images from the angiography imaging device (2701).

21. The system of any one of claims 18 to 20, further comprising an ECG module interfacing with an x-ray device (2701), the intravascular device (2702) or the combined device (2703) to enable a data processing module (2704) to refine an ECG signal.

22. The system of any one of claims 18 to 21, wherein the x-ray images, intravascular data, and ECG signals are input to the combination device (2703) at least in part via a frame grabber, an A / D converter, a DICOM transfer protocol, or a combination thereof.

23. The system of any one of claims 18 to 22, wherein the intravascular device comprises an intraluminal wire having a sensor at its tip for collecting intraluminal data via a console while the intraluminal wire is pulled through a tubular organ, the sensor being suitable for collecting intraluminal data being pressure or images selected from the group consisting of FFR, iFR, CFR, IVUS, OCT data.

Images