Intravascular Ultrasound Imaging

Abstract

An image processing apparatus 10 is disclosed having a processor device 16 configured to receive image data corresponding to a region of interest 1 of a patient's cardiovascular system, the image data comprising a time sequence 15 of intravascular ultrasound images 150 acquired during different phases of at least one cardiac cycle of the patient, the intravascular ultrasound images imaging overlapping volumes of the patient's cardiovascular system. The processor device further includes a spatial reordering process for the time sequence of intravascular ultrasound images, the spatial reordering process evaluating the image data to determine at least one spatial criterion 6 (V) associated with the time sequence of intravascular ultrasound images. ref ), estimating a distance to the at least one spatial reference for each of the intravascular ultrasound images in the time sequence, and rearranging the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on the estimated distances, and generating an output having the spatial sequence of intravascular ultrasound images. A method and a computer program product for configuring an image processing apparatus accordingly are also disclosed.

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus having a processor device configured to receive image data comprising a time sequence of intravascular ultrasound (IVUS) images corresponding to different phases of at least one cardiac cycle, at least some of the intravascular ultrasound images corresponding to different intravascular locations.

[0002] The present invention further relates to a computer-implemented method for processing such image data, including a time sequence of intravascular ultrasound images.

[0003] The invention further relates to a computer program product for carrying out such a method on a processor unit of an image processing device. [Background technology]

[0004] IVUS imaging is a valuable technique for obtaining internal images of a patient's cardiovascular system, such as the patient's arteries or heart. IVUS images can aid in assessing the condition of the cardiovascular system, for example, in detecting and quantifying the size of stenosis, i.e., plaque buildup, and in aiding in the positioning of medical implants, such as stents.

[0005] To obtain IVUS images, a minimally invasive medical device, such as a catheter or guidewire with an ultrasound probe attached to its tip, is inserted into the patient's cardiovascular system, typically an artery, and then IVUS images are captured at regular intervals while the minimally invasive medical device is slowly pulled back. In this way, the captured cross-sectional IVUS images of the cardiovascular system can help provide valuable insight into the condition over the length of the imaged cardiovascular system.

[0006] Such a time sequence 15 of IVUS images 150 is shown diagrammatically in FIG. 1, where, by way of non-limiting example, a capture time T=T 0 , T=T 1, T=T 2 , and T=T n 1. The IVUS image 150 is shown as a sequence of four IVUS images 150 captured at regular intervals, each denoted by . Each IVUS image 150 captures a cross-sectional view of a portion 1 of the patient's cardiovascular system, such as an artery. An abnormality 3, such as a plaque build-up or stenosis, is seen in the cross-sectional IVUS image of the portion 1 of the cardiovascular system, which allows for the evaluation of characteristics of the abnormality 3. The temporal capture frequency of the IVUS images is typically in the range of a few Hz, e.g., 5-50 Hz, such that during a single cardiac cycle (i.e., a single heartbeat) of the patient's heart, the time sequence 15 includes multiple IVUS images captured during different phases of the cardiac cycle. This is more commonly referred to as a non-gated sequence of IVUS images.

[0007] Theoretically, such a high density of the IVUS image of the portion 1 of the cardiovascular system would allow a 3D reconstruction of the portion 1 with a high longitudinal resolution in addition to the in-plane resolution of the IVUS image, especially when the pullback speed of the minimally invasive medical device is low, e.g., less than 1 mm / s. However, in practice, the minimally invasive device is also displaced relative to the patient's cardiovascular system as a function of the phase of the cardiac cycle. For example, during diastole, this displacement is negligible and the overall motion of the minimally invasive medical device is governed by the pullback speed of the device, resulting in a well-defined motion in a well-defined direction. In contrast, during systole, the displacement of the minimally invasive medical device relative to the patient's cardiovascular system also includes a component caused by the patient's cardiac cycle, resulting in an unknown displacement of the minimally invasive medical device in an unknown direction. At present, the cause of such cardiac cycle-induced displacement of the minimally invasive device, which is added to the displacement caused by the pullback of the device, is not fully understood. Without wishing to be bound by theory, such displacement could be caused by changes in the geometry of the patient's cardiovascular system, the blood flow through the patient's cardiovascular system, or a combination of such potential causes.

[0008] As a result, the order of the IVUS images 150 in the time sequence 15 does not equate to a spatially ordered volume of the IVUS images 150. This is shown diagrammatically in FIG. 2, from which it can be seen that the spatial order of the IVUS images 150 along the length of the portion 1 of the patient's cardiovascular system does not correspond to their temporal order, as indicated by the labels T0-T6, where higher numbers indicate IVUS images acquired at later time points. This means that the temporal order of the ungated IVUS images 150 cannot be easily interpreted, as this may provide a distorted cross-sectional view of the portion 1 of the patient's cardiovascular system. This is shown in the images of FIG. 3, in which the left image shows an anatomically correct longitudinal view of a synthetic stent in a synthetic vessel, while the middle image is constructed from a simulated ungated sequence of IVUS images 150.

[0009] For this reason, evaluation of such IVUS images 150 is typically performed on a gated volume of IVUS images 150, where only IVUS images corresponding to the same phase of the cardiac cycle are grouped together across multiple cardiac cycles. This results in anatomically reliable images such as those shown in the right image of Figure 3, but with a significant loss of longitudinal resolution due to the fact that IVUS images 150 in different phases of the cardiac cycle are not taken into account.

[0010] An example of such gated acquisition of IVUS images is disclosed in "Accurate Visualization and Quantification of the Coronary Vasculature by 3-D / 4-D Fusion from Biplane Angiography and Intravascular Ultrasound" by Andreas Wahle et al., EBios 2000: Biomonitoring and Endoscopy Technologies; pages 144-155. In this paper, the authors disclose a system for the geometrically correct reconstruction of IVUS images by fusion with biplane angiography. The vessel cross-section and tissue properties are obtained from IVUS, while the 3D position is derived by geometric reconstruction from the angiographic projections. ECG-based timing ensures proper matching of the image data with the selected cardiac phase. Fusion is performed separately for each cardiac phase, thus generating 4D data as a set of 3D reconstructions. However, it is not straightforward to create such 4-D data from non-gated IVUS image sequences. Summary of the Invention [Problem to be solved by the invention]

[0011] The present invention aims to provide an image processing device configured to convert a time sequence of intravascular ultrasound (IVUS) images corresponding to different phases of at least one cardiac cycle into a spatially ordered sequence of these images.

[0012] The present invention further seeks to provide a computer-implemented method for converting such a time sequence of intravascular ultrasound images into a spatially ordered sequence of these images.

[0013] It is a further object of the present invention to provide a computer program product for implementing such a method on a processor unit of an image processing apparatus. [Means for solving the problem]

[0014] According to one aspect, an image processing apparatus is provided having a processor device configured to: receive image data corresponding to a region of interest of a patient's cardiovascular system, where the image data includes a time sequence of intravascular ultrasound images acquired at different phases of at least one cardiac cycle of the patient, the intravascular ultrasound images imaging overlapping volumes of the patient's cardiovascular system; perform a spatial reordering process of the time sequence of intravascular ultrasound images by evaluating the image data to select at least one spatial feature associated with the time sequence of intravascular ultrasound images, estimating a distance to the at least one spatial feature for each of the intravascular ultrasound images of the time sequence, and reordering the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on the estimated distances; and generate an output having the spatial sequence of intravascular ultrasound images.

[0015] Such spatially permuted sequences of IVUS images are used to generate high-resolution, low-distortion visualizations of regions of interest of the patient's cardiovascular system, such as the coronary arteries. In particular, at least some embodiments of the present invention exploit the feature that the depth of view of each intravascular ultrasound image is large compared to the displacement (pullback) speed of the invasive medical device, such that there is a large overlap of the imaged volumes of the patient's cardiovascular system between different intravascular ultrasound images of the sequence. This allows, for example, a first-order approximation of each intravascular ultrasound image of the sequence imaging the same volume of the patient's cardiovascular system, which allows the spatial displacement of the intravascular ultrasound images relative to each other to be estimated using a motion estimation algorithm.

[0016] For example, in a first main embodiment of the present invention, the time sequence of intravascular ultrasound images covers a plurality of cardiac cycles, and the processor device is further configured, within the spatial reordering process, to evaluate the time sequence of intravascular ultrasound images by gating the intravascular ultrasound images into a plurality of intravascular ultrasound image groups, each of the intravascular ultrasound images corresponding to approximately the same phase of the cardiac cycle; and to select one of the intravascular ultrasound image groups as the spatial reference; and the operation of estimating a distance from the spatial reference for each of the remaining intravascular ultrasound images of the time sequence includes, for each remaining intravascular ultrasound image group (Vi), an operation of estimating the distance for each intravascular ultrasound image of that remaining intravascular ultrasound image group relative to the intravascular ultrasound image at the spatial reference using a motion estimation algorithm.

[0017] The main advantage of this embodiment is that the non-gated IVUS images are spatially ordered without the need for a secondary imaging technique such as angiography. This not only simplifies the IVUS image processing, but also limits the patient's exposure to potentially harmful radiation, thereby improving the safety of the medical procedure in which the IVUS images are captured. Furthermore, by grouping the IVUS images into gated groups, the insight that for a given phase of multiple cardiac cycles, the cycle-induced displacement of the probe involving the IVUS images is unidirectional and only changes slowly allows a number of constraints to be applied to the motion estimation of the IVUS images in the IVUS image group, thereby reducing the risk of inaccurate estimation of the displacement of the IVUS images in the group relative to a spatial reference. A further advantage is that the spatial ordering of the non-gated sequence of IVUS images does not depend on the detection of an anatomical landmark to be used as a reference for spatial ordering, which typically needs to be provided by a secondary imaging technique such as angiography imaging. Instead, one of the gated groups of IVUS images can be used as a spatial reference. For it has been found that such spatial references provide an accurate spatial reordering of the IVUS images, due to the aforementioned similarity of the various IVUS images in the sequence, thereby making it possible to determine the displacement of such images relative to one another using motion estimation techniques. Indeed, due to the large depth of field of the IVUS images relative to the distance between them in the time sequence, the IVUS images are treated as imaging the same anatomical landscape with a longitudinal shift between them.

[0018] In order to obtain a reliable spatial reference, it is preferable that gating errors in the acquisition of IVUS images do not significantly affect the accuracy of this static reference. Such gating errors may occur when it is difficult to achieve the capture of IVUS images in a particular phase of the cardiac cycle, for example when the heart is beating rapidly. For this reason, the processor unit may be further configured to select as the spatial reference a group of intravascular ultrasound images including intravascular ultrasound images captured during the diastole of the cardiac cycle, since during the diastole the cardiac condition is stable for some time during which the spatial displacement of the IVUS images resulting from the gating errors is negligible.

[0019] As another example, the processor unit is further configured to determine, for each intravascular ultrasound image group with a subsequent temporally adjacent intravascular ultrasound image group, a difference between each intravascular ultrasound image of the intravascular ultrasound image group and an intravascular ultrasound image of a corresponding cardiac cycle of the subsequent temporally adjacent intravascular ultrasound image group; sum the determined differences to obtain a further group difference; and select the intravascular ultrasound image group that shows the smallest further group difference as the spatial reference. This is another suitable approach to find a reliable spatial reference, since finding a gated IVUS image group with a temporally adjacent group of IVUS images with a minimum spatial displacement ensures that a small temporal gating error will only result in a small change (shift) with respect to the captured IVUS image. This is a suitable alternative method to find a spatial reference that is less sensitive to gating errors, such as a spatial reference in diastole, which is advantageous when cardiac phase information is not available to the processor unit.

[0020] In estimating the distance of each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to the intravascular ultrasound image in a spatial reference, the processor unit is configured to determine a group-level distance, i.e., a single distance common to all IVUS images in the remaining group of IVUS images, which is a simple and fast approach that provides good results when a region of interest of a patient's cardiovascular system imposes cardiac cycle-induced displacements of an invasive medical device that are approximately constant across the region of interest.

[0021] However, such an approach may be improved in scenarios where such cardiac cycle-induced displacement of the invasive medical device varies across the region of interest due to changes in the anatomy, such as narrowing or widening of a coronary artery through which the invasive medical device is displaced, or the minimally invasive medical device entering a stiffer or more flexible portion of the anatomy that would change the response of the anatomy to the cardiac cycle. To this end, the motion estimation algorithm may be configured to estimate an optimal distance for each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to the intravascular ultrasound image in a spatial reference. That is, the displacement distance for each IVUS image in the remaining group of IVUS images is individually optimized relative to the IVUS image in a spatial reference to more accurately compensate for such changes in the magnitude of the cardiac cycle-induced displacement of the invasive medical device.

[0022] As will be appreciated from the above, the spatial location of the gated volume of IVUS images, i.e., the IVUS image group, is determined by translating the IVUS image group along a translation coordinate or multiple translation coordinates, as described above, which typically correspond to or approximate the translation coordinates of the invasive medical device used to capture the IVUS images, with translations individually optimized for each IVUS image of the group, and by utilizing similarities between the translated IVUS images in the IVUS image group under investigation and the IVUS images in the IVUS image group in the spatial reference, i.e., the reference IVUS image group.

[0023] The accuracy of this approach is further improved by the fact that the processor unit may be configured to estimate the intravascular distance from the spatial reference for each remaining intravascular ultrasound image group of the time sequence by identifying an intravascular ultrasound image group that is spatially adjacent to the spatial reference; enhancing the spatial reference by merging the spatially adjacent intravascular ultrasound image group with the spatial reference; and spatially relocating the intravascular ultrasound image of the enhanced spatial reference. Such an enhancement process allows improving the resolution of the spatial reference. In principle, such an improvement in resolution can be achieved using any of the spatially relocated IVUS image groups, but since there is an increased risk that, for example, the relocation of a systolic IVUS image group is less accurate in the presence of gating errors within the group, it is preferable that such an enhancement process is performed using spatially adjacent IVUS image groups, for example IVUS image groups corresponding to adjacent cardiac phases.

[0024] In a further embodiment, once the time sequence of IVUS images has been spatially reordered as in the above embodiment, the processor unit may be adapted to repeat the spatial reordering process, which typically involves gating the spatially reordered time sequence of IVUS images into a number of gated IVUS image groups, repeating the selection of a spatial criterion, estimating the displacement of the remaining gated IVUS image groups relative to the spatial criterion, and then spatially reordering (permuting) based on the thus estimated distances to further improve the spatial reordering of the time sequence of IVUS images. The processor unit is configured to repeat this spatial reordering process several times, for example in an iterative manner such that the process is terminated once a fixed number of iterations have been performed or when the spatial reordering no longer changes the determined spatial positions of the IVUS images of the time sequence.

[0025] To further improve the accuracy of the spatial reordering process performed on the time sequence of IVUS images, the processor unit may be further configured to apply a 2D lateral motion compensation algorithm to the time sequence of intravascular ultrasound images prior to gating the intravascular ultrasound images into a plurality of groups of intravascular ultrasound images. Such (2-D) motion compensation compensates for lateral motion in the cardiovascular region of interest present in the time sequence of IVUS images, e.g., lateral motion introduced by the pullback, such that the only remaining dominant motion is the longitudinal motion corresponding to the pullback direction of the invasive medical device.

[0026] According to another main embodiment of the present invention, the image data further comprises: a time series of fluoroscopic (X-ray fluoroscopic) images of the region of interest captured at a viewing angle through which an invasive medical device used to capture the time series of intravascular ultrasound images is visible, wherein each fluoroscopic image is captured at the same time point in the at least one cardiac cycle as a corresponding intracardiac ultrasound image in the time series of intravascular ultrasound images; and a time series of separately recorded angiographic images of the patient region of interest captured at said viewing angle, wherein each angiographic image is captured at substantially the same phase of the at least one cardiac cycle as a corresponding fluoroscopic image in the time series of fluoroscopic images; and the processor device further comprises: temporally registering each intravascular ultrasound image in the time series of intravascular ultrasound images to each fluoroscopic image in the time series of fluoroscopic images; the fluoroscopy images are temporally registered to angiography images of the time sequence of angiography images; the fluoroscopy images are temporally registered to angiography images of the time sequence of angiography images; the invasive medical device is identified in the fluoroscopy images; the invasive medical device is identified in the fluoroscopy images; the invasive medical device is identified in the fluoroscopy images; the invasive medical device is identified in the fluoroscopy images; the invasive medical device is identified in the fluoroscopy images; This ensures that IVUS images can be accurately spatially realigned even in the presence of large changes in the geometry of the region of interest of a patient's cardiovascular system caused by phase changes in the patient's cardiac cycle, because the anatomical landmarks act as stable anchors within the image data and are therefore relied upon regardless of the actual geometry of the region of interest.In other words, even if the 2D projection of the region of interest of the patient's cardiovascular system changes shape dramatically during the patient's cardiac cycle, the relative positions of anatomical landmarks across multiple segments remain intact and are therefore reliable. Thus, such a co-registration approach achieves progressive and monotonic sampling of the region of interest of the patient's cardiovascular system.

[0027] The location of the invasive medical device through the patient's cardiovascular system relative to the registered fluoroscopic and angiographic images is identified in any suitable manner, such as from the registered fluoroscopic and angiographic images, or alternatively, by utilizing its distance from a device, such as an infusion catheter, through which it extends. Other suitable registration techniques for registering a minimally invasive medical device relative to the registered fluoroscopic and angiographic images will be apparent to those skilled in the art.

[0028] In a particular embodiment, the processor unit is further configured to spatially and temporally register each fluoroscopic image of the time sequence of fluoroscopic images to an angiographic image of the time sequence of angiographic images using a cardiac road mapping algorithm, which is a straightforward approach to such a registration process.

[0029] The processor unit may be further configured to extract a path of the invasive medical device through the patient's cardiovascular system from the registered fluoroscopic and angiographic images by extracting a centerline of the path through the patient's cardiovascular system In this manner, the path of the invasive medical device through the patient's cardiovascular system is determined with high accuracy.

[0030] According to another aspect, a computer-implemented method for processing a time sequence of intravascular ultrasound images corresponding to different phases of at least one cardiac cycle is provided, the method comprising the steps of: receiving image data corresponding to a region of interest of a patient's cardiovascular system, the image data comprising the time sequence of intravascular ultrasound images, the intravascular ultrasound images imaging overlapping volumes of the patient's cardiovascular system; performing a spatial relocation process of the time sequence of intravascular ultrasound images by evaluating the image data and selecting at least one spatial reference associated with the time sequence of intravascular ultrasound images; estimating a distance to the at least one spatial reference for each of the intravascular ultrasound images of the time sequence; realigning the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on the estimated distances; and generating an output comprising the spatial sequence of intravascular ultrasound images.

[0031] Such methods readily allow for the reordering of ungated time sequences of IVUS images into a spatial sequence from which an anatomically reliable 3D representation of a region of interest of a patient's cardiovascular system can be generated.

[0032] In a first embodiment, the time sequence of intravascular ultrasound images covers a plurality of cardiac cycles, and the method further comprises the steps of: evaluating the time sequence of intravascular ultrasound images by gating the intravascular ultrasound images into a plurality of intravascular ultrasound image groups, each group consisting of intravascular ultrasound images corresponding to approximately the same phase of the cardiac cycle, within the spatial rearrangement process; selecting one of the intravascular ultrasound image groups as the spatial reference; and estimating a distance from the spatial reference for each remaining intravascular ultrasound image of the time sequence by selecting one of the remaining intravascular ultrasound image groups (V iestimating a distance for each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to the intravascular ultrasound image in the spatial reference using a motion estimation algorithm. Among other advantages previously mentioned, this configuration facilitates spatial reordering of a time sequence of IVUS images without the need to locate anatomical landmarks within a region of interest of the patient's cardiovascular system, as such localization typically requires a secondary sequence of images, e.g., angiographic images, in which such anatomical landmarks can be identified.

[0033] In a second embodiment, the image data further comprises: a time series of fluoroscopic images of the region of interest captured at a viewing angle at which an invasive medical device used to capture the time series of intravascular ultrasound images is visible, wherein each fluoroscopic image is captured at the same time in the at least one cardiac cycle as a corresponding intravascular ultrasound image in the time series of intravascular ultrasound images; and a separately stored time series of angiographic images of the patient region of interest captured under said viewing angle, wherein each angiographic image is captured at substantially the same phase of the at least one cardiac cycle as a corresponding fluoroscopic image in the time series of fluoroscopic images, the method further comprising the steps of: temporally registering each intravascular ultrasound image in the time series of intravascular ultrasound images with a fluoroscopic image in the time series of fluoroscopic images; and registering each fluoroscopic image in the time series of fluoroscopic images with a vascular ultrasound image in the time series of angiographic images. The method includes the steps of: temporally registering an intravascular ultrasound image to a contrast image; identifying an invasive medical device in the fluoroscopic image; extracting a path of the identified invasive medical device through the patient's cardiovascular system from the registered fluoroscopic and angiographic images; identifying a set of anatomical landmarks from angiographic images of the time sequence of angiographic images that are common to the angiographic images; dividing the extracted path of the identified invasive medical device through the patient's cardiovascular system into a plurality of path segments, each path segment defining a spatial reference, each path segment being connected by an adjacent pair of the anatomical landmarks; and at least partially reordering the time sequence of intravascular ultrasound images to the spatial sequence of intravascular ultrasound images based on a distance along a path segment from the time sequence of intravascular ultrasound images to at least one of the anatomical landmarks connecting the path segment. Such a registration method achieves a gradual and monotonous sampling of a region of interest of the patient's cardiovascular system.

[0034] According to yet another aspect, there is provided a computer program product comprising a computer readable storage medium having computer readable program instructions embodied therein which, when executed on a processor unit of an image processing device, cause the processor unit to perform the method of any of the embodiments described herein. Such a computer program product may be used, for example, to upgrade or otherwise modify existing image processing devices, thereby avoiding the need to more expensively replace such existing image processing devices with new devices adapted to perform such methods. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 shows a schematic diagram of a time sequence of IVUS images. [Diagram 2] FIG. 2 illustrates diagrammatically the spatial distribution of a time sequence of IVUS images along a portion of a patient's cardiovascular system. [Diagram 3] FIG. 3 shows images of a synthetic stent in a synthetic vessel captured in different ways. [Figure 4] FIG. 4 illustrates a schematic of an IVUS imaging system in accordance with an exemplary embodiment. [Diagram 5] FIG. 5 illustrates, in block diagram form, a schematic of an IVUS imaging system in accordance with an exemplary embodiment. [Figure 6] FIG. 6 is a flowchart of a method implemented by a processor unit of an image processing apparatus according to an embodiment. [Figure 7] FIG. 7 shows a schematic of an embodiment of this method. [Figure 8] FIG. 8 shows a schematic of an embodiment of this method. [Figure 9] FIG. 9 shows images of a synthetic stent in a synthetic vessel captured and processed with different methods. [Figure 10] FIG. 10 is a flowchart of a method implemented by a processor unit of an image processing device according to another embodiment. [Figure 11] FIG. 11 illustrates a schematic of one embodiment of this method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Embodiments of the invention will now be described in more detail, by way of non-limiting example, with reference to the accompanying drawings, in which:

[0037] It should be understood that the drawings are only schematic and are not drawn to scale, and that the same reference numbers are used throughout the drawings to denote the same or similar parts.

[0038] FIG. 4 shows a schematic diagram of an IVUS imaging system 100, in particular a two-dimensional (2D) IVUS imaging system or a three-dimensional (3D) IVUS imaging system. The IVUS imaging system 100 may be applied for intravascular examination of a region of interest 1 of a patient's cardiovascular system, such as a portion of an artery, such as a coronary artery, a peripheral artery, etc. The IVUS imaging system 100 comprises an invasive medical device 5, such as a catheter or a guidewire, including an ultrasound probe 14 with at least one transducer array having a number of transducer elements for transmitting and / or receiving ultrasound waves. In one example, each of the transducer elements transmits ultrasound waves in the form of at least one transmit impulse of a certain pulse duration, in particular a number of consecutive transmit pulses. The transducer elements may be arranged in a linear array in the case of the 2D IVUS imaging system 100, or in a two-dimensional array, in particular in the case of the 3D IVUS imaging system 100, to provide a multiplanar or three-dimensional image. The ultrasound probe 14 may be attached to a suitable location on the invasive medical device 5, for example, on or proximal to the tip of the invasive medical device 5.

[0039] Additionally, the IVUS imaging system 100 includes an image processing device 10 including a processor unit 16 that controls the provision of 2D or 3D image sequences via the IVUS imaging system 100. As described in more detail below, the processor unit 16 controls the acquisition of data via the transducer array of the ultrasound probe 14 as well as signal and image processing to form 2D or 3D IVUS image sequences from echoes of ultrasound beams received by the transducer array of the ultrasound probe 14. Additionally, the processor unit is responsible for spatially reordering the time sequence 15 of IVUS images 150 into a spatially ordered sequence of IVUS images 150 in accordance with an embodiment of the present invention as described in more detail below.

[0040] The IVUS imaging system 100 further comprises a display device 18 (hereinafter also referred to as display 18) for displaying (spatially reordered) 2D or 3D image sequences to a user, and an input device 20 which may comprise keys or a keyboard 22 and further input devices such as a trackball 24. The input device 20 may be connected to the display 18 or may be directly connected to the processor unit 16.

[0041] The ultrasound system 100 may further include a data storage device 60, e.g., one or more memory devices, hard disks, optical disks, etc., in which the processor unit 16 stores image frames, e.g., a time sequence of IVUS images and / or a spatially permuted sequence of IVUS images, for later evaluation.

[0042] 5 shows a schematic block diagram of an IVUS imaging system 100 including a processor unit 16 of an image processing device 10 configured to process intravascular ultrasound (IVUS) images acquired from an ultrasound probe 14 on an invasive medical device 5 for intravascular examination, such as a catheter or guidewire. The ultrasound probe 14 includes, for example, a CMUT transducer array 26. The transducer array 26 alternatively includes piezoelectric transducer elements formed of a material such as PZT or PVDF.

[0043] The transducer array 26 is coupled to a microbeamformer 28 in the probe, which controls the transmission and reception of signals by the CMUT array cells or piezoelectric elements. The microbeamformer can at least partially beamform signals received by groups or "patches" of transducer elements, as described in U.S. Patents 5,997,479 (Savord et al.), 6,013,032 (Savord et al.), and 6,623,432 (Powers et al.). The microbeamformer 28 can be coupled to a transmit / receive (T / R) switch 30 by a probe cable or probe wire, which may be integral with the invasive medical device, and which switches between transmit and receive to protect the main beamformer 34 from high energy transmit signals when the microbeamformer 28 is not used and the transducer array 26 is operated directly by the main beamformer 34. The transmission of ultrasound beams from the transducer array 26 under the control of the microbeamformer 28 is directed by a transducer controller 32 coupled to the microbeamformer 28 by a T / R switch 30 and a main system beamformer 34 which receives input from a user's operation of the user interface or control panel 22. One of the functions controlled by the transducer controller 32 is the direction in which the beam is steered and focused. The beam is steered straight ahead (at a right angle) from the transducer array 26 or at a different angle for a wider field of view. The transducer controller 32 is coupled to control a DC bias control 58 for the CMUT array. The DC bias control 58 sets the DC bias voltage(s) applied to the CMUT cells.

[0044] On receive, the partially beamformed signals generated by the microbeamformer 28 are coupled to a main beamformer 34, which combines the partially beamformed signals from the individual patches of transducer elements into a fully beamformed signal. For example, the main beamformer 34 may have 128 channels, each of which receives partially beamformed signals from a dozen or even hundreds of patches of CMUT transducer cells or piezoelectric elements. In this way, the signals received by the thousands of transducer elements of the transducer array 26 effectively contribute to a single beamformed signal.

[0045] The beamformed signals are coupled to a signal processor 36, which may form part of the processor unit 16 of the image processing device 10 according to an embodiment of the present invention. The signal processor 36 processes the received echo signals in various ways, such as band-pass filtering, decimation, I and Q component separation, and harmonic signal separation, which act to separate linear and nonlinear signals to allow identification of nonlinear (harmonic of fundamental frequency) echo signals returning from microbubbles contained in the tissue and / or contrast agent previously administered to the body of the patient 12. The signal processor 36 also performs additional signal enhancements such as speckle reduction, signal combining, and noise removal. The band-pass filters in the signal processor 36 are tracking filters whose passbands slide from higher to lower frequency bands as echo signals are received from increasing depths, thereby removing noise at higher frequencies from greater depths where those frequencies lack anatomical information.

[0046] The processed signals are forwarded to a B-mode processor 38 and a Doppler processor 40, which may also form part of the processor unit 16 of the image processing device 10 according to an embodiment of the present invention. The B-mode processor 38 uses detection of the amplitude of the received ultrasound signals to image structures within the body, such as the tissue of organs and blood vessels within the body. B-mode images of the body structures are formed in either a harmonic imaging mode, a fundamental imaging mode, or a combination of both, as described in U.S. Patent No. 6,283,919 (Roundhill et al.) and U.S. Patent No. 6,458,083 (Jago et al.).

[0047] The Doppler processor 40 processes the time-distinct signals from tissue motion and blood flow to detect material motion, such as the flow of blood cells, within the image field. The Doppler processor 40 typically includes a wall filter with parameters that can be set to pass and / or reject echoes returned from selected types of material within the body. For example, the wall filter is set to have a passband characteristic that passes relatively low amplitude signals from materials with higher velocities while rejecting relatively strong signals from materials with lower or zero velocities. This passband characteristic passes signals from flowing blood while rejecting signals from nearby stationary or slow moving objects, such as the walls of the heart. The inverse characteristic passes signals from moving tissues of the heart while rejecting blood flow signals, for what is called tissue Doppler, which images, detects and describes tissue motion. The Doppler processor 40 receives and processes sequences of time-discrete echo signals from different points within the image field, and a sequence of echoes from a particular point is called an ensemble. An ensemble of echoes received in rapid succession over a relatively short period of time is used to estimate the Doppler shift frequency of the flowing blood, whose correspondence to velocity indicates blood flow velocity, while an ensemble of echoes received over a longer period of time is used to estimate the velocity of slower flowing blood or slowly moving tissue.

[0048] The structural and motion signals generated by the B-mode and Doppler processors 38, 40 are then forwarded to the scan converter 44 and multi-planar reformatter 54. The scan converter 44 arranges the received spatially related echo signals into a desired image format. For example, the scan converter 44 arranges the echo signals into a two-dimensional (2D) sector format or a pyramidal three-dimensional (3D) image. The scan converter 44 overlays the B-mode structural image with colors corresponding to the motion of each point in the image field according to their Doppler estimated velocity to generate a color Doppler image depicting the tissue and blood flow motion within the image field.

[0049] In a 3D imaging system, a multi-planar reformatter 54 converts echoes received from points in a common plane within a volumetric region of the body into an ultrasound image of that plane, as described in U.S. Patent No. 6,443,896 (Detmer). A volume renderer 52 converts the echo signals of the 3D data set into a sequence of projected 3D images 56 over time as viewed from a given reference point, as described in U.S. Patent No. 6,530,885 (Entrekin et al.). The 3D image sequence 56 is transferred from the scan converter 44, multi-planar reformatter 54 and volume renderer 52 to the image processor 42 for further enhancement, buffering, temporary storage and display on the display 18.

[0050] In addition to being used for imaging, the blood flow values ​​produced by the Doppler processor 40 and the tissue structure information produced by the B-mode processor 38 may be transferred to a quantification processor 46 forming part of the processor unit 16. This quantification processor 46 produces measurements of different flow conditions, such as the volumetric velocity of blood flow, as well as structural measurements, such as dimensions of anatomical abnormalities within the patient's vasculature, such as stenosis, plaque buildup within the patient's arteries, etc. The quantification processor 46 receives input from the user control panel 22, such as points within the anatomical structures of the image where measurements are to be taken.

[0051] Output data from the quantification processor 46 may be transferred to a graphics processor 50 forming part of the processor unit 16 for reproducing graphics and values ​​of measurements along with the images on the display 18. The graphics processor 50 generates graphic overlays for display along with the ultrasound images. These graphic overlays include standard identifying information such as the patient name, date and time of the image, imaging parameters, etc. For these purposes, the graphics processor 50 receives inputs such as the patient name from the user interface 22. The user interface 22 is coupled to the transmit controller 32 for controlling the generation of ultrasound signals from the transducer array 26 and, therefore, the generation of images generated by the transducer array and the ultrasound system. The user interface 22 may also be coupled to a multiplanar reformatter 54 for selection and control of planes of a multiplanar reformatted (MPR) image that may be used to perform quantified measurements in the image field of the MPR image in the case of a 3D imaging system.

[0052] It should be noted again that the above-mentioned IVUS imaging system 100 is merely described as one possible example for the application of the image processing device 10. It should be noted that the above-mentioned IVUS imaging system 100 does not necessarily have to include all of the components described above. On the other hand, the ultrasound system 100 may include additional components as required. It should be noted that the above-mentioned components do not necessarily have to be realized as hardware, but may also be realized as software. As shown diagrammatically in FIG. 5, the above-mentioned components may be included in a common or single entity, and do not all have to be realized as separate entities.

[0053] As explained in more detail above, in order to reconstruct an undistorted 3D image of the region of interest 1 of the patient's cardiovascular system from the time sequence 15 of IVUS images 150, the time sequence 15 of IVUS images 150 needs to be spatially reordered to compensate for the displacement of the ultrasound probe 14 on the invasive medical device 5 during certain phases of the cardiac cycle. Without such compensation, the IVUS images 150 would appear to jump back and forth within the patient's cardiovascular system when visualizing the region of interest 1 based on the time sequence 15.

[0054] In a first main embodiment, the processor unit 16 of the image processing device 10 is configured to perform a method 200 to achieve such spatial reordering, a flow chart of which is shown in FIG. 6. At this point, it is noted that the processor unit 16 is configured to realize the method 200 and / or 300 entirely in software by executing computer program instructions loaded into the processor unit 16. Alternatively, the processor unit 16 may be configured to implement the method 200 and / or 300 partly in software and partly in hardware, for which the processor unit 16 may comprise one or more processing units, for example application specific integrated circuits, hard-coded to perform certain aspects of the method 200 and / or 300. As a further alternative, the processor unit 16 may be configured to perform the method 200 and / or 300 entirely in hardware.

[0055] In operation 201, the processor unit 16 receives a time sequence 15 of IVUS images 150. The time sequence 15 may be a raw sequence obtained directly from an ultrasound probe 14 of an invasive medical device 5, such as a catheter or guidewire, or alternatively, the processor unit 16 may receive the time sequence 15 from the data storage device 60 as a previously stored sequence of IVUS images 150. The latter scenario facilitates post-procedure evaluation of the IVUS images 150, for example, in scenarios where a physician is not present during the procedure in which the IVUS images 150 are captured.

[0056] The processor unit 16 further performs, in some embodiments, a process 203 in which the IVUS images 150 are pre-processed. Such pre-processing is desirable, for example, when at least some of the images exhibit, in addition to a longitudinal motion along the length of the region of interest 1, a cardiac cycle-induced (2D) lateral motion in the acquisition plane that the processor unit 16 is to compensate for. It would therefore be desirable to compensate for such 2D lateral motion prior to the longitudinal motion compensation of the IVUS images 150 to make the subsequent longitudinal motion compensation as easy as possible. A straightforward approach to such lateral motion compensation is to compensate for the motion between successive pairs of the IVUS images 150. Many suitable 2D motion estimation methods, such as block matching, spline-based motion estimation, parametric motion estimation, diffeomorphic motion estimation, etc., may be used for such lateral motion compensation. Such techniques are known per se to those skilled in the art and therefore will not be described in further detail here, for the sake of brevity only. However, it should be understood from the above that the process 203 is entirely optional and may be omitted from the method 200 without departing from the teachings of the present application.

[0057] In operation 205, the processor unit 16 gates the sequence 15 of IVUS images 150 into multiple groups or volumes of IVUS images 150, where within each group or volume, the IVUS images 150 correspond to the same phase of a different cardiac cycle. A number of techniques can be implemented to determine the relationship between the IVUS images 150 and the phase of the patient's cardiac cycle. For example, if the pullback of the invasive medical device 5 involves the acquisition of the patient's ECG, the cardiac phase can be inferred by interpolation from cyclical features in the ECG, such as the QRS peaks.

[0058] As another example, the phase of the cardiac cycle may be derived from the IVUS image 150 itself, as suggested, for example, by Aura Hernandez et al., "Image-based ECG sampling of IVUS sequences," 2008 IEEE Ultrasonics Symposium, ISBN 978-1-4244-2428-3. Such phase determination methods typically rely on the extraction of some periodically varying indicative information from the IVUS image 150, such as, for example, correlation between successive images whose cyclic variations indicate a particular phase of the cardiac cycle, the amplitude of an estimated lateral motion of the IVUS image 150 (which may also be determined in process 203), the sum of gradients within the IVUS image 150, etc.

[0059] The processor unit 16 then proceeds to operation 207, in which the processor unit selects a reference group or volume from the plurality of groups or volumes of gated IVUS images 150. This reference is used to reposition the remaining groups or volumes of gated IVUS images 150 relative to the selected reference to achieve spatial reordering of the time sequence 15 of IVUS images 150. In a theoretical situation where the cardiac phase associated with the IVUS images 150 is fully defined, the selection of this spatial reference would be arbitrary. However, in practice, there is a gating error in the group of gated IVUS images 150. Typically, this error can be up to 5-10%. For example, if 20-30 IVUS images 150 are captured during a single cardiac cycle, a gating error in such a group or volume may cause the IVUS images 150 in such a group to be off by one or two phases from the correct phase.

[0060] Such gating errors are of little concern when the patient's heart is at rest, such as during diastole, because in such a scenario the gating errors are not associated with large longitudinal displacements of the ultrasound probe 14. However, during systole, the patient's heart is moving so fast that large longitudinal displacements, for example up to 1 mm, may occur between successive IVUS images 150. Furthermore, the magnitude of such cardiac-induced longitudinal motion may vary greatly depending on the geometry of the portion of the cardiovascular system in which the ultrasound probe 14 resides, which may result in unpredictable errors in the spatial reference if the spatial reference corresponds to a phase of the cardiac cycle during systole. It is therefore desirable to select as the spatial reference a group of gated IVUS images 150 such that any gating errors present within the group do not affect the accuracy or reliability of the spatial reference. An acceptable spatial reference is, for example, a group of gated IVUS images 150 corresponding to a cardiac phase associated with diastole when the heart is at rest, as previously described.

[0061] The selection of such spatial reference is based on available ECG data, as described above. Alternatively, in the absence of such ECG data, the selection of spatial reference is obtained by the processor unit 16 determining the difference between each IVUS image 150 of the IVUS image group and the IVUS image 150 of the corresponding cardiac cycle of the subsequent temporally adjacent IVUS image group by subtracting the two images from each other. As can be seen from the above, when the two temporally adjacent IVUS images 150 to be subtracted from each other correspond to the diastole of the cardiac cycle, their difference is small, i.e., the subtraction result is small. On the other hand, in the case of a relatively large displacement of the ultrasound probe 14 during the systole, the difference (subtraction result) between such temporally adjacent IVUS images 150 becomes much larger, even if the effect of possible gating errors is ignored. Once the subtraction operation is completed for all the temporally adjacent IVUS images 150 in the two groups, the processor unit 16 sums up the determined differences to obtain a group difference. In this manner, the group difference is determined for all temporally adjacent IVUS image groups, and then the processor unit 16 selects the intravascular ultrasound image group exhibiting the smallest group difference as the spatial reference to obtain a stable and reliable spatial reference.

[0062] Furthermore, the processor unit 16 continues to execute an operation 209 in which the processor unit 16 estimates the intravascular distance between the spatial reference and each of the remaining groups of gated IVUS images 150 for each of the remaining IVUS images of the time sequence. This can be achieved using readily available motion estimation algorithms, since each IVUS image images a specific volume of the patient's cardiovascular system, in which case the field depth of an IVUS image is typically much larger than the displacement between adjacent IVUS images in the time sequence, or between IVUS image groups. In other words, there is a large overlap of imaged content between the IVUS images of such IVUS image groups, which is exploited by motion estimation techniques to estimate the displacement of one IVUS image relative to another. This is explained in detail with the aid of Figs. 7 and 8, which show two groups or volumes of gated IVUS images 150, i.e., spatial reference volumes V ref and the reference volume V ref A selected volume V that is spatially matched to i The task of the motion estimation algorithm is to estimate the spatial criterion V ref for the selected volume V i Displacement of, i.e., V i and V ref This means finding the distance between the spatial criterion V ref The selected volume for V i By identifying commonalities 155 between the IVUS images 150 in the same image and mapping these commonalities 155 to each other, e.g., spatial reference V ref IVUS images selected for group V i This can be achieved by spatially displacing the selected volume V i and spatial criteria V ref is equal to the distance between

[0063] Figure 7 shows the spatial reference V refand a selected group of gated IVUS images 150 without longitudinal displacement of the selected group, where the bottom series shows the difference (V ref -V i ) (dark grey indicates positive difference values ​​and light grey indicates negative difference values). In FIG. 8, the selected group of gated IVUS images 150 are longitudinally displaced in the direction of the block arrow in FIG. 7. This longitudinal displacement may be performed in stages, and for each stage, a group difference value between the spatial reference and the selected group of gated IVUS images 150 is determined as described above. From these determined differences, the processor unit 16 selects the longitudinal displacement value for which this group difference is smallest as an estimate of the longitudinal displacement of the selected group of gated IVUS images 150 relative to the spatial reference. However, those skilled in the art will appreciate that the motion estimation algorithm is not limited to determining the difference between such IVUS image groups, but may instead utilize any suitable function or measure to determine the quality of the alignment of these IVUS image groups. Furthermore, those skilled in the art will appreciate that the displacement between the groups may be modeled using motion models of different complexity, from a unique global transformation to an affine field, from a field decomposed on a spline to a dense motion field. Finally, the person skilled in the art estimates said motion field in any suitable manner known to him / her, such as by exhaustive, iterative, learning-based methods, etc.

[0064] The processor unit 16 checks in operation 211 whether the longitudinal displacements of all groups of gated IVUS images 150 relative to the spatial reference have been determined. If not, the processor unit returns to operation 209 until all longitudinal displacements have been estimated, after which the processor unit 16 proceeds to operation 213.

[0065] In step 213, the processor unit 16 spatially arranges the groups of gated IVUS images 150 based on their determined longitudinal displacements relative to the spatial reference. Once all the gated IVUS images 150 have been arranged in this way, the values ​​of the pixels at the positions (x,y) are seen as 1-D signals (in Z) sampled at different positions identified in the spatial re-arrangement of the gating groups of IVUS images 150 described above. A 3D volume is now interpolated by the processor unit 16 from the sampled pixel values ​​using known interpolation techniques, which, as previously explained, are known per se and will not be described in further detail for the sake of brevity. In step 215, the processor unit 16 generates an output of the spatially ordered gating groups of IVUS images 150, which takes the form of a volumetric image of the region of interest 1 interpolated from these spatially ordered data groups of IVUS images 150. This output is sent to the display device 18 for displaying the output or to the data storage device 60 for later retrieval. After generation of the output by processor unit 16, method 200 may end.

[0066] 9 shows a pair of images of a synthetic stent in a synthetic artery. The image on the left is the anatomically correct one, while the image on the right is an interpolated image generated from multiple spatially permuted groups of gated IVUS images 150. As can be seen, the interpolated image provides a high quality approximation of the anatomically correct image, thereby demonstrating that the spatial reordering method 200 as described above results in a high quality (volumetric) image that closely approximates the anatomically correct image using non-gated IVUS images 150. This is compared to the center image of FIG. 3, which shows the output without the present invention being used.

[0067] The method 200 implemented by the processor unit 16 of the image processing device 10 can optionally be improved in several ways. i Spatial Criterion Vref The estimation of displacement for the selected IVUS image group V i The present invention is not limited to using the same displacement for each IVUS image in the patient's cardiac cycle. Such a common displacement may provide accurate results when the imaged region of interest of the patient's cardiovascular system exhibits few geometric changes such that the magnitude of cardiac cycle-induced displacements of an invasive medical device, e.g., ultrasound probe 14, throughout the region of interest is approximately constant during a particular phase of the patient's cardiac cycle. However, in many scenarios, the imaged region of interest of the patient's cardiovascular system exhibits more substantial geometric changes that cause the cardiac cycle-induced displacements of an invasive medical device during a particular phase of the cardiac cycle to vary as a function of these geometric changes. Thus, the spatial reference V ref Selected IVUS images for Group V i The displacement of an IVUS image within a patient is typically not constant.

[0068] Therefore, as a first improvement, the selected group V i Rather than applying a single longitudinal displacement to all of the gated IVUS images 150 in a systematic manner, the longitudinal displacement is applied to each of the gated IVUS images in group V. i The individualized displacement of the IVUS images 150 may be optimized separately for each individual IVUS image 150 within the gated IVUS image group V. Such individualized displacement of the IVUS images 150 may be used, for example, to compensate for local changes in cardiac cycle induced displacement of the invasive medical device 5. For example, gradual changes in the geometry or flexibility of the region of interest 1 of the patient's cardiovascular system may cause such local changes. This may be achieved by the processor unit 16 by optimizing the displacement of the gated IVUS image group V. i , to capture such gradual local changes. This can be accomplished in any suitable manner. For example, the selected group V iThe longitudinal displacements applied to the IVUS images 150 may be modeled over a spline basis, and then the spline coefficients are estimated. The nodes of such a spline may have any suitable density, for example, nodes are provided every 2-5 mm. This is, of course, only a non-limiting example of a suitable motion estimation technique, and many more techniques will be readily apparent to those skilled in the art. For example, the Damon algorithm may be used by the processor unit 16, which estimates one displacement value per IVUS image, and subsequently filters these values ​​with a Gaussian filter. In this way, the V i and V ref Instead of estimating one unique displacement value to align the remaining IVUS image groups V, a set of displacement coefficients is obtained, which corresponds to the i Spatial Criterion V ref This increases the flexibility of the mapping operation to ensure a more accurate mapping onto the vertices. Other techniques are readily available to those skilled in the art.

[0069] Selected IVUS images Group V i The determination of the optimal displacement distance for each individual IVUS image in the selected image group V of the gated IVUS images 150 is i In such an embodiment, the node density is further utilized to correct gating errors in a selected IVUS image group V i A spline-based model is used in which the phase of the patient's cardiac cycle associated with

[0070] As explained above, gating errors during diastole are largely insensitive to cardiac cycle-induced displacements, whereas gating errors during systole may lead to large cardiac cycle-induced displacements of the IVUS images 150 that are indicative of such gating errors. Thus, a more constrained (e.g., spline-based) model of longitudinal displacement is used (e.g., having a lower spline node density, such as every 5 mm) for the IVUS image group associated with the (near) diastole of the patient's cardiac cycle, while a more flexible model (spline-based model) of longitudinal displacement is used (e.g., having a higher spline node density, such as every 2 mm) for the IVUS image group associated with the (near) systole of the patient's cardiac cycle. In this way, the selected IVUS image group V i The estimated individual displacements of the IVUS images 150 in are used to compensate for the gating errors. Again, many suitable alternatives to the spline-based approach will be readily apparent to those skilled in the art.

[0071] In yet another refinement, the processor unit 16 identifies a spatially adjacent group of the gated IVUS images 150 relative to a spatial reference Vref and compares the spatially adjacent group of the gated IVUS images 150 relative to the spatial reference Vref. ref For example, by merging with the selected group V i 150 individual IVUS images in the spatial reference V ref IVUS images 150 are merged with corresponding images in the selected IVUS image group V i The spatial positions of the IVUS images 150 are i The spatial reference V from which the images are merged ref The spatial reference V is calculated by averaging with the spatial location of the IVUS image 150 in ref In this way, more information is available for longitudinal motion compensation of the group selected after the gated IVUS images 150 to the spatial reference V refwill be captured. Although it is theoretically possible to select any of the groups of gated IVUS images 150 for such merging processing, by selecting a group that is spatially close to the spatial reference V ref two volumes or groups related to the diastolic phase that are more easily distorted are obtained, and as much information as possible is captured by the spatial reference V in order to facilitate the spatial rearrangement of the more difficult "systolic" groups of the gated IVUS images 150. ref This enhancement process is repeated multiple times to merge M of the N groups of gated IVUS images 150 (M < N) into the spatial reference V. The spatial reference thus obtained is used for the spatial rearrangement of the remaining N - M groups of the gated IVUS images 150 as described in more detail above. ref

[0072] Another improvement implemented is that after the spatial reordering of the time series 15 of the IVUS images in process 213, the processor device 16 returns to process 205, where the spatially reordered time series 15 of the IVUS images in process 213 is gated again, and the aforementioned processes 207, 209, and 211 are repeated, in particular when the IVUS images are spatially rearranged in an individually optimized manner within group V i it is possible to further improve the spatial rearrangement of the IVUS images 150, and in this way, further reduce the remaining spatial positioning errors of the IVUS images 150 from the previous spatial rearrangement cycle. The processor device 16 repeats this iterative process over a certain number of iterations, or alternatively, the processor device 16 compares the spatially sorted sequence of the actual iterative IVUS images 150 with the spatially sorted sequence of the IVUS images 150 of the previous iteration, and ends the iterative process after the difference between these spatially sorted sequences of the IVUS images 150 falls below a defined threshold.

[0073] The above-mentioned first main embodiment of the image processing device 10, i.e. the implementation of the method 200 by its processor unit 16, has the advantage that no additional visualization techniques are required in addition to IVUS imaging in order to generate a spatially reliable 3D image of a region of interest 1 of the patient's cardiovascular system. Moreover, no landmark detection in such region of interest 1 is required in order to accurately position the IVUS images 150 with respect to such detected landmarks. However, in the second main embodiment of the invention, such additional visualization techniques and landmark detection are exploited by the processor unit 16 in order to obtain a spatially reordered sequence of IVUS images 150.

[0074] At this point, it is noted that it is known from the article "Image-based Co-Registration of Angiography and Intravascular Ultrasound Images" by Wang et al., IEEE transactions on medical imaging, Vol. 32 (12), 2013, pp. 2238-2249, to provide a system for detecting the coronary artery branch under investigation in a reference angiography image. During the pullback of the IVUS transducer, the system acquires both ECG-triggered fluoroscopy images and IVUS images, and automatically tracks the position of the medical device in fluoroscopy. More specifically, both ECG-triggered fluoroscopy images and gated IVUS images are acquired in a time-synchronous manner, and the acquisition of the fluoroscopy images is triggered in the end-diastolic phase by the ECG. However, extending such a gated approach to non-gated IVUS image acquisition is not trivial. For example, due to cardiac phase-induced changes in the geometry of a patient's cardiovascular system, the extension of such mapping to non-gated IVUS imaging results in a rather discontinuous (wobbling or jumping) visualization of the region of interest 1 when the individual non-gated IVUS images 150 mapped to the angiogram image are shown in rapid succession. Furthermore, due to the changing geometry of the region of interest 1 during the cardiac cycle, spatial reordering of the time sequence 15 of non-gated IVUS images 150 is not straightforward because the positions of the IVUS image slices 150 relative to one another are unclear.

[0075] At least some of these problems are addressed by a second main embodiment of the present invention, as explained in more detail with the aid of Fig. 10, which shows a flow chart of a method 300 implemented by the processor device 16 for this purpose. According to the method 300, the processor device 16 receives in operation 301 a time-synchronized sequence of fluoroscopic (X-ray) images and a time sequence 15 of non-gated IVUS images 150, the latter generated using an ultrasound probe 14 on an invasive medical device 5, as previously described. For each IVUS image 150 acquired at a particular phase of the cardiac cycle, a fluoroscopic image is acquired at the same time, i.e. at the same phase of the same cardiac cycle.

[0076] In process 303, the processor unit 16 receives a series of angiographic images, typically acquired at the same viewing angle as the fluoroscopic images, e.g., the same C-arm view, so that the fluoroscopic images can be easily mapped onto the angiographic images. As is known, the angiographic images provide excellent detail of the topology of the patient's cardiovascular system, while the fluoroscopic images provide excellent detail of the invasive medical device 5 within the patient's cardiovascular system. To further facilitate such mapping, the angiographic images are typically captured at approximately the same phase of the cardiac cycle as the fluoroscopic images, e.g., in a time-synchronous manner. The angiographic images are acquired at any suitable time, such as before acquisition of the fluoroscopic and IVUS images 150. In an exemplary embodiment, the processor unit 16 receives a series of previously recorded angiographic images from the data storage device 60. For completeness, process 303 is shown following process 301 in FIG. 10, but it will be understood from the above that these processes may be performed by the processor unit 16 in reverse order or simultaneously.

[0077] The processor unit 16 then proceeds to process 305 where each of the IVUS images 150 is matched in time to its fluoroscopic image counterpart, i.e., a fluoroscopic image captured at the same time or in the same phase of the cardiac cycle. The processor unit 16 then spatially and temporally matches each of the fluoroscopic images to its angiographic image counterpart, for example, using a cardiac roadmapping algorithm.

[0078] As explained above, the invasive medical device 5 including the ultrasound probe 14 is clearly visible in the fluoroscopic images, thus, after image-based registration, the processing unit 16 detects the ultrasound probe 14 in each of the fluoroscopic images in process 307. As is known, for this purpose, the processor unit 16 uses hand-crafted detection filters that emphasize features defining the appearance characteristics of the ultrasound probe 14 and / or part of the invasive medical device 5. Alternatively, the processor unit 16 uses learning algorithms to create such detection filters. Such techniques are known per se and therefore, for the sake of brevity only, will not be described in further detail. As a result, by exploiting the mapping of the fluoroscopic images to the angiographic images, the ultrasound probe 14 is accurately positioned on the corresponding injection angiogram, i.e. an angiogram captured with an injected contrast agent and at the same cardiac phase of the fluoroscopic image in which the ultrasound probe 14 is visible. Thus, in this manner, the processor unit obtains an association of each IVUS image 150 with a particular angiographic image, as well as the spatial location of the ultrasound probe 14 within the angiographic image, expressed for example in (x,y) pixel coordinates.

[0079] However, it should be understood that embodiments of the present invention are not limited to such cardiac roadmapping algorithms. Instead, it may only be required to map the fluoroscopic images with the angiographic images in time. If only the position of the minimally invasive medical device 5 is important, perfect spatial registration of the fluoroscopic images with the angiographic images may not be required. For example, an injection catheter used to introduce the minimally invasive medical device 5 into the patient's cardiovascular system is detected in both the fluoroscopic and angiographic images, and then the distance of the minimally invasive medical device 5 to a reference point on the injection catheter is extracted from the fluoroscopic image, and then the distance thus extracted is reported in the angiographic image. In this way, the registration of the minimally invasive medical device 5 with the fluoroscopic and angiographic images is achieved without (perfect) spatial registration of the fluoroscopic and angiographic images.

[0080] The processor device 16 is further configured to execute step 309 of the method 300, in which the processor device 16 extracts a pullback centerline of the invasive medical device 5 through the region of interest 1 of the patient's cardiovascular system. For example, the processor device 16 may be configured to evaluate the successively registered ultrasound probes 14 placed in the angiogram. The probe positions may be depicted as dotted lines on the associated angiogram along the pullback centerline. This representation may be incomplete (i.e., some dots may be missing) and may further include potential outliers. The processor device 16 constructs a centerline from this representation by joining more distal dots to more proximal dots, with the constraint that the centerline must pass through as many dots (probe points) as possible, while at the same time staying within the introduced region of interest 1, e.g. a portion of the coronary artery. This corresponds in effect to the generation of an energy map reflecting these constraints. The centerline is propagated along the region of interest 1 using a suitable algorithm, such as a suitably adapted fast marching algorithm.

[0081] The processor device 16 is further configured to execute operation 311 of the method 300, in which the processor device 16 detects a number of anatomical landmarks (i.e., vascular landmarks) in each angiographic image using a suitably configured detector. As explained above, the construction of such a detector is performed manually or (semi-)automatically by the processor device 16 using deep learning algorithms. Such techniques are known per se and therefore, for the sake of brevity, will not be described in further detail. Such anatomical landmarks (landmarks) related to the patient's vessels may include, for example, vessel bifurcations, stenoses, stents, etc. Once anatomical landmarks are detected, the processor device 16 further uses a distance filter to ensure that only anatomical landmarks within a defined distance of the previously determined pullback centerline are retained, since only anatomical landmarks in the immediate vicinity of the pullback centerline are likely to be relevant.

[0082] The processor unit 16 then evaluates the detected anatomical landmarks and associates the same landmarks across multiple angiographic images, for example by providing the same labels to the same landmarks in the different angiographic images. The determination of whether the landmarks in the different angiographic images are indeed the same is based on the type of the landmark, the local appearance, the spatial location, etc. The processor unit 16 further discards the landmarks that are not present in each of the angiographic images, so that a set of landmarks remains common to all the angiographic images. These common landmarks provide spatial waypoints through the region of interest 1 of the patient's cardiovascular system that can be used to spatially reorder the time sequence 15 of IVUS images 150.

[0083] For this purpose, the processor device 16 is further configured to execute operation 313 of the method 300, in which the processor device 16 divides the pullback centerline into a plurality of segments, each centerline segment being connected by a pair of spatially separated anatomical landmarks. An exemplary centerline segment 6 is shown diagrammatically in FIG. 11, where the centerline segment 6 is connected by spatially separated anatomical landmarks, here a first branch 2 and a second branch 2'. In this way, the pullback centerline is divided into a plurality of centerline segments that are coherent from one cardiac phase to another, such that these centerline segments can be used as spatial references for spatial reordering of the temporally ordered IVUS images 150 by association with the position of the ultrasound probe 14 in the registered fluoroscopic images of the IVUS images 150.

[0084] To this end, the processor unit 16 is further configured to execute operation 315 of the method 300, in which the processor unit 16 determines a spatial position of the ultrasound probe 14 on the pullback centerline segment 6. The ultrasound probe 14 may appear at different locations on the pullback centerline segment 6, for example, positions 7 and 9 in FIG. 11, in which case the spatial order of the IVUS images 150 associated with these different probe positions is derived by determining the probe position on the pullback centerline segment 6 with respect to its proximity to one of the anatomical landmarks that form the boundaries of the pullback centerline segment 6. For example, in FIG. 11, position 7 may be determined to be located 15% of the total length of the pullback centerline segment 6 from the branch 2, while position 9 may be determined to be located 65% of the total length of the pullback centerline segment 6 from the branch 2. In this way, position 7 is determined to follow position 9 (assuming a pullback in the direction from branch 2' to branch 2'), even though a direct comparison between the registered fluoroscopic and angiographic images was not possible due to the significant changes in the geometry of the region of interest 1 across different phases of the cardiac cycle, as explained earlier.

[0085] Having determined the position of the ultrasound probe 14 along each pullback centerline segment, the processor unit 16 executes step 317 of the method 300, in which the processor unit 16 spatially reorders the time sequence 15 of non-gated IVUS images 150 based on the determined position of the ultrasound probe 14 along each pullback centerline segment. Since each probe position is derived from a particular fluoroscopic image registered with the particular IVUS image 150 such that the IVUS image 150 can be associated with the particular probe position accordingly. Finally, in step 319, the processor unit 16 generates an output of the spatially ordered gating group of IVUS images 150, which takes the form of a volumetric image of the region of interest 1 interpolated from these spatially ordered IVUS images 150. This output is sent to the display device 18 for displaying the output or sent to the data storage device 60 for later retrieval. After the processor unit 16 generates this output, the method 300 may end.

[0086] It should be noted that the image processing device 10 according to the embodiments of the present invention may be provided as a stand-alone device. Alternatively, the image processing device 10 may be provided together with an invasive medical device 5, such as a catheter or guidewire carrying an ultrasound probe 14, to form an IVUS imaging system. The above-described embodiments of the methods 200 and 300 may be realized by computer readable program instructions embodied on a computer readable storage medium that, when executed on a processor unit of the ultrasound system 100, causes the processor unit to perform the methods 200 and / or 300. Any suitable computer readable storage medium may be used for this purpose, such as an optically readable medium, such as a CD, DVD or Blu-ray disc, a magnetically readable medium, such as a hard disk, or an electronic data storage device, such as a memory stick.

[0087] The computer-readable storage medium may be a medium accessible via a network, such as the Internet, such that the computer-readable program instructions may be accessed via the network. For example, the computer-readable storage medium may be a network-attached storage device, a storage area network, a cloud storage device, etc. The computer-readable storage medium may be an Internet-accessible service from which the computer-readable program instructions are obtained. In an embodiment, the ultrasound imaging device 10 is configured to create a new computer-readable storage medium by retrieving computer-readable program instructions from such a computer-readable storage medium and storing the retrieved computer-readable program instructions in a data storage device 60, such as a memory device forming part of the data storage device arrangement, which is accessible to the processor unit 16 such that the processor unit 16 may retrieve the computer-readable program instructions from the data storage device 60 for execution.

[0088] It should be noted that the above-described embodiments illustrate rather than limit the invention, and that those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims. In the claims, reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprises" does not exclude the presence of elements or steps other than those listed in a claim, and the word "a" preceding an element does not exclude the presence of a plurality of such elements. Can be implemented by means of hardware comprising several distinct elements. In a device claim enumerating several methods, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. An image processing apparatus having a processor device, the processor device comprising: receiving image data corresponding to a region of interest of a patient's cardiovascular system, where the image data comprises a time sequence of intravascular ultrasound images acquired at different phases of at least one cardiac cycle of the patient, the intravascular ultrasound images imaging overlapping volumes of the patient's cardiovascular system; performing a spatial reordering process on the time sequence of intravascular ultrasound images by evaluating the image data to select at least one spatial fiducial associated with the time sequence of intravascular ultrasound images, estimating a distance to the at least one spatial fiducial for each of the intravascular ultrasound images in the time sequence, and reordering the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on the estimated distances; generating an output having a spatial sequence of the intravascular ultrasound images; The image data is a time series of fluoroscopic images of the region of interest captured at a viewing angle within which an invasive medical device used to capture the time series of intravascular ultrasound images is visible, each fluoroscopic image being captured at the same time in the at least one cardiac cycle as a corresponding intravascular ultrasound image in the time series of intravascular ultrasound images; a separately recorded time sequence of angiographic images of the region of interest captured at the viewing angle, each angiographic image captured at substantially the same phase of the at least one cardiac cycle as a corresponding fluoroscopic image in the time sequence of fluoroscopic images; and The processor unit includes: temporally registering each intravascular ultrasound image of the time sequence of intravascular ultrasound images to a fluoroscopic image of the time sequence of fluoroscopic images; temporally registering each fluoroscopic image of the time sequence of fluoroscopic images to an angiographic image of the time sequence of angiographic images; identifying an invasive medical device within the patient's cardiovascular system in the fluoroscopic image; extracting from the registered fluoroscopic and angiographic images a path of the identified invasive medical device through the patient's cardiovascular system; identifying a set of anatomical landmarks common to the angiographic images from the angiographic images of the time sequence of angiographic images; dividing the extracted path of the identified invasive medical device through the patient's cardiovascular system into a plurality of path segments, each path segment defining a spatial reference, where each path segment is connected by adjacent pairs of the anatomical landmarks; reordering the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images at least in part based on distances along path segments of the intravascular ultrasound images from the time sequence to at least one of the anatomical landmarks connecting the path segments. Image processing device.

2. the time sequence of intravascular ultrasound images covering a plurality of cardiac cycles; The processor unit further includes, in the spatial rearrangement process, evaluating a time sequence of the intravascular ultrasound images by gating the intravascular ultrasound images into a plurality of groups of intravascular ultrasound images, each group being an intravascular ultrasound image corresponding to substantially the same phase of the cardiac cycle; selecting one of the group of intravascular ultrasound images as the spatial reference; and estimating a distance from the spatial reference for each of the remaining intravascular ultrasound images of the time sequence includes, for each of the remaining group of intravascular ultrasound images, estimating the distance using a motion estimation algorithm for each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to an intravascular ultrasound image at the spatial reference. The image processing device according to claim 1 .

3. The image processing apparatus of claim 2 , wherein the motion estimation algorithm estimates an optimal distance for each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to the intravascular ultrasound image in the spatial reference.

4. 4. The image processing apparatus of claim 2 or 3, wherein the processor unit selects as the spatial reference a group of intravascular ultrasound images having intravascular ultrasound images captured during a diastolic phase of the cardiac cycle.

5. The processor unit determines an intravascular distance from the spatial reference for each of the remaining group of intravascular ultrasound images: identifying a group of intravascular ultrasound images that are spatially adjacent with respect to said spatial criteria; enhancing the spatial reference by merging the group of spatially adjacent intravascular ultrasound images with the spatial reference; Spatially relocating the enhanced spatially referenced intravascular ultrasound image. The image processing device according to claim 2 , wherein the estimation is performed by:

6. 6. The image processing apparatus of claim 2, wherein the processor unit repeats the spatial reordering process for a spatial sequence of the intravascular ultrasound images.

7. 7. The image processing apparatus of claim 2, wherein the processor unit is further configured to apply a motion compensation algorithm to the time sequence of intravascular ultrasound images before gating the intravascular ultrasound images into a plurality of groups of intravascular ultrasound images.

8. 8. The image processing apparatus of claim 1, wherein the processor unit is adapted to spatially and temporally register each fluoroscopic image of the time sequence of fluoroscopic images to an angiographic image of the time sequence of angiographic images using a cardiac roadmapping algorithm.

9. 9. The image processing device of claim 1, wherein the processor extracts a path of the invasive medical device through the patient's cardiovascular system by extracting a centerline of the path through the patient's cardiovascular system from the registered fluoroscopic and angiographic images.

10. An image processing apparatus having a processor device, receiving image data corresponding to a region of interest of a patient's cardiovascular system, the image data comprising a time sequence of intravascular ultrasound images acquired during different phases of at least one cardiac cycle of the patient, the intravascular ultrasound images imaging overlapping volumes of the patient's cardiovascular system; performing a spatial reordering process on the time sequence of intravascular ultrasound images by evaluating the image data to select at least one spatial fiducial associated with the time sequence of intravascular ultrasound images, estimating a distance to the at least one spatial fiducial for each intravascular ultrasound image in the time sequence, and reordering the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on the estimated distances; and generating an output having a spatial sequence of the intravascular ultrasound images, the computer program comprising: The image data is a time series of fluoroscopic images of the region of interest captured at a viewing angle within which an invasive medical device used to capture the time series of intravascular ultrasound images is visible, each fluoroscopic image being captured at the same time in the at least one cardiac cycle as a corresponding intravascular ultrasound image in the time series of intravascular ultrasound images; a separately recorded time sequence of angiographic images of the region of interest captured at the viewing angle, each angiographic image captured at substantially the same phase of the at least one cardiac cycle as a corresponding fluoroscopic image in the time sequence of fluoroscopic images; and The computer program further comprises: temporally registering each intravascular ultrasound image of the time sequence of intravascular ultrasound images to a fluoroscopic image of the time sequence of fluoroscopic images; temporally registering each fluoroscopic image of the time sequence of fluoroscopic images to an angiographic image of the time sequence of angiographic images; identifying an invasive medical device within the patient's cardiovascular system in the fluoroscopic image; extracting a path of the identified invasive medical device through the patient's cardiovascular system relative to the registered fluoroscopic and angiographic images; identifying a set of anatomical landmarks common to the angiographic images from the angiographic images of the time sequence of angiographic images; dividing the extracted path of the identified invasive medical device through the patient's cardiovascular system into a plurality of path segments, each path segment defining a spatial reference, each path segment being connected by an adjacent pair of the anatomical landmarks; and at least partially reordering the time sequence of intravascular ultrasound images into a spatial sequence of intravascular ultrasound images based on a distance along a path segment of the intravascular ultrasound images from the time sequence to at least one anatomical landmark connecting the path segment. Computer program.

11. The time sequence of intravascular ultrasound images covers a plurality of cardiac cycles, and the computer program further comprises, during the spatial reordering process: evaluating the time sequence of intravascular ultrasound images by gating the intravascular ultrasound images into a plurality of groups of intravascular ultrasound images, each group being an intravascular ultrasound image corresponding to substantially the same phase of the cardiac cycle; selecting one of the group of intravascular ultrasound images as the spatial reference; and estimating a distance from the spatial reference for each of the remaining intravascular ultrasound images of the time sequence includes, for each of the remaining group of intravascular ultrasound images, estimating the distance using a motion estimation algorithm for each intravascular ultrasound image of the remaining group of intravascular ultrasound images relative to an intravascular ultrasound image at the spatial reference.

11. A computer program product as claimed in claim 10.

12. 12. The computer program product of claim 11, further comprising causing the processor unit to repeat the spatial reordering process for a spatial sequence of the intravascular ultrasound images.

13. A computer readable storage medium comprising a computer program according to any one of claims 10 to 12.

Images