Method for visualizing dynamic anatomical structures

Abstract

The invention relates to a method, a computer program and a user interface for visualizing a dynamic anatomical structure (1). The method comprises: (a) providing a sequence of three-dimensional medical images (M1, M2, M3,..., M Z ) of a dynamic anatomical structure (1) across a time period (T), (b) providing a dynamic model (14), in particular a surface of the anatomical structure, (c) determining a volume of interest (40) containing an anatomical feature of interest (3) within each of the three-dimensional images, wherein the volume of interest (40) follows the position and / or shape of the anatomical feature of interest (3) over the time period, and wherein the volume of interest (40) is smaller than the complete field of view of the three-dimensional medical images (M1, M2, M3,..., M Z ), and (d) providing a three-dimensional visualization environment (50, 70), wherein a visualization (45) corresponding to a particular time point comprises (i) a volume rendering of the volume of interest (40) of the three-dimensional image, and (ii) a visualization of the dynamic model (14) in the same coordinate system. Preferably, the three-dimensional visualization environment (50, 70) allows displaying the dynamic model (14) and the volume rendered volume of interest (40) for each three-dimensional image over the time period in a cine mode.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method of visualizing dynamic anatomical structures, to a related computer program and user interface. BACKGROUND

[0002] Medical imaging techniques provide three-dimensional (3D) image data of a human or animal body. However, the images are usually viewed on a two-dimensional (2D) screen. Thus, there is a general risk of misinterpreting the relative spatial relationships between anatomical structures represented on medical 3D data when viewing and analyzing the 3D dataset in orthogonal planes on a 2D screen. A common source of error is the selection of an incorrectly placed or distorted measurement plane for analyzing and measuring anatomical structures.

[0003] 3D volume rendering is a collection of techniques for displaying 2D projections of a 3D image dataset. However, although 3D volume rendering helps to form an idea 3D model of the anatomical structures, it strongly depends on the image quality as well as on the chosen settings (thresholds, smoothing, etc.). It often contains image errors and thus carries the risk that the user no longer critically questions or verifies the volume rendered representation of the anatomical structures. However, when used for complex pathologies (e.g. tumors, ruptured tendons or calcified biological prostheses), the approach of segmenting a 3D volume with one or more thresholds is still of interest.

[0004] On the other hand, simplified models of anatomical structures enjoy high reputation in clinical practice as they are able to compensate for poor image quality and are easier and faster to interpret. An example of a dynamic surface model is the TOMTEC 4D or 4D MV, a feature of the commercially available software of TOMTEC Imaging Systems GmbH, which is a dynamic surface model of the right ventricular cavity. However, they are not suitable for all pathologies as complex and delicate structures are not properly captured and displayed.

[0005] Thus, current approaches of segmenting and displaying medical image data of complex anatomical structures, e.g. cardiac 3D volume data, either simplify the anatomical properties too much by using surface or shape models or use threshold-based segmentation which can lead to false conclusions. Important geometric correlations and information, which are necessary e.g. when planning cardiac interventions, can thus be lost or ignored.

[0006] Some studies involve 3D printing of medical models based on CT or 3D echo data. However, the 3D printing process is time-consuming and requires expertise, is expensive, cannot be included in the daily clinical routine and cannot represent the dynamic movement of the anatomical structures, especially for 3D echo.

[0007] Virtual reality (VR) has already been used to visualize medical image data. For example, Thomas S. In "A new virtual reality approach for planning of cardiac interventions" (Artificial Intelligence in Medicine 22 (2001), 193-2014), researchers disclosed virtual reality visualization of cardiac magnetic resonance (MR) data. The disclosed method includes optimized respiratory-compensated 3D MR scanning, segmentation, model generation, and interactive virtual reality visualization. Segmentation results in a set of contours in parallel planes, defining different anatomical parts of the cardiovascular system. A 3D model is created by connecting these contours. The model is viewed using shutter glasses combined with a "holobench" (a setup of two monitors at a 90° angle to each other).

[0008] Cristian A. Linte et al. disclosed a virtual reality environment for visualizing real-time intraoperative echocardiography in “Virtual reality-enhanced ultrasound guidance: A novel technique for intracardiac interventions” (Computer Aided Surgery, March 2008, 13(2), 82-94). It can present surgeons with preoperative images and magnetically track the position of surgical instruments to enhance intraoperative ultrasound imaging. A feature-based registration technique is used to register preoperative images with intraoperative TEE (transesophageal echocardiography) data.

[0009] US2014 / 052001A1 discloses the use of both B-mode data representing tissue and flow data representing regurgitation jets to automatically detect the mitral valve using machine learning classifiers. A series of classifiers can be used, such as using one classifier to determine the location and orientation of the valve region, another classifier to determine the regurgitation orifice, and a third classifier to locate the mitral valve anatomy. One or more features can be computed based on the orientation of the valve region for some of the classifiers. Once the mitral valve is detected, it is plotted as a mesh model and overlaid on a rendering of the heart.

[0010] US2016220311 Al discloses a processor that acquires image data from a medical imaging system. The processor generates a first model from the image data. The processor generates a computational model that includes cardiac electrophysiology and cardiac mechanics estimated from the first model. The processor performs a test on the computational model to determine an outcome of a therapy. The processor superimposes the outcome on an interventional image. Using interventional imaging, the first cardiac model can be updated / superimposed during therapy to visualize its impact on the patient's heart.

[0011] US2008194957 Al discloses a method for producing a three-dimensional image of an object. The method includes providing a model of the object, acoustically penetrating a region of the object from a source transducer external to the object, receiving return echoes from the object at a receiving transducer external to the object, processing the return echoes and generating a hybrid image of the object, the hybrid image including the region of the object responsive to the model of the object and the region of the object responsive to the return echoes.

[0012] Lin W. et al. in "Visualisation of cardiac dynamics using physics-based deformable model" (visual communications and image processing; 20-1-2004-201-2004; San Jose, vol. 3976, 15 February 2000 (2000-02-15), pages 210-217, XP008019251, DOI: 10.1117 / 12.383043 ISBN: 978-1-62841-730-2) disclose the creation of an image showing the dynamic motion of the left ventricle. Thus, a first method is provided according to which a surface model is created. Furthermore, a second method is provided according to which a physics-based deformable model is created in which a surface mesh is deformed by following the trajectory of connected corresponding vertices in the surface model at successive time points.

[0013] It is therefore an object of the present invention to provide a method of visualizing a dynamic anatomical structure which minimizes the risk of misinterpretation of image data and in particular of misinterpretation or incorrect measurement of the relative spatial relationship between anatomical features. SUMMARY

[0014] To better address one or more of the concerns addressed above, according to a first aspect of the present invention, a method of visualizing a dynamic anatomical structure is provided. The method comprises the following steps:

[0015] a) providing a sequence of three-dimensional medical images across a time period, each three-dimensional medical image in the sequence showing a dynamic anatomy at a time point during the time period;

[0016] b) providing a dynamic model of at least part of the anatomy, wherein the dynamic model has been derived from and registered to the sequence of three-dimensional medical images;

[0017] c) determining a volume of interest within each of the three-dimensional images containing an anatomical feature of interest, wherein the volume of interest follows the position and / or shape of the anatomical feature of interest over the time period, and wherein the volume of interest is smaller than the complete field of view of the three-dimensional medical images; and

[0018] d) providing a three-dimensional visualization environment for displaying the dynamic anatomy over the time period, wherein the visualization corresponding to a particular time point within the time period comprises:

[0019] (i) a volume rendering of the volume of interest of the three-dimensional image corresponding to the particular time point; and

[0020] (ii) a visualization of the dynamic model at the particular time point and in the same coordinate system as the volume rendering of the volume of interest.

[0021] Preferably, the three-dimensional visualization environment allows displaying the dynamic model and the volume rendered volume of interest for each three-dimensional image across the time period in "movie mode", i.e. in "movie mode" or animation, which means dynamically showing a sequence of visualizations at e.g. a frame rate of 5-100 visualizations per second.

[0022] Thus, when viewing medical 3D volume data, the present invention combines two important approaches: The first approach is a dynamic, computer generated model of at least part of the anatomy. Such models have the advantage that they show a simpler version / abstraction of the anatomy, making it easier to navigate and interpret the anatomy, are not very dependent on image quality, and they do not have "holes", i.e. do not contain artifacts etc.

[0023] The second approach is volume rendering, which has the advantage that it is suitable for more complex / unusual anatomy or highly individual structures, like leaflet cusp valves, stenosis, calcifications, biological prostheses, ruptured tendons etc. that cannot be modeled by software like TOMTEC's 4DMV. In addition, parameters like threshold, opacity, contrast can be adjusted "on the fly", i.e. with immediate effect while watching a dynamic sequence of volume renderings. On the other hand, volume rendering is strongly dependent on image quality, and thus can be too complex for easy interpretation.

[0024] The present invention provides a combination of the advantages of both methods: volume rendering is only used for those parts of the anatomy where it is really useful and necessary. The region where volume rendering is used (= VOI) can be minimized to the actual feature of interest, resulting in a better overview, while the feature of interest (e.g. a valve) is not moved out of focus. This can be done by using the points of the dynamic model to adjust the position of the VOI equally dynamically.

[0025] Thus, the present invention provides the excellent overview and navigation opportunities provided by a dynamic model (e.g. shape / surface model) combined with the adjustable and highly individualized benefits provided by volume rendering, while volume rendering is only used where necessary.

[0026] The dynamic anatomy can be any moving object within a human or animal body. In particular, it is a structure that is affected by a periodic motion, such as a breathing motion or a heartbeat. Thus, the method is particularly suitable for visualizing the anatomy within the torso, e.g. the heart, the lungs, the ribs, the liver, the kidneys, etc. The anatomy can be an organ or a part of an organ of a human or animal body, such as a heart, but can also be a blood vessel or a bone structure. The method of the present invention is particularly suitable for hollow organs and organs comprising a lumen, such as the heart.

[0027] The dynamic anatomy is captured in a sequence of three-dimensional medical images obtained from a human subject across a time period, wherein the images can be pre-operative images, but can also be intra-operative images. The sequence of 3D medical images can be referred to as 4D images. The three-dimensional (3D) images are typically digital images, e.g. in the DICOM standard, i.e. a three-dimensional array of voxels, each containing a gray value. Such 3D medical images are typically obtained from a field of view containing the dynamic anatomy using a medical imaging modality, such as MR, computed tomography (CT), positron emission tomography (PET) or ultrasound (US). When the anatomy is a heart, ultrasound and in particular transesophageal echocardiography (TEE) can be advantageously used. A 3D image from the time sequence of 3D images is also referred to as "frame" in the following. The 3D images can be acquired at a frame rate of e.g. 5-100, preferably 20-60 images per second, in order to allow a smooth representation of the dynamically moving anatomy, where it is displayed in a cine mode. The time period is typically at least one period of the periodic motion, e.g. at least one heartbeat.

[0028] The dynamic model of at least part of the anatomy is in particular a simplification of the anatomy, e.g. a triangulated surface model of a particular interface within the anatomy, e.g. a blood-tissue interface of a blood vessel or a heart chamber. The model can comprise a plurality of points of a line or surface across each frame. It can also be a mathematical model, e.g. a parametric model such as a surface or volume spanned by a spline curve. The model is dynamic, i.e. it follows the movement of the anatomy over a period of time. The purpose of the dynamic model is to visualize at least part of the anatomy, e.g. one or several chambers of a moving heart, without obstructing the view of the user with too much detail. Such a simplified model is thus useful in providing orientation to the user, e.g. when planning an intervention or taking measurements of a particular part of the anatomy.

[0029] The dynamic model can be extracted from the 4D ultrasound image data, e.g. by speckle tracking techniques, to automatically follow three-dimensional endocardial contours throughout the cardiac cycle, thereby generating a dynamic surface model of one or several heart chambers, in particular at least part of the left ventricle and optionally the left atrium. In addition, it can be a shape or surface model that is fitted to the respective 3D medical image.

[0030] The present invention is particularly useful for viewing and analyzing a particular anatomical feature of interest, which is typically a part of the anatomy. Such a feature of interest is contained in a volume of interest (VOI) that is smaller than the complete field of view of the 3D medical image. It can be a part of an organ that constitutes the anatomy, in particular a part with a complex anatomy, e.g. a heart valve. In useful embodiments, the anatomical feature of interest is the mitral valve, the tricuspid valve, the aortic valve or the pulmonary valve. In other embodiments, the anatomical feature of interest can be other important blood vessels, such as coronary blood vessels, or another structure, such as a tumor.

[0031] For a more detailed view of this anatomical feature of interest, the present application provides a volume rendered volume of interest (VOI) which contains the anatomical feature of interest, but preferably not much more than the anatomical feature of interest, i.e. the VOI is as small as possible. In other words, the size and / or shape of the VOI is adapted to match the size and / or shape of the anatomical feature of interest as closely as possible, and this preferably over the entire image sequence. Thus, a VOI containing the anatomical feature of interest is determined within each of the three-dimensional images, wherein the volume of interest follows the position and / or shape of the anatomical feature of interest over the time period. The VOI is a sub-volume from the 3D image and is defined, for example, by a set of enclosing surfaces enclosing the volume belonging to the VOI. The VOI contains voxels having different grey values. When volume rendering the VOI, one or several settings / parameters like a threshold determine which voxels within the VOI are displayed and how they are displayed. The VOI according to the present application can be a cuboid, a cylinder, a sphere or have an irregular shape. The volume of interest is typically defined based on the position and / or shape of the anatomical feature of interest and can have a fixed size over the image sequence, e.g. a box or a cylinder having a predetermined edge length and diameter. The predetermined length and diameter are preferably chosen such that the size of the VOI corresponds to the expected size of the anatomical feature of interest. In some embodiments, the size can change over the time period depending on the size of the anatomical feature of interest on each 3D image. The purpose of the VOI is to define a volume containing the feature of interest on each 3D image. Thus, by volume rendering only the image content within such a VOI, a very good visualization of the feature of interest is obtained, while the view is not obstructed by other volume rendered parts of the anatomy which are less interesting.

[0032] In useful embodiments, the dynamic model covers or borders the anatomical feature of interest, i.e. it is in spatial relation thereto, such as the left ventricle being in spatial relation to the mitral valve. The anatomical feature can also be part of the dynamic model. Thus, each volume rendered VOI is based on the corresponding 3D medical image, and for example, the center of each volume rendered VOI of the sequence is at a fixed relative position with respect to the position of the anatomical feature of the dynamic model at the corresponding time point within the time period.

[0033] The step of determining the VOI can be performed as part of providing the dynamic model. In the case of the mitral valve, for example, a dynamic surface model of the left ventricle will include a set of (landmark) points on the mitral annulus. The mitral annulus constitutes the anatomic junction between the ventricle and the left atrium and serves as the insertion site for the leaflet tissue. Thus, the dynamic model will include a definition of the position of the mitral valve, which can be used to define the VOI for each 3D image in the sequence. In other embodiments, the position of an anatomical feature of interest can be defined by segmentation techniques and feature / speckle tracking techniques, where a certain landmark point is identified on one image and then tracked within the entire image sequence. The position can be defined by a single point in the three-dimensional image space, but can also be defined by a set of points, or the position of a geometric structure, or even the position of a volume.

[0034] To allow the user to view and analyze the anatomical feature of interest, a three- dimensional visualization environment is provided for visualizing the dynamic anatomical structure over a time period. The visualization corresponding to a particular time point within said time period comprises at least two different kinds of visualizations / renderings. These are displayed in the same coordinate system, i.e. they are shown in correct relative spatial position and orientation with respect to each other. In case the two different visualizations spatially overlap, they can overlap or be superimposed with each other, e.g. both are rendered as transparent, or one can be considered more important than the other and can cover the other. The at least two visualization objects are:

[0035] (i) a volume rendering of the VOI, i.e. the image content of the three-dimensional medical image within the VOI is volume rendered and displayed. Since the VOI follows the position and / or shape of the anatomical feature of interest, this volume rendering should be essentially a volume rendering of the anatomical feature of interest (and possibly the immediate surrounding, but no more). Thus, the user is given a detailed and unobstructed view of the particular anatomical feature of interest, without losing the benefits of volume rendering, since volume rendering is only applied where it works. In useful embodiments, the settings chosen for volume rendering, such as threshold, smoothing, etc., are automatically adjustable and / or user adjustable.

[0036] (ii) secondly, a visualization of the dynamic model is rendered, in particular a three- dimensional visualization of the dynamic model at the same time point as the three-dimensional image from which the VOI is volume rendered. Thereby, the user is given further orientation and navigation information. For example, when analyzing the mitral valve in the VOI, he is able to simultaneously track the left ventricular outflow tract (LVOT). This is important when planning an interventional procedure such as a valve replacement (e.g. transcatheter aortic valve implantation (TAVI) or transcatheter aortic valve replacement (TAVR)) or a mitral valve replacement, where the LVOT can not be obstructed.

[0037] This visualization can be displayed in a cinematic mode (also referred to as movie or film mode). In cinematic mode, a sequence of visualizations corresponding to a sequence of 3D medical images is shown to the user, preferably at a suitable frame rate of about 5-100, preferably 20-60 frames per second, in order to give the user a good impression of the moving dynamic model and the moving features of interest. In a useful application, for example, a beating heart and the corresponding valve opening and closing are visualized, wherein the time period is at least one heartbeat.

[0038] Thereby, the movement of the volume rendered VOI (e.g. the part of the heart valve and the LVOT) is locked to the moving structure of the surface model (e.g. the mitral annulus) and thus dynamically moves throughout the cardiac cycle. Thus, the present invention prevents the relevant part of the anatomy (feature of interest) from moving out of the VOI and ensures that the current situation can be analyzed, measured and interpreted faster and with more confidence. The visualization environment of the present invention can be used for viewing and analyzing complex dynamic anatomies, in particular for planning interventions and / or determining the correct size, shape and position of an implant to be implanted in a future intervention.

[0039] The 3D visualization of the dynamic model is typically a rendering of the dynamic shape or surface model, wherein the rendering can be done by techniques available from computer graphics, including shading, ray casting, ambient occlusion, etc.

[0040] Volume rendering can be performed by any volume rendering technique known in the art, for example as described in US 2005 / 0253841 Al, which is incorporated herein by reference. Typically, a camera position and a viewing direction in space need to be defined to perform volume rendering. Also, some techniques define an opacity and a color for each voxel. In some volume rendering techniques, the volume can be viewed by extracting isosurfaces (surfaces of equal gray value) from the volume and rendering them, for example as a polygon mesh, or by directly rendering the volume as a data block. The marching cubes algorithm is a commonly used technique to extract isosurfaces from volume data. Another common technique is volume ray casting. In this technique, a ray is generated for each desired image pixel. Using a simple camera model, the ray starts at the camera's protection center (typically the viewing position or viewpoint) and passes through the image pixel on an imaginary image plane floating between the camera and the volume to be rendered. The ray is then sampled at regular or adapted intervals throughout this volume. At each sampling point, the data is interpolated, a transfer function is applied to form an RGBA sample, the result is added to the ray's cumulative RGBA, and the process is repeated until the ray leaves the volume. The process is repeated for each pixel on the screen to form a complete image.

[0041] According to a particularly useful embodiment, the three-dimensional visualization environment is a virtual reality environment. By "virtual reality" is meant any computer-generated visualization that provides a true three-dimensional experience of the depicted structure. Thus, the virtual reality (VR) environment of the present application provides in particular visual feedback, but can also allow other types of sensory feedback, such as auditory feedback. The VR environment can also be an augmented reality environment, in which the user still sees the real environment, but VR objects (e.g. volume renderings and dynamic models) overlay or are superimposed on real objects, or a mixed reality, in which real-world objects are superimposed on a virtual scene. The visualization of the volume-rendered VOI and the dynamic model together can form a visualization object, preferably a virtual reality object.

[0042] A virtual reality environment is typically realized by presenting stereoscopic images to the user, i.e. each eye sees a different image, so that the brain puts the two different images together into a real three-dimensional scene. Such binocular images can be presented on any VR display, such as a virtual reality headset or a multi-projection environment, in conjunction with shutter glasses, or a screen that intermittently shows the two images.

[0043] In a VR environment, the volume-rendered VOI and the dynamic model can be displayed by stereoscopic rendering: where for two viewing positions with a slight spatial offset (i.e. one viewing position for the left eye and one viewing position for the right eye), the volume-rendered (or otherwise rendered) visualization / image is computed twice. When the two so-computed visualizations are shown to the user on each eye (e.g. on a VR headset), the user gets a real three-dimensional (VR) impression. Thereby, the volume-rendered VOI and the dynamic surface model can be converted to VR.

[0044] In a preferred embodiment, a person using the VR environment of the present application is able to "look around" the artificial world, move around in it, and interact with virtual objects, features or items. This effect is typically created by a VR headset, which comprises a head-mounted display with a small screen in front of each eye, but can also be created by a specially designed room with multiple large screens. In order for the user to move around in the VR environment, position and orientation information has to be transmitted by the headset to the electronic device (e.g. computer) that generates the VR environment, so that the visualization moves in unison with the user's head movements. In order for the user to interact with virtual features in the VR environment, hand movements also have to be tracked, which can be done by a hand-held VR controller. However, this last feature as well as the transmission of position / orientation information are optional in order for the user to walk around in the virtual scene.

[0045] In useful embodiments, the user is able to scale / scale the visualizations / models in the VR environment, adjust visualization parameters and rendering settings, and / or grab displayed objects, in particular visualizations of volume rendered VOIs and / or dynamic models. Since they are locked to each other, they are preferably grabbed and moved together. In addition, in useful embodiments, the VR environment comprises lights that the user can grab and move in the VR environment in order to influence the lighting of volume rendered VOIs and surfaces. In useful embodiments, also the brightness of the scene, in particular of the movable lights, can be adjusted. Another useful embodiment uses a VR environment in which several users can together be in one scene.

[0046] The virtual reality environment offers the advantage that the user can view and analyze the visualized objects with great confidence since he gets a real three-dimensional view of the anatomical structure. In addition, since he is able to walk around and even possibly enter into it, he is able to display the visualized objects (e.g. a visualization of a human heart) with a huge magnification in order to completely fill the space in front of the user. Thus, the user has a particularly good overview and can make measurements with high accuracy. In addition, the handling of user input events is particularly easy and intuitive in a VR environment. In a VR environment using VR controllers, actions such as turning and / or adjusting settings of a volume rendered VOI, which can be quite tricky on a two-dimensional screen, are very intuitive and fast.

[0047] However, the present invention can also advantageously be used with non-VR visualization environments. The following features and embodiments are useful for both VR and non-VR visualization environments without specific reference to VR (virtual reality).

[0048] In useful embodiments, the dynamic anatomical structure is a human or animal heart and the anatomical feature of interest is a part of the heart, such as a heart valve or a coronary vessel. This is particularly useful if the sequence of 3D images has been acquired by ultrasound (such as TEE), since this technique provides dynamic images at high frame rates, is inexpensive compared to other image modalities like CT or MRI, and has minimal risk and radiation exposure for the patient. In yet another useful application, the dynamic model is a dynamic surface model of one or several heart chambers and the anatomical feature is a heart valve. In useful embodiments, the dynamic surface model is a model of the endocardium of (parts of) the left ventricle and the left atrium and the feature of interest is the mitral valve.

[0049] The present invention can be used in particular for planning minimally invasive heart surgery, such as surgery on a heart valve or heart valve replacement. New minimally invasive approaches, like transcatheter valve replacement, can be used for patients that were previously considered inoperable and / or not suitable for open heart surgery. Some transcatheter valve replacements, e.g. TAVR, use a fully collapsible bioprosthetic valve. However, it is crucial for the success of these interventions that the existing pathology / geometry is fully analyzed and understood and that the new valve is carefully selected, sized and positioned to ensure that it works properly and does not obstruct the LVOT or coronary arteries. This is especially true for valve-in-valve (ViV) interventions. In a minimally invasive ViV procedure, therefore, a dysfunctional valve, sometimes a bioprosthetic mitral valve, is replaced by a new valve. The replacement valve is thus positioned inside the old valve, destroying the old valve upon deployment. It is therefore of utmost importance that the valve is positioned correctly and has the correct size. In particular, it is important that the new mitral valve does not obstruct the left ventricular outflow tract (LVOT). For valve-in-valve intervention planning, therefore, the volume rendered VOI contains the mitral valve and preferably also the LVOT. The left ventricle is represented by a dynamic surface model and includes the definition of a set of landmark points on the mitral annulus. These points can be used as a basis for defining the VOI, which accordingly moves with the heartbeat, preventing the mitral valve from moving out of the volume of interest. In a useful embodiment, the user can measure, e.g., the diameter of the mitral valve and accordingly select the best fitting valve from a library.

[0050] Thus, when the anatomical feature is a heart valve, the present invention allows for an excellent visualization of the valve in the surface rendered VOI. The rest of the heart chamber, e.g. the left or right ventricle and / or atrium, is represented by a simplified surface model only, which pumps dynamically with the heartbeat, but does not hinder the view on the anatomical structure of interest, the valve.

[0051] In useful embodiments, the dynamic model is a dynamic surface model, and is derived from a sequence of three-dimensional medical images by image segmentation of the three-dimensional images, or by image segmentation in one three-dimensional image and speckle or feature tracking in subsequent images. Image segmentation is the process of partitioning a digital image into multiple segments or sets of pixels / voxels, and is often used to locate boundaries. Thus, segmentation is the process of assigning a label to every voxel in a 3D image, so that voxels with the same label share certain characteristics. Image segmentation can be performed using thresholding, i.e. all voxels above or below a certain threshold are assigned the same label. Other methods include clustering methods, edge detection or region growing methods. In the case of the heart, segmentation can be used to separate blood-filled cavities (in particular the ventricles and atria) from cardiac tissue such as the heart wall and valves. When the boundary between, for example, a blood-filled cavity and the tissue of the heart wall is detected, a surface model can be established by, for example, selecting a number of voxels on the boundary and connecting them to a wire mesh model or a triangulated surface model. This can be performed for each 3D image in the sequence. According to another useful method, only one image in the sequence is segmented, for example as described above, to extract a static surface model. A dynamic model is then derived from this static model by feature tracking. Thereby, certain characteristic features of the anatomical structure are selected in one 3D image of the sequence. Such characteristic features are then tracked from image to image on the sequence, which is preferably done automatically using a feature tracking algorithm. In the case of ultrasound images, 3D speckle tracking is preferably used. This is a method of feature tracking using the speckle artifact of M-mode ultrasound images for tracking, since speckle is due to interference effects between overlapping echoes, and thus its occurrence is related to the respective anatomical structure. Thus, 3D ultrasound speckle can be tracked from image to image like any anatomical feature, and thus can be used to derive a dynamic model of the anatomical structure. In particularly useful embodiments, speckle tracking is used to derive a dynamic surface model of the left ventricle. Such a surface model includes the mitral annulus, and the mitral annulus in turn is used to define and lock the VOI containing the anatomical feature of interest, i.e. the mitral valve.

[0052] According to useful embodiments, the position and / or orientation of the volume of interest is determined over the time period by identifying corresponding positions and / or orientations of the feature of interest in the dynamic model. Thus, when the feature of interest is part of a structure modelled by a dynamic model, tracking the position and / or orientation of the VOI over the sequence of images is simplified by using the dynamic model, for example by using certain landmark points that are part of the model.

[0053] More generally, according to useful embodiments, a volume of interest can be defined by identifying a set of landmark points of an anatomical feature, by defining an approximating surface across the set of landmark points, and by defining a volume extending above and / or below and / or sideways of the approximating surface. Thus, the volume of interest is locked to the anatomical feature, and if the anatomical feature is part of a structure modeled by a dynamic surface model, the VOI will move with the dynamic model and in particular with the anatomical feature of interest. The set of landmark points can for example be several points on a feature surface or boundary, such as an endocardial or epicardial surface. According to preferred embodiments, the set of landmark points is a point on the annulus of the mitral valve or another heart valve. The set of landmark points can also be referred to as a point cloud. The approximating surface is preferably a surface which can have a predetermined shape, such as a spherical, ellipsoidal, elliptical or oval shape, and which is fitted to the point cloud by a fitting technique in order to obtain a best fit (e.g. least squares) to the point cloud. The surface can advantageously be planar for efficient processing, but it can also be non-planar for best fitting the set of landmark points. The approximating surface thus defined is then used to determine the VOI by defining a volume enclosing the approximating surface and preferably extending a predetermined length above and / or below and / or sideways of the approximating surface. In useful embodiments, the VOI extends up to a predetermined length for a characteristic of the feature of interest (e.g. 1 cm above and below the approximating surface and a safety width to the sides, e.g. 1 cm, for the mitral valve) to ensure that the feature of interest is fully contained in the VOI. By using a predetermined length / width, the processor or computer controlling the visualization can automatically define the VOI from the set of landmark points on the feature of interest for each 3D image in the sequence. When the feature of interest is the mitral valve, the approximating surface will be an approximating circle in the mitral valve plane. Thus, by defining a volume extending a predetermined length above and below said approximating surface and possibly a predetermined width towards the sides, a VOI can be selected which contains the mitral valve and possibly additionally the LVOT but without containing further obstructing anatomical structures. Thereby, volume rendering provides an unobstructed and accurate view of the anatomical feature of interest (e.g. the mitral valve).

[0054] According to useful embodiments, input tools are provided with the three-dimensional visualization environment.

[0055] In a non-VR visualization environment, the input tool can be a keyboard, a pointing device such as a mouse, a trackball, a touchpad or a touch-sensitive display, which is typically used in conjunction with an interactive panel comprising buttons, sliders etc. viewed on the screen. Such buttons or sliders can be actuated, for example, by the user with his finger or the pointing device, e.g. the user can move a cursor over the screen to actuate the input tool. With such an input tool, the user can, for example, zoom in and out of the visualization, adapt the visualization parameters / settings such as volume rendering threshold, smoothing, illumination and contrast, start and hold a cine mode, and perform measurements on the volume rendered VOI. In particularly useful embodiments, the input tool allows the user to select points and make measurements on the anatomy. For example, the user can select two points on the volume rendering, and the distance between such points will be automatically calculated. This feature is useful when planning an intervention, e.g. selecting an implant. In some embodiments, the user can be able to "grab" a visualized object by means of the pointing device, i.e. a touch on the mouse or touch-sensitive display, and thereby move or tilt it.

[0056] In a VR environment, such an input tool is preferably implemented by a virtual controller, which allows the user to at least grab and move objects within the virtual reality environment by gestures. In addition, the VR controller can comprise buttons or sliders, which the user can select by. In a VR environment, a user wearing a VR headset and holding at least one VR controller in one hand, preferably one VR controller in each hand, sees a static or dynamic visualization of the anatomy comprising the volume rendered VOI and the dynamic model in the VR environment. Preferably, he also sees the controllers at a position and orientation corresponding to the current hand position and orientation. Thus, the VR environment offers the user the possibility to move the controllers towards the visualization, grab it by pressing a specific button and move, turn or tilt the visualized object with his hand movements, as he would do with real world objects. Thus, the user has 18 degrees of freedom (six degrees of freedom for each of the VR headset and the two VR controllers, i.e. three rotational and three translational degrees of freedom) to correctly and intuitively view and analyze the visualized object. This is very similar to the natural way of interacting with objects.

[0057] According to a useful embodiment, the input tool as described above allows the user to select a plane in the three-dimensional volume visualized. The method then preferably comprises a step of displaying a multi-planar reconstruction (MPR) of the selected plane of at least one of the three-dimensional medical images of said sequence, in particular at a location in said three-dimensional visualization environment corresponding to the selected plane. A multi-planar reconstruction is an image reconstructed from several original image planes. For example, in CT, a stack of usually transversal images is acquired. Thus, if one wants to see a cross-sectional plane intersecting the stack of images in an orientation different from the transversal one, the user can select the desired orientation and create the MPR by interpolating for example from the respective nearest pixels in the various transversal slices. Displaying the MPR in addition to the volume rendered VOI and the dynamic model allows the user to see the anatomy in more detail. In a virtual reality environment, thanks to the 18 degrees of freedom (VR headset and two controllers), the correct positioning of the MPR plane in the 3D volume is very fast and verifiable, and the measurements on the MPR plane or within the volume rendered part become more precise and reliable.

[0058] According to a useful embodiment, the VR controllers allow the user to adjust parameters by means of gesture control. For example, the user selects a parameter by touching it in the VR environment using a hand movement. He can then use gestures to actuate for example a virtual slider, or simply move the controller horizontally (or vertically) to adjust the parameter without reference to any slider. Suitable parameters are related to the visualization and can be chosen from among volume rendering threshold, smoothing, lighting intensity, size, opacity of the visualized object, start and hold cine mode, etc.

[0059] In a useful embodiment, the three-dimensional visualization environment further comprises displaying a computer graphics object corresponding to a medical device, in particular an implant, in the same coordinate system as the volume rendering and the dynamic model. The computer graphics object is for example a three-dimensional representation of geometrical data, e.g. a 3D structure defined by vertices, such as a polyhedron. The computer graphics object is preferably locked to the movement of the anatomical structure, i.e. once it is placed in a certain position and orientation with respect to the volume rendered VOI in any one frame. When the user starts the cine mode, the processor controlling the visualization remembers the relative position and orientation of the computer graphics object with respect to the volume rendered VOI and will keep this relative position and orientation. In case the computer graphics object represents a new valve, such new valve can be locked to the movement of the annulus, e.g. the mitral annulus. Preferably, this can be done using 3D speckles, as described above. Thus, important dynamic information over the entire cardiac cycle is delivered and the valve can be optimally positioned, avoiding or limiting any obstruction of outflow. Locking the position of the computer graphics object to the position in the dynamic model can be done by assigning one or several of the set of landmark points to the computer graphics object, which then has a fixed relative position to such landmark points over the time period.

[0060] According to a further embodiment, the user is provided with an input tool allowing the user to move and tilt the computer graphics object with respect to the visualization of the volume rendered VOI and / or the dynamic model. Thus, the user can not only measure, but also "try" the selected implant or implant size, e.g. a replacement valve, to see if it fits the anatomical feature, e.g. the mitral valve. For example, the user can select the best fitting valve from a library and place the valve, or rather the computer graphics object corresponding to the valve, inside the visualization object for an initial check. In a particularly useful embodiment, the computer graphics object looks similar to what the medical device will look like on the intervention X-ray images (fluoroscopy images), since minimally invasive interventions are almost always done under fluoroscopy control. Thus, the user can visualize the scene in three dimensions and also gets an idea about what the implant will look like on the fluoroscopy images. The computer graphics object is preferably three-dimensional, it can for example be a simplified model of the implant, e.g. in the form of a wire mesh or an object defined by a set of simple surfaces.

[0061] The computer graphics object can also be a measuring device, such as a measuring tape or a ruler.

[0062] All the embodiments described herein are applicable to a "conventional" visualization environment that can be implemented on a computer screen, a tablet or a display and in a VR environment. However, the VR environment is particularly advantageous as it provides a real 3D view and the most intuitive / quick user experience / processing, as well as 6, 12 or 18 degrees of freedom of the user's movement with respect to the visualized objects.

[0063] Preferably, the method according to the application is executed by a processor that can be included in any electronic device able to control a display, in particular a VR display such as a VR headset or a projection display. Such a digital device can be a computer, a PC, a server, a television, a tablet, a smartphone, a laptop, a handheld device, etc. The processor can also be part of a cloud computer, a workstation or a console of a medical imaging device, in particular an ultrasound scanner.

[0064] According to another aspect, the application provides a computer program comprising program code instructions which, when executed by a processor, cause the processor to perform the method of the application. The computer program can be any code, in particular a code adapted for computer graphics applications, in particular for VR programming.

[0065] In yet another aspect, the application relates to a computer readable medium comprising the computer program as defined above. The computer readable medium can be any digital data storage device such as a USB stick, a hard disk, a CR-ROM, an SD card or an SSD card. Of course, the computer program does not need to be stored on such a computer readable medium to be supplied to a customer, but can be downloaded via the Internet.

[0066] According to yet another aspect, the application relates to a user interface configured to visualize a dynamic anatomical structure. The dynamic anatomical structure has been captured on a sequence of 3D medical images over a time period, each 3D medical image of the sequence showing the dynamic anatomical structure at a time point during the time period. The user interface comprises:

[0067] a) a three-dimensional visualization environment configured to provide a three- dimensional visualization of the dynamic anatomical structure over the time period, and

[0068] a processor configured for generating a visualization corresponding to a particular time point within the time period, the visualization comprising:

[0069] (i) a display of a volume rendering of a volume of interest within the three- dimensional image corresponding to the particular time point, the volume of interest containing an anatomical feature of interest; and

[0070] (ii) a display of a dynamic model of at least part of the dynamic anatomy at the specific point in time and in the same coordinate system as the volume rendering of the VOI,

[0071] wherein the VOI follows the position and / or shape of the anatomical feature of interest over the time period when the dynamic model and the volume rendered VOI are displayed in cine mode, and wherein the VOI is smaller than the full field of view of the three-dimensional medical image.

[0072] Any feature or useful embodiment described in connection with the method of the application is also applicable to the user interface.

[0073] The user interface is for example a system comprising at least a screen or display (VR or non-VR display) and input tools as generally described above, allowing the user to interact with the content of the display, for example by adjusting visualization parameters / settings, zooming, annotating and / or moving or tilting the displayed objects.

[0074] In a preferred embodiment, the three-dimensional visualization environment is a virtual reality environment and the display of the volume rendering and the dynamic model is a virtual reality display, in particular on a virtual reality headset.

[0075] In an advantageous embodiment, the virtual reality environment comprises at least one input tool, wherein the input tool is a virtual reality controller allowing the user to perform one or more of the following actions:

[0076] grabbing and moving objects displayed in the virtual reality environment;

[0077] taking measurements on the anatomical structure;

[0078] adjusting parameters used in the visualization by means of gesture control; and

[0079] annotating the anatomical structure, wherein the annotations are locked to positions in the dynamic model when the dynamic model and the volume rendered VOI are displayed in cine mode.

[0080] The virtual reality environment can be implemented using a commercially available VR device, such as the HTC VIVE or VIVE Pro virtual reality system, which comprises a VR headset, two VR controllers, two position trackers and optionally a speaker system (manufactured by HTC Corporation, Taoyuan City 330, Taiwan). BRIEF DESCRIPTION OF DRAWINGS

[0081] Useful embodiments of the application will now be described with reference to the accompanying drawings. In the drawings, like elements or features are designated with the same reference signs. In the drawings:

[0082] Figure 1 A schematic cross-section (4-chamber view) through a human heart is shown;

[0083] Figure 2 A dynamic surface model of the left ventricle is shown;

[0084] Figure 3 A schematic representation of a sequence of medical images is shown;

[0085] Figure 4 A schematic diagram of a dynamic surface model including a reference position for a volume rendered volume of interest is shown;

[0086] Figure 5 A schematic representation of a volume rendering of a three-dimensional ultrasound image of a heart including a volume of interest is shown;

[0087] Figure 6 A schematic representation of a dynamic model and a tracked volume of interest is shown;

[0088] Figure 7 A representation of a three-dimensional visualization according to an embodiment of the invention is shown;

[0089] Figure 8 A virtual reality environment according to an embodiment of the invention is shown;

[0090] Figure 9 A larger view of a three-dimensional visualization according to an embodiment of the invention is shown;

[0091] Figure 10 A user interface according to a first embodiment of the invention is shown;

[0092] Figure 11 A user interface according to a second embodiment of the invention is shown;

[0093] Figure 12 A flowchart illustrating a method according to an embodiment of the invention is shown.

[0094] Reference signs

[0095] 1 heart

[0096] 2 left atrium

[0097] 3 mitral valve

[0098] 4 left ventricle

[0099] 5 aortic valve

[0100] 6 aorta

[0101] 7 right atrium

[0102] 8 tricuspid valve

[0103] 9 right ventricle

[0104] 10 pulmonary valve

[0105] 11 pulmonary artery

[0106] 12 heart wall

[0107] 13 septum

[0108] 14 dynamic surface model

[0109] 18 bag model of left ventricle

[0110] 16 mitral annulus

[0111] 20 LVOT

[0112] 22 vertex defined model

[0113] 24 wire mesh

[0114] M1, M2, M3,..., M z sequence of medical images

[0115] 26 model of left atrium

[0116] 30 approximated surface / model of mitral annulus

[0117] 31 landmark point

[0118] 32 leaflet of mitral valve

[0119] 34 volume rendering

[0120] 36 heart

[0121] 40 VOI volume rendered

[0122] 42 valve opening

[0123] 44 arrow

[0124] 45 visualization

[0125] 50 VR environment

[0126] 52 VR horizon line

[0127] 54 volume rendering in form of VR objects

[0128] 56 VR light

[0129] 60, 60a, 60b VR controller

[0130] 62 button

[0131] 64 VR scale

[0132] 66, 66a, 66b computer graphics objects corresponding to valve implants

[0133] 70 screen

[0134] 71 input panel

[0135] 72 computer

[0136] 73 processor

[0137] 74 keyboard

[0138] 75 hard disk

[0139] 76 mouse

[0140] 80 user

[0141] 82 VR headset

[0142] 84 position sensor

[0143] 86 VR controller

[0144] 90-100 method steps DETAILED DESCRIPTION

[0145] In order to better visualize the preferred application of the visualization method and user interface of the present invention, Figure 1 The structure of a human heart 1 is illustrated. Blood from the lungs flows into the left atrium 2 and from there through the mitral valve 3 into the left ventricle 4. From there, it is pumped through the aortic valve 5 into the aorta 6. This section is also called the left ventricular outflow tract (LVOT). Blood from the body flows into the right atrium 7 and is pumped through the tricuspid valve 8 into the right ventricle 9. From there, it is pumped through the pulmonary valve 10 into the pulmonary artery 11. The heart wall 12 consists of muscle tissue surrounding the heart chambers 2, 4, 7 and 9. The left and right ventricles are separated by a septum 13. From Figure 1 It is evident from the above that the heart has a complex shape and in addition constantly moves with the heartbeat, i.e. it is a dynamic anatomical structure. Thus, it is difficult and error-prone to visualize the shape of, for example, the mitral valve 3 in order to plan a valve replacement.

[0146] In order to better visualize the left ventricular activity, it is possible to use Figure 2A dynamic surface model 14 is indicated. This surface model 14 represents a simplified model of the blood-tissue boundary of the left ventricle 4. It essentially has the shape of a bag 18 with two openings 16, 20 at the top: the opening 16 represents the mitral annulus, while the opening 20 represents the left ventricular outflow tract (LVOT) in which the aortic valve is located. The dynamic surface model 14 can be derived from a sequence of three-dimensional ultrasound images of the heart, in particular by tracking anatomical features or by tracking speckles from one image to the next. Thereby, for each three-dimensional image a surface model can be generated consisting of a set of vertices 22 across a wire mesh 24. Such a dynamic surface model 14 can be viewed in movie mode, i.e. like a movie, so that the viewer gets the impression of the left ventricle moving with the heartbeat. Obviously, similar dynamic surface models can be formed from other parts of the human anatomy, in particular other heart chambers or blood vessels.

[0147] Figure 3 A sequence of ultrasound images M1, M2, M3,..., M z of a heart is shown. Z is the number of images acquired during one heart cycle, i.e. in time T, where T is about 0.5 to 1.5 seconds. This shows two-dimensional images, however, preferably three-dimensional images are acquired at each time point t i The three-dimensional medical images can be formed from a stack of two-dimensional images. Such a sequence of images M1, M2, M3,..., M z may be acquired, for example, by moving an echocardiogram of the heart, e.g. with a TEE probe.

[0148] According to an embodiment of the present application, a VOI containing an anatomical feature of interest is defined on the three-dimensional images over a time period. As Figure 4 indicated, such a VOI can be determined by means of a set of landmark points or a reference position 30 on the dynamic surface model 14. In case the feature of interest is the mitral annulus, a suitable reference is the model of the mitral annulus 30. In Figure 4 this is indicated by the annular object 30. The mitral annulus 30 is located between the surface models of the left atrium 26 and the left ventricle 18. The LVOT 20 as well as the leaflets 32 of the mitral valve are also visible in Figure 4 this. Thus, the plane of the mitral annulus 30 can form a reference position for the VOI for volume rendering, which thus moves together with the surface model 14 over the heart cycle.

[0149] Figure 5 Volume rendering 34 of the three-dimensional images is illustrated, in this case volume rendered heart 36. As from Figure 5It is apparent that the volume rendering of a three-dimensional ultrasound image is often difficult to interpret as it contains many anatomical details. Therefore, according to the present application, a volume of interest 40 is defined and only the image content within this VOI 40 is submitted to the volume rendering process. In this case, the VOI contains the mitral valve 3, wherein the opening of the valve is shown at 42. The VOI 40 can be box-shaped or cuboid, as shown in Figure 5 It can also have any other three-dimensional shape, it can have the shape of a sphere, an ellipsoid, a cylinder, etc. For an application where the anatomical feature of interest is the mitral valve, the VOI can have the shape of a box or a cylinder, or even an irregular shape defined with a length extending above and below the plane of the mitral annulus. By defining the VOI in a fixed relative position with respect to the position of the anatomical feature, in particular on a dynamic model at each time point within a time period, the VOI is locked to the movement of the moving anatomical feature of interest (e.g. the mitral annulus) in the surface model during the entire time period (e.g. a cardiac cycle). Thereby, the feature of interest (e.g. the mitral valve 3) does not move out of the volume rendered VOI 40.

[0150] Thus, the visualization 45 corresponding to a particular time point provided by the three-dimensional visualization environment according to embodiments of the present application can look like Figure 6 as shown in Fig. 5: The visualization 45 comprises a visualization of a dynamic model 14 comprising bag-shaped surface models of the left ventricle 18 and the left atrium 26. For example, by segmentation on one 3D image, and by means of feature tracking or speckle tracking methods on further images, the mitral annulus 30 is determined on each of the three-dimensional images. Thereby, and a set of landmark points 31 is defined for the mitral annulus. An annular model 30 of the mitral annulus is fitted to the set of landmark points, and thereby an approximated surface is defined, which is a plane crossed by the fitted annulus 30, which in this case is spherical or ellipsoidal and planar. A VOI box 40 is defined with respect to the approximated surface 30 of the mitral annulus 30 on each of the three-dimensional images, and thereby moves with the moving heart, as indicated by arrow 44. Inside the box-shaped VOI, the three-dimensional images are volume rendered, preferably with an adjustable threshold, and thus, when viewed in movie mode, the volume rendering is locked to the mitral annulus.

[0151] This is further illustrated in Figure 7 Fig. 6, Figure 7A VR environment 50 according to an embodiment of the application is shown. When using such a user interface, the user will find himself in a virtual reality environment, including e.g. a virtual horizon 52 and virtual lights 56. Alternatively, he might find himself in a closed room. The visualization of the dynamic anatomy 45 will float in free space in front of the user, so he will see a three-dimensional visualization of the dynamic surface model 14, which looks like a transparent tube with the shape of the left ventricle and possibly the left atrium. At the location of the mitral valve, the VOI 40 is locked to the motion of the heart. Since the user will not actually see the contours of the VOI box 40, this box is indicated in dashed lines. What the user will see is a virtual reality object 54, and this corresponds to a volume or surface rendering of the image content inside the VOI box 40, i.e. a volume rendering 54 of the mitral valve 3. Both the surface model 14 and the volume rendering 54 will move with the heartbeat, where the user can start and stop the movie mode at any point in time within a time period, where the time period spanned by the sequence is at least one heartbeat. In a useful embodiment, the user can actuate an input tool, i.e. a virtual reality controller 60. This can have a button 62 which the user can press to start and stop the movie mode, or grab and move or tilt the visualization or virtual reality object 45. The user holds the VR controller 60 in his hand, and ideally will see the controller at a position in front of him corresponding to the real hand position.

[0152] In Figure 8 Further advantageous features of the VR environment 50 are illustrated in Fig. 6. In a useful embodiment, the user not only sees the controller 60a and the virtual reality object 45 including the volume rendered VOI and the dynamic surface model, but can see additional tools or VR objects. For example, a tool 64 can be a ruler or a scale for measuring the dimensions of the anatomical structure, e.g. the mitral valve. Alternatively, the VR controllers 60a, 60b held in either hand of the user can be used directly to perform measurements on the three-dimensional visualization 45 by the user selecting points in the VR space, where the processor calculates the distance therebetween. Based on such measurements, the best fitting valve can be selected from a library. Thus, the VR environment 50 can contain VR compatible computer graphics objects 66a, 66b representing implants, in this case mitral valve implants, which can be grabbed by the user with the controllers 60a, 60b and "tried on" the mitral valve represented in the visualization 45. Thus, the user is also able to try out the position of the valve 66a, 66b, so that he will be able to insert the valve at the correct position during the actual intervention. Thus, the VR objects 66a, 66b are composed of elements of the valve implant visible in the fluoroscopic images, in this case wire structures. Such valve prostheses can be used in valve-in-valve surgery and are directly deployed into the old valve in a minimally invasive procedure.

[0153] Figure 9 An enlarged view of the VR environment 50 is illustrated, showing the horizontal line 52 and the surface model 14 of the left ventricle. Also shown is the volume rendered valve 54. In this visualization 45, the user has selected and grabbed the new valve 66 (or rather the VR object corresponding thereto) and has placed it inside the three-dimensional visualization 45 for initial inspection. The new valve is then locked to the movement of the mitral annulus using 3D speckle tracking. The remaining LVOT 20 and possible obstruction can be measured and evaluated over the entire cardiac cycle. Furthermore, the placement of the VR object corresponding to the valve can be optimized in dynamic movement. The placement of the valve can be adjusted while moving within a simulated cardiac cycle.

[0154] Figure 10 A user interface according to a non-VR embodiment of the application is shown. In this setup, the dynamic visualization 45 is on a regular computer screen 70, and the visualization is merely a rendering on the two-dimensional screen 70. The screen can comprise a panel 71 of buttons and sliders which allow the user to tilt, zoom, move or otherwise manipulate the visualization 45. Also in such a user interface, a volume rendered VOI with a position of a feature of interest locked to a dynamic model on a dynamic anatomy, such as a beating heart, is a useful tool. The display can be controlled by a computer 72, such as a PC, comprising a processor 73 and a hard disk 75. The user interface can have input tools, such as a keyboard 74 and / or a mouse 76.

[0155] However, in the preferred embodiment, the user interface is a virtual reality interface, as Figure 11 is shown. Such an interface is implemented by a virtual reality headset 82 worn by the user 80. The headset 82 is connected to the computer 72 by a cable or by a wireless connection. Such a virtual reality headset 82 comprises an internal display separate for each eye, and a position sensor 84 which tracks head movement. In case an augmented reality environment is to be presented, such a headset can also comprise a camera. Furthermore, the user 80 holds a VR controller 86 in his hand, wherein the controller 86 also comprises a position sensor (not shown) and buttons or other input elements. Such a virtual reality controller 86 allows the user to grab and move objects displayed in the virtual reality environment 50. The VR headset can for example be an HTC VIVE headset and corresponding VR controllers.

[0156] Figure 12A flowchart illustrating a method according to an embodiment of the application is shown. In step 90, a sequence of three-dimensional medical images showing e.g. a moving heart is provided, the sequence spanning a time period corresponding to one heartbeat. In step 92, a dynamic model of at least part of the heart is provided, e.g. by fitting a surface model to part of the anatomy, or by generating a surface model (by segmentation and speckle tracking of an image). Thereby, the dynamic model is automatically registered with the sequence of images, i.e. the model can be shown at the correct anatomical position on the 3D images. In step 94, the position of an anatomical feature of interest can be determined over the time period and in particular by means of a reference structure such as the mitral annulus. Furthermore, a volume of interest (VOI) containing the anatomical feature of interest is defined in each of the three-dimensional images in step 94, such that the VOI follows the position and / or shape of the anatomical feature of interest over the time period. In step 96, a three-dimensional visualization environment is provided for visualizing the dynamic anatomy over the time period, as described above. Such a visualization environment can comprise input means in the form of a virtual reality controller, wherein input events from the user can occur in step 98. Such input events can then be used to change the visualization environment shown in step 96, e.g. by changing the lighting or threshold of the surface rendering. Depending on such input events 98, further objects can be shown in the visualization environment in step 100, e.g. a representation of a valve implant.

[0157] While the application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the application is not limited to the disclosed embodiments.

Claims

1. A method of visualizing a dynamic anatomical structure (1), the method comprising the steps of: a) providing a sequence of three-dimensional medical images (M1, M2, M3,..., M Z ) across a time period (T), each three-dimensional medical image in the sequence showing a dynamic anatomical structure (1) at a point in time during the time period; b) providing a dynamic model (14) of at least part of the dynamic anatomical structure, wherein the dynamic model is a simplification of the dynamic anatomical structure that follows the movement of the dynamic anatomical structure over the time period and that has been derived from and registered to the sequence of three-dimensional medical images; c) determining a volume of interest (40) within each of the three-dimensional medical images containing an anatomical feature of interest (3), wherein the volume of interest (40) follows the position and / or shape of the anatomical feature of interest (3) over the time period, and wherein the volume of interest (40) is smaller than the complete field of view of the three-dimensional medical images (M1, M2, M3,..., M Z ) and wherein the volume of interest is defined by identifying a set of landmark points of the anatomical feature of interest, by defining an approximated surface across the set of landmark points, and by defining a volume extending above and / or below and / or aside the approximated surface; and d) providing a three-dimensional visualization environment (50, 70) for displaying the dynamic anatomical structure over the time period, wherein a visualization (45) corresponding to a particular time point within the time period comprises: (i) a volume rendering of only the volume of interest (40) of the three- dimensional medical image corresponding to the particular time point; and (ii) a visualization of the dynamic model (14) at the particular time point and in the same coordinate system as the volume rendering of the volume of interest.

2. The method of claim 1, wherein, The three-dimensional visualization environment (50) is a virtual reality environment.

3. The method of claim 1 or 2, wherein, The dynamic anatomical structure (1) is a human or animal heart and the anatomical feature of interest (3) is a part of the heart.

4. The method of claim 3, wherein, The dynamic model (14) is a dynamic surface model of one or several heart chambers (2, 4, 7, 9) and the anatomical feature is a heart valve.

5. The method of claim 1 or 2, wherein, The dynamic model (14) is a dynamic surface model and the dynamic model (14) is derived from the sequence of three-dimensional medical images by segmenting all three- dimensional medical images or by segmenting in one three-dimensional medical image and speckle or feature tracking in subsequent images.

6. The method of claim 1 or 2, wherein, The position and / or orientation of the volume of interest (40) is determined over the time period by identifying a corresponding position (30) and / or orientation of the anatomical feature of interest (3) in the dynamic model (14).

7. The method of claim 1 or 2, wherein, The volume of interest (40) is determined by identifying a set of landmark points (31) of the anatomical feature in the dynamic model (14) or in the three-dimensional medical images and by defining an approximated surface (30) spanning the set of landmark points for each three-dimensional medical image and by determining the volume of interest (40) as a volume extending above and / or below and / or to the side of the approximated surface, wherein the landmark points (31) follow the position and / or shape of the anatomical feature of interest (3) over the time period.

8. The method of claim 1 or 2, wherein, The step of providing a three-dimensional visualization environment comprises: displaying a computer graphics object (66, 66a, 66b) corresponding to a medical device, in particular to an implant, in the same coordinate system as the volume rendering (54) and wherein the computer graphics object is locked to a position in the dynamic model (14) when the dynamic model and the volume rendered VOI (40, 54) are displayed in movie mode, and The user is provided with input tools (60, 60a, 60b) that allow the user to move and tilt the visualization of the volume rendering (54) and the dynamic model relative to the computer graphics objects (66, 66a, 66b) corresponding to the medical device.

9. The method of claim 1 or 2, wherein, The step of providing a three-dimensional visualization environment (50, 70) comprises: The user is provided with input tools (60, 60a, 60b, 71, 74, 76) that allow the user to select points within the dynamic anatomy and to take measurements of the dynamic anatomy.

10. The method of claim 2, wherein, The virtual reality environment comprises at least one input tool, wherein the input tool is implemented by a virtual reality controller and allows the user to grab and move objects within the virtual reality environment using gestures.

11. The method of claim 2 or 10, wherein, The virtual reality environment comprises at least one input tool, wherein the input tool is implemented by a virtual reality controller and allows the user to adjust parameters used in the visualization by means of gesture control, in particular to adjust settings such as thresholds used in performing volume rendering of the volume of interest (40).

12. A computer readable medium having program code instructions stored therein, the program code instructions, when executed by a processor, enabling the processor to perform the method according to any one of claims 1 to 11.

13. A user interface configured for visualizing a dynamic anatomical structure (1) that has been captured on a sequence of three-dimensional medical images (M1, M2, M3,..., M Z ) across a time period (T), each three-dimensional medical image in the sequence showing the dynamic anatomical structure (1) at a point in time during the time period, The user interface comprises: a) a three-dimensional visualization environment (50, 70) configured to provide a three-dimensional visualization (45) of the dynamic anatomy over a time period, and a processor (73) configured for generating a visualization (45) corresponding to a particular time point within the time period, the visualization (45) comprising: (i) a display of a volume rendering of only a volume of interest (40) within the three-dimensional medical image corresponding to the particular time point, the volume of interest (40) containing an anatomical feature of interest (3); and (ii) a display of a dynamic model (14) of at least a portion of the dynamic anatomy at the particular time point and in the same coordinate system as the volume rendering of the volume of interest (40), wherein the dynamic model is a simplification of the dynamic anatomy that follows the movement of the dynamic anatomy over the time period, wherein the volume of interest follows the position and / or shape of the anatomical feature of interest (3) over the time period, and wherein the volume of interest (40) is smaller than the complete field of view of the three-dimensional medical image (M1, M2, M3,..., M Z ) and wherein the volume of interest is defined by identifying a set of landmark points of the anatomical feature of interest, by defining an approximated surface spanning the set of landmark points, and by defining a volume extending above and / or below and / or aside the approximated surface.

14. The user interface of claim 13, wherein, The three-dimensional visualization environment (50) is a virtual reality environment and the display of the volume rendering (54) and the dynamic model (14) is a virtual reality display, in particular on a virtual reality head mounted device (82).

15. The user interface of claim 14, wherein, The virtual reality environment comprises at least one input tool (86), wherein the input tool is a virtual reality controller that allows the user to perform one or more of the following actions: grabbing and moving objects displayed in the virtual reality environment; taking measurements of the dynamic anatomy; adjusting parameters used in the visualization by means of gesture control; and annotating the dynamic anatomy (1), wherein the annotations are locked to locations in the dynamic model (14) when the dynamic model and the volume rendered volume of interest (40) are displayed in movie mode.

Images