Method of visualizing dynamic anatomical structure

Abstract

To provide a method for visualizing a dynamic anatomical structure, a program and an interface for minimizing the risk of misinterpreting image data, and in particular the risk of misinterpreting or incorrectly measuring relative spatial relationship between anatomical features.SOLUTION: A method comprises: providing a sequence of three-dimensional medical images of a dynamic anatomical structure 1 spanning a time period (T); providing a dynamic model 14; determining a volume of interest 40 containing a volume of interest 3 within each of three-dimensional images, the volume of interest 40 following the position and / or the shape of an anatomical feature of interest 3 across the time period, the volume of interest 40 being smaller than the complete field of view of three-dimensional medical images; and providing three-dimensional visualization environments 50, 70. A visualization 45 corresponding to a particular point in time comprises a volume rendering of the volume of interest 40 of the three-dimensional image, and a visualization of the dynamic model.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a method for visualizing dynamic anatomical structures, related computer programs, and user interfaces.

Background Art

[0002] Medical imaging techniques provide three-dimensional (3D) image data of the human or animal body. However, the images are usually viewed on a two-dimensional (2D) screen. Thus, there is a risk of misinterpreting the relative spatial relationships between anatomical structures represented on medical 3D data when the 3D dataset is observed and analyzed within an orthogonal plane on a 2D screen. A frequent cause of error is the selection of inaccurately positioned or skewed measurement planes for performing the analysis and measurements of anatomical structures.

[0003] 3D volume rendering is a set of techniques used to display 2D projections of 3D image datasets. However, while 3D volume rendering helps in forming a mental 3D model of anatomical structures, it is strongly dependent not only on the settings selected (thresholding, smoothing, etc.) but also on the image quality. This often incorporates image errors and thus entails the risk that the user no longer questions or verifies critical aspects of the volumetric rendering of anatomical structures. Nevertheless, the approach of dividing a 3D volume with one or more thresholds remains interesting when used for complex pathologies (e.g., tumors, cord ruptures, or calcified native valves).

[0004] On the one hand, simplified models of anatomical structures are highly regarded in the clinical setting because they can compensate for poor image quality and enable easier and faster interpretation. An example of a dynamic surface model is the commercially available software 4D LV-ANALYSIS (registered trademark) by TOMTEC Imaging Systems GmbH, which is a dynamic surface model of the right ventricular cavity, or TOMTEC Beutel (registered trademark), which is a feature of 4D MV. However, they are not suitable for all pathologies because they do not capture complex and delicate structures and do not display them appropriately.

[0005] Therefore, current approaches for segmenting and displaying medical image data of complex anatomical structures, such as cardiac 3D volume data, either overly simplify anatomical features by using surface models or shape models, or use threshold-based segmentation that can lead to incorrect conclusions. Thus, important geometric correlations and information required, for example, when planning a cardiac intervention, may be lost or overlooked.

[0006] Some studies function with 3D printing of medical models based on CT or 3D echo data. However, the 3D printing process requires a lot of time and expertise, is expensive, cannot be included in daily clinical routines, and cannot represent the dynamic movement of anatomical structures (especially in the case of 3D echo).

[0007] Virtual reality (VR) has been used when visualizing medical image data. For example, Thomas S. Sorensen et al. disclose in "A New Virtual Reality Approach for Planning Heart Interventions" (Artificial Intelligence in Medicine 22 (2001), 193 to 2014) the virtual reality visualization of cardiac magnetic resonance (MR) data. The disclosed method includes an optimized respiratory compensation 3D MR scan, segmentation, model generation, and interactive virtual reality visualization. The segmentation resulted in a set of contours in parallel planes that define different anatomical parts of the cardiovascular system. A 3D model is created by connecting these contours. The model is observed using shutter glasses in combination with a "holobench", which is a setup of two displays that form an angle of 90 degrees with each other.

[0008] Cristian A. Linte et al. disclose in "virtual reality enhanced ultrasound guidance A New Technique for Heart Interventions" (Computer Aided Surgery, March 2008, 13(2), 82 to 94) a virtual reality environment for visualizing real-time intraoperative echocardiography. The preoperative image and the position of the magnetically tracked surgical instrument can be presented to the surgeon to enhance the intraoperative ultrasound image. The preoperative image is aligned with the intraoperative TEE (transesophageal echocardiography) data using feature-based registration techniques.

[0009] U.S. Patent Application Publication No. 2014 / 052001 discloses using both B-mode data representing tissue and flow data representing regurgitant jets to automatically detect the mitral valve using a machine learning classifier. A series of classifiers can be used. For example, one classifier can be used to determine the position and orientation of the valve region, another classifier can be used to determine the regurgitant orifice, and a third classifier can be used to position the anatomical structure of the mitral valve. One or more features for some of the classifiers may be calculated based on the orientation of the valve region. When the mitral valve is detected, it is rendered as a mesh model and overlaid on the rendering of the heart.

[0010] US 2016220311 A1 discloses a processor that acquires image data from a medical imaging system. The processor generates a first model from the image data. The processor generates a computational model that includes cardiac electrophysiology and cardiac mechanics estimated from the first model. The processor executes tests on the computational model to determine the results of treatment. The processor overlays the outcome on the interventional image. Using interventional imaging, the initial heart model can be updated / registered during treatment to visualize its effect on the patient's heart. US2008194957 A1 discloses a method for generating a three-dimensional image of an object. The method includes providing a model of the object, irradiating a region of the object with ultrasonic waves from an external source transducer of the object, receiving an echo return from the object at a receiving transducer external to the object, processing the echo return, and generating a hybrid image of the object that includes a target region responsive to the model of the object and a target object region responsive to the echo return.

[0011] Lin W. et al. in "Visualisation of cardiac dynamics using physics based deformable mode" (Visual Communications and Image Processing 20.1.2004 20.1.2004 San Jose, vol. 3976, February 15, 2000 (2000.02.15), pp. 210 to 217, XP008019251, DOI 10.1117 / 12.383043 ISBN 978 1 62841 730 2) discloses generating images that display the dynamic motion of the left ventricle.

[0012] Accordingly, a first method is provided in which a surface model is created. Further, a second method is provided in which, in consecutive time point surface models, a physics-based deformable model is provided in which the surface mesh is deformed by following the trajectories of corresponding vertices that are connected.

Summary of the Invention

Problems to be Solved by the Invention

[0013] Accordingly, an object of the present invention is to provide a method for visualizing a dynamic anatomical structure that minimizes the risk of misinterpreting or inaccurately measuring the risk of interpreting image data, particularly the relative spatial relationships between anatomical features.

Means for Solving the Problems

[0014] According to a first aspect of the present invention, a method for visualizing a dynamic anatomical structure is provided to better solve one or more of the above problems. This method includes a) providing a sequence of three-dimensional medical images over a period, wherein each three-dimensional medical image of the sequence shows a dynamic anatomical structure at a time point during the period, b) providing a dynamic model of at least a part of the anatomical structure, the dynamic model being derived from and registered with the sequence of the three-dimensional medical images; c) determining a volume of interest containing an anatomical feature of interest in each of the three-dimensional images, the volume of interest following the position and / or the shape of the anatomical feature of interest during the period, the volume of interest being smaller than the full field of view of the three-dimensional medical images; d) providing a three-dimensional visualization environment for displaying the dynamic anatomical structure during the period, the visualization corresponding to a particular point in time within the period comprising: (i) volume rendering of the volume of interest of the three-dimensional image corresponding to the particular point in time; and (ii) visualization of the dynamic model in the same coordinate system as the volume rendering of the volume of interest and the particular point in time. It has. Preferably, the three-dimensional visualization environment is in the "cine mode", i.e., the "cinematic mode" or in a video, enabling the display of a dynamic model and a volume-rendered volume of interest for each three-dimensional image, which means that a series of visualizations are dynamically shown at a frame rate of, for example, 5 to 100 visualizations per second. Thus, the present invention combines two important approaches when viewing medical 3D volume data. The first approach is that at least a partially dynamic computer-generated model of the anatomical structure. Such a model shows a simpler version / abstraction of the anatomical structure, making it easier to navigate and interpret the anatomical structure, having advantages such as not being overly dependent on image quality, not having "holes", i.e., not incorporating artifacts. The second approach is volume rendering, which has the advantage of being suitable for more complex / anomalous anatomical structures or very individual structures such as valve tips, stenoses, calcifications, biological prostheses, ruptured cords, etc., which cannot be modeled by software such as 4D MV by TOMTEC. Furthermore, parameters such as thresholds, opacities, and contrasts can be adjusted "live", i.e., with an immediate effect while viewing the dynamic sequence of volume rendering. On the other hand, volume rendering is strongly dependent on image quality and thus may be too complex for easy interpretation.

[0015] The present invention provides a combination that offers the advantages of both approaches, where volume rendering is actually useful and is used only for the parts of the anatomical structures that are actually necessary. The region where volume rendering is used (=VOI) can be minimized with respect to the actual features of interest, so that a better overview can be obtained without the features of interest (e.g., valves) being out of focus. This can be done by also dynamically adjusting the position of the VOI using the points of the dynamic model. Thus, the present invention provides the excellent overview and navigation opportunities provided by a dynamic model, such as a shape / surface model, in combination with the adjustable and highly individualized advantages provided by volume rendering, while volume rendering is used only when necessary.

[0016] The dynamic anatomical structure may be any moving object within the body of a human or animal. In particular, it is a structure that is affected by periodic movements such as respiratory movements and heartbeats. Therefore, this method is particularly suitable for visualizing anatomical structures within the torso such as the heart, lungs, ribs, liver, kidneys, etc. The anatomical structure may be an organ or part of an organ of the body of a human or animal such as the heart, but it may also be a blood vessel or a bone structure. The method of the present invention is particularly suitable for hollow organs and organs containing cavities such as the heart.

[0017] The dynamic anatomical structure is captured in a series of three-dimensional medical images obtained from a human subject over a period of time, which may be preoperative images or intraoperative images. The series of 3D medical images can be referred to as 4D images. The three-dimensional (3D) images are typically digital images, for example, in the DICOM standard, that is, they include a three-dimensional array of voxels, and each voxel includes a grayscale value. Such 3D medical images are typically obtained from a field of view including the dynamic anatomical structure using a medical imaging modality such as MR, computed tomography (CT), positron emission tomography (PET), or ultrasound (US). When the anatomical structure is the heart, ultrasound, particularly transesophageal echocardiography (TEE), can be advantageously used. One 3D image from the time sequence of 3D images is also referred to as a "frame" hereinafter. The 3D images may be acquired at a frame rate of, for example, 5 to 100, preferably 20 to 60 images per second to enable a smooth representation of the dynamically moving anatomical structure, and in this case, it is displayed in cine mode. The period is typically at least one cycle of periodic motion, for example, at least one heartbeat.

[0018] At least a dynamic model of a part of the anatomical structure is in particular a simplified model of the anatomical structure, for example, a triangular surface model of a specific interface within the anatomical structure, for example, the blood-tissue interface of a blood vessel or a heart chamber. The model can include several points over the lines or surfaces of each frame. It can also be a mathematical model, such as a parameterized model like a surface or volume spanned by spline curves. This model is dynamic, that is, this model follows the movement of the anatomical structure over this period. The purpose of the dynamic model is to visualize at least a part of the anatomical structure, for example, one or some heart chambers of the moving heart, without disturbing the user's view too much. Therefore, such a simplified model is useful, for example, when planning an intervention or when making measurements on a specific part of the anatomical structure, to provide direction to the user.

[0019] The dynamic model is extracted from 4D ultrasound image data, for example, by speckle tracking techniques, to automatically track the three-dimensional endocardial contour over the entire cardiac cycle, thereby generating a dynamic surface model of one or more cardiac chambers, particularly the left ventricle, and optionally at least a portion of the left atrium. Further, it may be a shape or surface model adapted to each 3D medical image.

[0020] The present invention is particularly useful for observing and analyzing specific anatomical features of interest that are typically part of an anatomical structure. Such features of interest are included in a volume of interest (VOI) that is smaller than the full field of view of the 3D medical image. It may be a specific part of an organ that makes up the anatomical structure, a part having a complex anatomical structure such as a cardiac valve. In a useful embodiment, the anatomical feature of interest is the mitral valve, tricuspid valve, aortic valve, or pulmonary valve. In other embodiments, the anatomical feature of interest may be another important blood vessel such as a coronary vessel, or another structure such as a tumor.

[0021] To view such an anatomical feature of interest in more detail, the present invention provides a volume rendering region of interest (VOI) that includes said anatomical feature, but preferably not much more than the anatomical feature of interest, i.e., the VOI is as small as possible. In other words, the size and / or shape of the VOI is adapted to be as close as possible to the size and / or shape of the anatomical feature of interest, preferably matching across the entire sequence of images. Thus, a VOI containing the anatomical feature of interest is determined within each of the three-dimensional images, and the volume of interest follows the position and / or shape of the anatomical feature of interest over time. The VOI is a sub-volume from a 3D image and is defined, for example, by a set of closed surfaces surrounding the volume belonging to said VOI. The VOI contains voxels with different gray values. When volume rendering the VOI, one or more settings / parameters, such as a threshold, determine how the voxels within the VOI are displayed. The VOI according to the present invention can have a rectangular parallelepiped, cylindrical, ellipsoidal, or irregular shape. The volume of interest is typically defined based on the position and / or shape of the anatomical feature of interest and can have a fixed size across a series of images, for example, a box or cylinder with a predetermined edge length and diameter. Said predetermined length and diameter are preferably selected such that the size of the VOI corresponds to the expected size of the anatomical feature of interest. In some embodiments, the size may vary over time depending on the size of the anatomical feature of interest on each 3D image. The purpose of the VOI is to define the volume on each 3D image that contains the feature of interest. Thus, by volume rendering only the image content within such a VOI, a very good visualization of the feature of interest can be obtained without the view being obstructed by other volume-rendered parts of the anatomical structures of less interest.

[0022] In useful embodiments, the dynamic model covers or abuts the anatomical features of interest, i.e., the dynamic model is spatially related to the anatomical features of interest, such as the left ventricle being in a spatial relationship with the mitral valve. The anatomical features may be part of the dynamic model. Thus, each volume rendered VOI is based on the corresponding 3D medical image. For example, the center of the VOI for each volume rendering of the sequence is in a fixed relative position with respect to the position of the anatomical features of the dynamic model at the corresponding time point within the period.

[0023] The step of determining the VOI may be performed as part of the step of providing the dynamic model. In the case of the mitral valve, for example, the dynamic surface model of the left ventricle includes a set of (landmark) points on the mitral annulus. The mitral annulus forms the anatomical junction between the ventricle and the left atrium and is the insertion site of the valve leaflet tissue. Thus, the dynamic model includes the definition of the position of the mitral valve, and this can be used to define the VOI for each 3D image within the sequence. In other embodiments, the position of the anatomical features of interest may be defined by segmentation techniques and feature / speckle tracking techniques, where specific landmark points are identified on one image and then tracked throughout the sequence of images. The position can be defined by a single point in the three-dimensional image space, but can also be defined by a set of points, or the position of a geometric structure, or the position of a volume.

[0024] To enable the user to view and analyze the anatomical features of interest, a three-dimensional visualization environment is provided for visualizing the dynamic anatomical structures over a period of time. The visualizations corresponding to specific points in time within the period include at least two different types of visualizations / depictions. These are in the same coordinate system, i.e., they are displayed in the correct relative spatial positions and orientations with respect to each other. If two different visualizations spatially overlap, they may be overlaid or superimposed on each other, for example, both may be drawn transparently, or one may be considered more important than the other and may overwrite the other. At least two visualized objects are: (i) the volume rendering of the VOI. That is, the image content of the three-dimensional medical image within the VOI is volume rendered and displayed. Since the VOI follows the position and / or shape of the anatomical feature of interest, this volume rendering should essentially be a volume rendering of the anatomical feature of interest (and, if possible, the immediate surroundings), but no more. This gives the user a detailed and unobstructed view of the specific anatomical feature of interest without losing the advantages of volume rendering, as the volume rendering is only applied where it counts. In useful embodiments, the selected settings for volume rendering, such as thresholds, smoothing, etc., are adjustable automatically and / or by the user. At least two visualized objects are: (ii) Second, the visualization of the dynamic model. In particular, a three-dimensional visualization of the dynamic model at the same time point as the three-dimensional image in which the VOI is volume rendered is shown. This provides the user with additional direction and navigation information. For example, when analyzing the mitral valve within the VOI, he can simultaneously track the left ventricular outflow tract (LVOT). This is important when planning an intervention procedure such as valve replacement, e.g., transcatheter aortic valve implantation (TAVI) or transcatheter aortic valve replacement (TAVR), or replacement of the mitral valve where the LVOT may not be occluded.

[0025] Such visualization can be displayed in a cine mode, also referred to as a cinematic mode or movie mode. In the cine mode, in order for the user to be given a good impression of the moving dynamic model along with the movement characteristics of interest, a series of visualizations corresponding to a sequence of 3D medical images are preferably shown at an appropriate frame rate on the order of 5 to 100 frames per second, preferably 20 to 60 frames per second. In a useful application, for example, the beating heart and the opening and closing of each valve are visualized, and the period is at least one heartbeat.

[0026] Thereby, the movement of the volume-rendered VOI (e.g., the part of the heart valve and the LVOT) is locked to the movement structure of the surface model (e.g., the mitral annulus), and thus moves dynamically throughout the cardiac cycle. Accordingly, the present invention prevents relevant parts of the anatomical structure (features of interest) from deviating from the VOI, ensuring that the current situation can be analyzed, measured, and interpreted more quickly and reliably. The visualization environment of the present invention can be used to observe and analyze complex dynamic anatomical structures, particularly for planning an intervention and / or determining the correct size, shape, and position of an implant to be implanted in a future intervention. The 3D visualization of the dynamic model is typically the rendering of a dynamic shape or surface model, and the rendering may be performed by techniques available from computer graphics, including shading, ray casting, ambient occlusion, etc.

[0027] Volume rendering can be performed by any volume rendering technique known in the art, such as that described in U.S. Patent Application Publication No. 2005 / 0253841, which is incorporated herein by reference. Typically, to perform volume rendering, it is necessary to define a camera position and viewing direction within the space. Also, some techniques define the opacity and color of all voxels. In some volume rendering techniques, isosurfaces (surfaces of equal grayscale value) are extracted from the volume and they are viewed by rendering them, for example, as a polygonal mesh or by directly rendering the volume as a block of data. The marching cubes algorithm is a common technique for extracting isosurfaces from volume data. Another common technique is volume ray casting. In this technique, a ray is generated for each desired image pixel. Using a simple camera model, the ray starts from the center of projection of the camera (usually the display position or the eye point) and passes through the image pixel on the virtual image plane floating between the camera and the volume being rendered. Next, the ray is sampled at regular or adaptive intervals throughout the volume. The data is interpolated at each sampling point, a transfer function is applied to form an RGBA sample, the result is added to the cumulative RGBA of the ray, and the process is repeated until the ray exits the volume. This process is repeated for each pixel on the screen to form the completed image.

[0028] According to a particularly useful embodiment, the three-dimensional visualization environment is a virtual reality environment. "Virtual reality" means any computer-generated visualization that provides a true three-dimensional experience of a depicted structure. Thus, the virtual reality (VR) environment of the present invention particularly provides visual feedback, but can also enable other types of sensory feedback such as hearing. The VR environment may be an augmented reality environment in which the user views the real environment statically, or it may be a mixed reality in which VR objects (e.g., volume renderings and dynamic models) are superimposed on real objects or vice versa, or real objects are superimposed on the virtual scene. The visualization of the volume-rendered VOI and the dynamic model can together form the object to be visualized, preferably a virtual reality object.

[0029] The virtual reality environment is generally realized by presenting stereoscopic images to the user, i.e., since each eye sees a different image, the brain combines the two different images into a true three-dimensional scene. Such binocular images can be presented in connection with shutter glasses, on any VR display such as a virtual reality headset or a multi-projection environment, or on a screen that intermittently shows two images.

[0030] In the VR environment, the volume-rendered VOI and the dynamic model may be displayed by stereoscopic rendering, where the visualization / image to be volume-rendered (or rendered by other means) is calculated twice for two viewing positions with a slight spatial offset, i.e., one viewing position for the left eye and one viewing position for the right eye. When the two visualizations calculated in this way are shown to the user, one for each eye, e.g., on a VR headset, the user obtains a true three-dimensional (VR) impression. Thereby, the volume-rendered VOI and the dynamic surface model can be converted into VR.

[0031] In a preferred embodiment, a person using the VR environment of the present invention can "look around" the artificial world, move around in it, and interact with virtual objects, features, or items. This effect is generally generated by a VR headset equipped with a head-mounted display having a small screen in front of each eye, but can also be generated through a specially designed room having a plurality of large screens. In order for the user to move around within the VR environment, position and orientation information must be transmitted by the headset to an electronic device (e.g., a computer) that generates the VR environment so that the visualization moves consistently with the movement of the user's head. In order for the user to interact with virtual features within the VR environment, hand movements that can be performed by a hand-held VR controller must also be tracked. However, this last feature is optional, as is the transmission of position / orientation information for the user to walk around within the virtual scene.

[0032] In a useful embodiment, the user can zoom / scale the visualization / model within the VR environment, adjust visualization parameters and rendering settings, and / or capture the visualization of the objects being displayed, particularly the volumetrically rendered VOI and / or dynamic models. Since they are locked to each other, they are preferably grasped and moved together. Further, in a useful embodiment, the VR environment includes a lamp that the user can grasp and move within the VR environment so as to affect the illumination of the volumetrically rendered VOI and the surface. In a useful embodiment, the brightness of the scene, particularly the brightness of the movable lamp, may also be adjusted. A further useful embodiment uses a VR environment in which several users can come together in one scene.

[0033] The virtual reality environment provides the advantage that the user can view and analyze the object to be visualized with high reliability because the user can obtain a true three-dimensional view of the anatomical structure. Furthermore, since the user can walk around and, in some cases, even into their surroundings, the object to be visualized (e.g., visualization of the human heart) can be displayed very enlarged so as to completely fill the space in front of the user. Thus, the user has a particularly good overview and can make measurements with high accuracy. Additionally, the processing of user input events is particularly easy and intuitive in a VR environment. Operations such as rotating and / or adjusting the settings of the volume-rendered VOI are very difficult to handle on a 2D screen but are very intuitive and fast in a VR environment using a VR controller.

[0034] However, the present invention can also be advantageously used in non-VR visualization environments. When VR (virtual reality) is not specifically mentioned, the following features and embodiments are useful for both VR and non-VR visualization environments.

[0035] In a useful embodiment, the dynamic anatomical structure is the heart of a human or animal, and the anatomical features of interest are parts of the heart such as heart valves or coronary blood vessels. When a sequence of 3D images is acquired by ultrasound such as TEE, this technique provides dynamic images at a high frame rate, is inexpensive compared to other imaging modalities such as CT or MRI, and is particularly useful because the risk and radiation exposure to the patient are minimal. In a further useful application, the dynamic model is a dynamic surface model of one or several heart chambers, and the anatomical feature is a heart valve. In a useful embodiment, the dynamic surface model is a model of the endocardium of the left ventricle and a part of the left atrium, and the feature of interest is the mitral valve.

[0036] The present invention can be particularly used when planning minimally invasive cardiac surgery such as cardiac valve surgery or cardiac valve replacement. New minimally invasive methods such as transcatheter valve replacement can be used in patients who were previously considered inoperable and / or not suitable for open heart surgery. Some transcatheter valve replacements (e.g., TAVR) use a fully collapsible bioprosthetic valve. However, it is extremely important for the success of these interventions that the existing pathology / shape is analyzed, fully understood, the new valve is carefully selected to ensure proper functioning and non-occlusion of the LVOT or coronary arteries, and the size and position are determined. This is particularly applicable to valve-in-valve (ViV) interventions. Thereby, in a dysfunctional valve procedure, the mitral valve is replaced with a new valve by a minimally invasive ViV method. Thereby, the replacement valve is placed inside the old valve, and the old valve is destroyed while being deployed. Therefore, it is important to place the valve in the correct position and make it the correct size. In particular, it is important that the new mitral valve does not occlude the left ventricular outflow tract (LVOT). Therefore, for the valve in the valve intervention plan, the VOI to be volume rendered includes the mitral valve and preferably also the LVOT. The left ventricle is represented by a dynamic surface model and includes the definition of a set of landmark points on the mitral valve annulus. These points can be used as a basis for defining the VOI, which moves with the heartbeat accordingly, thereby preventing the mitral valve from falling out of the volume of interest. In a useful embodiment, the user can measure, for example, the diameter of the mitral valve and thus select the optimal valve from the library.

[0037] Accordingly, when the anatomical feature is a heart valve, the present invention enables excellent visualization of the valve on the surface being the VOI. The remaining parts of the heart chamber, such as the left or right ventricle and / or atrium, are represented only by a simplified surface model that dynamically pumps with the heartbeat and does not obstruct the view of the anatomical structure of interest, the valve. In a useful embodiment, the dynamic model is a dynamic surface model derived from a sequence of three-dimensional medical images by image segmentation across the three-dimensional image or by image segmentation in one three-dimensional image and speckle or feature tracking in subsequent images. Image segmentation is the process of dividing a digital image into multiple segments or sets of pixels / voxels and is typically used to identify boundaries. Thus, segmentation is the process of assigning labels to all voxels within a 3D image such that voxels with the same label share certain characteristics. Image segmentation may be performed using thresholding, i.e., the same label is assigned to all voxels above or below a certain threshold. Other methods include clustering methods, edge detection, or region growing methods. In the case of the heart, segmentation may help to separate the blood-filled chambers, particularly the ventricles and atria, from heart tissues such as the heart wall and valves. For example, when the boundary between the blood-filled chamber and the tissue of the heart wall is detected, the surface model may be constructed, for example, by selecting a number of voxels on the boundary and connecting them to a wire mesh model or a triangulated surface model. This process may be performed for each 3D image within the sequence. According to another useful method, only one image within the sequence is segmented, for example, to extract a static surface model as described above. The dynamic model is then derived from this static model by feature tracking. Thereby, certain characteristics of the anatomical structure are selected in one 3D image of the sequence. Such features are then tracked from image to image across the sequence, preferably automatically using a feature tracking algorithm. In the case of ultrasound images, 3D speckle tracking is preferably used.This is due to the interference effect between echoes where the speckles overlap. Thus, since the generation of speckles is related to each anatomical structure, it is a feature tracking method that uses the characteristic speckle artifacts of M-mode ultrasound images for tracking. Thus, 3D ultrasound speckles can be tracked like any anatomical feature from image to image and thereby can be used to derive a dynamic model of the anatomical structure. In a particularly useful embodiment, speckle tracking is used to derive a dynamic surface model of the left ventricle. Such a surface model includes the mitral annulus, which defines and locks a VOI that includes the mitral valve, a region of interest anatomical feature.

[0038] According to a useful embodiment, the position and / or orientation of the volume of interest is determined over time by identifying the corresponding position and / or orientation of the feature of interest in the dynamic model. Thus, if the feature of interest is part of the structure modeled by the dynamic model, tracking the position and / or orientation of the VOI over a series of images is simplified by using the dynamic model, for example, by using a specific landmark point that is part of the model.

[0039] More generally, according to useful embodiments, the volume of interest can be defined by identifying a set of landmark points of anatomical features, by defining an approximate surface over the set of landmark points, and by defining a volume that extends above and / or below and / or to the sides of the approximate surface. Thereby, the volume of interest is locked to the anatomical features, and if the anatomical features are part of a structure modeled by a dynamic surface model, the VOI moves with the dynamic model, especially with the anatomical features of interest. The set of landmark points can be, for example, characteristic surfaces or boundaries, such as some points on the endocardium or epicardium. According to a preferred embodiment, the set of landmark points is a point on the mitral annulus or the annulus of another heart valve. The set of landmark points may also be referred to as a point cloud. The approximate surface can preferably have a predetermined shape such as spherical, spherical, elliptical or ellipsoidal, and is a surface fitted to the point cloud by fitting techniques to obtain the best fit (e.g., least squares) to the point group. The surface may advantageously be planar for efficient handling, but may also be non-planar so as to best fit the set of landmark points. Then, using the approximate surface thus defined, the VOI is determined by defining a volume that encloses the approximate surface and preferably extends a predetermined length above and / or below and / or to the sides of the approximate surface. In a useful embodiment, to ensure that the feature of interest is completely contained within the VOI, a predetermined length characteristic of the feature of interest, for example, in the case of the mitral valve, extends 1 cm above and below the approximate surface and, for example, 1 cm security width towards the sides. By using the predetermined length / width, a processor or computer that controls the visualization can automatically define the VOI from the set of landmark points on the feature of interest for each 3D image of the sequence. If the feature of interest is the mitral valve, the approximate surface is an approximate circle in the plane of the mitral valve. Thus, by defining a volume that extends a predetermined length above and below the approximate surface and, optionally, extends towards the sides by a predetermined width, a VOI can be selected that further includes the mitral valve and, optionally, the LVOT but does not include anatomical structures that interfere further.Thereby, volume rendering provides an unobstructed and accurate view of the anatomical features of interest, such as the mitral valve.

[0040] According to a useful embodiment, an input tool is provided together with the three-dimensional visualization environment.

[0041] In a non-VR visualization environment, the input tool can be a pointing device such as a keyboard, mouse, trackball, touchpad, or touch-sensitive display, which are usually used together with an interactive panel equipped with buttons, sliders, etc. visible on the screen. Such buttons or sliders can be actuated, for example, by the user with their finger or a pointing device. For example, the user can move the cursor on the screen to actuate the input tool. With such an input tool, the user can, for example, perform zoom-in and zoom-out of the visualization, adapt visualization parameters / settings such as volume rendering thresholds, smoothing, lighting, and contrast, start and hold the cine mode, and perform measurements on the VOI to be volume rendered. In a particularly useful embodiment, the input tool enables the user to select points and perform measurements on anatomical structures. For example, the user can select two points on the volume rendering, and the distance between such points is automatically calculated. This feature is useful when planning an intervention, for example, when selecting an implant. In some embodiments, the user can "grab" the object being visualized with a pointing device, i.e., a mouse or touch on a touch-sensitive display, thereby being able to move or tilt it.

[0042] In a VR environment, such input tools are preferably realized by a virtual controller that enables the user to at least grasp and move an object within the virtual reality environment by hand gestures. Further, the VR controller can include buttons or sliders that the user can select. In the VR environment, a user wearing a VR headset and holding at least one VR controller (preferably a VR controller for each hand) in one hand views a static or dynamic visualization of an anatomical structure consisting of a volumetrically rendered VOI and a dynamic model. Preferably, the user can also view the controller at a position and in a direction corresponding to the current hand position and direction. Thus, the VR environment provides the user with the possibility of moving the controller towards the visualization like a real-world object, grasping the controller by pressing a specific button, and moving, rotating, or tilting the visualized object with hand movements. Thereby, the user has 18 degrees of freedom (6 degrees of freedom, i.e., 3 rotational and 3 translational degrees of freedom for each of the VR headset and the two VR controllers) and can accurately and intuitively view and analyze the visualized object. This is very similar to a natural way of interacting with an object.

[0043] According to an advantageous embodiment, the input tool described above enables the user to select a plane within the visualized three-dimensional volume. The method then preferably includes the step of displaying a multi-planar reconstruction (MPR) of at least one selected plane of the sequence of three-dimensional medical images, in particular at a position within the three-dimensional visualization environment corresponding to the selected plane. A multi-planar reconstruction is an image reconstructed from several original image planes. For example, in CT, usually a stack of transverse images is acquired. Therefore, if a cross-section intersecting the stack of images in a direction different from the transverse direction is seen, the user can select the desired direction, and the MPR is created, for example, by interpolating from the respective closest pixels within the various transverse slices. By displaying the MPR in addition to the volume-rendered VOI and the dynamic model, the user can view the anatomical structure in more detail. In a virtual reality environment, thanks to 18 degrees of freedom (VR headset and two controllers), the correct positioning of the MPR plane grippable within the 3D volume becomes very fast, verifiable, and the measurements on the MPR plane or within the volume-rendered part become more accurate and reliable.

[0044] According to an effective embodiment, the VR controller enables the user to adjust parameters by gesture control. For example, the user selects a parameter by touching a parameter using hand movements in the VR environment. The user can then use gestures to, for example, actuate a virtual slider or simply move the controller horizontally (or vertically) to adjust the parameter without referring to any slider. Suitable parameters may be related to visualization and may be selected from, for example, volume rendering thresholds, smoothing, light intensity, size, opacity of the visualized object, start and hold of cine mode, etc.

[0045] In an advantageous embodiment, the three-dimensional visualization environment also includes displaying computer graphic objects corresponding to medical devices, in particular implants, in the same coordinate system as the volume rendering and the dynamic model. The computer graphic object is, for example, a three-dimensional representation of geometric data, for example a 3D structure defined by vertices such as a polyhedron. The computer graphic object is preferably locked to the movement of the anatomical structure, i.e., it is once placed in a specific position and orientation with respect to the VOI that is volume rendered in any one frame. When the user starts the cine mode, the processor controlling the visualization memorizes the relative position and orientation of the computer graphic object with respect to the volume rendered VOI and maintains this relative position and orientation. If the computer graphic object represents a new valve, such a new valve can be locked to the movement of the valve annulus, for example the mitral valve annulus. Preferably, this can be done using 3D speckles as described above. Thereby, important dynamic information over the entire cardiac cycle is delivered and the valve can be optimally positioned, thereby avoiding or limiting any obstruction of the outflow. Locking the position of the computer graphic object to a position within the dynamic model can be done by assigning one or several of a set of landmark points to the computer graphic object, and the computer graphic object then has a relative position fixed with respect to such landmark points over a time period.

[0046] According to a further embodiment, an input tool is provided to the user, which enables the user to move and tilt computer graphical objects with respect to the VOI to be volume rendered and / or the visualization of the dynamic model. Thereby, the user can not only measure the selected implant or implant size, e.g., a replacement valve, but also "try it out" to see if it fits anatomical features such as, for example, the mitral valve. For example, the user can select an optimal valve from a library and, for an initial examination, place the valve, or the computer graphic object corresponding to the valve, inside the object to be visualized. In a particularly useful embodiment, since the computer graphical object is such that minimally invasive interventions are almost always performed under fluoroscopic control, it appears the same as a medical device would appear on an intervention X-ray image (fluoroscopic image). Thus, the user can visualize the scene in three dimensions and, moreover, obtain an idea of how the implant would appear on a fluoroscopic image. The computer graphical object is preferably three-dimensional and can be, for example, a simplified model of an implant in the form of a wire mesh or an object defined by a simple set of surfaces.

[0047] The computer graphical object may be a measuring device such as a measuring tape or a yardstick.

[0048] All embodiments described herein are applicable to both a "conventional" visualization environment that can be realized on a computer screen, a tablet computer or a display, and a VR environment. However, the VR environment is particularly advantageous as it provides not only 6, 12, or 18 degrees of freedom for the user to move with respect to the object to be visualized, but also a true 3D view and the most intuitive / rapid user experience / handling.

[0049] Preferably, the method according to the present invention is executed by a processor that can be incorporated into any electronic device capable of controlling a display, particularly a VR display such as a VR headset or a projection display. Such digital devices may be a computer, a PC, a server, a television set, a tablet computer, a smartphone, a laptop, a handheld device, etc. The processor may be part of a cloud computer, a workstation, or a control console of a medical imaging device, particularly an ultrasonic scanner.

[0050] According to another aspect, the present invention provides a computer program including program code instructions that, when executed by a processor, cause the processor to execute the method of the present invention. The computer program may be any code, particularly code suitable for a computer graphics application, particularly code suitable for VR programming.

[0051] In a further aspect, the present invention is directed to a computer-readable medium comprising the computer program as defined above. The computer-readable medium may be any digital data storage device such as a USB stick, a hard disk, a CRROM, an SD card, or an SSD card. Of course, the computer program need not be stored on such a computer-readable medium for supplying to a user, but may be downloadable via the Internet. According to a further aspect, the present invention is directed to a user interface configured to visualize a dynamic anatomical structure. The dynamic anatomical structure is captured on a sequence of 3D medical images over a period, and each of the 3D medical images in the sequence shows the dynamic anatomical structure at a certain point in time during that period. The user interface is a) a three-dimensional visualization environment configured to provide a three-dimensional visualization of the dynamic anatomical structure during the period; a processor configured to generate a visualization corresponding to a specific point in time within the period, wherein the visualization is (i) A display of volume rendering of a volume of interest within the three-dimensional image corresponding to the specific time point, wherein the volume of interest includes an anatomical feature of interest, the display, and (ii) A display of a dynamic model of at least a part of the dynamic anatomical structure in the same coordinate system as the volume rendering of the volume of interest and at the specific time point, wherein the volume of interest follows the position and / or the shape of the anatomical feature of interest during the period, and the volume of interest is smaller than the entire field of view of the three-dimensional medical image, the display and has.

[0052] Any feature or useful embodiment described in connection with the method of the present invention is also applicable to the user interface.

[0053] The user interface is a system comprising, for example, at least a screen or a display (VR or non-VR display) and usually an input tool, and as described above, the user can, for example, adjust visualization parameters / settings, zoom, annotate, and / or move or tilt the object being displayed, thereby enabling interaction with the content of the display.

[0054] In a preferred embodiment, the three-dimensional visualization environment is a virtual reality environment, and the display of volume rendering and the dynamic model is particularly a virtual reality display on a virtual reality headset.

[0055] In an advantageous embodiment, the virtual reality environment has at least one input tool, and the input tool enables the user to

[0056] grasp and move an object displayed in the virtual reality environment,

[0057] measure the anatomical structure,

[0058] adjust the parameters used for the visualization by gesture control, and

[0059] An operation of annotating the anatomical structure, wherein the annotation is locked to a position within the dynamic model when the dynamic model and the volume-rendered volume of interest are displayed in cine mode. A virtual reality controller that enables one or more of the following to be executed.

[0060] The virtual reality environment can be realized using a commercially available VR device such as the HTC VIVE or VIVE Pro virtual reality system, which includes a VR headset, two VR controllers, two position trackers, and optionally a loudspeaker system (manufactured by HTC Corporation, Taoyuan City 330, Taiwan).

[0061] Next, useful embodiments of the present invention will be described with reference to the accompanying drawings. Similar elements or features are denoted by the same reference numerals in the drawings.

Brief Description of the Drawings

[0062]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0063] To better visualize the visualization method of the present invention and the preferred application of the user interface, FIG. 1 shows the structure of the human heart 1. Blood coming from the lungs flows into the left atrium 2, and from there through the mitral valve 3 into the left ventricle 4. From there, it is sent into the aorta 6 through the aortic valve 5. This part is also called the left ventricular outflow tract (LVOT). Blood coming from the body is sent into the right atrium 7 and through the tricuspid valve 8 into the right ventricle 9. From there, it is sent into the pulmonary artery 11 through the pulmonary valve 10. The heart wall 12 is made of muscle tissue surrounding the heart cavities 2, 4, 7 and 9. The left ventricle and the right ventricle are separated by a septum 13. As is clear from FIG. 1, the heart has a complex shape and is constantly moving with each heartbeat, that is, it is a dynamic anatomical structure. Therefore, it is difficult to visualize the shape of the mitral valve 3, etc. in order to plan valve replacement, and errors are likely to occur.

[0064] To better visualize left ventricular activity, the dynamic surface model 14 shown in FIG. 2 can be used. This surface model 14 represents a simplified model of the blood-tissue boundary of the left ventricle 4. This essentially has the shape of a bag 18 with two openings 16, 20 at the top, where the opening 16 represents the mitral valve annulus, while the opening 20 represents the left ventricular outflow tract (LVOT) where the aortic valve is located. The dynamic surface model 14 can be derived from a series of three-dimensional ultrasound images of the heart, in particular by tracking anatomical features or by tracking speckles from one image to the next. Thereby, for each three-dimensional image, a surface model can be generated that consists of a set of vertices 22 spanning a wire mesh 24. Such a dynamic surface model 14 can be viewed in cine mode, i.e., like a movie, so that the viewer can obtain an impression of the movement of the left ventricle with each heartbeat. Clearly, a similar dynamic surface model may be formed from other parts of the human anatomical structure, in particular other heart chambers or blood vessels.

[0065] FIG. 3 shows a schematic representation of a series of ultrasonic images M 1 , M 2 , M 3 ,... M Z of the heart. Z is the number of images acquired during one cardiac cycle, i.e., at time T, where T is about 0.5 to 1.5 seconds. The drawings show two-dimensional images, but preferably three-dimensional images are acquired at each time point t i . The three-dimensional medical images may be formed by stacking two-dimensional images. Such a series of images M 1 , M 2 , M 3 ,... M Z can be obtained, for example, by using a TEE probe and an echocardiogram of the beating heart.

[0066] According to one embodiment of the present invention, a VOI including an anatomical feature of interest is defined on a three-dimensional image over a period of time. As shown in FIG. 4, such a VOI can be determined by a set of landmark points or reference positions 30 on the dynamic surface model 14. When the feature of interest is the mitral annulus, an appropriate reference is the model of the mitral annulus 30. In FIG. 4, this is represented by a ring-shaped object 30. The mitral annulus 30 is between the surface model of the left atrium 26 and the left ventricle 18. The LVOT 20, like the mitral valve leaflets 32, is also seen in FIG. 4. Thus, the plane of the mitral annulus 30 can form a reference position for the volume-rendered VOI, and the VOI moves with the surface model 14 over the cardiac cycle accordingly.

[0067] FIG. 5 shows a three-dimensional image, in this case a volume rendering 34 of the volume-rendered heart 36. As is apparent from FIG. 5, volume renderings of three-dimensional ultrasound images are generally difficult to interpret because they contain a lot of anatomical details. Thus, according to the present invention, a volume of interest 40 is defined and only the image content within this VOI 40 is submitted to the volume rendering process. In this case, the VOI includes the mitral valve 3 and the valve opening is indicated at 42. The VOI 40 may be box-shaped or cubic, as shown in FIG. 5. It may also have any other three-dimensional shape, such as a sphere, an ellipsoid, a cylinder, etc. In an application where the anatomical feature of interest is the mitral valve, the VOI may have the shape of a box or a cylinder, or an irregular shape that extends only the length defined above and below the plane of the mitral annulus. By defining the VOI at a fixed relative position with respect to the position of the anatomical feature, particularly with respect to the dynamic model at each time point within the period, the VOI is locked to the movement of the moving anatomical feature of interest (e.g., the mitral annulus) within the surface model over the entire period, e.g., the cardiac cycle. Thereby, the feature of interest (e.g., the mitral valve 3) does not fall out of the volume-rendered VOI 40.

[0068] Thus, visualization 45 corresponding to a particular point in time provided by the three-dimensional visualization environment according to an embodiment of the present invention may appear as shown in FIG. 6, and visualization 45 includes visualization of the dynamic model 14 including the bag-shaped surface models of the left ventricle 18 and the left atrium 26. The mitral annulus 30 is determined on each of the three-dimensional images, for example, by segmentation on one 3D image and by feature tracking or speckle tracking methods on additional images. Thereby, a set of landmark points 31 is defined for the mitral annulus. The ring-shaped model 30 of the mitral annulus is fitted to the set of landmark points, thereby defining an approximate surface that is the plane spanned by the fitted ring 30, where the approximate surface is spherical or elliptical and planar in this case. The VOI box 40 is defined with respect to the proximal surface 30 of the mitral annulus ring 30 in each of the three-dimensional images, and thus moves with the beating heart as indicated by the arrow 44. Inside the box-shaped VOI, the three-dimensional image is preferably volume rendered with an adjustable threshold, and thus the volume rendering is locked to the mitral annulus when viewed in cine mode.

[0069] This is further shown in FIG. 7, which shows a VR environment 50 according to an embodiment of the present invention. When using such a user interface, the user will be in a virtual reality environment, for example, including a virtual horizon 52 and a virtual lamp 56. Alternatively, he may be in a closed room. The visualization of the dynamic anatomical structure 45 floats within the free space in front of the user, and thus the user will see a three-dimensional visualization of the dynamic surface model 14, which appears like a transparent blood vessel having the shape of the left ventricle and perhaps the left atrium. At the position of the mitral valve, the VOI 40 is locked to the movement of the heart. Since the user does not actually see the contour of the VOI box 40, the box is shown by a dashed line. What the user will see is the virtual reality object 54, which corresponds to the volume or surface rendering of the image content within the VOI box 40, i.e., the volume rendering 54 of the mitral valve 3. Both the surface model 14 and the volume rendering 54 move with the heartbeat, and the user can start and stop the cine mode at any point in time during the period, and the period covered by the sequence is at least one heartbeat. In a useful embodiment, the user can operate an input tool, i.e., a virtual reality controller 60. This can have buttons 62 that the user can press to start and stop the cine mode, or to grab, move, or tilt the visualization or the virtual reality object 45. The user holds the VR controller 60 in his hand and ideally looks at the controller in front of the user at a position corresponding to the position of the hand in real life.

[0070] Further advantageous features of the VR environment 50 are shown in FIG. 8. In a useful embodiment, the user views not only the controller 60a and the virtual reality object 45 including the VOI rendered in volume and the dynamic surface model, but also further tools or VR objects. For example, the tool 64 may be a yardstick or a ruler for measuring the dimensions of an anatomical structure, such as the mitral valve. Alternatively, the VR controllers 60a, 60b held in either of the user's hands can be directly used to perform measurements on the three-dimensional visualization 45 by the user selecting a point in the VR space, and the processor calculates the distance therebetween. Based on such measurements, the best-fit valve can be selected from the library. Thus, the VR environment 50 can include VR-compatible computer graphical objects 66a, 66b representing grafts, in this case mitral valve grafts, that the user can grasp with the controllers 60a, 60b and "try on" the mitral valve represented in the visualization 45. Thereby, the user can also try the positions of the valves 66a, 66b so that the valve can be inserted into the correct position during the actual intervention. Thus, the VR objects 66a, 66b are composed of fluoroscopic images, in this case the elements of the valve implant visible in the wire structure. Such artificial valves can be used for transcatheter valve-in-valve procedures and are expanded directly inside the old valve with a minimally invasive procedure.

[0071] FIG. 9 shows an enlarged view of the VR environment 50 showing the horizontal line 52 of the left ventricle and the surface model 14. The valve 54 rendered in volume is also shown. In this visualization 45, the user selects, grasps and places a new valve 66 (or the corresponding VR object) inside the three-dimensional visualization 45 for an initial inspection. The new valve is then locked to the movement of the mitral valve annulus using 3D speckle tracking. The remaining LVOT 20 and possible obstructions can be measured and evaluated over the entire cardiac cycle. Further, the placement of the VR object corresponding to the valve can be optimized in dynamic motion. The placement of the valve can be adjusted while simulating the movement within the cardiac cycle.

[0072] Figure 10 shows a user interface according to a non-VR embodiment of the present invention. In this setup, the dynamic visualization 45 is on a conventional computer screen 70, and the visualization is simply a rendering on the 2D screen 70. The screen can include a panel 71 of buttons and sliders that allow the user to tilt, zoom, move, or otherwise manipulate the visualization 45. Also, in such a user interface, it is a useful tool to have a volume-rendered VOI locked to the position of an interesting feature on a dynamic model of a dynamic anatomical structure such as a beating heart. The display can be controlled by a computer 72 such as a PC that includes a processor 73 and a hard disk 75. The user interface can have input tools such as a keyboard 74 and / or a mouse 76.

[0073] However, in a preferred embodiment, the user interface is a virtual reality interface as shown in FIG. 11. Such an interface is realized by a virtual reality headset 82 worn by the user 80. The headset 82 is connected to the computer 72 via either a cable or a wireless connection. Such a virtual reality headset 82 includes an internal display for each eye, as well as a position sensor 84 that tracks head movement. Such a headset can also include a camera if an augmented reality environment is presented. Further, the user 80 holds a VR controller 86 in their hand, and the controller 86 includes a position sensor (not shown) as well as buttons or other input elements. With such a virtual reality controller 86, the user can grasp and move an object displayed in the virtual reality environment 50. The VR headset can be, for example, an HTC VIVE headset and the corresponding VR controller.

[0074] FIG. 12 shows a flow diagram illustrating a method according to an embodiment of the present invention. At step 90, for example, a series of three-dimensional medical images showing a beating heart is provided, and this series of images spans a period corresponding to one heartbeat. At step 92, a dynamic model of at least a part of the heart is provided, for example, by fitting a surface model to a part of the anatomical structure or by generating a surface model by segmenting an image and feature / speckle tracking. Thereby, the dynamic model is automatically aligned with the sequence of images, i.e., the model can be shown in the correct anatomical position on the 3D image. At step 94, the position of the anatomical feature of interest may be determined over a period, in particular by a reference structure such as the mitral annulus. Further, at step 94, a volume of interest (VOI) including the anatomical feature of interest is defined within each three-dimensional image, and as a result, the VOI follows the position and / or shape of the anatomical feature of interest over a period. At step 96, as described above, a three-dimensional visualization environment is provided for visualizing the dynamic anatomical structure over a period. Such a visualization environment can include an input tool in the form of a virtual reality controller, and input events from the user can occur at step 98. Next, such input events can be used to change the visualization environment shown in step 96, for example, by changing the lighting or threshold of surface rendering. According to such input event 98, at step 100, a representation of a further object, for example a valve implant, can be shown in the visualization environment.

[0075] Although the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive, and the invention is not limited to the disclosed embodiments.

Description of Reference Numerals

[0076] 1 Heart 2 Left Atrium 3 Mitral Valve 4 Left Ventricle 5 Aortic Valve 6 major arteries 7 right atrium 8 tricuspid valve 9 right ventricle 10 pulmonary valve 11 pulmonary artery 12 heart wall 13 septum 14 dynamic surface model 18 left ventricular bag model 16 mitral annulus 20 LVOT 22 vertex definition model 24 wire mesh M 1 , M 2 , M 3 , … M Z Sequence of medical images 26 left atrial model 30 Approximate surface / model of mitral annulus 31 landmark 32 mitral valve leaflet 34 volume rendering 36 heart 40 VOI to be volume rendered 42 valve opening degree 44 arrow 45 visualization 50 VR environment 52 VR horizon 54 Volume rendering as a VR object 56 VR lamp 60, 60a, 60b VR controllers 62 button 64 VR reference 66, 66a, 66b Computer graphic objects corresponding to valve implants 70 screen 71 input panel 72 computer 73 processor 74 keyboard 75 hard disk 76 mouse 80 users 82 VR headsets 84 position sensors 86 VR controllers 90 to 100 method steps

Claims

1. 1. A method for visualizing dynamic anatomical structures, the method comprising: a) providing a sequence of three-dimensional medical images over a period of time, each three-dimensional medical image of the sequence showing a dynamic anatomical structure at a time during the period of time; b) providing a dynamic model of at least a portion of the anatomical structure, the dynamic model being derived and registered from the sequence of the three-dimensional medical images; c) determining a volume of interest including an anatomical feature of interest in each of said three-dimensional images, said volume of interest following said position and / or said shape of said anatomical feature of interest during said time period, said volume of interest being smaller than said complete field of view of said three-dimensional medical image; d) providing a three-dimensional visualization environment for displaying said dynamic anatomical structure during said time period, said visualization corresponding to a particular time point within said time period comprising: (i) a volume rendering of the volume of interest of the three-dimensional image corresponding to the particular time point; (ii) visualizing the dynamic model in the same coordinate system as the volume rendering of the particular time point and the volume of interest; and The method comprising:

2. The method of claim 1 , wherein the three-dimensional visualization environment is a virtual reality environment.

3. 3. The method of claim 1, wherein the dynamic anatomical structure is a human or animal heart and the anatomical feature of interest is a part of the heart.

4. The method of claim 3 , wherein the dynamic model is a dynamic surface model of one or more heart chambers and the anatomical features are heart valves.

5. 5. The method of claim 1 , wherein the dynamic model is a dynamic surface model and is derived from the sequence of 3D medical images by segmentation across all 3D images or by segmentation in one 3D image and speckle or feature tracking in the subsequent images.

6. 6. The method of claim 1, wherein the position and / or orientation of the volume of interest is determined during the time period by identifying a corresponding position and / or orientation of the feature of interest in the dynamic model.

7. 7. The method of claim 1 , wherein the volume of interest is determined by identifying a set of landmark points of the anatomical feature in the dynamic model or the three-dimensional images, the landmark points being determined according to the position and / or the shape of the anatomical feature of interest during the period, and by defining an approximation surface across the set of landmark points for each three-dimensional image, and determining the volume of interest as a volume extending above and / or below and / or to the sides of the approximation surface.

8. The step of providing a three-dimensional visualization environment includes: - displaying a computer graphic object corresponding to the medical device, in particular an implant, in the same coordinate system as said volume rendering, said computer graphic object being locked to a position within said dynamic model when said dynamic model and said volume rendered VOI are displayed in cine mode; providing an input tool to a user, the input tool enabling the user to move and tilt the computer graphic object relative to the volume rendering and the visualization of the dynamic model relative to the medical device; 8. The method according to claim 1 , further comprising:

9. The step of providing a three-dimensional visualization environment includes: Providing an input tool to a user, the input tool enabling the user to select points in the anatomical structure and to measure on the anatomical structure; 9. The method according to claim 1 , further comprising:

10. 3. The method of claim 2, wherein the virtual reality environment has at least one input tool, the input tool being realized by a virtual reality controller and enabling a user to grasp and move objects in the virtual reality environment using hand gestures.

11. 11. The method of claim 2 or 10, wherein the virtual reality environment has at least one input tool, which is realized by a virtual reality controller and allows a user to adjust parameters used for the visualization by gesture control, in particular settings such as thresholds used when performing volume rendering on the volume of interest.

12. A computer program comprising program code instructions which, when executed by a processor, enable the processor to carry out a method according to any of claims 1 to 11.

13. 1. A user interface configured for visualizing a dynamic anatomical structure, the dynamic anatomical structure being captured on a sequence of three-dimensional medical images over a period of time, each three-dimensional medical image of the sequence showing the dynamic anatomical structure at a time point during the period of time; The user interface includes: a) a three-dimensional visualization environment configured to provide a three-dimensional visualization of the dynamic anatomical structure over a period of time; a processor configured to generate a visualization corresponding to a particular point in time within the time period, the visualization comprising: (i) displaying a volume rendering of a volume of interest within the three-dimensional image corresponding to the particular time point, the volume of interest including an anatomical feature of interest; and (ii) displaying a dynamic model of at least a portion of the dynamic anatomical structure in the same coordinate system as the volume rendering of the volume of interest and at the particular time point, the volume of interest following the position and / or the shape of the anatomical feature of interest during the time period, the volume of interest being smaller than the complete field of view of the three-dimensional medical image; A user interface having:

14. The user interface of claim 13 , wherein the three-dimensional visualization environment is a virtual reality environment and the display of the volume rendering and the dynamic model is on a virtual reality display, in particular a virtual reality headset.

15. The virtual reality environment has at least one input tool, the input tool being configured to allow a user to: a grasping and moving action of an object displayed in the virtual reality environment; measuring said anatomical structure; adjusting the parameters used for said visualization by gesture control; and an operation of annotating the anatomical structure, the annotations being locked into position within the dynamic model when the dynamic model and the volume-rendered volume of interest are displayed in cine mode; 15. A user interface according to claim 13 or 14, which is a virtual reality controller enabling one or more of the following to be performed:

Images