Method, computer program and 3D medical image data evaluation device for analyzing 3D medical image data - Patents.com

Abstract

A method for displaying and analysing 3D medical image information is provided, comprising the steps of receiving 3D medical image data of an anatomical structure, generating a 3D rendering 1 of the anatomical structure based on the 3D medical image data, displaying the 3D rendering 1 to a user 20 in a visualization environment 10, receiving a user's command indicating a first point 4 on or in the 3D rendering 1, providing a 2D frame 2 at a position in the visualization environment 10 corresponding to the first point 4, a first side of the 2D frame 2 facing the user 20, displaying an MPR view 3 in the 2D frame 2, the MPR view 3 being based on the 3D medical image data at the position of the 2D frame 2 and sized to correspond to a size of the 2D frame 2, and providing a transparent cropper 9 between the user 20 and the first side of the 2D frame 2 to crop out a portion of the 3D rendering 1. Further provided is a computer program and a 3D medical image information evaluation device.

Description

[Technical field]

[0001] The present invention relates to a method for analysing 3D medical image data, a computer program and a 3D medical image data evaluation device. [Background technology]

[0002] There are multiple modalities to acquire three-dimensional (3D) medical image data of the human or animal body, such as 3D ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), 3D echocardiography (3DE), and cardiac computed tomography (CCT). One way to display the acquired 3D medical image data is to visualize it as a 3D volume generated by rendering the 3D image data. The visualized 3D volume is therefore also called a 3D rendering. The 3D rendering is displayed on a two-dimensional (2D) screen in a virtual reality environment (VR) or an augmented reality environment (AR). In particular, the upcoming development of consumer-friendly AR / VR headsets will enable users to step into the virtual space and view the 3D volume stereoscopically in "true 3D" using a headset with two separate screens and two separately calculated images for the left and right eyes. This, in combination with a very precisely tracked hand controller or hand tracking, will enable a very natural interaction with the 3D content. In medical 3D data imaging, VR analysis is associated with faster navigation and more accurate measurements with less variability compared to traditional imaging.

[0003] In any case, 3D volume rendering or navigation using multiplanar reconstruction (MPR) planes are the two most common ways to view medical 3D data. MPR is considered to be the "ground truth" of the dataset and is used to display specific planes cut through the 3D volume and to make measurements. On the other hand, 3D volume rendering combined with VR analysis provides a volumetric visualization of the dataset, but measurements on 3D volume rendering are very sensitive to the chosen volume threshold, potentially resulting in inaccurate measurements and erroneous clinical conclusions.

[0004] In a virtual reality environment, the volume rendering and the MPR are displayed simultaneously. However, there is a problem that the opaque MPR surface often obstructs the view of the 3D volume (i.e., anatomical structures) behind it, or vice versa.

[0005] It is therefore desirable to have a solution that optimally combines the visual overview capabilities of a 3D volume with the precise measurement capabilities of MPR.

[0006] US Patent Application Publication No. 2007 229500 A1 shows how a combination of direct volume rendering and MPR rendering can be used to render an MPR image in the context of the surrounding material. By shifting the MPR, the user interactively changes the incision area to reveal the interior of the image volume.

[0007] WO2017 / 212063A1 shows a medical imaging and visualization system with a user interface that enables a user to visualize a volume rendering of a three-dimensional dataset, manipulate the volume rendering to dynamically select MPR planes within the 3D dataset, and generate B-mode images at the selected MPR planes.

[0008] US Patent Application Publication No. 2012 / 0308095A1 shows a system for adjusting the visualization of volume data of an object as an image with respect to diagnostically relevant medical information related to the environment of the area under investigation. Thus, at least one slice area is specified based on slice information in the volume data. A first mapping of the volume data is used to visualize the at least one slice area on a display. A second mapping, different from the first mapping, is used to visualize areas adjacent to the slice area.

[0009] However, the problem remains that an opaque MPR surface often obstructs the view of the anatomical structures behind it, thereby immediately increasing the risk of setting an incorrectly positioned MPR surface and therefore inaccurate measurements. Summary of the Invention [Problem to be solved by the invention]

[0010] It is therefore an object of the present invention to provide a global visualization of 3D medical image data while allowing accurate measurement or analysis of the data set. [Means for solving the problem]

[0011] This object is solved by a method for analysing 3D medical image information comprising the features of claim 1, by a computer program comprising the features of claim 14 and by a device for evaluating 3D medical image information comprising the features of claim 15.

[0012] According to one aspect of the present invention, A method for analyzing 3D medical image information is provided, the method comprising: receiving 3D medical image data of an anatomical structure; generating a 3D rendering of the anatomical structure based on the 3D medical image data; displaying the 3D rendering to a user in a visualization environment; receiving a user command indicating a first point on or within the 3D rendering; providing a 2D frame at a location in the visualization environment corresponding to the first point, a first side of the 2D frame facing the user; displaying an MPR view in a 2D frame, the MPR view being based on 3D medical image information at a location of the 2D frame and sized to correspond to a size of the 2D frame; providing a transparent cutout between the user and a first side of the 2D frame to cut off a portion of the 3D rendering; has.

[0013] In contrast to the state of the art, the present invention provides a different approach of using both 3D rendering and MPR views to analyze 3D medical image information. In the prior art, either 3D rendering or MPR views are used to analyze medical image information, while in the present invention, both 3D rendering and MPR views are used simultaneously to analyze 3D medical image information as a whole. That is, in the prior art, both methods are used alternately. Thus, the advantages of each analysis method are combined to improve the analysis process of 3D medical image information.

[0014] In particular, the 2D frame in which the MPR view is shown can be positioned anywhere in the 3D rendering without being completely superimposed on the 3D rendering. In particular, the 3D rendering around the 2D frame is still visible (i.e., the 3D rendering is not completely hidden by the 2D frame). Thus, the user still gets a spatial overview of the 3D medical image information by looking at the 3D rendering. This is important because the position of the MPR view in the 3D rendering is critical for clinical decisions and is highly error-prone since a high level of experience and spatial imagination is required to select the correct position of the MPR view in the 3D rendering. According to the invention, the user has an improved orientation by looking at both the MPR view and the 3D rendering in order to correctly position the 2D frame. Preferably, the area covered by the MPR view (i.e., in the 2D frame) is smaller than the 3D rendering and represents a very limited section of the 3D medical image information. In other words, the visible area of ​​the MPR view is limited to the area of ​​the 2D frame.

[0015] For example, the user rotates or moves the 3D rendering to get a better orientation and to accurately position the 2D frame within the 3D rendering. In other words, the user orients himself with the 3D rendering, thereby accurately placing the 2D frame in the right position of the 3D rendering to analyze in detail this area (e.g., area of ​​interest) of the 3D rendering. With the crop, the user has a nearly unobstructed view of the MPR view. That is, the 3D rendering is only partially cropped so that the user still sees a part of the 3D rendering that is not cropped. That is, the 3D rendering is cropped to provide a nearly unobstructed view of the MPR view within the 2D frame. Thus, the analysis of the 3D medical image data is simplified and made more intuitive since the user can see both the 3D rendering and the MPR view at the same time, so that the user can analyze the MPR view in detail while still having an overview of the entire 3D rendering.

[0016] The 3D medical image data is provided, for example, as a digital image of the DICOM standard, i.e. as a digital image comprising a three-dimensional array of voxels, each voxel comprising a grayscale value. Such 3D medical image data is generally obtained from a field of view including the anatomical structure, for example, using a medical imaging modality such as MR, computed tomography (CT), positron emission tomography (PET), or ultrasound (US). When the anatomical structure is the heart, ultrasound and in particular transesophageal echocardiography (TEE) is advantageously used. For example, the 3D medical image data is acquired over a period of time to obtain four-dimensional (4D) medical image data, where the fourth dimension is time. In this case, one 3D image from the 4D medical image data is also referred to as a "frame" in the following. The 3D images are acquired at a frame rate of, for example, 5 to 100 images / s, preferably 20 to 60 images / s, to allow a smooth representation of dynamically moving anatomical structures. The period is typically at least one cycle of cyclical motion, for example at least one heart beat.The 3D medical image data is received either directly from a modality that acquired the medical image data or from a clinical image database.

[0017] The generation of the 3D rendering uses an adjustable threshold to display only certain voxels within the threshold range. Color shading and special lighting techniques are used to create the impression of depth on the 2D screen. The 3D rendering is generated by a volume rendering process performed using volume rendering techniques known in the art, for example as described in US Patent Application Publication No. 2005 / 0253841 A1, which is incorporated herein by reference. To perform a volume rendering, it is usually necessary to define the camera position and the viewing direction in space. This position corresponds to the user's viewing position. Furthermore, some techniques define the opacity and color of every voxel. A common technique is volume ray casting. In this technique, a ray is generated for each desired image pixel. Using a simple camera model, the ray starts from the center of the camera's projection (usually the viewing position or viewpoint) and passes through the image pixel on a virtual image plane that hovers between the camera and the volume to be rendered. The ray is then sampled at regular or adapted intervals throughout the volume. The data is interpolated at each sample point, a transfer function is applied to form an RGBA sample, and the result is added to the ray's accumulated RGBA, and the process is repeated until the ray has left the volume. The process is repeated for every image pixel on the screen to form the completed image. A 3D rendering can be thought of as a three-dimensional volume being visualized.

[0018] Volume rendering is performed according to the techniques described in "The Visualization Handbook", edited by Charles H. Hansen and Christopher R. Johnson, Elsevier Butterworth Heinemann 2005, in particular "Overview of Volume Rendering" by Arie Kaufmann starting on page 127, which is incorporated herein by reference.

[0019] The visualization environment is defined by a visualization coordinate system with coordinates X, Y, and Z (world coordinates). In addition, the 3D rendering is defined in an object coordinate system with coordinates X', Y', and Z' (object coordinates). Moreover, the 3D medical image data is defined relative to an original coordinate system with coordinates X", Y", and Z" (3D medical image coordinates). The coordinate systems are shifted relative to each other. As a result, objects such as the 3D rendering, the 2D frame, the surface mode, etc. are defined in separate coordinate systems, as they are shifted relative to each other. On the other hand, the user's viewing position is defined by the visualization coordinate system.

[0020] The visualization environment is a VR / AR visualization environment or a non-VR visualization environment. In a non-VR visualization environment, the input tools used by the user are a keyboard, a pointing device, such as a mouse, a trackball, a touchpad, or a touch-sensitive display, which are typically used in conjunction with an interactive panel including buttons, sliders, etc. displayed on a screen. Such buttons or sliders are, for example, actuated by the user with a finger or a pointing device, e.g., the user moves a cursor on the screen to actuate the input tool. By such input tools, the user can, for example, zoom in and out the visualization, adapt visualization parameters / settings such as threshold, smoothing, lighting, and contrast of the 3D rendering, start and hold cine mode, and perform measurements on the 3D rendering and / or MPR view. In a particularly useful embodiment, the input tools allow the user to set points on an anatomical structure and perform measurements. For example, the user selects two points on the 3D rendering and / or MPR view, and the distance between such points is automatically calculated. This feature is useful when planning an intervention, e.g., when choosing an implant. In some embodiments, the user can "grab" a visualized object by a pointing device, i.e., a mouse, or by touch on a touch-sensitive display, thereby moving or tilting it. The object may be a simulated implant or other object that must be individually matched to the patient.

[0021] In a VR environment, such an input tool is preferably realized by a virtual controller that allows the user to grab and move objects in the virtual reality environment at least by hand gestures. "Move" in this context also means rotate. The VR controller further allows the user to change the size and / or shape of the VR object. Furthermore, the VR controller includes buttons or sliders by which the user makes selections. In the VR environment, a user wearing a VR headset and holding at least one VR controller in one hand (preferably one in each hand) sees in the VR environment a static or dynamic visualization of the anatomical structure consisting of 3D renderings and MPR views. Preferably, the user also sees the controller in a position and orientation corresponding to the current hand position and orientation. Thus, the VR environment allows the user to move the controller towards the visualization, grab it by pressing a specific button, and move and / or rotate the visualized object by the user's hand movements as in the case of real world objects. Thereby, the user has 18 degrees of freedom (6 for the VR headset and each of the two VR controllers, i.e. 3 rotation and 3 translation) to correctly and intuitively view and analyze the visualized objects, which closely resembles the natural way of interacting with objects.

[0022] For example, a command may be issued by a user analyzing the 3D medical image data, by any of the user actions described above. a) initiating a workflow to analyze 3D medical image data; b) moving a pointer of a pointing device (e.g., a mouse, a VR controller, a touchpad, a trackpad, a trackwheel, etc.) to a specific location within the visualization environment, pressing a button, and / or remaining at that location for a predefined period of time; c) a gesture made by a user, and / or d) Pressing the bottom (e.g., the virtual bottom) It is.

[0023] In any case, the command results in the definition of a first point having a spatial location within at least one of the coordinate systems defined above. For example, the first point is defined in an original coordinate system X", Y", and Z" and then transformed to an object coordinate system X', Y', and Z'. That is, the first point is located at predetermined coordinates within the visualization environment. Preferably, the coordinates define a spatial location (x, y, z) within the visualization environment.

[0024] Case a) involves the initiation of the execution of the method on a computer or other machine including a processor. That is, the command is a command to start the execution of the method. In this case, the first point indicated by the user has a predefined position defined in the visualization coordinate system by coordinates X, Y, and Z in the visualization environment. Preferably, the coordinates define a spatial location in the visualization environment. This allows the user to perform the analysis immediately upon starting the execution of the method. This makes the workflow particularly user-friendly. In addition, the workflow is a measurement workflow that is activated by the user when the user wants to measure distances, areas, surfaces, etc. of anatomical structures. Thus, the first point (and therefore the 2D frame) is provided in the visualization environment only when the user wants to execute a particular workflow (e.g. perform a measurement). Until such a command, there is no 2D frame in the visualization environment and the user sees the entire 3D rendering without any obstruction. Hence, the user gets an impression of the entire 3D rendering before starting the execution of a particular workflow. The initial position of the first point is set before starting the method.

[0025] Case b) is considered as actively issuing a command directly indicating the location of the first point. For example, the user moves a controller (in the case of a VR environment) or other pointing device to a certain spatial location in the visualization environment and presses a button on the controller when the location is reached. Alternatively, the user moves the pointer of the pointing device (mouse, trackpad, etc.) to such a location and activates the button. The first point is defined accordingly in at least one of the coordinate systems. This allows the user to actively control when the command is given and where the first point is located in the visualization environment. That is, the spatial location where the first point should be defined is directly and individually controlled by the user. Hence, an individual work process is provided.

[0026] Case c) is a predefined gesture by the user (e.g., a nod of the head, a movement with one or two arms, etc.) that is recognized by the system in which the method is executed. A receiver is therefore provided to recognize such a gesture by the user. If the system includes a VR headset, the receiver is provided in the headset. Alternatively or additionally, a camera is provided that is configured to receive the gesture by the user. In case c), the first point specified by the user is a given spatial location in the visualization environment, or the spatial location is derived by the gesture itself. For example, the gesture specifies a spatial location in the visualization environment.

[0027] Depending on the first point, a 2D frame is provided in the visualization environment. That is, based on the first point, the spatial position of the 2D frame is derived directly or indirectly from the position of the first point. The 2D frame is provided on or in the 3D rendering. In other words, the 3D rendering shows the anatomical structures contained in the 3D medical image data. The 3D rendering is therefore a closed volume with a contour. The 2D frame is on the surface of the 3D rendering (i.e. on the surface of a closed volume), inside the 3D rendering, outside the 3D rendering, or partly inside and partly outside the 3D rendering.

[0028] In case d), the user taps a virtual button for issuing a first point, etc., without directly defining the spatial location of the first point. In this case, the first point is first defined by the coordinates X, Y, and Z in the visualization coordinate system. This is done, for example, when the user does not know where exactly to position the 2D frame in the 3D rendering and first wants to have an MPR view in place to find such a location in the 3D rendering.

[0029] A 2D frame extends only in two dimensions. That is, the 2D frame does not extend in a third direction (e.g., in the depth direction or in the z direction). Alternatively, the 2D frame has the same thickness as the MPR view (e.g., the same thickness as the MPR slice). In other words, the 2D frame has the same dimensions as the MPR view. The 2D frame is considered a picture frame, where the picture is the MPR view, surrounded by a thin line or border to better indicate the location of the 2D frame. There are at least four embodiments for providing a 2D frame in a visualization environment, which may be combined with any of a) to d) above. I) The 2D frame is centered on the first point indicated by the user's command, i.e. the 2D frame is positioned such that the center of gravity of the 2D frame is at the location of the first point. II) As mentioned above, the 2D frame is centered on a first point indicated by the user's command. Moreover, the first point has a fixed spatial position in the same coordinate system (e.g., in the object coordinate system) as the 3D rendering. In other words, if the 3D rendering is moved and / or rotated, the first point and therefore the 2D frame are also moved. That is, the three degrees of freedom of the 2D frame (up-down, front-back, left-right) are fixed relative to the 3D rendering. However, the 2D frame is still moved in three degrees of freedom, namely roll, yaw, and / or pitch. This is advantageous when a specific part of the anatomical structure is to be examined. Thus, the user positions the first point by command at such section, and the 2D frame is provided at that specific position, regardless of the rotation or movement of the 3D rendering. In addition, the 2D frame is positioned in the visualization environment and two points are fixed relative to the object coordinate system. One of these points is the first point and the second point is defined such that the 2D frame faces the user. The 2D frame is therefore only moved in the rotational degree of freedom, which is particularly easy for the user, since he can immediately start analysing the MPR views in the 2D frame. III) As mentioned above, the 2D frame is centered on the first point indicated by the user's command. Furthermore, a distance between the user and the 2D frame is determined. Such a distance is the distance from the user's virtual position to the 2D frame. In particular, this distance is the distance between the user and the center of gravity of the 2D frame or the average distance between all points of the 2D frame and the user. In other words, the 2D frame is defined in the same coordinate system (e.g., the visualization coordinate system) as the user's viewing position. In the further examination process, the distance between the user and the 2D frame is fixed. This means that the 2D frame can be considered as a head-up display for the user, since it always has the same distance from the user, regardless of whether the user changes his position relative to the 3D rendering. This allows for a very intuitive navigation within the visualization environment, which is particularly advantageous when viewing large 3D medical image data. This is particularly useful in combination with d) above. IV) As mentioned above, the 2D frame is centered on the first point indicated by the user's command. The position of the 2D frame is then fixed in the X, Y, and Z coordinates of the visualization coordinate system. In other words, the 3D rendering can be moved, rotated, etc., but the 2D frame is fixed in the X, Y, and Z coordinates of the visualization coordinate system. In particular, the 3D rendering (defined by the object coordinate system) can be dragged through the 2D frame without affecting the position of the 2D frame.

[0030] The 2D frame has a polygonal outer contour (e.g., a rectangle, a cuboid, a triangle, etc.). Alternatively, the 2D frame has a rounded contour (e.g., a circle, an ellipse, etc.). Furthermore, the 2D frame has a predefined size. That is, the 2D frame encloses a predefined two-dimensional space. The predefined size is set manually by the user before indicating the command or automatically depending on the size of the 3D medical image data. For example, the size is set such that the 2D frame covers at least 1-20%, preferably 4-10%, of the area of ​​the 3D rendering in the initial visualization. The area of ​​the 3D rendering corresponds to the area of ​​the image plane on which the 3D medical image data is rendered. Alternatively, the initial size of the 2D frame is 1-20%, preferably 4-10%, of the area of ​​one side of a cuboid corresponding to the 3D medical image data. As a result, the workflow runs smoothly since the 2D frame is automatically adapted to the 3D medical image data.

[0031] In any case, the 2D frame is adaptive, i.e. the size and / or shape of the 2D frame is adjusted after it is provided in the visualization environment. For example, the user increases or decreases the size and / or shape of the 2D frame so that the region of interest is sufficiently covered by the 2D frame and the 3D rendering is sufficiently visible around the 2D frame. Thus, the configuration of the 2D frame is individually adapted to the preferences of each individual user.

[0032] Furthermore, the 2D frame may have a virtual handle. The handle is a virtual control panel that is also shown in the visualization environment. In particular, the user can change further settings of the 2D frame by manipulating the virtual control panel. For example, the visualization properties of the MPR view are set via the virtual control panel (e.g., brightness, contrast, etc.). Using the handle, the user changes the position of the 2D frame in the visualization environment. In other words, the user virtually grasps the handle and freely moves the 2D frame in any of the three dimensions in the visualization environment. For example, the user moves the 2D frame forward / backward, up / down, and left / right through the 3D rendering. In addition, the user additionally or alternatively moves the 2D frame using all three rotational degrees of freedom.

[0033] The 2D frame faces the user so that the user sees what is shown in the 2D frame (e.g., the MPR view). That is, the 2D frame is not oriented such that the user sees only one edge of the 2D frame. Preferably, the 2D frame is automatically oriented so that it faces the user when it is set up. When the 2D frame is set up, the initial orientation of the 2D frame is always the same. The initial orientation is pre-determined by each user. That is, some users individually set their own orientation of the 2D frame before starting the analysis process. Thus, the method is individualized according to the preferences of each user. The orientation of the 2D frame is then prepared according to the user performing the analysis process. Furthermore, this orientation is adjusted by the user in real time while the user is analyzing the 3D medical image data. This allows the user to set the field of view to best suit their personal preferences. For example, initially, one rotational degree of freedom (i.e., roll) is fixed or can only be moved in a range between 0° and 5°. That is, the 2D frame is movable in two rotational degrees of freedom (i.e., yaw and pitch). Thus, usability is improved since the user has a better overview when the 2D frame does not move randomly. Workflow is more efficient, especially when the 2D frame has a non-circular shape.

[0034] The MPR view is displayed in a 2D frame. The MPR view is derived directly from the 3D medical image data at a specific position of the 2D frame. That is, if the 2D frame is moved relative to the 3D rendering (and / or relative to the 3D medical image data), the MPR view is updated based on the 3D medical image data. The MPR view displays all the gray values ​​(or the interpolated gray values) of the planes cut in the 3D medical image data with the position and orientation of the 2D frame, and is seen as the "ground truth" of the 3D medical image data. That is, in the MPR view, no information is lost due to various display adjustments, such as the threshold of the 3D rendering. As a result, the MPR view corresponds exactly to the 3D medical image data acquired from the patient. As a result, the measurements made in the MPR view are based on the best information available. The MPR view is generated by multiplanar reconstruction, which is a process of transforming the 3D medical image data from one anatomical plane (usually a transverse plane) to another (orthogonal or oblique) plane, usually using an interpolation of pixel values, preferably a nearest neighbor interpolation. It is used for thin slices as well as projections. For example, MPR views are well suited to show bronchial anatomy, since they are not orthogonal to the direction of scanning. Furthermore, curved MPR views are used for evaluating blood vessels. This type of reconstruction helps to straighten vascular bends, thereby helping to visualize the entire vessel with one or multiple MPR views. After the vessel is "straightened", measurements such as cross-sectional area, length are taken. It is very useful in pre-operative evaluation of surgical procedures. Moreover, MPR views are opaque, i.e., the user cannot see what is behind the MPR view.

[0035] In other words, the MPR view is used to view a particular part of the 3D medical image data in more detail. The MPR view includes said anatomical features, but preferably does not include much more than the anatomical features of interest, i.e. the MPR view is as small as possible and just large enough to surround the features of interest. In other words, the position, size and / or shape of the MPR view (i.e. also of the 2D frame), in case of a sequence of 3D image data, is preferably adapted to match as closely as possible the position, size, orientation and / or shape of the anatomical features of interest throughout the entire sequence of images. In particular, the MPR view occupies a smaller viewing area (e.g. 2-20%) compared to the 3D rendering. That is, a person looking at the viewing area sees both the MPR view and the 3D rendering at the same time. For example, the 3D rendering is a representation of an anatomical structure and the anatomical features of interest are part of or adjacent to said anatomical structure. As an example, the MPR view covers less than 1 / 5 of the 3D rendering in the user's viewing direction. As a result, the user has an overview of the anatomy and knows where exactly the MPR view is positioned, thus preventing misplacement of the MPR view prior to measurements within the MPR view.

[0036] The MPR view is based on 3D medical image data and therefore always contains all the information available regardless of the adjustment of the threshold or other parameters of the 3D rendering.

[0037] According to a useful embodiment, an input tool as described above allows the user to select a plane in the 3D rendering. The method then preferably comprises the step of displaying a multiplanar reconstruction (MPR) of at least one selected plane of the three-dimensional medical images of the sequence, in particular at a position in the three-dimensional visualization environment corresponding to the selected plane. By displaying the MPR view in addition to the 3D rendering, the user can see the anatomical structures in more detail. In the virtual reality environment, thanks to the 18 degrees of freedom (VR headset and two controllers), the correct positioning of the touchable / graspable MPR view in the 3D rendering can be verified very quickly, and measurements on the MPR view or in the volume-rendered part (i.e. in the 3D rendering) become more precise and more reliable.

[0038] A clipping body or a clipping body is a body that has a volume but is transparent so that the user can see through it. In other words, voxels within the clipping body are set to have a grayscale value of zero. In another embodiment, all voxels within the clipping body are excluded from the volume rendering and treated as not existing. In this case, even with a threshold of 0, the 3D rendering is not present within the clipping body. The clipping body is defined by an exterior surface that encloses the volume of the clipping body. That is, the part of the 3D rendering that is located within the volume is clipped (i.e., cut out). The position of the clipping body is determined relative to the position of the 2D frame. That is, wherever a 2D frame is provided (see above), the clipping body is also provided. Thus, the user can easily see the 2D frame (i.e., the MPR view shown in the 2D frame). If there are more than one 2D frames in the visualization environment, there are the same number of clipping bodies in the visualization environment. In other words, the clipping body partially cuts off the 3D rendering between a first side of the 2D frame and the user. That is, the cropping body forms a kind of tunnel through the 3D rendering to the 2D frame. The user sees the 2D frame through said tunnel. This allows the user to easily see the MPR view displayed in the 2D frame and furthermore to see the 3D rendering around the 2D frame. Furthermore, the cropping body adapts its size, shape and / or position to the position of the 2D frame. For example, if the 2D frame is moved, the cropping body is moved accordingly such that the user still has an unobstructed view in the 2D frame. Thus, if the 3D rendering is moved and / or rotated, the cropping body adapts such that the user has an unobstructed view in the 2D frame. That is, if the 2D frame is moved further into the 3D rendering, the cropping body expands its size to partially crop the 3D rendering on a first side such that the user has an unobstructed view in the 2D frame.The crop also adapts its size and shape to the size and shape of the 2D frame, respectively. Furthermore, the 3D rendering and crop are updated instantly (e.g., 60-90 updates / sec) based on the user's movement (i.e., changes in the user's viewing direction) and / or the size / shape / position of the 2D frame.

[0039] The present invention is particularly useful for viewing and analyzing a particular anatomical feature of interest that is usually part of an anatomical structure. Such a feature of interest is particularly contained in a volume of interest (VOI) that is smaller than the full line of sight of the view of the 3D rendering. It is a part of an organ that constitutes an anatomical structure, particularly a part that has a complex anatomical structure such as a heart valve. In a useful embodiment, the anatomical feature of interest is the mitral valve, the tricuspid valve, the aortic valve, or the pulmonary valve. In other embodiments, the anatomical feature of interest is a heart chamber, or another important blood vessel such as a coronary vessel, or another structure such as a tumor.

[0040] In order to allow the user to see and analyze anatomical features of interest, the visualization environment is provided for visualizing at least two different types of visualizations / representations, which are in the same coordinate system, i.e., they are displayed in the correct relative spatial position and orientation with respect to each other. The at least two visualized objects are: (i) 3D rendering of the 3D medical image data, whereby the user is given a detailed and unobstructed view of the entire anatomical structure contained in the 3D medical image data. In a useful embodiment, selected settings / parameters for volume rendering such as thresholding, smoothing, etc. are adjusted automatically and / or manually by the user. (ii) Secondly, MPR view shown in the 2D frame, which gives the user more detailed information of the 3D medical image data at a specific location of the 2D frame. For example, when analyzing the mitral valve by examining the MPR view in the 2D frame, the user can simultaneously follow the progress of the surrounding parts of the heart, such as the left ventricular outflow tract (LVOT). This is important when planning an interventional procedure such as valve replacement, e.g., transcatheter aortic valve implantation (TAVI) or transcatheter aortic valve replacement (TAVR) or mitral valve replacement, where the LVOT is not obstructed.

[0041] Preferably, the first side of the 2D frame faces the user so as to be at least substantially orthogonal to the user's line of sight. Substantially orthogonal is considered to be oriented at an angle between 80° and 100°, preferably between 88° and 92°, to the user's line of sight, i.e. to the line of sight of the 3D rendering. In this range, the 2D frame (and therefore the MPR view) looks convenient for the user. On the other hand, if the 2D frame is tilted at an angle outside the range defined above, the MPR view displayed in the 2D frame will not look correct due to the tilt of the MPR view. Furthermore, measurements made with such a tilted 2D frame will be rather inaccurate. The first range defined above provides in particular sufficient visibility of the MPR view for correctly positioning the 2D frame in the 3D rendering. The second range provides an advantageous orientation of the MPR view for accurate measurements. As mentioned above, the 2D frame automatically orients itself towards the user so as to always orient itself within the above range, regardless of whether the user changes the line of sight and / or the 2D frame is moved. This simplifies user experience since the user does not have to worry about correct alignment of the 2D frame, but functions in a way that is most suitable for the analysis of the 3D rendering. In addition, the user issues a command so that a predefined view is shown by the MPR view. In other words, upon receiving such a command, the 3D rendering is automatically rotated so that a predefined view is shown by the MPR view in the 2D frame. Such predefined views are the parasternal long axis view, the parasternal short axis view (i.e., a 90° rotation of the 3D rendering relative to the parasternal long axis view), the two-chamber view, the three-chamber view, and / or the four-chamber view.

[0042] According to one embodiment, the cropper is positioned directly adjacent to the 2D frame so as to crop the 3D rendering between the first side of the 2D frame and the user. In this case, any part of the 3D rendering positioned between the 2D frame and the user is cropped (cut off). Thus, the entire MPR view is visible to the user. In other words, no part of the 3D rendering obstructs the direct viewing of the entire MPR view in the 2D frame. This is particularly useful when the MPR view is to be analyzed in great detail. For example, measurements or other analysis methods are easily performed in the MPR view. Thus, such operations are not hindered by the part of the 3D rendering between the first side of the 2D frame and the user.

[0043] According to an alternative embodiment, the crop is not directly adjacent to the 2D frame, and the ratio between the shortest distance from the crop to the 2D frame and the length of the perimeter of the 2D frame is between 0.1 and 0.9, preferably between 0.1 and 0.6, and most preferably between 0.1 and 0.4. In other words, the crop does not extend to the 2D frame and is therefore not directly adjacent to the 2D frame. That is, a part of the 3D rendering is located between the first side of the 2D frame and the user. For example, the part of the 3D rendering extends through the MPR view. For simplicity, the MPR view is considered as a base and a part of the 3D rendering is placed from that base to the beginning of the crop. The extent of such placement is defined by the distance between the 2D frame and the crop. In this case, the user sees the MPR view and a part of the 3D rendering at the location of the MPR view. The above defined extent defines the extension of the 3D rendering from the first side of the 2D frame towards the user with respect to the perimeter of the 2D frame. In other words, the larger the perimeter, the larger the expansion of the 3D rendering on the first side. This is useful because if the user creates a large 2D frame, the user will also see a larger part of the 3D representation than if the user creates a much smaller 2D frame. As a result, the expansion of the 3D rendering adapts to the size of the 2D frame set by the user.

[0044] The 0.1 to 0.9 range is particularly useful for positioning the 2D frame. Because part of the 3D rendering extends through the MPR view, the user easily orients himself to correctly place the 2D frame in the visualization environment. Because the MPR view is opaque, the visibility of part of the 3D rendering on the first side of the 2D frame helps the user know what is behind the 2D frame (i.e., on the second side of the 2D frame). As a result, less movement of the 2D frame back and forth is required to correctly position it.

[0045] In addition, the range of 0.1-0.6 is particularly useful for checking whether the additional model provided matches the anatomical structure. The model is a surface model representing a medical device such as an artificial heart valve, a pacemaker, or other device intended to be inserted into the human or animal body. The model has handles, which allow the user to adapt the model or parts of the model to the anatomical structure. It is helpful that the 3D rendering is partially visible on the first side of the 2D frame.

[0046] The range of 0.1 to 0.4 is particularly useful for calibrating the threshold of the 3D rendering for the 3D medical image data. It should be noted that the visualization of the 3D rendering depends on the threshold setting. Therefore, the MPR view is used to calibrate the 3D rendering to find the threshold setting that best matches the 3D medical image data. In the above mentioned range, the extension of the 3D rendering through the MPR view is only slight. This allows the user to adjust the 3D rendering that builds from the first side of the 2D frame to the contour represented by the MPR view. That is, the threshold of the 3D rendering is changed until the 3D rendering that rises from the first side of the 2D frame matches the contour of the anatomical structure shown in the MPR view. The threshold that best represents the 3D medical image data is easily set accordingly. According to another embodiment, the distance between the cutout and the 2D frame is variably adjusted. Thus, each user individually adjusts how far the 3D rendering extends on the first side of the 2D frame.

[0047] The method further includes a step of receiving a user command to resize a 2D frame in the visualization environment, wherein the MPR view in the 2D frame is updated to the new size of the 2D frame and the size of the transparent clipping body is adapted to the new size of the 2D frame.

[0048] Preferably, the method further comprises a step of receiving a user's command to change the position of the 2D frame in the visualization environment, where the MPR view in the 2D frame is updated to the new position of the 2D frame and the position of the transparent cutout is adapted to the new position of the 2D frame. The user's command is similar to the above-mentioned command for providing a 2D frame in the visualization environment. For example, the 2D frame has a virtual handle, and the 2D frame is grasped by the virtual handle and moved through the visualization environment. Furthermore, the user inputs the coordinates where the 2D frame should be positioned. Such input is realized by using a keyboard or any other input medium. In addition, the 2D image is rotated and / or scaled. The MPR view shown in the 2D frame is automatically updated based on the new position and / or size of the 2D frame relative to the 3D medical image data. The updated MPR view is still based on the 3D medical image data. Therefore, the transfer of inaccuracies of the 3D rendering to the MPR view is avoided. That is, at least one 2D frame has a fixed orientation facing the user, regardless of the user's gaze direction, and the MPR view in the 2D frame is automatically updated when the 3D rendering is moved relative to the 2D frame or vice versa.

[0049] Preferably, the 2D frame is provided on an analysis surface that extends in two dimensions through the 3D rendering, the 2D frame covering only a portion of the analysis surface. The analysis surface is transparent so that the user's view is not obstructed by the analysis surface. The 2D frame is above or parallel to the analysis surface. Since the 2D frame covers only a portion of the 3D rendering, the 2D frame is smaller than the analysis surface.

[0050] According to one embodiment, the transparent crop body has an adaptable cross-section, the method further comprising receiving a user's command indicating a ratio at which the size of the cross-section changes with increasing distance from the 2D frame. As described above, the crop body defines a volume extending between the 2D frame and the user. Wherever the volume of the crop body is present, the 3D rendering is cropped. Moreover, the crop body additionally crops out any elements displayed in the visualization environment between the 2D frame and the user, such as surface models, etc. Since the crop body is transparent, the user sees through it the 2D frame and thus the MPR view displayed in the 2D frame. The crop body is defined by a number of cross-sections stacked on top of each other to form the crop body, each cross-section being parallel to the 2D frame. In one embodiment, the crop body has a constant cross-section, i.e. each cross-section has the same dimensions. In other words, the crop body has a cylindrical shape. In another embodiment, the cross-sections have different dimensions. In this way, by varying the cross-section, crop bodies of different shapes are realized. According to one embodiment of the present invention, the cross-section is adaptable to change the shape of the crop body during the execution of the examination. That is, the user issues commands to change the shape and / or dimensions of the cutting body. As a default, all sections of the cutting body have the same dimensions. In this case, the distal section closest to the 2D frame and the proximal section closest to the user's viewing position have the same dimensions. However, any other configuration of sections may be set as a default. For example, the cutting body has a funnel shape, i.e. the distal section is the smallest section and each section towards the proximal section is larger than the previous one. This is also possible the other way around in the opposite direction, the proximal section being the smallest. Thus, the cutting body is adapted to the user's preferences and the type of examination to be performed. For example, for spatial orientation, it is advantageous to have a cutting body as small as possible (to see as much of the 3D rendering as possible, but also to see the MPR view).

[0051] Moreover, the distal section has the same shape and dimensions as the 2D frame. It is therefore guaranteed that the 2D frame is easily visible to the user through the cutout body. Furthermore, the cutout body is changed in shape and then returned to its initial position. The change of the cutout body allows, for example, the cutout body to be given a movement that continuously changes its shape and continuously returns to its initial position. Thus, the user gets a very good spatial impression of the entire anatomical structure. In addition, the cutout body continuously increases in size until everything is cut out from the 3D rendering of the first side of the 2D frame. In this case, the user faces the entire surface of the analysis plane. This is advantageous for taking measurements of the anatomical structure or for obtaining a 2D view of the anatomical structure.

[0052] Preferably, the method further comprises receiving a user's command to rotate and / or move the 3D rendering so that it has a new position and / or orientation in the visualization environment, where the MPR view in the 2D frame is updated corresponding to the new position of the 3D rendering and / or the shape and / or size of the cutout is updated corresponding to the new position of the 3D rendering. That is, the 3D rendering is moved and / or rotated by the user. This is necessary to inspect posterior anatomical features of the anatomy. Furthermore, by moving the 3D rendering, the 3D perception of the user is improved. As mentioned above, the 2D frame is provided in the visualization environment according to one of the options I) to IV) (see overview above). If the 2D frame is not fixed to the 3D rendering (as in case II), the 2D frame stays in its position while the 3D rendering is moved and / or rotated. However, the MPR view in the 2D frame is updated when the 3D rendering is moved and / or rotated relative to the 2D frame. Similarly, the cropping body is updated. That is, due to the movement and / or rotation of the 3D rendering, a larger part of the 3D rendering on the first side (i.e., between the user and the 2D frame) needs to be cut out to provide visibility of the MPR view. Therefore, the cropping body is adapted in shape and / or size to cut out a larger part of the 3D rendering on the first side of the 2D frame. As a result, a smooth workflow is achieved in which the user always has a good view into the MPR view, regardless of the position of the 3D rendering relative to the 2D frame. In addition, the MPR view is constantly updated based on the 3D medical image (i.e., based on the ground truth).

[0053] Preferably, the method comprises the steps of: providing a second 2D frame at a position in the visualization environment corresponding to the second point, a first side of the second 2D frame facing the user, the second 2D frame preferably being at least substantially orthogonal to a line of sight direction of the user; displaying a second MPR view in a second 2D frame, the second MPR view being based on the 3D medical image information at a location of the second 2D frame and sized to correspond to a size of the second 2D frame; providing a second transparent cutout between the user and the first side of the second 2D frame for cutting off a portion of the 3D rendering; The second 2D frame and / or the second crop body have the same properties as the first 2D frame and the first crop body described above. This also applies to the publication of the second 2D frame and / or the second crop body. Furthermore, further 2D frames are added to the visualization environment. For example, three, four, five and six 2D frames are provided.

[0054] Further 2D frames are useful for analysing and comparing two small features of an anatomical structure that are separated from one another, where an overview of the anatomical structure is guaranteed and at the same time the features are examined in great detail.

[0055] Preferably, the method further comprises receiving a user's command to individually modify the shape and / or size of at least one 2D frame, i.e. each 2D frame is adapted individually (i.e. the shape and / or size of the 2D frame is modified). Thus, the 2D frame is adapted to the size of the respective feature under inspection. For example, the shape of the cross section is modified from circular to polygonal and vice versa. Further, a third, fourth and fifth frame are provided. In other words, a plurality of 2D frames are provided having the same characteristics as the first 2D frame and / or a plurality of 2D frames are provided similar to the first 2D frame.

[0056] Preferably, the method comprises the steps of: receiving a user command indicating a measurement along a measurement path beginning at a first point set by the user, where a 2D frame surrounds the first point and the measurement path, dynamically adapting a size and / or shape of the 2D frame to the measurement path; receiving a user command indicating where the measurement path is to end; Thus, the user enters a measurement mode. In this mode, the user issues a command to indicate a first point to provide a first 2D frame. The measurement is started at the first point. The user then indicates a second point to which the distance should be measured. In this way, the diameter of a heart valve or blood vessel is measured. For example, the user presses a button on the pointing device to indicate the location of the first point. The user then moves the pointing device to the second point while still pressing the button. At the same time as the second point is reached, the user releases the button to indicate the location of the second point.

[0057] Further, the 2D frame is provided at the first position as described above. In addition, the 2D frame extends itself to the second point to cover the measurement path. As a result, the 2D frame is created after the measurement, covering the first point and the second point, as well as the space between the first point and the second point. In this case, the user looks at the MPR view in the 2D frame to better find the second point (e.g., the boundary of a vessel or cavity). As a result, the quality of the measurement is improved.

[0058] If a single point is set, the 2D frame defined by this point still rotates around all three rotational degrees of freedom (roll, pitch, yaw). As soon as a second point is defined for the 2D frame, two of these degrees of freedom are removed and the 2D frame area rotates only around one axis. If a third point is set, the 2D frame is rigid and the degrees of freedom do not increase, i.e. only a transparent cutout adapts to the rotation and / or translation of the 3D rendering or to changes in the user's viewing position to be able to freely view this rigid 2D frame (i.e. MPR view) at any time.

[0059] Alternatively, two 2D frames are provided, one at the first point (start of measurement) and one at the second point (target of measurement). In this case, in the middle of the two points, the 3D rendering is still visible. Therefore, the user has a spatially full understanding of the space between the points. When two points are set by the user, the shortest distance between these two points is automatically measured each time. The first point used to provide the 2D frame of the first point is the same as the first point used as the starting point for the measurement. Alternatively, these points are separate points that can be specified separately by the user. Preferably, the visualization environment is a VR or AR environment. According to a useful embodiment, the three-dimensional visualization environment is a virtual reality environment. By "virtual reality" is meant a computer-generated visualization that provides a true three-dimensional experience of the depicted structure. Thus, the virtual reality (VR) environment of the present invention provides in particular visual feedback, but also allows other types of sensory feedback, such as hearing. A VR environment is also an augmented reality environment, where the user still sees a real environment, but VR objects (e.g., volume renderings and dynamic models) are overlaid or superimposed on real objects, or mixed reality, where real-world objects are superimposed on a virtual scene. The volume-rendered VOI and the visualization of the dynamic model together form a visualized object, preferably a virtual reality object.

[0060] Virtual reality environments are generally achieved by presenting a user with stereoscopic images, i.e., each eye sees a different image, so that the brain assembles the two different images into a true three-dimensional scene. Such binocular images are presented on a VR display, such as a virtual reality headset or multiple projection environment, or on a screen that intermittently shows the two images in conjunction with shutter glasses.

[0061] In a VR environment, the volume-rendered VOI or the dynamic model is displayed by stereoscopic rendering, where a volume-rendered (or otherwise rendered) visualization / image is calculated twice for two viewing positions with a small spatial offset, i.e., one viewing position for the left eye and one viewing position for the right eye. When the two such calculated visualizations are shown to a user, for example, one for each eye in a VR headset, the user gets a true three-dimensional (VR) impression. Thereby, the volume-rendered VOI and the dynamic surface models can be transformed, viewed and analyzed in VR.

[0062] In a preferred embodiment, a person using the VR environment of the present invention can "look around" the artificial world, moving around in it and interacting with virtual objects, features or items. This effect is typically created by a VR headset that includes a head-mounted display with a small screen in front of each eye, but also by specially designed rooms with many large screens. In order for a user to move around in a VR environment, position and orientation information must be sent by the headset to the electronic device that generates the VR environment (e.g., a computer or processing unit in the headset) so that the visualization is moving in line with the user's head movements. In order for the user to interact with virtual features in the VR environment, hand movements must also be tracked, which is done by handheld VR controllers. However, this last feature is optional, as is sending out position / orientation information for the user to walk around in the virtual scene.

[0063] A virtual reality environment offers the advantage that the user sees and analyzes visualized objects with great confidence, since the user gets a true three-dimensional view of the anatomical structures. Moreover, as the user walks around and possibly even steps into the visualized object, the user has the visualized object (e.g. a visualization of a human heart) displayed at a very large magnification so as to completely fill the space in front of the user. Hence, the user gets a particularly good overview and makes measurements with high accuracy. Moreover, the handling of user input events is particularly easy and intuitive in a VR environment. Actions such as rotating an object and / or adjusting the settings of a volume-rendered VOI, which are rather tricky on a two-dimensional screen, are very intuitive and fast in a VR environment using a VR controller.

[0064] However, the present invention may also be advantageously used in non-VR visualized environments, and unless VR (Virtual Reality) is explicitly mentioned, the features and embodiments described herein are useful in both VR and non-VR visualized environments.

[0065] Preferably, the method further comprises providing a clipping plane extending through the entire 3D rendering and receiving a user's command to clip the 3D rendering on one side of the clipping plane, the clipping plane having an individual orientation and position within the visualization environment. On one side of the clipping plane, at least the 3D rendering is clipped. In addition, other models or visualizations in the visualization environment are clipped on one side of the clipping plane. In other words, the clipping plane represents a 2D boundary in the visualization environment, on one side of the clipping plane, an object such as the 3D rendering is mapped, while on the other side, at least the 3D rendering is clipped or cut out. The clipping plane has an extension in two dimensions throughout the visualization environment. Thus, it is guaranteed that all parts of the 3D rendering are clipped, even if the boundary of the 3D rendering is not precisely determined. Alternatively, the clipping plane simply extends throughout the entire 3D rendering. This ensures that other visualizations in the visualization environment are still visible to the user. The cropping plane is freely moved based on the user's commands. That is, in contrast to a 2D frame facing the user, the cropping plane is freely positioned and / or oriented within the visualization environment. For example, the user views a 3D rendering in a first direction and partially crops the 3D rendering in a second direction orthogonal to the first direction. As a result, the user reduces the 3D rendering to an efficient size required for the examination. This further simplifies the examination and ensures clear visualization of important parts of the 3D medical image data.

[0066] Preferably, the method comprises the steps of: automatically segmenting a surface model from the 3D medical image data, the surface model corresponding to at least a portion of an anatomical structure; displaying the surface model within a visualization environment; receiving a user command to fit the surface model to an MPR view in the 2D frame; For example, the surface model is automatically segmented from the 3D medical image data. That is, based on the 3D medical image data, the ventricles of the heart are at least partially represented by the surface model. The surface model is then used for further examination or planning work. Therefore, it is preferable that the surface model represents the 3D medical image data as best as possible. One way to check whether the surface model matches the 3D medical image data is to compare the surface model with a 3D rendering. However, due to the rendering process, which determines the 3D rendering depending on which threshold is set, there are inaccuracies in the 3D rendering. On the other hand, using only the MPR view is complicated because spatial orientation is not easy for the user. According to this embodiment, the surface model is checked by using both the 3D rendering and the MPR view (shown in a 2D frame). For example, the user issues a command to display the surface model in addition to the 3D rendering and the 2D frame in the visualization environment. The user then positions the 2D frame so that the MPR view is at least partially on the surface model. As a result, the user checks whether the surface model corresponds to each feature (e.g., boundary) shown by the MPR view. Since the 3D rendering is visible around the 2D frame, the user can easily orient himself / herself. Thus, the surface model is checked in an efficient and accurate manner. According to another embodiment, the user adjusts the surface model to perfectly match the 3D medical image data. For example, the user moves or adjusts the surface model to match the 3D medical image data. In this context, for example, control points that define the shape and size of the surface model are moved.

[0067] According to another aspect, the invention provides a computer program comprising program code instructions which, when executed by a processor, enable the processor to carry out the method of the invention. The computer program may be any code, in particular code suitable for computer graphic applications, in particular VR programming.

[0068] In a further aspect, the present invention is directed to a computer readable medium comprising a computer program as defined above. The computer readable medium is any digital data storage device, such as a USB stick, a hard disk, a CD-ROM, an SD card or an SSD card. Of course, the computer program does not have to be stored on such a computer readable medium to be provided to the customer, but may be downloadable via the internet.

[0069] According to another aspect, the invention provides a 3D medical image data evaluation device adapted to carry out the method of the invention, the evaluation device comprising: an interface for receiving 3D medical image information; a computing unit for carrying out the method of the present invention; a visualization device for providing a visualization environment and displaying at least 3D renderings and 2D frames; Includes.

[0070] All embodiments described herein are applicable to both "traditional" visualization environments realized on computer screens, tablet computers, or displays, as well as to VR environments. However, VR environments are particularly advantageous as they offer a true 3D view and the most intuitive / fast user experience / operation, as well as 6, 12, or 18 degrees of freedom that a user can move relative to visualized objects.

[0071] Preferably, the method according to the invention is performed by a processor integrated in an electronic device capable of controlling a display, in particular a VR display such as a VR headset, or a projection display. Such digital devices are computers, PCs, servers, television sets, tablet computers, smartphones, laptops, handheld devices, etc. The processor is also part of a cloud computer, a workstation, or a control console of a medical imaging device, in particular an ultrasound scanner.

[0072] Individual features of the above-described embodiments may be combined with other embodiments or with other features of other embodiments to form new embodiments. The advantages stated with respect to the individual features also apply to such new embodiments. Furthermore, advantages and features stated with respect to the method also apply to the device and vice versa.

[0073] Useful embodiments of the present invention will now be described with reference to the accompanying drawings, in which like elements or features are designated with the same reference numerals, and in which: [Brief description of the drawings]

[0074] [Figure 1] FIG. 1 shows a schematic visualization of the prior art. [Diagram 2] FIG. 1 shows a schematic visualization of the prior art. [Diagram 3] FIG. 2 is a schematic diagram illustrating 3D rendering and MPR views according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of a measurement process according to one embodiment of the present invention. [Diagram 5] FIG. 4 is a schematic diagram of a measurement process according to another embodiment of the present invention. [Figure 6] FIG. 2 is a schematic diagram of an analysis process according to one embodiment of the present invention. [Figure 7] FIG. 2 is a schematic diagram of an analysis process according to another embodiment of the present invention. [Figure 8] FIG. 2 is a schematic diagram of an analysis process according to another embodiment of the present invention. [Figure 9] FIG. 2 is a schematic diagram of an analysis process according to another embodiment of the present invention. [Figure 10A] FIG. 1 is a schematic diagram of a 3D rendering and analysis process using different thresholds, according to an embodiment of the present invention; [Figure 10B] FIG. 1 is a schematic diagram of a 3D rendering and analysis process using different thresholds according to an embodiment of the present invention. [Figure 11] FIG. 2 is a schematic diagram of a calibration process according to an embodiment of the present invention. [Figure 12] FIG. 2 is a schematic diagram of an adaptation process according to one embodiment of the present invention. [Figure 13] 1 is a schematic diagram of a 3D medical image data evaluation device according to an embodiment of the present invention; [Figure 14] 2 is a schematic diagram of a 3D medical image data evaluation device according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0075] Fig. 1 illustrates a state-of-the-art visualization method. In particular, Fig. 1 illustrates a 3D rendering 101 and an MPR view 100 that extends through the 3D rendering 101. In front of the MPR view 100, the 3D rendering is invisible to a user looking at the MPR view 100. Behind the MPR view 100, the 3D rendering is also invisible due to the mandated MPR view 100. Thus, the user can hardly use the 3D rendering 101 to orient himself / herself.

[0076] 2 also shows a state-of-the-art visualization method. In this case, the 3D rendering 101 is not cut off in front of the MPR view 100. However, the 3D rendering 101 obstructs a clear view of the MPR view 100. In this case, the user can orient himself / herself towards the 3D rendering 101 but can barely see the MPR view 100.

[0077] Fig. 3 illustrates a visualization method according to an embodiment of the present invention. In particular, in Fig. 3 a 3D rendering 1 in a visualization environment 10 is shown. Within the visualization environment 10 a 2D frame 2 is provided in which an MPR view 3 is shown. Fig. 3 corresponds to a user's viewpoint. The 2D frame 2 is provided to face the user. In other words, the 2D frame 2 is placed and oriented within the visualization environment such that the user always sees the MPR view 3 in the 2D frame 2. The 2D frame 2 and / or the 3D rendering 1 are moved and / or rotated relative to each other. Nevertheless, the 2D frame 2 automatically orients itself to face the user. In addition, the user may change the viewing direction, and even in this case the 2D frame 2 orients itself to face the user and the user can see the MPR view 3. Furthermore, whenever the 2D frame 2 changes its position relative to the 3D rendering 1, the MPR view 3 shown in the 2D frame 2 is updated based on the new relative position. It should be noted that the MPR view is determined based on the 3D medical image data, thereby representing the ground truth of the 3D medical image data. In this embodiment, the visualization environment 10 is a VR environment and the user controls the method using at least one controller 11. Furthermore, in this embodiment, the 2D frame 2 has a rectangular shape. However, the 2D frame 2 can also have other shapes that are best suited to the examination to be performed. Moreover, by grasping the 2D frame 2, the user adjusts the size of the 2D frame 2. That is, the 2D frame 2 is enlarged or reduced according to the user's preference. A cropper 8 (see FIG. 7) is provided that at least partially crops the 3D rendering on a first side of the 2D frame 2 so as to always have an unobstructed view of the MPR view 3. As a result, the user can see the MPR view 3 regardless of how deep the 2D frame 2 is moved into the 3D rendering 1. In addition, three coordinate systems are shown in FIG. 3. The visualization coordinate system defines the coordinates X, Y and Z in which the viewing position of the user and the user's controller 11 is defined.Then an object coordinate system X', Y' and Z' is defined in which the 3D rendering 1 and the 2D frame 2 are defined. Finally, an original coordinate system X", Y" and Z" is provided in which the 3D medical image data is defined. In this embodiment the coordinate systems are rotated and / or translated with respect to each other. For ease of explanation, in further figures the coordinate systems are not shown, but are present as well.

[0078] Fig. 4 shows four sub-figures illustrating a visualization method according to another embodiment of the invention. In particular, in Fig. 4.1, the first point 4 set by the user is furthermore the starting point of the measurement. That is to say, the first point 4 is the starting point of the measurement path 5. In addition, a 2D frame 2 is provided in the visualization environment 10 to show the first point 4 in the MPR view 3. The user then extends the measurement path 5 (see Fig. 4.2). The 2D frame 2, and therefore the MPR view 3, are adapted according to the extension of the measurement path 5. Then, in Fig. 4.3, the boundary of the vessel to be measured is visible in the MPR view 3. Hence, the user provides the second point 6 exactly on the boundary. In this way, the measurement path extends between the first point 4 and the second point 6. In Fig. 4.4, the completed measurement path 5 is shown by the MPR view 3. The user then adjusts the 2D frame 2 individually. For example, the user may zoom in or out of the 2D frame to get a better impression of the vicinity of the measurement path 5 .

[0079] Fig. 5 shows a visualization method according to another embodiment of the invention. The method of Fig. 5 differs from the method of Fig. 4 in that the 2D frame 2 is not extended along the measurement path, but there is a first 2D frame 2 provided at a first point 4 and a second 2D frame 7 provided at a second point 6. The first 2D frame 2 shows a first MPR view 3 and the second 2D frame 7 shows a second MPR view 8. In this way, the measurement path 5 extends through the 3D rendering 1. In this case, the user still sets the first and second points precisely by seeing their positions in the first 2D frame 2 and the second 2D frame 7, but in addition has an improved impression of the spatial extension of the 3D rendering 1, since the user sees the spatial extension of the 3D rendering 1 between the first point 4 and the second point 6. It should be noted that the second 2D frame 7 has the same properties as the first 2D frame. In addition, the second point 6 is issued by the user in the same way as the first point 4.

[0080] FIG. 6 illustrates a further analysis method according to an embodiment of the present invention. As mentioned above, in this embodiment, the visualization environment is a VR environment. The user uses the controller 11 to control commands. In this embodiment, the user enters an analysis mode in which a 2D frame is provided in the visualization environment. For ease of explanation, the 2D frame 2 is considered as a table tennis racket. In other words, the 2D frame 2 has a handle with which the user virtually grips the 2D frame 2. The 2D frame 2 is thus fixed to the controller 11, so that the 2D frame 2 is easily moved and / or rotated by simply moving the controller 11. In this way, the user virtually moves the controller 11 through the 3D rendering 1, for example to explore a particular feature. As mentioned above, the MPR view 3 is shown in the 2D frame 2 and is automatically adapted to the respective position of the 2D frame 2 relative to the 3D rendering 1.

[0081] FIG. 7 shows a schematic diagram of an analysis process according to another embodiment of the invention. The method of FIG. 7 substantially corresponds to the method shown in FIG. 3. In FIG. 7 the scene is shown in a side view. Thus, a cutout body 9 is seen. The cutout body 9 is provided on a first side of the 2D frame 2 so as to cut out the 3D rendering. Hence, the user 20 has an unobstructed view of the MPR view 3 in the 2D frame 2. In addition, in this embodiment, the 2D frame 2 is virtually fixed to the controller 11 of the user 20, as described above in relation to FIG. 6. As described above, the 2D frame is moved through the 3D rendering 1 to find features and to position the first measurement point 4. The cutout body has an adaptable shape. That is to say, the cross-section perpendicular to the line of sight of the user 20 and defining the cutout body 9 has an individual shape and size. In this embodiment, the cutout body 9 is sized and shaped so that the user has an unobstructed view of the MPR view 3 in the 2D frame 2. That is, the crop volume adapts to the new position of the 2D frame 2 relative to the 3D rendering 1. In summary, both the crop volume 9 and the MPR view 3 adapt to the new position of the 2D frame 2 to ensure visibility of the MPR view 3 to the user 20.

[0082] FIG. 8 basically corresponds to FIG. 7, but with the difference that a second 2D frame 7 is provided showing a second MPR view 8. A second cutout 12 is therefore further provided on a first side of the second 2D frame 7. As mentioned above, each 2D frame 2, 7 is oriented to face the user. Each cutout 9, 12 is therefore also oriented (i.e. sized and shaped) to face the user 20. In other words, each cutout is provided such that the user 20 has an unobstructed view of the respective MPR view 3, 8. The cutouts 9, 12 therefore have different sizes and shapes when the two 2D frames 2, 7 are positioned in different spatial positions in the visualization environment.

[0083] Figure 9 basically corresponds to Figure 7, with the difference that the 2D frame 2 has another shape as in the embodiment of Figure 3. In addition, the cutout body 9 has a tapered shape and the cross section of the cutout body 9 closest to the user 20 has a larger size compared to the cross section of the cutout body 9 closest to the 2D frame 2. Thus, the user 20 sees the 3D rendering 1 at the wall of the cutout body 9. This provides the skilled person with further information to better understand the anatomical structure shown by the 3D rendering.

[0084] 10A and 10B are schematic illustrations of the 3D rendering 1 and the analysis process with different thresholds according to an embodiment of the present invention. In FIG. 10A, the 3D rendering 1 as described above with a threshold of zero is shown, i.e. all voxels are visible in the 3D rendering 1. Thus, the anatomical structures are hardly visible to the user 10 in the 3D rendering. Nevertheless, the MPR views 3, 8 shown in the 2D frames 2, 7 are visible because the cutouts 9, 12 cut out the 3D rendering 1. In addition, the first point 4, the second point 6 and the measurement path 5 are visible in FIG. 10A and 10B. In FIG. 10B, the threshold is set to a value of 89 / 225, i.e. all voxels with values ​​higher than 89 and lower than 225 are rendered to generate the 3D rendering 1. As a result, the anatomical structures are better estimated. This example is intended to show how sensitive the 3D rendering 1 is to the threshold setting. On the other hand, the MPR views 3, 8 represent the ground truth of the 3D medical image data and are not dependent on any settings that would deteriorate the meaningfulness of the MPR views 3, 8. Therefore, the inventive combination of both display formats shows immense accuracy and operator convenience.

[0085] Moreover, according to a further embodiment of the invention, the accuracy of the MPR views 3, 8 is used to calibrate the 3D rendering 1. In FIG. 11 the 3D rendering 1 is shown. Furthermore, a 2D frame 2 is provided in which the MPR view 3 is shown. In contrast to the previously described embodiment, in this embodiment the cutout body (not shown in FIG. 11) does not reach up to the 2D frame 2. On the other hand, the cutout body 8 allows the 3D rendering to reach partially through the 2D frame 2 and thus through the MPR view 3. In other words, the cutout body is spaced a predefined distance from the 2D frame. This ensures that the user 20 sees the MPR view 3 and the part of the 3D rendering 1 that is allowed to reach from a first side of the 2D frame towards the user (i.e. this depends on the distance of the cutout body to the first side of the cutout body).

[0086] In this embodiment, the user sees the MPR view 3 as a black and white representation of the 3D medical image data of the first side of the 2D frame. In addition, the user sees a 3D rendering (i.e., the 3D rendering 1 extending through the 2D frame) that is assembled from the first side of the 2D frame. The outer contour of the intersection of the 3D rendering 1 with the MPR view is used to calibrate the threshold of the 3D rendering based on the MPR view. In other words, the threshold is adapted such that said outer contour of the intersection matches the contour shown in the MPR view. As a result, the threshold is determined such that the 3D rendering 1 correctly represents the anatomical structure. This further improves the accuracy of the analysis of the 3D medical image data.

[0087] In Fig. 12 a further embodiment of the invention is shown diagrammatically. This embodiment substantially corresponds to the embodiment of Fig. 7, with the difference that in this embodiment a surface model 13 is shown in the visualization environment 10. The surface model 13 is generated automatically based on 3D medical image data. The surface model 13 has been generated to correspond to the features of the anatomical structure. Using the MPR view 3, a user 20 checks that the surface model 13 indeed corresponds correctly to said features.

[0088] 13 shows a user interface according to a non-VR embodiment of the invention. In this setup, the visualization environment 10 is on a conventional computer screen 23 and the visualization is simply a rendering on the screen 23 in two dimensions. The screen includes a panel of buttons and sliders 24 that allow the user to tilt, zoom, move or otherwise manipulate the 3D rendering 1, the 2D frame 2, etc. Furthermore, in such a user interface, having the 3D rendering and MPR view 3 is a useful tool. The display is controlled by a computer 25, such as a PC, which includes a processor 26 and a hard disk 27. The user interface has input tools such as a keyboard 28 and / or a mouse 29.

[0089] However, in a preferred embodiment, the user interface is a virtual reality interface as shown in Fig. 14. Such an interface is realized by a virtual reality headset 21 worn by a user 20. The headset 21 is connected to a computer 25 via a cable or wireless connection. Such a virtual reality headset 21 includes separate internal displays for each eye as well as a position sensor 22 for tracking the head movements. Such a headset further includes a camera if an augmented reality environment is to be presented. Furthermore, the user 20 holds in his hands a VR controller 11, which further includes a position sensor (not shown) as well as buttons or other input elements. Such a virtual reality controller 11 allows the user to grab and move objects displayed in the visualization environment. The VR headset is for example an HTC VIVE headset and a corresponding VR controller.

[0090] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustrations and description are to be considered illustrative or exemplary and not descriptive, and the invention is not limited to the disclosed embodiments. [Explanation of symbols]

[0091] 1. 3D Rendering 2, 7 2D Frame 3, 8 MPR visualization 4. First Point 5 Measurement Path 6. Second point 9, 12 Cutouts 11 Controller 13 Surface Model 20 users 21 Headset 22 Position Sensor 23 Computer Screen 24 Panel 25 Computers 26 processors 27 Hard Disk 28 Keyboard 29 Mouse

Claims

1. A method for analyzing 3D medical image data, comprising: receiving 3D medical image data of an anatomical structure; generating a 3D rendering of the anatomical structure based on the 3D medical image data; displaying the 3D rendering to a user in a visualization environment; receiving a user command indicating a first point on or within the 3D rendering; providing a 2D frame at a position in the visualization environment corresponding to the first point, wherein a first side of the 2D frame faces the user; displaying a MPR view within the 2D frame, wherein the MPR view is based on the 3D medical image data at the position of the 2D frame and is sized to correspond to the size of the 2D frame; providing a transparent cut-off body between the user and the first side of the 2D frame to partially cut off the 3D rendering; receiving a user command to change the size of the 2D frame in the visualization environment while the 3D rendering is displayed around the 2D frame, wherein the MPR view within the 2D frame is updated to the new size of the 2D frame and the size of the transparent cut-off body is adapted to the new size of the 2D frame. A method as described above.

2. The method according to claim 1, wherein the first side of the 2D frame faces the user such that it is at least substantially orthogonal to the user's line of sight.

3. The cut-off body is positioned directly adjacent to the 2D frame to cut off the 3D rendering between the first side of the 2D frame and the user, or the cut-off body is not directly adjacent to the 2D frame, and the ratio between the shortest distance from the cut-off body to the 2D frame and the outer perimeter of the 2D frame is between 0.1 and 0.9, preferably between 0.1 and 0.6, and most preferably between 0.1 and 0.

4. The method according to claim 1 or 2.

4. A step of receiving a user command to change the position of the 2D frame in the visualization environment, wherein the MPR view within the 2D frame is updated to a new position of the 2D frame, and the position of the transparent cut body is adapted to the new position of the 2D frame, the method according to any one of claims 1 to 3 further comprising the step of receiving the command.

5. The method according to any one of claims 1 to 4, wherein the 2D frame is provided on an analysis plane that extends two-dimensionally through the 3D rendering, and the 2D frame covers only a part of the analysis plane.

6. The method according to any one of claims 1 to 5, wherein the transparent cut body has an adaptable cross-section, and the method further comprises a step of receiving a user command indicating a ratio in which the size of the cross-section changes with an increase in the distance from the 2D frame.

7. The method is A step of receiving a user command to rotate and / or move the 3D rendering so that the 3D rendering has a new position and / or orientation within the visualization environment, wherein the MPR view within the 2D frame is updated corresponding to the new position of the 3D rendering, and / or the shape and / or size of the cut body is updated corresponding to the new position of the 3D rendering, the step of receiving the command The method according to any one of claims 1 to 6 further comprising.

8. Receiving a user command indicating a second point on or within the 3D rendering; Providing a second 2D frame at a position in the visualization environment corresponding to the second point, wherein a first side of the second 2D frame faces the user, and the second 2D frame is preferably at least substantially orthogonal to the user's line of sight direction, the step of providing the second 2D frame; Displaying a second MPR view within the second 2D frame, wherein the second MPR view is based on the 3D medical image data at the position of the second 2D frame and is sized to correspond to the size of the second 2D frame, the step of displaying the second MPR view; providing a second transparent clipping body between the user and the first side of the second 2D frame to partially cut off the 3D rendering The method according to any one of claims 1 to 7, further comprising

9. The method according to any one of claims 1 to 8, further comprising receiving a user command to individually change the shape and / or size of at least one of the 2D frames.

10. Receiving a user command indicating a measurement along a measurement path starting at the first point set by the user, wherein the 2D frame surrounds the first point and the measurement path, and dynamically adapting the size and / or shape of the 2D frame to the measurement path, receiving the command; receiving a user command indicating the position where the measurement path ends The method according to any one of claims 1 to 9, further comprising

11. The method according to any one of claims 1 to 10, wherein the visualization environment is a VR environment or an AR environment.

12. The method according to any one of claims 1 to 11, further comprising providing a clipping plane extending through the entire 3D rendering and receiving a user command to cut off the 3D rendering on one side of the clipping plane, wherein the clipping plane has individual orientations and positions within the visualization environment.

13. The method comprises automatically segmenting a surface model from the 3D medical image information, the surface model corresponding to at least a part of the anatomical structure, segmenting the surface model; displaying the surface model within the visualization environment; receiving a user command to adapt the surface model to an MPR view within the 2D frame The method according to any one of claims 1 to 12, further comprising

14. A computer program comprising program code instructions that, when executed by a processor, enable the processor to execute the method according to any one of claims 1 to 13.

15. A 3D medical image information evaluation device for executing the method according to any one of claims 1 to 13, wherein the 3D medical image information evaluation device An interface for receiving 3D medical image information, a computing unit for executing the method according to any one of claims 1 to 13, a visualization device for providing the visualization environment and displaying at least the 3D rendering and the 2D frames, A 3D medical image information evaluation device comprising:

Images