Method, computer program, user interface, and system for analyzing medical image data in virtual multi-user collaboration

Abstract

A method for analyzing medical image data 34 in virtual multi-user collaboration is provided, the medical image data 34 being analyzed by at least two users A, N, C, S, each having their own workspace 30, the workspace being VR and / or AR and / or MR, the method comprising the steps of: providing medical image data 34 including 3D or 4D image information; loading the medical image data 34 into each user's workspace 30 and simultaneously displaying a visualization of the medical image data 34 to each user; enabling each user individually and independently of each other to modify the visualization of the medical image data 34 to obtain an individual visualization of the medical image data 34 in each workspace 30 of each user; enabling at least one user to perform an analysis process on the medical image data 34 in their own workspace; displaying a result of the analysis process in the workspace 30 in which the analysis process was performed; and synchronizing the result of the analysis process with at least one other workspace 30 in real time, so that each workspace 30 displays the result of the analysis process in its own individual visualization of the medical image data 34. Also provided is a computer program relating to the above method. Additionally, a user interface and a system for use during the execution of the above method are provided.

Description

[Technical Field]

[0001] The present invention relates to a method for analyzing medical image data in a virtual multi-user collaboration, a computer program relating to said method, a user interface for use in said method, and a system configured to carry out said method. [Background technology]

[0002] Medical imaging technologies provide three-dimensional (3D) and sometimes four-dimensional (4D) medical image data of the human or animal body. However, image data is typically viewed and analyzed on a two-dimensional (2D) screen. Therefore, when medical image data is analyzed and interpreted on a 2D screen, there is always a risk of misinterpreting the relative spatial relationships between anatomical structures depicted in 3D or 4D medical image data. Specifically, when a team of experts simultaneously analyzes the same medical image data, there is a high risk of misunderstanding and misinterpreting the spatial relationships, resulting in incorrect clinical decisions.

[0003] Recent developments in the fields of microprocessors, cameras, and sensors have enabled affordable consumer VR headsets, which have been widely adopted by the mass market. Virtual reality (VR) technology provides users with complete immersion in a virtual environment and a stereoscopic 3D representation of 3D content.

[0004] Highly intuitive control via tracked headsets and hand controllers (with a combined 18 degrees of freedom), primarily driven by the gaming and entertainment industries, allows users to interact with virtual objects in a very natural way, enhancing convenience and ease of use. Existing multi-user VR applications are avatar-based, allowing multiple users to simultaneously occupy the same scene, for example, manipulating an object model. Unfortunately, these avatar-based multi-user solutions do not adequately meet the more stringent requirements of specialized medical VR collaboration, which requires meaningful and time-efficient evaluation of virtual 3D datasets, objects (e.g., cross-sections), and measurements. Furthermore, known techniques appear inadequate for expert users (e.g., clinicians) who need to analyze and evaluate 3D content independently. Therefore, a multi-user VR collaboration solution is needed that can fully utilize the potential of VR and enable multiple clinicians to work efficiently and simultaneously on the same 3D dataset with all the benefits the VR environment offers. Furthermore, to increase the collaboration efficiency of expert user teams, it would be desirable to automatically share the results of each user's analysis and evaluation bidirectionally.

[0005] David W. Shattuck, in his paper "VR-framework for multiuser virtual reality environment for visualizing neuroimaging data," Healthcare Technology Letters, received August 13, 2018, accepted August 20, 2018, describes how a VR framework can support multiple simultaneous users operating in the same virtual space by utilizing a client-server model in which one user acts as the server and manages the system display. Each user's VR environment is driven by a separate network computer connected to the user's headset. Each client computer may have access to a copy of the data to be displayed, located on a local drive or a shared network drive. The server listens for clients on a TCP / IP port and establishes a network connection with each client. Once connected, the server periodically sends a small data object containing view state information, which each client uses to update its display. This object contains data necessary to synchronize the view, such as rotation, scale, volume position, and cut plane. Each client sends pose information for its headset and controllers to the server, which broadcasts this information to other clients for display. Each client view renders models of the headsets and controllers of other users in the system, allowing users to interact directly with each other and helping to avoid real-world collisions between users working in a shared physical space.

[0006] In "Texture-based Volume Visualization for Multiple Users on the World Wide Web," Klaus Engel describes a texture-based volume visualization tool that enables remote access to radiology data and supports a multi-user environment. This tool enables shared viewing and manipulation of three-dimensional medical volume datasets across heterogeneous networks. The volume dataset is transmitted from a server to multiple different client machines and visualized locally using a Java-enabled web browser. Data reduction and compression schemes are proposed to reduce network traffic. The application enables interactively movable viewpoint-dependent and orthogonal clipping planes. On the client side, users can join a visualization session and synchronize viewpoints and other visualization parameters to achieve the same view of the volume dataset. Interesting parts of the dataset can be marked for other users by placing tags in the visualization. To support collaboration, users communicate using a chat applet or by using existing videoconferencing tools.

[0007] In "Bridging Multiple User Interface Dimensions with Augmented Reality," Dieter Schmalstieg describes an experimental user interface system that uses collaborative augmented reality to incorporate true 3D interaction into productivity environments. This concept is extended to bridging multiple user interface dimensions by including multiple users, multiple host platforms, multiple display types, multiple concurrent applications, and multi-context (i.e., 3D document) interfaces in a heterogeneous, distributed environment. Contexts encapsulate live applications with 3D (visual) and other data, while locales are used to organize geometric reference systems. Separating geometric relationships (locales) from semantic relationships (contexts) allows for great flexibility in display configuration. Multiple users work on separate hosts. Multiple users can share a context but arbitrarily layout the context representation (3D window) according to screen format and personal preferences. This is made possible by defining separate locales, since the position of the 3D window is not shared across locale boundaries. In other words, multiple users can view a shared object from different perspectives.

[0008] EP 3496046 A1 relates to a method for displaying medical image data on at least one display medium for a group of at least two interactive observers, the method comprising the steps of providing medical image data comprising at least one 3D or 4D image dataset of a particular examination area of ​​a patient, and providing the possibility to display the medical image data in an interactive virtual environment, each interactive observer having their own virtual position and being able to change their virtual position and optionally their viewing angle independently of the other observers.

[0009] US2014 / 0033052A1 discloses a method and system for displaying a hologram. One exemplary embodiment provides a system including a light source, an image generating unit that generates an image through interaction of light approaching the image generating unit from the light source, an eyepiece, and a mirror that directs light from the image onto the surface of the eyepiece. The surface of the eyepiece has the shape of a solid of revolution formed by rotating a plane curve at least 180° around an axis of rotation. As a result, a hologram is generated that can be viewed from various sides. Furthermore, the hologram can be manipulated by a user.

[0010] However, the above problem is not solved, and collaboration efficiency is not significantly improved because users must actively share their analysis results based on the commands of one or more users. As a result, it takes repetitive work steps to provide users with relevant information obtained by other users. Summary of the Invention [Problem to be solved by the invention]

[0011] Therefore, the present invention aims to provide a method for analyzing medical image data in virtual multi-user collaboration, in which each user has full control over the visualization of the medical image data in their own workspace, while at the same time the efficiency of the collaboration remains high, i.e., a team of expert users can work simultaneously on the same medical image data. According to another object of the present invention, selectable parts of individual works can be automatically shared between users. The present invention also aims to provide respective computer programs, user interfaces configured to be used to perform the above method, and a system for virtual collaboration for interacting with 3D medical image data, which still allows users to have full control over the visualization of the medical image data. [Means for solving the problem]

[0012] In order to better address one or more of the above concerns, in a first aspect of the present invention a method for analysing medical image data is presented in claim 1. Useful embodiments are described in the dependent claims.

[0013] According to this first aspect, the present invention is directed to a method for analyzing medical image data in a virtual multi-user collaboration, comprising: The medical imaging data is analyzed by at least two users, Each user has their own workspace, which is an XR workspace. The method is: providing medical image data including 3D or 4D image information; loading the medical image data into each user's workspace and simultaneously displaying a visualization of the medical image data to each user; - enabling each user individually and independently of each other to modify the visualization of the medical image data to obtain an individual visualization of the medical image data in each workspace of each user; enabling at least one user to perform an analysis process of medical image data in his / her workspace; displaying the results of the analytical process in the workspace in which the analytical process was executed; and synchronizing the results of the analytical process with at least one other workspace in real time, so that each workspace displays the results of the analytical process in a respective individual visualization of the medical image data.

[0014] In the context of the present invention, the term XR refers to X-reality or cross-reality, a collective term that includes at least virtual reality (VR), augmented reality (AR), mixed reality (MR), and cinematic reality (CR). XR is defined as the experience of a virtual environment, often combined with real-world objects, or a combination of the real world and a virtual environment or object, implemented by technology. X-reality encompasses a variety of hardware and software, such as sensory interfaces, applications, and infrastructure, that enable content creation for virtual reality (VR), mixed reality (MR), augmented reality (AR), and cinematic reality (CR). Users use these tools to create new forms of reality by bringing digital objects into the physical world and physical objects into the digital world. In this specification, XR refers to various technologies such as VR, AR, or MR. To provide X-reality, computer-generated visualizations that provide a true three-dimensional experience of the depicted structure are often achieved, particularly through the use of screens or glasses that display slightly different images to each eye. Additionally, in many cases, a user may interact with an XR through gestures or other actions, such as walking around the XR or grabbing virtual objects. While the XR of the present invention may provide particularly visual feedback, other types of sensory feedback, such as auditory or haptic feedback, may also be provided to the user. Preferably, the XR of the present invention uses only the user's gestures (e.g., gestures of the user's fingers, arms, legs, and / or head) or movements as input to provide the user with sensory feedback. Notably, the present invention may not require an eye tracker to provide the user with such feedback. That is, the visualization of the present invention may be shown to the user from multiple different viewpoints without the need for such an eye tracker.

[0015] In VR, the real environment is typically invisible to the user and is completely obscured by the virtual world. This effect is usually created by a VR headset, which consists of a head-mounted display with a small screen placed in front of the eyes, but can also be created by a specially designed room with multiple larger screens. People using virtual reality equipment can look around the artificial world, move around within it, and interact with virtual features or items.

[0016] Generally, mixed reality (MR) is a new environment and visualization that blends the real and virtual worlds, allowing physical and digital objects to coexist and interact in real time. More specifically, MR is defined as the experience of a real environment combined with virtual objects created by technology; for example, a VR headset uses its cameras to create a virtual reality that at least partially corresponds to the real environment. Thus, virtual objects are overlaid on top of the real environment, possibly obscuring or obscuring real objects. In AR, the real world remains visible, with virtual objects overlaid (i.e., superimposed) on top of it. This effect can be created by specialized glasses, such as Microsoft HoloLens, that allow the user to see the real environment but use cameras to create a 3D model of the real environment so that virtual objects can be overlaid on top of it via the glasses.

[0017] Thus, MR or AR may include the real world as a backdrop for the displayed visualization. For example, two users, each wearing AR glasses, can be in the same room, each with their own individual view of the medical image data (i.e., each user with their own visualization of the medical image data), and can interact with each other in the real world and in AR.

[0018] The visualization can be a representation (i.e., an image) of a real object (e.g., a human or animal heart). That is, the visualization is a model of the real object, and the parameters of the visualization can be modified relative to the real object (e.g., size, contrast, color, partially enlarged areas, etc.). To visualize medical image data, the technique of volume ray casting can be used. In this technique, a ray is generated for each desired pixel. Using a simple camera model, the ray starts from the camera's center of projection (usually the viewpoint or eyepoint) and passes through a pixel on an imaginary plane floating between the camera and the volume being rendered. The ray is sampled throughout the volume at regular or adjustable intervals. The data is interpolated at each sample point, and a transfer function is applied to form an RGBA sample. The result is added to the ray's cumulative RGBA; this process is repeated until the ray exits the volume. This process is repeated for all pixels on the screen to form the complete visualization.

[0019] For example, medical image data may be visualized using volume rendering. Volume rendering of visualizations may be performed by techniques described in "The Visualization Handbook," edited by Charles H. Hansen and Christopher R. Johnson, Elsevier Butterworth Heinemann (2005), in particular the chapter "Overview of Volume Rendering" by Arie Kaufmann beginning on page 127, which is incorporated herein by reference.

[0020] Technically, XR is achieved by presenting a user with stereoscopic images—that is, different images are displayed for each eye, and the brain combines the two different images to create a true three-dimensional scene. Such binocular images can be displayed on any XR display, for example, a VR headset, AR glasses, or a screen that displays two images intermittently, such as in a multi-projection environment or with shutter glasses. In XR, visualizations can be displayed using stereoscopic rendering. A visualization is calculated twice for two viewpoints with a slight spatial offset: one for the left eye and one for the right eye. When these two visualizations are displayed for the user's left and right eyes, respectively, in a VR headset, for example, the user gets the impression of true three-dimensionality (VR). This allows visualizations to be transformed, observed, and analyzed in XR. XR allows users to look around an artificial world, move around within it, and interact with virtual objects, features, or items. This effect is typically created by an XR headset with a head-mounted display that has small screens placed in front of each eye, but can also be created by a specially designed room with multiple large screens. For a user to move around in an XR, the headset must send position and orientation information to an XR-generating electronic device (e.g., a computer) so that the visualization follows the user's head movements.

[0021] Medical image data can be dynamically rendered (e.g., volume rendering or surface rendering) to be visualized as a visualization in each XR workspace. More specifically, volume rendering can be based on spatial intensity data (i.e., voxel data). Depending on the resolution of the medical image data, intensity data can exist for each point in space or for each region in space. In other words, volume rendering renders each available image data piece and requires significantly more computational power. On the other hand, surface rendering renders only one "layer" (i.e., the visible surface) and does not render image data that exists behind the layer. Surface-rendered medical image data can be a computer graphical model consisting of multiple triangular surfaces. As a result, surface rendering requires less computational power. To increase efficiency during the rendering process while providing sufficient visualization data density, both techniques can be combined, with the region of interest being rendered using volume rendering and the surrounding region being rendered using surface rendering.

[0022] In a preferred embodiment, the medical image data is visualized using a volume-rendered object, which has the advantage of being suitable for more complex anatomical structures or highly individual structures, such as valve leaflets, stenoses, calcifications, bioprostheses, or ruptured chordae. Alternatively or additionally, the medical image data may be visualized using a dynamic computer-generated model of at least a portion of the anatomical structure. Such a model has the advantages of representing a simpler version / abstraction of the anatomical structure, facilitating navigation and interpretation of the anatomical structure, and being less dependent on the quality of the medical image data. The model (i.e., a simplified version of the anatomical structure) may be a triangular mesh surface model of a specific interface within the anatomical structure, such as the blood-tissue interface of a blood vessel or heart chamber. The model may include multiple points that span a line or surface in each frame. The model may also be a mathematical model, e.g., a parameterized model such as a surface spanned by a spline curve or a volume. The model is dynamic, i.e., it follows the movement of the anatomical structure over time. The purpose of a dynamic model is to visualize at least a portion of an anatomical structure, such as one or more chambers of a moving heart, without obscuring the user's view with excessive detail. Therefore, such simplified models are useful for providing guidance to a user, for example, when planning an intervention or taking measurements on a particular portion of the anatomical structure. The 3D visualization representation of a dynamic 3D model is typically a rendering of a dynamic shape or surface model, which may be performed by computer graphics-enabled techniques such as shading, ray casting, and ambient occlusion. Volume rendering may be performed by any volume rendering technique known in the art, such as those described in US 2005 / 0253841 A1, which is incorporated herein by reference.

[0023] Medical image data also includes information about anatomical structures, which may be organs or parts of organs in a human or animal body, such as the heart, blood vessels, or bone structures. More specifically, medical image data includes a 3D scatter plot containing points in a 3D coordinate system, where each point has its own x, y, and z components within the 3D coordinate system. In addition to the above, medical image data may also include digital image data, such as DICOM-standard digital image data, i.e., digital image data containing a three-dimensional array of voxels, each voxel having a grayscale value. Such medical data (3D medical images) may be acquired from a field of view containing dynamic anatomical structures using medical imaging modalities such as MR, computed tomography (CT), positron emission tomography (PET), or ultrasound (US). When the anatomical structure is the heart, ultrasound, particularly three-dimensional echocardiography, may be preferably used. That is, different medical image data may be derived from ultrasound images with different frequencies, computed tomography images with different acceleration voltages, or images with or without contrast agents. Furthermore, the medical image data may include 4D medical image data, where the fourth dimension is time. Hereinafter, one 3D image from a time series of 3D images forming a 4D medical image may also be referred to as a "frame." That is, the 4D medical image data includes visualizations of 3D images over time, i.e., a sequence of visualizations is dynamically displayed, e.g., at a frame rate of 60-100 visualizations / second. That is, the 4D medical image data may be visualized as a moving image (i.e., in cine mode), e.g., to visualize the movement of a patient's chest. When the 4D medical image data is visualized, the sequence may be rendered like a movie. To enable smooth depiction of dynamically moving anatomical structures, the 3D images may be acquired at a frame rate of, e.g., 5-100 images / second, preferably greater than 30 images / second. The medical image data may include additional medical and / or administrative information, e.g., patient information, current treatment, etc. The medical image data may include multiple image data, which are preferably spatially registered to each other so that they can be superimposed on one another.Each user can select the image data to be displayed in their workspace. Thus, medical image data can be considered shared content that can be shared by all users. The shared content can include raw data, e.g., digital image data, and / or a 3D scatter plot including multiple points in a 3D coordinate system. Additionally or alternatively, the shared content can include preprocessed data, such as slices, segmentations, or surface models based on the medical image data. Such preprocessed data can be considered model data. Thus, the visualizations in each workspace can include a visualization of the raw data (i.e., 3D data) and a visualization of the model data (i.e., preprocessed data). By combining the model data and raw data as shared content, the amount of raw data processed in each workspace is typically less, enabling collaboration using a standard PC without requiring significant computing power. In other words, the computationally intensive calculations required to generate preprocessed data can be performed centrally, and the results can be provided to each user as shared content. Providing shared content in each workspace can provide users with a complete picture of the issues being discussed during collaboration. For example, raw data may be used to visualize the most important organ or part of an organ (i.e., the region of interest), and model data may be used to visualize the area surrounding the region of interest. For example, raw data may be used to visualize a heart valve, and model data may be used to visualize the rest of the heart. As a result, optimal efficiency of the visualization can be achieved. Also, each user may individually switch off / on the model data to see only the visualization of the raw data. As a result, each user can individually decide whether or not they need both visualizations to analyze medical image data.

[0024] A medical imaging modality may be capable of acquiring up to 30 volumes (i.e., medical image data representing an area under examination) per second, and thus the rendered 3D object may be updated each time new medical image data becomes available. In other words, the rendered 3D object may be dynamically updated. This is particularly useful when a patient is being examined simultaneously while collaboration is taking place, for example, when the patient is in an emergency condition.

[0025] It should be noted that each user shares the same medical image data with all other users. Furthermore, since the results of the analysis process are combined with the medical image data shared by all users, the results of the analysis process are also shared by all other users. Meanwhile, the visualization displayed in each workspace can be individually modified by each user to obtain an individual view of the medical image data. In other words, according to the present invention, the medical image data and the results of the analysis process are the same and shared by all users, and the display of this content (i.e., the visualization of the medical image data and the results) is left to the user. As a result, according to the present invention, each user has maximum freedom in examining the medical image data as they wish, while the results of other users' analysis processes are displayed in real time within their own personalized visualization (i.e., within their own individual views of the medical image data). Therefore, the medical image data and the results of the analysis process are separated from the visualization of the medical image data in each workspace. In other words, while each user has their own individual view of the medical image data, the medical image data and the results are the same for all users.

[0026] A user is a participant in a collaboration session and may specifically be a physician. For example, an interdisciplinary cardiac team (IHT) may use the present invention, and the team may consist of an interventional cardiologist, a cardiovascular surgeon, a care / nurse coordinator, an OR / catheterization lab nurse, an imaging specialist, and a cardiac anesthesiologist. Each user uses their own XR workspace. Alternatively, at least two users may share the same workspace, for example, in cases where one user is a teacher and another user is a student.

[0027] Each XR workspace may be physically connected to another XR workspace via a cable or wireless connection. Each workspace may have its own personal computer including a processor. Each computer may function as a client computer. Alternatively, at least some workspaces may be part of a single client computer. That is, multiple XR workspaces may be located in different locations or may be located in the same room, for example.

[0028] The step of enabling each user to individually and independently modify the visualization in their XR workspace is performed so that other users cannot see how the visualization (i.e., individual views of medical image data) has been modified in another XR workspace. Specifically, the viewpoint position and viewing direction in space may be changed. Also, the position or orientation of a plane cutting the 3D medical image data (referred to herein as a "cutting plane") and the mode in which such a plane is displayed in the visualization may be individually adjusted. For example, a user may select one of several so-called "cutting plane" modes described below. In one mode, the cutting plane is not displayed, while in another mode, the cutting plane cuts the 3D image volume and is overlaid on the image content. In another mode, the cutting plane displays a corresponding multiplanar reconstruction (MPR) of the 3D image data. Furthermore, the opacity and color of the visualization may be changed. In general, visualization parameters of the visualization, such as volume rendering parameters, may be modified. Specifically, the visualization parameters may include thresholds, opacity, contrast, etc. Furthermore, the visualization can be adjusted "live," that is, instantly while viewing the visualization.

[0029] According to one aspect of the present invention, the results of the analytical process are synchronized in real time with at least one other workspace, and each workspace displays the results of the analytical process in its own individual visualization of the medical image data. Because the results of the analytical process belong to the medical image data shared by all users (see also the general description above), synchronization in this case means sending the data obtained in the analytical process from each workspace to the shared medical image data. That is, the results of the analytical process are displayed to all other users as soon as they are created. In other words, immediately after the analytical process is performed, each other user can see the results in their own visualization of the medical image data. For example, each user can see other users' annotations live in their own visualization of the medical image data. This eliminates the "handoff" of medical image data between multiple users, resulting in less interaction between users and faster sharing of measurement results and annotations.

[0030] According to one embodiment, no indications are shown in the workspace other than visualizations that indicate the location and / or gaze direction of other users, e.g., by displaying each user's "avatar," so that portions of the visualizations are not obscured and / or obstructed by such indications.

[0031] For example, performing the analysis process may include selecting a plane within the visualization. In that case, the process preferably includes displaying a multiplanar reconstruction (MPR) of the selected plane of the visualization, particularly at a location within the three-dimensional visualization environment corresponding to the selected plane. A multiplanar reconstruction is an image reconstructed from several original image planes. For example, in CT, a stack of transverse images is typically acquired. Also, during the analysis process, a user may measure the diameter of the mitral valve and select the most appropriate valve from a library based on the measurement. Thus, when displaying a cross-section that intersects a stack of images in a different orientation from the transverse direction, the user selects the desired orientation, and the MPR is created, for example, by interpolating from the nearest pixels in the various transverse slices. Displaying the MPR in addition to the visualization allows the user to view the anatomical structures in greater detail. In a virtual reality environment, thanks to the 18 degrees of freedom (XR headset and two controllers), the graspable MPR plane can be positioned very quickly within the 3D volume, the positioning can be verified, and measurements on the MPR plane or within the visualization become more accurate and reliable. Additionally, an analytical process may be performed by each user individually in their own workspace. That is, the execution of the analytical process is not shared among other users. For example, the movement or trajectory of a real tool (e.g., a controller or another real object held by a user) or a virtual tool (e.g., a virtual tape measure or a visualization representation of an object used to perform the analytical process) within the workspace during the analytical process may be visualized only within the workspace in which the analytical process is performed. Thus, only the measurement results may be shared among all users. That is, during the analytical process, a user is not distracted or interrupted in performing his or her own individual analytical process by another user performing an analytical process. Additionally, an analytical process may be performed simultaneously by at least two users. Because the operations (i.e., substeps) performed during the analytical process are not shared among users, users do not interfere with each other's execution of the analytical process.For example, two or more users may simultaneously measure the diameter of the mitral valve of a human heart without disturbing each other.

[0032] Note that only the results of the analytical process are shared and visualized in each workspace, e.g., to notify each user that the results are available. Furthermore, each user can perform their own analytical process at their own pace and accuracy without immediately revealing their approach to analyzing the features of the visualization to all other users. For example, a user may try several different methods for analyzing their individual visualization before sharing the analysis results with all users. Thus, each user has the freedom to perform the analytical process in their own way without being observed by other users. As a result, each user's working atmosphere may improve, and the accuracy of the analytical process results may increase. For example, if a user who performed the analytical process is not satisfied with their performance, the user can perform the analytical process again without providing excessive information (i.e., inappropriate measurements) to other users. Preferably, a user can signal whether they have finished their analytical process to initiate synchronization of the analytical process results with all other users (i.e., workspaces). The signal may be performed by pressing a virtual button or by ending the analytical process. For example, the user may signal that the analysis process is finished by disabling a virtual tool used during the analysis process.

[0033] For example, the result of the analysis process may be at least one measurement of a feature of the visualization representation of the medical image data, such as a distance, a diameter, or a thickness. More specifically, the measurement may be displayed by a first dot (starting point) at a first location where the distance measurement should begin, a second dot (ending point) at a second location where the distance measurement should end, and a line connecting the first and second dots. Specifically, the result of the analysis process may be a three-dimensional annotation, such as a 3D line that can be drawn freehand by a user in their workspace. In other words, the result of the analysis process may not be a planar object, but a 3D object in a 3D coordinate system extending in any of three dimensions. For example, a user may retrace the flow of blood through the heart or trace the course of a coronary artery.

[0034] The first aspect of the present invention provides the advantage that each user can generate their own preferred visualization of medical image data, i.e., each user has their own individual view of the medical image data. Furthermore, each user can analyze and view the visualization of medical image data in their own way (e.g., individual viewing direction, cutting plane position, threshold, and speed), i.e., some users may require more time for certain procedures compared to other users. Because there are no avatars in the visualization, parts of the medical image data are not hidden behind such avatars. Furthermore, inexperienced users can follow the results created by other users in real time to better understand subjects that they do not understand.

[0035] Furthermore, XR offers the advantage that users can confidently view and analyze visualizations because they obtain a true three-dimensional view of the anatomical structures represented by the medical image data. Furthermore, because users can walk around and even enter the visualized object (e.g., a visualization of a human heart), the user can view the visualized object at a very large magnification, completely filling the space in front of the user. Thus, users have a particularly good overview and can perform measurements with high precision. Furthermore, handling user input events is particularly easy and intuitive in XR. Actions that can be quite tricky on a two-dimensional screen, such as rotating an object or adjusting visualization settings, can be very intuitive and quick in XR, preferably using an XR controller.

[0036] In one embodiment, the medical image data includes images, preferably 4D images, of the human heart.

[0037] The present invention may be particularly useful in planning minimally invasive cardiac procedures, such as heart valve surgery or heart valve replacement. New minimally invasive procedures, such as transcatheter valve replacement, can be used for patients previously considered inoperable or unsuitable for open-heart surgery. Some transcatheter valve replacement procedures (e.g., TAVR) use fully collapsible biological valves. However, the success of these interventions requires a thorough analysis and understanding of the existing pathology / geometry, as well as careful selection, sizing, and placement of the new valve to ensure proper function and avoid obstruction of the left ventricular outflow tract (LVOT) or coronary arteries. This is particularly true for valve-in-valve (ViV) interventions, whereby a dysfunctional valve (which may be a biological mitral valve) is replaced with a new valve using minimally invasive ViV. The new valve is positioned inside the old valve, replacing it as it deploys. Therefore, it is important that the valve is correctly positioned and sized. It is particularly important that the new mitral valve does not obstruct the LVOT. Therefore, for ViV intervention planning, the medical image data includes the mitral valve, the LVOT, and the left ventricle.

[0038] In a preferred embodiment of the present invention, the display of the results of the analytical process can be selectively and individually enabled and disabled by users in their workspaces.

[0039] Each result of the analysis process may be located at a specific location within the 3D coordinate system in which the medical image data is defined (i.e., the location may be defined by x, y, and z coordinates). That is, the result may be located at a specific location relative to the medical image data. Because the medical image data and the results are shared by all users, the results are displayed within each workspace regardless of how the users individually modify the visualization (i.e., individual views of the medical image data). To reduce the possibility that the results obscure other objects displayed in the workspace, the results may be visualized with relatively thin lines. Nevertheless, situations may arise in which the results obscure portions of the visualization and / or other results. Therefore, according to the above-described embodiment of the present invention, the results may be disabled so that they disappear, providing visibility to objects located behind the results.

[0040] Because each user may focus on different aspects of the region of interest, it is important for each user to be able to individually determine which results obstruct sufficient visibility into the region of interest and therefore should be disabled. That is, each user may selectively disable results created by themselves or other users. Also, enabling or disabling a result may not affect at least one other workspace, and enabling and disabling of results may be performed independently in each workspace. Furthermore, the method may allow a user to adjust the transparency of a result so that objects placed behind the result can be seen through the result. In particular, after a result is disabled, a user may re-enable the result and redisplay the result in their workspace. To this end, a disabled result may be indicated with a small arrow or the like to provide the user with information about where the result was originally located. As a result, a user can easily retrieve the result in order to re-enable it.

[0041] According to this embodiment, the visualization in each workspace can be further personalized in that each user can precisely customize both the visualization (i.e., individual view of medical image data) and the results to meet their own workability requirements.

[0042] In another preferred embodiment of the present invention, at least one workspace may be an AR workspace. That is, both the visualization and the results are displayed in the workspace, with the real environment visible as the background of the workspace. For example, the real environment may be the user's surrounding area (e.g., an operating room or a clinic). Furthermore, according to one embodiment, at least one visualization parameter in the AR workspace, in particular the transparency and / or color of the visualization representation of the medical image data and / or the results of the analysis process, may be automatically adjusted so that the user can see the visualization representation of the medical image data and / or the results of the analysis process superimposed on the real environment at a target contrast.

[0043] More specifically, visualization parameters can be values ​​that define how medical image data and / or results are visualized. Therefore, changing these values ​​changes the visualization accordingly. For example, visualization parameters can include transparency values, i.e., the transparency of an object. For example, if an object's transparency is low, the area covered by the object is invisible. On the other hand, if the object's transparency is high, the area covered by the object may be at least slightly visible. Visualization parameters can also include the color of the visualization, i.e., the hue or tone of the visualization. More specifically, hue is one of the primary characteristics of color (also known as color appearance parameters). Hue can usually be quantitatively represented by a single number (often corresponding to the angular position around the center or neutral point or axis of a color space coordinate diagram (e.g., a chromaticity diagram) or a color wheel) or by its dominant wavelength or the dominant wavelength of its complementary color. Other color appearance parameters are colorfulness, saturation (also known as intensity or chroma), lightness, and brightness. In this embodiment, among other things, these parameters may be automatically adjusted to achieve a target contrast within the workspace. The target contrast is also referred to as optimal contrast. The contrast within the workspace is defined as the difference between the brightest and darkest points within the workspace, and the real-world environment is also taken into account in the AR workspace. The contrast within the workspace may also be defined as a contrast ratio or a dynamic range. The visualization and / or results of the medical image data may be adjusted so that the contrast within the AR workspace is the target contrast. In other words, if the real-world environment (background) is relatively bright, the brightness of the visualization and / or results is also increased to reduce the overall contrast to the target contrast. The target contrast may be a predetermined value defined as being most appropriate in terms of detectability and conspicuity of the visualization. Alternatively, the target contrast may be individually adjusted and / or set by the user to meet the user's personal preferences.

[0044] According to one embodiment, each workspace may have its own virtual environment in which visualizations of medical image data and results of the analysis process are displayed. The virtual environment may be the background on which the visualizations are displayed. Furthermore, according to the above embodiment, the method may further include a step of allowing each user to individually and independently adjust the visualization parameters of the virtual environment and adjust the contrast within the workspace, preferably by setting the transparency and / or color of the virtual environment. The virtual environment may be the background in front of which the visualizations are placed. In other words, the virtual environment may be the periphery surrounding the visualizations. For example, when a VR workspace is used, the virtual environment may include colored surfaces with a predetermined color and brightness to provide a comfortable working environment for the user. That is, the virtual environment may be adjusted to provide a target contrast within the workspace. Furthermore, the user may individually adjust the virtual environment. Also, when an AR workspace is used, the real environment is also displayed as a background within the workspace. However, even an AR workspace may have a virtual environment with a certain transparency so that the user can see the real environment. However, the transparency value of the virtual environment may be adjustable to darken a very bright real environment so that the contrast within the workspace is the target contrast. This is particularly useful when the AR workspace is used in a relatively bright environment, such as an operating room. In other words, the virtual environment of the AR workspace allows the user to recognize the real environment while automatically adapting the AR workspace to the conditions of the real environment. Furthermore, the AR workspace can be easily adapted to various lighting conditions in the real environment. As a result, the AR workspace can be used appropriately regardless of the lighting conditions of the real environment.

[0045] Preferably, the virtual environment may include at least one virtual control element. For example, the virtual control element may be a slide bar and / or a button. The control element may be used to execute method steps such as changing the visualization and / or to perform other management operations such as saving images, loading medical image data, and communicating with other users. The control element may also provide a drop-down menu listing several predefined control commands.

[0046] In another preferred embodiment, the step of enabling each user to individually and independently modify the visualization of the medical image data may include the users using a VR controller to interact with virtual features in the workspace. Specifically, hand gestures are used to modify the visualization, preferably by grabbing objects in the workspace. In the XR workspace, a user wearing an XR headset and holding at least one controller in one hand (preferably one in each hand) can see the controller along with the visualization. The controller may be a handheld device including additional operating elements such as a button, a trackball, or a touchpad. When a VR workspace is used, a representation of the controller (e.g., a virtual controller) may be depicted in the VR workspace to inform the user where the controller is located in the VR workspace. Preferably, the user can also see the virtual controller in a position and orientation corresponding to the current hand position and orientation. Thus, the XR workspace allows the user to move the controller toward the visualization, grab it by pressing a specific button, and move and / or rotate the visualized object with hand movements, just as they would with a real-world object. This allows the user to have 18 degrees of freedom (6 degrees of freedom for each of the XR headset and the two controllers, i.e., 3 rotational and 3 translational degrees of freedom) to observe and analyze the visualized objects correctly and intuitively. This is very similar to normal interactions with objects. The movements of the controllers can be tracked and the corresponding movements of the display can be visualized in the VR workspace. On the other hand, when using the AR workspace, the real controllers may remain visible. However, even in the AR workspace, virtual controllers may be visualized to make the controllers more visible.

[0047] The position information may be acquired by at least one sensor, such as an acceleration sensor provided in the controller. More generally, the position and / or orientation of one or more controllers may be tracked by a camera on the VR headset. The data output by the sensor or camera may be input to a processor that processes the data and controls commands executed by manipulating the controller and the visualization of the virtual controller. Depending on the sensor output, the processor may also determine whether the gesture is slow or fast. Rotational movement may also be detected. As a result, the processor may perform corresponding processing.

[0048] Operations performed by the controller may include zooming in / out of the visualization, shrinking / enlarging the visualization in the workspace, adjusting visualization and rendering settings / parameters, and / or grabbing the visualization of the displayed object, in particular the medical image data.

[0049] Furthermore, the controller may receive hand gestures performed by the user, such as a grab gesture, a zoom-in gesture (two fingers moving closer together), a zoom-out gesture (two fingers moving apart), and a wipe gesture. For example, a visualization may be virtually grabbed, rotated, or moved using the controller. In an XR workspace, a real controller may be visualized as a virtual controller, allowing the user to at least grab and move objects in the workspace with hand gestures. "Move" or "move" in this context may mean rotation. The controller may also allow the user to change the size and / or shape of objects in the workspace, particularly the visualization. According to a useful embodiment, the controller allows the user to change any one or more of the position, size, shape, and / or orientation of the visualization. In particular, the position and / or orientation of the visualization (i.e., an individual view of medical image data) may be adjusted. However, in a useful embodiment, the size of the visualization may also be changed to better analyze the visualization. Preferably, such adjustments are made using the controller by grabbing the visualization with hand gestures and changing its position, size, and / or orientation, just as one would with a real-world object. According to one useful embodiment, the controller allows the user to adjust parameters through gesture control. For example, the user selects a particular parameter by touching it with a hand movement within the XR workspace. The user may then use gestures to manipulate, for example, a virtual slider, or to adjust a parameter without reference to a slider by moving the controller horizontally (or vertically). Suitable parameters may be associated with the visualization and may be selected from volume rendering thresholds, smoothing, lighting intensity, size, opacity of visualized objects, and starting / stopping cine mode, etc.

[0050] As a result, users can intuitively perform operations within the AR / VR workspace, such as moving objects or manipulating control elements, thereby increasing the efficiency of such collaboration.

[0051] Also, in a useful embodiment, the workspace may include a lamp that the user can grab and move within the workspace to affect the lighting of the visualization. In a useful embodiment, the brightness of the scene, and in particular the brightness of the movable lamp, may be adjusted. Also, by adjusting the position of the lamp, the direction of illumination of the visualization may be changed. This is particularly useful for illuminating cavities in the visualized anatomical structure.

[0052] Preferably, the step of enabling each user to individually and independently modify the visualization representation of the medical image data includes manipulating the visualization representation to rotate the visualization representation, crop a portion of the visualization representation, change rendering parameters of the visualization representation, change image settings of the visualization representation, change the contrast of the visualization representation, change the voxel intensity threshold of the visualization representation, and / or change the size of the visualization representation. In other words, the visualization representation is manipulated, for example, rotated, if the user wants to view a different portion or area of ​​the visualization representation. That is, each user has their own individual view of the medical image data. Similarly, each user may individually crop a portion of the visualization representation to obtain a visualization representation that displays only the portion of the visualization representation that is of interest to that user. The rendering parameters of the visualization representation may include the type of rendering (i.e., volume rendering and / or surface rendering), and the image settings may include surface color and shading options (i.e., the position of the light source in space). Users may also configure the settings of their individual visualization representations so that the medical image data is visualized as slices or in other formats. There may also be multiple different visualization presets in which the medical image data is displayed in a plan view, a right-hand front view, a left-hand front view, a back view, a bottom view, an oblique view, etc. The presets may be preset or may be predefined by each user.

[0053] Preferably, at least one user may adopt changes to the visualization of the medical image data made by another user. That is, by adopting another user's visualization (i.e., individual view of the medical image data), the visualization is sent to the workspace of at least one other user, so that at least two users have the same individual view of the medical image data. This process is particularly useful when an experienced user (e.g., a teacher) is currently explaining something to other users (e.g., students). As a result, the other users may be able to learn from the more experienced user. In other words, the same individual view of the medical image data as the experienced user is provided to one or more other users. To show the user how the individual view has changed, the switch from one individual view to another may be performed continuously. Alternatively, the switch between individual views may be shown by visualizing the switch as a bird's-eye view, i.e., the change between the individual views is shown as a top view.

[0054] Alternatively, a user may force at least one other user to adopt his / her modifications of the visualization of medical image data. That is, the user can make other users see exactly what the user sees (i.e., have the same individual view), which may be particularly suitable when a user wants other users to see a specific part of a workflow. Thus, collaboration efficiency is improved. In another preferred embodiment, only one user can decide which modifications of the visualization of medical image data can be shared with other users (i.e., the user may be a presenter and / or teacher), while other users may have limited capabilities and may not be allowed to share modifications of the visualization of medical image data (e.g., the user may be a participant and / or student).

[0055] In another preferred embodiment, the step of enabling at least one user to perform an analysis process on the medical image data in their workspace may further include taking 3D measurements, performing MPR mode, and / or inserting annotations. That is, the results of the analysis process may be one or more annotations regarding a particular aspect of the visualization, such as background information on a previously performed treatment or a user's comments related to an abnormality. Because the results belong to the medical image data shared by all users, each result may be visualized in real time as a thin line in each workspace. That is, each user sees the results in their own individual view immediately after the results are acquired. The 3D measurement may be a measurement in 3D dimensional space, i.e., the distance between two distinct points in each of the x-, y-, and z-coordinates. Also, the 3D measurement may include tracing an object having a non-planar irregular shape, for example, by measuring the distance spanned by multiple points in space, e.g., multiple points along the irregularly shaped object, such as the mitral valve annulus. Such non-planar measurements cannot be performed on a 2D screen, but can be easily performed in the XR workspace of the present invention. MPR mode can be multiplanar reconstruction, a two-dimensional image reconstruction technique. In MPR, coronal, sagittal, oblique, or curved planes are calculated and displayed from the transverse plane to aid the user in determining anatomical orientation. For example, oblique planes are useful for cardiac imaging (four-chamber view, short-axis view), and curved reconstruction along structures with multiple bends is useful for representing blood vessels (e.g., coronary arteries) or ureters. Obtaining high-quality MPR reconstructions requires acquiring overlapping medical image data (e.g., acquired by CT) with small layer thicknesses. Furthermore, small layer thicknesses should be selected to prevent step artifacts during image reconstruction. Image noise can be reduced by summing several layers. Thresholding can be used to adjust the threshold boundaries of each voxel to improve visualization of the region of interest.

[0056] Preferably, the step of enabling at least one user to perform an analysis process of the medical image data in their workspace may further include placing at least one model of a medical device within the visualization of the medical image data to determine the operating position of the medical device, specifically an implant. The model of the medical device may be a computer graphic object. That is, a model of the medical device may be additionally displayed within the XR workspace. The medical device may be an implant or any other device placed within a human or animal body. The computer graphic object may be, for example, a representation of geometric data, such as a 3D structure defined by vertices, e.g., a polyhedron. The computer graphic object is preferably locked to the movement of the anatomical structure, i.e., once placed at a specific position and orientation relative to the visualization in any one frame. When the user initiates cine mode, the processor controlling the visualization stores the relative position and orientation of the medical device with respect to the visualization and maintains this relative position and orientation. If the medical device represents a new valve, such new valve may be locked to the movement of a valve annulus, e.g., the mitral valve annulus. Preferably, this may be performed using 3D speckles. This provides important dynamic information throughout the cardiac cycle, allowing the valve to be optimally positioned (i.e., in its optimal position) to avoid or reduce outflow obstruction. The controller also allows the user to move and tilt the computer graphic object relative to the visualization. This allows the user to not only take measurements but also "try on" the fit of a selected implant (e.g., a replacement valve) or implant size to the anatomical mechanism (e.g., the mitral valve). For example, the user can select the best-fitting valve from the library and place that valve, or its corresponding computer graphic object, within the visualization for an initial inspection.In one useful embodiment, the computer graphic object corresponds to a CAD model of a medical device, e.g., a CAD model used in the design and / or manufacture of the medical device, or more preferably, a simplified model of the medical device, e.g., a simplified model of a heart valve.

[0057] In another embodiment, since minimally invasive interventions are almost always performed under fluoroscopic control, the computer graphic object resembles the appearance of the medical device on an interventional X-ray image (fluoroscopic image). Thus, the user can visualize the scene in three dimensions and still get an idea of ​​how the implant will look on a fluoroscopic image. The computer graphic object is preferably three-dimensional and can be, for example, a simplified model of the implant, such as a wire mesh or a model of an object's shape defined by a set of simple surfaces.

[0058] According to a preferred embodiment of the present invention, the operating position of a model of a medical device is determined by dynamically visualizing the medical device in operation, preferably in combination with 4D image information. In other words, with a model of a medical device (e.g., a prosthetic mitral valve) positioned in its intended position, 4D medical image data (i.e., a sequence of 3D medical image data in cine mode, for example) is used to visualize the study object, e.g., the heart, in operation. The dynamic movement of the medical device (e.g., during a heartbeat) can be based on the tracked 4D image data. For example, specific landmarks interacting with the medical device are tracked throughout the sequence of 3D image data, and the virtual position of the medical device is adjusted accordingly. The user can then examine how the specific model of the medical device operates with the specific anatomical structure during the examination.

[0059] In summary, the present invention in a first aspect provides a anywhere XR collaboration tool that offers an interconnected communication concept where medical image data and results of the analysis process are shared by all users, while each user still has their own individual view of the medical image data (i.e. their own copy / visualization representation of the medical image data) in their workspace.

[0060] According to a second aspect of the present invention there is provided a computer program comprising the features of claim 12. The computer program comprises program code instructions which, when executed by a processor, enable the processor to carry out the method described above. The computer program may be any code, in particular code suitable for computer graphic applications, in particular code suitable for XR programming.

[0061] Also, a computer-readable medium containing the computer program may be provided. The computer-readable medium may be any digital storage device, such as a USB stick, a hard disk, a CD-ROM, an SD card, or an SSD card. Of course, the computer program does not need to be stored on a consumer-oriented computer-readable medium, but may be downloadable via the Internet.

[0062] Preferably, the method according to the present invention is performed by a processor that may be integrated into any electronic device capable of controlling a display, in particular an XR display such as an XR headset or a projected display. Such digital equipment may be a computer, PC, server, television, tablet computer, smartphone, laptop, or handheld device. The processor may also be part of a cloud computer, a workstation, or the control console of a medical imaging device, in particular an ultrasound scanner.

[0063] According to a third aspect of the present invention there is provided a user interface configured for use in performing the above method, the user interface comprising: an XR display device, in particular a VR headset, for displaying to a user in real time in a workspace a visualization of medical image data and the results of an analytical process; The workspace is an XR workspace, the workspace comprises a virtual environment in which a visualization representation of the medical image data and the results of the analytical process are displayed; The user interface further includes a tracked controller configured for use during the performance of the method to input commands by user gestures, the commands including: This includes selectively and individually enabling or disabling the display of the results of the analytical process.

[0064] Any of the features of the advantageous embodiments described in relation to the method according to the invention also apply to the user interface.

[0065] A user interface is, for example, a system that typically includes at least a screen or display and input elements, such as XR controllers and / or buttons or sliders, that allow a user to interact with the content of the display, for example by adjusting visualization parameters / settings, zooming, annotating, and / or moving or tilting the visualization.

[0066] Preferably, the commands may further include individually and independently adjusting the contrast within the user's workspace, preferably by setting the transparency and / or color of the virtual environment.

[0067] According to a fourth aspect of the present invention, there is provided a system for analysing medical image data in virtual multi-user collaboration, the system comprising a processor configured to perform the above method and at least two of the above user interfaces connected to the processor.

[0068] Any features of the useful embodiments described in relation to the method according to the invention also apply to the system.

[0069] The virtual reality environment can be realized using commercially available VR equipment, such as the HTC VIVE Pro VR system (manufactured by HTC Corporation, 330 Taoyuan City, Taiwan), which includes a VR headset, two VR controllers, and two position trackers, or the Oculus Rift S (Facebook Technologies, LLC, Oculus). The headset does not require a separate position tracker in the room because the position tracking function is provided by the headset itself. [Brief explanation of the drawings]

[0070] Useful embodiments of the present invention will now be described with reference to the accompanying drawings, in which similar elements or features are provided with the same reference numerals. [Figure 1] FIG. 1 shows a schematic cross-sectional view of the human heart (four-chamber view). [Figure 2] FIG. 2 shows a schematic diagram of a sequence of medical images. [Figure 3] FIG. 3 is a diagram illustrating the principle of virtual multi-user collaboration according to an embodiment of the present invention. [Figure 4] FIG. 4 illustrates a user interface according to one embodiment of the present invention. [Figure 5] FIG. 5 shows a flow diagram illustrating a method according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0071] To better visualize a preferred application of the method according to the present invention for analyzing medical image data in virtual multi-user collaboration, FIG. 1 illustrates the structure of a human heart 1. Blood from the lungs flows into the left atrium 2, from which it flows through the mitral valve 3 into the left ventricle 4. From there, blood is pumped through the aortic valve 5 into the aorta 6, also known as the left ventricular outflow tract (LVOT). Blood from the torso flows into the right atrium 7, from which it flows through the tricuspid valve 8 into the right ventricle 9. From there, blood is pumped through the pulmonary valve 10 into the pulmonary artery 11. The heart wall 12 is composed of muscular tissue surrounding the heart chambers 2, 4, 7, and 9. The left and right ventricles are separated by a septum 13. From FIG. 1, it can be seen that the heart has a complex shape and is constantly moving with the heartbeat, i.e., it is a dynamic anatomical structure. Therefore, analyzing a complex structure such as the mitral valve 3 to plan a valve replacement is difficult and prone to error. That is, the heart 1 in Figure 1 represents an example of 3D medical image data obtained by MR, computed tomography (CT), positron emission tomography (PET), ultrasound (US), or transesophageal echocardiography (TEE). To obtain 4D medical image data, time is used as the fourth dimension.

[0072] Figure 2 shows a sequence of ultrasound images M1, M2, M3, M of heart 1 based on medical image data. Z 1 shows a schematic diagram of a cardiac cycle, where Z is the number of images acquired during one cardiac cycle, i.e., during time T, where T is approximately 0.5 to 1.5 seconds. For simplicity, the drawing shows two-dimensional images, but in accordance with the present invention, at each time point t i A three-dimensional medical image may be formed by a stack of two-dimensional images. A four-dimensional medical image may be formed by a plurality of three-dimensional medical images displayed in succession. Such an image sequence M1, M2, M3, ... M Z can be obtained by performing echocardiography of a moving heart using, for example, a TEE probe. Z represents 4D medical image data.

[0073] In one embodiment of the present invention, an interdisciplinary cardiac team (IHT) is planning a cardiac surgery and will discuss complex aspects before the actual surgery. To ensure that all participants in the IHT have the same understanding of the patient's anatomy and the planned procedure, the team has scheduled a virtual multi-user collaboration. During the collaboration, each member of the team is in a different location. The IHT consists of an interventional cardiologist C, a cardiac anesthesiologist A, an OR / catheterization lab nurse N, and a cardiac surgeon S. Figure 3 illustrates the principle of virtual multi-user collaboration according to an embodiment of the present invention.

[0074] As shown in FIG. 3 , each participant / user has their own workspace WS. 3D or 4D medical image data 34 is provided and loaded to be visualized in each workspace 30. As a result, an individual visualization 32 of the medical image data 34 is generated and provided for each workspace. In other words, each user has their own individual view 32 of the medical image data 34. Then, each user begins working on the visualization. First, each user modifies the visualization 34 to meet their own requirements for analyzing the visualization of the medical image data.

[0075] There are significant individual differences in how different users view and analyze datasets (e.g., zoom level, threshold / contrast / brightness settings, preferred viewing angle, etc.). Therefore, each user has their own separate copy 32 of the visualization of medical image data 34, which they can resize, rotate, move, and change image settings using a controller 36. These manipulations / view settings are not transmitted to other users. For example, interventional cardiologist C prefers the visualization to be displayed approximately 40 cm in size and approximately 50 cm from their head, with a high threshold set to hide the leaflets of the mitral valve 4, and a viewing direction from the left atrium 2 through the mitral valve 4 to the left ventricle 4. Meanwhile, cardiac anesthesiologist A prefers the visualization to be displayed as large as possible (100–120 cm) so that he or she can move his or her head within the visualization, a lower threshold set to easily see the leaflets, and a lateral viewing direction in the two-chamber view. OR / catheterization lab nurse N and cardiac surgeon S also prefer more individual views of the visualization. This modification of the visualization 32 is performed individually and independently by each user to obtain an individual visualization (i.e., an individual view of the medical image data), whereby modifications of the individual views of the medical image data by other users are not visible to any user.

[0076] Furthermore, if the medical image data changes due to available new measurements, the visualizations based on the medical image data are automatically updated. That is, in each workspace 30, the visualizations are updated based on the medical image data shared by all users. Another reason why new medical image data may become available is, for example, in the case of a patient in an emergency, when collaboration occurs and the acquired data is provided directly while the patient's organs are being analyzed. Furthermore, each user can see which version of the medical image data the visualizations are based on in their workspace 30. Each user can also independently switch between different versions of the visualizations.

[0077] The analysis process then begins, and each user performs the analysis process in their own workspace 30 while having their own individual view of the medical image data. That is, each user uses the controller to create their own measurements / annotations, which are the result of the analysis process in their own workspace 30. In other words, each user controls the medical image data by adding the results of the analysis process directly to the medical image data via their own workspace 30. At the same time, the measurements / annotations entered via the controller are sent to the medical image data shared by all users. As a result, each user sees the measurements / annotations of other users live in the visualization of their own medical image data. As a result, each user can immediately see and receive the measurements / annotations of other users and, in some cases, use them in their own approach.

[0078] For example, the same image dataset is displayed simultaneously to all four users. Each user (C, A, N, and S) creates their own measurements and annotations. Each user sees the other users' annotations live on their own visualization of the image dataset. As an example, cardiologist C and surgeon S may have different ideas about which procedure is correct, but anesthesiologist A, having seen both approaches, has another idea that meaningfully combines S's approach with N's. Using the method of the present invention, S and N create different measurements and annotations, and A can see both of them. Thus, A can annotate his improved suggestions on his own visualization of the image dataset. For each user, A's new annotations are displayed live on their own model / visualization, and each user agrees on them. This allows the interdisciplinary cardiac team to meet more efficiently.

[0079] Because the measurements / annotations are sent directly to (i.e., synchronized with) the medical image data 34 shared by all users, the measurements / annotations may obscure or cover portions of the visualization in another user's workspace 30. Thus, if portions of the visualization are obscured and / or covered in one user's workspace 30, the user may disable the measurements / annotations. Alternatively, the user may adjust the measurements / annotations to be transparent.

[0080] That is, the present invention allows for fewer unnecessary interactions during collaboration, faster sharing of measurements / annotations, and unrestricted views of the region of interest for each user, regardless of the number of participants in a VR session. Furthermore, because individual understanding of anatomical structures differs, differences in perception of reference points and structures within a medical 3D dataset inevitably lead to inter-observer variability, different measurements (procedures), different assumptions, and different interpretations of anatomical correlations. However, the present invention avoids these issues by allowing each user to individually modify the visualization representation according to their own working preferences. Furthermore, users can see their own and other users' changes live in their own individual views, which can be manipulated and adjusted according to their own personal viewing preferences.

[0081] In another embodiment of the present invention, at least one workspace 30 is an AR workspace. In other words, the process is the same as that of the previous embodiment, except that the user views both the visualization and the real environment. The real environment forms the background of the visualization. In this embodiment, the AR workspace has a virtual environment interposed between the visualization and the real environment to allow the user to fully view and analyze the visualization. As a result, the real environment can be blocked by increasing the opacity of the virtual environment. As a result, the AR workspace 30 can automatically set the opacity / transparency of the virtual background to provide a desired contrast within the workspace 30. AR workspaces are particularly useful when a user simultaneously interacts with a person (e.g., a patient) in the real environment. Additionally, the virtual environment automatically adapts to changing lighting conditions. Alternatively, in other embodiments, each user can individually tailor the virtual environment to meet their needs and / or preferences.

[0082] According to another embodiment, a user can synchronize their individual view with another user, such as a teacher demonstrating critical measurements or placement of a medical device to students in a classroom. That is, the individual view 32 of the visualization is shared with one or more other participants in the collaboration. The user can position themselves at an optimal viewpoint and watch live and in real 3D as another (e.g., more experienced) user performs a specific measurement or annotation or positions a device, such as a prosthetic heart valve, within the visualization to determine its optimal location. For example, a “teacher” can first observe (from the student’s perspective) how a student performs a specific annotation / measurement, and then, if they want to demonstrate a particular part of the intervention or measurement strictly from their own perspective, they can synchronize the individual views of one or more students with their own. In this case, similar to GoToMeeting, one user assumes the role of moderator (“presenter / teacher”), and all other participants (“participants / students”) have more limited capabilities.

[0083] In a preferred embodiment, the user interface is a VR interface, as shown in FIG. 4 . Such an interface is realized by a virtual reality headset 82 worn by a user 80. The headset 82 is connected to the computer 72 via a cable or wireless connection. Such a virtual reality headset 82 includes separate internal displays for each eye and a position sensor 84 for tracking head movement. If an augmented reality environment is to be presented, the headset may also include a camera. The user 80 also holds a VR controller 86 in his or her hands. The controller 86 also includes a position sensor (not shown) and further includes buttons or other input elements. Such a virtual reality controller 86 allows the user to grasp and move objects displayed in the virtual reality environment 50. The VR headset may be, for example, an HTC VIVE headset and corresponding VR controller.

[0084] FIG. 5 shows a flow diagram illustrating a method according to one embodiment of the present invention. In step 90, medical image data is provided, for example, including 3D or 4D image information showing a moving heart. If 4D medical image data is provided, the sequence spans a period corresponding to one heartbeat. In step 92, the medical image data is loaded into each user's workspace 30, and a visualization of the medical image data is simultaneously displayed to each user, for example, by generating a volume-rendered or surface-rendered visualization. In step 94, each user individually and independently modifies the visualization of the medical image data, resulting in an individual visualization of the medical image data (i.e., their own individual view of the medical image data) in each user's workspace 30. In step 96, at least one user can perform an analysis process on the medical image data in their own workspace 30. Such an analysis process includes making measurements and annotations located in a 3D coordinate system. In step 98, the results of the analysis process are displayed in the workspace 30 in which the analysis process was performed. In step 100, the results of the analysis process are synchronized in real time with at least one other workspace 30, with each workspace 30 displaying the results of the analysis process in a respective individual visualization of the medical image data. Steps 96-100 are performed simultaneously, i.e., the measurements and annotations made in step 96 are implemented directly into the medical image data 34 shared by each user, so that the measurements and annotations are directly visible in each workspace.

[0085] While the invention has been illustrated and described in detail in the drawings and foregoing, such illustration and description are to be considered illustrative or exemplary rather than descriptive. The invention is not limited to the disclosed embodiments.

Claims

1. 1. A method for analyzing medical image data in a virtual multi-user collaboration, comprising: The medical image data is analyzed by at least two users; Each user has their own workspace, said workspace being an XR workspace; The method comprises: Providing medical image data including 3D or 4D image information; loading said medical image data into said workspace of each user and displaying a visualization of said medical image data to each user simultaneously; - enabling each user individually and independently of each other to modify the visualization representation of the medical image data to obtain an individual visualization representation of the medical image data in each workspace of each user; - enabling at least one user to perform an analysis process of the medical image data in his / her workspace using real or virtual tools, the execution of the analysis process not being shared with at least one other workspace; displaying results of the analytical process in the workspace in which the analytical process was performed; and synchronizing the results of the analytical process in real time with at least one other workspace, such that each workspace displays the results of the analytical process in a respective individual visualization representation of the medical image data.

2. The method for analyzing medical image data in virtual multi-user collaboration according to claim 1 , wherein the analysis process of the medical image data is performed by at least two users simultaneously.

3. A method for analyzing medical image data in virtual multi-user collaboration as described in claim 1 or 2, wherein the step of displaying the results of the analysis process can be selectively and individually enabled or disabled by users in their own workspaces.

4. At least one workspace is an AR workspace or an MR workspace; A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 3, wherein at least one visualization parameter in the AR workspace or the MR workspace, in particular the transparency and / or color of the visualization representation of the medical image data and / or the results of the analysis process, is automatically adjusted so that the user can see the visualization representation of the medical image data and / or the results of the analysis process superimposed on a real environment at a target contrast.

5. each workspace having its own virtual environment in which the visualization representation of the medical image data and the results of the analysis process are displayed; The method comprises: A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 4, further comprising a step of enabling each user individually and independently to adjust visualization parameters of the virtual environment by setting the transparency and / or color of the virtual environment and to adjust contrast within the workspace.

6. The method for analyzing medical image data in virtual multi-user collaboration according to claim 5 , wherein the virtual environment includes at least one virtual control element.

7. A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 6, wherein the step of enabling each user individually and independently to modify the visualization representation of the medical image data includes using a controller to perform the modification of the visualization representation using hand gestures by grabbing an object in the workspace.

8. A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 7, wherein the step of enabling each user individually and independently to modify the visualization representation of the medical image data includes manipulating the visualization representation to rotate the visualization representation, crop a portion of the visualization representation, change rendering parameters of the visualization representation, change image settings of the visualization representation, change the contrast of the visualization representation, change a voxel intensity threshold of the visualization representation, and / or change the size of the visualization representation.

9. At least one user may adopt the modifications of the visualization representation of the medical image data made by another user; or A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 8, wherein one user can force at least one other user to adopt his / her own changes to the visualization representation of the medical image data.

10. A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 9, wherein the step of enabling at least one user to perform an analysis process of the medical image data in his or her own workspace further includes obtaining 3D measurements, performing an MPR mode, and / or inserting annotations.

11. The step of enabling at least one user to perform the analysis process of the medical image data in his workspace further comprises placing at least one model of a medical device in the visualization representation of the medical image data in order to determine an operating position of a medical device, in particular an implant, A method for analyzing medical image data in virtual multi-user collaboration as described in any one of claims 1 to 10, wherein the operating position of the model of the medical device is determined by dynamically visualizing the medical device in operation in combination with 4D image information.

12. A computer program comprising program code instructions which, when executed by a processor, enable said processor to carry out the method according to any one of claims 1 to 11.

13. A user interface for use in carrying out the method of any one of claims 1 to 11, said user interface comprising: an XR display device, in particular a VR headset, for displaying to a user in real time in a workspace a visualization representation of the medical image data and the results of the analysis process, the workspace is an XR workspace, the workspace includes a virtual environment in which the visualization representation of the medical image data and the results of the analysis process are displayed; The user interface further includes a tracked controller configured for use during the method to input commands by gestures of the user, the commands comprising: A user interface including selectively and individually enabling or disabling the display of the results of the analytical process.

14. The command further comprises: The user interface of claim 13 , including individually and independently adjusting contrast within the workspace of the user by setting transparency and / or color of the virtual environment.

15. 1. A system for analyzing medical image data in virtual multi-user collaboration, comprising: A processor for executing the method according to any one of claims 1 to 11; A system comprising at least a user interface according to claim 13 or 14, connected to said processor.

Images