Computer-implemented method for generating a representation of a volumetric dataset representing a medical volume for use in a rendering process

Abstract

A computer-implemented method for generating a representation of a volumetric dataset representing a medical volume for use in a rendering process is described. The method involves generating a multitude of function sets representing the volumetric dataset in various rendering configurations and, based on these function sets, determining given functions with a respective 3D position relative to the volumetric dataset and one or more visual attribute values ​​to be stored in the representation. The given functions are stored in the representation in conjunction with selection data designed to allow the selection of one or more of the given functions for use in the rendering process. The selection data is based on one or more of the rendering configurations.A computer-implemented method for rendering an image of a volumetric dataset representing a medical volume using such a representation is also described.

Description

field of technology

[0001] The present invention relates to a computer-implemented method for generating a representation of a volumetric data set representing a medical volume for use in a rendering process. The present invention also relates to a method for rendering an image of a volumetric data set representing a medical volume using such a representation. Furthermore, the present invention relates to a device and a set of machine-readable instructions for implementing both methods, and to a machine-readable medium comprising such instructions. State of the art

[0002] In computer-implemented imaging of medical volume data, the immediate provision of interactive, high-quality visualizations of medical data can present challenges related to data size, particularly when rendering data generated by high-resolution imaging modalities such as 7T MRI, photon-counting CT, and phase-contrast CT. The enormous amounts of data generated by these imaging modalities can pose challenges in transferring and / or rendering the image data due to limitations such as bandwidth, rendering performance, and storage capacity. Such problems can be particularly evident in the context of clinical workflows and anatomy learning platforms, where computing and time resources may be constrained. Brief description of the invention

[0003] According to a first aspect of the present invention, a computer-implemented method for generating a representation of a volumetric data set representing a medical volume is provided for use in a rendering process.The method comprises generating a multitude of function sets representing a volumetric dataset in various rendering configurations; determining, based on the multitude of function sets, given functions with a respective 3D position relative to the volumetric dataset and one or more respective visual attribute values, to be stored in a representation of the volumetric dataset for use in a rendering process; and storing, in the representation of the volumetric dataset, the given functions in conjunction with selection data designed to allow a selection of one or more of the given functions for use in the rendering process, the selection data being based on one or more of the rendering configurations.

[0004] The different rendering configurations can include a first rendering configuration, which represents a first section of the volumetric dataset with a first resolution, and a second rendering configuration, which represents a second section of the volumetric dataset that differs from the first section and has a second resolution that is higher than the first resolution.

[0005] The given functions can be Gaussian functions, and the representation of the volumetric dataset can be a Gaussian splat representation.

[0006] One or more of the different rendering configurations can differ from each other with respect to one or more of the following: a clipping layer setting; a transfer function setting; a lighting setting; and a bounding box setting.

[0007] The procedure may involve comparing one or more functions from different sets of the multitude of function sets; and determining the given functions to be stored in the representation of the volumetric data set may be based on a result of the comparison.

[0008] The comparison can involve determining respective similarities between functions of the different sets of functions, and the determination of one or more given functions can be based on the determined similarities.

[0009] The volumetric dataset can be a volumetric 4D dataset, and the different rendering configurations can represent different time steps of the volumetric 4D dataset.

[0010] Generating the multitude of function sets can involve: generating each set of the multitude of function sets based on a respective multitude of rendered images showing the volumetric dataset from different viewpoints in a given rendering configuration.

[0011] The method can involve generating a large number of rendered images using a respective rendering process with one or more rendering parameters. The method can also involve determining the selection data based on one or more of these rendering parameters.

[0012] The procedure may include identifying combinations of viewpoints and respective rendering configurations of the various rendering configurations designed, when used in a rendering process, to provide images that differ from one another with respect to a visible aspect of the volumetric dataset; and identifying, based on the identified combinations, the respective different viewpoints for the respective multitude of images that show the volumetric dataset in the given rendering configuration.

[0013] One or more of these respective rendering processes can be a physical rendering process.

[0014] According to a second aspect of the present invention, a computer-implemented method for rendering an image of a volumetric dataset representing a medical volume is provided. The method comprises: performing a rendering process using a representation of a volumetric dataset representing a medical volume, wherein the representation of the volumetric dataset includes given functions with a respective 3D position relative to the volumetric dataset and one or more respective visual attribute values; wherein the given functions are stored in the representation in conjunction with selection data designed to allow, during the rendering process, a selection of one or more of the given functions for use in the rendering process;and wherein the rendering process comprises: selecting, based on the selection data and one or more parameters of the rendering process, one or more functions of the given functions for use in the rendering process; and rendering an image of the volumetric dataset using the selected one or more functions of the given functions.

[0015] According to a third aspect of the present invention, a set of machine-readable instructions is provided which, when executed by a processor, cause a method according to the first aspect of the present invention and / or a method according to the second aspect of the present invention to be carried out.

[0016] According to a fourth aspect of the present invention, a machine-readable medium is provided comprising: a representation of a volumetric data set representing a medical volume, wherein the representation can be generated by a method according to the first aspect of the present invention; and / or a set of machine-readable instructions according to the third aspect of the present invention.

[0017] According to a fifth aspect of the present invention, a device is provided comprising: a processor; and a storage device comprising: a representation of a volumetric data set representing a medical volume, wherein the representation can be generated by a method according to the first aspect of the present invention; and / or a set of machine-readable instructions according to the third aspect of the present invention. Brief description of the drawings

[0018] The present invention is now described only by way of example and with reference to the accompanying drawings, in which the following applies: Fig. Figure 1 is a flowchart illustrating a computer-implemented procedure for generating a representation of a volumetric dataset representing a medical volume for use in a rendering process; Fig. Figure 2 is a flowchart that describes a computer-implemented procedure for rendering an image of a volumetric dataset using a method according to Fig. 1 generated representation illustrated; Fig. 3 is a schematic drawing that illustrates aspects of an exemplary procedure according to Fig. 1 illustrates; and Fig. 4 A schematic drawing showing a setup for carrying out exemplary procedures described herein. Detailed description

[0019] Fig. Figure 1 is a flowchart illustrating an exemplary computer-implemented procedure 100 for generating a representation of a volumetric data set representing a medical volume for use in a rendering process.

[0020] The volumetric dataset can comprise a discrete sampling of a scalar field. The volumetric dataset can be obtained, for example, by loading data from memory, sensors, and / or other sources. The medical volume represented by the volumetric dataset can, for example, contain a patient or part of a patient, such as a human or animal patient.

[0021] In general, any suitable scanning modality can be used to create the volumetric dataset. The scanning modality might include, for example, the use of computed tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET). In some examples, the scanning modality might include a high-resolution imaging modality, such as 7T MRI, photon counting CT, or phase-contrast CT. In some examples, a scanning modality that includes the use of single-photon emission computed tomography (SPECT), ultrasound, or another scanning modality might be used. In some examples, the volumetric dataset might contain data from several such scanning modalities. For example, multiple sets of volumetric data, such as PET, CT, and MRI, could be combined to provide the volumetric dataset.Scan data can be provided in the form of multiple two-dimensional (2D) scans or can be formatted from a single scan. In some examples, the volumetric dataset may comprise a stack of 2D image slices. The 2D image slices may include compressed slice data, for example, in JPEG2000 format. In some examples, the volumetric dataset is a DICOM dataset created by scanning at least one section of a patient using one or more scanning modalities.

[0022] The volumetric dataset can comprise data formatted as a multitude of voxels. The voxels can be arranged in a uniform or non-uniform grid, or in another type of geometry (such as polar coordinates). The voxels can be isotropic or anisotropic. Each voxel typically represents a scalar value obtained by sampling a scalar field, although in some examples, the volumetric dataset may contain data with respect to a non-scalar field. The type of value represented by each voxel can depend on the means by which the volumetric dataset is obtained. For example, if a CT scanner is used to create the volumetric dataset, the dataset may contain Hounsfield values. The values ​​represented by each voxel can be represented with a given precision, such as 8 bits or 16 bits.

[0023] In some examples, the volumetric dataset can be a 4D volumetric dataset. The volumetric dataset can, for example, comprise a series of 3D volume datasets, each representing the medical volume at a different time step. Each 3D volume dataset can have any of the features described above, such as volumetric datasets.

[0024] In block 102, the procedure involves generating a multitude of function sets that represent the volumetric dataset in various respective rendering configurations.

[0025] The various rendering configurations can differ from each other, for example, in one or more of the following: a clipping layer setting; a transfer function setting; a lighting setting; or a bounding box setting. Additionally or alternatively, the different configurations can refer to different time steps in a volumetric 4D dataset.

[0026] In one example, the different rendering configurations represent different sections of the volumetric dataset. These sections can be displayed at different resolutions. For instance, in a first rendering configuration, a first section of the volumetric dataset, representing, say, the entirety of the medical volume, can be displayed at a higher resolution, while in a second rendering configuration, a second section of the volumetric dataset, representing, say, an area of ​​interest within the medical volume, can be displayed at a higher resolution. The first and second sections of the volumetric dataset can correspond to different bounding box settings.The second resolution can, for example, be higher than the first resolution, so that certain areas of interest in the volumetric dataset can be displayed at a higher resolution.

[0027] In another example, the volumetric dataset can be rendered in a first rendering configuration with a first clip layer setting. A second rendering configuration can correspond to a second clip layer setting. For example, a given clip layer might be placed in a different position in the second rendering configuration compared to its position in the first.

[0028] In other examples, any other parameter or any combination of parameters may differ between the various rendering configurations.

[0029] Generating the multitude of function sets representing the volumetric dataset in the various rendering configurations can involve generating each set of function sets based on a specific multitude of rendered images. These images can depict the volumetric dataset from different perspectives within the given rendering configuration.

[0030] As an example, an initial set of images used to generate a first set of function sets can depict the volumetric dataset in an initial rendering configuration from a variety of different perspectives. The initial rendering configuration can correspond to an initial set of rendering parameters for a rendering process used to render the first set of images of the volumetric dataset. For example, the initial rendering configuration can correspond to an initial given combination of bounding box, clipping level, and transfer function settings used in a rendering process to generate the first set of images.

[0031] Images representing the volumetric dataset in a given rendering configuration can be generated by performing a rendering process using a given set of rendering parameters. This rendering process can be, for example, a physical rendering process, a path-tracing rendering process, or, in other examples, a non-physical rendering process, such as raycasting. The rendering process can be repeated from a variety of different viewpoints to obtain images of the volumetric dataset in the given rendering configuration. This process can be repeated for one or more other rendering configurations to obtain corresponding sets of images representing the volumetric dataset in each of the different rendering configurations.Different bounding box settings can be used, for example, to render different sections of the volumetric dataset, e.g., with different resolutions; different clip layer settings can be used to render the volumetric dataset with a different clip layer position or orientation; and / or different transfer function settings can be used to render the volumetric dataset with different visual characteristics.

[0032] A given set of functions generated at Block 102 provides a representation of the volumetric dataset in the given rendering configuration, corresponding to the given set of functions. For example, a given set of high-fidelity images can be rendered using a path-tracing process that shows the volumetric dataset in a given configuration from a limited number of angles. A set of functions can be generated from the multitude of images that provides a 3D representation of the volumetric dataset in the given rendering configuration. These functions can be used in a rendering process, such as a rasterization process, to render images of the volumetric dataset in the given rendering configuration from any given angle.The functions can provide a compact representation of the volumetric dataset in the given rendering configuration. For example, the set of functions, when stored, can occupy significantly less space than the volumetric dataset itself.

[0033] In the rendering configuration of the volumetric dataset used to render images to generate a given set of functions, the volume can be segmented and color, opacity, and segmentation mask-specific transfer functions can be applied.

[0034] In some examples, a set of functions representing the volumetric dataset in a given rendering configuration can be a point cloud representation. That is, each function can represent a point in a point cloud with a 3D position relative to the volumetric dataset. Each function can have one or more visual attribute values, such as shape, color, and opacity.

[0035] In certain such examples, the set of functions representing the volumetric dataset in a given rendering configuration is a set of Gaussian functions that form a Gaussian splat representation of the volumetric dataset in that rendering configuration. In an example where a given set of functions is a Gaussian splat comprising a set of Gaussian functions, each function may contain the following parameters. - A mean value µ, which can be interpreted as a 3D position in the volumetric dataset, which includes, for example, Cartesian x, y, z coordinates; - A covariance Σ, which can be interpreted as a form of the Gaussian function; - An opacity σ(α), which can be a sigmoid function, that is applied to map the parameter to the interval [0, 1]. - One or more color parameters, for example three values ​​for (R, G, B) or spherical harmonic coefficients.

[0036] A set of such Gaussian functions, representing the volumetric dataset in the given rendering configuration, can be developed by performing a training process using the images depicting the volumetric dataset in that configuration. The training process can involve using a differentiable rendering technique to optimize the parameters of the Gaussian functions and the number of Gaussian functions in the set, for example, using a gradient descent technique. The training process might, for instance, involve initializing a set of Gaussian functions representing the rendering configuration and rendering images from the Gaussian function set using a rasterization technique.Rasterization techniques can involve GPU sorting and rasterization of projected 2D Gaussian functions and the use of a pixel shader to evaluate and blend the 2D projections in image space. In one particular exemplary rasterization technique for rendering from a Gaussian splat representation, the screen space is tiled, and the Gaussian functions visible in the view cone truncated for each tile are determined. These Gaussian functions are then labeled according to their depth. For each tile, the associated Gaussian functions are loaded into memory, and for a given pixel color and alpha (i.e., opacity), values ​​are accumulated by iterating the associated Gaussian functions from front to back with respect to depth until a target alpha saturation for the pixel is reached.

[0037] The images rendered from the Gaussian functions can be compared with the original rendered images of the volumetric dataset in the given rendering configuration. The parameters of the Gaussian functions can be adjusted to better represent the original images. Successive iterations of this process can be performed to optimize the set of Gaussian functions for representing the volumetric dataset in the given rendering configuration, for example, using a stochastic gradient descent technique.Generating a set of Gaussian functions representing a given rendering configuration can, for example, use any of the techniques described in the publication: 3D Gaussian Splatting for Real-Time Radiance Field Rendering, Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, SIGGRAPH 2023, (ACM Transactions on Graphics) and / or in the publication Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis, Simon Niedermayr, Josef Stumpfegger, Rüdiger Westermann, CVPR 2024.

[0038] In Block 104, Procedure 100 involves determining, based on the multitude of function sets, given functions to be stored in a representation of the volumetric dataset for use in a rendering process. The given functions to be stored have a respective 3D position relative to the volumetric dataset and one or more visual attribute values, such as color, shape, and opacity.

[0039] In some examples, the given functions to be stored in the volumetric dataset representation are functions from the multitude of function sets generated at Block 102. For example, the given functions might be selected from the multitude of function sets generated at Block 102. The given functions might be Gaussian functions, and the volumetric dataset representation in which the given functions are stored might be a Gaussian splat representation.

[0040] In an example where a first set of functions from the plurality of function sets represents a first section of the volumetric dataset with a first resolution, and a second set of functions from the plurality of function sets represents a second section of the volumetric dataset with a second resolution, one or more functions from each of the first and second sets can be identified as functions to be stored in the representation of the volumetric dataset. In some such examples, some or all of the functions from the first and second sets can be selected as functions to be stored in the representation.

[0041] In some examples, the representation of the volumetric dataset may comprise different subsets of functions that represent the volumetric dataset in different rendering configurations. For example, in an example like the one described above, where the first and second rendering configurations represent the volumetric dataset with different clip layer settings, a first subset of the given functions may represent the volumetric dataset in the first rendering configuration with a first clip layer setting, and a second subset of the given functions may represent the volumetric dataset in the second rendering configuration with a second clip layer setting.Similarly, if the volumetric dataset is a 4D dataset encompassing different time steps, a first subset of the given functions can represent the first time step of the volumetric dataset, and a second subset of the given functions can represent the second time step of the volumetric dataset.

[0042] In some examples, the given functions to be stored can be determined by comparing functions from the multiple function sets representing the different rendering configurations. Determining the given functions at block 104 to be stored in the representation of the volumetric dataset can be based on the result of such a comparison. For example, the comparison might involve identifying similarities between functions from the different function sets. The functions to be stored in the representation can then be determined based on these identified similarities.

[0043] As an example, functions from a second set of functions representing a first rendering configuration can be compared with functions from a first set of functions representing a second rendering configuration to determine their similarities. All functions from the second set that are found to have a given degree of similarity to a function from the first set, such as a similarity metric above a given threshold, can be identified. In such an example, all functions from the first set can be identified as a first subset of the given functions to be stored.Meanwhile, from the second set of functions, only those functions that do not meet the given degree of similarity with respect to the functions of the first set (or, in other words, are sufficiently different from the functions of the first set) are identified as a second subset of the given functions to be stored. In this way, the second subset of the given functions can represent the second rendering configuration more efficiently by encoding only a difference between the second and first rendering configurations.In some examples, such a technique can be used to determine the given functions to efficiently represent different rendering configurations, which might represent, for example, different clip layer settings, a different bounding box or transfer function settings, different time steps in a volumetric 4D dataset, or more than one of these in combination. Examples of such techniques are described in more detail below.

[0044] In some examples, certain functions to be stored might not be functions selected from the various function sets representing different rendering configurations, but rather might be determined from these sets in another way. For instance, two or more functions from the different function sets might be used to determine a given function to be stored. For example, a function from one function set and a function from a second function set might be combined to obtain a given function to be stored in the volumetric dataset representation.

[0045] In block 106, procedure 100 includes storing, in the representation of the volumetric dataset, the given functions in conjunction with selection data. The selection data is designed to allow the selection of one or more of the given functions for use in the rendering process. The selection data is based on one or more of the rendering configurations.

[0046] In some examples, the selection data includes a specific label for each of the given functions. Each label can be used to determine whether the given function should be selected when rendering images from the volumetric dataset representation. For example, each function might be associated with a label indicating a subset of the given functions to which it belongs. If the given functions include, for instance, functions representing respective sections of the volumetric dataset at different resolutions, each function might be labeled according to the section of the volumetric dataset it represents and / or the resolution it corresponds to.In some such examples, the functions representing the different sections of the volumetric dataset at varying resolutions may be arranged in a hierarchy, with each function labeled according to a level in the hierarchy to which it corresponds. Alternatively or additionally, a given function may be labeled according to a time step in a sequence of time steps to which it corresponds, or according to any one or more parameter settings, such as a clip level setting, a transfer function setting, or a lighting setting.

[0047] Fig. Figure 2 is a flowchart illustrating an exemplary computer-implemented procedure 200 for rendering an image of a volumetric dataset representing a medical volume. Procedure 200, in block 202, involves performing a rendering process using a representation of the volumetric dataset. The representation of the volumetric dataset can include any of the features described above by procedure 100. Fig. 1 generated representation. The representation can be changed by a Fig. The described procedure has been generated 100 times.

[0048] In one example, the rendering process involves selecting, based on the selection data and one or more rendering parameters, one or more of the given functions. The selected one or more functions can be used to generate visual parameter values ​​for an image of the volumetric dataset.

[0049] The rendering parameters of the rendering process, used to select one or more functions, can correspond to any setting representing a specific rendering configuration for displaying the volumetric dataset. For example, one or more clip level settings, transfer function settings, or settings indicating a region of the volumetric dataset to be visualized—such as a bounding box or zoom setting—can be used to select one or more functions for rendering an image. Alternatively or additionally, a rendering parameter specifying a time step of a 4D volumetric dataset to be visualized can be used to select from the available functions.

[0050] The selected functions can be used to generate an image using any suitable rendering process. In one example, the rendering process involves a rasterization process. The rasterization process can have any of the features described above regarding the rasterization process used to generate the various sets of functions on which the functions stored in the representation are based.

[0051] Since the given functions are stored in the representation of the volumetric dataset in conjunction with the selection data, suitable functions for visualizing the volumetric dataset in a given rendering configuration can be selected during the rendering process. For example, a user can adjust the rendering parameters by zooming in or otherwise selecting areas of interest within the volumetric dataset. A suitable rendering parameter, such as the zoom level or a selected bounding box setting, can then be used to select, based on the selection data, the given functions to render an image of the area of ​​interest.As another example, a user can adjust a clip level setting, and this setting can be used to select, based on the selection data, the given functions suitable for representing the volumetric dataset with the desired clip level setting. As yet another example, during a visualization of 4D volume data, a time step setting can be adjusted to specify a time step to be visualized, and this setting can be used to select the given functions that represent the volumetric dataset at that time step.

[0052] In this way, a compact representation of a volumetric dataset can be provided to enable a user to visualize the volumetric dataset in a computationally efficient and storage-efficient manner, while also offering a high degree of interactivity for the visualization.

[0053] The functions that provide the visualization of the volumetric dataset can be designed to deliver a representation that is compact in terms of storage and enables fast rendering of images of the volumetric dataset in a given rendering configuration from any desired perspective. For example, the functions can provide a Gaussian splat representation of the volumetric dataset in several different rendering configurations. Selection data can enable interactive visualization by allowing the user to choose appropriate functions to visualize the volumetric dataset in a desired rendering configuration. The rendering configuration to be visualized can be selected by the user or otherwise adjusted during rendering, for example, by changing a rendering parameter of the rendering process used to generate the images.

[0054] Because the representation of the volumetric dataset can be smaller than the volumetric dataset itself, it can enable the representation to be transferred to devices where it may be impractical or impossible to provide the entire volumetric dataset, for example, due to limited storage or bandwidth. Furthermore, in some examples, the representation can allow high-quality visualizations to be rendered by devices with limited rendering performance or memory size restrictions, as the device can render a high-quality visualization from the representation, for example, using a rasterization process, without having to perform a more computationally intensive rendering process such as path tracing.

[0055] As such, according to some examples, high-quality interactive visualizations of even the largest volumetric datasets can be delivered to devices such as mobile devices, immersive devices like light field displays, and AR / VR headsets. This can offer improved portability of such visualizations, for example, by enabling a surgeon's mobile device in an operating room to generate a high-quality interactive visualization, or for students to learn and explore on a website, social media, or a personal mobile or immersive display device. Alternatively, powerful rendering devices that handle large volumetric datasets and require high-quality rendering, e.g.,Based on path tracking, the methods described herein offer advantages in terms of reduced memory and computational requirements for generating high-quality interactive visualizations. For example, such visualizations can be generated in less time and with reduced memory usage compared to using more computationally intensive rendering processes.

[0056] Fig. 3 is a schematic drawing that illustrates aspects of an exemplary procedure according to Fig. Figure 1 illustrates how to generate a representation of a volumetric dataset.

[0057] In the example of Fig. In the first step, parameters are defined for performing a rendering process on a volumetric dataset representing medical volume data. The rendering process can be, for example, a physical rendering process, such as a Monte Carlo path tracing method. The parameters can include one or more bounding box settings, transfer function settings, lighting settings, and clip layer settings for the rendering process. These parameters provide an initial rendering configuration of the volumetric dataset. This initial rendering configuration can be designed to enable the transmission of specific visual information about the volumetric dataset.For example, the volumetric dataset can represent a patient's heart, and the initial rendering configuration 310 can be defined by a suitable bounding box that encloses the area to be included in the visualization, as well as a suitable transfer function, lighting, and other settings to provide a representation of the heart that conveys relevant visual information to a viewer. As an example, the initial rendering configuration 310 can include one or more clip layer settings that are appropriately positioned along with suitable transfer function settings to allow a specific internal structure of the heart to be shown in detail.

[0058] In some examples, the parameters defining the initial rendering configuration 310 can be manually set by a user. Additionally or alternatively, one or more of these parameters can be set automatically, for example, based on automatically generated landmarks and / or automatic contouring—that is, automatic identification of structures in the volumetric dataset that correspond to specific structures within the medical volume. Optimizations of the rendering parameters can, for example, utilize differentiable rendering methods.

[0059] In a second step, an initial set of several images 320 of the first rendering configuration 310 is rendered using the rendering parameters defined in the first step. These images 320 are generated to cover a range of different viewing directions from which the first rendering configuration 310 can be viewed. The images 320 can be rendered as high-resolution images, for example, using a physically based path tracing process. In examples where automatically generated landmarks or features identified by automatic contouring are present, the viewing directions can be optimized to display these landmarks or features in the best possible way. The set of images 320 can, for example, include multiple zoom-level renderings of specific features of interest in the volumetric dataset.The rendering parameters used to render the set of 320 images can be stored with each rendered image along with any information specific to a particular set of the 320 rendered images, such as camera position and direction.

[0060] In a third step, an initial function set 330, representing the initial rendering configuration 310, is created from the rendered images 320. The initial function set 330 can be designed to provide an approximation of the volumetric dataset in the initial rendering configuration 310. Each function in the initial function set 330 can have a 3D position relative to the volumetric dataset and has one or more visual attribute values, such as shape, color, and opacity. In one particular example, the functions in the initial function set 330 are Gaussian functions, so the initial function set 330 forms an initial Gaussian splat representation that represents the volumetric dataset in the initial rendering configuration 310. In certain examples, additional compression can be performed to reduce the memory size of the initial function set 330.

[0061] In addition to generating the first function set 330, one or more additional function sets, such as additional Gaussian splat representations, are generated. Each of these additional function sets can be generated in the same way as described above for the first function set 330. For example, an additional function set can be generated that is designed to represent a specific area of ​​interest within the volumetric dataset. This area of ​​interest can lie within the section of the volumetric dataset represented by the first function set 330. The area of ​​interest can, for example, represent a portion of the medical volume that is desired to be visualized in detail as a close-up.

[0062] In the Fig. In the example shown, an area of ​​interest 345 of the volumetric dataset is identified to be represented by an additional function set. In some examples, the area of ​​interest 345 may be a region that is desired to be represented at a higher resolution than that at which the volumetric dataset is represented in the first rendering configuration 310, which is represented by the first function set 330. In some examples, the area of ​​interest 345 can be identified by a user. In other examples, the volumetric dataset may already include representations of various areas of interest that can be represented at higher resolutions.For example, if the volumetric dataset was generated by a hierarchical imaging process, where the volumetric dataset comprises volume data arranged in a hierarchy, with areas of interest being sampled with increasing resolution, such as in hierarchical phase contrast CT imaging.

[0063] The first, second, and third steps described above can then be performed to generate a second set of functions 360 representing the area of ​​interest 345. That is, a second rendering configuration 340 is defined with a suitable set of rendering parameters for rendering images of the area of ​​interest 345, e.g., at high resolution. A second set of images 350 is generated by performing the rendering process, which can have any of the features described above in relation to the rendering process used to generate the first set of images 320. The rendering process can, for example, be a physical rendering process capable of generating high-resolution images of the area of ​​interest 345. The rendering parameters are stored in conjunction with the second set of images 350. The second set of images 360, e.g.,A second Gaussian splat representation is generated from the second set of images, number 350. These steps can be repeated for all other areas of interest within the volumetric dataset.

[0064] The first function set 330, the second function set 360, and all other generated function sets representing further areas of interest are used to determine given functions 380, which are to be stored in a representation 370 of the volumetric dataset. The given functions 380 are stored in the representation 370 in conjunction with selection data. In the Fig. In the third example, all functions from the different function sets 330 and 360 are determined as given functions 380, which are to be stored in representation 370. In other examples, only a subset of the functions from function sets 330 and 360 can be selected to be stored as functions 380 in representation 370. In still other examples, the stored functions 380 can be determined by mixing or otherwise combining functions from the different function sets 330 and 360.

[0065] Representation 370 is designed to be used to render images of the volumetric dataset using a further rendering process. Functions can, for example, be selected from the stored function 380 based on the selection data and used in a rasterization process to render images of the volumetric dataset. The selection data can, for example, display the respective bounding box settings, resolution information, and any other relevant information that was used to generate images 320 and 350, from which a specific set of functions 330 and 360 was generated.

[0066] In some examples, the functions 380 are stored hierarchically in representation 370. In such examples, the selection data can indicate a level in the hierarchy to which a specific function corresponds. The first function set 330, for example, can represent the entirety of the volume with a first, lowest resolution. The functions of the first function set 330 can accordingly be marked in representation 370 as belonging to a first level in the hierarchy. The second function set 360 can represent the area of ​​interest 345 with a second resolution that is higher than the first. The functions of the second function set 360 can accordingly be marked as belonging to a second level in the hierarchy. In some examples, there may be further areas of interest at the second level in the hierarchy.There may also be further levels in the hierarchy that represent one or more additional areas of interest.

[0067] Such a hierarchical representation can also be used to interactively render an image of the volumetric dataset from any viewpoint. For example, if a given view of an image, which is to be rendered using the further rendering process, allows the entire volume to be visualized, the first set of functions 330, which is marked as belonging to the first level in the hierarchy, can be selected from the stored functions 380 and used to render an image of the entire volume from the given viewpoint.If the user changes the view for the image to be rendered so that it focuses on the area of ​​interest 345 in the second level of the hierarchy, then the second set of functions 360 is selected from the stored functions 380 and used in the further rendering process to render an image of the area of ​​interest 345 with the higher resolution.

[0068] In some examples, a zoom level can be used to determine the appropriate level in the hierarchy from which to select the functions to use when rendering the image using the subsequent rendering process. In certain examples, the visualization can allow a smooth transition between different levels in the hierarchy by blending images rendered by functions from different levels. For example, if a user zooms into area of ​​interest 345, the zoom level can be used to smoothly transition between an image of the entire volume generated by functions corresponding to the first function set 330 at the first level in the hierarchy (e.g., a lower-resolution image) and an image of area of ​​interest 345 generated by functions corresponding to the second function set 360 at the second level in the hierarchy (e.g., a higher-resolution image).B. to a higher-resolution image). In some examples, areas of interest, such as the first area of ​​interest 345, which are represented by further sets of functions, can be marked in the visualization to allow a user to identify them. For example, areas of interest can be marked by a specific color or framed by a box.

[0069] Further parameter settings corresponding to different rendering configurations can be represented by various subsets of functions in Figure 370 in conjunction with appropriate selection data. For example, multiple rendering configurations may be defined that differ only in the position of the clip plane. The positions of the clip planes may be defined manually by a user during keyframe generation or automatically selected based on segmentation, landmark information, or application-specific optimal clip plane positioning. In a cardiac workflow, for instance, the clip plane might be automatically positioned using segmentation information or anatomical landmarks to show the heart chambers or valves. Each of these rendering configurations can be represented by a corresponding set of functions.These respective sets of functions can be generated in the same way as described above with reference to the first or second set of functions 330 and 360, and stored in representation 370 in conjunction with appropriate selection data. For example, if a clip layer position differs between rendering configurations, the respective clip layer settings can be stored in representation 370 along with the sets of functions. When using another rendering process to render an image from representation 370, a user adjustment of the clip layer setting to be visualized can allow the appropriate set of functions representing that clip layer setting to be selected from the functions stored in representation 370.

[0070] In some examples, a subset of these function sets can be selected to be stored in Representation 370, rather than storing all the functions representing the various parameter settings in Representation 370. For example, the functions of the function sets representing the different parameter settings, such as different clip level settings, can be compared to identify functions that are shared across the parameter settings. This might involve, for example, comparing functions from different parameter settings using a similarity metric. Functions shared across different parameter settings can be marked as global and stored in Representation 370 as belonging to a global function set.Additionally, one or more further sets of functions corresponding to each parameter setting can be stored in representation 370. Each of these sets can represent a difference between the set of functions representing the respective parameter setting and the global set of functions. For example, for a given clip level setting, only those functions that are sufficiently different from a function in the global set of functions can be stored in representation 370. These functions can be stored in conjunction with a label indicating the clip level setting to which they correspond. Each different parameter setting can therefore be represented compactly by storing in representation 370, in addition to the global set of functions, only those functions that are sufficiently different from the functions in the global set.

[0071] In such an example, when rendering an image of the volumetric dataset with a given parameter setting, such as clip layer position, all functions in the global set, as well as those stored for that parameter setting, can be selected. This can make the process of rendering an image from Figure 370 more efficient, since the functions in the global set can be reused between parameter settings. For example, adjusting a parameter setting of the visualization, such as moving a clip layer, might involve loading the subset of functions representing the new parameter setting into memory, rather than the subset representing the previous parameter setting. This can be more efficient than loading an entirely new set of functions corresponding to the new parameter setting into memory.As an example, such a technique can enable the 370-degree rendering to efficiently provide an interactive rendering process. The method can, for instance, efficiently provide a visualization that involves the interactive melting of a clip layer through the volume.

[0072] As an alternative to determining a global set of functions, all functions representing an initial parameter setting, such as an initial clip level setting, can be stored in representation 370. For example, each of several sets of functions corresponding to different parameter settings can be compared to the set of functions representing the initial parameter setting. Then, for each of these additional parameter settings, only those functions that are sufficiently different from the functions in the initial set can be stored.

[0073] Similar techniques to those described above regarding the representation of different rendering parameter settings can be used to represent different time steps in a volumetric 4D dataset. For example, distinct sets of functions can be generated to represent different time steps of a volumetric 4D dataset. These sets of functions can be stored along with selection data indicating the time step they correspond to in Figure 370. In some such examples, techniques like those described above can be used to efficiently encode the different time steps. For instance, a set of functions shared by all the functions of the different time steps can be stored as a global set of functions, with each of the time steps being encoded based on its difference from the global set of functions.Alternatively, all of the functions representing a first time step can be stored, with other time steps being represented only by those functions that differ from those of the first time step.

[0074] In some examples, certain functions may be encoded with differential values ​​that represent a difference between the function and a reference function, for example, a function from an earlier time step or a function at a different level in a hierarchy. In some such examples, each of the stored functions may be encoded with an additional value that represents a link to a reference function. Furthermore, in some examples, during rendering, a particular function may be transformed using interpolation to allow a smooth transition from one rendering configuration to another, such as from one time step to another or from one level in the hierarchy to another.

[0075] Any of these techniques can be used in combination. For example, in examples like the one described above, where a hierarchical representation of a volumetric dataset is generated in a specific rendering configuration, the functions stored for the different levels in the hierarchy can use a technique to encode a global set of functions along with a different set of functions for each level in the hierarchy, or a technique to encode a complete representation of a particular level in the hierarchy and, for other levels in the hierarchy, only encode a difference from those encoded for the particular level.

[0076] In some examples, the process of generating images 320, 350 and the respective sets of functions 330, 360 for representing different rendering configurations may involve techniques to reduce the amount of processing required. For example, if the different rendering configurations represent different positions of a clip layer, a set of combinations of clip layer position and target view can be determined to be used to generate the images for storing the functions in representation 370. Such a set of combinations comprises all possible combinations of clip layer position and target view. However, it may be the case that not all of the clip layer positions affect the image generated for a particular target view. For example, for a given set of target views, the surface affected by the clip layer may not be visible because, for instance, it is obscured by a surface.is obscured by other structures in the volume. As such, a slight change in the clip layer position for this target view will not affect the final rendered image. Such combinations of target view and clip layer can be omitted from the set of combinations used for rendering images. Images can then be generated for the determined set of combinations of clip layer position and target view. This can improve efficiency in the process of generating Representation 370 because it reduces the number of images that need to be rendered to generate the features to be stored in Representation 370. These images can then be used to generate a set of features to be stored in Representation 370 of the volumetric dataset, as described above. In such an example, generating the features involves, for example,Training a Gaussian splat representation, an iterative training of the function sets, using differentiable rendering to accurately represent the clipped target views when the same clip layer position is applied.

[0077] The exemplary methods described herein can be used in various scenarios where it is desirable to enable the interactive generation of images from the volumetric dataset without distributing the volumetric dataset itself. This can allow for the interactive rendering of high-quality images while limiting the computing power and / or memory requirements of a device for rendering the images. Furthermore, not distributing the volumetric dataset itself protects the source data of the volumetric dataset.

[0078] As an example, in a teacher-student scenario, an instructor may have prepared anatomical volume data, e.g., from a CT scan, with optimal keyframes for visualizing various anatomical structures and pathologies. The instructor may, for example, define optimal transfer function and clip layer settings for visualizing these structures and pathologies. For each of these keyframes, a 370° representation, as described above, can be generated using the procedures outlined herein and distributed to students for learning and exploration at home via a website, social media, or a personal mobile or immersive display device.For example, using the 370 display, students can generate high-quality images of the keyframe from any angle and also modify certain rendering parameter settings, such as the clip layer position, to provide an interactive visualization. This can enhance the learning experience.

[0079] As another example, in a clinical setting, a radiologist and a surgeon could collaborate to generate one or more keyframes with optimized parameters to depict a specific pathology. This could be done on a high-performance rendering device, such as a desktop application. A representation of these keyframes could then be generated using the procedure described above. This representation could then be transferred to a display device in the operating room, such as a tablet computer, stereo display, AR glasses, or similar, to enable the generation of a high-quality, interactive visualization for use in the operating room.

[0080] As another example, the representation 370 could be transferred to the local memory of an AR device. The AR device can then render images from the representation 370 using minimal bandwidth, memory, and rendering power requirements, allowing the visualization to be explored interactively.

[0081] In certain examples, additional functions for the 370 display can be transferred as needed, for example, if parameter settings are changed or if an area of ​​interest or a higher resolution area is being explored by the user who is interactively examining the visualization.

[0082] Fig. Figure 4 is a schematic drawing illustrating an exemplary system 401 in which an exemplary setup 404 can use the procedures described herein. The system 401 comprises a scanner 402, the setup 404, and a visualization unit 414. In examples, the system may have fewer components than, or additional components to, those described in Figure 401. Fig. The four illustrated examples include: For instance, System 401 can encompass a computer network, such as the Internet.

[0083] The scanner 402 can be any scanner for generating a data set that includes a volumetric data set 410, which, for example, could be a medical volumetric data set representing a section of a patient. The scanner 402 can be a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a positron emission tomography (PET) scanner, or the like. The scanner 402 is connected to the device 404, for example, via a wired or wireless connection. The scanner 402 can be configured to provide the device 404 with the volumetric data set 410.

[0084] The device 404 comprises one or more processors 406 and memory in the form of one or more storage devices 408. The device 404 may, for example, include a GPU. In this example, the device 404 is designed to perform a procedure according to the examples described above to generate a representation of the volumetric dataset 410 for use in rendering images of the volumetric dataset 410. The device 404 may, for example, include one or more processors to perform the various aspects of the procedure. The device 404 may, for example, include a volume rendering module to perform a process for rendering the images used to generate the representation of the volumetric dataset 410, for example, by performing a physical rendering process on the volumetric dataset 410 with different rendering parameter settings.

[0085] The storage medium 408 can comprise a machine-readable medium containing a set of machine-readable instructions which, when executed by the processor 406, cause the device 404 to perform an exemplary procedure described herein. The program can be stored on a computer-readable medium that can be read by the device 404 to execute the program. The device 404 can be configured to receive the volumetric data set 410 directly or indirectly, or otherwise acquire it from the scanner 402. The device 404 can be configured to use the volumetric data set 410 to render images and to use these images in a procedure such as described above to generate a representation of the volumetric data set. In other examples, the scanner 402 can be omitted, and the volumetric data set 410 can be obtained from an external source.

[0086] The device 404 can be configured to transmit information, for example, values ​​that define the representation of the volumetric dataset 410, such as a Gaussian splat plot containing associated selection data, to a visualization unit 414. The transmission can be direct or indirect, for example, via a wired connection, a wireless connection, or the internet.

[0087] The visualization unit 414 may include visualization software for displaying a two-dimensional projection of the volumetric dataset 410 using the representation generated by the device 404. For example, the visualization unit 414 may include a device for performing a rendering process, such as a rasterization process, to interactively render images from the representation of the volumetric dataset 410. The visualization unit 414 may include a display screen and one or more graphics hardware or software components. In some examples, the visualization unit 414 may be or include a mobile device. In some examples, the visualization unit 414 may include a virtual reality or augmented reality device. The visualization unit 414 may display a stereoscopic image in some examples.In other examples, the facility 404 or another facility between the facility 404 and the visualization unit 414 can perform a rendering process on the display and provide the generated data to the visualization unit 414 for display.

[0088] While in certain examples described above the functions stored in the representation of the volumetric data set are Gaussian functions, in other examples the functions may be of a different type; for example, the functions may represent one or more neuronal radiation fields.

[0089] In certain examples described above, the procedure for generating the representation of the volumetric dataset involves performing a rendering process to render the images of the volumetric dataset in the various rendering configurations used to generate the representation. In other examples, however, the images used to generate the representation can be obtained from a different source, such as an external rendering facility or suitable storage that contains pre-rendered images of the volumetric dataset in various rendering configurations.

[0090] The embodiments described above are to be understood as illustrative examples of the invention. Other embodiments are considered. It is understood that any feature described in relation to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment or any combination of any other embodiments. Furthermore, equivalents and modifications not described above may also be used without departing from the scope of protection of the invention as defined in the appended claims.

[0091] Regardless of the grammatical usage, the term includes persons of male, female or other gender identities.

Claims

[1] Computer-implemented method (100) for generating a representation of a volumetric data set (410) representing a medical volume for use in a rendering process, the method comprising: Generating (102) a multitude of function sets (330, 360) that represent a volumetric dataset (410) in different respective rendering configurations (310, 340); Determine (104), based on the multitude of function sets (330, 360), of given functions (380) with a respective 3D position in relation to the volumetric dataset (410) and a respective one or more visual attribute values, which are to be stored in a representation (370) of the volumetric dataset (410) for use in a rendering process; and Storing (106), in the representation (370) of the volumetric dataset (410), the given functions (380) in conjunction with selection data designed to allow a selection of one or more of the given functions (380) for use in the rendering process, wherein the selection data is based on one or more of the rendering configurations (310, 340). [2] Method according to claim 1, wherein the different rendering configurations (310, 340) comprise a first rendering configuration (310) representing a first section of the volumetric data set (410) with a first resolution, and a second rendering configuration (340) representing a second section (345) of the volumetric data set (410) different from the first section with a second resolution higher than the first resolution. [3] Method according to claim 1 or claim 2, wherein the given functions (380) are Gaussian functions and the representation (370) of the volumetric data set (410) is a Gaussian splat representation. [4] Method according to any one of claims 1 to 3, wherein one or more of the different rendering configurations (310, 340) differ from each other with respect to one or more of the following: a clipping layer setting; a transfer function setting; a lighting setting; and a bounding box setting. [5] Method according to any one of claims 1 to 4, comprising: Comparing one or more functions of different sets of the multitude of function sets (330, 360); and where determining the given functions (380) to be stored in the representation (370) of the volumetric data set (410) is based on a result of comparison. [6] Method according to claim 5, wherein the comparison includes determining respective similarities between functions of the different sets of functions (330, 360) and the determination of one or more of the given functions (380) is based on the determined similarities. [7] Method according to any one of claims 1 to 6, wherein the volumetric data set (410) is a volumetric 4D data set and the different rendering configurations (310, 340) represent different time steps of the volumetric 4D data set. [8] Method according to any one of claims 1 to 7, wherein generating the plurality of function sets (330, 360) comprises: Generating each set of the multitude of function sets based on each of the multitude of rendered images (320, 350) showing the volumetric dataset (410) from different viewpoints in a given rendering configuration (310, 340). [9] The method of claim 8, wherein the method comprises: Generating the multitude of sets of rendered images (320, 350) using a respective rendering process with one or more respective rendering parameters; and optionally: Determining the selection data based on one or more of the respective rendering parameters. [10] Method according to claim 8 or claim 9, comprising: Determining combinations of viewpoint and respective rendering configurations of the various rendering configurations (310, 340) that are designed, when used in a rendering process, to provide images that differ from one another with respect to a visible aspect of the volumetric dataset (410); and Determine, based on the determined combinations, the respective different viewpoints for the respective multitude of images (320, 350) that show the volumetric data set (410) in the given rendering configuration. [11] Method according to any one of claims 8 to 10, wherein one or more of the respective rendering processes is a physical rendering process. [12] Computer-implemented method (200) for rendering an image of a volumetric data set (410) representing a medical volume, the method comprising: Performing (202) a rendering process using a representation (370) of a volumetric dataset (410) representing a medical volume, wherein the representation (370) of the volumetric dataset (410) comprises given functions (380) with a respective 3D position in relation to the volumetric dataset (410) and a respective one or more visual attribute values; wherein the given functions (380) are stored in the representation (370) in conjunction with selection data designed to allow the selection of one or more of the given functions (380) for use in the rendering process; and wherein the rendering process comprises: selecting, based on the selection data and one or more parameters of the rendering process, one or more functions of the given functions (380) for use in the rendering process; and rendering an image of the volumetric data set (410) using the selected one or more functions of the given functions (380). [13] A set of machine-readable instructions which, when executed by a processor (406), cause a method (100) according to any one of claims 1 to 11 and / or a method (200) according to claim 12 to be carried out. [14] Machine-readable medium, comprising: a representation (370) of a volumetric data set (410) representing a medical volume, wherein the representation (370) can be generated by a method according to any one of claims 1 to 11; and / or a set of machine-readable instructions according to claim 13. [15] Institution (404), comprising: a processor (406); and a storage (408), comprising: a representation (370) of a volumetric data set (410) representing a medical volume, wherein the representation (370) can be generated by a method according to any one of claims 1 to 11; and / or a set of machine-readable instructions according to claim 13.

Images