A computer implementation method for generating volume datasets representing medical volumes for use in rendering processes.

Abstract

A computer implementation method for generating volume datasets representing medical volumes for use in rendering processes. [Solution] A computer implementation method for generating a display of a volume dataset representing medical volume for use in rendering. Multiple sets of functions representing the volume dataset are generated in different rendering configurations, and a given function is determined based on the multiple sets of functions, having a 3D position for each volume dataset and one or more respective visualization attribute values ​​to be stored during the display. The given function is stored in the display in association with selection data configured to allow selection of one or more of the given functions for use during the rendering process. The selection data is based on one or more rendering configurations. A computer implementation method for rendering an image of a volume dataset representing medical volume using the above display is also described.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for generating a display of a volume dataset representing a medical volume for use in a rendering process. The present invention also relates to a method for rendering an image of a volume dataset representing a medical volume using the above display. Further, the present invention relates to an apparatus for implementing any of these methods, a set of machine-readable instructions, and a machine-readable medium including the above instructions.

Background Art

[0002] When imaging medical volume data by computer implementation, there are problems in terms of data size in order to enable interactive or high-quality visualization so that medical data can be quickly available. In particular, there are problems when rendering data generated by high-resolution imaging modalities (for example, 7T MRI, photon-counting CT, phase-contrast CT). Due to the huge amount of data generated by these imaging modalities, there are problems during transfer and / or rendering of image data, for example, problems caused by limited bandwidth, limited rendering capabilities, and limited memory size. Such problems may become apparent especially in clinical workflows and learning platforms for anatomical structures where computational (computer processing) and time resources are limited.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In a first aspect of the present invention, there is provided a computer-implemented method for generating a display of a volume dataset representing a medical volume for use in a rendering (or drawing) process. In this method, a plurality of sets of functions representing the volume dataset are generated in different rendering configurations, Based on multiple sets of the above functions, a given function is determined that has the respective 3D position of the volumetric dataset and the respective visualization attribute values ​​of one or more volumestric datasets stored during the rendering process. The display of the volume dataset includes storing the given functions in association with selection data configured to allow selection of one or more of the given functions for use during the rendering process, wherein the selection data is based on one or more of the rendering configurations.

[0004] The different rendering configurations described above may include a first rendering configuration that represents a first portion of the volumetric dataset at a first resolution, and a second rendering configuration that represents a second portion of the volumetric dataset, different from the first portion, at a second resolution higher than the first resolution.

[0005] The given function above may be a Gaussian function, and the above representation of the volume dataset may be a Gaussian-splat representation.

[0006] One or more of the above different rendering configurations may differ from one or more of the clipping plane settings, transfer function settings, lighting settings, and bounding box settings.

[0007] The above method may include comparing one or more functions from different sets of the above sets of functions, and determining, based on the results of the above comparison, a given function to be stored during the display of the volume dataset.

[0008] The above comparison may include determining the similarity between different sets of functions, and based on the determined similarity, one or more of the given functions may be determined.

[0009] The volumetric dataset described above may be a 4D volumetric dataset, and the different rendering configurations described above may represent different time steps of the 4D volumetric dataset.

[0010] Generating multiple sets of the above functions may involve generating each set of the above functions based on each of multiple rendered images that show the volumetric dataset in a given rendering configuration from different perspectives.

[0011] The above method may include generating multiple sets of rendered images using each rendering process having one or more rendering parameters. The above method may also include determining the selected data based on one or more of the rendering parameters.

[0012] The above method may include determining a combination of rendering configuration and viewpoint for each of the different rendering configurations, which are configured to provide images that differ from each other in terms of the visibility of the volumetric dataset when used during the rendering process, and, based on the determined combination, determining a different viewpoint for each of the multiple images that show the volumetric dataset in a given rendering configuration.

[0013] One or more of the above rendering processes may be physical rendering processes (or physics-based rendering processes).

[0014] A second aspect of the present invention provides a computer implementation method for rendering an image of a volume dataset representing medical volume. The method performs a rendering process using a display of a volume dataset representing medical volume, wherein the display of the volume dataset includes a given function having a 3D position and one or more visualization attribute values ​​for each of the volume datasets, the given function is stored in the display and associated with selection data configured to allow selection of one or more of the given functions for use in the rendering process, the rendering process includes selecting one or more of the given functions for use in the rendering process based on the selection data and one or more parameters of the rendering process, and rendering an image of the volume dataset using the selected one or more of the given functions.

[0015] In a third aspect of the present invention, a set of machine-readable instructions is provided, which, when executed by a processor, is configured to perform either or both of the methods according to the first aspect of the present invention and the methods according to the second aspect of the present invention.

[0016] In a fourth aspect of the present invention, a machine-readable medium is provided, which is configured to include both or either a display of a volume dataset representing medical volume, which can be generated by a method according to the first aspect of the present invention, and a set of machine-readable instructions according to the third aspect of the present invention.

[0017] In a fifth aspect of the present invention, an apparatus is provided, comprising a processor and a memory device, the memory device being configured to include either or both of a volume dataset representation representing medical volumes and a set of machine-readable instructions according to a third aspect of the present invention, which can be generated by a method according to a first aspect of the present invention. The present invention will be described below with reference to the attached drawings, which are merely illustrative examples. [Brief explanation of the drawing]

[0018] [Figure 1] Figure 1 is a flowchart illustrating a computer implementation method for generating a volume dataset representing medical volume for use in rendering. [Figure 2] Figure 2 is a flowchart illustrating a computer implementation method for rendering an image of a volumetric dataset using the display generated according to the method in Figure 1. [Figure 3] Figure 3 is a schematic diagram showing an embodiment of the method illustrated in Figure 1. [Figure 4] Figure 4 is a schematic diagram showing an apparatus for carrying out the exemplary method described herein. [Modes for carrying out the invention]

[0019] Figure 1 is a flowchart showing an example of a computer implementation method 100 for generating a volume dataset (or set of volumetric data) representing medical volume (or volume used in medical treatment) for use in rendering (or drawing) processes.

[0020] Volumetric datasets may include discrete sampling of scalar fields. For example, volumetric datasets may be obtained by reading from memory, sensors, and / or other sources. The medical volume represented by the volumetric dataset may include, for example, a patient or a portion of a patient, where the patient may be, for example, a human patient or an animal patient.

[0021] In general, any suitable scanning modality (or scanning modality) can be used to generate a volumetric dataset. For example, the scanning modality can be computed tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET). In some examples, the scanning modality can be a high-resolution imaging modality such as 7T MRI (7 Tesla magnetic resonance imaging), photon counting CT, or phase-contrast CT. In some examples, the scanning modality can be single-photon emission computed tomography (SPECT), ultrasound, or another scanning modality. In some examples, the volumetric dataset may include data from multiple of the above scanning modalities. For example, multiple sets of volumetric data such as PET, CT, and MRI can be combined to obtain a volumetric dataset. The scan data may be provided in the form of multiple two-dimensional (2D) scans, or it may be formatted from the scans. In some examples, a volumetric dataset may include a group of 2D image slices. These 2D image slices may include, for example, compressed slice data in JPEG2000 format. In some examples, a volumetric dataset is a DICOM dataset created by scanning at least a portion of a patient using one or more scanning modalities.

[0022] A volume dataset can include data formatted as a plurality of voxels (or three-dimensional elements). The voxels may be arranged, for example, in a uniform or non-uniform grid (checkerboard pattern), or in other types of geometric shapes (e.g., polar coordinate format). The voxels may be isotropic or anisotropic. Each voxel can typically represent a scalar value obtained by sampling a scalar field, although in some examples, the volume dataset can also include data regarding a non-scalar field. The type of value represented by each voxel can be based on the means by which the volume dataset is acquired. For example, if a CT scanner is used to generate the volume dataset, the dataset can include Hounsfield values. The value represented by each voxel is represented with a given precision and may be represented, for example, in 8 bits or 16 bits.

[0023] In some examples, the volume dataset may be a 4D volume dataset. For example, the volume dataset can include a series of 3D volume datasets, where each set can represent a medical volume at a different time step. Each 3D volume dataset can, for example, have any of the features described above for the volume dataset.

[0024] The method 100 generates, at block 102, a plurality of sets of functions representing a volume dataset, where each rendering configuration is different.

[0025] The different rendering configurations may be different, for example, with respect to one or more of the clip plane setting, transfer function setting, lighting setting, and bounding box setting. In addition to or instead of that, the different rendering configurations may be associated with different time steps in the 4D volume dataset.

[0026] For example, different rendering configurations may represent different parts of a volumetric dataset. These different parts of the volumetric dataset may be represented at different resolutions. For instance, in a first rendering configuration, the first part of the volumetric dataset (e.g., representing the entire medical volume) may be represented at a first resolution, while in a second rendering configuration, the second part of the volumetric dataset (e.g., representing the region of interest of the medical volume) may be represented at a second resolution. The first and second parts of the volumetric dataset may correspond to different bounding box settings. The second resolution may be higher than the first resolution, resulting in, for example, a particular region of interest in the volumetric dataset being represented at a higher resolution.

[0027] In another example, in the first rendering configuration, the volumetric dataset may be represented by a first clipping plane setting. The second rendering configuration can correspond to a second clipping plane setting. For example, a given clipping plane may be positioned differently in the second rendering configuration compared to its position in the first rendering configuration.

[0028] In another example, any other parameter or any combination of parameters may differ between different rendering configurations.

[0029] Generating multiple sets of functions representing a volumetric dataset in different rendering configurations may involve generating each set of functions based on multiple rendered images. These multiple images can represent the volumetric dataset in a given rendering configuration from different viewpoints.

[0030] For example, a first set of images used to generate a first set of functions could represent a volumetric dataset in a first rendering configuration from multiple different viewpoints. The first rendering configuration could correspond to a first set of rendering parameters for the rendering process used to render the first set of images of the volumetric dataset. For example, the first rendering configuration could correspond to a first given combination of bounding boxes, clipping planes, and transfer function settings used in the rendering process to generate the first set of images.

[0031] An image representing a volumetric dataset in a given rendering configuration may be generated by performing a rendering process using a given set of rendering parameters. For example, this rendering process may be a physical (or physics-based) rendering process. For example, this rendering process may be a path tracing process. In another example, this rendering process may be a non-physical rendering process, such as ray casting. This rendering process may be repeated from several different viewpoints to obtain an image of the volumetric dataset in a given rendering configuration. This process may be repeated for one or more other rendering configurations to obtain multiple images representing the volumetric dataset in each of the different rendering configurations. For example, different parts of the volumetric dataset can be rendered at different resolutions using different bounding box settings. Alternatively, for example, the volumetric dataset can be rendered at different clipping plane positions or orientations using different clipping plane settings. Furthermore, the volumetric dataset can be rendered with different visual characteristics using different transfer function settings. Any one or any combination of these three methods can be used.

[0032] A given set of functions generated in block 102 provides a representation of the volumetric dataset in a given rendering configuration in which that set of functions corresponds. For example, a given set of high-fidelity (or high-quality) images can be rendered using path tracing, thereby showing the volumetric dataset in a given rendering configuration from a limited number of angles. A set of functions can be generated from multiple images that provide a 3D representation of the volumetric dataset in a given rendering configuration. These functions can be used in rendering processes, such as rasterization, to render images of the volumetric dataset in a given rendering configuration from any given angle. These functions can provide a compact representation of the volumetric dataset in a given rendering configuration. For example, a set of functions can occupy substantially less space when saved (or stored) compared to the volumetric dataset itself.

[0033] In rendering configurations for volumetric datasets used to render multiple images to generate a given set of functions, the volumetric dataset can be divided (or segmented) to which color, opacity, and transfer functions specific to the division mask may be applied.

[0034] In some examples, a set of functions representing a volumetric dataset in a given rendering configuration may be a point cloud representation. That is, each function can form points in the point cloud that have 3D locations relative to the volumetric dataset. Each function may have one or more visualization attribute values, such as shape, color, and opacity.

[0035] In the example above, the set of functions representing a volumetric dataset in a given rendering configuration is a set of Gaussian functions (or Gaussian functions) that form a Gaussian splat (or Gaussian splatting) of the volumetric dataset in a given rendering configuration. For example, if a given set of functions is a Gaussian splat containing a set of Gaussian functions, then each function may include the following parameters: - For example, mean μ, which can be interpreted as a 3D position in a volumetric dataset, including x, y, z Cartesian coordinates (or rectangular coordinates); - Covariance Σ that can be interpreted as the shape of a Gaussian function; -Opacity σ(α), for example, a sigmoid function applied to map the parameter to the interval [0,1], - One or more color parameters, for example, three values ​​of (R, G, B), or spherical harmonic coefficients.

[0036] A set of Gaussian functions representing a volumetric dataset in a given rendering configuration, as described above, can be obtained by performing a training process using images(s) that represent the volumetric dataset in the given rendering configuration. The training process may include optimizing the parameters of the Gaussian functions and the number of Gaussian functions in the set, for example, using a differentiable rendering method, such as gradient descent. For example, the training process may include initializing the set of Gaussian functions representing the rendering configuration and rendering an image from the set of Gaussian functions using a rasterization technique. The rasterization technique may include GPU sorting (or selection) and rasterization of projected 2D Gaussians, as well as the use of pixel shaders to evaluate and combine 2D projections in image space. In one particular example, a rasterization method for rendering from a Gaussian splat display involves tiling the screen space and determining a visually permissible Gaussian function within a conical view for each tile. These Gaussian functions are then labeled according to their depth. For each tile, the associated Gaussian function is loaded into memory and, for a given pixel color and alpha (i.e., opacity) value, is accumulated by traversing the associated Gaussian function back and forth with respect to depth until the target saturation of the alpha of that pixel (or object) is reached.

[0037] Images rendered from a Gaussian function can be compared to the original rendered images of the volumetric dataset in a given rendering configuration. The parameters of the Gaussian function can be adjusted to better represent the original image. Sequential iterations of this process can be performed to optimize the set of Gaussian functions for representing the volumetric dataset in a given rendering configuration, for example, by using probabilistic (or inferential) gradient descent. When generating the set of Gaussian functions for representing a given rendering configuration, any of the techniques described in one or both of the following documents can be used, for example. [Non-Patent Document 1] "3D Gaussian Splatting for Real-Time Radiance Field Rendering," by Bernhard Kerbl, Georges Kopanas, Thomas Leimkuhler, and George Drettakis, SIGGRAPH 2023 (ACM Transactions on Graphics). [Non-Patent Document 2] "Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis," by Simon Niedermayr, Josef Stumpfegger, and Rudiger Westermann, CVPR 2024.

[0038] In block 104, the method 100 determines, based on a set of functions, a given function to be stored during the display of the volumetric dataset used in the rendering process. Each of the given functions to be stored has a 3D position relative to the volumetric dataset and one or more visualization attribute values ​​(e.g., color, shape, or opacity).

[0039] In some examples, a given function stored during the display of a volumetric dataset is one of several functions from a set of functions generated in block 102. For example, a given function may be selected from several functions from a set of functions generated in block 102. The given function may be a Gaussian function, and the display of the volumetric dataset in which the given function is stored may be a Gaussian splat display.

[0040] In one example, among multiple sets of functions, the first set of functions represents the first part of the volumetric dataset at a first resolution, and among multiple sets of functions, the second set of functions represents the second part of the volumetric dataset at a second resolution. Then, one or more functions from each of the first and second sets may be determined to be the functions to be stored during the display of the volumetric dataset. In the above example, some or all of the functions from the first and second sets of functions may be selected to be the functions to be stored during their display.

[0041] In some examples, the display of a volumetric dataset may include different subsets of functions that represent the volumetric dataset in different rendering configurations. For example, as described above, if the first and second rendering configurations represent the volumetric dataset with different clipping plane settings, the first subset of a given function may represent the volumetric dataset in the first rendering configuration with the first clipping plane setting, and the second subset of a given function may represent the volumetric dataset in the second rendering configuration with the second clipping plane setting. Similarly, if the volumetric dataset is a 4D dataset with different time steps, the first subset of a given function may represent the first time step of the volumetric dataset, and the second subset of a given function may represent the second time step of the volumetric dataset.

[0042] In some examples, the given function to be stored may be determined by comparing functions from multiple sets of functions representing different rendering configurations. In block 104, determining the given function to be stored during the display of the volumetric dataset may be based on the results of that comparison. For example, the comparison may involve determining the similarity (or degree of similarity) between multiple functions from different sets of functions. The function to be stored during the display may be determined based on that determined similarity.

[0043] For example, the functions in a second set of functions representing a first rendering configuration may be compared with the functions in a first set of functions representing a second rendering configuration to determine their similarity. Functions in the second set of functions that are determined to have a given level of similarity (e.g., a similarity metric exceeding a given threshold) can be identified with functions in the first set of functions. For example, all functions in the first set of functions can be determined as a first subset of the given functions to be stored. On the other hand, only the functions in the second set of functions that do not reach a given level of similarity with respect to the functions in the first set of functions (in other words, functions that are sufficiently different from the functions in the first set of functions) can be determined as a second subset of the given functions to be stored. In this way, the second subset of the given functions can represent the second rendering configuration more efficiently by simply encoding the differences between the second and first rendering configurations. In some examples, the techniques described above may be used to determine a given function in an efficient manner to represent different rendering configurations (e.g., different clipping plane settings, different bounding box or transfer function settings, different time steps in a 4D volumetric dataset, or any two or more combinations thereof). Examples of such techniques are described in more detail below.

[0044] In some examples, some of the functions to be stored do not have to be selected from different sets of functions representing different rendering configurations, but may be determined from different sets of functions in other ways. For example, two or more functions from different sets of functions may be used to determine a given function to be stored. For example, a function from a first set of functions and a function from a second set of functions may be combined to obtain a given function to be stored during the display of a volumetric dataset.

[0045] In block 106, method 100 saves a given function in association with selected data while displaying a volumetric dataset. This selected data is configured to allow the selection of one or more of a given function to be used in the rendering process during the rendering process. This selected data is based on one or more rendering configurations.

[0046] In some examples, the selection data includes a label for each function of a given function. Each label may be used to determine whether a given function should be selected when rendering an image from a display of the volumetric dataset. For example, each function may be stored associated with a label indicating the subset of functions to which it belongs. For example, if a given function includes functions representing different parts of a volumetric dataset at different resolutions, each function may be labeled according to the part of the volumetric dataset it represents and / or the resolution it corresponds to. In some examples, a hierarchical structure of functions representing different parts of a volumetric dataset at different resolutions may be provided, in which case each function may be labeled according to the level of the hierarchy to which it corresponds. Alternatively, or additionally, a given function may be labeled according to a time step within a corresponding set of time steps, or according to any other parameter settings such as clip plane settings, transfer function settings, or illumination settings.

[0047] Figure 2 is a flowchart illustrating an example of a representative computer implementation method 200 for rendering an image of a volumetric dataset representing medical volume. In this method 200, the rendering process is performed in block 202 using a representation of the volumetric dataset. The representation of the volumetric dataset may include any of the above-described features of the representation generated by method 100 in Figure 1. The representation may be generated by method 100 as described with reference to Figure 1.

[0048] In one example, the rendering process described above selects one or more functions based on one or more rendering parameters and selected data. The selected functions can then be used to generate visual parameter values ​​for images of a volumetric dataset.

[0049] Rendering parameters for the rendering process used to select one or more functions may correspond to any settings that represent a specific rendering configuration in which the volumetric dataset is viewed (or visualized). For example, one or more functions for rendering the image may be selected using any one or more of the following: clipping plane settings, transfer function settings, or settings that indicate the region of the volumetric dataset to be viewed, such as bounding box settings or zoom settings. Alternatively or additionally, rendering parameters that indicate the time step of the 4D volumetric dataset to be viewed may be used to select from a given function.

[0050] The selected function may be used to generate an image using any appropriate rendering process. For example, the rendering process may include rasterization. The rasterization process may have any of the characteristics described above with respect to the rasterization process used to generate a different set of functions (on which the functions stored during display are based).

[0051] A given function is stored in association with selected data during the display of the volumetric dataset, so that during the rendering process, an appropriate function can be selected to visualize the volumetric dataset in a given rendering configuration. For example, the user can adjust rendering parameters by zooming in on or selecting a region of interest in the volumetric dataset. Then, using appropriate rendering parameters such as the zoom level or selected bounding box settings, a given function can be selected to be used to render an image of the region of interest based on the selected data. In another example, the user can adjust the clipping plane setting, which can then be used to select a given function appropriate to represent the volumetric dataset using the desired clipping plane setting, based on the selected data. In yet another example, during the visualization of 4D volumetric data, a time step setting indicating the time step being visualized can be adjusted, which can then be used to select a given function representing the volumetric dataset at a given time step.

[0052] In this way, by providing a compact display of volumetric datasets, users can visualize volumetric datasets in a computationally and memory-efficient manner, while also providing a high level of interactivity for visualization.

[0053] Functions that provide a representation of a volumetric dataset may, for example, provide a compact representation in terms of memory (or storage), and may enable fast rendering of an image of the volumetric dataset in a given rendering configuration from any viewpoint. For example, these functions may each provide a Gaussian splat representation of the volumetric dataset in several different rendering configurations. This selection data may enable interactive visualization by selecting an appropriate function to visualize the volumetric dataset in a desired rendering configuration. The rendering configuration to be visualized may be selected by the user or adjusted during rendering, for example by changing the rendering parameters of the rendering process used to render the image.

[0054] A display of a volumetric dataset can be reduced in size compared to the volumetric dataset itself. Therefore, even when providing the entire volumetric dataset is impractical or impossible due to limited memory or bandwidth, a display of the volumetric dataset can be transmitted to the device. Furthermore, in some cases, the display enables the rendering of high-quality visualizations using devices with limited rendering capabilities or memory size. This is because the device can render high-quality visualizations from the display using, for example, rasterization, without performing computationally intensive rendering processes such as path tracing.

[0055] Therefore, in some cases, high-quality interactive visualizations of the largest volume datasets can be provided to mobile devices or mobile devices, or immersive devices such as light field displays or AR / VR headsets. As a result, the mobility or portability of such visualizations is improved, for example, allowing surgeons to use high-quality interactive visualizations generated by mobile devices in the operating room, or enabling students to learn or conduct research using web pages, social media, personal mobile devices, or immersive display devices. Alternatively, even in the case of powerful rendering devices capable of processing large volume datasets and performing high-quality rendering (e.g., path tracing-based rendering), applying the method described herein offers advantages in terms of limited memory and the computational processing requirements required to generate high-quality interactive visualizations. For example, the visualizations described above can be generated in less time and with lower memory usage requirements compared to using more computationally intensive rendering processes.

[0056] Figure 3 is a schematic diagram illustrating an exemplary method according to Figure 1 for generating a display of a volumetric dataset.

[0057] In the example in Figure 3, the first step sets parameters for performing a rendering process on a volumetric dataset representing medical volumetric data. The rendering process may be a physical rendering process, such as Monte Carlo path tracing. The parameters may include one or more bounding box settings, transfer function settings, illumination settings, and clipping plane settings for the rendering process. These parameters provide a first rendering configuration of the volumetric dataset. This first rendering configuration 310 may be a rendering configuration of a volumetric dataset configured to convey predetermined visualization information about the volumetric dataset. For example, the volumetric dataset may represent a patient's heart, and the first rendering configuration 310 may be defined by a suitable bounding box enclosing the region included in the visualization, as well as a suitable transfer function, illumination, and other settings that provide a display of the heart to convey relevant visual information to the viewer. For example, the first rendering configuration 310 may include one or more appropriately placed clipping plane settings along with a suitable transfer function setting to allow detailed visualization of specific internal structures of the heart.

[0058] In some examples, the parameters defining the first rendering configuration 310 may be manually set by the user. In addition to this, or instead, the setting of one or more parameters defining the first rendering configuration 310 may be performed automatically, for example, based on automatically generated markers and / or automatic contours, that is, based on the automatic identification of structures in the volume dataset corresponding to specific structures of medical volumes. For example, differentiable rendering methods can be used to optimize the rendering parameters.

[0059] In the second step, a first set 320 of multiple images of the first rendering configuration 310 is rendered using rendering parameters set in the first step. These images 320 are generated to cover a range of various viewing directions (or orientations) in which the first rendering configuration 310 is visible. The images 320 can be rendered as high-resolution images, for example, using physical path tracing. If there are features identified by automatically generated markers or auto-contours, the viewing directions may be optimized to display these markers or features in the most optimal way. For example, the set of images 320 may include zoom levels that render multiple of a given feature of interest in the volumetric dataset. The rendering parameters used to render the set of images 320 may be associated with each rendered image and stored along with any information specific to one particular rendered image 320 (e.g., camera position and orientation).

[0060] In the third step, a first set 330 of functions representing a first rendering configuration 310 is created from the rendered image 320. This first set 330 of functions can be configured to provide an approximation of the volumetric dataset in the first rendering configuration 310. Each function in the first set 330 of functions may have a 3D position relative to the volumetric dataset and one or more visualization attribute values ​​such as shape, color, and opacity. In one particular example, the function(s) in the first set 330 of functions are Gaussian functions, so that the first set 330 of functions forms a first Gaussian splat representation of the volumetric dataset in the first rendering configuration 310. In other examples, additional compression may be performed to reduce the memory size of the first set 330 of functions.

[0061] In addition to generating the first set of functions 330, one or more additional sets of functions, for example, each additional Gaussian splat representation, are generated. These further sets of functions may each be generated in the same manner as described above for the first set of functions 330. For example, an additional set of functions may be generated that is configured to represent a particular region of interest of a volumetric dataset. This region of interest may lie inside the portion of the volumetric dataset represented by the first set of functions 330. For example, this region of interest may represent a portion of a medical volume that is desirable to be enlarged and visualized in detail.

[0062] In the example shown in Figure 3, the region of interest 345 of the volumetric dataset is identified as being represented by an additional set of functions. In some examples, the region of interest 345 may be a region that is desirable to be represented at a higher resolution than when the volumetric dataset was represented in the first rendering configuration 310, which is represented by a first set of functions 330. In some examples, the region of interest 345 may be identified by the user. In other examples, the volumetric dataset may already include representations of different regions of interest that can be represented at a higher resolution. For example, the volumetric dataset may be generated from a hierarchical imaging process, in which case the volumetric dataset may include volumetric data arranged in a hierarchical structure, and may include regions of interest scanned at a higher resolution, such as HiP (Hierarchical phase-contrast) CT images.

[0063] Next, the first, second, and third steps described above may generate a second set 360 of functions representing the region of interest 345. That is, the second rendering configuration 340 is defined by a suitable set of rendering parameters to render an image of the region of interest 345, for example, at high resolution. The second set 350 of images is generated by performing a rendering process, which may have any of the features described above for the rendering process used to generate the first set 320 of images. For example, the rendering process may be a physical rendering process capable of producing a high-resolution image of the region of interest 345. Its rendering parameters are stored in association with the second set 350 of images. The second set 360 of functions, for example, a second Gaussian splat representation, is generated from the second set 350 of images. These steps may be repeated for any further regions of interest in the volumetric dataset.

[0064] The first set of functions 330 and the second set of functions 360, as well as any other sets of generated functions representing further areas of interest, are used to determine a given function 380 to be stored in the display 370 of the volumetric dataset. The given function 380 is stored in the display 370 in association with the selected data. In the example shown in Figure 3, all functions from the different sets of functions 330 and 360 are determined to be the given function 380 to be stored in the display 370. In another example, only a subset of functions from the sets of functions 330 and 360 may be selected to be stored as the function 380 in the display 370. In yet another example, the stored function 380 may be determined by combining or integrating functions from the different sets of functions 330 and 360.

[0065] The above display 370 is configured to be used to render an image of the volumetric dataset using further rendering processes. For example, a function may be selected from the stored functions 380 based on selection data, and this may be used in a rasterization process to render an image of the volumetric dataset. This selection data may indicate each bounding box setting, resolution information, and any other relevant information used to generate images 320, 350, which are produced by a specific one of the sets of functions 330, 360.

[0066] In some examples, functions 380 are stored in a hierarchical structure in display 370. In such cases, the selected data may indicate the level in the hierarchy to which a particular function corresponds. For example, a first set of functions 330 may represent the entire volume at the first, lowest resolution. The functions in the first set of functions 330 may be labeled accordingly in display 370 as belonging to the first level in the hierarchy. A second set of functions 360 may represent the region of interest 345 at a second resolution higher than the first resolution. The functions in the second set of functions 360 may be labeled accordingly as belonging to the second level in the hierarchy. In some examples, there may be further regions of interest at the second level in the hierarchy. Also, there may be further levels in the hierarchy representing one or more further regions of interest as the resolution increases.

[0067] The hierarchical display described above can be used interactively to render images of a volumetric dataset from any viewpoint. For example, if a given view of an image to be rendered using further rendering processes allows for visualization of the entire volume, a first set of functions 330, labeled as belonging to a first level in the hierarchical structure, may be selected from the stored functions 380 and used to render an image of the entire volume from a given viewpoint. If the user changes the view of the rendered image to focus on a region of interest 345 at a second level in the hierarchical structure, a second set of functions 360 may be selected from the stored functions 380 and used for further rendering processes to render an image of the region of interest 345 at a higher resolution.

[0068] In some examples, a zoom level may be used to determine an appropriate level in the hierarchy to select the function to be used to render the image using further rendering processes. In some examples, the visualization may allow for smooth transitions between different levels in the hierarchy by combining images rendered by functions at different levels in the hierarchy. For example, when the user zooms in on the region of interest 345, the zoom level may be used to make a smooth combination between the image of the entire volume (e.g., a lower-resolution image) generated by the function corresponding to the first set 330 of functions at the first level of the hierarchy and the image of the region of interest 345 (e.g., a higher-resolution image) generated by the function corresponding to the second set 360 of functions at the second level of the hierarchy. In some examples, regions of interest, such as the first region of interest 345, represented by a further set of functions, may be marked during visualization so that they can be identified by the user. For example, regions of interest may be marked by a specific color or enclosed by a box.

[0069] Different rendering configurations and corresponding further parameter settings may be represented by different subsets of functions in display 370 associated with appropriate selection data. For example, multiple rendering configurations may be defined to differ only with respect to the position of the clip plane. The position of the clip plane may be manually defined by the user during key frame generation, or it may be automatically selected based on segmentation, landmark information, or the optimal clip plane position specific to the usage. For example, in a cardiac procedure flow, the clip plane may be automatically positioned using segmentation information or anatomical landmarks to reveal the ventricle or heart valve. Each of these rendering configurations may be represented by a set of functions. Each of these sets of functions may be generated in the same manner as described above with reference to the first or second set of functions 330, 360, and stored in display 370 associated with appropriate selection data. For example, if the position of the clip plane differs among multiple rendering configurations, each clip plane setting may be stored associated with a set of functions in display 370. When further rendering is used to render an image from display 370, adjustments to the clip plane settings visible to the user may allow a suitable set of functions representing those clip plane settings to be selected from the functions stored in display 370.

[0070] In some examples, instead of storing all functions representing different parameter settings in display 370, a subset of these functions may be selected to be stored in display 370. For example, functions from a set of functions representing different parameter settings (e.g., different clip plane settings) may be compared with each other to identify functions shared between parameter settings. This may include, for example, comparing functions for different parameter settings using a similarity metric. Functions shared between different parameter settings can be marked as global (or inclusive) and stored in display 370 as belonging to a global set of functions. Furthermore, one or more additional sets of functions corresponding to each parameter setting may be stored in display 370. Each of these sets may represent the differences between the set of functions representing each parameter setting and the global set of functions. For example, for a given clip plane setting, only functions that differ sufficiently from the functions in the global set of functions may be stored in display 370. These functions may be stored associated with labels indicating the corresponding clip plane settings. Therefore, different parameter settings can be displayed in a compact manner, with each setting containing only functions sufficiently different from those in the global set, in addition to the global set of functions, stored in display 370.

[0071] In the example above, when rendering an image of a volumetric dataset using a given parameter setting (e.g., the position of the clipping plane), all functions in the global set can be selected in the same way as those stored for the given parameter setting. This makes rendering images from Display 370 more efficient because the functions in the global set are reusable between parameter settings. For example, when adjusting the parameter settings of a visualization (e.g., moving the clipping plane), a subset of functions representing the new parameter setting may be loaded into memory instead of a subset of functions representing the previous parameter setting. This may be more efficient than loading an entirely new set of functions corresponding to the new parameter setting into memory. As an example, the technique above allows Display 370 to provide an interactive rendering process in an efficient way. For example, this method may provide a visualization that interactively shifts the clipping plane gradually through volume in an efficient way.

[0072] Instead of determining a global set of functions, all functions representing initial parameter settings (e.g., initial clip plane settings) may be stored in display 370. For example, each of several sets of functions corresponding to different parameter settings may be compared with the set of functions representing the initial parameter settings. And for each of these further parameter settings, only functions sufficiently different from the functions in the initial set may be stored.

[0073] Different time steps in a 4D volumetric dataset may be represented using techniques similar to those described above in relation to representing different rendering parameter settings. For example, each set of functions can be generated to represent different time steps in a 4D volumetric dataset. These sets of functions may be stored in display 370 in association with selected data that indicates the time steps they correspond to. In such cases, different time steps can be encoded in an efficient manner using the techniques described above. For example, a set of functions shared among all the functions of different time steps may be stored as a global set of functions, in which case each time step may be encoded based on its differences from the global set of functions. Alternatively, all the functions representing a first time step may be stored together with other time steps represented only by functions different from those of the first time step.

[0074] In some examples, a particular function may be encoded using a difference value representing the difference between the function and a reference function, for example, using a function from an earlier time step or a function at a different level in a hierarchical structure. In such examples, each of the stored functions 380 may be encoded using an additional value representing a link to the reference function. Furthermore, in some examples, during rendering, a particular function may be modified using interpolation to enable a smooth transition from one rendering configuration to another (for example, from one time step to another, or from one level in a hierarchical structure to another).

[0075] The techniques described above can all be used in combination with each other. For example, in the above example that generates a representation of the hierarchical structure of a volumetric dataset in a particular rendering configuration, one could apply a technique to encode a global set of functions along with different sets of functions for each level in the hierarchy for functions stored for different levels in the hierarchy, or one could apply a technique to encode all representations of a given level in the hierarchy and then encode only the differences between those encoded for the given level and the other levels in the hierarchy.

[0076] In some examples, techniques may be applied to reduce the amount of processing performed in the process of generating images 320, 350 and sets of functions 330, 360 for representing different rendering configurations. For example, if different rendering configurations represent different positions of the clipping plane, a set of combinations of clipping plane positions and target views may be determined and used to generate images that produce functions to be stored in display 370. One such set of combinations is all possible combinations of clipping plane positions and target views. However, not all clipping plane positions affect the images generated for a given target view. For example, for one of the given sets of target views, the surface affected by the clipping plane may be invisible, for example, because it is obscured by other structures in the volume. For such a target view, a slight change in the clipping plane position may not affect the final rendered image. Such combinations of target views and clipping planes can be omitted from the set of combinations used to render the images. Images can then be generated for the determined set of clipping plane positions and target view combinations. Reducing the number of images rendered to generate the functions to be stored in the display 370 can improve the efficiency of the process of generating the display 370. These images can then be used to generate a set of functions to be stored in the display 370 of the volumetric dataset in the manner described above. In one such example, the function generation (e.g., training a Gaussian splat display) may involve iteratively training the set of functions using differentiable rendering to accurately represent the clipped target view when applying the same clipping plane position.

[0077] The exemplary methods described herein can be used in a variety of situations, for example, where it is desirable to generate images of a volumetric dataset interactively without distributing the volumetric dataset itself. In this case, it is possible to render high-quality images interactively while limiting the computing power and / or memory requirements required of the device for rendering the images. Furthermore, when the volumetric dataset itself is not distributed, it becomes possible to protect the source data of the volumetric dataset.

[0078] For example, when a teacher-student scenario is used, the teacher may prepare anatomical volume data (e.g., CT scans), in which optimal keyframes may be used to visualize various anatomical structures and pathologies. For example, the teacher may set optimal transfer functions and clip plane settings to visualize these structures and pathologies. For each of these keyframes, a display 370 may be generated as described above, according to the method described herein, and distributed to the student so that it can be used for home learning and searching on a web page, social media, or personal mobile or immersive display device. For example, the student can use the display 370 to generate high-quality images of the keyframes from any viewpoint, and can also change predetermined rendering parameter settings (e.g., the position of the clip plane) to enable interactive visualization. This makes it possible to deepen the learning experience.

[0079] Another example is that in a clinical setting, radiologists and surgeons may collaborate to generate one or more keyframes with optimized parameters to visualize a specific medical condition. This may be done, for example, by a high-performance rendering device via a desktop application. Then, a display 370 of these keyframes may be generated according to the method described above. This display 370 may then be transferred to an in-operating visual device, such as a tablet computer, stereo display, or AR glasses, to generate high-quality interactive visualizations available in the operating room.

[0080] In yet another example, the display 370 may be transferred to the AR device's local storage (remote storage device). The AR device may then render an image from the display 370 using the minimum bandwidth, storage, and rendering performance requirements so that the visualization can be explored interactively.

[0081] In some cases, if parameter settings are changed, or if a region of interest or a high-resolution region is explored by a user interactively exploring the visualization, additional functions for display 370 may be transferred on demand.

[0082] Figure 4 is a schematic diagram of an exemplary system 401 in which the method described herein is used in an exemplary apparatus 404. This system 401 includes a scanner 402, apparatus 404, and visualization apparatus 414. In some examples, this system may include fewer components than those shown in Figure 4, or it may include more components. For example, this system 401 may include a computer network such as the Internet.

[0083] The scanner 402 may be any scanner capable of generating a dataset (e.g., a medical volumetric dataset representing a portion of a patient) that includes a volumetric dataset 410. The scanner 402 may be a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a positron emission tomography (PET) scanner, an ultrasound scanner, etc. The scanner 402 is connected to the device 404, for example, by wire or wirelessly. The scanner 402 can be configured to provide the volumetric dataset 410 to the device 404.

[0084] The device 404 includes one or more processors 406 and one or more memory units (or memory devices) 408. The device 404 may include, for example, a GPU. In this example, the device 404 is configured to perform the method described above and generate a representation of the volumetric dataset 410 used for rendering an image of the volumetric dataset 410. The device 404 may include one or more processors to perform various embodiments of the method. For example, the device 404 may include a volumetric rendering module that performs the process of rendering an image used to generate a representation of the volumetric dataset 410, and may perform a physical volumetric rendering process on the volumetric dataset 410 using, for example, different rendering parameter settings.

[0085] The storage unit 408 may include a machine-readable medium containing a set of machine-readable instructions, and when the machine-readable instructions are executed by the processor 406, the exemplary method described herein may be executed by the device 404. This program may be stored in a computer-readable (or computer-readable) medium, so that when read by the device 404, the program is executed. The device 404 may be configured to acquire the volumetric dataset 410 directly, indirectly, or from the scanner 402. The device 404 may use the volumetric dataset 410 to render images and use these images to generate a display of the volumetric dataset, as in the method described above. In another example, the scanner 402 is optional, and the volumetric dataset 410 may be acquired from an external source.

[0086] Device 404 can be configured to transmit information to visualization device 414, which may be, for example, values ​​that define the display of volumetric dataset 410, or, for example, a Gaussian splat display including relevant selection data. The transmission can be carried out directly or indirectly, for example, via a wired connection, a wireless connection, or the internet.

[0087] The visualization device 414 may be equipped with visualization software for displaying a two-dimensional projection of the volumetric dataset 410 using a display generated by device 404. For example, the visualization device 414 may be equipped with a device for performing rendering processes, such as rasterization, to interactively render an image from the display of the volumetric dataset 410. The visualization device 414 may be equipped with a display screen and one or more graphics hardware or software components. In some examples, the visualization device 414 may be a mobile device or may be equipped with a mobile device. In some examples, the visualization device 414 may be equipped with a virtual reality (VR) device or an augmented reality (AR) device. In some examples, the visualization device 414 may display stereo images (stereoscopic images). In yet another example, device 404, or another device between device 404 and the visualization device 414, may perform rendering processes on the display and provide generated image display (or visualization) data so that the visualization device 414 can display the image.

[0088] In some of the examples described above, the functions stored during the display of the volumetric dataset are Gaussian functions, but in other examples, these functions may be of a different kind; for example, they may represent one or more neural radii fields (NeRFs).

[0089] In the example of generating a display of a volumetric dataset described above, rendering is performed to render images of the volumetric dataset in different rendering configurations used to generate the display. However, in another example, the images used to generate the display may be obtained from a different source, for example, from an external rendering device, or from a suitable storage device (or storage) that stores pre-rendered images of the volumetric dataset in different rendering configurations.

[0090] The embodiments described above should be understood as illustrative examples of the present invention. Other embodiments are conceivable. Any feature described in any embodiment may be used alone, in combination with other features described, in combination with one or more features of any other embodiment, or in any combination of any other embodiment. Furthermore, equivalent or modified features not described above may be applied without departing from the scope of the present invention as defined in the appended claims.

[0091] Anything that is identical to a male, female, or other sex, independently of the use of grammatical terms, shall be included in the term.

Claims

1. A computer implementation method (100) for generating a volume dataset (410) representing medical volume for use in rendering processing, Multiple sets of functions (330, 360) representing the volumetric dataset (410) are generated in different rendering configurations (310, 340) (102), Based on a plurality of sets of the aforementioned functions (330, 360), a given function (380) having a 3D position for each volumetric dataset (410) and one or more visualization attribute values ​​stored in the display (370) of the volumetric dataset (410) used in the rendering process is determined (104). The display (370) of the volume dataset (410) includes storing the given function (380) in association with selection data configured to allow selection of one or more of the given function (380) for use in the rendering process during the rendering process, wherein the selection data is based on one or more of the rendering configurations (310, 340), method.

2. The method according to claim 1, wherein the different rendering configurations (310, 340) include a first rendering configuration (310) representing a first portion of the volumetric dataset (410) at a first resolution, and a second rendering configuration (340) representing a second portion (345) of the volumetric dataset (410) that is different from the first portion at a second resolution higher than the first resolution.

3. The method according to claim 1 or 2, wherein the given function (380) is a Gaussian function, and the representation (370) of the volumetric dataset (410) is a Gaussian splat representation.

4. The method according to any one of claims 1 to 3, wherein one or more of the different rendering configurations (310, 340) differ from one or more of the clipping plane settings, transfer function settings, lighting settings, and bounding box settings.

5. Compare one or more functions from different sets of the aforementioned sets of functions (330, 360), Based on the results of the above comparison, a given function (380) to be stored in the display (370) of the volume dataset (410) is determined. The method according to any one of claims 1 to 4.

6. The method according to claim 5, wherein the comparison includes determining the similarity between different sets of functions (330, 360), and based on the determined similarity, the one or more given functions (380) are determined.

7. The method according to any one of claims 1 to 6, wherein the volumetric data set (410) is a 4D volumetric data set, and the different rendering configurations (310, 340) represent different time steps of the 4D volumetric data set.

8. To generate multiple sets of the aforementioned functions (330, 360), This involves generating each set of multiple sets of the function based on each of multiple rendered images (320, 350) that show a volumetric dataset (410) in a given rendering configuration (310, 340) from different perspectives. The method according to any one of claims 1 to 7.

9. The aforementioned method includes, Using each rendering process having one or more rendering parameters, a set of the rendered images (320, 350) is generated. The method according to claim 8, comprising determining the selected data based on one or more rendering parameters.

10. The aforementioned method includes, When used during rendering, the combination of each rendering configuration (310, 340) and viewpoint is determined to provide images that differ from each other in terms of the visibility of the volumetric dataset (410), The method according to claim 8 or 9, further comprising determining a different viewpoint for each of the plurality of images (320, 350) showing the volumetric dataset (410) in a given rendering configuration, based on the determined combination.

11. The method according to any one of claims 8 to 10, wherein one or more of the rendering processes are physical rendering processes.

12. A computer method (200) for rendering an image of a volume dataset (410) representing medical volume, A rendering process is performed (202) using a display (370) of a volume dataset (410) representing medical volume, wherein the display (370) of the volume dataset (410) includes a given function (380) having each 3D position of the volume dataset (410) and one or more respective visualization attribute values. The given function (380) is associated with selection data configured to allow selection of one or more of the given functions (380) during the rendering process for use in the rendering process, and is stored in the display (370). The rendering process includes selecting one or more of the given functions (380) for use in the rendering process based on the selected data and one or more parameters of the rendering process, and rendering an image of the volumetric dataset (410) using the selected one or more of the given functions (380). method.

13. A set of machine-readable instructions, When executed by processor (406), The method according to any one of claims 1 to 11 (100), and The method according to claim 12 (200), This was designed to cause either or both of the following to be performed. A set of machine-readable instructions.

14. A machine-readable medium, A representation (370) of a volume dataset (410) representing medical volume, which can be generated by the method according to any one of claims 1 to 11, and A set of machine-readable instructions according to claim 13, Including both or either of the following: Machine-readable media.

15. A device (404) comprising a processor (406) and a memory device (408), The memory device (408) is A representation (370) of a volume dataset (410) representing medical volume, which can be generated by the method according to any one of claims 1 to 11, and A set of machine-readable instructions according to claim 13, Including both or either of the following: Device.

Images