Techniques for medical image rendering of optical properties per sampling point

Abstract

The present invention relates to techniques for medical image rendering of optical properties per sampling point. A technique is provided for volume rendering and / or surface rendering of a medical imaging dataset based on optical properties of each sampling point. An uncertainty indicator for each voxel and / or each surface element associated with a segmentation mask of the medical imaging dataset and / or anatomy included in the medical imaging dataset is received. The randomization of the one or more sampling points is scaled based on the received uncertainty indicators, and at least one optical property is determined for each sampling point. The volume is rendered based on the voxels and / or the surface is rendered based on the surface elements. The rendering is based on the determined at least one optical property of each sampling point.

Description

[0001] Related Applications

[0002] This application claims the benefit of EP 24152768.8, filed on January 19, 2024, which is hereby incorporated by reference in its entirety.

[0003] Technical Field

[0004] This document relates to a technique for volume rendering and / or surface rendering of medical imaging datasets based on the optical properties of each sampling point, particularly including methods, computing devices, systems including computing devices, and computer program products. Background Art

[0005] Modern artificial intelligence (AI)-based segmentation tools are revolutionizing the way doctors interact with imaging data in many clinical and educational applications. However, high-performance and high-quality three-dimensional (3D) volume rendering using binary masks generated from such algorithms remains a challenge, particularly in real-time imaging modalities and for image-based guidance applications. Fast rendering algorithms often result in scattered visual artifacts at the boundaries of segmentation classes, which may hinder doctors' spatial understanding. Hindering spatial understanding may in turn affect successful treatment planning and / or execution.

[0006] Smooth rendering using binary segmentation masks can first use a smooth distance transform on the binary volume to compute a signed distance field, whose zero (0) level set can be rendered as an isosurface. Other image smoothing operations can also be applied, including specialized interpolation methods and smooth surface extraction. While such conventional methods can produce high-quality results, they require additional preprocessing, thereby introducing undesirable latency, particularly making them unsuitable for real-time applications. Summary of the Invention

[0007] Accordingly, it is an object to provide a solution for high-performance rendering of medical image data for reducing latency, particularly latency due to preprocessing, and / or for avoiding disruptive changes to the rendering pipeline. The solution provided herein should be applicable to interventional image guidance procedures, e.g., surgical image guidance procedures.

[0008] This object is solved by a method for volume rendering and / or surface rendering of medical imaging datasets based on the optical properties of each sampling point, by a computing device, by a system including the computing device, and by a computer program (and / or computer program product). Advantageous aspects, features, and embodiments are described in the claims and the following description together with the advantages.

[0009] Below, the solution is described with respect to a method for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point and with respect to a computing device. Features, advantages or alternative embodiments herein may be assigned to other claimed objects (e.g., systems, computer programs or computer program products), and vice versa. In other words, the claims for a computing device and / or a system including a computing device may be improved with features described or claimed in the context of a method. In this case, the functional features of the method are embodied by the structural units of the system, and vice versa.

[0010] In terms of the method aspect, a computer-implemented method is provided for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point. The method includes the action of receiving an uncertainty index for each voxel and / or each surface element related to a segmentation mask of the medical imaging dataset and / or an anatomical structure included in the medical imaging dataset. The method further includes the action of scaling the randomization of one or more sampling points based on the received uncertainty index. Optionally, the scaling of the randomization may include scaling the distance to the sample center as a function of the received uncertainty index. The function may include a linear correlation.

[0011] The method further includes the action of determining at least one optical property of each sampling point.

[0012] At least one optical property may be derived from the output of a medical scanner through a transfer function. For example, in the case of a computed tomography (CT) scanner, the transfer function maps Hounsfield units (HU) to optical colors and opacity (e.g., through a one-dimensional 1D transfer function). Higher-order transfer functions may have further inputs, such as data gradients and / or curvatures. For example, the mapping may be set by an algorithm, may use program-specific presets, may be set through user interaction, or may be set by a combination of these methods.

[0013] The method further includes the action of rendering a volume based on voxels and / or rendering a surface based on surface elements. The rendering is based on at least one optical property of each sampling point determined.

[0014] With this technique, high-performance and high-quality three-dimensional (3D) volume rendering and / or surface rendering of medical imaging datasets can be performed, which can avoid visual artifacts, especially at the boundaries of anatomical structures and / or segmentation classes. Alternatively or additionally, destructive changes to the rendered volume and / or the rendered surface can be avoided. For medical practitioners (e.g., internists, radiologists, and / or surgeons), the spatial understanding of the rendered volume and / or the rendered surface can be improved. Further alternatively or additionally, the technique can obviate the need for complex preprocessing and / or help avoid delays. The technique can be particularly combined with real-time medical imaging, e.g., for medical image-based guidance (e.g., during surgical intervention and / or endoscopy).

[0015] An uncertainty metric (also referred to as an uncertainty value) can include (e.g., an uncertainty) measure value and / or an estimated value. The (e.g., uncertainty) value can represent the confidence of a label prediction (e.g., from a deep neural network-based segmentation algorithm), the subjective confidence of a practitioner's manual segmentation, and / or a value derived from data (e.g., signal-to-noise ratio (e.g., SNR) measurement).

[0016] For all voxels or voxel subsets (e.g., near the boundary of a segmented object) or for voxels belonging to a label and / or a segmentation mask, an uncertainty metric (and / or e.g., an uncertainty value) can be derived (e.g., calculated) or received.

[0017] For example, the (especially uncertainty) measure can have a value ranging between 0 and 1 (and / or [0,1]), where zero corresponds to no uncertainty and 1 corresponds to maximum uncertainty.

[0018] The method can use (especially one or more) sampling points. The sampling points can be randomized.

[0019] Randomization can include those positions of the sampling points (also referred to as sampling sites and / or sampling locations) that are used to calculate volume rendering integrals along observation rays offset in a randomized direction. The offset magnitude can be proportional to the uncertainty metric (also referred to as the (especially local) uncertainty value).

[0020] For sampling positions with a well-defined local surface (e.g., with a large voxel gradient magnitude), randomization can be limited to a plane perpendicular to the surface normal and / or the voxel gradient. For poorly defined local surfaces (e.g., with a voxel gradient magnitude below a predetermined threshold), the sampling position offset can include (or can be) a weighted average of volume-based and / or surface-based randomization, where the weighting is proportional to the voxel gradient magnitude.

[0021] The randomization position can be (e.g., always) determined (e.g., computed) both on a plane perpendicular to the voxel gradient and within a box or sphere around the voxel. Two types of randomization positions can be (e.g., always) averaged by weighting based on the gradient magnitude. If the gradient magnitude is large, e.g., near the surface of a bone, then plane sampling can be the major or sole contributor to the weighted average.

[0022] The voxel gradient can be (or can correspond to) a vector in the direction of the maximum change of scalar data (e.g., including opacity, color, reflectivity, one or more values representing dispersion, HU values, and / or CT numbers). The voxel gradient magnitude can indicate the amount of change in the scalar data. At the interface between tissue types (e.g., bone and muscle), the magnitude can be high, while within a single tissue type, the magnitude can be low. Alternatively or additionally, a large gradient magnitude can indicate that positions around the voxel can behave more like a surface in the context of a physical process or rendering, e.g., related to light scattering.

[0023] Scaling the randomization of one or more sampling points using an uncertainty metric can alternatively be labeled as random sampling and / or random (especially label) volume rendering (and / or surface rendering, especially in the case of a single sampling point).

[0024] For volume rendering, the uncertainty metric can be received (and / or provided) per voxel. Alternatively or additionally, for surface rendering, the uncertainty metric can be received (and / or provided) per surface element (and / or per pixel).

[0025] An anatomical structure can include one or more organs and / or one or more types of body tissue (e.g., soft tissue and / or bone tissue). Any body tissue can be assigned a value on the Hounsfield unit (HU) scale. The HU scale value for computed tomography (CT) used as a medical imaging modality can be labeled as the CT number.

[0026] A segmentation mask can include a binary mask for each class (also referred to as a label). A class can refer to an anatomical structure, such as an organ.

[0027] Segmentation classes and / or labels can refer to anatomical structures (e.g., organs, such as the liver and / or kidneys).

[0028] For surface rendering, the randomization can be scaled for a single sampling point. Alternatively or additionally, for volume rendering, the randomization can be scaled for multiple sampling points.

[0029] At least one optical property may include at least one rendering parameter, in particular a color (e.g., hue of a color) or a change thereof, opacity (e.g., change of opacity), and / or a combination thereof. Alternatively or additionally, at least one optical property may refer to how a position in a volume reacts with light in the context of a rendering algorithm, e.g., how it absorbs and scatters light.

[0030] By scaling the randomization of one or more sampling points, at least one parameter (e.g., segmentation class and / or HU scale value) of each voxel and / or each surface element can be modified. Due to the dependence on the uncertainty index, this effect may particularly occur at the boundaries between anatomical structures and / or at the boundaries between segmentation classes, where the value of the uncertainty index increases compared to inside the anatomical structures and / or inside the segmentation classes.

[0031] Rendering may be based on ray casting and / or Monte Carlo path tracing algorithms. Alternatively or additionally, rendering may include performing smooth shading based on an uncertainty index and / or subsurface scattering (SSS).

[0032] SSS can be used to perform smooth surface shading. The uncertainty index (also referred to as the uncertainty value) can be used to define a scattering parameter, which can be a scattering radius and / or an average path length between scattering events. A larger uncertainty (e.g., according to the uncertainty index) can cause the light to leave the surface at a greater distance from the entry point, which results in a smooth surface appearance on the final (e.g., rendered and / or displayed) image.

[0033] Conventional volume visualization or rendering methods based on ray casting (which are still used in many currently advanced visualization medical products) only simulate the emission and absorption of radiation energy along the primary viewing ray through volume data. According to the Beer-Lambert law, the radiation energy emitted at each point is absorbed along the ray to the observer position according to the absorption coefficient derived from the patient data. The renderer (a computing device with the implemented rendering algorithm) typically calculates the shading only using a standard local shading model (e.g., the Blinn-Phong model) at each point along the ray based on the local volume gradient (i.e., local illumination). Although fast, these methods do not simulate the complex light scattering and extinction (i.e., global illumination) associated with photorealism.

[0034] Monte Carlo path tracing is a global illumination algorithm that uses Monte Carlo integration to solve the rendering equation. It can produce highly realistic images, including for medical visualization. At the same time, due to the need to simulate hundreds to thousands of discrete light paths at each pixel or voxel, the computational requirements are very high. As more paths are simulated, the solution converges to an accurate estimate of the irradiance at each point from incident light in all directions. The renderer employs a mixture of volume scattering and surface-like scattering, which are modeled by a phase function and a bidirectional reflectance distribution function (BRDF) respectively based on properties derived from the anatomical structure. Generating a single image can take several minutes, so it is currently not suitable for real-time rendering. A variety of algorithms aim to address the performance challenges, including irradiance caching (which requires long pre-computation of lighting changes before real-time rendering is possible), artificial intelligence (AI)-based denoising, and light path generation. However, real-time rendering still cannot be achieved using conventional Monte Carlo path tracing.

[0035] Rendering can further use a smoothed distance transform and / or a signed distance field (SDF). Alternatively or additionally, rendering can further use interpolation and / or smoothed surface extraction.

[0036] The rendering action can include displaying the volume and / or surface using a screen, a stereoscopic display, a virtual reality (VR) display, and / or an augmented reality (AR) headset.

[0037] The received uncertainty metric can be based on noise measurements within and / or across segmentation masks and / or within and / or across local neighborhoods within the anatomical structure. Optionally, the noise measurements can include determining the signal-to-noise ratio (SNR), peak signal-to-noise ratio (PSNR), and / or contrast-to-noise ratio (CNR). Alternatively or additionally, the noise measurements can include determining the signal-to-interference-plus-noise ratio (SINR) and / or carrier-to-noise interference ratio (CNIR).

[0038] The noise measurements can provide higher values (e.g., values of noise) using an increasing number of label values (and / or segmentation classes). Alternatively or additionally, a low measured SNR may imply high noise in the sample neighborhood, which may be associated with high uncertainty in the segmentation (and / or coloring of the anatomical structure). A high SNR may imply a strong signal, and in this case, a more certain segmentation. Further alternatively or additionally, high noise measurements can be associated with high uncertainty and / or a large number of unique voxel classes and / or label values in the local neighborhood of the sample.

[0039] In high-frequency and / or large-overlap regions between segmentation masks and / or between anatomical structures, the voxels and / or surface elements have high uncertainty, e.g., corresponding to high values of the uncertainty metric.

[0040] High frequencies can refer to changes in the details of medical imaging data included in a medical imaging dataset, particularly with respect to signal processing performed in the frequency domain and / or the use of (particularly fast) Fourier transforms (particularly FFT). Alternatively or additionally, high frequencies can indicate and / or can include sharp transitions between labels.

[0041] The received uncertainty metric can be determined by a segmentation algorithm that provides a segmentation mask.

[0042] The uncertainty metric, e.g., based on noise measurements, can be determined (e.g., routinely) when performing segmentation on medical imaging data included in a medical imaging dataset.

[0043] For example, the act of receiving the uncertainty metric can include (e.g., simultaneously) receiving the segmentation mask.

[0044] Randomized scaling can include scaling the distance to the sample center as a function of the received uncertainty metric. The function can include a linear correlation.

[0045] The distance to the sample center can include the distance along a ray, can include motion in a random direction, and / or can be determined by the size of a box.

[0046] Computing a volume rendering integral along an observation ray requires sampling the optical properties of the volume at a particular set of positions (e.g., regularly spaced points along the observation ray).

[0047] For each sampling position, the position in (e.g., 3D) space can be perturbed based on the uncertainty metric (and / or uncertainty value), such that for regions of high certainty, when using Monte Carlo, values are averaged over a larger local neighborhood. The same process can be applied multiple times.

[0048] Randomized scaling can parameterize how the uncertainty metric (and / or uncertainty value) is associated with the rendering.

[0049] For example, in practice, perturbations can be imposed by creating a sphere and / or an axis-aligned bounding box (AABB) where the radius and / or diameter is proportional to the uncertainty metric (and / or uncertainty value). Random points within the sphere and / or AABB can be selected as sampling positions.

[0050] Alternatively or additionally, a unit sphere and / or a unit AABB can be sampled at the sample position and at a distance scaled to the original sample center based on the uncertainty metric (and / or uncertainty value).

[0051] For positions with well-defined local surfaces (e.g., having a large voxel gradient magnitude), the randomization can be restricted to a plane perpendicular to the surface normal and / or the voxel gradient, in which case the points are selected within a disk and / or rectangle in the perpendicular plane. Alternatively or additionally, for poorly defined local surfaces (e.g., having a voxel gradient magnitude below a predetermined threshold), the sampling position offset can comprise (or can be) a weighted average of volume-based and / or surface-based randomization, particularly weighted in proportion to the voxel gradient magnitude.

[0052] For example, when the (e.g., voxel) gradient magnitude is small, the weight for surface-like position randomization can be small and the weight for volume-like position randomization can be large.

[0053] In the case of weighted averaging, only the relative magnitudes of the weights may matter since they can (e.g., always) be normalized to sum to 1.

[0054] The method can further include the action of combining the optical properties of more than one sampling point. The rendering can be based on at least one optical property of the combined more than one sampling point.

[0055] For the case of more than one sampling point, particularly for volume rendering, the combination can be performed. Alternatively or additionally, the combination can include determining the (e.g., weighted) average of the optical properties (e.g., color and / or opacity of each voxel).

[0056] By combining, it is beneficial for smooth rendering of the segmentation mask and / or anatomical structures.

[0057] The rendering can include denoising. Optionally, the denoising can include sampling the medical imaging data included in the medical imaging dataset using a random number generator (RNG) sequence.

[0058] The rendering can be repeated using different seeds of the random number generation algorithm (and / or using altered randomization).

[0059] The RNG sequence can include a very large (e.g., millions and / or billions) sequence of random numbers, e.g., within the interval between 0 and 1 (and / or [0,1]), such as values like 0.01 (and / or 1 / 100) and / or 0.1 (and / or 1 / 10).

[0060] The randomized positions (e.g., the randomized positions of the sampling points) can be independent, particularly by definition, and / or for each voxel and / or each surface element (and / or pixel). Alternatively or additionally, a (particularly independent and / or different) randomized offset can be applied each time the sampling position is determined.

[0061] The technique can use three types of sampling. The first type of sampling is to sample the volumetric optical properties along the rays to compute the volume rendering integral. According to the technique, a sampling position offset is applied to these samples.

[0062] The second type of sampling is to sample each pixel in the final image multiple times to reduce noise from the randomized offset. Alternatively or additionally, statistical and / or AI-based denoising methods can be used. This sampling can be used especially in conjunction with a ray-casting based volume rendering algorithm.

[0063] Regarding the third type of sampling, when using Monte Carlo integration for photorealistic volume rendering, individual light paths through the volume from each pixel are sampled. The rendering process is repeated multiple times and the results are averaged, as in the case of ray casting. Statistical and / or AI-based denoising methods can be further used to reduce the image variance faster.

[0064] Medical imaging datasets can be acquired by medical scanners. Medical scanners can include X-ray devices, ultrasound (US) devices, positron emission tomography (PET) devices, computed tomography (CT) devices, single photon emission computed tomography (SPECT) devices, and / or magnetic resonance tomography (MRT) devices.

[0065] Volume rendering can be performed especially on medical imaging datasets acquired from medical scanners.

[0066] At least one optical property can include opacity, reflectivity, color, and / or at least one value indicating dispersion.

[0067] At least one optical property can especially include a combination of color and opacity.

[0068] Rendering can include applying a reconstruction filter. The reconstruction filter can include, for example, a trilinear filter, a cubic B-spline filter, a nearest neighbor filter, and / or any (especially other) higher-order reconstruction filter.

[0069] Rendering can include ray casting, Monte Carlo path tracing, subsurface scattering (SSS), and / or rendering a color grid.

[0070] SSS can include (and / or can be) a shadow effect computed as part of the rendered surface. The surface can be defined explicitly (e.g., as a mesh) and / or implicitly (e.g., as a (especially zero) level set and / or as an isosurface). Alternatively or additionally, the surface can include the surface of an organ and / or anatomical structure.

[0071] SSS can simulate the light diffusion under the surface of (especially segmented) anatomical structures (also referred to as objects).

[0072] SSS can be used to merge uncertainties to smooth the shadows and / or improve the image quality, especially when preserving the original surface geometry.

[0073] Alternatively or additionally, a change in the color grid can be used (e.g., to indicate the uncertainty in assigning voxels and / or surface elements to segmentation classes and / or anatomical structures).

[0074] SSS can be controlled by (especially a single) radius value. The radius value can be determined based on the received uncertainty metric.

[0075] The radius can determine the distribution of the volume. Alternatively or additionally, assuming multiple scattering of light below the surface, the radius can determine the maximum distance between the light exit point and the light entry point. Further alternatively or additionally, the radius can determine the size of a disk placed at the light entry position and oriented along the surface normal. Alternatively or additionally, the "disk normal" can be oriented along the surface normal, especially the plane of the disk can be perpendicular to the surface normal. The projection of the disk on the surface can define a patch, which can be sampled using various statistical methods (e.g., uniform sampling and / or Poisson sampling) to select the light exit position.

[0076] The received uncertainty metric can be mapped to at least one property of the SSS. In particular, the subsurface attenuation of one or more wavelength bands can be modulated according to the received uncertainty metric. For example, based on both the uncertainty (e.g., according to the uncertainty metric) and the color wavelength, the radius of the patch used for SSS sampling can be changed by different amounts. Thus, the shadows in regions of high uncertainty can become smoother, and both the (e.g., red) subsurface hues can be improved.

[0077] Alternatively or additionally, physically based rendering of the SSS effect can use the average path length between scattering events (e.g., instead of the radius).

[0078] Different (and / or independent) radii can be used for each color channel. For example, SSS for human skin materials can use a larger radius for red channel scattering to simulate blood flow below the skin surface (e.g., compared to the radii of the green and blue channels). Uncertainty handling can use different scalings of the radius for each color channel based on the material profile.

[0079] Whenever the word "radius" is used, the same content can apply to the diameter (e.g., the diameter of a disk and / or a box).

[0080] Alternatively or additionally, (e.g., local) surface patches can be determined around the (e.g., sampling) point where the light enters the surface.

[0081] A segmented mesh (e.g., representing a segmentation mask and / or an anatomical structure) can be determined by, e.g., a (neural) Marching Cube algorithm and / or a Marching Tetrahedron algorithm.

[0082] The Marching Cube algorithm can include a computer graphics algorithm for extracting a polygonal mesh (specifically, a polygonal mesh of an isosurface) from a 3D discrete scalar field whose elements are voxels. The application of such an algorithm can mainly involve medical visualization, such as CT and / or MRT medical imaging datasets, as well as special effects or 3D modeling using metaballs or other meta-surfaces. The Marching Cube algorithm can be intended to be used for 3D (specifically for volume rendering). A 2D version of such an algorithm (e.g., for surface rendering) can be referred to as a Marching Squares algorithm.

[0083] The Marching Cube algorithm can traverse the scalar field (and / or voxels), taking eight adjacent positions at a time (thus forming a hypothetical cube), and then determining one or more polygons required to represent the part of the isosurface passing through the cube. Then, the individual polygons can be fused into the desired surface.

[0084] Fusion can be done by creating an index into a precomputed array of 256 possible polygon configurations (2 8 = 256) in the cube and / or by treating each of the 8 scalar values as bits in an 8-bit integer. If the scalar value is higher than the isovalue (e.g., it is inside the surface), the appropriate bit can be set to 1, while if it is lower (e.g., outside), it can be set to 0. After examining all eight scalars, the final value can be the actual index into the polygon index array.

[0085] Each vertex of the generated polygon can be placed in the appropriate position along the edge by linearly interpolating between the two scalar values connected by the edge of the cube.

[0086] The gradient of the scalar field at each grid point (and / or sampling point) can also be (and / or can correspond to) the normal vector of the hypothetical isosurface passing through that point. Thus, these normals can be interpolated along each edge of the cube to find the normals of the generated vertices, which may be necessary for shading the resulting mesh using some illumination models.

[0087] Marching Tetrahedra can include an algorithm for rendering implicit surfaces in the field of computer graphics and can be considered a generalization of the Marching Cube algorithm.

[0088] In the marching tetrahedra algorithm, each cube can be divided into six irregular tetrahedra by cutting the cube in half three times and / or by making diagonal cuts through each of three pairs of opposite faces. In this way, the tetrahedra all share one of the main diagonals of the cube. Instead of having 12 edges, the cube has 19 edges: the original 12 edges, 6 face diagonals, and the main diagonal. As in the marching cubes algorithm, the intersection points of these edges with the isosurface can be approximated by linear interpolation of the values at the grid points (and / or sample points).

[0089] Adjacent cubes can share all the edges of the connecting face, including the same diagonals. This is an important property for preventing cracks in the drawn surface, because interpolation of two different diagonals of a face will usually give slightly different intersection points. An additional benefit is that up to five computed intersection points can be reused when processing neighboring cubes. This includes computed surface normals and / or other graphics properties at the intersection points.

[0090] Each tetrahedron has 16 possible configurations, falling into three categories: no intersection, intersection in one triangle, and intersection in two (adjacent) triangles. It is straightforward to enumerate all 16 configurations and map them to a list of vertex indices that define the appropriate triangle strips.

[0091] Using a diamond cubic crystal as a basis, the cube cell to be meshed can alternatively be cut into five (5) tetrahedra. The cube can be paired with another cube on each side, which has an opposite alignment of the tetrahedra around the centroid of the cube. Alternate vertices can have different numbers of tetrahedra intersecting them, resulting in a slightly different mesh depending on the location. When cut in this way, additional symmetry planes can be provided. Having tetrahedra around the centroid of the cube can also generate very open spaces around points outside the surface.

[0092] In the absence of user interaction (e.g., during a predefined time period), for rendering, multiple random paths can be accumulated and / or averaged over medical imaging data associated with different sampling points.

[0093] The absence of user interaction (especially with the renderer) can also be marked as offline (especially for performing rendering).

[0094] By accumulating multiple random paths (and / or averaging over medical imaging data associated with different sampling points and / or samples), the variance, especially the variance caused by noise, can be reduced and the quality of the rendered volume can be improved.

[0095] Surface rendering using uncertainty-based subsurface scattering (SSS) can (e.g., still) be performed using a statistical process, and / or (e.g., still) requires multiple paths to reduce variance and / or noise (e.g., even when using only one sampling point), especially like volume rendering.

[0096] Alternatively or additionally, there are rendering techniques that can produce approximate SSS without random sampling, e.g., screen space techniques, which can be suitable for interactive rendering, while more complex methods (e.g., according to this technique) can be used during offline rendering.

[0097] Alternatively or additionally, in the presence of user interaction, rendering can be performed without accumulating multiple random paths and / or averaging on medical imaging data associated with different sampling points. Optionally, after a predetermined period during which no user interaction is received, multiple paths can be accumulated and / or averaged on medical imaging data associated with different sampling points.

[0098] User interaction can be received by a user interface (UI), especially a graphical user interface (GUI). For example, user interaction can include rotating the rendered volume.

[0099] (Especially with respect to the renderer) The presence of user interaction can also be labeled as online and / or real-time (especially when performing rendering).

[0100] By skipping the accumulation of multiple random paths and / or skipping averaging, noisy medical imaging data can be rendered during user interaction.

[0101] Alternatively or additionally, interactive reconstruction of Monte Carlo image sequences can use a recurrent denoising autoencoder.

[0102] Further alternatively or additionally, temporal reprojection can be used.

[0103] A recurrent denoising autoencoder can combine spatial and temporal denoising using Monte Carlo samples from the current frame and previous frames.

[0104] Alternatively or additionally, temporal reprojection can be explicitly used to map pixels from a previously rendered image to the current image for weighted blending. If the previous image was rendered offline with high quality, blending pixels from the previous image can improve the quality compared to only denoising the interactive image.

[0105] Rendering may include non-uniform representations of volumes and / or surfaces, including at least part of a segmentation mask and at least part of an (especially shaded or colored or marked) anatomical structure, preferably according to selected HU scale values (e.g., especially specifically for rendering soft tissue or bone tissue).

[0106] The rendered volume may be displayed by a display device. The display device may include a (e.g., computer) screen, a projector (e.g., onto a surface in a (especially operating) room and / or onto a patient), and / or a virtual reality (VR, and / or augmented reality AR and / or extended reality XR) head-mounted device.

[0107] Segmentation may only be performed on selected organs (and / or selected anatomical structures) within a medical imaging dataset.

[0108] The computer-implemented method may use deep learning (DL).

[0109] Regarding the device aspect, a computing device (computer or processor) is provided for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point. The computing device includes a receiving module (instructions) configured to receive an uncertainty metric for each voxel and / or each surface element related to the segmentation mask of the medical imaging dataset and / or the anatomical structure included in the medical imaging dataset. The computing device includes a scaling module configured to scale the randomization of one or more sampling points based on the received uncertainty metric. Optionally, the scaling of the randomization may include scaling the distance to the sample center as a function of the received uncertainty metric.

[0110] The computing device further includes a determination module configured to determine at least one optical property of each sampling point. The computing device further includes a rendering module (renderer or graphics processing unit) configured to render a volume based on voxels and / or render a surface based on surface elements. The rendering is based on the at least one optical property of each sampling point determined.

[0111] Optionally, the computing device may include a combination module configured to combine the optical properties of more than one sampling point. The rendering may be based on the at least one optical property of more than one sampling point combined.

[0112] The computing device may be configured to perform any one of the actions of the method and / or include any one of the features described in the context of the method aspect.

[0113] Regarding the system aspect, a system is provided for volumetric rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point. The system includes a medical scanner configured to provide a medical imaging dataset. The system further includes a computing device according to the device aspect. The receiving module of the computing device is configured to receive an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of the medical imaging dataset provided by the medical scanner and / or an anatomical structure included in the medical imaging dataset.

[0114] Regarding a further aspect, a computer program product is provided. The computer program product includes program elements (instructions) that, when loaded into the memory of a computing device, cause the computing device (e.g., a computing device according to the device aspect) to perform the actions of a method according to the method aspect for volumetric rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point.

[0115] Regarding yet a further aspect, a non-transitory computer-readable medium is provided on which program elements are stored that can be read and executed by a computing device to perform, when the program elements are executed by the computing device, the actions of a method according to the method aspect for volumetric rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point.

[0116] Based on the following description and embodiments, the above properties, features, and advantages, as well as the ways to implement them, become clearer and easier to understand. The following description and embodiments will be described in more detail in the context of the accompanying drawings.

[0117] The following description does not limit the invention to the embodiments included. In different figures, the same components or parts may be labeled with the same reference signs. Generally, the figures are not drawn to scale.

[0118] It should be understood that the preferred embodiments of the present invention may also be any combination of the dependent claims or the above embodiments and the corresponding independent claims.

[0119] Referring to the embodiments described below, these and other aspects of the present invention will become apparent and will be elucidated. Description of the Drawings

[0120] Figure 1 is a flowchart of a method for volumetric rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point according to a preferred embodiment;

[0121] Figure 2is an overview of the structure and architecture of a computing device for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point, where such a computing device can be configured to perform Figure 1 the method of;

[0122] Figure 3A 、 Figure 3B and Figure 3C show an exemplary visual comparison of trilinear voxel reconstruction with and without statistical label map sampling and cubic B-spline (as a higher-order example) voxel reconstruction with random label map sampling;

[0123] Figure 4A 、 Figure 4B and Figure 4C show Figure 3B a first exemplary close-up of the anatomical structure in the upper left corner of the trilinear voxel reconstruction with statistical label map sampling, where Figure 4A shows the original, Figure 4B shows the statistical sampling, and Figure 4C shows the cumulative result;

[0124] Figure 5A 、 Figure 5B and Figure 5C show Figure 3B a second exemplary close-up of the anatomical structure in the lower right corner of the trilinear voxel reconstruction with statistical label map sampling, where Figure 5A shows the original, Figure 5B shows the statistical sampling, and Figure 5C shows the cumulative result;

[0125] Figure 6A and Figure 6B show an exemplary visual comparison of the direct rendering of the segmentation mesh obtained via marching cubes with label mapping, where only the surface normals are smoothed in Figure 6A and the visual surface artifacts are smoothed out by random SSS in Figure 6B ; and

[0126] Figure 7A 、 Figure 7B 、 Figure 7C and Figure 7D show a further exemplary visual comparison of the human torso, using only the shading structure in Figure 7A and only the segmentation structure in Figure 7B , and zooming in on the chest in Figure 7C and Figure 7D . Detailed Description

[0127] Figure 1Schematically illustrates an exemplary flowchart of a method for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point. The method is generally referred to by reference numeral 100.

[0128] Method 100 includes an action S102 of receiving an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of a medical imaging dataset and / or an anatomical structure included in the medical imaging dataset.

[0129] Method 100 further includes an action S104 of scaling the randomization of one or more sampling points based on the received uncertainty metrics. Optionally, the scaling of the randomization S104 may include scaling the distance to the sample center as a function of the received uncertainty metrics of S102.

[0130] Method 100 further includes an action S106 of determining at least one optical property of each sampling point.

[0131] Method 100 also further includes an action S110 of rendering a volume based on voxels and / or rendering a surface based on surface elements. The rendering S110 is based on at least one optical property of each sampling point determined in S106.

[0132] Optionally, method 100 may include an action S108 of combining the optical properties of more than one sampling point. The rendering S110 may be based on at least one optical property of more than one sampling point combined in S108.

[0133] Figure 2 Schematically illustrates an exemplary architecture of a computing device (computer) for volume rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point. The computing device is generally referred to by reference numeral 200.

[0134] Computing device 200 includes a receiving module 202 configured to receive an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of a medical imaging dataset and / or an anatomical structure included in the medical imaging dataset.

[0135] Computing device 200 may further include a scaling module 204 configured to scale the randomization of one or more sampling points based on the received uncertainty metrics. Optionally, the scaling of the randomization may include scaling the distance to the sample center as a function of the received uncertainty metrics.

[0136] Computing device 200 further includes a determining module 206 configured to determine at least one optical property of each sampling point.

[0137] The computing device 200 further includes a rendering module 210 configured to render a volume based on voxels and / or a surface based on surface elements. The rendering is based on at least one optical property of each determined sampling point.

[0138] Optionally, the computing device 100 may include a combining module 208 configured to combine the optical properties of more than one sampling point. The rendering may be based on at least one optical property of the combined more than one sampling point.

[0139] The receiving module 202 may be embodied by an input-output interface 212. Alternatively or additionally, the scaling module 204, the determining module 206, the rendering module 210, and the optional combining module 208 may be embodied by a processor (e.g., a central processing unit CPU and / or a graphics processing unit GPU). Further alternatively or additionally, the computing device 200 may include at least one memory 216.

[0140] The computing device 200 may be configured to execute method 100.

[0141] A system (not shown) may include the computing device 200 and a medical scanner configured to provide a medical image dataset for which uncertainty metrics are received.

[0142] The system may be configured to execute method 100.

[0143] Techniques (e.g., including method 100 and / or computing device 200) may also be labeled as having high-quality rendering of a binary label volume segmented according to uncertainty.

[0144] Techniques may perform high-quality rendering of segmentation data (and / or anatomical structures) via random sampling. In general embodiments, a binary mask for each label may be received for the purpose of modifying voxel classification and / or other parameters for each label class used for volume rendering (e.g., coloring with optical colors and / or changing opacity), visualizing the mask directly as colored voxels, and / or visualizing the mask directly as a surface. Specific applications may employ different methods for each label class. "Different methods" may refer to applying a combination of the techniques listed above in a single rendering based on segmentation, e.g., coloring voxels covered by a heart mask, reducing the opacity of voxels covered by a rib mask, and (especially simultaneously) rendering voxels as a mesh.

[0145] In a first act of an embodiment (e.g., a general embodiment), an uncertainty value (and / or uncertainty indicator) is estimated for each pixel based on noise measurements within a local neighborhood within each mask and across masks, where larger overlaps between voxels and masks in high-frequency detail regions have higher uncertainties. Alternatively or additionally, the uncertainty metric can be calculated as part of a segmentation algorithm and used directly as an estimated uncertainty value or in combination with the estimated uncertainty value.

[0146] In an embodiment, during rendering, each time the label volume is sampled, a random offset is applied, where the distance to the sample center is scaled proportionally to the uncertainty value.

[0147] In some embodiments, a smooth rendering that approximates the segmented data is rendered by combining images produced by different RNG sequences.

[0148] Figure 3A , Figure 3B and Figure 3C Results for a CT image acquired at 0.56 mm × 0.56 mm × 3 mm voxel resolution are shown along with overlapping masks for some of the segmentation classes. Figure 3A , Figure 3B and Figure 3C A visual comparison of using higher order voxel reconstruction filtering and random label mapping filtering is shown. The displayed anatomy includes a human thorax with visible lungs, ventricles, major blood vessels, and ribs.

[0149] exist Figure 3A In , trilinear voxel reconstruction is applied. Figure 3B In , trilinear voxel reconstruction is combined with random label map sampling. Figure 3C A higher order voxel reconstruction filter for anatomical data, namely cubic B-spline voxel reconstruction, combined with random label map sampling (and / or together with random label volume rendering) is shown. Figure 3A , Figure 3B and Figure 3C As can be seen in FIG. 1 , the rendered volumes vary in how sharply or smoothly the transitions between different anatomical structures occur.

[0150] Figure 4A , Figure 4B and Figure 4C as well as Figure 5A , Figure 5B and Figure 5C Focus on two different cropping areas (facing Figure 3B ), and shows the original rendering, the image with a single RNG seed, and the final accumulated result.

[0151] exist Figure 4A, Figure 4B and Figure 4C shows the clavicle extending from the posterior region of the volume and a cut on a cross - section in front of the drawn volume. A significant improvement in the visualization of different anatomical structures is visible when going from a single RNG seed image to the final cumulative result.

[0152] In Figure 5A , Figure 5B and Figure 5C cross - sections passing through the left ventricular (LV) endocardium and the left ventricular (LV) epicardium are shown as a bright approximately elliptical shape and a dark halo around the bright approximately elliptical shape, respectively. Again, a significant improvement in the visualization of the corresponding heart wall is visible when going from a single RNG seed image to the final cumulative result.

[0153] In Figure 4A , Figure 4B and Figure 4C the clavicle and Figure 5A , Figure 5B and Figure 5C the LV endocardium and LV epicardium in two exemplary cases, an important comparison of the techniques is made against images without using the uncertainty technique. For example, from Figures 4A to 4C the technique mixes the label colors (shown as different gray shades in the accompanying figures) and clearly identifies the volume of uncertainty.

[0154] In other words, Figure 4A , Figure 4B , Figure 4C and Figure 5A , Figure 5B , Figure 5C show different close - up views, illustrating the random sampling of the label mapping. All images use the Figure 3B trilinear voxel reconstruction shown. Figure 4A and Figure 5A each shows the original, Figure 4B and Figure 5B each shows the random sampling, and Figure 4C and Figure 5C each shows the cumulative result of their respective cropping regions and / or close - up views.

[0155] Alternatively or additionally, Figure 4A and Figure 5A illustrate conventional volume rendering without any uncertainty handling. In Figure 4B and Figure 5B uncertainty - based label sampling is applied (specifically according to the technique, e.g., in action S104 of method 100). Figure 4B and Figure 5BIt is redrawn multiple times and averaged to reduce the variance in the image. The final denoised result is as shown in Figure 4C and Figure 5C .

[0156] In an exemplary embodiment, for surface-based visualization of segmented data, the renderer employs subsurface scattering (SSS), which simulates the diffusion of light below the surface of the segmented object. In a preferred embodiment and instead of full optical path simulation, the amount of SSS is controlled by a single radius value that determines the surface patch around the point where the light enters the surface.

[0157] Randomized sampling of the patches determines the light exit points with larger patches, resulting in smoother surface shading. Cumulating the rendering results for different RGN sequences results in smooth shading of the surface under the limit. Note that the geometry of the segmentation is not (or does not need to be) modified, and the SSS radius is proportional to the uncertainty measurement.

[0158] "Under the limit" (especially as a mathematical term) can describe the behavior when the number of cumulative samples tends to infinity (e.g., the final rendering behavior). For an infinite number of samples, the image variance is 0 and there is no noise. In practice, when the noise level is acceptable (e.g., below a predetermined threshold), the accumulation of the rendering results for different RGN sequences is stopped.

[0159] Figure 6A and Figure 6B show exemplary results with a segmentation mesh generated by running the marching cubes algorithm and without additional processing of the geometry.

[0160] In Figure 6A and Figure 6B , a direct rendering of the segmentation mesh obtained via marching cubes with label mapping is shown. Compared to only smoothing the surface normals (briefly labeled as smooth normals) in Figure 6A , random SSS in Figure 6B smoothes out visual surface artifacts (also labeled as smooth shading) without making any modifications to the surface geometry. For example, the effect is visible when comparing the heart and stomach in each figure in Figure 6A and Figure 6B .

[0161] The rendering technique can be for two cases: offline rendering, where many random paths are accumulated to produce a reference quality result; and real-time rendering without accumulation, which instead uses temporal image reconstruction to produce a noise-free image.

[0162] In a preferred embodiment, deep learning based image reconstruction is used, in particular interactive reconstruction of a Monte Carlo image sequence using a recurrent denoising autoencoder (e.g., including NVIDIA DLSS and / or Intel XeSS). Alternatively or additionally, a dedicated AI-based denoiser and / or temporal reprojection method may be used. In a simple embodiment, accumulation of random passes is performed only when the user stops interacting with the system (e.g., including a computing device and for real-time applications, in particular a medical scanner), wherein a noisy image is displayed during the interaction.

[0163] Figure 7A , Figure 7B , Figure 7C and Figure 7D Example results of additional higher resolution CT scans rendered using the technique are shown.

[0164] Figure 7A and Figure 7B A human torso with colored structure and only segmented structure are shown respectively. Figure 7C and Figure 7D Similar images focused on the thorax, showing colored structures and showing only segmented structures, respectively, are shown.

[0165] The technique can produce smooth shading without blocking artifacts at the boundaries of segmented structures, e.g. Figures 7A to 7D Furthermore, the clarity of color mixing at the interface of the segmented structure (such as, for example, Figures 7A to 7D The different shades of grey in FIG. 4 provide a visual indication of the amount of uncertainty in the segmentation.

[0166] The described rendering technique can produce high-quality 3D visualizations, in particular using binary segmentation masks, without the need for complex pre-processing of the data. Modern image reconstruction techniques can make stochastic methods practical for real-time use and enable applications for real-time imaging modalities (such as CT), for example, for surgical guidance.

[0167] The presence of the described algorithm can be tested by rendering the phantom data with a carefully crafted segmentation mask. Uncertainty can be introduced by applying a noise filter to the binary segmentation mask and examining the amount of optical color and opacity smoothing applied during rendering. For example, by using image data with thick slices (e.g. Figure 6A and Figure 6B ) and evaluating the different stepping artifacts at surfaces parallel and perpendicular to the viewing direction, it is possible to detect cases of smooth shading for mesh surfaces.

[0168] SSS is conventionally used for physically-based modeling of materials, such as the scattering and absorption of light as it passes through human skin. In contrast, according to the technology, non-physical properties (in particular uncertainty according to an uncertainty metric) can be mapped to SSS properties.

[0169] The uncertainty metric (and / or uncertainty value) can be used to modulate the subsurface attenuation for one or more wavelength bands, e.g., by varying the radius of the patches used for SSS sampling by different amounts based on both the uncertainty (in particular according to the uncertainty metric) and the color wavelength, such that both the shadows in high uncertainty regions become smoother and a red subsurface shading is selected.

[0170] As long as not explicitly described otherwise, the individual embodiments or their individual aspects and features described with respect to the figures can be combined or exchanged with each other without restricting or expanding the scope of the described invention, as long as such combination or exchange is meaningful and in the sense of the present invention. The advantages described with respect to a particular embodiment of the present invention or with respect to a particular figure are also advantages of other embodiments of the present invention, as long as applicable.

Claims

1. A computer-implemented method for volumetric rendering and / or surface rendering of a medical imaging dataset based on the optical properties of each sampling point, the method comprising: Receiving an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of the medical imaging dataset and / or an anatomical structure included in the medical imaging dataset; Scaling the randomization of one or more of the sampling points based on the received uncertainty metric; Determining at least one optical property of each sampling point as randomized; And Rendering a volume based on the voxels and / or rendering a surface based on the surface elements, wherein the rendering is based on the at least one optical property of each sampling point determined.

2. The method according to claim 1, wherein the received uncertainty metric is based on noise measurements within the segmentation mask and / or across segmentation masks and / or within the anatomical structure and / or across anatomical structures in a local neighborhood.

3. The method according to claim 1, wherein the received uncertainty metric is determined by a segmentation algorithm that provides the segmentation mask.

4. The method according to claim 1, wherein the scaling of the randomization includes scaling the distance to a sample center as a function of the received uncertainty metric.

5. The method according to claim 1, further comprising: Combining the at least one optical property of more than one sampling point, wherein the rendering is based on the at least one optical property of the more than one sampling point combined.

6. The method according to claim 1, wherein the rendering includes denoising.

7. The method according to claim 6, wherein the denoising includes sampling the medical imaging data included in the medical imaging dataset using a sequence of random number generators.

8. The method according to claim 1, wherein the medical imaging dataset is acquired by a medical scanner, wherein the medical scanner comprises: An X-ray device; An ultrasound device; A positron emission tomography device; A computed tomography device; A single photon emission computed tomography device; Or A magnetic resonance tomography device.

9. The method according to claim 1, wherein the at least one optical property comprises at least one of the following: Opacity; Reflectivity; Color; and / or At least one value indicating dispersion.

10. The method according to claim 1, wherein the rendering includes applying a reconstruction filter.

11. The method according to claim 1, wherein the rendering includes ray casting, Monte Carlo path tracing, subsurface scattering, and / or rendering a color grid.

12. The method according to claim 11, wherein the received uncertainty metric is mapped to at least one property of the subsurface scattering.

13. The method according to claim 12, wherein the subsurface attenuation of one or more wavelength bands includes the at least one property and is modulated according to the received uncertainty metric.

14. The method according to claim 1, wherein In the absence of user interaction, for the rendering, multiple random paths are accumulated and / or averaged over medical imaging data associated with different sampling points.

15. The method according to claim 1, wherein In the presence of user interaction, the rendering is performed without accumulating multiple random paths and / or averaging over the medical imaging data set associated with different sampling points; wherein after a predetermined period of time without receiving user interaction, multiple paths are accumulated and / or averaged over the medical imaging data set associated with different sampling points.

16. The method according to claim 1, wherein the rendering includes rendering a non-uniform representation of a volume and / or a surface, the volume and / or the surface including at least a portion of the segmentation mask and at least a portion of the anatomical structure.

17. A system for performing volume rendering and / or surface rendering on a medical imaging data set based on the optical properties of each sampling point, the system comprising: a memory configured to store instructions; and a processor configured to execute the instructions, the instructions including: a receiving module configured to receive an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of the medical imaging data set and / or an anatomical structure included in the medical imaging data set; a scaling module configured to scale the randomization of one or more sampling points based on the received uncertainty metric; a determination module configured to determine at least one optical property of each sampling point; and a rendering module configured to render a volume based on the voxels and / or render a surface based on the surface elements, wherein the rendering is based on the at least one optical property of each sampling point determined.

18. A non-transitory computer-readable medium having program elements stored thereon that can be read and executed by a computing device for rendering medical imaging data based on the optical properties of each sampling point, the program elements including instructions for: receiving an uncertainty metric for each voxel and / or each surface element related to a segmentation mask of the medical imaging data set and / or an anatomical structure included in the medical imaging data set; scaling the randomization of one or more of the sampling points based on the received uncertainty metric; determining at least one optical property of each sampling point as randomized; and rendering a volume based on the voxels and / or rendering a surface based on the surface elements, wherein the rendering is based on the at least one optical property of each sampling point determined.

Images