Direct volume rendering device

Abstract

To enable volume rendering of 3D image datasets at high frame rates. A direct volume rendering apparatus for volume rendering a three-dimensional image dataset includes a two-dimensional image calculator configured to: calculate, for each primary ray of a plurality of primary rays that travel through the three-dimensional image dataset and impinge on a respective pixel of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image dataset is projected, a color and an opacity along the primary ray; subdivide each primary ray into a plurality of segments based on the opacity and / or color along the respective primary ray; calculate, for each segment, a segment illumination value according to a predetermined illumination value calculation rule; and calculate a two-dimensional image to be presented to a user by projecting the color along the primary ray and the segment illumination value onto pixels based on the predetermined projection rule.

Description

[Technical field]

[0001] The present invention relates to a direct volume rendering (DVR) apparatus, method and computer program for volume rendering of three-dimensional image datasets, in particular three-dimensional medical image datasets. [Background technology]

[0002] Modern computer graphics applications impose high requirements on the hardware and rendering methods used in terms of resolution, frame rate and image quality. For most applications, a frame rate of 25 frames per second (fps) constitutes a minimum requirement. The image resolution used is often 2K, i.e. 2048x1080 pixels or higher. Virtual reality (VR) and augmented reality (AR) applications make these requirements even greater. Here, a frame rate of 60 fps or higher is often said to be the minimum to provide an immersive and comfortable user experience. Furthermore, the required frame rate is potentially doubled due to the inherent stereo rendering of the corresponding headset. While many recent headsets have a relatively low resolution, next-generation headsets such as Microsoft's HoloLens 2 feature a native 2K resolution per eye. Therefore, for a comfortable user experience on such headsets, two 2K ​​images need to be rendered at least at approximately 60 fps. At the same time, expectations regarding image quality and realism are extremely high these days, boosted by the increasing speed of developments in various industries, such as gaming, medicine or entertainment. Global illumination is a key technique for generating immersive and convincing images, but is still often not possible in real time. Real-time implementations typically rely on approximations such as ambient occlusion (AO), as described, for example, in Zhukov et al., "An ambient light illumination model", Rendering Techniques '98, pp. 45-55 (1998), the contents of which are incorporated herein by reference, or many variations of which can produce similar results. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Zhukov et al., "An ambient light illumination model", Rendering Techniques '98, pages 45 to 55 (1998) [Non-Patent Document 2] MS Langer et al., "Depth discrimination from shading under diffuse lighting", Perception 29(6), pages 649 to 660 (2000) [Non-Patent Document 3] F. Lindemann et al., "About the influence of illumination models on image comprehension in direct volume rendering", IEEE Transactions on Visualization and Computer Graphics 17(12), pages 1922 to 1931 (2011) Summary of the Invention [Problem to be solved by the invention]

[0004] The competing requirements of high frame rates, high resolution and high image quality are also relevant in the field of medical visualization. Here, in addition to traditional surface rendering, DVRs are used to provide insight into the interior of the human body based on computed tomography (CT) and magnetic resonance (MR), which are important for various medical applications, e.g. pre-operative planning. Several studies have shown that realistic representations, including important visual cues such as shadows in addition to a pleasing appearance, improve spatial understanding and the ability to recognize anatomical structures, as described for example in MS Langer et al., "Depth discrimination from shading under diffuse lighting", Perception 29(6), 64-660 (2000) and F. Lindemann et al., "About the influence of illumination models on image comprehension in direct volume rendering", IEEE Transactions on Visualization and Computer Graphics 17(12), 1922-1931 (2011), the contents of which are incorporated herein by reference. However, not all methods that allow approximating global illumination (GI) in real time are immediately applicable to DVR. Moreover, the computational complexity for DVR is generally higher than for surface rendering, since every point of the visualized volume may contain useful information, whereas surface rendering only processes discrete surfaces. As a result, even approaches that are transferable to DVR are generally much slower than their surface rendering counterparts. Therefore, all methods for interactive computation of GI in DVR must rely on some form of precomputation and storage of lighting values ​​for each voxel of the visualization dataset for reuse during interactive rendering.A limitation of such precomputed solutions is that they require recomputation updates every time the transfer function is updated or the volume is clipped. However, interactive clipping and transfer function definition is a requirement for many visualization scenarios. These standard operations are only partially possible with existing approaches, often resulting in frame rate degradation.

[0005] It is therefore an object of the present invention to provide a DVR (Direct Volume Rendering) device, a method and a computer program for volume rendering of a three-dimensional image data set at a higher frame rate. [Means for solving the problem]

[0006] In a first aspect of the invention, a direct volume rendering apparatus for volume rendering a three-dimensional image dataset is provided. The direct volume rendering apparatus comprises: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image data set and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image data set is projected, color(s) and opacity(s) along the primary ray; Subdividing each primary ray into a number of segments based on the opacity(s) and / or color(s) along the respective primary ray; · a) defining illumination value determination positions within each segment; and b) calculating, for each illumination value determination position, a respective segment illumination value according to a predetermined illumination value calculation rule, calculating, for each segment, a segment illumination value according to a predetermined illumination value calculation rule; Projecting the color(s) and segment lighting value(s) along the primary ray to a pixel based on a predetermined projection rule. The present invention also includes a two-dimensional image calculator configured to calculate a two-dimensional image to be presented to a user by: Volume rendering refers to the direct display of three-dimensional data on a two-dimensional screen. In a second aspect of the invention, a direct volume rendering method for volume rendering a three-dimensional image dataset is provided. The direct volume rendering method includes the steps of: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image data set and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image data set is projected, color(s) and opacity(s) along the primary ray; subdividing each primary ray into a plurality of segments based on the opacity(s) and / or the color(s) along the respective primary ray; · a) defining illumination value determination positions within each segment; and b) calculating, for each illumination value determination position, a respective segment illumination value according to a predetermined illumination value calculation rule, calculating, for each segment, a segment illumination value according to a predetermined illumination value calculation rule; projecting the colors and segment illumination values ​​along the primary rays onto the pixels based on a predetermined projection rule. Contains It is characterized by (morphology 13 ). In a third aspect of the invention there is provided a computer program for volume rendering a three-dimensional image dataset. The computer program includes program code for causing the direct volume rendering apparatus of the present invention to execute a plurality of steps of the direct volume rendering method of the present invention when the computer program is executed on a computer that controls the direct volume rendering apparatus. Do Including It is characterized by (morphology 14 ). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] A preferred embodiment of the invention is now presented. (Mode 1) See the first aspect of the present invention above. (Mode 2) In the direct volume rendering apparatus according to Mode 1, the three-dimensional image data set includes a plurality of image elements; The two-dimensional image computer is assigning a color and an opacity to each image element of the 3D image data set; and For each primary ray, calculating color(s) and opacity(s) along the primary ray based on the color(s) and opacity(s) assigned to the image element(s) traversed by the primary ray. Preferably, for each primary ray, the filter is configured to provide color(s) and opacity(s) along the respective primary ray. (Mode 3) In the direct volume rendering apparatus according to Mode 1 or 2, the two-dimensional image computer subdividing each primary ray into a plurality of segments by defining segment boundaries; Preferably, segment boundaries are defined by positions along the primary ray where a function of opacity and / or color exceeds a predefined threshold. (Feature 4) In the direct volume rendering apparatus according to feature 3, The opacity function is preferably a function selected from the group consisting of an identity function, a linear function, an integral type function and a differential type function, applied directly to the opacity and / or color and / or applied to the accumulated opacity. (Feature 5) In the direct volume rendering apparatus according to feature 3 or 4, said two-dimensional image calculator being configured to provide a number of threshold values; Preferably, segment boundaries are defined by positions along the primary ray where a function of opacity and / or colour exceeds a predefined threshold of said number of thresholds provided. (Feature 6) In the direct volume rendering device according to feature 5, Preferably, the difference between adjacent provided thresholds is constant. (Feature 7) In the direct volume rendering apparatus according to feature 5 or 6, Preferably, the two-dimensional image calculator is configured to provide the number of threshold values ​​according to a predetermined linear or exponential threshold function. 。 ( form 8 ) The above form Any of 1 to 7 In the direct volume rendering apparatus according to the present invention, The two-dimensional image calculator is preferably configured to define illumination value determination positions within each segment such that the illumination value determination positions are within a quadrant of each segment located at the end of each segment where a primary ray is traced from a pixel position into the volume. (form 9 ) The above form Any of 1 to 8 In the direct volume rendering apparatus according to the present invention, Preferably, said predetermined lighting value calculation rules include at least one of: a) calculating an ambient lighting value; and b) calculating at least one directional lighting value. (form 10 ) Form citing Form 2 above 9 In the direct volume rendering apparatus according to the present invention, Preferably, the predetermined lighting value calculation rules include: a) providing an environment map defining a direction and intensity of a first incident light, and calculating an ambient lighting value by modifying the intensity of the first incident light based on an opacity assigned to an image element; and b) providing directional lighting information for the lighting value determination position representing the light intensity of a second incident light from a virtual light source, and calculating at least one directional lighting value by modifying the intensity of the second incident light based on an opacity assigned to an image element. (form 11 ) The above form 10 In the direct volume rendering apparatus according to the present invention, The ambient lighting value is defining several cones within a hemisphere about the illumination value determination location, the hemisphere being defined by a gradient of color and / or opacity assigned to the image element at the illumination value determination location, the hemisphere being oriented in the direction of the gradient, an end face of the hemisphere intersecting the illumination value determination location and perpendicular to the gradient, and the apex of the cone being located at the illumination value determination location, defining, for each cone, a number of spheres centred on the axis of the respective cone, the radii of the spheres being greater the greater the distance to said apex; determining an occlusion value for each sphere and combining the occlusion values ​​determined for all spheres of the respective cone to determine a (single) occlusion value for the whole cone; determining, for each cone, a modified light intensity based on the occlusion value determined for the respective cone and a light intensity of light incident through the respective cone as defined by the environment map; combining the modified light intensities determined for the cones of each illumination value determination location to determine an ambient illumination value for each illumination value determination location. It is preferably calculated by: (form 12 ) Above form 1~ 11 In the direct volume rendering apparatus according to the present invention, The 2D image calculator is preferably further configured to calculate, for each segment, a segment color based on the color and opacity along the respective segment, and to project the colors and segment illumination values ​​along the primary ray to the pixel by projecting the segment color and segment illumination value. (form 13 ) See the second aspect of the invention above. (form 14 ) See the third aspect of the invention above.

[0008] Thus, it is only necessary to calculate one lighting value per segment, which significantly reduces the amount of calculations required for the calculation of the two-dimensional image. Furthermore, it has been found (in the present invention) that it is possible to subdivide the primary rays so that a high image quality can be achieved for the two-dimensional image, even if only one lighting value per segment is calculated, by subdividing the primary rays based on the opacity and / or color along each primary ray. Therefore, high-quality volume rendering can be provided at high frame rates. In particular, high-quality volume rendering with approximate global illumination can be provided at sufficiently high frame rates, even for AR / VR applications, without imposing any restrictions on interactive transfer function definition and clipping.

[0009] A three-dimensional image dataset may be considered a volumetric image dataset since it spans three different spatial directions. The three-dimensional image dataset may also be provided for different times, in which case the image dataset may be considered a four-dimensional dataset.

[0010] The 3D image dataset comprises image elements, preferably voxels. The 2D image calculator is preferably configured to provide, for each primary ray, colors and opacities along the respective primary ray by a) assigning a color and an opacity to each image element of the 3D image dataset, and b) calculating, for each primary ray, the colors and opacities along the primary ray based on the colors and opacities assigned to the image elements traversed by the primary ray. The assignment of the colors and opacities to the image elements is preferably defined by a transfer function (TF). Furthermore, the DVR device preferably includes an input receiving unit configured to enable a user to modify the TF, and the 2D image calculator is configured to update pixels of the 2D image in response to the modification of the TF. Thus, when the TF is modified, the subdivision of each primary ray into multiple segments based on possibly modified opacity(s) and / or color(s), the calculation for each segment of a segment illumination value according to a predetermined illumination value calculation rule, and the projection of the color(s) and segment illumination value(s) along the primary ray(s) onto the pixel(s) based on a predetermined projection rule are repeated in response to the modification of the TF, thereby allowing the user to modify the TF and monitor, inter alia, in real time, how the DVR visualization changes.

[0011] In one embodiment, the opacity assigned to image elements and the opacity calculated along the primary rays are expressed as alpha values, which refer to the alpha channel that complements a color model, such as the RGB color model, thereby forming, for example, an RGBA color model.

[0012] Furthermore, the two-dimensional image calculator can be configured to calculate a two-dimensional image for a portion of the three-dimensional image dataset, the portion being defined by at least one clipping plane, in particular by a set of clipping planes, the direct volume rendering device including an input receiving device configured to allow a user to modify the at least one clipping plane, in particular the position and orientation of the at least one clipping plane, and the two-dimensional image calculator configured to update pixels of the two-dimensional image in response to the modification of the at least one clipping plane. Thus, the two-dimensional image calculator can be configured to calculate a two-dimensional image such that the two-dimensional image represents only a portion of the three-dimensional image dataset that is not clipped or removed by the at least one clipping plane. Accordingly, the two-dimensional image calculator can be configured to perform at least the steps of: subdividing each primary ray into a plurality of segments, calculating for each segment a segment illumination value, and projecting the colors and segment illumination values ​​along the primary rays onto pixels only for the portion of the three-dimensional image dataset. Furthermore, at least one of the steps of assigning a color and an opacity to each image element of the three-dimensional medical image and the step of defining primary rays through the three-dimensional image dataset may be performed only for a defined portion of the three-dimensional image dataset, where for each primary ray the colors and opacities along the primary ray are calculated based on the colors and opacities assigned to the image elements passed by the primary ray. Thus, when the clipping plane is modified, i.e. when the portion to be imaged is modified, the pixels of the two-dimensional image are updated. This allows a user to modify the clipping plane and monitor how the DVR visualization changes, particularly in real time.

[0013] Furthermore, preferably, the two-dimensional image calculator may be configured such that the primary rays start at the position of the virtual camera, and the direct volume rendering device includes an input receiving unit configured to allow a user to modify the position and / or the orientation of the virtual camera, and the two-dimensional image calculator is configured to update pixels of the two-dimensional image in response to the modification of the position and / or the orientation of the virtual camera. Accordingly, the two-dimensional image calculator may be configured to perform at least the steps of subdividing each primary ray into a number of segments when the position of the virtual camera is changed, calculating for each segment a segment illumination value, and projecting the color and the segment illumination value along the primary ray onto the pixel. This allows a user to modify the position and / or the orientation of the virtual camera and monitor how the DVR visualization changes, notably in real time.

[0014] The input-accepting unit may be a unit that accepts information from an input device, representing an input made by a user, or the input-accepting unit may be the input device itself, for example a keyboard, a computer mouse, a touchpad, a microphone, especially of a headset, or other input device. The two-dimensional image is displayed on an output unit, such as a monitor, a display of a headset, etc.

[0015] The three-dimensional image dataset is preferably a medical image dataset, for example it may be a CT image dataset, an MR image dataset or an image dataset of another imaging modality.

[0016] The two-dimensional image calculator may be configured to calculate two two-dimensional images based on the three-dimensional image dataset in two different viewing directions to enable stereoscopic viewing, whereby a user can view the three-dimensional image dataset in three dimensions, and the two two-dimensional images may be displayed on two displays, each of which is seen by a corresponding one eye, and the two displays may be displays of a headset.

[0017] The two-dimensional image calculator is configured to subdivide each primary ray based on the opacity(s) and / or color(s) along the respective primary ray. This typically results in a number of segments having different lengths. In other words, the two-dimensional image calculator is configured to subdivide each primary ray based on the opacity(s) and / or such that not all segments have the same length (there are also segments of different lengths). In particular, the two-dimensional image calculator is configured to subdivide each primary ray into a number of segments by defining segment boundaries, The segment boundaries are defined by the positions along the primary rays where a function of the opacity(s) and / or color(s) exceeds a predefined threshold. The opacity(s) function can be, for example, a function selected from the group consisting of identity, linear, integrating and derivatizing functions, derivative, tangent and square root functions, applied directly to the opacity(s) and / or color(s) and / or to the accumulated opacity. It has been found (in the present invention) that by using these functions it is possible to define segments that allow high quality imaging of the three-dimensional image dataset. In particular, these segments allow a fast and good approximation of the illumination along each primary ray.

[0018] Preferably, the two-dimensional image calculator is configured to provide several thresholds, and the segment boundaries are defined by positions along the primary ray where a function of opacity and / or color exceeds a predefined threshold of the provided several thresholds. The difference between adjacent provided thresholds may be constant. The two-dimensional image calculator may be configured to provide several thresholds, for example according to a predefined linear or exponential threshold function. For example, the maximum number of segments may be determined by a function of opacity and / or color. If TIFF0007661470000001.tif6150, some threshold The linear threshold function that defines TIFF0007661470000002.tif6150 is TIFF0007661470000003.tif9150, which also results in threshold(s) equidistancy.

[0019] In a preferred embodiment, the two-dimensional image calculator is configured to calculate, for each segment of each primary ray, a segment color and a segment opacity based on the colors and opacities calculated along the respective primary ray in the respective segment. In particular, the segment color and the segment opacity may result from the colors and opacities calculated along the respective primary ray in the respective segment by an accumulation, which corresponds to an integration. The integration may be achieved by using a Riemann sum with a constant sample distance, which results in an accumulation of samples along the ray. If the opacities calculated along the primary ray are represented by an alpha value, the segment opacity may mean the segment alpha value, and the accumulation may be given by alpha blending, also known as alpha compositing. Thus, when iterating over samples of 3D image data during ray tracing along a ray segment, these samples may be mapped, for example using TF, to color and alpha values, and the accumulation of these color and alpha values ​​along the ray segment may then be produced by alpha blending, which is a numerical approximation of the attenuation of light as it "travels" through a medium.

[0020] In a preferred embodiment, the two-dimensional image calculator is configured to calculate the segment illumination value by defining for each segment of each ray an illumination value determination position within the respective segment and calculating a respective illumination value for, i.e. at, each illumination value determination position according to a pre-determined illumination value calculation rule. Preferably, the two-dimensional image calculator is configured to define an illumination value determination position within each segment such that it is within a quadrant of the respective segment located at the end of the respective segment along which the primary ray is traced from the pixel position towards the volume, said quadrant preferably corresponding to the quadrant of the respective segment located at the end of the respective segment along which the primary ray is traced from the virtual camera towards the volume. This illumination value determination position is therefore preferably at the end of the respective segment directed away from the respective pixel. By selecting an illumination value determination position within each segment in this way, the quality of the finally calculated two-dimensional image can be further improved.

[0021] In one embodiment, the predetermined lighting value calculation rules include at least one of a) calculating an ambient lighting value and b) calculating at least one directional lighting value. Preferably, the predetermined lighting value calculation rules include calculating an ambient lighting value and at least one directional lighting value for each lighting value determination location, and calculating a segment lighting value by combining the ambient lighting value and the at least one directional lighting value. For example, combining the ambient lighting value and the at least one directional lighting value may correspond to an addition. In particular, the predetermined lighting value calculation rules include a) calculating an ambient lighting value by providing an environment map defining a direction and intensity of a first incident light and modifying the intensity of the first incident light based on an opacity assigned to an image element, and b) providing directional lighting information for the lighting value determination location representing the light intensity of a second incident light from a virtual light source and calculating at least one directional lighting value by modifying the intensity of the second incident light based on an opacity assigned to an image element. In one embodiment, the environment map may also indicate the direction and color of the first incident light.

[0022] In one embodiment, the ambient lighting value is determined by: a) defining several cones within a hemisphere about the lighting value determination location, the hemisphere being defined by a gradient of color and / or opacity assigned to the image element at the lighting value determination location, the hemisphere being oriented in the direction of the gradient, the end faces of the hemisphere intersecting the lighting value determination location and perpendicular to the gradient, and the vertices of the cones being located at the lighting value determination location; b) defining, for each cone, a number of spheres centered on the axis of the respective cone, the radius of the sphere being larger the greater the distance to the corresponding vertex; c) defining a number of spheres centered on the axis of the respective cone, the radius of the sphere being larger the greater the distance to the corresponding vertex; a) determining an occlusion value for each sphere and combining the occlusion values ​​determined for all spheres of the respective cone to determine an occlusion value for the entire respective cone, d) determining for each cone a modified light intensity based on the occlusion value determined for the respective cone and the light intensity of the first incident light through the respective cone defined by the environment map, and e) combining the modified light intensities determined for the cones of the respective lighting value determination location to determine an ambient lighting value for the respective lighting value determination location. The combination of the modified light intensities determined for the cones of the respective lighting value determination location to determine an ambient lighting value for the respective lighting value determination location is preferably a weighted sum of these modified light intensities depending on the inclination of each cone relative to the respective gradient.

[0023] In a preferred embodiment, the environment map is adapted to the cones, i.e. it defines the total intensity of the first incident light in each cone, which further reduces the required calculations, since only one light intensity value per cone needs to be modified by using only one occlusion value determined for each cone.

[0024] The cones are preferably evenly distributed within each hemisphere, in one embodiment five or six cones are used in each hemisphere.

[0025] The spheres are preferably defined such that the radius of one sphere is twice the radius of the sphere immediately preceding it. Furthermore, the radius of each sphere is preferably equal to the distance of the sphere to the cone apex multiplied by the sine of half the cone opening angle. Thus, the spheres are as large as possible while still fitting into the cone.

[0026] Preferably, the occlusion value of the sphere is determined based on an opacity assigned to the center of the sphere according to volumetric data representative of the original three-dimensional image data having a level of detail corresponding to the distance between the center of the sphere and the apex of a cone defining the sphere. Preferably, the volumetric data representative of the original three-dimensional image data is pre-calculated and then stored for each of a predetermined number of levels of detail. By determining the occlusion value of the sphere based on such volumetric data, the amount of calculations required for the occlusion value calculation can be further reduced without substantially compromising the rendering quality.

[0027] The opacity assigned to a given image element of the volumetric data for a given level of detail corresponds, for example, to the sum or average of the assigned opacities of the image elements of the volumetric data for the next higher level of detail that correspond to the given image element, i.e., that give rise to the given image element.

[0028] In a preferred embodiment, the occlusion value of the spheres is modified based on the ratio of the distance between the centers of the spheres defined for the cones to the image element size, in particular the voxel size, in the original 3D image data.

[0029] The occlusion values ​​determined for the spheres can, for example, be added to result in an occlusion value for the corresponding cone. The number of spheres considered can depend on the size of the data set or on a predefined maximum ray length to limit the size of the considered neighborhood. In any case, the spheres can only be considered until the cone axis leaves the 3D image data or penetrates a clipping plane.

[0030] As mentioned above, a modified light intensity may be determined for each cone based on the occlusion values ​​determined for the different cones and the intensity of the first incident light through the respective cones as defined by the environment map. For example, the modified light intensity determined for each cone may correspond to the product of the occlusion value determined for the respective cone and the light intensity of the first incident light through the respective cone as defined by the environment map. As further mentioned above, the modified light intensities determined for the cones corresponding to each illumination value determination location may be combined to determine an ambient lighting value for each illumination value determination location, but preferably the combination of the modified light intensities determined for the cones of each illumination value determination location to determine an ambient lighting value for each illumination value determination location is a sum of these modified light intensities weighted in response to the inclination of each cone relative to the respective gradient. The weights may be given, inter alia, by the cosine of the slope for each gradient of each cone.

[0031] In a preferred embodiment, at least one direct lighting value is determined by: a) defining several cones within a hemisphere about a lighting value determination location, the hemisphere being defined by a gradient of the colors and / or opacities assigned to the image elements at the lighting value determination location, the hemisphere pointing in the direction of the gradient, an end face of the hemisphere intersecting the lighting value determination location and perpendicular to the gradient, and the apexes of the cones being located at the lighting value determination location; b) defining for each cone a number of spheres centered on the axis of the respective cone, the hemisphere being defined by a gradient of the colors and / or opacities assigned to the image elements at the lighting value determination location, the hemisphere pointing in the direction of the gradient, an end face of the hemisphere intersecting the lighting value determination location and perpendicular to the gradient, and the apexes of the cones being located at the lighting value determination location; the radius is larger the greater the distance to the vertex, c) determining an occlusion value for each sphere and combining the determined occlusion values ​​for all spheres of the respective cone to determine an occlusion value for the entire cone, d) determining for each cone a directional light contribution indicative of a fraction of the light intensity of the second incident light from the virtual light source associated with the respective cone, and e) determining a modified light intensity as at least one directional light value based on the determined occlusion values ​​and the directional light contributions for the several cones. Preferably, the directional light contribution for the cone depends on the direction of the virtual light source determined by the directional lighting information and on the direction of the respective cone axis, in particular on the angle between the direction of the virtual light source determined by the directional lighting information and the direction of the respective cone axis. For example, the directional light contribution may be highest for the cone that contains the virtual light source direction determined by the directional lighting information, and may be smaller the greater the inclination of each cone axis with respect to the virtual light source direction. The directional light contribution may also depend on the opening angle of each cone, which may be the same for all cones. For example, to avoid illumination discontinuities that may occur when a virtual light source generating a second incident light moves out of the solid angle covered by any one cone in the hemisphere into another cone, the contribution may desirably be selected to ensure a gradual (soft) transition between adjacent cones without any gaps or large jumps.The modified light intensity, and therefore at least one directional lighting value, can be obtained by multiplication of the light intensity of the second incident light with a weighted sum of the occlusion values ​​of said several cones, where the weights depend on the direction of the virtual light source determined by the directional lighting information and on the direction of the respective cone axis, in particular on the angle between the direction of the virtual light source determined by the directional lighting information and the direction of the respective cone axis, possibly further on the opening angle of the respective cone.

[0032] The 2D image calculator may further be configured to calculate, for each segment, a segment color based on the color(s) and opacity(s) along the respective segment, and to project the color(s) and segment illumination value(s) along the primary ray to the pixel(s) by projecting the segment color(s) and segment illumination value(s). A corresponding projection rule defines how the compositing of the color(s) and segment illumination value(s) is performed. For example, the compositing of the color(s) and segment illumination value(s) according to the projection rule may be an addition of the illuminated segment color(s) for all segments, while the illuminated segment color for a segment (i.e. the final color value for the segment taking illumination into account) may be calculated by going to the multiplication of the segment color with the respective segment illumination value. Since the segment colors are already known from the initial ray tracing and accumulation of colors and opacities along the segments of the primary ray, e.g. using alpha blending, the final projection step, i.e. performing an addition to obtain the final illuminated pixel value in the 2D image, becomes extremely simple.

[0033] In another aspect of the invention, there is provided a direct volume rendering method for volume rendering a three-dimensional image dataset, the direct volume rendering method comprising the steps of: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image data set and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image data set is projected, color(s) and opacity(s) along the primary ray; subdividing each primary ray into a plurality of segments based on the opacity(s) and / or the color(s) along the respective primary ray; Calculating for each segment a segment lighting value according to a predetermined lighting value calculation rule; projecting the colors and segment illumination values ​​along the primary rays onto the pixels based on a predetermined projection rule. Includes.

[0034] In a further aspect of the invention there is provided a computer program for volume rendering a three-dimensional image dataset, the computer program comprising program code means for causing a DVR device according to any of claims 1 to 19

[13] to perform the steps of the DVR method according to claim 20

[14] , when the computer program is executed on a computer controlling the DVR device.

[0035] The system of claim 1, the method of claim 14 and the computer program of claim 15 are to be understood as having similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0036] It is to be understood that a preferred embodiment of the invention can also be any combination of the dependent claims with the respectively corresponding independent claims.

[0037] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments (or examples) described hereinafter. [Brief description of the drawings]

[0038] [Figure 1] 1 is a schematic illustrative example of one embodiment of a DVR device. [Diagram 2] 1 is a schematic illustrative example of subdivision of a primary ray based on opacity along the primary ray; [Diagram 3] Schematic illustrative example of subdivision of primary rays based on opacity values ​​by using an equidistance threshold. [Figure 4] A schematic illustration of the subdivision of a hemisphere into 6 cones used to calculate a segment illumination value for a position within a segment of a primary ray. [Diagram 5] A schematic illustration of sampling of one cone with varying sampling density along the secondary ray. [Figure 6] 1 is an example flow chart illustrating an embodiment of a DVR method. EXAMPLES

[0039] 1 shows a schematic illustrative example of a DVR device 1 for volume rendering of a three-dimensional medical image dataset. The DVR device 1 includes a two-dimensional image calculator 2 configured to calculate a two-dimensional image to be shown to a user.

[0040] The DVR device 1 further includes an input device 3 such as a keyboard, a computer mouse, a touchpad, a microphone, especially of a headset, or other input device. The 2D image is shown on an output unit 4 such as a monitor, a display of a headset, etc.

[0041] The 2D image calculator 2 assigns a color and an opacity to each voxel of the 3D medical image and defines primary rays through the 3D image dataset. The primary rays originate from a virtual camera position and traverse through respective pixels of the 2D image in the 2D image plane onto which the 3D medical image dataset is to be projected. For each primary ray, the colors and opacities along the primary ray are calculated based on the colors and opacities assigned to the voxels traversed by the primary ray.

[0042] Furthermore, the two-dimensional image calculator 2 subdivides each primary ray into segments based on the opacity values ​​along said primary ray. In this embodiment, the two-dimensional image calculator 2 is configured to subdivide each primary ray into segments by defining segment boundaries, a segment boundary being defined by a position along the primary ray at which the accumulated opacity values ​​exceed a respective predefined threshold. Adjacent predefined thresholds preferably have a certain distance, which may be, for example, TIFF0007661470000004.tif9150, where s is a counter for each segment, and s max is the maximum number of segments, which can be predetermined or automatically determined by the two-dimensional image calculator 2.

[0043] Accumulated opacity represents the attenuation of radiance that "travels" through a medium. For example, an accumulated opacity of 0.5 means that 50% of the radiance of a ray is absorbed by the medium. It is based on the physical laws of light, and in volume rendering, the physical basis for simulating the propagation of light in a volume has been formulated, for example, by the emission-absorption model in N. Max's paper "Optical Models for Direct Volume Rendering", IEEE Transactions on Visualization and Computer Graphics, 1(2), pages 99 to 108 (1995), which is incorporated herein by reference. Usually, accumulated opacity is calculated by a technique called alpha blending, which is a numerical approximation of the physical laws formulated in N. Max's paper. Preferably, the cumulative opacity is calculated according to the product of equation (4) below.

[0044] Fig. 2 shows a schematic illustrative example of a primary ray 10 subdivided into a number of segments. Four of these segments, I, II, III, IV, are shown, with their respective cumulative opacity exceeding a threshold at boundaries 11, 12, 13, 14. It should be noted, however, that the threshold 14 may not be met (reached) since early ray termination may be applied, in which case ray tracing is terminated (stopped) after reaching some predefined cumulative opacity value, such as 0.99 or 0.96. In Fig. 2, α represents the volumetric opacity for each colored region, and the values ​​along the primary ray 10 represent the cumulative opacity.

[0045] FIG. 3 shows a further example to explain the subdivision of a primary ray. In this figure, the horizontal axis represents the position along the corresponding primary ray, and the vertical axis represents the accumulated opacity. Furthermore, in this figure, the lines 20...23 represent equidistant thresholds. As can be seen in this figure, each time the accumulated opacity reaches one of the thresholds 20...23, the boundaries of the corresponding segments I', II', III', IV' are defined. It should be noted that FIG. 2 and FIG. 3 are based on different opacity values, i.e. FIG. 3 does not correspond to the situation shown in FIG. 2.

[0046] The two-dimensional image calculator 2 is further configured to calculate, for each segment, a segment illumination value by defining a position within the respective segment and calculating for each position a respective illumination value according to a predetermined illumination value calculation rule. Preferably, the two-dimensional image calculator 2 is configured to define the position within each segment such that the position is within a quarter of the respective segment that is located at the end of the respective segment with respect to a primary ray that is traced from the virtual camera / pixel position towards the volume. In FIG. 2, this "quarter" is on the right side of each segment I, II, III, IV, and in the figure, the respective positions for which the respective illumination values ​​are calculated are depicted by hemispheres. In this example, the respective positions are at the respective ends of each segment I, II, III, IV.

[0047] In this embodiment, the predetermined lighting value calculation rules include calculating an ambient lighting value and directional lighting values ​​for each lighting value determination location, and calculating a segment lighting value by combining the ambient lighting value and the directional lighting values. In particular, the predetermined lighting value calculation rules include: a) providing an environment map defining a direction and intensity of a first incident light, and calculating an ambient lighting value by modifying the intensity of the first incident light based on the opacity values ​​assigned to the voxels; and b) providing directional lighting information for the lighting value determination location representing the light intensity of a second incident light from virtual light sources, and calculating the directional lighting values ​​by modifying the intensity of the second incident light based on the opacity values ​​assigned to the voxels.

[0048] In this embodiment, for each lighting value determination location, a corresponding hemisphere is defined. The hemisphere is defined by a gradient of opacity assigned to the voxels at the lighting value determination location, the hemisphere is oriented in the direction of the gradient, and the end face of the hemisphere intersects the lighting value determination location and is perpendicular to the gradient. Furthermore, within each hemisphere, a number of cones 30 are defined. The cones 30 are preferably evenly distributed within each hemisphere. In this embodiment, six cones are used within each hemisphere, as shown in FIG. 4.

[0049] Furthermore, for each cone 30, spheres are defined that are centered on the axis 40 of the respective cone 30, the radii of which are larger the greater the distance to the respective apex, as shown in Figure 5. The axis 40 of each cone 30 can also be thought of as a secondary ray.

[0050] An occlusion value is determined for each sphere, and the occlusion values ​​determined for all spheres of each cone 30 are combined to determine an occlusion value for the entire respective cone 30. Furthermore, for each cone 30, a modified light intensity is determined based on the occlusion value determined for that cone 30 and the light intensity of the first incident light passing through that cone 30 as defined by the environment map, and the modified light intensities determined for the cones 30 of each lighting value determination location are combined to determine an ambient lighting value for each lighting value determination location. To determine an ambient lighting value for each lighting value determination location, the combination of the modified light intensities determined for the cones 30 of each lighting value determination location is preferably a weighted sum of these modified light intensities, each modified intensity being weighted depending on the inclination angle for each gradient of each corresponding cone. The weight may be given, inter alia, by the cosine of the respective inclination angle.

[0051] In a preferred embodiment, the environment map is adapted to the cones 30, i.e. it defines the total intensity of the first incident light in each cone 30. This reduces the required calculations, since it is only necessary to modify only one light intensity value per cone 30 respectively by using only one occlusion value determined for that cone 30.

[0052] The spheres are defined such that the radius of each sphere is twice the radius of the sphere immediately preceding it, and furthermore, the radius of each sphere is equal to the distance from the sphere to the apex of the cone multiplied by the sine of half the opening angle of the cone.

[0053] Each direct lighting value is calculated by determining a modified light intensity based on a) the determined occlusion (value) for a cone 30 containing a ray of the second incident light from the respective virtual light source and b) the light intensity of the second incident light passing through the cone 30 as defined by the directional lighting information. For example, the modified light intensity, and therefore each directional lighting value, is obtained by multiplying the determined occlusion (value) by the light intensity of the second incident light.

[0054] In particular, a modified light intensity and thus at least one directional lighting value can be determined based on the occlusion values ​​determined for all cones in the hemisphere by multiplying the light intensity of the second incident light by a weighted sum for the occlusion values ​​for all cones in the hemisphere, where the weights depend on the respective angles between the direction of the second incident light and the respective cone axis and on the respective cone opening angle.

[0055] After one corresponding segment illumination value is determined for each segment, i.e. for each respective illumination value determination position, the two-dimensional image calculator projects the colors and segment illumination values ​​along the primary rays to the pixels based on a predetermined projection rule. In particular, the two-dimensional image calculator 2 is configured to calculate one segment color for each segment based on the colors and opacities along the respective segments, and to project the colors and segment illumination values ​​along the primary rays to the pixels by projecting the segment colors and segment illumination values. The projection rule may be, for example, multiplying each segment color by the respective segment illumination value and adding the resulting products. However, other projection rules could also be used.

[0056] The computation of each segment illumination value includes, as described above, the computation of ambient occlusion (AO), i.e., the occlusion of the incident ambient light at each illumination value determination location by the voxels of the 3D (image) data set used to determine the ambient illumination value. To compute AO for volume data, multiple AO samples per primary ray can be used to obtain semi-transparent volume data, and in this embodiment, the segment-based algorithm can include at least three basic steps: (i) generation of level of detail (LOD) occlusion volumes, (ii) gathering segment information from the original volume data during initial ray tracing, and (iii) illumination computation and final composition, i.e., projection. In the following paragraphs, the steps of one embodiment are described based on problems encountered during the development of DVR devices.

[0057] The physical basis for simulating the propagation of light in a volume can be given by an emission-absorption model, such as the emission-absorption model described in the above-mentioned article by N. Max. According to this article by Max and the further article by D. Joensson, "A survey of volumetric illumination techniques for interactive volume rendering", Computer Graphics Forum, 33(1), pp. 27-51 (2014), the contents of which are hereby incorporated by reference, the point Ray from TIFF0007661470000005.tif6150 TIFF0007661470000006.tif6150 along the point The attenuation radiance reaching TIFF0007661470000007.tif6150 is given by equation (1) (but ignoring emission): TIFF0007661470000008.tif12150, where TIFF0007661470000009.tif6150 is Ray TIFF0007661470000010.tif6150 represents samples along the line, TIFF0007661470000011.tif6150 is the background energy entering the volume, TIFF0007661470000012.tif6150 is the scattering coefficient and the absorption period is The file is TIFF0007661470000013.tif9150.

[0058] In formula (1), TIFF0007661470000014.tif6150 is TIFF0007661470000015.tif6150 to material shading function, TIFF0007661470000016.tif6150 in all directions TIFF0007661470000017.tif6150 is the incident radiance from every sample position in the view ray. All incidence directions on the unit sphere for TIFF0007661470000018.tif6150 Direction from TIFF0007661470000019.tif6150 Radiance scattered to TIFF0007661470000020.tif6150 Represents TIFF0007661470000021.tif11150.

[0059] To be efficient enough for real-time rendering, in DVR the integral in equation (1) is approximated by computing a Riemann sum for the primary rays and compositing these samples using alpha blending, resulting in a discrete volume rendering equation: TIFF0007661470000022.tif873 is obtained.

[0060] Here, the final pixel color for a ray traveling through a volume is the product of the alpha-blended opacity values ​​from the preceding samples. TIFF0007661470000023.tif6150 and all samples along that ray Color values ​​for TIFF0007661470000024.tif6150 The local illumination is now calculated for each step, and the sample color is determined by multiplying it by TIFF0007661470000025.tif6150. TIFF0007661470000026.tif6150. However, this is affected by global lighting effects and This would impair scattering (represented by the term TIFF0007661470000027.tif6150) because computing these effects is still very expensive since it again involves evaluating the incident radiance over the whole unit sphere for every step along the ray. A common approximation is to simply consider point or directional light sources and compute simple Phong shading. This would of course still impair occlusion of nearby structures and global lighting effects, which is why approximating local occlusion with AO and incident light with a High Dynamic Range Image (HDRI) environment map is desirable. TIFF0007661470000028.tif6150 is all directions Since this is the incident radiance from TIFF0007661470000029.tif6150, TIFF0007661470000030.tif6150 is the Radiance environment map TIFF0007661470000031.tif6150. Furthermore, in this embodiment, the gradient The current point, defined by TIFF0007661470000032.tif6150 Only the upper hemisphere of TIFF0007661470000033.tif6150 is considered. In particular, the gradient is calculated based on the opacity values ​​of neighboring voxels. The resulting vector then defines a plane in three-dimensional space, i.e., the plane is perpendicular to the gradient and intersects the point at which the calculation is currently being performed. The hemisphere is preferably the one on the side of the plane that lies in the direction of the gradient.

[0061] For TIFF0007661470000034.tif6150, this is then TIFF0007661470000035.tif891 results.

[0062] In this formula, TIFF0007661470000036.tif6150 is a vector TIFF0007661470000037.tif6150 is a radiance value defined as the value of the radiance environment map at the location where the radiance values ​​of the radiance environment map intersect with the sphere on which they are defined. Further, TIFF0007661470000038.tif6150 is a visibility function that gives a value between 0 and 1, and the value is the vector The smaller the occlusion in the direction of TIFF0007661470000039.tif6150, the larger it becomes. An incident ray along TIFF0007661470000040.tif6150 is occluded and its position If TIFF0007661470000041.tif6150 is not reached, the visibility function gives a value of 0. If an incident ray along TIFF0007661470000042.tif6150 is not occluded at all, the visibility function gives a value of 1.

[0063] The radiance map (given as a 360°×180° HDRI panoramic image) is desired to provide the incident radiance for one cone with only one required sampling operation. For this purpose, pre-convolved lighting is employed as described in O. Klehm et al., "Bent Normals and Cones in Screen-Space", VMV 2011 - Vision, Modeling and Visualization, pp. 177-182 (2011), the contents of which are incorporated herein by reference. Thus, the radiance map can be calculated from the "standard" environment map with filtering / blurring (this filtering can be performed by free third-party software tools) corresponding to the required cone radius. If support for multiple cone radii is desired, the radiance map could also contain multiple LOD levels with increasing filter size / blurring. The radiance map can then be imported into the renderer / shader as a "CubeMap texture" and sampled using standard GLSL commands. A CubeMap is defined as an infinite sphere around the scene, and is scaled in one direction. Provides the incident radiance value for the intersection of TIFF0007661470000043.tif6150 with this sphere.

[0064] TIFF0007661470000044.tif6150 is the current location TIFF0007661470000045.tif6150 and incident light direction Assuming the visibility function for TIFF0007661470000046.tif6150, and using the proposed lighting model, the complete formula can be written as: TIFF0007661470000047.tif12150

[0065] However, in other embodiments, other lighting models can be used, in particular the lighting model can easily be combined or extended to more sophisticated models and even directional or point light sources, see also further below in this regard.

[0066] AO (or Since a brute-force approach in which the radiance (item 6150) is calculated for every step along each ray is not feasible, the 2D image calculator 2 is configured to perform a segment-based calculation in which every primary ray is divided into multiple segments. Each of these segments contains the alpha-blended accumulated radiance for the corresponding part of the ray, so that the final radiance can be easily fetched by simply adding them together. This is similar to the idea of ​​local piecewise integration described in the paper by F. Hernell et al., "Interactive global light propagation in direct volume rendering using local piecewise integration", SPBG'08 Proceedings of the Fifth Eurographics / IEEE VGTC Conference on Point-Based Graphics, pp. 105-112 (2008), the contents of which are incorporated herein by reference. However, in this embodiment, the subdivision is opacity-driven and is performed on the primary rays instead of the secondary rays.

[0067] The two-dimensional image calculator 2 is configured to subdivide each primary ray into segments based on the accumulated opacity along the respective primary ray, i.e. the length of the segments depends on the accumulated alpha value during tracing of the (hence varying) length of the primary ray. Preferably, the two-dimensional image calculator 2 is configured to subdivide each primary ray into segments by defining segment boundaries, the segment boundaries being defined by positions along the primary ray at which the accumulated opacity exceeds predefined thresholds. The difference between adjacent predefined thresholds may be constant. In general, they may be linearly distributed, but may also be distributed in other ways, such as exponentially. Thus, the current segment is Let TIFF0007661470000049.tif6150 be the current (actual) number of segments, TIFF0007661470000050.tif6150 as the maximum number of segments, The next cumulative opacity threshold (which in the simplest case is just TIFF0007661470000051.tif9150) is reached, it is changed.

[0068] The maximum number of segments may be set by the user, however, preferably the maximum number of segments is predetermined or automatically determined by the DVR device and not set by the user.

[0069] Each time a segment is changed, the current position in the volume TIFF0007661470000052.tif6150, i.e., the illumination value determination position, and the current cumulative color given by equation (4). TIFF0007661470000053.tif6150 (i.e. each position TIFF0007661470000054.tif6150 and the respective segment colors of the respective segments are stored for later calculation of the segment illumination value and synthesis of the final pixel value by adding together the attenuated segments, as will be explained in detail below with reference to equation (7). The separation of the initial ray marching and the segment illumination value calculation has the advantage that the ray marching, which is usually parallelized in thread groups working in lockstep, is not hindered by the segment illumination value calculation, since the segments per ray can also vary, and thus faster threads will not have to wait for slower threads. In fact, the maximum number of segments is only used if a ray reaches all the defined opacity thresholds. The AO is only calculated once per segment, which allows for inherent adaptivity for the ambient occlusion / lighting calculation. The pixel color for the current pixel of the 2D image TIFF0007661470000055.tif6150 is therefore a discretized volume rendering equation (4) TIFF0007661470000056.tif, which corresponds to having 6,150 segments, TIFF0007661470000057.tif is given by splitting it into 6150 partial sums as follows: TIFF0007661470000058.tif9147

[0070] By making the decision by cumulative opacity, the (DVR) device is able to identify regions or segments that have similar opacity to some extent and place the segment boundaries near the sudden increase in the (cumulative) opacity. In this way, and by placing the illumination value determination positions at the boundaries in the last quadrant of each segment, well-defined surface-like structures receive a reasonable illumination approximation. However, even for soft tissues, which are characterized by a slow and continuous increase in cumulative opacity, these regions are quite homogeneous and This heuristic works because the specific positioning of TIFF0007661470000059.tif6150 is not critical.

[0071] Segment-based lighting is based on the assumption that illumination does not change too much within a segment and is easily reusable for all points of the segment. However, there are other ways in which segment illumination values ​​can be made available. In principle, an illumination value could be interpolated for each position in the segment based on the segment illumination value of the current segment and the segment illumination values ​​of the two adjacent segments. Common interpolation schemes such as nearest neighbor, linear or cubic interpolation could be used. A practical approach to use segment illumination values ​​for all points of a segment is similar to nearest neighbor interpolation. This preferred approach allows a simple interpolation that gives very good results.

[0072] Since the Riemann sum of the primary rays has already been calculated in equation (4), and the AO values ​​and illumination duration are considered constant for each segment, we can use them and their corresponding segment colors in the later compositing steps. TIFF0007661470000060.tif6150 can then be multiplied to get the final illuminated pixel value. Thus, the segments are only relevant for the lighting calculations and do not change the primary ray tracing step. The final compositing equation, which is a numerical approximation for equation (6), is given by the following equation (8): TIFF0007661470000061.tif9115 Where: TIFF0007661470000062.tif6150 is the ambient occlusion term TIFF0007661470000063.tif6150 and incident light Current segment consisting of TIFF0007661470000064.tif6150 Lighting for TIFF0007661470000065.tif6150.

[0073] Since the alpha values ​​per segment are already alpha blended during the initial ray tracing, it is not necessary to alpha blend the segments to obtain the final pixel value. The maximum number of segments is determined by the complexity of the underlying data, but usually this can be very low to obtain a reasonable approximation of the occlusion. Typical values ​​for this parameter range from 5 to 30 segments. This mainly depends on the complexity of the underlying data and the TF. A value of 5 segments can already give good results, but values ​​above 30 usually do not give any further improvement in visual quality. For a typical size dataset with a resolution of 512x512x512 voxels and a typical TF, a maximum number of segments of 10 already gives good results close to the path traced one. Thus, in a preferred embodiment, the maximum number of segments per ray is 10.

[0074] In this embodiment, the two-dimensional image calculator 2 is adapted to use voxel cone tracing AO as the AO model. TIFF0007661470000066.tif6150 Orientation Let the visibility function for TIFF0007661470000067.tif6150 be the point The average accessibility of TIFF0007661470000068.tif6150 is defined as: TIFF0007661470000069.tif11150

[0075] This term is used to determine the directional visibility of the point can be approximated by tracing additional rays, or secondary rays, over the hemisphere of TIFF0007661470000070.tif6150. In surface rendering, the visibility function TIFF0007661470000071.tif6150 is normally binary since there are only two possibilities: the ray either intersects with other geometry or it does not (unless transparent surfaces are also considered). For volume rendering, i.e., for the present disclosure, this is reformulated to fit for a continuous visibility function: Here, visibility is continuous and the visibility of the secondary rays is TIFF0007661470000072.tif6150, which forms the basis of an emission-absorption model such as that described in the above-mentioned paper by Max et al. Thus, the AO term for a segment is TIFF0007661470000073.tif6150 spanning the hemisphere The quadratic ray of TIFF0007661470000074.tif6150 can be approximated by equation (10): TIFF0007661470000075.tif894

[0076] Since AO is evaluated at multiple positions along each ray, this immediately results in a very high amount of required secondary rays that need to be traced through the volume. Since the performance penalty for this secondary ray tracing is severe due to frequent memory accesses, in this embodiment, other schemes are used to reduce the required number of samples. Additionally, since indirect light and AO are very infrequent, it has been found (in the present invention) that it is not necessary to use the highest resolution volume to calculate occlusion. In fact, the further away from the initial sample position, the wider the representation of the volume can be. The 2D image calculator 2 is therefore adapted to generate additional occlusion volumes that contain an exact copy of the main volume at multiple resolutions. Preferably, the multi-resolution volumes are recalculated when the TF is changed in order to account for the full dynamic changes of the TF. This adds only a small performance penalty, since downsampling of the volume can be performed very efficiently in the graphics processing unit (GPU) of the 2D image computer 2. Sampling of the environment at each initial sample location can therefore be designed as an LOD method that significantly accelerates the tracing of secondary rays. This is illustrated in FIG. 5, where a secondary ray 40 at the center of the cone 30 is sampled at multiple sample locations 41, 42, 43, but with a spatial density of 100%. The greater the distance to that location 44, which corresponds to TIFF0007661470000076.tif6150, the smaller the image.

[0077] The 2D image calculator 2 is configured to calculate AO in a manner similar to that described in the article "Interactive indirect illumination using voxel cone tracing", C. Crassin et al., Computer Graphics Forum 30(7), pp. 1921-1930 (2011), the contents of which are incorporated herein by reference. However, in contrast to the method described in this article, the 2D image calculator 2 does not require voxelization of the scene, since the volume data is already voxelized. Additionally, the 2D image calculator 2 does not use hierarchical data structures such as octrees for the illumination calculations, since no neighbor information per voxel is required, thus significantly reducing the complexity of the algorithm. Furthermore, in the present invention, preferably, the voxel cone tracing method is applied for the calculation of occlusion, rather than indirect illumination, as in Crassin's original approach. However, the 2D image calculator 2 hampers the positive aspects of that approach, mainly the absence of noise and the significant improvement in performance.

[0078] Similar to the technique described in the article "Ambient occlusion using cone tracing with scene voxelization" by ECD Favera et al., 25th SIBGRAPI Conference on Graphics, Patterns and Images, pp. 142-149 (2012), the contents of which are incorporated herein by reference, the 2D image calculator 2 is preferably configured to use 6 cones with equal apertures of 67° per cone for occlusion gathering, since there is no specular / high frequency information and the layout of 6 cones 30 introduces smaller gaps than the original 5-cone layout proposed in the above-mentioned article by Crassin et al. These cones are shown exemplarily in FIG. 4. It has been found (in the present invention) that a layout with 3 cones or only 1 cone does not significantly improve the performance and produces visually worse results. This is why 2D image calculator 2 preferably uses six cones, although other numbers of cones and different cone apertures can be used.

[0079] Now, to calculate the occlusion value per cone, the 2D image calculator 2 performs the tracing of the corresponding quadratic rays through the volume and approximates the cone integral. For this, a multi-resolution occlusion volume is used by summing the occlusion values ​​from the corresponding LOD levels along the cone to get the final approximate occlusion. A step size is then defined such that the extent of the cell of the current LOD level fits into the diameter of the cone. This can therefore be thought of as packing spheres into the cone with a doubling of the radius for each new sphere, since the size of each cell in the multi-resolution volume doubles from each LOD level to the next LOD level (see Figure 5). This then becomes the step size for each LOD level. About TIFF0007661470000077.tif6150 Cone Aperture TIFF0007661470000078.tif6150, the current distance from the start of the cone This results in the following equation (11), given in voxels, for calculating TIFF0007661470000079.tif6150: TIFF0007661470000080.tif14150

[0080] This results in an exponential increase in step length that allows the 2D image calculator 2 to sample a large neighborhood around each point with a very small number of required samples, and allows it to not only simulate local AO, but also to simulate both large-scale occlusion effects and sharp details close to the current point. Larger Step Sizes Each sample to fit TIFF0007661470000081.tif6150 The opacity of TIFF0007661470000082.tif6150 is in standard step width TIFF0007661470000083.tif6150. See also Crassin et al., supra. TIFF0007661470000084.tif8150

[0081] The 2D image calculator 2 is in this embodiment arranged to perform the interpolation such that each location of the volume is sampled trilinearly, which ensures a smooth interpolation result, although in other embodiments the interpolation can of course be performed in other ways.

[0082] The 2D image calculator 2 can be configured to use HDRI environment maps to illuminate the scene, i.e. to calculate the lighting values ​​by using the HDRI environment map. This has the advantage that the generated 2D images to be displayed can be used in AR scenarios, since the HDRI environment map can generate very natural and realistic lighting. As mentioned above, the 2D image calculator 2 is configured to use simplified lighting calculations by calculating lighting and occlusion only once per segment. The 2D image calculator 2 can use the already calculated occlusion values ​​for each cone to calculate a directional lighting approximation by evaluating the incident light for each cone and then attenuate it with the corresponding occlusion value. All this requires is very few additional texture lookups to the environment map, yet still results in a reasonable approximation of the incident light for the current segment.

[0083] Now, to calculate the incident light per cone, the 2D image calculator 2 could be configured to actually integrate again over the solid angle of each cone, for example by sampling the environment map multiple times. However, the 2D image calculator 2 is preferably configured to avoid this by using pre-convolution lighting as described in the above-mentioned article by Klehm. For this purpose, mipmapped HDRI maps can be used, where each mipmap effectively acts as a pre-integrated incident light with the highest mipmap representing the radiance for a cone radius of preferably 90°. Thereby, the incident light per cone is fetched by sampling the environment map once per mipmap level corresponding to the cone radius. Note that if only one constant (fixed) cone aperture angle is used, the pre-calculation of one map level is sufficient.

[0084] TIFF0007661470000085.tif6For a layout with 150 cones, the points TIFF0007661470000086.tif6150, i.e., per lighting determination position, hence points The illumination per segment, i.e. the illumination value, in which TIFF0007661470000087.tif6150 resides is therefore given by the following formula: TIFF0007661470000088.tif11114 Where: TIFF0007661470000089.tif6150 is currently the incident light for cone K. If the environment map is the only light source, TIFF0007661470000090.tif6150 is the direction of the secondary ray The light from the environment in TIFF0007661470000091.tif6150 and the cone aperture corresponding to the corresponding mipmap level of the environment map Defined by TIFF0007661470000092.tif6150 as: TIFF0007661470000093.tif6150

[0085] Thus, for example, a constant cone aperture that is the same for all cones Regarding TIFF0007661470000094.tif6150, TIFF0007661470000095.tif6150 is valid It only depends on TIFF0007661470000096.tif6150.

[0086] In a similar way, this can be extended, with very little computational effort, to approximate directional lighting values ​​such as incident light from light sources such as directional light sources (incident light has parallel rays) or point light sources (light coming from a single point, incident light rays are not parallel) and their respective shadows. For this purpose, the shadow values ​​for each light source are approximated as an interpolation of the occlusion values ​​of the cones already calculated (see equation (13)). Direction For the incoming light from TIFF0007661470000097.tif6150, the incoming light per (unshadowed) cone K is: This can be calculated using the following formula: TIFF0007661470000098.tif6150.

[0087] Where: TIFF0007661470000099.tif6150 is a smoothing function that avoids abrupt transitions at the cone boundaries. TIFF0007661470000100.tif6150 is the light source intensity. For both, i.e. directional light sources (multiple) and point light sources (multiple), the illumination value is determined at the position Incident light direction for TIFF0007661470000101.tif6150 TIFF0007661470000102.tif6150 is clearly defined. In the case of a directional light source, TIFF0007661470000103.tif6150 is the nature of the light source. For point light sources, TIFF0007661470000104.tif6150 is given by the vector from TIFF0007661470000105.tif6150 to the position of the light source. TIFF0007661470000106.tif6150 and TIFF0007661470000107.tif6150 are both unit vectors with magnitude 1. Thus, in one embodiment, 2D image calculator 2 is configured to calculate illumination values ​​according to this formula:

[0088] TIFF0007661470000108.tif7147

[0089] Since the contribution of each light source for every cone is calculated, a realistic lighting approximation with soft shadows is provided without introducing noise and with very little additional computation. In fact, since the dot product can be evaluated very fast on a GPU and the accumulated alpha has already been calculated, this approach is valid for almost any number of light sources while still being able to calculate soft shadows for each of them. This method can also be extended to incorporate other types of light sources, such as area light sources (i.e. light sources with rectangular or circular emitters) or light sources of arbitrary shape, for example by treating the points on the emitting surface similar to point light sources. An alternative approach is to calculate the incoming light of such light sources as ( TIFF0007661470000109.tif6150 and TIFF0007661470000110.tif6150 and The occlusion values ​​of the cone tracing cones may also be approximated as cones (called incident light cones) defined by an opening angle derived from the size of the light source and the distance to the light source, and the occlusion values ​​of the cone tracing cones may be modulated by these incident light cones based on the intersection of these two types of cones.

[0090] To further improve the 2D image calculator 2, several optimizations can be considered. The debatable but most important part of the algorithm is the definition of segment boundaries that determine the positions along the ray where lighting and AO (ambient occlusion) will be calculated. This amounts to analysing the progress of the accumulated opacity along the ray. As mentioned before, it is desirable to set the segment boundaries preferably before the bursts of accumulated opacity to best capture hard structures and boundaries. However, for homogeneous regions, it is desirable to distribute the segment boundaries as sparsely as possible. A linear division of the segment boundaries already gives good results, but in fact a segment positioning based on the derivative of the accumulated opacity, which corresponds to the current opacity, may give even better results. The 2D image calculator can therefore be configured to subdivide each primary ray into segments based on the derivative of the accumulated opacity. The 2D image calculator can be configured to use any function(s) that determines the segment boundary(s). Furthermore, the 2D image calculator can be configured to define the segment boundary(s) after the initial ray tracing, when the full evolution of the accumulated opacity is known, rather than during the initial ray tracing. In general, the better the segment boundary(s) are chosen, the more likely the algorithm will converge to the correct solution.

[0091] Additionally, the performance can be further improved by introducing a more adaptivity-based idea. Based on the observation that low opacity regions have a smaller contribution to the final pixel result than high opacity regions, and that high opacity often represents (characterizes) solid structures, the 2D image calculator can be configured to use a sampling density that depends on (is determined by) the current opacity of a point. Thus, the 2D image calculator can be configured to use sampling for the AO calculation in which low opacity regions are sampled with less accuracy than high opacity regions. The 2D image calculator can be configured to use different cone sample patterns for this, i.e., for example, 3 cones for low accuracy and 6 cones for higher accuracy. The approximation / calculation of directional and ambient light can be used with any cone layout and aperture.

[0092] A 2D image calculator can be formed by implementing the above techniques in a GLSL fragment shader of a generic ray-casting-based direct volume renderer in a rasterized graphics API such as OpenGL, thus using the processing power of a graphics processing unit (GPU). The ray-casting-based direct volume renderer can be, for example, the "Giga Voxel volume renderer" in the well-known "MeVis-Lab" software for medical imaging. However, the described performance improvements based on the introduced segment-based adaptive illumination sampling can also be used in any other CPU-based, GPU-rasterized or GPGPU (general purpose GPU)-based (e.g. using CUDA) volume renderer implementations.

[0093] As mentioned above, the two-dimensional image calculator preferably uses segment-based rendering to greatly reduce the required lighting calculations by identifying similar regions in the underlying data and assuming constant lighting values ​​for these regions. This results in a significant reduction in the required lighting calculations and the variable number of calculations per ray. The inventors have further found that in the case of rasterization-based implementations, performance can be further improved if the lighting calculations, including AO and incident light evaluation, are separated from the initial ray tracing. This is caused by the fact that fragment shader thread groups operate in lockstep, and faster threads have to wait for slower threads. The separation of the initial ray tracing and the lighting calculations is important, since the position along the example where the lighting is calculated can be significantly different even for adjacent rays. Therefore, in a preferred embodiment, the two-dimensional image calculator is configured to only define the segments and collect the relevant data for the later lighting calculations during the initial ray tracing. For every segment, in one embodiment, this data consists of the segment's start position within the volume, the current accumulated color, and the alpha-blended accumulated opacity. In other embodiments, just the positions where the lighting values ​​are to be calculated are stored, i.e., in other embodiments, just the lighting value determination positions are stored. These values ​​can be stored in a one-dimensional matrix with one element per segment. Because it is possible that a ray does not reach full opacity, and therefore not all segments are used, the 2D image calculator also preferably includes an additional counter that counts the number of valid segments for the later lighting calculation steps.

[0094] The 2D image calculator may also be configured to, after ray tracing, calculate a final color value for the current pixel by iterating over multiple effective segments and multiplying the stored color value of each segment by the calculated lighting approximation, i.e. the calculated lighting value and the calculated AO value, and adding them together according to equation (8) above.

[0095] As mentioned above, the 2D image calculator is configured to calculate AO similar to the voxel cone tracing algorithm as described in the above-mentioned article by Crassin et al., but adapted for DVR. As in the article by Crassin et al., the 2D image calculator thus uses volumes at multiple levels of detail and resolution, but does not structure it using an octree or other hierarchical data structure. Thus, the generation of the LOD representation is simple. An OpenGL-based implementation may employ, for example, the built-in LOD functionality of OpenGL during the generation of the 3D texture of the volume. However, the inventors have found that the subsequent sampling of LOD levels of the volume is faster if the downsampling is performed manually and each LOD level is stored in a separate sampler. Furthermore, they have found that a small number of LOD levels, such as 4 or 7, is already sufficient, depending on the size and complexity of the underlying data. It is also possible to automatically calculate the number of LOD levels based on the size of the 3D image. Furthermore, the 2D image calculator is preferably configured to generate the LOD levels in a manner similar to that described in the above-mentioned article by Crassin et al., by using a rasterization pipeline. More precisely, the 2D image calculator 2 is preferably configured to employ a volume renderer that renders a volume slice by slice in a frame buffer object (FBO) to obtain a separate occlusion volume. By using another volume renderer instance that bakes the occlusion volume, even very complex datasets with multiple secondary volumes, multiple clipping planes, multiple additional TFs, etc. can be covered.Since no lighting or shading is calculated for the occlusion volume, this additional rendering step is extremely fast and can be easily performed on the fly when the TF is being changed by the user. And this multi-resolution volume generation can be efficiently performed on the GPU of the 2D image calculator 2 via a GLSL shader that performs undersampling of the previous LOD level, thereby generating the next LOD level that has exactly half the size of the previous level in all dimensions. All LOD levels can be stored in a one-channel 16-bit texture, such as an R16 texture, or even an RGBA16 texture, which allows colorized shadows. And each LOD level can be bound to the main shader as a separate sampler.

[0096] Furthermore, the interpolation can be performed efficiently, for example by utilizing OpenGL hardware filtering when sampling the higher level volumes between two slices. Preferably, a 3x3 Gaussian filter kernel is additionally applied during downsampling, similar to Crassin et al., to ensure smooth results. These steps are repeated until the desired number of LOD levels in the multi-resolution volume is generated or further reduction of the volume is not possible. The 2D image calculator recalculates the LOD levels when the TF is changed resulting in a fully dynamic change of the TF. This also significantly simplifies the algorithm, since no dynamic update scheme or similar mechanisms are required.

[0097] The two-dimensional image calculator is preferably arranged to calculate two two-dimensional images at different viewing directions to enable stereoscopic viewing.

[0098] The 2D image calculator facilitates advanced illumination effects in high frame rate DVRs by combining a new sampling scheme with voxel cone tracing. The sampling scheme reduces the amount of illumination calculations along the primary viewing gray, while voxel cone tracing reduces the complexity of the illumination calculations themselves. This combination even allows the definition of TFs or clipping planes in real time without any loss of image quality.

[0099] In the above embodiments, the 2D image calculator combines adaptive illumination sampling with image-based lighting, AO, and direct illumination with shadow casting techniques, although other illumination techniques such as sub-surface scattering could also be used.

[0100] The input device 1 of the DVR device 1 is preferably configured to allow the user to modify (change) a) the TF and / or b) the position and / or orientation (direction) of one or more clipping planes and / or c) the position and / or orientation (direction) of the virtual camera. Accordingly, the 2D image calculator 2 is preferably configured to allow fully interactive real-time modification of the TF and / or clipping planes and / or virtual camera and to calculate the AO in the DVR. At the core of the corresponding algorithm, the 2D image calculator 2 employs an adaptive sampling scheme that splits the rays into multiple segments, thus facilitating a required reduction in the amount of illumination calculations along the viewing rays, i.e. along the primary rays. The illumination calculations themselves are efficiently calculated using an adaptation of voxel cone tracing. This requires minimal pre-computation and allows a noiseless on-the-fly calculation of AO in combination with image-based illumination and soft shadows for almost any number of directional or point light sources.

[0101] A 2D image calculator can be easily created by incorporating the described algorithm into existing ray-casting based volume rendering pipelines. For the primary target use case, which is AR, a 2D image calculator represents a good compromise by combining high frame rates with noise-free, realistically illuminated images. It especially lends itself to the combination with image-based lighting, which can facilitate a deeper integration of the rendered image with the real environment.

[0102] The concept of segment-based ray casting embodied in a 2D image computer can be used for all scenarios where the iterative calculations along a ray need to be reduced. Therefore, segment-based rendering can be used in many domains and algorithms, such as global illumination algorithms. Furthermore, the above lighting terms can be extended to include more sophisticated effects or shading models, covering for example colorized shadows, subsurface scattering or even full global illumination.

[0103] The AO calculation performed by the 2D image calculator, called "Adaptive Volumetric Illumination Sampling" and which may be abbreviated as AVIS, is preferably a fully GPU-based real-time approach for calculating AO in ray-casting-based DVR. It allows AO to be calculated in real-time for DVR and supports interactive TF updates and clipping of visualization data at high frame rates and high resolutions. To this end, the proposed approach combines cone tracing AO with an adaptive sampling scheme that facilitates a reduction in the amount of computation and produces noise-free AO with only minimal recomputations. The described approach can be integrated into existing production volume rendering pipelines and is preferably used in an AR setting in the context of medical volume rendering. Preferably, it is designed for medical application scenarios and performs well even for complex datasets.

[0104] The idea of ​​using obscurance - or occlusion - as a means to approximate indirect lighting was first proposed in the above mentioned paper by S. Zhukov et al. Their method, now known as ambient occlusion, uses the amount of occlusion at a surface point to modulate the ambient light coming from a distributed pseudo-light source. Typically, AO is calculated by ray casting: multiple rays are traced to sample a hemisphere. For each ray the amount of occlusion is calculated, but the occlusion value at a surface point is the average of all the occlusion values.

[0105] Although AO is only an approximation of global illumination, it is still computationally expensive. One strategy to allow real-time rendering of AO is to precompute the AO values ​​or bent normals and store them in a texture or a 3D grid. However, this only allows for rigid transformations.

[0106] Fully dynamic AO is realized in current graphics hardware as a 2.5D image-space process, the so-called Screen Space Ambient Occlusion (SSAO), which is based on operations on deferred shading buffers for depth and surface normals, but is only prevalent in surface-based rendering.

[0107] Work on real illumination models initially focused on surface rendering, but eventually gave rise to similar work in the field of DVR. In addition to shadow casting approaches such as those described in U. Behrens et al., "Adding shadows to a texture-based volume renderer", IEEE Symposium on Volume Visualization (2002), M. Hadwiger et al., "GPU-Accelerated deep shadow maps for direct volume rendering", Graphics Hardware, pp. 49-52 (2006) and T. Ropinski et al., "Efficient shadows for GPU-based volume raycasting", International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (2008), the contents of which are incorporated herein by reference, the application of ambient occlusion (AO) methods to volume rendering has gained much importance. One of the first approaches to AO for DVR was "Vicinity Shading" by A. Stewart, described in "Vicinity shading for enhanced perception of volumetric data", Proceedings of the IEEE visualization conference, pp. 355-362 (2003), the contents of which are incorporated herein by reference. In this paper, local illumination was calculated by examining the occlusion degree of neighboring voxels, and preprocessing was required for all modifications in terms of the scene and an additional buffer to store the results.

[0108] In the paper "Local ambient occlusion in direct volume rendering" by F. Hernell et al., IEEE Transactions on Visualization and Computer Graphics, 16(4), pp. 548-559 (2010), the contents of which are incorporated herein by reference, a local AO is described that first downsamples and then calculates occlusion from a local neighborhood by packing the volume into blocks with multiple resolutions in a precomputation step. The AO is then stored in an additive ambient light texture that is progressively updated by a fixed (regular) geometric pattern that determines one AO ​​sample direction and ray directions per frame. The sampling along the primary and secondary rays follows an adaptive scheme where the step length is based on the resolution of the underlying blocks. This allows efficient LOD-based traversion and sampling of the volume and allows skipping empty spaces. Although this approach produces plausible results, allows clipping of the data, and does not rely on gradients, it suffers from discontinuity artifacts and is excessively slow for real-time applications, especially since the underlying multi-resolution volume must be recomputed for every change in TF.

[0109] Similarly, in "Interactive volumetric lighting simulating scattering and shadowing", IEEE Pacific visualization symposium, pp. 169-176 (2010), the contents of which are incorporated herein by reference, T. Ropinski et al. generate an illumination volume by front-to-back accumulation of the chromaticity of the volume and scattering of the light in back-to-front order in a second pass, using cones for both light transportation and scattering. With the appropriate shading model, this results in scattering and diffuse, specular and translucent materials, while the volume can be recalculated on the fly if the transfer function or light displacement changes. However, the restriction of having only one dynamic point or directional light is quite restrictive.

[0110] The concept of reducing the required samples by splitting rays into pieces is studied in F. Hernell et al., "Interactive global light propagation in direct volume rendering using local piecewise integration", SPBG'08 Proceedings of the Fifth Eurographics / IEEE VGTC Conference on Point-Based Graphics, pp. 105-112 (2008), the contents of which are incorporated herein by reference, where piecewise integration is presented. Unlike the approach used by the 2D image generation unit 2, in this paper the segments have a constant size and are not opacity driven. Furthermore, in this paper piecewise integration is only employed for light rays (or secondary rays) during the generation of multi-resolution LOD-based shadow volumes, but not for primary rays. Using the resulting shadow volume, the paper then calculates local first-order scattering effects in a manner similar to the local ambient occlusion described in F. Hernell et al., "Efficient ambient and emissive tissue illumination using local occlusion in multiresolution volume rendering", Proceedings of Eurographics / IEEE Volume Graphics 2007, pp. 1-8 (2007), the contents of which are incorporated herein by reference. In the last step, a single ray casting pass is used to render the final image. This approach allows interactive variation of the TF, but does not support clipping planes.Regarding lighting, this method does not support environment maps and only supports up to four point lights.

[0111] In the paper "Historygrams: Enabling interactive global illumination in direct volume rendering using photon mapping", IEEE transactions on visualization and computer graphics, volume 18, pp. 2364-2371 (2012), the contents of which are incorporated herein by reference, the idea of ​​splitting rays into segments / parts is proposed to generate an adaptive update scheme for the photon mapping approach for volume rendering. For each primary ray and each ray from a light source, the ray will be split into segments where photon-medium interactions (scattering, absorption, etc.) occur. Every ray stores the history of the parameters used when the segment was last updated, so that when the TF is changed by the user, only the affected segments are recalculated. Similarly, every photon stores the complete history of the light path. In this paper, they are called "historygrams" and one is stored per object. Furthermore, the primary ray segments are formed with a uniform size and are not opacity dependent. Furthermore, this approach is not possible in real-time, and constantly changing camera and light positions, as occurs in AR scenarios, are problematic, especially since photon updates are extremely expensive.

[0112] T. Kroes et al., "Exposure render: An interactive photo-realistic volume rendering framework", PLoS ONE, 7(7), https: / / doi.org / 10.1371 / journal.pone.0038586 (2012), the contents of which are incorporated herein by reference, show the feasibility of fully dynamic global illumination in DVR based on path tracing. The method was adapted by Siemens Healthineers and named Cinematic Rendering. Typically, path tracing, which is an interactive algorithm, converges to a low-noise result within a few seconds. However, convergence to a smooth result can only be achieved by capturing a larger number of samples from the same view. Although the process can be fixed, for example by using a denoising filter, path tracing is still too slow for applications with high requirements for accuracy and speed, as in medical situations and AR scenarios. However, path tracing still produces the most accurate results, since it is the most physically accurate method.

[0113] The document "Adaptive Temporal Sampling for Volumetric Path Tracing of Medical Data". Computer Graphics Forum, 38(4), pp. 67-76 (2019), the contents of which are incorporated herein by reference, describes a method for making path tracing available for virtual and AR scenarios. In the described approach, temporal reprojection, as known from surface-based rendering, was applied to path-traced volume rendering by storing additional information in a screen-space history buffer and by employing an adaptive sampling approach. This allows already calculated values ​​to be reused and samples to be spent only when necessary. Although this improves the performance of path-traced volume rendering, the resulting images still suffer from noise and blur, which is undesirable, especially in medical volume rendering. In addition, changes in TFs and clipping planes are only indirect and only supported to a certain extent.

[0114] In the above mentioned paper by Crassin et al. a "voxel cone tracing" technique is proposed, a technique for traditional surface rendering that allows smooth and noise-free global illumination in real time with good quality. The fully GPU-based approach works with almost no precomputation except for the need to voxelize the scene and store the information in a hierarchical octree structure called the "sparse voxel octree" that contains the scene at different LODs. In this case, the basic algorithm includes three basic steps: first, the scene is rendered from light positions to assign incident light information to the octree; second, light is propagated through the volume to yield radiance for each octree node; third, the scene is rendered again from the camera's viewpoint and for each visible surface point, but the final gathering using cone tracing is performed as described in J. Amanatides, "Ray Tracing With Cones", Computer Graphics (ACM), 18(3), pp. 129-135 (1984), the contents of which are also incorporated herein by reference. A separate cone for the specular part of the bidirectional reflectance distribution function (BRDF) takes into account specular materials. An example of the use of cone tracing to calculate AO is also given in Crassin et al.

[0115] In the following, an embodiment of a DVR method for volume rendering of a three-dimensional image dataset will be exemplarily described with reference to the flow chart shown in FIG.

[0116] In step 101, a three-dimensional image dataset is provided, preferably a medical image dataset such as a CT or MR image dataset. In step 102, a color and an opacity are assigned to each voxel of the three-dimensional image, and in step 103, for each primary ray of a plurality of primary rays passing through the three-dimensional image data and impinging on each pixel of the two-dimensional image in the two-dimensional image plane onto which the three-dimensional image dataset is to be projected, a color and an opacity are calculated along the primary ray based on the color and the opacity assigned to the voxel passed by the primary ray. In step 104, each primary ray is subdivided into a plurality of segments based on the opacity along the respective primary ray, and in step 105, for each segment, a segment illumination value is calculated according to a predetermined illumination value calculation rule. In step 106, the color and the segment illumination value are projected along the primary rays to the pixels based on the predetermined projection rule to generate a two-dimensional image. It is also possible that steps 103 and 104 are integrated, i.e., opacity(s) and color(s) along the primary rays are calculated and segments are determined already during initial ray casting. In particular, in a preferred embodiment, opacity(s) and color(s) along the primary rays are calculated and segments are determined during initial ray casting, and furthermore, positions within the respective segments at which a segment lighting value should be calculated are determined. Thus, lighting value determination positions can be determined already during initial ray casting.

[0117] In step 107 it is checked whether the assignment of colour(s) and opacity(s) to the voxels, i.e. the corresponding transfer functions, have been amended by the user using the input device 3. If so, the method continues from step 103. Furthermore, in step 107 it is checked whether the user has modified the clipping planes or the position and / or orientation of the virtual camera using the user device 3. In this case the method can also continue from step 103. Furthermore, in step 107 it is checked whether the user has indicated via the input device 3 that the DVR method should be stopped. If so, the method ends in step 108.

[0118] Thus, the user can modify the TF, clipping planes and / or virtual camera position, and the generated 2D image is continuously updated so that the user can see how the modifications have affected the visualization in real time.

[0119] In one embodiment, for a (one) determined segment, the lighting value calculation for one sample point, i.e., lighting value determination position, is given as the sum of (1) an ambient light term and any number of (2) directional light terms, as described below.

[0120] (1) The ambient light term is calculated as the light coming from all directions over a hemisphere of sample points. For this purpose, an environment map is used to identify the light coming from all directions in the form of a 360° by 180° panorama. This light is modulated by shadowing as calculated by the ambient occlusion according to the calculations described in Zhukov et al., supra. More specifically, voxel cone tracing as described in Crassin et al., supra, is used to calculate the ambient occlusion. This means that the hemisphere is sampled by a small number of cones (typically 5 or 6) evenly distributed so that the hemisphere is more or less covered by the resulting circles on its surface. A minimum overlap is allowed. Preferably, 6 cones with an opening angle of 67° are used. These cones are defined in a fixed reference frame oriented by the gradient of the 3D image dataset at the sample points. Each cone is traced through the 3D image dataset to compute the occlusion resulting from the parts of the volume covered by this cone. The following description is for one of these cones: The tracing of a cone is performed by sampling an additional level of detail (LOD) occlusion volume precomputed based on the original 3D image dataset and the TF. The idea behind this is that starting from the apex of the cone and following the cone along its main axis, the cross section of the cone gets larger. Instead of sampling the original 3D image dataset at many points inside the entire volume of the cone to compute a representative occlusion value, a few (2-3) spheres or spherical areas on the axis of the cone are sampled, each sphere having a larger radius than the sphere before it.However, the radius of the sampling spheres is preferably always exactly doubled so that the voxels of each LOD always double their size as well. The sampling points along the ray are determined automatically since they should also match the current radius of the cone. The sampling of these spheres is performed by fetching values ​​from the additive LOD occlusion volumes. The voxel size of the levels grows: each level doubles the voxel size with respect to the previous level. Fetching a value from a lower resolution level means getting the integration of the samples in a larger area, i.e. getting the occlusion of a larger area. This is used to calculate occlusion values ​​for each sphere with exactly one sample. The combination of the few sphere occlusion values ​​results in the occlusion value for the whole cone. The number of spheres (or steps along rays) depends on the size of the neighborhood to be considered for the occlusion calculation and the size of the 3D image dataset. In any case, the spheres can only be considered until the cone axis leaves the 3D image data or penetrates a clipping plane. The occlusion value is used to modulate the light from the environment map that will reach the sample point along the directions through this cone. Fetching this light from the environment map would again require taking multiple samples from the map. Instead, pre-convolved lighting is used as described in Klehm et al., supra. Similar to the LOD occlusion volume, an additional representation of the environment map is used that pre-integrates the light that will fall in through a cone with a given opening angle, i.e. a given aperture. Thus, only one sample needs to be taken from the panorama at a position corresponding to the direction of the cone.The sample values ​​are modulated by the occlusion value of the cone to yield the amount of incoming ambient light for the solid angle covered by the cone. By repeating the process for all cones and calculating a weighted sum of the results (where the weighting factor depends on the inclination of each cone to its respective gradient), the ambient light for the whole hemisphere, i.e. the modulated first incoming light, is calculated. In particular, the weighting factor can be given by the cosine of the inclination of each cone to its respective gradient.

[0121] (2) The directional light term for a given light source is calculated based on a common local illumination model, e.g. the Phong model, and the respective shadow value. The shadow value is approximated by reusing the occlusion values ​​calculated for the six cones (see (1)). Thus, the shadow value for a light source is a weighted sum of the cone occlusion values, where the weight depends on the angle between the light direction and the cone axis as well as the cone opening angle. The shadow value is used to modulate the result of the local illumination model. This process is repeated for all light sources.

[0122] The DVR apparatus and method preferably uses ambient occlusion instead of global illumination to increase the update speed and therefore the achievable frame rate.

[0123] In the above embodiments the image dataset is a medical image dataset, however in other embodiments the image dataset may be other types of datasets such as image datasets used in the fields of materials science, flow visualization, etc.

[0124] While the above embodiment(s) use transfer functions to assign opacity and color to voxels, in other embodiments, opacity and color may be provided without the use of transfer functions. For example, a voxel or other type of image element may already contain opacity and color, but the opacity and color for a primary ray may be determined, for example, by interpolation based on the opacity and color of the image element.

[0125] In the above embodiments, each primary ray is subdivided into segments based on the opacity along each primary ray, in particular the cumulative opacity, but alternatively or additionally, this subdivision can also be performed based on the color along each primary ray. For example, each color at each location along each primary ray, typically including three values ​​for red, green and blue components, can be mapped to one value. This can be achieved, for example, by calculating the average of the three values ​​or by calculating a weighted sum that allows for the consideration of human color perception, such as Y=0.2125×R+0.7145×G+0.0721×B. Here, Y is a value to which the three values ​​should be mapped, and R, G and B are three values ​​for the red, green and blue components, as described, for example, in the article "Rehabilitation of gamma" by Charles A. Poynton, Photonics West'98 Electronic Imaging, International Society for Optics and Photonics (1998), the contents of which are incorporated herein by reference. Then, the detection of the segment boundaries can be performed based on the values ​​obtained by the mapping and thus representing the respective colors along each primary ray. A mathematical function is applied to the obtained values ​​representing the colors, and the results can be compared with threshold values ​​to determine the segment boundaries, as described above, especially with respect to the cumulative opacity. Also, if the original opacities are to be used to determine the segment boundaries, then a mathematical function is applied to the resulting numerical values ​​representing the original opacities, and the result can be compared to threshold values ​​to determine the segment boundaries, as described above, particularly with respect to cumulative opacity.In one embodiment, the mathematical function is an integral function, i.e., integrals, where values ​​representing the colors or original opacities are summed along each primary ray and each segment boundary is defined by the sum exceeding a respective threshold. The thresholds that provide a very fast overall DVR processing can be determined by calibration, during which the thresholds are determined such that the speed of the DVR processing is increased, and in particular maximized.

[0126] The mathematical function can also be other functions such as the derivative of opacity, color or cumulative opacity. The derivative is particularly useful for detecting strong changes, which typically correspond to tissue boundaries, and segment boundaries can be located at the locations of these strong changes. Thus, the segment boundaries can be determined by thresholding the derivative, which defines how much the respective parameter (color, opacity or cumulative opacity) has changed along each primary ray. The derivative can be determined based on the difference between subsequent values ​​along the primary ray.

[0127] Although one embodiment described above includes segment illumination values ​​based on a particular version of ambient occlusion (voxel cone tracing) and an approximation of directional (direct) light (parallel and point lights) also based on voxel cone tracing, the segment illumination values ​​could be determined in other ways. For example, any method for the calculation of light and shadowing for a given position could be used instead. Any alternative method would benefit from a reduction in the number of calculations. As an example, the well-established Deep Shadow Maps method described in the above-mentioned paper by M. Hadwiger could be used. Instead of sampling the shadow map for each point along the ray, it would only be necessary to sample it once per segment, which would speed up rendering.

[0128] Although it has been described that only one lighting value needs to be calculated per segment, it is also possible that more than one lighting value is calculated per segment, in which case lighting values ​​are nevertheless determined not for all sampling positions along each primary ray, i.e. not for all positions to which colors and opacities are assigned. For example, lighting values, i.e. segment lighting values, are determined for less than 50 percent of the sampling positions in each segment, more preferably for less than 30 percent, even more preferably for less than 20 percent. Alternatively, a fixed number of lighting values ​​per segment could be used, which would allow to limit the number of lighting value calculations even more efficiently. Thus, instead of defining one lighting value determination position, it would be possible to use multiple lighting value determination positions in a segment in order to increase the accuracy of the method, but nevertheless the reduction in the amount of calculations required for the calculation of the two-dimensional image would be significant. For example, multiple lighting value determination positions could be defined in each quarter of each segment. For the lighting value determination locations, lighting values ​​can be calculated according to a predetermined lighting value calculation rule. These could be combined (e.g. by calculating their average) into one segment lighting value, or if one segment lighting value is determined for each quadrant, it could be used exclusively for that quadrant. Also, the segment lighting values ​​could be interpolated between the lighting value determination locations. In that way, one lighting value per sample for the primary ray could be approximated.

[0129] In one embodiment above, a projection rule is described that includes multiplying the segment illumination value with the segment color, but other projection rules could also be used. For example, the segment illumination value could be multiplied with one, some or all colors along each primary ray and in each segment, and the resulting products could be summed up. Thus, in this alternative embodiment, the segment illumination value is, for example, reused for all samples in each segment. According to a further projection rule, the segment illumination values ​​are interpolated between the segment illumination positions such that for each sample position, and therefore for each color, a respective interpolated illumination value is determined, and each color is multiplied with the respective interpolated illumination value at each position along the primary ray, i.e. at each sampling position, and the resulting products are summed up. In general, the phrase "projecting colors and segment lighting values ​​to pixels along primary rays" refers to the process of combining colors and segment lighting values ​​along each primary ray such that a respective pixel value is determined for each primary ray, which combining may include pre-processing of the colors and segment lighting values, such as the interpolation described above, and post-processing of the pixel values.

[0130] It is possible that one primary ray is traced for a pixel, or that multiple primary rays are traced for a pixel in order to reduce aliasing artifacts: in the latter case, the described method is performed multiple times for a limited number of positions within the area of ​​the corresponding pixel, instead of once for only one position, preferably at the center of the corresponding pixel.

[0131] Other variations of the above embodiment(s) can be understood and effected by those skilled in the art in practicing the invention of this application, from a study of the drawings, the detailed description, and the appended claims.

[0132] In the claims, the term "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0133] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0134] The procedures performed by the 2D image calculator, such as determining opacity(s) and color(s) for the primary rays, subdividing the primary rays into segments, calculating segment illumination values, projecting the colors and segment illumination values, etc., can also be performed by any number of other units or devices. These procedures and / or the control of the DVR device according to the DVR method can be realized as program code means of a computer program and / or as dedicated hardware.

[0135] The computer program may be stored / distributed on a suitable medium, such as an optical recording medium or a solid-state medium, provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0136] Any reference signs to the drawings in the claims should not be construed as limiting the scope of protection.

[0137] The invention relates to a direct volume rendering device for volume rendering a three-dimensional image dataset. For each primary ray of a plurality of primary rays, which proceed through the three-dimensional image dataset and impinge on a respective pixel of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image dataset is to be projected, colours and opacity values ​​are provided along the primary ray. Each primary ray is subdivided into a plurality of segments based on the opacity values ​​and / or colours along the respective primary ray, and for each segment a segment illumination value is calculated according to a pre-determined illumination value calculation rule. The colours and segment illumination values ​​are finally projected along the primary rays based on the pre-determined projection rule onto the pixels.

[0138] Possible embodiments of the present invention are now described. [Appendix 1] A direct volume rendering device for volume rendering three-dimensional image datasets. The direct volume rendering apparatus comprises: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image data set and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image data set is projected, color(s) and opacity(s) along the primary ray; Subdividing each primary ray into a number of segments based on the opacity(s) and / or color(s) along the respective primary ray; Calculating for each segment a segment lighting value according to a predetermined lighting value calculation rule; Projecting the color(s) and segment lighting value(s) along the primary ray to a pixel based on a predetermined projection rule. a two-dimensional image calculator configured to calculate a two-dimensional image to be shown to a user by Includes. [Appendix 2] In the above-mentioned direct volume rendering apparatus, particularly as described in appendix 1, the three-dimensional image dataset includes a plurality of image elements; The two-dimensional image computer is assigning a color and an opacity to each image element of the 3D image data set; and For each primary ray, calculating color(s) and opacity(s) along the primary ray based on the color(s) and opacity(s) assigned to the image element(s) traversed by the primary ray. for each primary ray, is configured to provide color(s) and opacity(s) along the respective primary ray. [Appendix 3] In the direct volume rendering apparatus described above, particularly in appendix 1 or 2, The two-dimensional image calculator subdivides each primary ray into a plurality of segments by defining segment boundaries; A segment boundary is defined by a location along a primary ray where a function of opacity and / or color exceeds a predefined threshold. [Appendix 4] In the direct volume rendering apparatus described above, particularly in appendix 3, The opacity function is a function selected from the group consisting of an identity function, a linear function, an integral type function, and a differential type function, which is applied directly to the opacity and / or color and / or applied to the accumulated opacity. [Appendix 5] In the direct volume rendering apparatus described above, particularly in appendix 3 or 4, The two-dimensional image calculator is configured to provide a number of threshold values; A segment boundary is defined by a position along the primary ray where a function of opacity and / or color exceeds a predetermined threshold of said number of thresholds provided. [Appendix 6] In the direct volume rendering apparatus described above, particularly in appendix 5, The difference between adjacent provided thresholds is constant. [Appendix 7] In the direct volume rendering apparatus described above, particularly as described in appendix 5 or 6, The two-dimensional image calculator is configured to provide the number of threshold values ​​according to a predetermined linear or exponential threshold function. [Appendix 8] In the direct volume rendering apparatus described above, particularly in any one of appendices 1 to 7, The two-dimensional image calculator calculates, for each segment, Defining lighting value determination positions within each segment; For each lighting value determination position, calculating each segment lighting value according to a predetermined lighting value calculation rule. The segment illumination value is calculated by: [Appendix 9] In the above-mentioned direct volume rendering apparatus, particularly as described in appendix 8, The two-dimensional image calculator is configured to define illumination value determination positions within each segment such that the illumination value determination positions are within a quadrant of each segment located at the end of each segment where a primary ray is traced from a pixel position into the volume. [Appendix 10] In the direct volume rendering apparatus described above, particularly as described in appendix 8 or 9, The predetermined lighting value calculation rules include at least one of a) calculating an ambient lighting value and b) calculating at least one directional lighting value. [Appendix 11] In the direct volume rendering apparatus according to the above appendix 10, which cites appendix 2 in particular, The predetermined lighting value calculation rules include: a) providing an environment map defining a direction and intensity of a first incident light, and calculating an ambient lighting value by modifying the intensity of the first incident light based on an opacity assigned to an image element; and b) providing directional lighting information for the lighting value determination position representing the light intensity of a second incident light from a virtual light source, and calculating at least one directional lighting value by modifying the intensity of the second incident light based on an opacity assigned to an image element. [Appendix 12] In the above-mentioned direct volume rendering apparatus, particularly as described in appendix 11, The ambient lighting value is defining several cones within a hemisphere about the illumination value determination location, the hemisphere being defined by a gradient of color and / or opacity assigned to the image element at the illumination value determination location, the hemisphere being oriented in the direction of the gradient, an end face of the hemisphere intersecting the illumination value determination location and perpendicular to the gradient, and the apex of the cone being located at the illumination value determination location, defining, for each cone, a number of spheres centred on the axis of the respective cone, the radii of the spheres being greater the greater the distance to said apex; determining an occlusion value for each sphere and combining the occlusion values ​​determined for all spheres of the respective cone to determine a (single) occlusion value for the whole cone; determining, for each cone, a modified light intensity based on the occlusion value determined for the respective cone and a light intensity of light incident through the respective cone as defined by the environment map; combining the modified light intensities determined for the cones of each illumination value determination location to determine an ambient illumination value for each illumination value determination location. It is calculated by: [Appendix 13] In the direct volume rendering apparatus described above, particularly in appendices 1 to 12, The 2D image calculator is further configured to calculate, for each segment, a segment color based on the color and opacity along the respective segment, and to project the colors and segment illumination values ​​along the primary ray to the pixel by projecting the segment color and the segment illumination value. [Appendix 14] A direct volume rendering method for volume rendering of 3D image datasets. The direct volume rendering method includes the steps of: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image data set and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image data set is projected, color(s) and opacity(s) along the primary ray; subdividing each primary ray into a plurality of segments based on the opacity(s) and / or the color(s) along the respective primary ray; Calculating for each segment a segment lighting value according to a predetermined lighting value calculation rule; projecting the colors and segment illumination values ​​along the primary rays onto the pixels based on a predetermined projection rule. Includes. [Appendix 15] A computer program for volume rendering of three-dimensional image datasets. The computer program includes program code means for causing the direct volume rendering device described above, particularly any of Appendices 1 to 13, to execute the steps of the direct volume rendering method described in Appendices 14, when the computer program is executed on a computer controlling the direct volume rendering device.

Claims

1. 1. A direct volume rendering apparatus for volume rendering a three-dimensional image dataset, comprising: The direct volume rendering device (1) comprises: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image dataset and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image dataset is projected, color(s) and opacity(s) along the primary ray; Subdividing each primary ray into multiple segments based on the opacity(s) and / or color(s) along the respective primary ray; a) defining lighting value determination positions within each segment; and b) calculating for each segment a segment lighting value according to a predetermined lighting value calculation rule by calculating, for each lighting value determination position, a respective segment lighting value according to the predetermined lighting value calculation rule; Projecting the color(s) and segment illumination value(s) along the primary ray to a pixel based on a predetermined projection rule. a two-dimensional image calculator (2) configured to calculate a two-dimensional image to be shown to a user by Contains A direct volume rendering device comprising:

2. 2. The direct volume rendering apparatus according to claim 1, the three-dimensional image data set includes a plurality of image elements; The two-dimensional image computer (2) - assigning a color and an opacity to each image element of the 3D image data set; and For each primary ray, calculating color(s) and opacity(s) along the primary ray based on the color(s) and opacity(s) assigned to the image element(s) traversed by the primary ray. and for each primary ray, providing color(s) and opacity(s) along the respective primary ray by A direct volume rendering device comprising:

3. 3. The direct volume rendering apparatus according to claim 1, The two-dimensional image calculator (2) comprises: subdividing each primary ray into a plurality of segments by defining segment boundaries (a plurality of segments); Segment boundaries are defined by positions along the primary ray where a function of opacity and / or color exceeds a predefined threshold. A direct volume rendering device comprising:

4. 4. The direct volume rendering apparatus according to claim 3, The opacity function is a function selected from the group consisting of an identity function, a linear function, an integral type function, and a differential type function, which is applied directly to the opacity and / or color and / or applied to the cumulative opacity. A direct volume rendering device comprising:

5. 5. The direct volume rendering apparatus according to claim 3, said two-dimensional image calculator (2) being configured to provide several threshold values; A segment boundary is defined by a position along the primary ray where a function of opacity and / or color exceeds a predetermined threshold of said number of thresholds. A direct volume rendering device comprising:

6. 6. The direct volume rendering apparatus according to claim 5, The difference between adjacent provided thresholds is constant A direct volume rendering device comprising:

7. 7. The direct volume rendering apparatus according to claim 5, said two-dimensional image calculator (2) being adapted to provide said threshold values ​​according to a predefined linear or exponential threshold function; A direct volume rendering device comprising:

8. The direct volume rendering apparatus according to any one of claims 1 to 7, said two-dimensional image calculator (2) being configured to define illumination value determination positions within each segment such that the illumination value determination positions lie within a quadrant of each segment located at an end of each segment along which a primary ray is traced from a pixel location into the volume; A direct volume rendering device comprising:

9. A direct volume rendering apparatus according to any one of claims 1 to 8, The predetermined lighting value calculation rules include at least one of: a) calculating an ambient lighting value; and b) calculating at least one directional lighting value. A direct volume rendering device comprising:

10. In the direct volume rendering apparatus according to claim 9, The predetermined lighting value calculation rules include: a) providing an environment map defining a direction and intensity of a first incident light, and calculating an ambient lighting value by modifying the intensity of the first incident light based on an opacity assigned to an image element; and b) providing directional lighting information for the lighting value determination location representing a light intensity of a second incident light from a virtual light source, and calculating at least one directional lighting value by modifying the intensity of the second incident light based on an opacity assigned to an image element. A direct volume rendering device comprising:

11. 11. The direct volume rendering apparatus according to claim 10, The ambient lighting value is - defining several cones within a hemisphere about the illumination value determination location, the hemisphere being defined by a gradient of color and / or opacity assigned to the image element at the illumination value determination location, the hemisphere being oriented in the direction of the gradient, the end faces of the hemisphere intersecting the illumination value determination location and perpendicular to the gradient, the apex of the cone being located at the illumination value determination location, - defining for each cone a number of spheres centred on the axis of the respective cone, the radii of the spheres being greater the greater the distance to said apex; determining an occlusion value for each sphere and combining the occlusion values ​​determined for all spheres of the respective cone to determine a (single) occlusion value for the entire cone; determining, for each cone, a modified light intensity based on the occlusion value determined for the respective cone and the light intensity of the light incident through the respective cone as defined by the environment map; Combining the modified light intensities determined for the cones of each lighting value determination location to determine an ambient lighting value for the respective lighting value determination location. To be calculated by A direct volume rendering device comprising:

12. The direct volume rendering apparatus according to any one of claims 1 to 11, the two-dimensional image calculator (2) is further configured to calculate, for each segment, a segment color based on the color and opacity along the respective segment, and to project the colors and segment illumination values ​​along the primary rays to the pixels by projecting the segment color and the segment illumination value. A direct volume rendering device comprising:

13. 1. A direct volume rendering method for volume rendering a three-dimensional image dataset, comprising: The direct volume rendering method comprises the steps of: providing, for each primary ray of a plurality of primary rays passing through the three-dimensional image dataset and impinging on respective pixels of a two-dimensional image in a two-dimensional image plane onto which the three-dimensional image dataset is projected, color(s) and opacity(s) along the primary ray; - subdividing each primary ray into a number of segments based on the opacity(s) and / or the color(s) along the respective primary ray; a) defining lighting value determination positions within each segment; and b) calculating for each segment a segment lighting value according to a predetermined lighting value calculation rule by calculating, for each lighting value determination position, a respective segment lighting value according to the predetermined lighting value calculation rule; Projecting the colors and segment illumination values ​​along the primary rays onto the pixels based on a predetermined projection rule. Contains A direct volume rendering method comprising:

14. 1. A computer program for volume rendering of a three-dimensional image dataset, comprising: The computer program includes a program code for causing the direct volume rendering apparatus according to any one of claims 1 to 12 to execute the steps of the direct volume rendering method according to claim 13, when the computer program is executed on a computer that controls the direct volume rendering apparatus. A computer program comprising:

Images