Method and visualization system for visualizing a three-dimensional object

Abstract

The invention relates to a computer-implemented method for visualizing a three-dimensional object represented by volume data (VD) to a user in the form of a two-dimensional visualization image (VB) composed of a plurality of visualization pixels (VP). The method has the following steps: providing an image synthesis algorithm which is composed for visualizing the three-dimensional object by mapping the volume data onto the visualization pixels; detecting a current viewing direction of the user; determining a focal point region (FB), wherein the focal point region is defined as a region of the visualization image on which the user is focused based on the current viewing direction; adjusting the image synthesis algorithm such that a computational effort required for mapping the visualization pixels outside the focal point region is less than a computational effort required for mapping the visualization pixels within the focal point region; and mapping the volume data onto the visualization pixels by means of the adjusted image synthesis algorithm.

Description

Technical Field

[0001] This invention pertains to the field of volumetric rendering, specifically the display or visualization of three-dimensional volumes or objects. Background Technology

[0002] Modeling, reconstructing, or visualizing three-dimensional objects has wide applications in fields such as medicine (e.g., CT, PET), physics (e.g., electronic structure of macromolecules), or geophysics (properties and locations of strata). Typically, the object to be examined is irradiated (e.g., by means of electromagnetic waves or sound waves) to examine its properties. Scattered radiation is detected, and bodily characteristics are determined from the detected values. Results are typically generated in physical variables (e.g., density, tissue type, elasticity, velocity), and the values ​​of these physical variables are determined for the body. Here, a virtual mesh is typically used, and the values ​​of the variables are determined at the mesh points of this virtual mesh. These mesh points are often called voxels. The term "voxel" is a portmanteau of "volume" and "pixel." A voxel corresponds to the spatial coordinates of a mesh point, which are associated with the value of the variable at that location. This typically involves physical variables, which can be represented as scalar or vector fields, i.e., the corresponding field values ​​are associated with the spatial coordinates. By interpolating voxels, the values ​​of variables or fields at any point on the object (i.e., any location point on the object being examined) can be obtained.

[0003] To visualize volumetric data, a three-dimensional representation of the object or body being examined is generated from voxels on a two-dimensional display surface (such as a screen or the disc or lens of so-called "augmented reality glasses"). In other words, voxels (defined in three dimensions) are mapped onto pixels (defined in two dimensions) of the two-dimensional visualization image. Pixels of the visualization image are also referred to as visualization pixels below. This mapping is commonly referred to as volumetric rendering (volume reconstruction). Related to the execution of volume reconstruction is how the information contained in the voxels is reproduced using pixels.

[0004] One of the most widely used methods for volumetric representation is so-called ray casting (see Levoy, “Display of Surfaces from Volume Data,” IEEE Computer Graphics and Applications, 8th edition, Vol. 3, May 1988, pp. 29–37). In ray casting, simulated rays emanating from the eye of an imaginary observer are sent through the body or object being examined. Along the ray, RGBA values ​​are determined for sampling points from voxels, and pixels for a two-dimensional image are synthesized by means of alpha composites or alpha blends. Here, in the representation of RGBA, the letters R, G, and B represent the color components red, green, and blue, which constitute the color contribution of the corresponding sampling point. A represents the ALPHA value, which represents a measure of transparency at the sampling point. The corresponding transparency is used when superimposing the RGB values ​​at the sampling point onto the pixel. Illumination effects are typically considered using a lighting model within a framework called “shading.”

[0005] Another method for stereoscopic rendering is the so-called path-tracing method (see Kajiya, “The rendering equation,” ACM SIGGRAPH Computer Graphics, 20th edition, Vol. 4, August 1986, pp. 143-150). Here, for each visualized pixel, multiple simulated rays are projected onto volume data, and then the simulated rays interact with the volume, i.e., reflect, refract, or absorb, wherein at least one random ray is generated each time (except in the case of absorption). Thus, each simulated ray finds its path through the volume data. The more virtual rays used per visualized pixel, the closer it gets to the ideal image. The manner and method described in EP 3 178 068B1 can be applied in particular here. The contents of EP 3 178 068B1 are incorporated herein by reference in their entirety.

[0006] A major technical hurdle in implementing systems for interactive stereo rendering is the fast, efficient, and local processing of large volumes of data. Fast volumetric rendering is particularly challenging on devices with limited computing or graphics capabilities. Examples of such devices are mobile devices, such as smartphones or tablets, or "standalone" augmented reality, mixed reality, or virtual reality devices (such as augmented reality glasses that project visualizations onto a user's field of vision). These devices are typically equipped with energy-efficient processors ("System on a Chip"), which have significantly lower computing power compared to modern PCs. Furthermore, because augmented reality, mixed reality, and virtual reality applications require high frame rates, typically 60Hz, 90Hz, or even 120 / 144Hz, such systems quickly reach the limits of their computing power when it comes to mapping volumetric data onto visualizations in real time. This is especially important in medical imaging, where very large volumetric datasets occur regularly. For example, Siemens... A typical CT scanner can generate volumetric datasets of 512×512×4096 pixels. The capacity available for visualizing such datasets with high image quality is helpful for medical professionals in diagnosis. However, due to the limited computational power of multiple visualization systems, image frame rates are finite, which often severely limits the practical use and interactivity of existing solutions. While image frame rates can be improved by reducing the computational cost of visualizing images, the quality of the visualization is often no longer sufficient to achieve the intended purpose, such as the diagnosis of medical image datasets. Summary of the Invention

[0007] Therefore, the objective of this invention is to provide an improved method and apparatus for visualizing volumetric data. In particular, it is necessary to address the problem of achieving shorter computation time for visualizing images while maintaining acceptable quality, thereby enabling high image frame rates, for example, on devices with energy-efficient processors.

[0008] According to the present invention, the proposed objective is achieved by means of a method, apparatus, computer program product, or computer-readable storage medium according to embodiments of the invention. Advantageous improvements are described in the following description.

[0009] In the following description, the solution according to the invention is described not only in terms of the claimed device but also in terms of the claimed method. The features, advantages, or alternative embodiments mentioned herein can also be applied to other claimed subjects, and vice versa. In other words, physical embodiments (such as those for devices) can also be improved by incorporating the features described or claimed in connection with the method. Here, the corresponding functional features of the method are constituted by corresponding physical modules.

[0010] Furthermore, the solution according to the invention for the stated objective is described not only in terms of methods and apparatus for visualizing three-dimensional volumes but also in terms of methods and apparatus for adjusting trained functions. Here, features and alternative implementations of data structures and / or functions in the methods and apparatus for determining can be applied to similar data structures and / or functions in the methods and apparatus for adjusting. Here, a feature of similar data structures is particularly the use of the prefix "training". Furthermore, the trained functions used in the methods and apparatus for visualizing three-dimensional volumes can be adjusted and / or provided, in particular, by methods and apparatus for adjusting trained functions.

[0011] According to one aspect of the present invention, a computer-implemented method is provided for visualizing a three-dimensional object represented by volumetric data to a user in the form of a two-dimensional visualization image composed of a plurality of visualization pixels (in other words, for stereoscopic rendering of the three-dimensional object). First, the method includes the step of providing an image compositing algorithm configured to visualize the three-dimensional object by mapping volumetric data onto visualization pixels. A next step includes detecting the user's current viewing direction relative to the visualization image. Based on this, in another step, a focal region is determined, wherein the focal region is defined as the area of ​​the visualization image that the user focuses on based on the current viewing direction. As another step, the method includes adjusting the image compositing algorithm such that the computational cost for visualization pixels outside the focal region is less than the computational cost for visualization pixels within the focal region. Finally, the step of mapping volumetric data onto visualization pixels is performed using the adjusted image compositing algorithm.

[0012] Volumetric data can contain multiple voxels. A voxel (“volume pixel” or 3D pixel) is a volume element that represents a value on a regular grid in three-dimensional space. A voxel is analogous to a pixel representing two-dimensional image data. Like a pixel, a voxel itself does not typically contain its position in space (the voxel's coordinates), but rather the voxel's coordinates are derived based on the voxel's position relative to other voxels (i.e., the voxel's position within the data structure that forms a single volumetric image). The value of a voxel can represent different physical properties of a 3D object, such as, for example, local density. For example, in a computed tomography (CT) scan, this value, expressed in Henle units, represents the opacity of the mapped material with respect to X-rays. In this way, volumetric data describes a 3D object within a volume of an object. Volumetric data can particularly describe the (especially non-uniform) density of a 3D object within a volume of an object.

[0013] Volumetric data can be provided, in particular, through medical imaging methods. These methods may be based on, for example, X-ray fluoroscopy, computed tomography (CT), magnetic resonance imaging (MR), ultrasound, and / or positron emission tomography (PET). Correspondingly, the three-dimensional object can be the patient's body or a body part. Here, the three-dimensional object may include one or more of the patient's organs.

[0014] Image compositing algorithms can be understood, in particular, as computer program products configured to map volumetric data onto a two-dimensional projection plane, reproduce the volume of a three-dimensional volume, or perform stereoscopic rendering of a three-dimensional volume. Here, the projection plane is given by a visualized image. The image compositing algorithm may have program components in the form of one or more instructions for a processor used to compute the visualized image. The visualized image consists of multiple visualized pixels. Here, the resolution of the visualized image with respect to the visualized pixels is, in particular, spatially constant, uniform, or spatially consistent. For example, other terms used for image compositing algorithms include "renderer," "rendering algorithm," or "volume renderer." Image compositing algorithms can be provided, for example, by storing the image compositing algorithm in a storage device or loading it into the working memory of a suitable data processing device, or generally by providing the image compositing algorithm for application.

[0015] Here, the image synthesis algorithm can be implemented individually or in combination as different methods for visualizing volumetric datasets. For example, the image synthesis algorithm may have a ray casting module and / or a path tracing module. The image synthesis algorithm maps volumetric data onto visualization pixels. Here, for each visualization pixel, the computation of the mapping (i.e., the value of the corresponding visualization pixel) is related to the computational cost; the higher the level of detail and quality of the mapping produced by the image synthesis algorithm, the higher the computational cost. Here, the image synthesis algorithm is specifically configured such that the computational cost for each visualization pixel can be dynamically set. In particular, the computational cost can be set individually for each visualization pixel. By adjusting the computational cost of each visualization pixel, the total computational cost for creating the visualization image can ultimately be set.

[0016] To detect a user's current viewing direction, a detection system can be used to detect and evaluate the user's eye position, pupil position, or head position. Eye position, pupil position, or head position can be detected, for example, by a camera system and evaluated in a suitable image evaluation algorithm. If this information is related to the position of the visualization image displayed to the user, it can be determined which region or pixel (focal region or focal pixel) of the visualization image the user is observing. This relationship can be achieved, for example, through appropriate registration of the visualization image's position with the user's position and / or the position of the detection system used to detect the user's current viewing direction.

[0017] Determining the focus region can be based on identifying the focus pixel from the visualized pixels, where the focus pixel is defined as the pixel aligned with the user's current viewing direction. The focus region can include only the focus pixel from a plurality of visualized pixels or the region defined around the focus pixel.

[0018] Now, we can selectively adjust the image compositing algorithm using knowledge of the focal region as follows: high-quality display of 3D objects can be achieved within the focal region, while reducing the quality outside the focal region. To this end, the image compositing algorithm is adjusted as follows: the computational cost for visualization pixels within the focal region is greater than the computational cost for visualization pixels outside the focal region.

[0019] If a volumetric dataset is mapped using an image compositing algorithm adjusted in this way, or if a visualization image is computed using such an algorithm, a visualization image with a first image quality in the focal region and a lower image quality outside the focal region is produced. Here, the lower image quality can—at the same resolution for the visualization pixels—manifest as a lower level of detail in the mapping or a reduction in the mapping effect considered during rendering.

[0020] Therefore, in other words, when mapping volumetric datasets, computational power is saved precisely where it can be saved for the user—that is, in areas where the user of the visualized image is not focused and is at most only peripherally perceived. In this way, the claimed method achieves a balance between providing the user with high-quality visualized images and reducing the computational cost required for this. Conversely, this makes it feasible to compute visualized images on portable terminal devices whose computational capabilities are limited by the system, and to achieve an acceptable image frame rate. Thus, the proposed method considers technical conditions outside the terminal device when image quality and image frame rate are required. Furthermore, it also considers the technical conditions of the terminal device itself, namely, its typically limited computational capabilities.

[0021] According to one implementation, the visualization image has a uniform spatial resolution for its visualization pixels. In other words, the visualization pixels have a uniform, spatially constant resolution with respect to the 3D object to be mapped. Therefore, to adjust for computational costs when creating the visualization image, the resolution of the visualization pixels is not adapted.

[0022] This has the following advantages: computational costs can be adjusted directly when rendering volumetric data, without subsequently combining visualizations of different resolutions. This allows for smoother transitions between different quality levels that are less noticeable to the user.

[0023] According to one implementation, in the adjustment step, the image synthesis algorithm is adjusted such that the computational cost is reduced based on the distance from the focal region (or the focal pixel).

[0024] By varying the computational cost based on the distance, a level of computational cost that is easily achievable in the calculation can be reduced to the aforementioned extent. Here, the computational cost can decrease continuously (or persistently) as the distance from the focal region increases. For example, the image synthesis algorithm can be adjusted so that the computational cost decreases linearly, quadratically, exponentially, etc. By continuously varying the computational cost, the reduction in computational cost of the visualized image regarding the region surrounding the focal region, and the accompanying quality loss, can be made less noticeable to the user.

[0025] Alternatively, the image synthesis algorithm can be configured such that the computational cost decreases discontinuously at one or more discrete levels as the distance from the focal region increases. In other words, one or more concentric rings of regions (not necessarily circular) can be generated in the visualized image. These rings then have reduced computational cost starting from the interior, i.e., from the focal region. This discontinuous change in computational cost, compared to the continuous adjustment of computational cost described above, can selectively lead to simplified computations in the image synthesis algorithm, thereby further reducing the overall computational cost.

[0026] The following sections present some implementation methods for adjusting computational cost based on the image processing algorithm used. These measures can be combined with each other to achieve a demand-appropriate adjustment of computational cost.

[0027] According to one implementation, the image synthesis algorithm has a first mapping algorithm and a second mapping algorithm different from the first mapping algorithm. Here, "different" can specifically mean that the mapping algorithms are based on different methods or principles for sampling volumetric data. The computational cost per pixel of the first mapping algorithm can be systematically higher than that of the second mapping algorithm. Therefore, the first mapping algorithm generally also provides better image quality than the second mapping algorithm. In the adjustment step, it is now proposed that the first mapping algorithm be pre-registered for mapping volumetric data to visual pixels in the focal region, and the second mapping algorithm be pre-registered for mapping visual pixels outside the focal region. Thus, during mapping, the first mapping algorithm is used for the focal region, while the second mapping algorithm is used for regions of the visual image outside the focal region. Therefore, the higher-value (but more computationally expensive) first mapping algorithm can at least be used for the focal region observed by the user, even if the available computational power does not allow it to be applied to the entire visual image. The first mapping algorithm can be implemented, in particular, as a path-tracking method, while the second mapping algorithm can be implemented as a ray-casting method. Here, the advantage of the path tracing method is that multiple global illumination effects important for the true mapping of volumetric data are automatically considered through multiple scattering, and do not need to be implemented individually—as in light projection. For example, this is applicable to ambient occlusion or illumination effects caused by extended light sources.

[0028] According to one implementation, the image synthesis algorithm is based on the tracking of simulated rays in the object volume expanded by volume data, and adjusting the image synthesis algorithm includes adjusting the rate at which the object volume is sampled along the simulated rays (the so-called sampling rate).

[0029] An exemplary image synthesis algorithm based on tracing simulated rays through an object volume is the ray casting or path tracing method. In the ray casting method, the sampling rate determines the spatial frequency of the sampling points in the object volume. The higher the sampling rate, the more points are sampled in the object volume for the visualized pixel. The more points sampled, the better the quality of the mapping (or in other words, the image quality), but the higher the computational cost per visualized pixel.

[0030] In path tracing, the sampling rate can be understood as a relative measure of the spatial frequency of scattering events ("bounces") within an object volume. Thus, the sampling rate involves not only the simulated rays initially "injected" into the volume but also the simulated scattered rays. The same applies here: a higher sampling rate results in better mapping quality, but also higher computational cost per visualized pixel.

[0031] Here, the sampling rate for each visualization pixel can be reduced, in particular, according to the distance (continuous or discontinuous) between the corresponding visualization pixel and the focal region, especially reduced to the minimum sampling rate.

[0032] For the image synthesis algorithm mentioned, the sampling rate represents a highly efficient and easily controllable parameter that influences the computational cost in visualizing the image. Here, the optional limitation of the minimum sampling rate ensures that the minimum quality is not compromised.

[0033] According to one implementation, multiple different visualization modules are implemented in the image synthesis algorithm. These different visualization modules consider different mapping effects when mapping volumetric data. Specifically, the visualization modules are configured such that the contribution of each visualization module to the computational cost of each visualized pixel can be individually adjusted and / or each visualization module can be individually activated and disabled. Then, in the adjustment step, the contribution of each visualization module / mapping effect to the computational cost is selectively adjusted and / or individual visualization modules are selectively activated and / or disabled.

[0034] Here, "implementation" can specifically mean that the image compositing algorithm has components and / or program elements that allow the computation of associated mapping effects. Adjusting the contribution to computational cost is accompanied by adjusting the quality of the mapping effects. Mapping effects can be selected, for example, from: ambient occlusion, shadows, translucency, bleed, surface shading, complex camera effects, and / or illumination caused by arbitrary ambient light conditions. By selectively adjusting or activating (disabling) individual mapping effects, the algorithm can flexibly respond to available computational power and the required image frame rate.

[0035] According to one implementation, priorities are assigned to visualization modules (e.g., in the form of priority order or sequence) (e.g., by determining priorities based on mapping effects), wherein in the adjustment step, selective adjustment and / or activation and / or disabling are performed based on priorities.

[0036] Here, adjustments can be made such that the quality of mapping effects with higher priority (e.g., shadows) is set higher than that of mapping effects with lower priority (e.g., bleed). The priority determination process can be understood as creating an order for mapping effects based on how strongly a potential loss of quality would affect user perception.

[0037] By prioritizing mapping effects, those that are important to the user can be preferentially processed relative to less important ones. If computational cost must be reduced, the quality of higher-priority mapping effects can be reduced less drastically than that of lower-priority ones.

[0038] According to one implementation, a visualization module for taking into account the mapping effect of ambient occlusion is implemented in the image synthesis algorithm, wherein adjusting the image synthesis algorithm includes adjusting the computational cost when considering ambient occlusion.

[0039] In the literature, the effect of ambient occlusion is also referred to as "ambient occlusion." To account for ambient occlusion, for example in light projection, multiple ambient occlusion test rays (ambient occlusion test rays) can be sent to each sampling point to determine whether ambient occlusion exists. Here, adjusting computational costs when considering ambient occlusion can particularly include adjusting the number of ambient occlusion test rays (ambient occlusion test rays).

[0040] According to one implementation, the image synthesis algorithm has a path tracking algorithm, wherein in the adjustment step, the number of scattering events for each simulated ray is adjusted.

[0041] For example, the number of scattering events can be affected by the number of Monte Carlo iterations and the premature termination of scattered rays when an opacity threshold is reached or a maximum number of scattering events is determined. Adjusting the number of scattering events can effectively regulate the computational cost in path tracing methods.

[0042] Alternatively or additionally, the adjustment steps may include adjusting the number of simulated rays that sample the visualized pixels during path tracing. The more lines of sight considered, the better the quality of the mapping, but the higher the computational cost.

[0043] According to one implementation, the image synthesis algorithm has a ray casting algorithm, wherein the number of shadow rays is adjusted in the adjustment step.

[0044] In light projection, the lighting effect at each sampling point is calculated by sending illumination rays or shadow rays that connect the sampling point to a point on the light source. Because this connection may be obscured by an object or can pass through a milky medium, it is generally referred to here as a shadow ray. The more shadow rays there are, the more realistic the reproduction of the local lighting effect, and the more realistic the values ​​assigned to the associated visual pixels. However, computational cost also increases with the number of sampled shadow rays. Therefore, adjusting the number of shadow rays in a light projection system represents an effective adjustment screw for controlling computational cost.

[0045] According to one implementation, adjusting the image synthesis algorithm includes adjusting the coloring quality.

[0046] Here, shading (vignetting) means associating color and transparency values ​​with individual points using a lighting model. This applies to sampling points in light projection and scattering areas in path tracing. To calculate color and transparency values, transmission and reflection components must be calculated. This requires calculating surface normals. For this, the local gradient of the density of a 3D object is calculated based on volume data. In particular, if there are only small variations in volume properties, the quality of the gradient calculation (also referred to here as gradient quality) has a significant impact on shading quality. Therefore, adjusting shading quality can be done by adjusting the gradient quality used for shading calculations. Shading quality can be adjusted in particular so that the gradient quality used for visualization pixels decreases as the distance from the focal region increases. Higher gradient quality generally means higher computational cost for calculating the gradient. For example, in gradient calculations, central difference provides better conclusions about local slopes than simple forward difference, but central difference is more computationally expensive. To further improve gradient quality, higher-order differences can be considered when calculating finite differences; however, this further increases computational cost. Adjusting shading quality or gradient quality represents the effective feasibility of adjusting the computational power of each visualization pixel.

[0047] According to one embodiment, the method further includes adapting the resolution of the volume data such that, during the mapping step, the resolution of the volume data for visualization pixels outside the focal region is lower than the resolution of the volume data for visualization pixels within the focal region.

[0048] Here, volume data can be configured with different resolution levels (so-called Mipmapping). High-resolution volume data is sampled for the focal region, while lower resolution is used for regions outside the focal region. Using different resolutions for the volume data has no impact on the resolution of the visualized image. The object volume itself can be mapped to a spatially constant distribution of visualized pixels; however, depending on the region of the visualized image, object information is resolved to varying degrees. In other words, if the resolution of the volume data is reduced, the number of voxels per volume element decreases. Fewer interactions of simulated rays are automatically derived, thereby reducing computational cost. Therefore, computational cost can be effectively reduced by appropriately selecting the mipmap level.

[0049] According to one embodiment, the method further includes the step of determining the time budget and / or available computing power for creating the visualization image, wherein the adjustment step is performed taking into account the time budget and / or available computing power.

[0050] Therefore, in other words, the computational cost of each visualized pixel is selectively adapted to the available time budget and / or computational power. This allows for optimal utilization of available computational power. In particular, it ensures that the mapping quality is not unnecessarily degraded. Available computational power can be understood as the currently free computational power of the device performing the computer-implemented method. Available computational power can be provided by querying current system resources (e.g., processor utilization and / or working memory utilization). The time budget can be given by the boundary conditions used to display the mapping to the user. For example, the boundary conditions can be given by a minimum image frame rate. For example, if 60 frames per second (60Hz display) are to be displayed, a new frame must be computed at least every 16 milliseconds. Therefore, the time budget is 16 milliseconds. For a 100Hz display, the time budget is 10 milliseconds. For a 144Hz display, the time budget is approximately 7 milliseconds. If, for stereoscopic displays, such as in augmented reality glasses, two separate images must be computed per frame, the time budget for a 60Hz display is 8 milliseconds, and the time budget for a 144Hz display is 3.5 milliseconds. According to one implementation, the time budget is between 3.5 milliseconds and 42 milliseconds, and more particularly between 7 milliseconds and 16 milliseconds.

[0051] According to one embodiment, the method further includes the step of applying a filtering algorithm to a visualized image.

[0052] By applying filtering algorithms, quality loss outside the focal region can be at least partially compensated with relatively low computational cost. Furthermore, it can smooth transitions between regions of a visualization image with different qualities or levels of detail. For example, Gaussian filters, low-pass filters, bilinear filters, or combinations of these filters can be used as filtering algorithms. Here, according to one embodiment, the image filter is applied to the entire visualization image, i.e., to the entire region. According to another embodiment, the image filter is applied only to the region outside the focal region.

[0053] According to one embodiment, the method further includes the step of generating an improved visualization image by applying a trained function to the visualization image, wherein the trained function is designed such that the trained function improves the image quality of the visualization image, in particular by correcting the effects of adjustments made by the image synthesis algorithm.

[0054] Therefore, in other words, the trained function is applied to the visualized image. The trained function typically maps input data to output data. Here, the output data can also be related to one or more parameters of the trained function. One or more parameters of the trained function can be determined and / or adjusted through training. Determining and / or adjusting one or more parameters of the trained function can be based, in particular, on a pair of training input data and associated training output data, where the trained function is applied to the training input data to produce training mapping data. In particular, determination and / or adjustment can be based on a comparison of the training mapping data and the training output data. Generally, a trainable function, i.e., a function with parameters that have not yet been adjusted, is also referred to as the trained function.

[0055] Other terms used for the function being trained are a training mapping rule, a mapping rule with training parameters, a function with training parameters, an artificial intelligence-based algorithm, and a machine learning algorithm. An example of a trained function is an artificial neural network. The term "neural network" can also be used as an alternative to "neural network." Neural networks are essentially constructed like biological neural networks, such as the human brain. Artificial neural networks specifically include an input layer and an output layer. The artificial neural network may also include multiple layers between the input and output layers. Each layer includes at least one, preferably multiple, nodes. Each node can be understood as a biological processing unit, such as a neuron. In other words, each neuron corresponds to an operation applied to the input data. Nodes in one layer can be connected to nodes in other layers via edges or connections, especially directed edges or connections. These edges or connections define the data flow between nodes in the network. Edges or connections are associated with parameters commonly referred to as "weights" or "edge weights." These parameters can adjust the importance of the output of a first node to the input of a second node, where the first and second nodes are connected by an edge.

[0056] In particular, neural networks can be trained. Specifically, training a neural network is performed using "supervised learning" techniques, based on training input data and associated training output data. This involves feeding known training input data into the neural network and comparing the output data generated by the network with the associated training output data. As long as the output data of the final network layer does not sufficiently correspond to the training output data, the artificial neural network learns and independently adjusts the edge weights used for each node.

[0057] In particular, the function being trained can also be a deep artificial neural network. According to one implementation, the function being trained can be a so-called image-to-image neural network, that is, a neural network that maps image data onto image data. The input data will then be a visualized image, and the output data will be an improved visualized image. For example, a convolutional neural network or a deep convolutional neural network can be used. Here, the neural network has one or more convolutional layers and one or more deconvolutional layers. The neural network can also include pooling layers.

[0058] Neural networks can be used particularly effectively for image processing by using convolutional and / or deconvolutional layers because, despite the multiple connections between nodes, only a small number of edge weights (i.e., edge weights corresponding to the values ​​of the convolutional kernels) must be determined during training. This allows for improved accuracy of the neural network with the same amount of training data.

[0059] According to another aspect, a computer-implemented method is provided for providing a trained function for improving a two-dimensional visualization image composed of multiple visualization pixels. The method comprises the following steps:

[0060] - Provide training input data, wherein the training input data has a first visualization image of a three-dimensional object, the first visualization image being generated by an image synthesis algorithm based on volume data with a first total computational cost, the volume data representing the three-dimensional object;

[0061] - Provide training output data, wherein the training output data has a second visualization image of a three-dimensional object, the second visualization image being generated by an image synthesis algorithm based on volume data with a second total computational cost, wherein the first total computational cost is less than the second total computational cost;

[0062] - To produce improved visualizations by applying the trained function to the training input data;

[0063] - Compare the improved visualization images with the training output data;

[0064] - Adjust the trained function based on comparison.

[0065] The described training method has the following advantages: the training output data can be generated by the image synthesis algorithm by adjusting its parameters. This not only enables the trained function to be specifically aligned with the image synthesis algorithm, but also ensures that training datasets can be generated in virtually any quantity, which can significantly improve the training of the trained function.

[0066] Here, according to one implementation, the step of providing training input data may in particular include the following steps:

[0067] - Determine the focal area in the first visualization image, and

[0068] - By using an image synthesis algorithm, volume data is mapped onto the visualization pixels of the first visualization image, making the computation cost for calculating visualization pixels outside the focal region less than the computation cost for calculating visualization pixels within the focal region.

[0069] According to one implementation, the step of providing training output data may include:

[0070] - By using an image synthesis algorithm, volume data is mapped onto the visualization pixels of the second visualization image, such that all visualization pixels of the second visualization image are calculated at least as much as the visualization pixels in the focal region of the first visualization image.

[0071] According to another aspect, a training system is provided for providing a function to be trained, the training system comprising one or more method steps of the above-described method for providing the function to be trained.

[0072] According to another aspect, a visualization device is provided for visualizing a three-dimensional object represented by volumetric data, wherein the visualization is presented to a user in the form of a two-dimensional visualization image composed of a plurality of visualization pixels. The device has an interface and a computing unit. The interface is configured to receive volumetric data and detection information, the detection information including information about the user's current viewing direction. The computing unit is configured to determine a focal region based on the detection information, wherein the focal region is defined as the area of ​​the visualization image that the user is focusing on. The computing unit is further configured to map the volumetric data used to create the visualization image onto the visualization pixels, and hereby adjust the mapping such that the computational cost for visualization pixels outside the focal region is less than the computational cost for visualization pixels within the focal region.

[0073] The computing unit can be configured as a central or distributed computing unit. The computing unit can have one or more processors. The processors can be configured as a central processing unit (CPU) and / or a graphics processing unit (GPU). The computing unit can be configured as a so-called system-on-a-chip (SoP) that controls all functions of the device. Alternatively, the computing unit can be implemented as a local or cloud-based processing server. To map volumetric data, the computing unit can be configured to apply the aforementioned image compositing algorithm and adapt the image compositing algorithm according to the aforementioned adjustment steps.

[0074] Interfaces typically serve as a means of exchanging data between a computing device and other components. These interfaces can be implemented as one or more individual data interfaces, which may have hardware and / or software interfaces, such as a PCI bus, USB interface, Fire-Wire interface, ZigBee interface, or Bluetooth interface. The interface may also have an interface to a communication network, which can be a local area network (LAN), such as an intranet or a wide area network (WAN). Correspondingly, one or more data interfaces may have a LAN interface or a wireless LAN interface (WLAN or Wi-Fi).

[0075] The advantages of the proposed device substantially correspond to the advantages of the proposed method. The features, advantages, or alternative embodiments can also be adapted to other claimed subjects, and vice versa.

[0076] According to another aspect, a visualization system is provided, which has the aforementioned visualization device for visualizing a three-dimensional object represented by volume data. Furthermore, the visualization system includes a detection device configured to detect detection information and provide it to a computing unit via an interface.

[0077] According to one embodiment, the detection device has one or more optical detection devices, particularly a camera device.

[0078] According to one embodiment, the visualization system has a reproduction device configured to receive a visualization image from a computing unit via an interface and display it to a user.

[0079] The reproduction device may have a screen device with one or more screens and / or a projection device. The projection device may be configured to project a visualized image onto a glass or lens located in the user's field of view.

[0080] According to one implementation, the visualization system is configured as a "wearable device," and more particularly as augmented reality glasses.

[0081] In another aspect, the present invention relates to a computer program product comprising a program that can be directly loaded into the memory of a programmable computing unit and having program mechanisms, such as libraries and auxiliary functions, so that when the computer program product is executed, a method for visualizing three-dimensional objects, particularly according to the foregoing aspects, is performed.

[0082] Furthermore, in another aspect, the present invention relates to a computer program product comprising a program that can be directly loaded into the memory of a programmable computing unit and having program mechanisms, such as libraries and auxiliary functions, so that when the computer program product is executed, a method for providing trained functions, particularly according to the foregoing aspects, is performed.

[0083] Here, a computer program may include: software having source code, which must also be compiled and linked or only needs to be interpreted; or executable software code that, in order to be executed, must also be loaded into a processing unit. Through the computer program product, the method can be executed quickly, identically, repeatably, and robustly. The computer program product is configured such that it can perform the method steps according to the invention by means of a computing unit. Here, the computing unit must have prerequisites, such as, for example, a corresponding working memory, a corresponding processor, a corresponding graphics card, or a corresponding logic unit, so that the corresponding method steps can be executed efficiently.

[0084] The computer program product is stored, for example, on a computer-readable storage medium or stored on a network or server, from which it can be loaded into the processor of a corresponding computing unit, which can be directly connected to the computing unit or constitute part of the computing unit. Furthermore, control information of the computer program product can be stored on a computer-readable storage medium. The control information of the computer-readable storage medium can be configured such that when a data carrier is used in the computing unit, the control information executes the method according to the invention. Examples of computer-readable storage media are DVDs, magnetic tapes, or USB flash drives, on which electronically readable control information, especially software, is stored. When the control information is read from the data carrier and stored in the computing unit, all embodiments of the above-described method according to the invention can be executed. Therefore, the invention can also be based on the computer-readable medium and / or the computer-readable storage medium. The advantages of the proposed computer program product or associated computer-readable medium substantially correspond to the advantages of the proposed method. Attached Figure Description

[0085] Other features and advantages of the invention will become apparent from the following description of embodiments with reference to the schematic diagrams. Modifications mentioned in this context can be combined with each other to form new embodiments. In different figures, the same reference numerals are used for the same features.

[0086] The attached diagram shows:

[0087] Figure 1 A schematic diagram illustrating an implementation of a system for visualizing three-dimensional objects is shown.

[0088] Figure 2 A flowchart illustrating a method for visualizing a three-dimensional object according to an embodiment is shown.

[0089] Figure 3 A schematic visualization of a three-dimensional object according to an embodiment is shown.

[0090] Figure 4 Showing through Figure 3 The visualization shown is a schematic curve illustrating the change in image quality or computational cost along a cross-section of line AB.

[0091] Figure 5 A schematic visualization of a three-dimensional object according to an embodiment is shown.

[0092] Figure 6 Showing through Figure 5 The visualization shown is a schematic curve illustrating the change in image quality or computational cost along a cross-section of line AB.

[0093] Figure 7 The trained function used to generate improved visualizations is shown.

[0094] Figure 8 A schematic diagram illustrating an implementation of a system for providing a trained function, and

[0095] Figure 9 A flowchart illustrating a method for providing a trained function for improving the visualization of a three-dimensional object, according to an embodiment. Detailed Implementation

[0096] Figure 1 The diagram illustrates a system 1 for visualizing a three-dimensional object according to an embodiment. System 1 includes a computing unit 20, an interface 30, a detection device 40, a reproduction device 50, and a storage unit 60. The computing unit 20 is configured to calculate a visual image VB of the three-dimensional object based on volume data VD describing the object. The volume data VD can be provided to the computing unit 20 from the storage unit 60 via the interface 30.

[0097] Storage unit 60 can be configured as a central or distributed database. Storage unit 60 can be, in particular, part of a server system. Storage unit 60 can be, in particular, part of a so-called Picture Archiving and Communication System (PACS). Volumetric data VD can be generated by medical imaging methods. For example, volumetric data VD can be generated by X-ray fluoroscopy, computed tomography (CT), magnetic resonance imaging (MR), ultrasound, and / or positron emission tomography (PET). For example, volumetric data VD can be formatted in DICOM format. Here, DICOM stands for Digital Imaging and Communication in Medicine and represents an open standard for storing and exchanging information in the management of medical image data. Volumetric data VD has a three-dimensional dataset composed of multiple volume pixels, so-called voxels. Voxel values ​​can be obtained, for example, by the aforementioned medical imaging methods and generally represent a measure of the local density of a three-dimensional object at the location of a voxel. Providing volumetric data VD can, for example, include loading the volumetric data VD into a working memory (not shown) of computing unit 20.

[0098] The computing unit 20 is configured to consider the user's viewing direction when calculating the visualization or mapping of a 3D object. To this end, detection information EI, containing information about where the user is looking, is provided to the computing unit 20 via interface 30. The detection information EI is provided by a detection device 40. The detection device 40 is designed to detect the user's viewing direction. For this purpose, the detection device may have an optical detection mechanism, such as a camera, that detects the user's eye portion and / or pupil position and / or head position. The detection information EI may then, for example, contain image data of the eye portion, pupil position, and / or head position.

[0099] If a visualization image VB is calculated, it should be displayed to the user. For this purpose, the visualization image VB is provided to the playback device 50 via interface 30. The playback device 50 may have one or more screen devices and / or projection devices that constitute a means for reproducing the visualization image VB for the user. In particular, the playback device 50 may be implemented as an image display system for augmented reality glasses. Alternatively, the playback device 50 may have a screen from a PC, laptop, tablet, or smartphone.

[0100] Interface 30 may have one or more individual data interfaces that ensure data exchange between components 20, 40, 50, and 60 of system 1. The one or more data interfaces may have hardware interfaces and / or software interfaces, such as a PCI bus, USB interface, Fire-Wire interface, ZigBee interface, or Bluetooth interface. The one or more data interfaces may have an interface to a communication network, which may be a local area network (LAN), such as an intranet or a wide area network (WAN). Correspondingly, the one or more data interfaces may have a LAN interface or a wireless LAN interface (WLAN or Wi-Fi).

[0101] The computing unit 20, interface 30, detection device 40, and reproduction device 50 are preferably integrated into the augmented reality glasses. The computing unit 20 can then be configured as the processor of the augmented reality glasses. Here, the data connection to the storage unit 60 can particularly be configured as a wireless interface.

[0102] To calculate the visualization image VB based on input data, the calculation unit 20 may have different elements 21, 22, 23, and 24. Element 21 can be understood as a focal region calculation module. Element 21 may implement or control an algorithm that constitutes a calculation of the focal region FB within the visualization image VB where the user is focusing their attention, based on detection information EI. For this purpose, the detection information EI is evaluated by element 21 and correlated with the displayed or to-be-displayed visualization image VB. For this purpose, for example, appropriate registration may be utilized—for example, between the detection device 40 and the reproduction device 50. Here, the focal region FB can be defined as the region surrounding the focal pixel FP where the user's attention is focused (see also...). Figure 3 and Figure 5 ).

[0103] Component 22 can be understood as a volumetric rendering engine. Here, component 22 can implement or control an image compositing algorithm configured to map volumetric data VD onto a visualization image VB or pixels of the visualization image VB. Hereinafter, pixels of the visualization image VB are referred to as visualization pixels VP. To calculate the mapping to visualization pixels VP, the image compositing algorithm can have different visualization modules, which are selectively and particularly for each visualization pixel, and / or the computational cost of the visualization modules is configurable. The visualization modules may, for example, involve different schemes for calculating the visualization image VB. For example, the image compositing algorithm may include a ray casting module in which the visualization pixels VP are calculated by means of ray casting. Furthermore, the image compositing algorithm may have a path tracing module in which the visualization pixels are calculated according to a path tracing method. Additionally, the visualization modules may involve supplementary mapping effects. Particularly in the ray casting method, the supplementary mapping effects may include, for example, effects of ambient occlusion, shadow effects, light transmission effects, color bleeding effects, surface shadows, complex camera effects, and / or lighting effects caused by arbitrary ambient lighting conditions. Thus, element 22 is configured such that the computational cost and thus the quality of the mapping can be selectively set for each individual visualization pixel VP.

[0104] The aforementioned feature is used in element 23 to appropriately adjust the computational cost of each visualization pixel VP. Here, the computational cost of each visualization pixel VP is set such that each visualization pixel VP has a higher computational cost in the focal region FB compared to outside the focal region. This ensures sufficient mapping quality within the focal region FB while simultaneously reducing the total computational cost for visualizing the image.

[0105] Element 24 can be understood as a filtering element. Element 24 is configured to apply an image filter to the visualization image VB calculated by the adjusted image synthesis algorithm, thereby compensating for quality loss or artifacts that may be caused by computationally expensive dynamic adjustments. Here, the image filter can be implemented in particular as a trained function configured to optically improve the visualization image VB.

[0106] The subdivision of computing unit 20 into elements 21-24 is used only to more simply explain how computing unit 20 operates and should not be construed as limiting. Elements 21-24, or their functions, may also be combined in a single element. Element 21-24 may also be understood in particular as a computer program product or computer program segment that, when executed in computing unit 20, implements one or more of the method steps described below.

[0107] exist Figure 2 The diagram shows a schematic flowchart of a method for visualizing a 3D object. The order of the method steps is neither limited by the shown order nor by the chosen numbering. Therefore, the order of the steps can be interchanged if necessary, and individual steps can be omitted.

[0108] The optional first step S1 aims to provide volume data VD. Here, the provision can be achieved by retrieving volume data VD from storage unit 60 and / or loading volume data VD into computing unit 20.

[0109] Another optional step S10 aims to provide an image synthesis algorithm. Here, the provision can be achieved by storing the image synthesis algorithm in any memory (e.g., storage unit 60 or any internal memory of computing unit 20) and / or requesting the image synthesis algorithm from any memory and / or loading the image synthesis algorithm into computing unit 20.

[0110] In the next step S20, the user's current viewing direction is detected. For this purpose, detection information EI can first be provided by the detection device 40, and then the detection information can be appropriately evaluated. This can be achieved, for example, by means of an image evaluation mechanism, depending on the type of detection information EI. The current viewing direction can be particularly related to the display surface used for visualizing the image VB, and thus to the currently displayed or future to be displayed visualizing the image VB. In this way, the current viewing direction indicates which part or which visual pixel VP of the visualizing image VB the user is currently focusing their attention on.

[0111] In the next step S30, the focal region FB in the visualized image VB is calculated based on the detected current viewing direction. Here, the focal region FB can be limited to the area surrounding the focal pixel FP, or it can be limited to the focal pixel FP itself. Here, the focal region FB is particularly circular, but alternative shapes are also conceivable.

[0112] If the focal region FB is known, the image synthesis algorithm is adjusted in the next step S40 so that the computational cost of each visualization pixel VP dynamically adapts to the user's current viewing direction, the time budget for calculating the visualization image VB, and the available computing power. Here, the adjustment is performed such that each visualization pixel VP for the focal region FB allows for a higher computational cost than for visualization pixels VP outside the focal region. Specifically, the computational cost of each visualization pixel VP can be set such that the computational cost of each visualization pixel VP decreases as the distance from the focal region FB increases. For this purpose, different measures are provided in step S40. For example, these measures can be applied to the visualization modules implemented in the image synthesis algorithm, and these measures can turn the visualization modules on or off or limit the quality of the visualization modules. Here, different priorities can be determined for the visualization modules, so that visualization modules whose display quality is less important than other visualization modules are preferentially limited in order to reduce computational cost.

[0113] For example, the following measures can be taken, and all of the measures described can be combined with each other:

[0114] - As the distance from the focal region FB increases, the sampling rate of the simulated ray in the ray projection or path tracing decreases until the minimum sampling rate is reached;

[0115] - As the distance from the focal region FB increases, the sampling rate of scattered rays in path tracking decreases;

[0116] - As the distance from the focal region FB increases, the sampling rate of the shadow ray decreases;

[0117] - As the distance from the focal region FB increases, the ambient occlusion quality (ambient light occlusion quality) is reduced by decreasing the ambient occlusion test ray;

[0118] - As the distance from the focal region FB increases, the quality of the gradient used for shading calculations decreases (from high to low: higher order difference, central difference, forward difference);

[0119] - As the distance from the focal region FB increases, the shading quality decreases;

[0120] - As the distance from the focal region FB increases, the maximum number of scattering events in path tracking decreases;

[0121] -Start from a specific distance from the focal region FB, cut off the shadow ray calculation;

[0122] -Starting from a specific distance from the focal area FB, the ambient occlusion calculation is cut off;

[0123] - Based on the distance from the focal area FB, a hybrid rendering method is used, such as path tracing in the focal area FB, shaded ray casting outside the focal area FB, and unshaded ray casting in the periphery (i.e., unshaded ray casting).

[0124] - In path tracing, the number of Monte Carlo iterations is reduced as the distance from the focal region (FB) increases;

[0125] - Starting at a specific distance from the focal region FB, earlier rays terminate as the sampling increases (sampling ends when the opacity threshold is reached); and / or

[0126] - Starting from a specific distance from the focal region FB, a low-resolution Mipmap level of volumetric data texture is used as the distance increases.

[0127] Depending on the available time budget and computing power, it is also possible to adapt the measures mentioned above.

[0128] Then, in step S50, a visualization image VB is created using an appropriately adjusted image synthesis algorithm. An exemplary visualization image is shown in... Figure 3 and Figure 5 This is illustrated in the diagram. Then, corresponding to the computational cost per visualized pixel, regions of the visualized image have different qualities. Typically, the quality in the focal region FB will be the highest, and then the quality decreases as the distance from the focal region FB increases. This is shown in... Figure 4 and Figure 6 This is illustrated schematically. Here, Figure 4 and Figure 6 The curves showing the changes in image quality or computational cost for a cross-section of the visualized image VB are shown respectively. Figure 4 Show along Figure 3 The cross section of line AB, and Figure 6 Show along Figure 5 The cross-section of line AB. (As from...) Figures 3 to 6 It is also known that reductions in computational cost or image quality can be designed continuously or abruptly. A combination of both is also conceivable. For example, a continuous reduction is achieved if the quality factor, such as the sampling rate, is continuously reduced. Jumps in image quality and computational cost can be generated by switching the visualization module and its modeled effects on or off at intervals, or by switching between different basic mapping methods at intervals. Of course, continuous reductions can take different forms. Therefore, reductions such as linear, quadratic, exponential, or logarithmic reductions can be implemented as a function of the distance from the focal region FB.

[0129] To at least partially compensate for the quality loss, artifacts, or discontinuities caused by adjusting computational costs, step S60 proposes processing the visualization image VB generated in step S50 with an image filter. Here, the image filter may include conventional image filters, such as, for example, Gaussian filters and / or low-pass filters and / or bilinear filters. Furthermore, the image filter may have a trained function TF, which is trained to improve the visualization image VB generated by adjusting computational costs specific to the viewing orientation. Step S60 is optional and may be omitted.

[0130] In step S70, the completed visualization image VB is provided to the playback device 50 via interface 30. In step S80, the playback device 50 displays the visualization image VB to the user. Steps S70 and S80 are optional and can be omitted.

[0131] Step S90 is an optional repetition step. After displaying the visualization image VB to the user, the process jumps back to step S20 via step S90 and re-detects the user's viewing direction. Based on this, a new focal region FB is then calculated in step S30, and the visualization image VB is updated in steps S40, S50, and possibly S60, S70, and S80. By repeating the steps, the calculation of the visualization image VB can continuously track the user's current viewing direction.

[0132] Figure 7 An exemplary diagram of a trained function TF is shown, which can be used in step S60 to improve the visualization image VB. The trained function TF can be configured as an image-to-image network 100, trained to identify and compensate as much as possible for the effects that inevitably occur in the visualization image VB by adjusting computational costs in step S40. The trained function TF takes the visualization image VB generated in step S50 as input and provides an improved visualization image VVB as output. According to one embodiment, the trained function TF has a neural network, and in particular a convolutional neural network (CNN) or a deep convolutional neural network. The trained function TF then has one or more convolutional layers and one or more deconvolutional layers.

[0133] like Figure 7 As shown, the function TF trained as a neural network can have multiple consecutive layers 101-107. Each layer includes at least one, preferably multiple, nodes. Essentially, each node can be viewed as performing a mathematical operation that associates one or more input values ​​with an output value. Nodes in each layer can be connected to all or only a portion of the nodes in previous and / or subsequent layers. Two nodes are "connected" when their inputs and / or outputs are connected. An edge or connection is associated with a parameter commonly referred to as a "weight" or "edge weight". Figure 7 In the example shown, layer 101 is the input layer. The input values ​​for the nodes of layer 101 are preferably the visualized pixels VP of the visualized image VB. Layer 107 is the output layer. The output values ​​for the nodes of output layer 107 can be the pixel values ​​of the improved visualized image VBB. A number of hidden layers 102-106 are located between input layer 101 and output layer 107. Different layers can be used, such as, for example, convolutional layer 102, pooling layer 103, upsampling layer 105, deconvolutional layer 106, or so-called fully connected layer 104. The convolutional layer convolves the input and passes the result of the input to the next layer by moving the image filter on the input. The pooling layer 103 reduces the dimensionality of the data by aggregating the output of a group of nodes in one layer to the individual nodes in the next layer. To reconstruct the improved visualization image VVB, upsampling layer 105 and deconvolution layer 106 reverse the actions of convolutional layer 102 and pooling layer 103. Fully connected layer 104 connects each node of the previous layer to the nodes of the subsequent layer, so that essentially each node receives an "opinion".

[0134] The trained function TF learns by adjusting the weights or weight parameters (e.g., edge weights) of each layer and node. For example, the trained function TF can be trained using supervised learning methods. Backpropagation can be used, for instance. During training, the trained function TF is applied to the training input data to produce corresponding output values, the target values ​​of which are known in the form of the training output data. The difference between the output values ​​and the training output data can be used to introduce a cost function or loss function as a measure of how well or poorly the trained function TF performs the task provided to it. The goal of training is to find the (local) minimum of the cost function by iteratively adjusting the parameters (e.g., edge weights) of the trained function TF. This ultimately enables the trained function TF to provide acceptable results on a (sufficiently) large subset of the training input data. The optimization problem can be performed using stochastic gradient descent or other schemes known in the art.

[0135] Figure 8 An embodiment of a system 200 for training or providing a trained function TF is shown. The system includes a processor 210, an interface 220, working memory 230, a storage device 240, and a database 250. The processor 210, interface 220, working memory 230, and storage device 240 can be configured as a computer 290. The processor 210 controls the operation of the computer 290 during the training of the trained function TF. The processor 210 can be configured to perform... Figure 9 The method steps are shown. Instructions can be stored in working memory 230 or storage device 240 and / or loaded into working memory 230 when execution is desired. Storage device 240 can be configured as local memory or remote memory, and remote memory can be accessed via a network. Figure 9 The method steps shown can be defined by a computer program product stored in working memory 230 and / or storage device 240.

[0136] Database 250 can be implemented as a cloud storage or local storage, and is connected to computer 290 via wireless or wired interface 220. Database 250 can also be part of computer 290. Database 250 serves as an archive for (training) volumetric data T-VD and / or first and second visualization images T-VB-1, T-VB-2 derived therefrom. Furthermore, database 250 can serve as an archive for one or more trained functions TF.

[0137] exist Figure 9 The diagram shows a schematic flowchart of a method for providing a trained function TF for improving visualization images VB. The order of the method steps is neither limited by the shown order nor by the selected numbering. Therefore, the order of the steps can be interchanged if necessary, and individual steps can be omitted.

[0138] The first step T10 aims to provide the trained function TF. Here, the trained function TF can be provided to the processor 210 from the database 250 via interface 220. The trained function TF may have been pre-trained, meaning that one or more parameters of the trained function TF have been adjusted using the described training method and / or other training methods. Alternatively, one or more parameters of the trained function may not be adjusted using training data, and one or more parameters may be preset using constant values ​​and / or random values. All parameters of the trained function TF may not be adjusted using training data, and all parameters may be preset using constant values ​​and / or random values.

[0139] The second step, T20, aims to provide training input data. Since the trained function TF should improve the quality of the visualization image VB in use, optimizing the computational cost of the visualization image while considering the focal region FB, the appropriate training input data is precisely this visualization image VB. The visualization image VB is hereinafter referred to as the first visualization image T-VB-1. According to one embodiment, the first visualization image T-VB-1 can be generated synthetically based on (training) volume data T-VD. Here, the type of training volume data T-VD essentially corresponds to volume data VD. To create the first visualization image T-VB-1, the focal region can be determined manually. Here, "manually" can mean that the focal region is not based on the user's actual viewing direction, but is determined arbitrarily and particularly randomly. Then, the training volume data T-VD is mapped to the visualization pixels VP of the first visualization image T-VB-1 by means of an image synthesis algorithm, more precisely, such that the computational cost for calculating the visualization pixels VP outside the focal region FB is less than the computational cost for calculating the visualization pixels VP inside the focal region FB. Here, in particular, a combination of... Figure 2The method steps are described. Thus, a "simulated" first visualization image T-VB-1 is obtained, which exhibits a quality degradation from the focal region FB towards the peripheral region of the first visualization image T-VB-1. To improve the data basis used for training the trained function TF, on the one hand, multiple different training volume datasets T-VD can be used. On the other hand, each training volume dataset T-VD can define multiple focal regions VB with different settings. Furthermore, the computational cost measures within the image synthesis algorithm for setting each visualization pixel VP can be varied for different first visualization images T-VB-1. Thus, the first visualization image T-VB-1 exhibits quality loss, artifacts, and possible discontinuities, as shown in its reference... Figures 2 to 6 The methods described may also occur in practice. Providing the first visualization image T-VB-1 to the processor 210 can be achieved by retrieving a pre-generated first visualization image T-VB-1 from the database 250 via interface 220. Alternatively, the first visualization image T-VB-1 may be generated by the processor 210 specifically based on the training volume data T-VD. In this case, the training volume data T-VD may be provided to the processor 210 via interface 220.

[0140] Step T30 aims to provide training output data. Here, the training output data is also a visualization image VB, which is hereinafter referred to as the second visualization image T-VB-2. Here, the second visualization image T-VB-2 is the target image, which instructs the trained function TF in which direction the first visualization image T-VB-1 should be improved as much as possible. Here, the second visualization image T-VB-2 is associated with each of the first visualization images T-VB-1. The second visualization image T-VB-2 is also generated by an image synthesis algorithm based on the training volume data T-VD. However, the image synthesis algorithm is not limited in terms of computational cost. More precisely, the second visualization image T-VB-2 is preferably generated by the image synthesis algorithm with a computational cost that is substantially the same (high) for all visualization pixels VP. The second visualization image T-VB-2 is generated with a higher total computational cost than the first visualization image T-VB-1. Providing the second visualization image T-VB-2 to the processor 210 can be achieved by calling the pre-generated second visualization image T-VB-2 from the database 250 via interface 220. Alternatively, the second visualization image T-VB-2 can be generated by the processor 210 specifically based on the training volume data T-VD. In this case, the training volume data T-VD can be provided to the processor 210 via interface 220.

[0141] In the next step T40, the training input data, namely the first visualization image T-VB-1, is input into the trained function. Based on this, the trained function TF calculates the improved visualization image VVB, which should correct the effects of the image synthesis algorithm as much as possible.

[0142] In the next step T50, the improved visualization image VVB is compared with the associated second visualization image T-VB-2. Then, based on this comparison, the trained function TF can be adjusted in step T60. This can be achieved, for example, based on a cost function that penalizes the deviation between the improved visualization image VVB and the second visualization image T-VB-2. One or more parameters of the trained function TF can then be adjusted to minimize the cost function, for example, by means of back propagation. In one embodiment, the cost function can be based on the pixel-wise difference between the improved visualization image VVB and the second visualization image T-VB-2, for example, based on the sum of squared deviations. To minimize the cost function, comparisons are performed on different pairs of groups consisting of the improved visualization image VVB and the second visualization image T-VB-2 until a local minimum of the cost function is reached and the trained function TF works satisfactorily.

[0143] Even if not explicitly stated, various embodiments, sub-aspects of embodiments, or features thereof may be combined or interchanged with each other meaningfully and in the sense of the invention, without departing from the scope of the invention. Unless explicitly stated otherwise, the advantages described with reference to the embodiments of the invention also apply to other embodiments if applicable.

[0144] The following points are also part of the publicly available information:

[0145] 1. A computer-implemented method for visualizing a three-dimensional object represented by volumetric data (VD) to a user in the form of a two-dimensional visualization image (VB) composed of a plurality of visualization pixels (VP), wherein the method comprises the following steps:

[0146] Provide (S10) an image synthesis algorithm configured to visualize a three-dimensional object by mapping volume data (VD) onto visualization pixels (VP);

[0147] Detect (S20) the user's current viewing direction;

[0148] Determine (S30) the focal region (FB), wherein the focal region (FB) is defined as the area of ​​the visualized image (VB) that the user focuses on based on the current viewing direction;

[0149] The image synthesis algorithm is adjusted (S40) such that the computational cost required for mapping visual pixels (VP) outside the focal region (FB) is less than the computational cost required for mapping visual pixels (VP) within the focal region (FB); and

[0150] The volume data (VD) is mapped (S50) onto the visualization pixels (VP) by means of the adjusted image synthesis algorithm, so as to produce a visualization image (VB).

[0151] 2. According to method 1, the following steps are also included:

[0152] Display (S80) a visual image (VB) in the user interface (40).

[0153] 3. Based on any of the methods mentioned above, where:

[0154] The steps of detection (S20), determination (S30), adjustment (S40), mapping (S50), and optional display (S80) are executed multiple times in a loop.

[0155] 4. Based on any of the methods mentioned above, where:

[0156] Image synthesis algorithms include ray casting and / or path tracing methods; and

[0157] Adjusting the image synthesis algorithm involves adjusting the sampling rate of volumetric data (VD) along rays simulated according to ray projection and / or path tracing methods.

[0158] 5. Based on any of the methods mentioned above, where:

[0159] Multiple different visualization modules are implemented in the image synthesis algorithm, and each of the different visualization modules considers different mapping effects when mapping volumetric data (VD);

[0160] Image synthesis algorithms, especially those using ray casting methods; and

[0161] In the adjustment (S40) step, different visualization modules are selectively adjusted and / or selectively activated and / or disabled.

[0162] 6. According to the method in 5, where:

[0163] Assign priorities to visualization modules; and

[0164] In the adjustment (S40) step, selective adjustment and / or activation and / or disabling are performed based on priority.

[0165] 7. Based on method 5 or 6, the mapping effect is selected from:

[0166] Considerations include ambient occlusion, shadows, light transmittance, color bleeding, surface shading, complex camera effects, and / or arbitrary ambient lighting conditions.

[0167] 8. Based on any of the methods mentioned above, where:

[0168] The adjustment (S40) step includes adjusting the gradient quality during shading calculations.

[0169] 9. Based on any of the methods mentioned above, where:

[0170] Image synthesis algorithms include ray casting methods; and

[0171] The adjustment steps include:

[0172] Adjust the number of environmental occlusion test rays at each sampling point as determined by the light projection method; and / or

[0173] Adjust the number of shadow rays at each sampling point determined based on the light projection method.

[0174] 10. Based on any of the methods mentioned above, where:

[0175] Image synthesis algorithms include path tracking methods; and

[0176] The adjustment steps include:

[0177] Adjust the number of scattering events per ray simulated using the path tracing method; and / or

[0178] Adjust the number of rays sampled for each visual pixel (VP) based on the path tracing method.

[0179] 11. A computer-implemented method for visualizing a three-dimensional object represented by volumetric data (VD) for a user in the form of a two-dimensional visualization image (VB), wherein the method comprises the following steps:

[0180] Provide (S10) an image synthesis algorithm configured to visualize a three-dimensional object by mapping volume data (VD) onto a visualization image (VB);

[0181] Detect (S20) the user's current viewing direction;

[0182] Determine (S30) the focal region (FB), wherein the focal region (FB) is defined as the area of ​​the visualized image (VB) that the user focuses on based on the current viewing direction;

[0183] The image synthesis algorithm is adjusted (S40) so that the image quality of the visualization image (VB) outside the focal region (FB) is lower than the image quality within the focal region (FB); and

[0184] The volume data (VD) is mapped (S50) onto the visualized pixels (VP) using the adjusted image synthesis algorithm.

[0185] 12. According to the method in 11, where:

[0186] In the adjustment (S40) step, the image synthesis algorithm is adjusted such that the image quality decreases as the distance from the focal region (FB) increases, or more precisely, decreases at least partially continuously and or discontinuously.

[0187] 13. A computer-implemented method for providing a trained function (TF) for improving a two-dimensional visualization image (VB) composed of a plurality of visualization pixels (VP), the visualization image (VB) mapping a three-dimensional object represented by volume data (VD);

[0188] The method comprises the following steps:

[0189] Provide a first visualization image (T-VB-1) of a three-dimensional object (T20), the first visualization image (T-VB-1) being generated by an image synthesis algorithm by mapping volume data (VD) onto visualization pixels (VP) with a first total computational cost;

[0190] Provide a second visualization image (T-VB-2) of a three-dimensional object (T30), the second visualization image (T-VB-2) being generated by an image synthesis algorithm by mapping volume data (VD) onto visualization pixels (VP) at a second total computational cost, wherein the first total computational cost is less than the second total computational cost;

[0191] An improved visualization (VBB) is generated by applying the trained function (TF) to the first visualization image (T-VB-1);

[0192] The improved visualization (VBB) is compared with the second visualization (T-VB-2) (T50); and

[0193] The trained function (TF) is adjusted based on comparison (T60).

[0194] 14. According to the method of 13, the step of providing (T20) the first visualization image (T-VB-1) includes:

[0195] Determine the focal region (FB) of the first visualization image (T-VB-1); and map volume data (VD) onto the visualization pixels (VP) of the first visualization image (T-VB-1) using an image synthesis algorithm, such that the computational cost for calculating the visualization pixels (VP) outside the focal region (FB) is less than the computational cost for calculating the visualization pixels within the focal region (FB).

[0196] 15. According to the method of 14, the step of providing the second visualization image includes:

[0197] By using an image synthesis algorithm, volume data (VD) is mapped onto the visualization pixels (VP) of the second visualization image (T-VB-2), such that all visualization pixels (VP) of the second visualization image (T-VB-2) are calculated at least with the computational cost of the visualization pixels (VP) within the focal region (FB) of the first visualization image (T-VB-1).

Claims

1. A computer-implemented method for visualizing a three-dimensional object represented by volumetric data for a user in the form of a two-dimensional visualization image having a plurality of visual pixels, wherein the method comprises the following steps: (S10) Provide an image synthesis algorithm configured to visualize the three-dimensional object by mapping the volume data onto the visualization pixels; Detect (S20) the user’s current viewing direction, wherein the user’s eye position or pupil position or head position is detected and evaluated; Determine (S30) the focus area, wherein the focus area is defined as the area of ​​the visualization image that the user is focusing on based on the current viewing direction; The image synthesis algorithm is adjusted (S40) so that the computational cost required for mapping the visualization pixels outside the focal region is less than the computational cost required for mapping the visualization pixels within the focal region. as well as The volume data is mapped (S50) onto the visualized pixels using the adjusted image synthesis algorithm. The visualized image has a uniform spatial resolution for the visualized pixels. The image synthesis algorithm described therein has a first mapping algorithm and a second mapping algorithm that is different from the first mapping algorithm; In the adjustment (S40) step, the first mapping algorithm is pre-registered for mapping the volume data to visual pixels within the focal region, and the second mapping algorithm is pre-registered for mapping the volume data to visual pixels outside the focal region. The first mapping algorithm implements a path tracking method; and the second mapping algorithm implements a ray projection method.

2. The method according to claim 1, wherein: In the adjustment (S40) step, the image synthesis algorithm is adjusted such that the computational cost decreases as the distance from the focal region increases.

3. The method according to claim 2, wherein: The computational cost decreases at least partially and continuously as the distance from the focal region increases, and / or The computational cost decreases at least partially and discontinuously as the distance from the focal region increases.

4. The method according to any one of claims 1 to 3, wherein: The image synthesis algorithm is based on the tracking of simulated rays in the object volume expanded by the volume data; as well as Adjusting the image synthesis algorithm includes adjusting the rate at which the object volume is sampled along the simulated ray.

5. The method according to any one of claims 1 to 3, wherein: The image synthesis algorithm implements multiple different visualization modules, each of which considers different mapping effects when mapping the volume data. as well as In the adjustment (S40) step, the different visualization modules are selectively adjusted and / or selectively activated and / or disabled.

6. The method according to claim 5, wherein: Assign a priority to the visualization module; as well as In the adjustment (S40) step, the selective adjustment and / or activation and / or disabling is performed based on the priority.

7. The method according to any one of claims 1 to 3, wherein: The adjustment (S40) step includes adjusting the coloring quality.

8. The method according to any one of claims 1 to 3, further comprising the following steps: The resolution of the volume data is adapted such that, in the mapping (S50) step, the resolution of the volume data for visualization pixels outside the focal region is lower than the resolution of the volume data for visualization pixels within the focal region.

9. The method according to any one of claims 1 to 3, wherein: The adjustment (S40) step further includes determining the time budget and / or available computing power for creating the visualization image by mapping the volume data; and The adjustment (S40) is performed by the image synthesis algorithm based on the time budget and / or the available computing power.

10. The method according to any one of claims 1 to 3, further comprising the following steps: An improved visualization image is generated (S60) by applying a trained function to the visualization image, wherein the trained function is designed to improve the image quality of the visualization image.

11. The method according to claim 10, The trained function is designed such that it improves the image quality of the visualized image by correcting the effect of the adjustment (S40) of the image synthesis algorithm.

12. The method according to any one of claims 1 to 3, further comprising the following steps: The filtering algorithm is applied to the visualized image.

13. The method according to claim 12, The filtering algorithm described herein includes Gaussian filter elements, bilinear filter elements, and / or low-pass filter elements.

14. A visualization system (1) for visualizing three-dimensional objects represented by volume data, wherein: It visualizes for users in the form of a two-dimensional image composed of multiple visualized pixels; The visualization system (1) has an interface (30), a detection device and a computing unit (20); The interface (30) is configured to receive the volume data and detection information, the detection information including information about the user's current viewing direction; The detection device is designed to detect the user's viewing direction, wherein the detection information of the detection device includes image data of the user's eyes, pupil position, and / or head position; and The computing unit (20) is also configured to: The focus region is determined based on the detection information, wherein the focus region is defined as the area of ​​the visualization image that the user is focusing on; The volume data is mapped onto the visualization pixels to create the visualization image, and the mapping is adjusted such that the computational cost for visualization pixels outside the focal region is less than the computational cost for visualization pixels within the focal region. The visualized image has a uniform spatial resolution for the visualized pixels. An image synthesis algorithm is loaded into the computing unit. This image synthesis algorithm is configured to visualize the three-dimensional object by mapping the volume data onto the visualization pixels. The image synthesis algorithm described therein has a first mapping algorithm and a second mapping algorithm that is different from the first mapping algorithm; The first mapping algorithm is pre-registered for mapping the volume data to visual pixels within the focal region, and the second mapping algorithm is pre-registered for mapping the volume data to visual pixels outside the focal region. The first mapping algorithm implements a path tracking method; and the second mapping algorithm implements a ray projection method.

15. A computer program product having a computer program that can be directly loaded into the memory of a visualization system (1), the computer program product having a program segment that, when executed by the visualization system (1), performs all the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program segment readable and executable by a visualization system (1) such that, when the program segment is executed by the visualization system (1), all steps of the method according to any one of claims 1 to 13 are performed.

Images