System and method for multi-volume rendering

Abstract

Systems (100), methods, and computer program products for direct multi-volume rendering. The 3D data specifies a three-dimensional scene having a plurality of volumes (290). Each voxel of the specific volume has a scalar value that is a result of an imaging process for measuring a physical property of an imaged object. A multi-ray generator (230) generates a view multi-ray in the direction of a particular pixel of a projected image, wherein the particular pixel represents a 2D projection of all scene voxels that intersect the view multi-ray behind the projected image. A volumetric ray travel processor (240) processes each view ray in its associated volume using volumetric ray travel, where, in each ray travel step, scalar values for a corresponding voxel are mapped to color values and transparency values for the corresponding voxel. A projection image updater (250) updates the value of a particular pixel in the projection image by combining the individual voxel colors and transparency values of the individual view rays. After updating with all intersecting voxel values of voxels closer to the view point than the specific voxel, updating with a value of a specific voxel intersecting with the specific view ray is performed.

Description

Technical Field

[0001] The present invention relates generally to rendering images showing three-dimensional scenes, and more particularly to volume rendering having multiple volumes. Background Art

[0002] In many software applications, such as medical applications, computer graphics algorithms are used to generate 3D representations of a patient's body parts. Furthermore, in computer games executed in a virtual 3D scene, or in other 3D applications, the three-dimensional scene needs to be rendered as an image and shown to a human user of the application (multiple images). The rendering of such a three-dimensional scene, particularly in medical applications, is typically performed using an image-based volume rendering technique, which is often referred to as volumetric ray marching. Ray marching uses volumetric data. Other techniques, such as ray tracing and ray casting, calculate the intersection of rays with a surface model of a 3D object. Ray marching does not use such a surface model of the 3D object. It only knows the cuboid bounding box of the volume and cannot directly calculate the intersection of the view ray with the surface of the 3D object within the volume. To do this, ray marching performs small ray marching steps and samples the corresponding voxels.

[0003] In its basic form, the volumetric ray marching algorithm consists of four steps:

[0004] -Ray casting: For each pixel of the final image, a line of sight (view ray) is sent through the volume to be drawn. Typically, the volume corresponds to a simple geometry (such as a cuboid).

[0005] Sampling: Selects equally spaced sampling points or samples along the portion of the view ray that lies within the volume. Typically, the volume is not necessarily aligned with the view ray. Therefore, the sampling points do not necessarily coincide with the center of a voxel. Because of this, the sample values ​​can be interpolated from neighboring voxels.

[0006] - Shadows: For each sample point, the sample value is mapped to RGBA material color, opacity / transparency, and optionally other properties.

[0007] - Combine: Combine all sample points along the view ray to get the final color value of the pixel currently being processed.

[0008] The fundamental paper disclosing the concept of ray marching is: Levoy, Marc. "Display of surfaces from volume data." IEEE Computer graphics and Applications 8.3 (1988): 29-37. Using existing technical solutions, it is a challenge to efficiently perform direct multi-volume rendering, especially for three-dimensional scenes with overlapping volumes. Summary of the Invention

[0009] Therefore, there is a need to provide an efficient solution for direct multi-volume rendering of three-dimensional scenes. Direct rendering as used herein means rendering images directly from raw data. This term will be further explained in the context of different raw data types.

[0010] The above problems are solved by embodiments as claimed in the independent claims: a computer-implemented method for direct multi-volume rendering; a computer program product having instructions, which instructions are configured to cause the performance of the method when the computer program method is executed; and a computer system, which can load and execute the computer program product to perform the method.

[0011] A computer system for direct multi-volume rendering is hereinafter referred to as an MVR system. The MVR system includes functional modules that can perform the functions for executing the various steps of the method disclosed herein. The MVR system receives 3D data specifying a three-dimensional scene via an appropriate interface. "3D" here means three-dimensional, and "2D" means two-dimensional. As used herein, a 3D scene includes multiple volumes to be rendered by the MVR system. A typical 3D dataset is a series of 2D slice images acquired, for example, by computed tomography (CT), magnetic resonance imaging (MRI), microCT, or other 3D imaging techniques. The 3D data received by the MVR system corresponds to the raw data mentioned above. This discretely sampled 3D dataset is typically referred to as a volume. The 3D scene includes multiple volumes. Therefore, each volume includes multiple voxels. Each voxel in a particular volume has a scalar value that is the result of a volumetric imaging process used to measure a physical property of the imaged object. In this context, the imaged object is a body undergoing a volumetric imaging process. Examples of imaging objects include biological samples (e.g., samples from humans, animals, or plants) or non-biological samples (e.g., machine or device components). In other words, the raw data for direct mapping is a collection of voxels in a volume, where each voxel has such a scalar value. Examples of such imaging processes (imaging techniques) include, but are not limited to: computed tomography imaging, magnetic resonance imaging, optical coherence tomography imaging, positron emission tomography imaging, single photon emission computed tomography, and 3D ultrasound imaging, or any image acquisition or generation process that produces a volumetric image. Using appropriate imaging processes, physical properties such as absorption of X-ray radiation, proton number, light transmittance, positron emission from an injected radiopharmaceutical, changes in acoustic impedance, or other physical body properties can be measured. Those skilled in the art know which imaging techniques can be used to measure the corresponding physical properties of the imaging object.

[0012] The MVR system includes a multi-ray generator module that generates view multi-rays at a viewing position in the direction of a specific pixel of a projected image. The viewing position typically refers to the position of a virtual camera or virtual reality device used by a user to observe a 3D scene. A projected image is a 2D projection of the 3D scene, onto which the 3D scene is projected as seen from the viewing projection. A projected image is the image displayed on a display device such as a monitor. When a virtual reality device is used to display the image, two projected images are generated suitable for creating a 3D VR effect. The projected images are located between the viewing position and the scene. Typically, the projected images are located at a viewing distance of the order of 10 cm in front of the viewing position. Thus, a specific pixel (through which a view multi-ray is sent into the 3D scene) represents the 2D projection of all scene voxels intersected by the view multi-ray behind the projected image. A view multi-ray represents multiple individual view rays with the same starting point and direction, each associated with one of the volumes in the scene. In other words, each individual view ray (hereinafter also referred to as a view ray) belongs to exactly one of the volumes loaded into the scene.

[0013] The MVR system also has a volumetric ray marching processor module that uses volumetric ray marching to individually process each view ray in its associated volume. Individually processing a view ray in its associated volume includes determining the color and transparency values ​​in the associated volume at the corresponding ray marching step. That is, ray marching steps for a corresponding view ray are performed only within the associated volume. Therefore, for a particular view ray, the number of ray marching steps of a given step size (i.e., the distance between two ray marching steps) is limited by the boundaries of the associated volume. For example, the first ray marching step for a particular view ray may occur at a multiple of the step size at which the particular view ray first interacts with the associated volume. In the current ray marching step, the view ray interacts with its associated volume, either when the view ray first intersects the associated volume or when a corresponding position outside the associated volume is already associated with a visible color value. In some cases, due to voxel interpolation, even marched ray positions outside the volume—but very close to it—may already result in visible color values. At each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of that corresponding voxel. In one embodiment, such mapping can be implemented using predefined transfer functions. There can be different transfer functions for each volume (or even for different sub-regions within a particular volume), depending on the imaging technology used to create the corresponding relevant volume. For example, in the case of a CT scan, the scalar values ​​are measured in Hounsfield units (HU). The corresponding transfer function can map HU values ​​corresponding to air (black) and soft tissue (gray) to transparent, and map HU values ​​corresponding to bone (bright) to white / beige. In general, the scalar values ​​can be interpreted as grayscale values, which are mapped to color values ​​and transparency values ​​(or corresponding opacity values) according to the transfer function. An alternative color mapping method is disclosed in the detailed description.

[0014] The projection image updater module of the MVR system updates the value of a specific pixel in the projection image (the pixel intersected by the current view multi-ray) by combining the mapped voxel colors and transparency values ​​across individual view rays. Thus, updating the value of a specific voxel intersecting a specific view multi-ray is performed after updating with the values ​​of all intersecting voxels closer to the view point than the specific voxel. In other words, combining the color / transparency values ​​of voxels with a greater distance to the viewing position must wait until combining the color / transparency values ​​of voxels with a shorter distance is complete. As a result, view rays associated with more distant volumes can only be processed after processing of view rays associated with more distant volumes is complete. Updating pixel values ​​can be implemented such that the combined value after each ray marching step is stored in a corresponding local variable of the system, and the value of this variable is ultimately written to the specific pixel in the projection image once ray marching is complete for the corresponding view multi-ray.

[0015] The MVR system performs the generating, processing, and updating steps for pixels of the projected image that have not yet been updated. For example, the described functionality can be repeated in a loop for all pixels of the projected image. Because the processing of the first view multi-ray is independent of the processing of the second view multi-ray, multiple view multi-rays can also be processed in parallel. For example, in practical applications of the methods disclosed herein, hundreds or even thousands of view rays are typically processed in parallel by a graphics processing unit (GPU). Parallelization techniques are known in the art to distribute processing to multiple cores, pipeline processing, etc.

[0016] In one embodiment, volumetric ray marching can use empty space skipping because the individual processing of a particular view ray is accelerated by skipping as many ray marching steps as possible associated with fully transparent voxels inside the associated volume. The following paper discloses an empty space skipping method using distance mapping:

[0017] -Zuiderveld,Karel J.,Anton HJ Koning,and Max A.Viergever."Acceleration of ray-casting using 3-D distance transforms."Visualization inBiomedical Computing'92.Vol.1808.SPIE,1992.

[0018] -Sramek, Milos. "Fast surface rendering from raster data by voxeltraversal using chessboard distance." Proceedings Visualization'94. IEEE, 1994.

[0019] In one embodiment, the MVR system may also include a volume registration module to register one or more of the multiple volumes relative to each other. Registering a first volume relative to a second volume means that the origin of the positioning system of the first volume is known in the second volume (and vice versa). Once all volumes are properly registered, the volumes display the same anatomical points in the same position in the global coordinate system. This allows the positions of two or more volumes to be fixed relative to each other. For example, the position of a volume showing an MR scan of a patient's brain can be fixed relative to a volume showing a CT scan of the patient's skull, so that the visual representation makes anatomical sense.

[0020] To perform ray marching, it may be advantageous to transform each view ray from the global coordinate system (i.e., the coordinate system associated with the 3D scene) to the local coordinate system of its associated volume (i.e., the coordinate system of the associated volume, which is translated, rotated, and scaled relative to the global coordinate system) at the beginning of the corresponding ray marching process. Using this transformation allows the ray marching steps in each volume in each local coordinate system to be processed simply based on the number of steps performed, without the need to calculate the individual distances of each volume from the viewing position in the global coordinate system. As mentioned, this transformation can be performed once at the beginning of the corresponding ray marching process and does not need to be repeated in each ray marching step.

[0021] In one embodiment, a computer program product for direct multi-volume rendering is provided. The computer program product comprises computer-readable instructions that, when loaded into a memory of a computing device (e.g., an MVR system) and executed by at least one processor of the computing device, cause the computing device to perform the computer-implemented methods disclosed herein. Thus, the functional modules of the MVR system can be implemented as software modules that implement the functions, such that corresponding method steps are executed when the MVR system is run.

[0022] Other aspects of the invention will be realized and attained by means of the elements and combinations particularly described in the appended claims.It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention described. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1Ashows view multi-rays for rendering multiple volumes of a 3D scene in a projected image according to an embodiment;

[0024] FIG1B shows voxels of a volume with a local coordinate system;

[0025] Figure 2 is a simplified block diagram of an MVR system according to an embodiment;

[0026] Figure 3 is a simplified flow chart illustrating a computer-implemented method for direct multi-volume rendering according to an embodiment;

[0027] Figures 4A to 4D Examples of transfer functions for mapping scalar voxel values ​​to color and transparency values ​​are shown;

[0028] Figure 5 An example with pixels in a projected image and their respective view multi-rays is shown;

[0029] Figure 6 An example of ray travel in a volume including scene objects is shown;

[0030] Figure 7 shows an example of using distance maps for empty space skipping; and

[0031] Figure 8 is a diagram illustrating examples of a general-purpose computer device and a general-purpose mobile computer device that can be used in embodiments of the present invention. DETAILED DESCRIPTION

[0032] Figure 1A A view multi-ray 20 is shown for directly rendering multiple volumes 110, 120, 130, 140 of a 3D scene in a projected image 30 according to an embodiment. The figure illustrates the conceptual approach behind the solution disclosed herein. Figure 2 and Figure 3 The method is described in the context of . Therefore, Figure 2 is suitable for implementation Figure 1A A simplified block diagram of a computer system 200 (MVR system) for performing the MVR method is shown, and Figure 3 A computer-implemented method 1000 is described that may be performed at runtime by the MVR system 200. To this end, the following portion of the description uses reference numerals that refer to these figures.

[0033] The MVR system 200 receives 1100 3D data specifying a three-dimensional scene 100 via interface 210. Those skilled in the art will appreciate suitable interfaces for loading multiple volumes 290 into the scene 100. The 3D data specifying the respective volumes may be stored in a remote data store (e.g., in a database on a remote device communicatively coupled to the system 200). Optionally, the system 200 may include a memory suitable for storing the 3D data for the volumes 290. As previously described, volumes are composed of so-called voxels. FIG1B illustrates some voxels pv_n of a volume Vn. The voxels of a volume have a rectangular parallelepiped shape. Of course, in the complete volume Vn, each voxel layer has the same number of voxels, typically arranged in a rectangular shape that results in the volume's rectangular parallelepiped shape. Voxels may have a cubic shape, but typically, a volume Vn is typically composed of layers of a 2D image, where each pixel of the 2D image is square in shape. In this example, it is assumed that the 2D image layers lie in the xn-yn plane of Vn's local coordinate system CSn. When observing the dotted voxels, it can be seen that the depth of the voxel in the zn dimension is different from the side (edge) length of the voxel in the xn-zn dimension. The distance over which the 2D image layer is captured (e.g., by any of the previously mentioned volumetric imaging techniques) corresponds to the depth of the voxel in the zn dimension. In general, the depth of a voxel is different from the edge length of the corresponding pixel in the underlying 2D image layer.

[0034] exist Figure 1A In the example of FIG, the MVR system 200 has loaded four volumes 110 (V1), 120 (V2), 130 (V3), and 140 (V4) in the scene 100. For simplicity, the volumes are illustrated as two-dimensional rectangles. However, in a virtual 3D scene, the volumes are of course three-dimensional cuboids. Figure 1AThe legend shows different row styles used to distinguish rows associated with the corresponding volume from rows associated with other volumes. For example, a dashed line style is used for all rows associated with V1. Each volume has its own local coordinate system. For simplicity, only the local coordinate system xIl-yIl-zIl of V1 is shown in the figure. However, V2, V3, and V4 also have their own local coordinate systems, which can generally be translated, rotated, and scaled relative to the other local coordinate systems. Those skilled in the art will understand how to use matrix multiplication with a transformation matrix for transforming from one coordinate system to another. Each voxel in a particular volume has a scalar value, which is the result of the imaging process used to measure a physical property of the imaged object. In this example, the voxels with scalar values ​​are located within the boundaries of the scene objects 111, 121, 131, and 141, and the scalar value indicates a scene object within the corresponding volume. Similar to the volumes, these three-dimensional scene objects are also illustrated as two-dimensional objects to simplify the graphical representation in the figure. Scene objects are defined by the visible voxels in the corresponding volume. Visual objects can be semi-transparent or opaque voxels, which will be drawn with the corresponding color value. Figure 1A In the example of , scene objects may represent tissues and / or organs of a patient, which are identified by respective imaging techniques by respective scalar voxel values ​​in their respective volumes.

[0035] In an alternative embodiment, the MVR system 200 has a volume registration module 220 adapted to register volumes 290 (eg, V1 to V4) relative to one another, as has been previously described.

[0036] The multi-ray generator 230 of the MVR system 200 generates 1200 view multi-rays 20 at the viewing position 10 as a starting point. In Figure 1, the left edge of the camera / eye symbol defines the viewing position. The view multi-rays 20 are generated in the direction of a specific pixel of the projected image 30 relative to the viewing position 10. The projected image 30 is at a viewing distance d from the viewing position 10 and is located between the viewing position 10 and the scene 100. Ultimately, the specific pixel represents the 2D projection of all scene voxels intersected by the view multi-ray behind the projected image 30. In other words, the view multi-ray 20 intersecting the specific pixel is used to sample the voxel values ​​of the 3D scene in the direction of the multi-ray, ultimately calculating the combined value reflected in the color value of the specific pixel in the projected image. Thus, the view multi-ray 20 represents multiple individual view rays 21, 22, 23, and 24 with the same starting point and direction. Each individual view ray is associated with one of the volumes in the scene. In the example, view ray 21 is associated with V1 110, view ray 22 is associated with V2 120, view ray 23 is associated with V3 130, and view ray 24 is associated with V4 140. For clarity and better visibility in the drawings, view rays 21 to 24 are shown as parallel arrows. However, they are identical in starting point and direction. That is, their positions in the scene are all the same. However, individual view rays can be distinguished by their unique assignment to one of the volumes. The starting point and direction of multi-ray 20 (and thus the starting point and direction of the individual view rays) are defined by the multi-ray generator in the global coordinate system xg-yg-zg.

[0037] Briefly turn to Figure 5 , the projected image 30 is shown as having three additional specific pixels 30-1, 30-2, 30-3, through which the view multi-rays 20-1, 20-2 and 20-3 generated respectively are sent into the 3D scene for sampling the volume voxels in the scene. In the example of the figure, the volumes are not shown. Instead, the 3D scene object 121-1 included in one of the volumes is shown (as an ellipsoid). In the example, only the multi-rays 20-1 and 20-2 intersect the voxels associated with the scene objects. In addition, a separate view ray of each multi-ray 20-1, 20-2 is associated with the volume that includes the voxels of the scene object 121-1. The final projected image 30 includes the 2D projection values ​​of all pixels of the image 30.

[0038] The volume ray marching processor 240 of the MVR system 200 uses volume ray marching to individually process each view ray 21, 22, 23, 24 in its associated volume. Advantageously, ray marching can be performed in the local coordinate system of each volume for performance optimization. This saves computationally expensive matrix multiplications after each ray marching step. The ray marching step for the corresponding view ray is performed only within the associated volume to sample the corresponding volume voxels. In particular, in the case of overlapping volumes (e.g., V1 overlaps with V2, and V1 overlaps with V3), the corresponding view ray performs sampling only within its associated volume. In one implementation, the first ray marching step for a particular view ray can occur at a multiple of the ray marching step length at which the particular view ray first intersects the associated volume. In the example of FIG1 , the first ray marching step for view ray 21 is indicated by the left ray marching point on view ray 21. The first ray marching step for view ray 22 is indicated by the left ray marching point on view ray 22, and so on. It should be noted that in regions with overlapping volumes, the ray-marching steps for each individual view ray are synchronized because they are performed at the same distance from the viewing position 10. In the example, the first ray-marching step for view ray 22 in V2 occurs at the same distance from the viewing position as the sixth view-marching step for view ray 21 in V1. The second ray-marching step for view ray 22 is the same distance from the viewing position as the seventh view-marching step for view ray 21, and so on. Similarly, the first ray-marching step for view ray 23 is the same distance from the viewing position as the sixteenth view-marching step for view ray 21, and so on.

[0039] In one implementation, at each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel via a predefined transfer function. Optionally, the transfer function can also map the potentially interpolated scalar value sampled at the corresponding location to other parameters, such as photometric or reflectivity properties. Ray marching processor 240 can provide one or more transfer functions TF 241 for mapping in the corresponding volume. In this example, view ray 21 is used to process only voxels of V1. A first transfer function TF1 can be used for voxel value mapping in V1. View ray 22 is used to process only voxels of V2. A second transfer function TF2 can be used for voxel value mapping in V2. That is, in overlapping volumetric regions, mapped voxel values ​​of different volumes at the same distance from the viewing position typically have different mapped voxel values, and individual view rays can calculate such volume-specific mapped voxel values ​​for their associated volumes using the corresponding transfer function at each ray marching step.

[0040] In an alternative implementation, at each ray marching step, the scalar value of the corresponding voxel (or interpolated position in the volume) is mapped to a transparency value only via a predefined transfer function and the color value is determined by applying an illumination model or a combination of different illumination models. Thereby, the system determines the color and intensity of light reflected by the surfaces of scene objects in the direction of the viewing position by taking into account different light sources and the material properties of the surfaces. Details can be found in the Wikipedia article "Bidirectional reflectance distribution function", which is available at https: / / en.wikipedia.org / wiki / Bidirectional_reflectance_distribution_function or in Montes, Rosana, and Carlos. An overview of BRDF models can be found in the paper "An overview of BRDF models." University of Grenada, Technical Report LSI-2012-001 (2012).

[0041] The properties to be considered depend on the chosen illumination model(s). A classic illumination model is the Blinn-Phong reflection model (see Blinn, James F. "Models of light reflection for computer synthesized pictures." Proceedings of the 4th annual conference on Computergraphics and interactive techniques 1977). Another commonly used illumination model is the Lambert model (see Lambert, Johann Heinrich. Photometria sive de mensura et gradibus luminis, colorum et umbrae. sumptibus vidvae E. Klett, typis CP Detleffsen, 1760).

[0042] The result of this alternative implementation is that the voxel color value is determined by the position and color of the light source and the orientation of the scene object surface relative to the light source. Surfaces facing the light source are colored by the light source, while the back surfaces remain dark. For example, for relatively simple CT scans, objects can be clearly identified.

[0043] In practical implementations, the transfer function approach can be combined with the illumination model approach, leading to an improved spatial perception of an observer of the projected image.

[0044] Now by using Figures 4A to 4D The examples in describe the concept of transfer functions in more detail. Figure 4A In the CT volume, the raw CT slices (images) 410 provided by CT scanning technology have grayscale pixel values. As a result, the voxels of the CT volume interpreted by combining multiple layers of raw CT slices also have corresponding grayscale scalar values ​​(raw data). In the CT volume, the brightness of the voxels corresponds to the x-ray absorption of the tissue at various body locations, with units of Hounsfield (HU). To produce a true 3D rendering of the volume, a transfer function 400a is defined that maps grayscale HU values ​​401 to color and transparency (or opacity) values. In the CT example, the transfer function 400a maps HU values ​​corresponding to air (black) and soft tissue (gray) to a transparency value 402 of "transparent", and maps HU values ​​corresponding to bone (bright) to a color value 403 of "white" (or alternatively, to "beige"). The direct rendering result 420 based on the raw voxel values ​​and the transfer function 400a is illustrated as a portion of a human skeleton.

[0045] Figure 4D A transfer function 400d (similar to transfer function 400a) is used to generate a direct rendering result 450 from a corresponding CT scan raw data volume. Figure 4B and 4C Direct rendering results 430 and 440 generated from the same CT scan raw data volume using different transfer functions 400b and 400c are shown, respectively. In the figures, the direct rendering results are shown in grayscale, but in reality, transfer functions 400b and 400c map various scalar (grayscale) values ​​to color values ​​corresponding to different shades of red. Defining different transfer functions for mapping voxel scalar values ​​to corresponding transparency / color values ​​allows for different rendering focuses on what physical body properties are to be rendered.

[0046] The projected image updater module 250 of the MVR system updates 1400 the value of a particular pixel in the projected image 30 by combining the corresponding voxel color and transparency values ​​(i.e., the mapped RGBA values ​​of the corresponding voxels) of individual view rays. Thus, updating each projected image pixel value using the value of a particular voxel intersecting a particular view point ray is performed after updating using the values ​​of all intersecting voxels of voxels closer to the view point than the particular voxel. In other words, an individual view ray is allowed to advance rapidly until it encounters an opaque voxel within its associated volume. However, processing of the opaque voxel must be synchronized with processing of all rays at the same location in the 3D scene to achieve a correct result for the particular pixel value. The generating 1200, individually processing 1300, and updating 1400 steps are performed 1500 by the corresponding modules of the MVR system 200 for all pixels of the projected image 30. For example, these steps can be repeated in a loop, with a check 1450 after each iteration to determine whether there are any pixels in the projected image that have not yet been updated. If the projected image is not yet complete, further loop iterations are initiated for the pixels that have not yet been updated. Once the projected image is complete, method 1000 ends. Alternatively, if the MVR system 200 supports corresponding parallel processing, multiple pixels of the projected image can be processed in parallel. As an example, the processing of each pixel can be implemented as a shader program executed by a graphics processing unit. Updating the corresponding projected image pixel value can advantageously be implemented so that the combined value is intermediately stored in a (local) variable during the processing of the projected image pixel. In other words, intermediately storing the combined value in a (local) variable during the processing of the projected image pixel and updating the corresponding projected image pixel only at the end of the processing can improve performance in some cases. Once the processing of the view multi-ray 20 is completed, the final value of the variable is written to the projected image 30 as the corresponding projected image pixel value. Storing the updated projected image pixel value in a local variable during the ray marching process is computationally more efficient than writing the current combined value to the projected image after each ray marching step.

[0047] Figure 6Direct volume rendering is described in more detail by way of example. With direct volume rendering (i.e., computing projection images directly from volumetric data), the MVR system does not know where visible tissue surfaces (e.g., scene objects SO6) are located within the corresponding volume. Therefore, simple ray-surface collision calculations cannot be performed. Direct rendering methods can only determine collisions of view rays with the bounding box BB6 of the corresponding volume (e.g., the CT volume). Therefore, multiple view rays VR6* (one for each pixel of the projection image) are sent from viewing position 10 through the 3D scene. Within the volume defined by bounding box BB6, ray marching is performed with each view ray. The length of a ray march step (i.e., the distance between two ray march steps) is on the order of the size of a voxel. For example, the length of a ray march step can be set to the length of the voxel diagonal or less. Using this small step size along the view ray, raw voxel data is read at each ray march step location (indicated by black and white dots), and a corresponding transfer function can be applied to determine the color and transparency (or, alternatively, opacity, where transparency value = 1 - opacity value) of the corresponding voxel. Typically, voxel color and transparency values ​​are represented in the so-called RGBA format, where the RGB channels represent the color value and the alpha value in the A channel represents the opacity (0% = fully transparent, 100% = fully opaque).

[0048] Typically, close to the bounding box BB6, the voxels are transparent (e.g., air). Therefore, ray marching can simply continue until the system finds a visible voxel. Typically, when a view ray intersects a visible voxel (e.g., bone or soft tissue), the projected image updater begins accumulating the color and transparency of the visible voxel with subsequent visible voxels. In other words, the color and transparency values ​​of the visible voxels along the view ray are combined into the color value of the corresponding pixel in the projected image. This accumulation continues until the opacity channel (alpha value in RGBA) of the current pixel reaches a threshold close to 100%. For example, the opacity threshold can be predefined in the range between 90% and 99.9%. Therefore, anything that the view ray may encounter from this point on will not be visible anyway, because it will be hidden by the previously intersected voxel. That is, the ray marching process can be stopped at this point. This effect is often referred to as early ray termination. In Figure 6 In the example shown, black dots represent ray marching steps where a view ray intersects an invisible voxel. White dots show ray marching steps where a view ray intersects a visible voxel belonging to scene object S06. For each white dot, the above-described accumulation (combining) of mapped voxel values ​​is performed. In the example, all view rays that penetrate scene object S06 continue in the ray marching step behind the object. That is, early ray termination is not active or none of the rays shown have reached the opacity threshold for early ray termination.

[0049] In the following, given that Figure 1A Detailed examples are described without intending to limit the scope of the claims in any way. In particular, when combining mapped voxel values ​​in regions with overlapping volumes, the combination is always performed in an order where the voxels closest to the viewing position are combined first. Advantageously, the individual view rays are processed in lockstep. That is, the view rays (particularly in overlapping volumetric regions) are processed at exactly the same speed and in the same direction.

[0050] Initially, view rays 21 to 24 are initialized with the generation of view multi-ray 20. The MVR system then determines the location of the first ray marching step for each (individual) view ray in each associated volume. In this example, assume that the first ray marching step is:

[0051] - occurs in V1 at 108 steps from the viewing position (position '8'),

[0052] - occurs in V2 at 1013 steps from the viewing position (position '13'),

[0053] - occurs in V3 at 1023 steps from the viewing position (position '23'), and

[0054] - occurs in V4 at 1029 steps from the viewing position (position '39').

[0055] These first ray marching step positions in each volume can be stored as the number of step lengths from the viewing position of the individual view rays. The minimum of these first ray marching step length positions is the viewing multi-ray distance from the viewing position (called the multi-ray position). That is, in this example, the system knows that all view rays have a first ray marching step position that has a distance of at least 8 step lengths from the viewing position. The multi-ray position is then incremented by '1' after each ray marching step is completed. Using this information, the MVR system can decide at any time whether the mapped voxel values ​​on a particular view ray can be combined without risk, or whether the system still has to wait to combine because there is a risk that the mapped voxels closer to the viewing position may be opaque and will hide all voxels at a greater distance from the viewing position, or may have non-transparent color values ​​that must be combined first to achieve the correct final pixel color. Moreover, in the case of semi-transparent pixels, the order in which the voxel values ​​are combined is relevant to achieving the correct pixel color.

[0056] The volume ray marching processor iterates over all view rays 21 to 24 (i.e., over all volumes V1 to V4 to be rendered into the projection image) at each ray marching step. The order of this iteration is irrelevant. Hereinafter, the iteration order V2 → V1 → V4 → V3 is assumed. In the examples described below, where multiple iterations are described under a single heading, it should be understood that, in each iteration, all volumes are processed in the aforementioned iteration order before the system performs the next iteration.

[0057] In an optional embodiment, the MVR system may store a distance map that may be calculated during the loading process of a volume. During this loading process, and given one or more transfer functions, the system may calculate the distance to the nearest visible voxel in each volume for each voxel in the volume and store the calculated distances in the distance map. During volume ray marching, the MVR system's ray marching processor may retrieve the distance to the next visible voxel from the distance map at each location in the corresponding volume to determine whether the next ray marching step can be performed at a multiple of the step size (ray marching jump). Once such a distance is less than or equal to the ray marching step size, the scalar data for the corresponding voxel is read and processed using the corresponding transfer function. The distance map allows transparent voxels in the volume to be skipped during ray marching, thereby speeding up ray marching processing.

[0058] Iteration 1

[0059] V2: View ray 22 begins with the first ray marching step at position '13'. When reading the distance map, the system recognizes that the visible voxel is near and cannot perform a ray marching jump. After reading the scalar voxel value and applying the transfer function, the ray marching processor determines that the voxel is transparent, so the mapped color / transparency value combination is not needed. As a result, view ray 22 moves to the next ray marching step at position '14'.

[0060] V1: For view ray 21, the situation is similar to view ray 22. View ray 22 moves to the next ray marching step at position '9'.

[0061] V4: For view ray 24, at the edge of V4, the system infers from the distance map that all voxels within a radius of two steps are transparent. Therefore, the position of view ray 24 can be safely increased from "39" to "41".

[0062] V3: For view ray 23, the situation is similar to view rays 22 and 21, and the position may be increased by one to position '24'.

[0063] After all view rays have been iterated, the multi-ray position is incremented by one from '8' to '9' for the next iteration. Because the ray marching step positions of all view rays for iteration 2 are already inside the corresponding scene objects after iteration 1, no further ray marching jumps are allowed (as long as the corresponding view rays are inside the scene objects) because the corresponding distance values ​​in the distance map are '0'.

[0064] Iteration 2

[0065] Starting again with V2, at the current position '14' of view ray 22, the ray marching processor reads the first visible voxel of V2 belonging to scene object 121. However, the current position of view ray 22's '14' is further from the view position than the current multi-ray position '9'. Therefore, for the corresponding pixel in the projected image, the corresponding mapped voxel value at the current position '14' must not yet be combined. The view ray 22 at the current position must wait until the multi-ray position also reaches '14'.

[0066] VI: View ray 21 has reached the visible voxel at the current position '9' belonging to scene object 111. The corresponding mapped voxel value is determined using the corresponding transfer function and combined with the corresponding projected image pixel, since there are no other view rays with current positions closer to the view position. The current position of view ray 21 is incremented to position '10'.

[0067] V4: The current position '41' of view ray 24 still intersects a transparent voxel (→ no combination of mapped values). The distance map indicates that no further ray marching jumps are possible. Therefore, the current position is incremented to position '42'.

[0068] V3: View ray 23 has reached the visible voxel at the current position '24' belonging to scene object 131. The corresponding mapped voxel value is determined using the corresponding transfer function. However, similar to view ray 22, the corresponding mapped voxel value at the current position '24' must not yet be combined with the corresponding pixel in the projection image. View ray 23 at the current position must wait until the multi-ray position also reaches '24'. In other words, before the mapped voxel value of view ray 23 can be combined with the projection image pixel value, volumes V1 and V2 closer to the viewing position must be processed up to position '24'.

[0069] After all view rays are processed in the second iteration, the multi-ray position is incremented to '10'.

[0070] Iterations 3, 4, and 5

[0071] V2: At its current position '14', view ray 22 must wait.

[0072] V1: View ray 21 has reached three additional visible voxels at the current position '11', '12', and '13', belonging to scene object 111. A corresponding mapped voxel value is determined at each position using a corresponding transfer function and combined with the corresponding projected image pixel after a corresponding ray marching step, since there are no other viewing rays with a current position closer to the viewing position. The current position of view ray 21 is incremented to position '14' in the fifth iteration. Although position '14' already overlaps with V2 and would allow combining, combining with the V2 voxel value does not occur because the voxel intersected by view ray 22 is transparent.

[0073] V4: At its current position '42', view ray 24 must wait.

[0074] V3: At its current position '24', view ray 23 must wait.

[0075] After all view rays have been processed in the third, fourth, and fifth iterations, the multi-ray position is at '14'.

[0076] Iterations 6 to 11

[0077] In the sixth iteration, view ray 21 has reached the current position of view ray 22. Both view rays are now at the same distance from the viewing position. Therefore, for V2 and V1, the respective mapped voxel values ​​are determined at ray march step positions '14' to '19' (the last positions of view ray 22 that intersected scene object 121) and combined into the respective image projection pixel values. In the 11th iteration, the current positions of view rays 21 and 22 are incremented to position '20'. Position '20' is already outside of V2. That is, the ray marching processor no longer considers view ray 22 for further processing.

[0078] V4 and V3: Watch rays 24 and 23 are still waiting.

[0079] After all view rays have been processed in the 11th iteration, the multi-ray position is incremented to '20'.

[0080] Iterations 12 to 14

[0081] V1: At positions '20' to '22', view ray 21 intersects only visible pixels of its associated volume (scene object 111). Therefore, the corresponding mapped voxel values ​​are combined with the corresponding image projection pixels. The current position of view ray 21 is incremented to position '23' in iteration 14.

[0082] V4 and V3: Watch rays 24 and 23 are still waiting.

[0083] After all view rays have been processed in the 14th iteration, the multi-ray position is incremented to '23'.

[0084] Iteration 15

[0085] V1: View ray 21 is now at position '23', where it still intersects a visible voxel of scene object 111. Although, at position '23', V1 overlaps with V3, only the mapped voxel value of view ray 21 is combined with the pixel value because the voxel in V2 at position '23' is not visible, and view ray 23 has advanced to position 24 (while the multi-ray position is still at position 23). The current position of view ray 21 is incremented to '24'.

[0086] Watch Rays 23, 24 still waiting.

[0087] The multi-ray position is incremented to '24'.

[0088] Iterations 16 and 17

[0089] V1: During iterations 16 and 17, view ray 21 reaches positions '24' and '25', which still intersect visible voxels of V1. Therefore, the respective mapped voxel values ​​of scene object 111 are combined with the respective projected image pixels. At the end of iteration 17, the current position of view ray 21 is set to '26'.

[0090] V4: View ray 24 still pending.

[0091] V3: The current position of view ray 23 is equal to the multi-ray position '24' at the start of iteration 16. Therefore, the mapped voxel value of view ray 23 can now be combined with the corresponding projected image pixel, because no other voxels closer to the viewing position can hide the visible voxels of scene object 131. The current position of view ray 23 is incremented to '25', and the same process continues for the visible voxels at this position. The current position of view ray 23 is set to '26' at the end of iteration 17.

[0092] The multi-ray position is also set to '26' at the end of the 17th iteration.

[0093] Iterations 18 to 21

[0094] V1: From position '26' onward, the remaining voxels inside V1 that are intersected by view ray 21 are all invisible (to the right of scene object 111). Therefore, no more combinations of mapped voxel values ​​are needed. The current position at the end of iteration 21 is incremented to '29', which is outside of V1. Therefore, view ray 21 is no longer considered for further ray marching processing after iteration 21. This example illustrates the situation when ray marching skipping is used. In this empty space skip implementation, the ray marching step that would have occurred at position '28' has been skipped.

[0095] V4: View ray 24 still pending.

[0096] V3: The corresponding voxel on view ray 23 is a visible voxel of scene object 131. Therefore, each mapped voxel value is combined with each projected image pixel. The current position of view ray 23 at the end of iteration 21 is set to '29'.

[0097] The multi-ray position at the end of iteration 21 is set to '29'.

[0098] Iteration 22

[0099] V4: View ray 24 is still pending.

[0100] V3: View ray 23 is at the last visible voxel of scene object 131. Therefore, the mapped voxel values ​​are combined and the current position is incremented to "30".

[0101] The multi-ray position is set to '30'.

[0102] Iteration 23

[0103] V4: View ray 24 still pending.

[0104] V3: At position '30', view ray 23 intersects the last (invisible) voxel of V3. Therefore, no value is combined with the projected image pixel. The current position increments to '31', which is outside of V3. Therefore, the ray marching processor no longer considers V3.

[0105] The multi-ray position is set to '30'. Alternatively, when leaving the volume, the multi-ray position may be updated with the minimum view ray position of the volume still to be processed, which in the current example is the current position '42' of view ray 24 .

[0106] Iteration 24

[0107] V4: V4 is now the only volume left for ray marching. Although multi-ray position '30' is still closer to the view position than view ray 24's current position '42', the ray marching processor can now continue processing view ray 24 because all volumes closer to the view position have already been processed. Therefore, the current multi-ray position from now on corresponds to the current position of view ray 24. The voxel at position '42' in V4 is the first visible voxel of scene object 141. Therefore, combining is performed, and the current position of view ray 24 is incremented to '43'.

[0108] Iterations 25 to 29

[0109] V4: At positions '43' to '47', view ray 24 intersects visible voxels belonging to scene object 141. After each ray marching step of view ray 24, the respective mapped voxel value is combined with the projected image pixel. At the end of the 29th iteration, the current position is incremented to '48'.

[0110] At position '48', the last voxel in volume V4 is already an invisible voxel outside scene object 141, and no combining occurs. Thus, the ray marching process for the entire multi-ray 20 has been successfully completed, and all mapped voxel values ​​for volumes V1 through V4 have been combined with the corresponding projected image pixels in the correct order, ensuring that voxels farther from the viewing position will not erroneously affect projected image pixel values ​​in situations where voxels farther from the viewing position would be hidden by voxels closer to the viewing position. However, processing of separate view arrays allows for efficient processing of 3D scenes with multiple volumes, particularly in scenes with overlapping volumes. Of course, the multi-volume rendering method disclosed herein is equally applicable to non-overlapping volumes (e.g., two volumes side by side). In this case, the volumes can be rendered one after the other, starting with the volume farthest from the viewing position, and using standard alpha compositing to obtain the final image. This is efficient because at each point in the 3D scene, there is only one volume to influence the color of the view ray traversing that portion of the 3D scene. However, for overlapping volumes with semi-transparent overlapping scene objects, a simple per-volume approach no longer works. In contrast, the multi-ray approach described above can be used to render all volumes of a 3D scene at once, thus solving this problem. Furthermore, the decoupling of the individual view rays associated with the volumes allows for faster rendering of some more complex situations where the view ray may have to be slowed down in different areas of the scene. Typical reasons for ray slowing down (i.e., not being able to make larger steps by skipping through, for example, empty space) are noisy areas in the volume (e.g., due to low-quality images or many tiny structures), or when a view ray passes through structures close to visible voxels without actually passing through anything visible.

[0111] In particular, in medical applications, multiple volumes with different sources of 3D data can be superimposed on one another in a 3D scene. For example, a CT volume and an MRI volume can be superimposed in a 3D scene, resulting in overlapping volumes. This is typically the case when the corresponding scans were directed at the same part of the patient's body (e.g., the patient's head or chest). CT typically produces higher-resolution images without geometric distortion, which is well-suited for visualizing bone. However, certain other tissues or pathologies cannot be visualized well using CT data. To close this gap, an MRI volume can be generated in addition to the CT volume. For example, an MRI volume can capture a tumor and be co-registered with the CT volume so that it appears in the correct position relative to the bone. In another use case, pre-operative and post-operative CT scans can be presented side by side to assess surgical outcomes. In yet another use case, a single CT volume can be separated into multiple smaller volumes, each containing one or more of the scene objects in the original CT volume. The scene objects in one smaller volume can then be individually moved relative to the scene objects in the other smaller volumes. This can be used, for example, to test fracture correction by rearranging the bone (i.e., by rearranging the smaller volume containing the bone portion as a scene object).

[0112] Standard empty space skipping techniques (e.g., Euclidean distance mapping, min-max octrees) can be used to accelerate view rays in empty regions of a volume. Empty space skipping is applied to each individual view ray. This means that view rays for different associated volumes (for the same projected image pixel) may become unsynchronized and some view rays may be ahead of other view rays. It has been described above how to deal with this problem by ensuring that visible voxels closer to the viewing position are processed before visible voxels at a greater distance. With empty space skipping, performance optimization can be achieved by using additional information to skip as much invisible regions of the corresponding volume as possible (e.g., invisible voxels that typically represent air). Empty space skipping techniques are known in the art. However, in the method disclosed herein, empty space skipping is applied independently to each individual view ray of the view multi-ray.

[0113] One possible approach for empty-space skipping is to precompute the 3D data when it is first loaded into the scene. For each individual voxel in a given volume, the distance to the nearest non-transparent voxel can be precomputed. In other words, the system determines the radius of the largest sphere that can be placed at that location containing only invisible voxels. This data structure is called a distance map. Figure 7Two visible scene objects 710 and 720 are shown in volume 700. The camera at the center of sphere 711 represents the viewing position. View ray 701 shows a ray traversing the volume corresponding to a single pixel in the projected image. As previously described, in one embodiment of the ray marching process, a distance map can be used at each ray marching step to determine whether a larger ray marching jump is possible across empty regions of volume 700 while ensuring that no visible voxels are skipped. The first ray marching step is located at the center of sphere 711 at the viewing distance in front of the camera. The value stored in the pre-calculated distance map at this location is the length of dashed line r1, which is the radius of the largest sphere that can be placed at this location without containing any visible voxels. The closest visible voxel to the center of sphere 711 is part of scene object 720 and is located at the intersection of dashed line r1 and sphere 711. Therefore, at the first ray marching step, a jump can be made equal to the length of r1 along view ray 701, ending at the center of sphere 712. At step 2, the value read from the distance map is the length of dashed line r2, which corresponds to the distance between the center of sphere 712 and the nearest visible voxel (part of scene object 710). Therefore, at step 2, a jump equal to the length of r2 can be made. This process is repeated until a value is read from the distance map that does not warrant an empty space jump. This is typically a distance less than or equal to the ray marching step length. In the example shown, this process is repeated for sphere 713 with radius r3, sphere 714 with radius r4, and three additional smaller spheres. At the last illustrated position along view ray 701, the distance to the nearest visible voxel (part of scene object 720) is too small to produce an empty space jump. Therefore, at this point, the ray marching process reverts to the default implementation without empty space skipping as described in the previous section. It should be noted that in the case of empty space skipping, the distance map is always read before falling back to the default implementation. Therefore, empty space jumps can occur in non-intersecting segments along the view ray. For example, a view ray can pass through a semi-transparent scene object, in which case the empty space can be used to skip before and after the scene object, rather than when passing through the scene object.

[0114] Figure 8 is a diagram illustrating an example of a general purpose computer device 900 and a general purpose mobile computer device 950 that can be used with the techniques described herein. In an exemplary embodiment, the computing device 900 is related to the MVR system 200 (see Figure 2Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices such as virtual reality glasses. Computing device 950 can be used to display projected images as a result of direct multi-volume rendering to a user. In the case where the user uses a virtual reality display device, two projected images reflecting different perspectives of the user's eyes are presented. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the embodiments of the present invention described and / or claimed in this document.

[0115] Computing device 900 includes a processor 902, memory 904, storage device 906, a high-speed interface 908 connected to memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and storage device 906. Each of components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and can be mounted on a common motherboard or other suitable mounting methods. Processor 902 can process instructions for execution within computing device 900, including instructions stored in memory 904 or on storage device 906, to display graphical information of a GUI on an external input / output device, such as a display 916 coupled to high-speed interface 908. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used as appropriate. Furthermore, multiple computing devices 900 can be connected, with each device providing a portion of the necessary operations (e.g., as a server group, a blade server group, or a multi-processor system).

[0116] Memory 904 stores information within computing device 900. In one embodiment, memory 904 is one or more volatile memory units. In another embodiment, memory 904 is one or more non-volatile memory units. Memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0117] The storage device 906 can provide mass storage for the computing device 900. In one embodiment, the storage device 906 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 904, the storage device 906, or a memory on the processor 902.

[0118] The high-speed controller 908 manages bandwidth-intensive operations for the computing device 900, while the low-speed controller 912 manages less bandwidth-intensive operations. This functional allocation is merely exemplary. In one implementation, the high-speed controller 908 is coupled to the memory 904, the display 916 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 910, which can accept various expansion cards (not shown). In an embodiment, the low-speed controller 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, ZigBee, WLAN, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networked device such as a switch or router, for example, through a network adapter.

[0119] The computing device 900 can be implemented in many different forms, as shown. For example, it can be implemented as a standard server 920, or multiple implementations in a group of such servers. It can also be implemented as part of a rack server system 924. In addition, it can be implemented in a personal computer such as a laptop computer 922. Alternatively, components from the computing device 900 can be combined with other components in a mobile device (not shown). Each of such devices can contain one or more of the computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 communicating with each other.

[0120] The computing device 950 includes a processor 952, a memory 964, an input / output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The device 950 may also be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and several of these components may be mounted on a common motherboard or in other suitable ways.

[0121] The processor 952 is capable of executing instructions within the computing device 950, including instructions stored in the memory 964. The processor can be implemented as a chipset including separate or multiple analog and digital processors. The processor can provide, for example, coordination of other components of the device 950, such as control of a user interface, applications executed by the device 950, and wireless communications performed by the device 950.

[0122] The processor 952 can communicate with the user through a control interface 958 and a display interface 956 coupled to the display 954. The display 954 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 956 may include appropriate circuits for driving the display 954 to present graphics and other information to the user. The control interface 958 can receive commands from the user and convert them for submission to the processor 952. In addition, an external interface 962 in communication with the processor 952 can be provided to enable near-area communication of the device 950 with other devices. The external interface 962 can, for example, provide wired communication in some implementations, or provide wireless communication in other implementations, and multiple interfaces can also be used.

[0123] Memory 964 stores information within computing device 950. Memory 964 can be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 984 can also be provided and connected to device 950 via expansion interface 982, which can include, for example, a SIMM (Single In-line Memory Module) card interface. Such expansion memory 984 can provide additional storage space for device 950, or can also store applications or other information for device 950. Specifically, expansion memory 984 can include instructions for executing or supplementing the above-mentioned processes, and can also include security information. Thus, for example, expansion memory 984 can act as a security module for device 950 and can be programmed with instructions that allow for the safe use of device 950. In addition, security applications can be provided along with additional information via a SIMM card, such as placing identification information on a SIMM card in a manner that cannot be hacked.

[0124] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as the memory 964, the expansion memory 984, or memory on the processor 952, which may be received, for example, via the transceiver 968 or the external interface 962.

[0125] The device 950 can communicate wirelessly via a communication interface 966, which may include digital signal processing circuitry, where necessary. The communication interface 966 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communication may occur, for example, via a radio frequency transceiver 968. In addition, short-range communication may occur, for example using Bluetooth, WiFi, ZigBee, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 980 may provide additional navigation and location-related wireless data to the device 950, which may be used as appropriate by applications running on the device 950.

[0126] Device 950 may also communicate audibly using audio codec 960, which may receive spoken information from a user and convert it into usable digital information. Audio codec 960 may also generate audible sounds for the user, such as through a speaker in a handset of device 950. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 950.

[0127] The computing device 950 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a cellular phone 980. It can also be implemented as part of a smart phone 982, a personal digital assistant, or other similar mobile device.

[0128] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0129] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0131] The systems and techniques described herein can be implemented in a computing device that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0132] Computing devices may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0133] A number of embodiments have been described, however, it will be understood that various modifications can be made without departing from the scope of the invention.

[0134] Furthermore, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. Additionally, other steps can be provided or eliminated from the described flows, and other components can be added to or removed from the described systems.

Claims

1. A computer-implemented method (1000) for direct multi-volume rendering, the method comprising: receiving (1100) 3D data specifying a three-dimensional scene (100) comprising a plurality of volumes (110, 120, 130, 140), wherein each volume comprises a plurality of voxels (pv_n), wherein each voxel of a particular volume has a scalar value that is a result of an imaging process for measuring a physical property of an imaged object; generating (1200) a view multi-ray (20) at a viewing position (10) in the direction of a particular pixel of a projected image (30) located between the viewing position (10) and the scene (100), wherein the particular pixel has a value representing the projection of all scene voxels intersected by the view multi-ray behind the projected image (30), the view multi-ray (20) representing a plurality of individual view rays (21, 22, 23, 24) having the same starting point and the same direction, wherein each individual view ray is associated with one of the volumes (110, 120, 130, 140) in the scene (100); iterating all individual view rays using volumetric ray marching in each ray marching step for individually processing (1300) each view ray (21, 22, 23, 24) in its associated volume, wherein the ray marching step for the respective view ray is performed only in the associated volume, and in each ray marching step the scalar value of the respective voxel is mapped to a color value and a transparency value for the respective voxel; updating (1400) the value of a particular pixel in the projected image (30) after each ray marching step by combining the corresponding voxel colors and transparency values ​​of the individual view rays, wherein updating with the value of the particular voxel intersecting the particular view ray is performed after updating with all intersecting voxel values ​​of voxels closer to the viewpoint than the particular voxel, such that processing of opaque voxels occurs synchronously with processing of all individual view rays at the same location in the three-dimensional scene; and The generating (1200), processing (1300) and updating (1400) steps are performed (1500) on pixels of the projected image (30) that have not yet been updated. 2 . The method of claim 1 , wherein volume ray marching uses empty space skipping in that separate processing of a particular view ray skips ray marching steps associated with fully transparent voxels within the associated volume. 3 . The method according to claim 1 , wherein the imaging process is an image acquisition or image generation process resulting in a volumetric image.

4. The method according to any one of the preceding claims, wherein the imaging process is selected from any one of the following: computed tomography imaging, magnetic resonance imaging, optical coherence tomography imaging, positron emission tomography imaging, single photon emission computed tomography imaging and 3D ultrasound imaging.

5. The method according to any one of claims 1 to 4, wherein the physical property is selected from the group consisting of: absorption of X-ray radiation, number of protons, light transmittance, positron emission of an injected radiopharmaceutical, and acoustic impedance.

6. A method according to any preceding claim, wherein two or more of the plurality of volumes are registered relative to each other.

7. The method of any one of the preceding claims, wherein at the start of the ray marching process for each view ray, the respective view ray is transformed from a global coordinate system (xg, yg, zg) of its associated volume (110, 120, 130, 140) to a local coordinate system (xn, yn, zn).

8. A method according to any of the preceding claims, wherein the first view rays (21) are individually processed (1300) using a first predefined transfer function for mapping scalar values ​​of voxels of the corresponding associated volume (110) to corresponding color values ​​and transparency values, and the second view rays (22) are individually processed (1300) using a second predefined transfer function for mapping scalar values ​​of voxels of the corresponding associated volume (120) to corresponding color values ​​and transparency values.

9. A computer program product for direct multi-volume rendering, the computer program product comprising computer readable instructions which, when loaded into a memory of a computing device and executed by at least one processor of the computing device, cause the computing device to perform the method according to any one of the preceding claims.

10. A computer system (200) for direct multi-volume rendering, comprising: An interface (210) adapted to receive 3D data specifying a three-dimensional scene (100) comprising a plurality of volumes (290), wherein each volume comprises a plurality of voxels (pv_n), wherein each voxel of a particular volume has a scalar value that is a result of an imaging process for measuring a physical property of an imaged object; a multi-ray generator (230) configured to generate, at a viewing position (10), a view multi-ray (20) in the direction of a particular pixel of a projected image (30) located between the viewing position (20) and the scene (100), wherein the particular pixel has a value representing the projection of all scene voxels intersected by the view multi-ray behind the projected image (30), the view multi-ray (20) representing a plurality of individual view rays (21, 22, 23, 24) having the same starting point and the same direction, wherein each individual view ray is associated with one of the volumes (110, 120, 130, 140) in the scene (100); a volumetric ray marching processor (240) configured to iterate all individual view rays using volumetric ray marching in each ray marching step for individually processing each view ray (21, 22, 23, 24) in its associated volume, wherein the ray marching step for a respective view ray is performed only in the associated volume, and in each ray marching step a scalar value of a respective voxel is mapped to a color value and a transparency value of the respective voxel; as well as A projection image updater (250) is configured to update the value of a specific pixel in the projection image (30) after each ray marching step by combining the corresponding voxel color and transparency value of the individual view rays, wherein the updating with the value of the specific voxel intersecting the specific view ray is performed after the updating with all intersecting voxel values ​​of voxels closer to the observation point than the specific voxel, so that the processing of opaque voxels is performed synchronously with the processing of all individual view rays at the same position in the three-dimensional scene.

11. The system of claim 10, wherein the volumetric ray marching processor (240) uses empty space skipping as separate processing of a particular view ray skips ray marching steps associated with fully transparent voxels within the associated volume.

12. The system according to any one of claims 10 to 11, further comprising a volume registration module (220) configured to register one or more of the plurality of volumes with respect to each other.

13. A system according to any one of claims 10 to 12, wherein the volume ray marching processor (240) also includes a coordinate system converter 242, which is used to convert the corresponding view ray from the global coordinate system (xg, yg, zg) of its associated volume (110, 120, 130, 140) to the local coordinate system (xn, yn, zn) at the beginning of the ray marching process for each view ray for volume ray marching in the associated volume.

14. A system according to any one of claims 10 to 13, wherein the volume ray marching processor uses a first predefined transfer function when separately processing (1300) a first view ray (21) to map the scalar values ​​of the voxels of the corresponding associated volume (110) to corresponding color values ​​and transparency values, and uses a second predefined transfer function when separately processing (1300) a second view ray (22) to map the scalar values ​​of the voxels of the corresponding associated volume (120) to corresponding color values ​​and transparency values.

15. The system of any one of claims 10 to 14, wherein the 3D data for a particular volume further comprises a distance map that stores, for each voxel of the particular volume, a distance to a next opaque voxel in the volume, and wherein ray marching with a particular view ray performs a ray marching jump that includes a multiple of the distance between two ray marching steps according to the distance map such that the next ray marching step is performed close to the next opaque voxel.

Images