Systems and methods for multi-volume rendering

Abstract

A system (100), method, and computer program product for direct multi-volume rendering. The 3D data specifies a three-dimensional scene having multiple volumes (290). Each voxel in a particular volume has a scalar value that is the result of an imaging procedure for measuring physical properties of the imaged subject. A multi-ray generator (230) generates a view multi-ray in the direction of a particular pixel in the projection image, which represents the 2D projection of all scene voxels intersected by the view multi-ray after the projection image. A volume ray marching processor (240) processes each view ray in its associated volume using volume ray marching, and at each ray marching step, the scalar value of a corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. A projection image updater (250) updates the value of a particular pixel in the projection image by combining the color and transparency values ​​of each voxel in each view ray. Updating the value of a particular voxel intersecting a particular view ray is performed after updating the values ​​of all intersecting voxels that are closer to the viewpoint than the particular voxel.

Description

[Technical Field]

[0001] The present invention relates generally to rendering images showing three-dimensional scenes, and more particularly to rendering volumes using multiple volumes. [Background technology]

[0002] Many software applications, such as medical applications, use computer graphics algorithms to generate 3D representations of a patient's organs. Also, in computer games or other 3D applications that run within virtual 3D worlds, the 3D scene must be rendered into an image and displayed to the human user of the application(s). Rendering such 3D scenes, particularly in medical applications, is typically performed by using an image-based volume rendering technique, often referred to as volume ray marching. Ray marching uses volumetric data. Other techniques, such as ray tracing and ray casting, calculate the intersection of rays with surface models of 3D objects. Ray marching does not use such surface models of 3D objects. It only knows the rectangular bounding box of the volume and cannot directly calculate the intersection of view rays with the surface of 3D objects within the volume. For this reason, ray marching performs small ray marching steps to sample each voxel.

[0003] In its basic form, the volume ray marching algorithm involves four steps: - Ray casting: for each pixel in the final image, a line of sight (view ray) is sent through the volume to be rendered. Typically, the volume corresponds to a simple geometric object (e.g. a cuboid). - Sampling. Equidistant sampling points or samples are selected along the portion of the view ray that is within the volume. In general, the volume does not necessarily coincide with the view ray. Therefore, the sampling points do not necessarily coincide with the center positions of the voxels. Therefore, the values ​​of the samples may be interpolated from neighboring voxels. - Shading: For each sampling point, the sampling value is mapped to an RGBA material color, opacity / transparency, and optionally other properties. - Combination: All sampling points are combined along the view ray to form the final color value for the currently processed pixel.

[0004] The foundational paper disclosing the concept of ray marching is: Levoy, Marc. "Display of surfaces from volume data." IEEE Computer graphics and Applications 8.3 (1988): 29-37. Efficiently performing direct multi-volume rendering remains a challenge for prior art solutions, especially for 3D scenes with overlapping volumes. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need to provide an efficient solution for direct multi-volume rendering of three-dimensional scenes. Direct rendering, as used herein, means rendering an image directly from raw data. The term will be further explained in the context of different raw data types. [Means for solving the problem]

[0006] The aforementioned problem is solved by the embodiments claimed by the independent claims: a computer-implemented method for direct multi-volume rendering, a computer program product with instructions configured to cause said method to be performed when said computer program product is executed, and a computer system capable of loading and executing said computer program product to perform said method.

[0007] A computer system for direct multi-volume rendering is hereinafter also referred to as an MVR system. The MVR system includes functional modules capable of executing functions for performing each step of the method disclosed herein. Through an appropriate interface, the MVR system receives 3D data specifying a three-dimensional scene. 3D means three dimensions, and 2D means two dimensions, as used herein. The 3D scene includes multiple volumes to be rendered by the MVR system. A typical 3D dataset is, for example, a series of 2D slice images acquired by a computed tomography (CT), magnetic resonance imaging (MRI), X-ray microtomography (MicroCT), or other 3D imaging technique. The 3D data received by the MVR system corresponds to the aforementioned raw data. Such individually sampled 3D datasets are typically referred to as volumes. The 3D scene includes multiple volumes, each of which includes multiple voxels. Each voxel of a particular volume has a scalar value that is the result of a volumetric imaging procedure for measuring a physical property of the imaging subject. As used herein, the imaging subject is the object that is the subject of the volumetric imaging procedure. Examples of imaging subjects include biological samples (e.g., samples from humans, animals, or plants) or non-biological samples (e.g., components of machines or devices). In other words, the raw data used for direct rendering is a collection of voxels in a volume, with each voxel having a scalar value. Examples of such imaging procedures (imaging techniques) include, but are not limited to, computed tomography, magnetic resonance tomography, optical coherence tomography, positron emission tomography, single-photon emission computed tomography, and 3D ultrasound imaging, or any image acquisition or generation process that results in a volumetric image. Using an appropriate imaging procedure, physical properties can be measured, such as, for example, absorption of X-ray radiation, number of protons, optical transparency, positron emission of injected radiopharmaceuticals, acoustic impedance changes, or other object properties. Those skilled in the art know which imaging techniques can be used to measure the respective physical properties of the imaging subject.

[0008] The MVR system includes a multi-ray generator module for generating view multi-rays at a viewpoint in the direction of a specific pixel of the projection image. The viewpoint typically represents the position of a virtual camera or virtual reality device used by a user to observe the 3D scene. A projection image is a 2D projection of the 3D scene as seen from the viewpoint projection. The projection image is an image displayed on a display device, such as a monitor. In the case of using a virtual reality device to display images, two projection images suitable for generating a 3D VR effect are generated. The projection images are positioned between the viewpoint and the scene. Typically, the projection images are positioned in front of the viewpoint at a viewing distance of approximately 10 cm. In relation to that, a specific pixel (through which the view multi-ray is transmitted into the 3D scene) represents the 2D projection of all scene voxels intersected by the view multi-ray behind the projection image. A view multi-ray represents multiple individual view rays with the same starting point and the same direction, each associated with one of the volumes in the scene. That is, each individual view ray (hereinafter also referred to as view ray) belongs to exactly one of the volumes loaded in the scene.

[0009] The MVR system further includes a volume ray marching processor module that uses volume ray marching to individually process each view ray within its associated volume. Individually processing each view ray within its associated volume includes determining color and transparency values ​​within the associated volume at each ray marching step. That is, the ray marching steps for each view ray are performed only within the associated volume. As a result, the number of ray marching steps with a given step length (i.e., the distance between two ray marching steps) for a particular view ray is limited by the boundaries of the associated volume. For example, the first ray marching step for a particular view ray may occur at multiple step lengths where the particular view ray first intersects with the associated volume. A view ray intersects with its associated volume at the current ray marching step where the view ray first intersects with the associated volume, or if a respective location outside the associated volume is already associated with a visible color value. In some cases, even ray marching locations outside the volume—but very close to the volume—may already have visible color values ​​due to voxel interpolation. At each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. In one embodiment, such mapping can be implemented using a predefined transfer function. There may be different transfer functions for each volume (or even for different subregions within a particular volume) depending on the imaging technology used to create each associated volume. For example, in the case of a CT scan, the scalar values ​​are measured in Hounsfield units (HU). Each transfer function may map HU values ​​corresponding to air (black) and soft tissue (gray) to transparent and HU values ​​corresponding to bone (bright) to white / beige. Typically, the scalar values ​​can be interpreted as grayscale values ​​that are mapped to color and transparency values ​​(or corresponding opacity values) according to the transfer function. Alternative color mapping approaches are disclosed in the detailed description.

[0010] The projection image updater module of the MVR system updates the value of a specific pixel (a pixel intersected by the current view multi-ray) in the projection image by combining the mapped voxel values ​​and transparency values ​​on each view ray. Therefore, updating the value of a specific voxel intersecting with a specific view ray is performed after updating the values ​​of all intersecting voxels that are closer to the viewpoint than the specific voxel. In other words, combining the color / transparency values ​​of voxels at a greater distance to the viewpoint must wait until combining the color / transparency values ​​of voxels at a shorter distance is complete. As a result, view rays associated with more distant volumes can be processed only after processing of view rays associated with less distant volumes is completed. The pixel value update can be implemented so that the combined values ​​after each ray marching step are stored in a respective local variable of the system after each ray marching step, and the value of this variable is finally written to the specific pixel in the projection image once ray marching is completed for the corresponding view multi-ray.

[0011] The MVR system performs the generation, processing, and update steps for pixels of the projection image that have not yet been updated. For example, the described functions can be repeated for all pixels of the projection image in a loop. Since the processing of a first view multi-ray is independent of the processing of a second view multi-ray, it is also possible to perform multiple view multi-rays in parallel. For example, in practical applications of the approach disclosed herein, hundreds or even thousands of view rays are typically processed in parallel by a graphics processing unit (GPU). Parallelization techniques are known in the art, such as for distributing processing to multiple cores and for pipelining processing.

[0012] In one embodiment, volume ray marching may use spatial skipping, in that the individual processing of a particular view ray is accelerated by skipping as many of the ray marching steps as possible inside the associated volume that are associated with fully transparent voxels. A spatial skipping approach using distance maps is disclosed in the following paper: -Zuiderveld,Karel J., Anton HJ Koning, and Max A. Viergever. "Acceleration of ray-casting using 3-D distance transforms." Visualization in Biomedical Computing'92.Vol.1808.SPIE,1992. -Sramek, Milos. "Fast surface rendering from raster data by voxel traversal using chessboard distance." Proceedings Visualization'94. IEEE, 1994.

[0013] In one embodiment, the MVR system may further include a volume registration module for registering one or more of the multiple volumes relative to one another. Registering a first volume with respect to a second volume means that the origin of the local coordinate system of the first volume is known within the second volume (and vice versa). Once all volumes are properly registered, the volumes represent the same anatomical point at the same location in the global coordinate system. This allows the positions of two or more volumes to be fixed relative to one another. For example, the position of a volume representing an MR scan of a patient's human brain can be fixed with respect to a volume representing a CT scan of the patient's skull so that the virtual representation makes anatomical sense.

[0014] To perform ray marching, it may be convenient to transform each view ray from the global coordinate system (i.e., the coordinate system associated with the 3D scene) to the local coordinate system of its associated volume (i.e., the coordinate system of the associated volume that is translated, rotated, and scaled relative to the global coordinate system) at the start of each ray marching process. Using such a transformation allows ray marching steps to be processed at various volumes in their respective local coordinate systems simply based on the number of steps performed, without the need to calculate the respective distances from the viewpoint position in the global coordinate system for the various volumes. As mentioned above, this transformation can be done once at the start of each ray marching process and does not need to be repeated for each ray marching step.

[0015] In one embodiment, a computer program product for direct multi-volume rendering is provided. The computer program product has computer-readable instructions that, when loaded into a memory of a computing device (e.g., an MVR system) and executed by at least one processor of the computing device, cause the computing device to perform the computer-implemented methods disclosed herein. In that regard, functional modules of the MVR system can be implemented as software modules that implement functions for performing respective method steps when executed by the MVR system.

[0016] Additional aspects of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as defined herein. [Brief explanation of the drawings]

[0017] [Figure 1A] 1 illustrates view multi-rays used to render multiple volumes of a 3D scene in a projection image, according to one embodiment. [Figure 1B] The voxels of the volume are shown with their local coordinate systems. [Figure 2] 1 is a simplified block diagram of an MVR system according to one embodiment. [Figure 3] 1 is a simplified flow diagram illustrating a computer-implemented method for direct multi-volume rendering, according to one embodiment. [Figure 4A] 1 shows an example of a transfer function for mapping scalar voxel values ​​to color and transparency values. [Figure 4B] 1 shows an example of a transfer function for mapping scalar voxel values ​​to color and transparency values. [Figure 4C] 1 shows an example of a transfer function for mapping scalar voxel values ​​to color and transparency values. [Figure 4D] 1 shows an example of a transfer function for mapping scalar voxel values ​​to color and transparency values. [Figure 5] 1 shows an example of pixels in a projection image and their respective view multi-rays. [Figure 6] 1 shows an example of ray marching in a volume containing scene objects. [Figure 7] 10 shows an example of using a distance map for spatial skipping. [Figure 8] 1 is a simplified diagram illustrating an example of a general purpose computing device and a general purpose mobile computing device that may be used in embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] FIG. 1A illustrates a view multiray 20 used to directly render multiple volumes 110, 120, 130, 140 of a 3D scene in a projected image 30, according to one embodiment. This diagram illustrates the conceptual approach behind the solution disclosed herein. This approach is explained in the context of FIGS. 2 and 3. In that regard, FIG. 2 is a simplified block diagram of a computer system 200 (MVR system) adapted to implement the MVR approach shown in FIG. 1A, and FIG. 3 illustrates a computer-implemented method 1000 that can be executed by the MVR system 200 at runtime. For this reason, the rest of the description will use reference numbers that refer to such figures.

[0019] The MVR system 200 receives 1100 3D data specifying the three-dimensional scene 100 via an interface 210. Those skilled in the art will recognize appropriate interfaces for loading multiple volumes 290 into the scene 100. The 3D data specifying each volume may be stored in remote data storage (e.g., in a database on a remote device communicatively coupled to the system 200). Alternatively, the system 200 may include a memory adapted to store the 3D data for the volumes 290. As previously mentioned, a volume is made up of so-called voxels. FIG. 1B illustrates several voxels pv_n of a volume Vn. The voxels of a volume have a rectangular shape. Of course, in a complete volume Vn, each voxel layer has the same number of voxels, typically arranged in a rectangle that defines the rectangular shape of the volume. While the voxels may have a rectangular shape, in general, a volume Vn is typically made up of layers of 2D images, where each pixel of the 2D image is square. In the example, the 2D image layer is assumed to lie in the xn-yn plane of Vn's local coordinate system CSn. Looking at the dotted voxels, it can be seen that the depth of the voxel in the zn dimension is different from the edge length of the voxel in the xn-zn dimension. The distance at which the 2D image layer is captured (e.g., by one of the volumetric imaging techniques mentioned above) corresponds to the depth of the voxel in the zn dimension. Typically, the depth of the voxel is different from the edge length of the respective pixel in the underlying 2D image layer.

[0020] In the example of FIG. 1A, the MVR system 200 has loaded four volumes 110 (V1), 120 (V2), 130 (V3), and 140 (V4) into the scene 100. For simplicity, the volumes are shown as two-dimensional rectangles. However, in a virtual 3D scene, the volumes are, of course, three-dimensional rectangular prisms. The legend in FIG. 1A indicates the different line styles used to distinguish the lines associated with each volume from those associated with other volumes. For example, a dotted line style is used for all lines associated with V1. Each volume has its own local coordinate system. For simplicity, only the local coordinate system xl1-yl1-zl1 of V1 is shown in the figure. However, V2, V3, and V4 also have their own local coordinate systems, which can generally be translated, rotated, and scaled relative to the other local coordinate systems. Those skilled in the art know how to use matrix multiplication with a transformation matrix to convert from one coordinate system to another. Each voxel in a particular volume has a scalar value that is the result of the imaging procedure for measuring a physical property of the imaged subject. In the example, voxels with scalar values ​​that represent a scene object in the respective volume are within the boundary of said scene object 111, 121, 131, 141. Like the volumes, these three-dimensional scene objects are also shown as two-dimensional objects for the purpose of simplifying the graphical representation in the figures. The scene objects are defined by the visible voxels in their respective volumes. Visible voxels may be semi-transparent or opaque voxels rendered with a corresponding color value. In the example of FIG. 1A, the scene objects may represent patient tissues and / or organs identified by the respective imaging techniques by their respective scalar voxel values ​​in their respective volumes.

[0021] In an optional embodiment, the MVR system 200 comprises a volume registration module adapted to register the volumes 290 (eg, V1-V4) relative to one another, as already mentioned above.

[0022] The multi-ray generator 230 of the MVR system 200 generates 1200 a view multi-ray 20 with the viewpoint position 10 as its starting point. In FIG. 1, the left endpoint of the camera / eye symbol defines the viewpoint position. The view multi-ray 20 is generated in a direction relative to the viewpoint position 10 of a specific pixel of the projection image 30. The projection image 30 is located within a viewing distance d from the viewpoint position 10 and between the viewpoint position 10 and the scene 100. Finally, the specific pixel represents the 2D projection of all scene voxels intersected by the view multi-ray after the projection image 30. In other words, the view multi-ray 20 intersecting the specific pixel is used to sample voxel values ​​of the 3D scene in the direction of the multi-ray to finally calculate a combined value reflected in the color value of the specific pixel of the projection image. In that respect, the view multi-ray 20 represents multiple individual view rays 21, 22, 23, 24 with the same starting point and the same direction. Each individual view ray is associated with one of the volumes in the scene. In the example, view ray 21 is associated with V1 110, view ray 22 is associated with V2 120, view ray 23 is associated with V3 130, and view ray 24 is associated with V4 140. For clarity and better visibility in the diagram, view rays 21-24 are shown as parallel arrows. However, they are identical in terms of starting point and direction; that is, their position in the scene is all identical. Nevertheless, the individual view rays are distinguishable by their unique assignment to one of the volumes. The starting points and directions of the multi-ray 20 (and, for that matter, the individual view rays) are defined by the multi-ray generator in the global coordinate system xg-yg-zg.

[0023] Referring briefly to FIG. 5, a projection image 30 is shown with three additional specific pixels 30-1, 30-2, and 30-3 through which the view multi-rays 20-1, 20-2, and 20-3 generated, respectively, are sent into the 3D scene to sample volumetric voxels within the scene. In this illustrative example, the volumes are not shown. Instead, a 3D scene object 121-1 constituted by one of the volumes is shown (as an ellipsoid). In the example, only multi-rays 20-1 and 20-2 intersect with voxels associated with the scene object. Furthermore, one individual view ray of each multi-ray 20-1, 20-2 is associated with a volume containing the voxels of the scene object 121-1. The final projection image 30 includes 2D projection values ​​for all pixels of the image 30.

[0024] The volume ray marching processor 240 of the MVR system 200 individually processes each view ray 21, 22, 23, 24 within its associated volume using volume ray marching. Advantageously, ray marching can be performed within each volume's local coordinate system for performance optimization. This saves computationally expensive matrix multiplications after each ray marching step. The ray marching step for each view ray is performed only within the associated volume to sample the respective volume voxels. In particular, in the case of overlapping volumes (e.g., V1 overlaps with V2, and V1 overlaps with V3), corresponding view rays are sampled only within their associated volumes. In one implementation, the first ray marching step for a particular view ray can occur at multiple ray marching step lengths, the first time the particular view ray intersects with its associated volume. In the example of FIG. 1, the first ray marching step for view ray 21 is indicated by the left ray marching dot on view ray 21. The first ray marching step of view ray 22 is indicated by the left ray marching dot on view ray 22, and so on. Note that in regions with overlapping volumes, the ray marching steps for each individual view ray are synchronous in that they occur at the same distance relative to the viewpoint position 10. In the example, the first ray marching step of view ray 22 in V2 occurs at the same distance relative to the viewpoint position as the sixth view marching step of view ray 21 in V1. The second ray marching step of view ray 22 occurs at the same distance relative to the viewpoint position as the seventh view marching step of view ray 21, etc. Similarly, the first ray marching step of view ray 23 occurs at the same distance relative to the viewpoint position as the sixteenth view marching step of view ray 21, etc.

[0025] In one embodiment, at each ray marching step, the scalar value of a corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel via a predefined transfer function. Optionally, the transfer function may also map the potentially interpolated scalar value sampled at the corresponding location to additional parameters, such as luminosity or reflectivity. The ray marching processor 240 may provide one or more transfer functions TF241 for mapping within each volume. In an example, view ray 21 is used to process only voxels in V1. A first transfer function TF1 may be used for voxel value mapping within V1. View ray 22 is used to process only voxels in V2. A second transfer function TF2 may be used for voxel value mapping within V2. That is, in overlapping volume regions, voxel values ​​of different volumes mapped at the same distance relative to the viewpoint position typically have different mapped voxel values, and individual view rays can calculate such volume-specific mapped voxel values ​​for their associated volumes using their respective transfer functions at each ray marching step.

[0026] In an alternative implementation, at each ray marching step, the scalar value of the corresponding voxel (or interpolated position within the volume) is mapped to only a transparency value via a predefined transfer function, and the color value is determined by applying an illumination model or a combination of different illumination models. In that regard, the system determines the color and light intensity reflected by the surface of a scene object toward the viewpoint by taking into account different light sources and surface material properties. More details can be found in the Wikipedia article "Bidirectional reflectance distribution function," available at https: / / en.wikipedia.org / wiki / Bidirectional_reflectance_distribution_function, or in the overview paper "An overview of BRDF models," by Montes, Rosana, and Carlos Urena, University of Grenada, Technical Report LSI-2012-001 (2012).

[0027] The characteristics to be taken into account depend on the illumination model(s) selected. The classical illumination model is the Blinn-Phong reflection model (see Blinn, James F. "Models of light reflection for computer synthesized pictures", Proceedings of the 4th annual conference on Computer graphics and interactive techniques 1977). Another commonly used illumination model is the Lambertian model (see Lambert, Johann Heinrich. Photometria sive de mensura et gradibus luminis, colorum et umbrae. sumptibus vidvae E. Klett, typis C.P. Detleffsen, 1760).

[0028] The result of this alternative implementation is that the color value of a voxel is determined by the position and color of the light source and the orientation of the scene object's surface relative to the light source. Surfaces facing the light source are colored by the light source, while surfaces behind it remain dark. For example, for a relatively simple CT scan, the object can be clearly identified.

[0029] In a practical implementation, the transfer function approach may be combined with the illumination model approach to provide improved spatial perception to the observer of the projected image.

[0030] The concept of transfer functions will now be explained in more detail using the example of Figures 4A-4D. In Figure 4A, raw CT slices (images) 410 provided by CT scanning techniques have grayscale pixel values. As a result, voxels of a CT volume, interpreted by combining multiple layers of raw CT slices, also have corresponding grayscale scalar values ​​(raw data). In a CT volume, the brightness of a voxel corresponds to the x-ray absorption of tissue at each body location in Hounsfield units (HU). To generate a realistic 3D rendering of the volume, a transfer function 400a is defined that maps grayscale HU values ​​401 to color and transparency (or opacity) values. In the CT example, transfer function 400a maps HU values ​​corresponding to air (black) and soft tissue (gray) to transparency value 402 ("transparent") and HU values ​​corresponding to bone (bright) to color value 403 ("white" or alternatively, "beige"). A direct rendering result 420 based on raw voxel values ​​and transfer functions 400a is shown for a portion of a human skeleton.

[0031] FIG. 4D uses transfer function 400d (similar to transfer function 400a) to generate direct rendering result 450 from the corresponding CT scan raw data volume. FIGS. 4B and 4C show direct rendering results 430, 440 generated from the same CT scan raw data volume using different transfer functions 400b and 400c, respectively. While the direct rendering results are shown in grayscale in the figures, transfer functions 400b and 400c actually map various scalar (grayscale) values ​​to color values ​​corresponding to different hues of red. Defining different transfer functions to map voxel scalar values ​​to corresponding transparency / color values ​​allows for different rendering focuses to be set regarding which types of object properties are rendered.

[0032] The projection image updater module 250 of the MVR system updates (1400) the value of a particular pixel in the projection image 30 by combining the color and transparency values ​​(i.e., the mapped RGBA values ​​of each voxel) of each individual view ray. In this regard, updating each projection image pixel value with the value of a particular voxel intersecting with a particular view ray is performed after updating with the values ​​of all intersecting voxels that are closer to the viewpoint than the particular voxel. That is, each view ray is allowed to proceed rapidly until it encounters a non-transparent voxel in its associated volume. However, processing of non-transparent voxels must be synchronized with all rays processing the same location in the 3D scene to achieve correct results for a particular pixel value. The generation (1200), individual processing (1300), and update (1400) steps are performed (1500) by each module of the MVR system 200 for all pixels in the projection image 30. For example, the steps may be repeated in a loop that checks 1450 whether there are still any unupdated pixels of the projection image after each iteration. If the projection image is not yet complete, a further loop iteration is initiated for the unupdated pixels. Once the projection image is complete, the method 1000 ends. Alternatively, multiple pixels of the projection image may be processed in parallel if parallel processing is supported by the MVR system 200. As an example, the processing of individual pixels may be implemented as a shader program executed by a graphics processing unit. The updating of each projection image pixel value may be advantageously implemented such that the combined value is stored intermediately in a (local) variable during the processing of the projection image pixel. In other words, storing the combined value intermediately in a (local) variable during the processing of the projection image pixel and updating each projection image pixel only at the end of the process can improve performance in some cases. Once the processing of the view multiray 20 is complete, the final value of that variable is written to the projection image 30 as the respective projection image pixel value.Storing updated projected image pixel values ​​in local variables during the ray marching process is computationally more efficient than writing the current combined values ​​to the projected image after each ray marching step.

[0033] FIG. 6 illustrates direct volume rendering in more detail with an example. In direct volume rendering (i.e., calculating projection images directly from volumetric data), the MVR system does not know where visible tissue surfaces (e.g., scene objects SO6) are located within each volume. Therefore, simple ray-surface collision calculations cannot be performed. The direct rendering approach can only determine collisions of view rays with the bounding box BB6 of each volume (e.g., a CT volume). Therefore, multiple view rays VR6* (one for each pixel in the projection image) are transmitted through the 3D scene from the viewpoint position 10. Ray marching is performed with each view ray within the volume defined by the bounding box BB6. The ray marching step length ls (i.e., the distance between two ray marching steps) is about the size of a voxel. For example, the ray marching step length can be set to be equal to or less than the diagonal length of a voxel. At such small steps along the view ray, the raw voxel data is read at each ray marching step position (indicated by black and white dots) and a respective transfer function can be applied to determine the color and transparency (or alternatively, opacity, where: transparency value = 1 - opacity value) for each voxel. Typically, the voxel color and transparency values ​​are represented in the so-called RGBA format, where the RGB channels represent the color value and the alpha value in the A channel represents the opacity (0% = fully transparent, 100% = fully opaque).

[0034] Typically, voxels close to bounding box BB6 are transparent (e.g., air). Therefore, ray marching can simply continue until the system finds a visible voxel. Typically, when a view ray intersects a visible voxel (e.g., bone or soft tissue), the projection image updater begins accumulating the color and transparency of the visible voxel with subsequent visible voxels. In other words, the color and transparency values ​​of visible voxels along the view ray are combined with the color values ​​of each pixel in the projection image. This accumulation continues until the opacity channel (alpha value in RGBA) of the current pixel reaches a threshold close to 100%. For example, the opacity threshold can be predefined in the range between 90% and 99.9%. As a result, anything the view ray may encounter from that point on will be hidden by previously intersected voxels and will not be visible anyway. That is, the ray marching process can be stopped at this point. This action is often referred to as early ray termination. In the example of Figure 6, the black dots represent ray marching steps where the view ray intersects an invisible voxel. White dots indicate ray marching steps where view rays intersect with visible voxels belonging to scene object S06. For each white dot, the aforementioned accumulation (combination) of mapped voxel values ​​is performed. In the example, all view rays penetrating scene object S06 continue the ray marching step after the object; that is, either early ray termination did not work or none of the illustrated rays reached the opacity threshold for early ray termination.

[0035] In the following, a detailed example is described in view of Figure 1A, without intending to limit the scope of the claims in any way. In particular, when combining mapped voxel values ​​within a region of overlapping volumes, the combination is always performed in an order where the voxels closest to the viewpoint position are combined first. Advantageously, the individual view rays are processed "in lockstep", i.e., the processing for the view rays (especially within overlapping volume regions) proceeds at exactly the same speed and in the same direction.

[0036] First, the view rays 21-24 are initialized by generating the view multi-ray 20. The MVR system then determines the position for the first ray marching step for each (individual) view ray within its associated volume. In the example, the first ray marching step is: -V1 occurs at 8 steps (position '8') from viewpoint position 10, -V2 occurs at 13 steps from viewpoint position 10 (position '13'), -V3 occurs at 23 steps from viewpoint position 10 (position '23'), -In V4, it occurs at 29 steps (position '39') from viewpoint position 10. It is assumed that these first ray marching step positions within each volume can be stored as a number of step lengths from the viewpoint position for each view ray. The minimum of these first ray marching step positions is the view multi-ray distance (called the multi-ray position) from the viewpoint position. That is, in the example, the system knows that all view rays have a first ray marching step position that is at least eight step lengths from the viewpoint position. The multi-ray position is then incremented by "1" after the completion of each ray marching step. Using this information, the MVR system can determine at any time whether the mapped voxel values ​​on a particular view ray can be combined without risk, or whether the system needs to wait further for combination because mapped voxel values ​​closer to the viewpoint position may be opaque and risk obscuring all voxels further away from the viewpoint position, or may have non-transparent color values ​​that need to be combined first to obtain the correct final pixel color. Also, in the case of semi-transparent pixels, the order in which voxel values ​​are combined is important to arrive at the correct pixel color.

[0037] The volume ray marching processor iterates over all view rays 21-24 at each ray marching step (i.e., over all volumes V1-V4 to be rendered in the projection image). The order of the iterations is irrelevant. In the following, the iteration order V2 → V1 → V4 → V3 is assumed. In the examples described below, where multiple iterations are described under a single heading, it should be understood that in each iteration, all volumes are processed in the aforementioned iteration order before the system performs the next iteration.

[0038] In an optional embodiment, the MVR system may store a distance map that can be calculated during the volume loading process. During this loading process, and given one or more transfer functions, the system may calculate, for each voxel in each volume, the distance to the nearest visible voxel within the volume and store the calculated distance in the distance map. During volume ray marching, the ray marching processor of the MVR system can obtain the distance to the next visible voxel at each location within each volume from the distance map to determine whether the next ray marching step can be performed with multiple step lengths (ray marching jumps). Once such distance is less than or equal to the ray marching step length, the scalar data of each voxel is loaded and processed with the corresponding transfer function. The distance map allows transparent voxels within the volume to be skipped during ray marching, thereby accelerating the ray marching process.

[0039] First Iteration V2: The view ray 22 begins the first ray marching step at position "13". Upon reading the distance map, the system recognizes that the visible voxels are close and a ray marching jump cannot be performed. After reading the scalar voxel values ​​and applying the transfer function, the ray marching processor determines that the voxel is transparent and therefore no combination of mapped color / transparency values ​​is necessary. As a result, the view ray 22 moves to the next ray marching step at position "14".

[0040] V1: For view ray 21, the situation is similar for view ray 22. View ray 22 moves to the next ray marching step at position "9".

[0041] V4: For view ray 24, at the edge of V4, the system derives from the distance map that all voxels within a radius of two steps length are transparent. Therefore, the position of view ray 24 can be safely increased from "39" to "41".

[0042] V3: For view ray 23, the situation is similar to view rays 22 and 21, the position can be incremented by 1 up to position "24".

[0043] After iterating for all view rays, the multi-ray position is incremented by 1 from "8" to "9" for the next iteration. After the first iteration, the ray marching step positions of all view rays for the second iteration are already within their respective scene objects, so further ray marching jumps are not allowed due to their respective distance values ​​in the distance map being "0" (as long as each view ray is inside the scene object).

[0044] Second Iteration Starting again with V2, at the current position "14" of the view ray 22, the ray marching processor reads the first visible voxel of V2, which belongs to the scene object 121. However, the current position "14" of the view ray 22 is further away from the viewpoint than the current multi-ray position "9". Therefore, each mapped voxel value at the current position "14" must not yet be combined with a respective pixel in the projection image. The view ray 22 at its current position must wait until the multi-ray position also reaches "14".

[0045] V1: View ray 21 reaches a visible voxel at current position "9" that belongs to scene object 111. Each mapped voxel value is determined using a respective transfer function and combined with each projected image pixel since there is no other view ray whose current position is closer to the viewpoint position. The current position of view ray 21 is increased to position "10".

[0046] V4: The current position "41" of view ray 24 still intersects with a transparent voxel (→ no combined mapped value). The distance map indicates that no further ray marching jumps are possible. Therefore, the current position is increased to position "42".

[0047] V3: View ray 23 reaches a visible voxel at current position "24" that belongs to scene object 131. Each mapped voxel value is determined using a respective transfer function. However, as with view ray 22, each mapped voxel value at current position "24" must not yet be combined with a respective pixel in the projection image. View ray 23 at its current position must wait until multi-ray position "24" also reaches. That is, volumes V1 and V2, which are closer to the viewpoint, must be processed up to position "24" before the mapped voxel values ​​of view ray 23 can be combined with the projection image pixel values.

[0048] After all view rays have been processed in the second iteration, the multiray position is incremented to "10".

[0049] 3rd, 4th and 5th iterations V2: At its current position "14", the view ray 22 needs to wait.

[0050] V1: View ray 21 reaches three further visible voxels at current positions "11," "12," and "13," which belong to scene object 111. Respective mapped voxel values ​​are determined at each position using the respective transfer functions and combined with the respective projected image pixels after each ray marching step, since there are no other view rays whose current positions are closer to the viewpoint position. The current position of view ray 21 is incremented to position "14" in the fifth iteration. Position "14" already overlaps with V2, which would allow combining, but no combining with V2 voxel values ​​occurs, since the voxels intersecting with view ray 22 were transparent.

[0051] V4: At its current position "42", the Viewray 24 needs to wait.

[0052] V3: At its current position "24", the view ray 23 needs to wait.

[0053] After all view rays have been processed in the third, fourth and fifth iterations, the multiray position is "14".

[0054] Iterations 6 to 11 In the sixth iteration, view ray 21 reaches the current position of view ray 22. Both view rays now have the same distance from the viewpoint position. Therefore, for V2 and V1, their mapped voxel values ​​are determined at ray marching step positions "14" through "19" (the last positions of view ray 22 intersecting with scene object 121) and combined with the projected pixel values ​​in the respective images. In the eleventh iteration, the current positions of view rays 21 and 22 are incremented to position "20". Position "20" is already outside V2; that is, the ray marching processor no longer considers view ray 22 for further processing.

[0055] V4 and V3: Viewray 24 and 23 are still waiting.

[0056] After all view rays have been processed in the 11th iteration, the multiray position is incremented to "20".

[0057] Iterations 12 to 14 V1: At positions "20"-"22", only the view ray 21 intersects with a visible pixel of its associated volume (scene object 111). Therefore, each mapped voxel value is associated with each image projection pixel. The current position of the view ray 21 is incremented to position "23" in the 14th iteration.

[0058] V4 and V3: Viewray 24 and 23 are still waiting.

[0059] After all view rays have been processed in the 14th iteration, the multiray position is incremented to "23".

[0060] 15th iteration V1: View ray 21 is currently at position "23" and still intersects with visible voxels of scene object 111. At position "23", V1 overlaps with V3, but voxels in V2 at position "23" are not visible, and view ray 23 has already advanced to position 24 (while the multi-ray position is still at position 23), so only the mapped voxel values ​​of view ray 21 are combined with pixel values. The current position of view ray 21 is incremented to "24".

[0061] Viewray 23 and 24 are still waiting.

[0062] The multi-ray position is incremented to "24".

[0063] Iterations 16 and 17 V1: In the 16th and 17th iterations, the view ray 21 reaches positions "24" and "25", where it still intersects with the visible voxels of V1. Thus, each mapped voxel value of the scene object 111 is combined with each projection image pixel. The current position of the view ray 21 is set to "26" at the end of the 17th iteration.

[0064] V4: Viewray 24 is still on hold.

[0065] V3: The current position of the view ray 23 is equal to the multi-ray position "24" at the start of the 16th iteration. Therefore, the mapped voxel values ​​of the view ray 23 can now be associated with each projection image pixel, since other voxels closer to the viewpoint position cannot occlude the visible voxels of the scene object 131. The current position of the view ray 23 is incremented to "25", and the same process continues for the visible voxels at this position. The current position of the view ray 23 is set to "26" at the end of the 17th iteration.

[0066] The multiray position is also set to "26" at the end of the 17th iteration.

[0067] Iterations 18 to 21 V1: From position "26" onwards, all remaining voxels inside V1 intersected by view ray 21 are invisible (to the right of scene object 111). Therefore, no more combining of mapped voxel values ​​is needed. The current position at the end of the 21st iteration is incremented to "29", which is outside V1. Therefore, view ray 21 is no longer considered for further ray marching processing after the 21st iteration. The example shows a situation using a ray marching jump. In this spatial skip implementation, the ray marching step that could already occur at position "28" is skipped.

[0068] V4: Viewray 24 is still on hold.

[0069] V3: The corresponding voxels on the view ray 23 are visible voxels of the scene object 131. Thus, each mapped voxel value is associated with each projection image pixel. The current position of the view ray 23 at the end of the 21st iteration is set to "29".

[0070] The multiray position at the end of the 21st iteration is set to "29".

[0071] 22nd iteration V4: Viewray 24 is still on hold.

[0072] V3: View ray 23 is at the last visible voxel of scene object 131. Hence, the mapped voxel values ​​are combined and the current position is incremented to '30'.

[0073] The multi-ray position is set to "30".

[0074] 23rd iteration V4: Viewray 24 is still on hold.

[0075] V3: At position "30", the view ray 23 intersects with the last (invisible) voxel of V3. Therefore, no value is associated with the projected image pixel. The current position is incremented to "31", which is outside V3. Therefore, V3 is no longer considered by the ray marching processor.

[0076] The multiray position is set to 30. Alternatively, when leaving a volume, the multiray position can be updated with the minimum of the view ray positions of the volumes not yet processed, which in the current example is the current position of view ray 24, 42.

[0077] 24th iteration V4: V4 is now the only volume remaining for ray marching. Although multi-ray position "30" is still closer to the viewpoint than the current position "42" of the view ray 24, the ray marching processor can now continue processing the view ray 24 since it has processed all volumes closer to the viewpoint. Therefore, the current multi-ray position henceforth corresponds to the current position of the view ray 24. The voxel in V4 at position "42" is the first visible voxel of the scene object 141. Therefore, a merge is performed and the current position of the view ray 24 is incremented to "43".

[0078] Repeats 25-29 V4: At positions "43" to "47", the view ray 24 intersects with visible voxels belonging to the scene object 141. Each mapped voxel value is combined with a projected image pixel after each ray marching step of the view ray 24. The current position is incremented to "48" at the end of the 29th iteration.

[0079] At position "48," the last voxel in volume V4 is already an invisible voxel outside the scene object 141, and no combining occurs. Therefore, the ray marching process for the entire multi-ray 20 has been successfully completed, and all mapped voxel values ​​from volumes V1 through V4 have been combined with their respective projection image pixels in the correct order to ensure that voxels far from the viewpoint cannot erroneously affect the projection image pixel values, even if they are obscured by voxels closer to the viewpoint. Nevertheless, processing individual view rays allows for efficient processing of 3D scenes with multiple volumes, especially in scenes with overlapping volumes. Of course, the multi-volume rendering approach disclosed herein works similarly for non-overlapping volumes (e.g., two adjacent volumes). In this case, the volumes can be rendered one after the other, starting with the volume farthest from the viewpoint, using standard alpha compositing to obtain the final image. This works because at each point in the 3D scene, only one of the volumes is present to affect the color of view rays passing through that part of the 3D scene. However, with overlapping volumes that have semi-transparent overlapping scene objects, the simple volume-by-volume approach no longer works. Instead, the multi-ray approach described above can be used to render all volumes of the 3D scene at once, thus solving this problem. Furthermore, decoupling individual view rays to their associated volumes allows for even faster rendering of some more complex cases where view rays may need to slow down in different areas of the scene. Typical reasons for rays slowing down (i.e., not being able to make greater progress, e.g., by using spatial skipping) are noisy regions in the volume (e.g., due to low-quality images or many small structures) or when view rays pass close to visible voxel structures without actually passing through anything visible.

[0080] Particularly in medical applications, multiple volumes with 3D data from different sources can be overlaid on top of each other in a 3D scene. For example, a CT volume and an MRI volume can be overlaid in a 3D scene so that the volumes overlap. This is typically the case when each scan covers the same part of a patient's body (e.g., the patient's head or chest). CT generally produces high-resolution images without geometric distortion, which are well-suited for visualizing bone. However, certain other tissues or pathologies are not well-visible using CT data. To fill this gap, an MRI volume can be generated in addition to the CT volume. For example, the MRI volume can capture the tumor and be registered with the CT volume so that it appears in the correct position relative to the bone. In a further use case, pre- and post-operative CT scans can be rendered side-by-side to evaluate surgical results. In yet a further use case, a single CT volume can be divided into multiple smaller volumes, each containing one or more of the scene objects in the original CT volume. The scene object(s) in one smaller volume can then be individually moved relative to the scene objects in the other smaller volumes. This can be used, for example, to check the correction of a fracture by repositioning the bone (ie, by repositioning a small volume containing the bone fragments as scene objects).

[0081] Standard spatial skipping techniques (e.g., Euclidean distance maps, minimax octrees) can be used to accelerate view rays within free regions of a volume. Spatial skipping is applied to individual view rays. This means that view rays (for the same projection image pixel) for different associated volumes may become out of sync, with some view rays possibly being further ahead than others. We have previously described how this problem can be addressed by ensuring that visible voxels closer to the viewpoint are processed before visible voxels that are further away. With spatial skipping, performance optimization can be achieved by using additional information to skip as many invisible regions of each volume as possible (e.g., invisible voxels that typically represent air). Spatial skipping techniques are known in the art. However, in the approach disclosed herein, spatial skipping is applied separately to each individual view ray of the view multi-ray.

[0082] One possible approach to spatial skipping is to perform some pre-calculation on the 3D data while it is being loaded into the scene for the first time. For each single voxel in a particular volume, the distance to the nearest non-transparent voxel can be pre-calculated. In other words, the system determines the radius of the largest sphere that can be placed at a location that contains only invisible voxels. This data structure is called a distance map. Figure 7 shows two visible scene objects 710, 720 in a volume 700. The camera at the center of the sphere 711 represents the viewpoint position. The view ray 701 represents a ray that passes through the volume and corresponds to a single pixel in the projected image. As mentioned above, in one embodiment of the ray marching process, the distance map can be used at each ray marching step to determine whether a larger ray marching jump is possible to skip over empty regions of the volume 700 while ensuring that no visible voxels are skipped. The first ray marching step is positioned at the center of the sphere 711 at the viewing distance in front of the camera. The value stored in the pre-computed distance map at this location is the length of dashed line r1, which is the radius of the largest sphere that can be placed at this location without containing any visible voxels. The visible voxel closest to the center of sphere 711 is part of scene object 720 and is at the intersection of dashed line r1 and sphere 711. Therefore, in the first ray marching step, a jump equal to the length of r1 can be made along view ray 701, ending at the center of sphere 712. In step 2, the value read from the distance map is the length of dashed line r2, which corresponds to the distance between the center of sphere 712 and the closest visible voxel (part of scene object 710). Therefore, in step 2, a jump equal to the length of r2 can be made. This process is repeated until a value is read from the distance map, which does not guarantee a spatial skip jump. This is typically a distance less than or equal to the ray marching step size. In the example shown, the process is repeated for sphere 713 of radius r3, sphere 714 of radius r4, and three additional smaller spheres.At the last example position along the view ray 701, the distance to the nearest visible voxel (part of the scene object 720) is too small to allow a spatial jump. Therefore, at this point, the ray marching process reverts to the default implementation without spatial skipping, as described in the previous section. Note that with spatial skipping, the distance map is always read before reverting to the default implementation. Therefore, it is possible for spatial jumps to occur in disjoint segments along the view ray. For example, the view ray may pass through a semi-transparent scene object, in which case spatial skipping can be used both before and after passing through the scene object, or not while passing through it.

[0083] FIG. 8 is a schematic diagram illustrating an example of a general-purpose computing device 900 and a general-purpose mobile computing device 950, which may be used with the techniques described herein. In an exemplary embodiment, computing device 900 is associated with MVR system 200 (see FIG. 2). Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices, such as virtual reality glasses. Computing device 950 may be used to display projected images to a user as a result of direct multi-volume rendering. In the case of a user using a virtual reality display device, two projected images are rendered that reflect different viewing angles of the user's eyes. The components shown here, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the scope of the present invention as described and / or claimed herein.

[0084] Computing device 900 includes a processor 902, memory 904, storage device 906, a high-speed interface 908 connecting to memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connecting to a low-speed bus 914 and storage device 906. Each of the components 902, 904, 906, 908, 910, and 912 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 902 processes instructions for execution within computing device 900, including instructions stored in memory 904 or on storage device 906, and can display graphical information for a GUI on an external input / output device, such as a display 916 coupled to high-speed interface 908. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types as desired. Multiple computing devices 900 may also be connected, with each device providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0085] Memory 904 stores information within computing device 900. In one implementation, memory 904 is a volatile memory device or devices. In another implementation, memory 904 is a non-volatile memory device or devices. Memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0086] The storage device 906 is capable of providing mass storage for the computing device 900. In one embodiment, the storage device 906 can be or include a computer-readable medium, such as a floppy disk drive, hard disk drive, optical disk drive, or tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 904, the storage device 906, or memory on the processor 902.

[0087] High-speed controller 908 manages operations that consume large amounts of bandwidth for computing device 900, while low-speed controller 912 manages operations that require less bandwidth. Such an allocation of functionality is merely exemplary. In one embodiment, high-speed controller 908 is coupled to memory 904, a display 916 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 910, which may accept various expansion cards (not shown). In an embodiment, low-speed controller 912 is coupled to storage device 906 and a low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, ZigBee, WLAN, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, through a network adapter.

[0088] Computing device 900, as shown in the figure, may be implemented in several different forms. For example, it may be implemented as a standard server 920, or multiple times within a group of such servers. It may also be implemented as part of a rack server system 924. In addition, it may be implemented within a personal computer, such as a laptop computer 922. Alternatively, components from computing device 900 may be combined with other components within a mobile device (not shown), such as device 950. Each such device may include one or more of computing devices 900, 950, and the entire system may be composed of multiple computing devices 900, 950 in communication with each other.

[0089] Computing device 950 includes, among other components, a processor 952, memory 964, input / output devices such as a display 954, a communication interface 966, and a transceiver 968. Device 950 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and some of the components may be mounted on a common motherboard or otherwise as desired.

[0090] The processor 952 can execute instructions within the computing device 950, including instructions stored in the memory 964. The processor may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor may be provided for coordination of other components of the device 950, such as, for example, the user interface, applications run by the device 950, and control of wireless communications by the device 950.

[0091] The processor 952 may communicate with a user through a control interface 958 and a display interface 956 coupled to a display 954. The display 954 may be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 956 may include appropriate circuitry for driving the display 954 to present graphical and other information to the user. The control interface 958 may receive commands from the user and convert them for submission to the processor 952. Additionally, an external interface 962 may be provided in communication with the processor 952 to enable near-field communication of the device 950 with other devices. The external interface 962 may be provided, for example, for wired communication in some implementations or for wireless communication in other implementations, although multiple interfaces may also be used.

[0092] Memory 964 stores information within computing device 950. Memory 964 may be implemented as one or more of a computer-readable medium or media, a volatile memory device or devices, or a non-volatile memory device or devices. Expansion memory 984 may also be provided and connected to device 950 through an expansion interface 982, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 984 may provide additional storage capacity for device 950 or may also store applications or other information for device 950. Specifically, expansion memory 984 may include instructions for performing or supplementing the processes described above and may also include security information. Thus, for example, expansion memory 984 may function as a security module for device 950 and be programmed with instructions that permit secure use of device 950. Additionally, secure applications may be provided via SIMM cards along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0093] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 964, expansion memory 984, or memory on processor 952, which may be received, for example, via transceiver 968 or external interface 962.

[0094] Device 950 may communicate wirelessly through communication interface 966, which may include digital signal processing circuitry if necessary. Communication interface 966 may provide communications based on various modes or protocols, such as GSM voice telephony, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communications may occur, for example, through wireless transceiver 968. In addition, short-range communications may occur, such as using Bluetooth, WiFi, ZigBee, or other such transceivers (not shown). In addition, GPS (Global Positioning System) receiver module 980 may provide additional navigation- and location-related wireless data to device 950, which may be used by applications executing on device 950, as needed.

[0095] Device 950 may also communicate audibly using an audio codec 960, which may receive audio information from a user and convert it into usable digital information. Audio codec 960 may also generate audible sounds for the user, such as through a speaker in the handset of device 950. Such audio may include audio from a voice call, recorded audio (e.g., voice messages, music files, etc.), and may also include audio generated by applications running on device 950.

[0096] The computing device 950 may be implemented in several different forms, as shown in the figure. For example, it may be implemented as a mobile phone 980. It may also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile device.

[0097] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0098] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0099] To provide for user interaction, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0100] The systems and techniques described herein can be implemented on a computing device that includes back-end components (e.g., as a data server), or includes middleware components (e.g., as an application server), or includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0101] Computing devices may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0102] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the scope of the present invention.

[0103] Additionally, the logic flows depicted in the figures do not require the particular order or sequence shown to achieve desired results. Additionally, other steps may be provided or steps may be removed from the described flows, and other components may be added or removed from the described systems.

Claims

1. receiving 3D data (1100) specifying a three-dimensional scene (100) comprising a plurality of volumes (110, 120, 130, 140), each volume comprising a plurality of voxels (pv_n), each voxel of a particular volume having a scalar value resulting from an imaging procedure for measuring a physical property of an imaging subject; generating (1200) a view multi-ray (20) at a viewpoint (10) in the direction of a specific pixel of a projection image (30), the projection image being located between the viewpoint (10) and the scene (100), the specific pixel having a value representing the projection of all scene voxels intersected by the view multi-ray after the projection image (30), the view multi-ray (20) representing a plurality of individual view rays (21, 22, 23, 24) of the same starting point and the same direction, each individual view ray being associated with one of the volumes (110, 120, 130, 140) within the scene (100); using volumetric ray marching to iterate over all individual view rays to individually process (1300) each view ray (21, 22, 23, 24) within its associated volume in each ray marching step, wherein the ray marching step for each view ray is performed only within the associated volume, and in each ray marching step, the scalar value of the corresponding voxel is mapped to a color value and a transparency value of the corresponding voxel; updating (1400) the value of the particular pixel in the projection image (30) by combining the color and transparency values ​​of the respective voxels of the individual view rays after each ray marching step, wherein updating with the value of a particular voxel intersecting with a particular view ray is performed after updating with the values ​​of all intersecting voxels of voxels closer to the viewpoint than the particular voxel, and processing of non-transparent voxels of overlapping volumes at the same location is performed synchronously for each individual view ray; performing (1500) the generating (1200), processing (1300) and updating (1400) steps on pixels of the projection image (30) that have not yet been updated; A computer-implemented method (1000) for direct multi-volume rendering, comprising:

2. 2. The method of claim 1, wherein volumetric ray marching uses spatial skipping in that the individual processing of a particular view ray skips ray marching steps inside the associated volume that are associated with fully transparent voxels.

3. 10. The method according to any of the previous claims, wherein the imaging procedure is an image acquisition or image generation process resulting in a volumetric image.

4. 10. The method of any of the previous claims, wherein the imaging procedure is selected from any of computed tomography, magnetic resonance tomography, optical coherence tomography, positron emission tomography, single photon emission computed tomography, and 3D ultrasound imaging.

5. The method according to any one of claims 1 to 4, wherein the physical property is selected from absorption of X-ray radiation, number of protons, optical transparency, positron emission of injected radiopharmaceuticals, and acoustic impedance.

6. 10. The method of any of the previous claims, wherein two or more of the plurality of volumes are aligned with respect to each other such that the origin of the local coordinate system of a first volume is known within a second volume, and vice versa.

7. 10. The method of claim 9, wherein at the start of the ray marching process for each view ray, the respective view ray is transformed from a global coordinate system (xg, yg, zg) into a local coordinate system (xn, yn, zn) of its associated volume (110, 120, 130, 140).

8. 13. The method of any of the previous claims, wherein the individual processing (1300) of the first view ray (21) uses a first predefined transfer function for mapping scalar values ​​of voxels of the respective associated volumes (110) to respective color and transparency values, and the individual processing (1300) of the second view ray (22) uses a second predefined transfer function for mapping scalar values ​​of voxels of the respective associated volumes (120) to respective color and transparency values.

9. 1. A computer program product for direct multi-volume rendering, comprising computer readable instructions that, when loaded into a memory of a computing device and executed by at least one processor of said computing device, cause said computing device to perform the method according to any one of the preceding claims.

10. an interface (210) adapted to receive 3D data specifying a three-dimensional scene (100) comprising a plurality of volumes (290), each volume comprising a plurality of voxels (pv_n), each voxel of a particular volume having a scalar value that is the result of an imaging procedure for measuring a physical property of an imaging subject; a multi-ray generator (230) configured to generate, at a viewpoint (10), a view multi-ray (20) in the direction of a specific pixel of a projection image (30), the projection image being located between the viewpoint (10) and the scene (100), the specific pixel having a value representing the projection of all scene voxels intersected by the view multi-ray after the projection image (30), the view multi-ray (20) representing a plurality of individual view rays (21, 22, 23, 24) of the same starting point and the same direction, each individual view ray being associated with one of the volumes (110, 120, 130, 140) within the scene (100); a volume ray marching processor (240) configured to use volume ray marching by iterating over all individual view rays to individually process each view ray (21, 22, 23, 24) within its associated volume in each ray marching step, wherein the ray marching step for each view ray is performed only within the associated volume, and wherein in each ray marching step, the scalar values ​​of the corresponding voxels are mapped to color and transparency values ​​of the corresponding voxels; a projection image updater (250) configured to update the value of the particular pixel in the projection image (30) by combining color and transparency values ​​of the respective voxels of the individual view rays after each ray marching step, wherein updating with the value of a particular voxel intersecting with a particular view ray is performed after updating with the values ​​of all intersecting voxels of voxels closer to the viewpoint than the particular voxel, and processing of non-transparent voxels of overlapping volumes at the same location is performed synchronously for each individual view ray; A computer system (200) for direct multi-volume rendering, comprising:

11. 11. The system of claim 10, wherein the volume ray marching processor (240) employs spatial skipping in that the individual processing of a particular view ray skips ray marching steps inside the associated volume that are associated with fully transparent voxels.

12. The system of any of claims 10 to 11, further comprising a volume registration module (220) configured to register one or more of the plurality of volumes with respect to one another, so that the origin of the local coordinate system of a first volume is known in a second volume, and vice versa.

13. 13. The system of claim 10, wherein the volume ray marching processor (240) further comprises a coordinate system transformer (242) that, at the start of the ray marching process for each view ray, transforms the respective view ray from a global coordinate system (xg, yg, zg) to a local coordinate system (xn, yn, zn) of its associated volume (110, 120, 130, 140) for volume ray marching within the associated volume.

14. 14. The system of claim 10, wherein the volume ray marching processor uses a first predefined transfer function for mapping scalar values ​​of voxels of the respective associated volumes to respective color and transparency values ​​when processing (1300) first view rays (21) individually, and uses a second predefined transfer function for mapping scalar values ​​of voxels of the respective associated volumes to respective color and transparency values ​​when processing (1300) second view rays (22) individually.

15. The system of any one of claims 10 to 14, wherein the 3D data for a particular volume further comprises a distance map storing, for each voxel of the particular volume, a distance to a next non-transparent voxel in the volume, and wherein ray marching for a particular view ray performs ray marching jumps including a plurality of the distances between two ray marching steps according to the distance map, so that the next ray marching step closest to the next non-transparent voxel is performed.

Images