Systems and methods for multi-volume rendering

Abstract

A system (100), method, and computer program product for direct multi-volume rendering. The 3D data specifies a three-dimensional scene having multiple volumes (290). Each voxel in a particular volume has a scalar value that is the result of an imaging procedure for measuring physical properties of the imaged subject. A multi-ray generator (230) generates a view multi-ray in the direction of a particular pixel in the projection image, which represents the 2D projection of all scene voxels intersected by the view multi-ray after the projection image. A volume ray marching processor (240) processes each view ray in its associated volume using volume ray marching, and at each ray marching step, the scalar value of a corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. A projection image updater (250) updates the value of a particular pixel in the projection image by combining the color and transparency values ​​of each voxel in each view ray. Updating the value of a particular voxel intersecting a particular view ray is performed after updating the values ​​of all intersecting voxels that are closer to the viewpoint than the particular voxel.

Description

Technical Field

[0001] The present invention generally relates to the rendering of images showing 3D scenes, and more particularly to the rendering of volumes using multiple volumes.

Background Art

[0002] In many software applications, such as medical applications, computer graphics algorithms are used to generate 3D representations of a patient's organs. Also, in computer games or other 3D applications running within a virtual 3D landscape, the 3D scene needs to be rendered into an image and displayed to the human user(s) of the application(s). The rendering of such 3D scenes, particularly in medical applications, is typically performed by using an image-based volume rendering technique often called volume raymarching. Raymarching uses volume data. Other techniques, such as raytracing and raycasting, calculate the intersection of rays with surface models of 3D objects. Raymarching does not use such surface models of 3D objects. It only knows the cuboid bounding box of the volume and cannot directly calculate the intersection of the view ray with the surface of the 3D object within the volume. For this reason, raymarching performs small raymarching steps to sample each voxel.

[0003] In its basic form, the volume raymarching algorithm includes four steps: - Raycasting. For each pixel in the final image, a viewing ray (view ray) is sent through the volume to be rendered. Typically, the volume corresponds to a simple geometric object (e.g., a cuboid). - Sampling. Equidistant sampling points or samples are selected along the portion of the viewray within the volume. Generally, the volume does not necessarily coincide with the viewray. Therefore, the sampling points do not necessarily coincide with the center position of the voxels. Thus, the sample values ​​may be interpolated from adjacent voxels. - Shading. For each sampling point, the sampling value is mapped to the RGBA material color, opacity / transparency, and optionally other properties. -Combine. All sampling points are combined along the viewray to obtain the final color value for the currently processed pixel.

[0004] The fundamental paper disclosing the concept of ray marching is Levoy, Marc. "Display of surfaces from volume data." IEEE Computer graphics and Applications 8.3(1988):29-37. Conventional solutions have a challenge in efficiently performing multi-volume rendering directly, especially for 3D scenes with overlapping volumes. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, there is a need to provide an efficient solution for direct multi-volume rendering of 3D scenes. As used herein, direct rendering means rendering images directly from raw data. This term will be further explained in the context of different raw data types. [Means for solving the problem]

[0006] The aforementioned problem is resolved by the embodiment claimed by the independent claim: a computer implementation method for direct multi-volume rendering, a computer program product comprising instructions configured to cause the computer program method to be executed when executing the computer program method, and a computer system capable of loading and executing the computer program product to execute the method.

[0007] A computer system for direct multi-volume rendering is also referred to below as an MVR system. An MVR system includes functional modules capable of performing functions to carry out each step of the method disclosed herein. Through a suitable interface, the MVR system receives 3D data specifying a three-dimensional scene. Hereinafter, 3D means three dimensions, and 2D means two dimensions. A 3D scene includes multiple volumes to be rendered by the MVR system, as used herein. A typical 3D dataset is, for example, a series of 2D slice images acquired by a computed tomography (CT), magnetic resonance imaging (MRI), X-ray microtomography (MicroCT), or other 3D imaging technique. The 3D data received by the MVR system corresponds to the aforementioned raw data. Such individually sampled 3D datasets are typically referred to as volumes. A 3D scene includes multiple volumes. In this regard, each volume includes multiple voxels. Each voxel in a particular volume has a scalar value which is the result of a volumetric imaging procedure for measuring the physical properties of the imaging subject. Hereinafter, the imaging subject is the object that is the subject of the volumetric imaging procedure. Examples of imaging subjects include biological samples (e.g., samples from humans, animals, or plants) or non-biological samples (e.g., components of machines or devices). In other words, the raw data used for direct rendering is a collection of voxels in a volume, each voxel having a scalar value such that. Examples of such imaging procedures (imaging techniques) include, but are not limited to, computed tomography, magnetic resonance imaging, optical coherence tomography, positron emission tomography, single-photon emission computed tomography, and 3D ultrasound imaging, or any image acquisition or generation process that results in a volumetric image. Using an appropriate imaging procedure, physical properties such as, for example, absorption of X-ray irradiation, number of protons, light transmittance, positron emission of injected radiopharmaceuticals, changes in acoustic impedance, or other material properties can be measured. Those skilled in the art will know which imaging techniques can be used to measure the respective physical properties of an imaging subject.

[0008] An MVR system has a multi-ray generator module to generate view multi-rays at the viewpoint position in the direction of specific pixels in the projected image. The viewpoint position typically represents the position of a virtual camera or virtual reality device used by the user to observe the 3D scene. The projected image is a 2D projection of the 3D scene as it appears from the viewpoint projection. The projected image is the image displayed on a display device, such as a monitor. In cases where a virtual reality device is used to display the image, two projected images are generated, which are suitable for generating 3D VR effects. The projected image is placed between the viewpoint position and the scene. Typically, the projected image is placed at a viewing distance of about 10 cm in front of the viewpoint position. With respect to this, a specific pixel (through which the view multi-ray is transmitted into the 3D scene) represents the 2D projection of all scene voxels intersecting the view multi-ray behind the projected image. A view multi-ray represents multiple individual view rays from the same starting point and in the same direction, each individual view ray associated with one of the volumes in the scene. In other words, each individual viewray (hereinafter also referred to as a viewray) belongs to exactly one of the volumes loaded into the scene.

[0009] The MVR system further includes a volume ray marching processor module that uses volume ray marching to process each view ray within its associated volume individually. Processing each view ray within its associated volume individually involves determining the color and transparency values ​​within the associated volume in each ray marching step. That is, the ray marching step for each view ray is performed only within the associated volume. As a result, the number of ray marching steps with a given step length (i.e., the distance between two ray marching steps) for a particular view ray is limited by the boundaries of the associated volume. For example, the first ray marching step for a particular view ray may occur over multiple step lengths, where the view ray first intersects the associated volume. A view ray intersects the associated volume in the current ray marching step where the view ray first intersects the associated volume, or where each location outside the associated volume is already associated with a visible color value. In some cases, even ray marching locations outside the volume—but very close to it—can already become visible color values ​​for voxel interpolation. At each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. In one embodiment, such mapping can be implemented using a predefined transfer function. There may be different transfer functions for each volume (or even for different subregions within a particular volume), depending on the imaging technique used to create each associated volume. For example, in the case of a CT scan, the scalar values ​​are measured in Hunsfield units (HU). Each transfer function may map HU values ​​corresponding to air (black) and soft tissue (gray) to transparent, and HU values ​​corresponding to bone (bright) to white / beige. Typically, the scalar values ​​can be interpreted as grayscale values ​​mapped to color and transparency values ​​(or corresponding opacity values) according to the transfer function. Alternative color mapping approaches are disclosed in the detailed description.

[0010] The MVR system's projection image updater module updates the value of a specific pixel in the projection image (a pixel intersecting the current view multiray) by combining the mapped voxel value and transparency value on the individual viewray. In this regard, updating the value of a specific voxel intersecting a particular viewray is performed after updating the values ​​of all intersecting voxels closer to the viewpoint than that specific voxel. In other words, combining the color / transparency values ​​of voxels at greater distances from the viewpoint must wait until the combining of color / transparency values ​​of voxels at shorter distances is complete. As a result, viewrays associated with more distant volumes can only be processed after the processing of viewrays associated with less distant volumes is complete. Pixel value updates are implemented so that the combined values ​​after each ray marching step are stored in the system's respective local variable after each ray marching step, and the value of this variable is finally written to the specific pixel in the projection image once ray marching is complete for the corresponding view multiray.

[0011] The MVR system performs generation, processing, and update steps for pixels of the projected image that have not yet been updated. For example, the described function may be iterated over for all pixels of the projected image in a loop. Since the processing of the first view multiray does not depend on the processing of the second view multiray, it is also possible to perform multiple view multirays in parallel. For example, in the practical application of the approach disclosed herein, typically hundreds or even thousands of view rays are processed in parallel by a graphics processing unit (GPU). Parallelization techniques are known in the art, such as distributing processing across multiple cores and pipeline processing.

[0012] In one embodiment, volume ray marching may utilize spatial skipping in which the processing of individual views of a particular view ray is accelerated by skipping as many ray marching steps as possible within the associated volume, which are associated with completely transparent voxels. A spatial skipping approach using distance maps is disclosed in the following paper: -Zuiderveld,Karel J., Anton HJ Koning, and Max A. Viergever. "Acceleration of ray-casting using 3-D distance transforms." Visualization in Biomedical Computing'92.Vol.1808.SPIE,1992. -Sramek, Milos. "Fast surface rendering from raster data by voxel traversal using chessboard distance." Proceedings Visualization'94. IEEE, 1994.

[0013] In one embodiment, the MVR system may further include a volume alignment module for aligning one or more volumes relative to each other. Aligning a first volume with respect to a second volume means that the origin of the local coordinate system of the first volume is known within the second volume (and vice versa). Once all volumes are correctly aligned, the volumes point to the same anatomical point at the same location in the global coordinate system. This makes it possible to fix the positions of two or more volumes relative to each other. For example, the position of a volume showing an MR scan of a patient's human brain can be fixed with respect to a volume showing a CT scan of the patient's skull so that the virtual representation makes anatomical sense.

[0014] To perform ray marching, it may be convenient to convert each view ray from the global coordinate system (i.e., the coordinate system associated with the 3D scene) to the local coordinate system of its associated volume (i.e., the coordinate system of the associated volume that is transformed, rotated, and scaled relative to the global coordinate system) at the start of each ray marching process. Using such a conversion allows ray marching steps to be processed simply based on the number of steps performed, without having to calculate the respective distances from the viewpoint position in the global coordinate system for each volume in its respective local coordinate system. As mentioned above, this conversion can be done once at the start of each ray marching process and does not need to be repeated at each ray marching step.

[0015] In one embodiment, a computer program product for direct multi-volume rendering is provided. The computer program product has computer-readable instructions that, when loaded into the memory of a computing device (e.g., an MVR system) and executed by at least one processor of the computing device, cause the computing device to perform the computer implementation method disclosed herein. In this regard, the functional modules of the MVR system can be implemented as software modules that implement functions for performing each method step when the MVR system is running.

[0016] Further aspects of the present invention will be realized and achieved by the elements and combinations specifically provided in the appended claims. It should be understood that both the above summary and the following detailed description are illustrative and descriptive, and do not limit the invention as described. [Brief explanation of the drawing]

[0017] [Figure 1A] According to one embodiment, a view multi-ray used to render multiple volumes of a 3D scene within a projected image is shown. [Figure 1B] The voxels of the volume are shown together with the local coordinate system. [Figure 2] It is a simplified block diagram of an MVR system according to an embodiment. [Figure 3] It is a simplified flowchart showing a computer-implemented method for direct multi-volume rendering according to an embodiment. [Figure 4A] An example of a transfer function for mapping scalar voxel values to color and transparency values is shown. [Figure 4B] An example of a transfer function for mapping scalar voxel values to color and transparency values is shown. [Figure 4C] An example of a transfer function for mapping scalar voxel values to color and transparency values is shown. [Figure 4D] An example of a transfer function for mapping scalar voxel values to color and transparency values is shown. [Figure 5] Examples of pixels in the projected image and their respective view multi-rays are shown. [Figure 6] An example of ray marching in a volume containing scene objects is shown. [Figure 7] An example of using a distance map for spatial skip is shown. [Figure 8] It is a schematic diagram showing examples of a general-purpose computer device and a general-purpose mobile computer device that can be used in embodiments of the present invention.

MODE FOR CARRYING OUT THE INVENTION

[0018] Figure 1A shows a view multi-ray 20 used to directly render multiple volumes 110, 120, 130, and 140 of a 3D scene in a projected image 30, according to one embodiment. This figure illustrates the conceptual approach behind the solution disclosed herein. This approach is explained in the context of Figures 2 and 3. In that regard, Figure 2 is a simplified block diagram of a computer system 200 (MVR system) adapted to implement the MVR approach shown in Figure 1A, and Figure 3 illustrates a computer implementation method 1000 that can be performed at runtime by the MVR system 200. For this reason, the following parts of the description will use reference numbers referring to such figures.

[0019] The MVR system 200 receives 3D data specifying a 3D scene 100 via interface 210 (1100). Those skilled in the art will know of appropriate interfaces for loading multiple volumes 290 into the scene 100. The 3D data specifying each volume may be stored in remote data storage (e.g., in a database on a remote device communicably coupled to the system 200). Alternatively, the system 200 may include memory adapted for storing 3D data for volumes 290. As previously mentioned, a volume consists of so-called voxels. Figure 1B shows some voxels pv_n of volume Vn. The voxels of a volume have a rectangular shape. Needless to say, in a complete volume Vn, each voxel layer has the same number of voxels arranged in a rectangle that typically forms the rectangular shape of the volume. While voxels can have a rectangular shape, generally, volume Vn typically consists of layers of 2D images where each pixel of the 2D image is a square. In the example, the 2D image layer is assumed to be located in the xn-yn plane of the local coordinate system CSn of Vn. Looking at the dotted voxels, we can see that the depth of the voxels in zn dimension is different from the edge length of the voxels in xn-zn dimension. The distance at which the 2D image layer is captured (e.g., by one of the volumetric imaging techniques described above) corresponds to the depth of the voxels in zn dimension. Typically, the depth of the voxels is different from the edge length of each pixel in the underlying 2D image layer.

[0020] In the example in Figure 1A, the MVR system 200 has loaded four volumes 110 (V1), 120 (V2), 130 (V3), and 140 (V4) into scene 100. For simplicity, the volumes are shown as two-dimensional rectangles. However, in a virtual 3D scene, volumes are, needless to say, three-dimensional cuboids. The legend in Figure 1A shows the different line types used to distinguish the lines associated with each volume from those associated with other volumes. For example, the dotted line style is used for all lines associated with V1. Each volume has its own local coordinate system. For simplicity, only the local coordinate system xl1-yl1-zl1 of V1 is shown in the figure. However, V2, V3, and V4 also have their own local coordinate systems, which can generally be transformed, rotated, and scaled relative to other local coordinate systems. Those skilled in the art know how to use matrix multiplication with transformation matrices for transformations from one coordinate system to another. Each voxel in a given volume has a scalar value that is the result of an imaging procedure for measuring the physical properties of the imaging subject. In the example, the voxels with scalar values ​​that represent a certain scene object within each volume lie within the boundaries of the scene objects 111, 121, 131, and 141. Like the volumes, these three-dimensional scene objects are also shown as two-dimensional objects for the purpose of simplifying the graphic representation in the figure. Scene objects are defined by visible voxels within each volume. Visible voxels can be translucent or opaque voxels rendered with corresponding color values. In the example in Figure 1A, the scene objects may represent patient tissues and / or organs identified by the respective imaging techniques by their respective scalar voxel values ​​within their respective volumes.

[0021] In an optional embodiment, the MVR system 200 has a volume alignment module adapted to align volumes 290 (e.g., V1 to V4) relative to each other, as already described above.

[0022] The multi-ray generator 230 of the MVR system 200 generates a view multi-ray 20 at viewpoint position 10 as the starting point (1200). In Figure 1, the leftmost point of the camera / eye symbol defines the viewpoint position. The view multi-ray 20 is generated in the direction of a specific pixel of the projected image 30 relative to viewpoint position 10. The projected image 30 is within viewing distance d from viewpoint position 10 and is located between viewpoint position 10 and scene 100. Finally, the specific pixel represents the 2D projection of all scene voxels that intersect the view multi-ray after the projected image 30. In other words, the view multi-ray 20 intersecting the specific pixel is used to sample voxel values ​​of the 3D scene in the direction of the multi-ray, ultimately calculating a combined value reflected in the color value of the specific pixel of the projected image. In this regard, the view multi-ray 20 represents multiple individual view rays 21, 22, 23, 24 having the same starting point and the same direction. Each individual view ray is associated with one of the volumes in the scene. In the example, viewray 21 is associated with V1 110, viewray 22 with V2 120, viewray 23 with V3 130, and viewray 24 with V4 140. For clarity and better visibility in the figure, viewrays 21-24 are shown as parallel arrows. However, they are identical with respect to their starting point and direction; that is, their positions in the scene are all identical. Nevertheless, individual viewrays are distinguishable by their unique assignment to one of the volumes. The starting point and direction of multiray 20 (and, with respect to it, individual viewrays) are defined by the multiray generator within the global coordinate system xg-yg-zg.

[0023] Referring briefly to Figure 5, the projected image 30 is shown with three further specific pixels 30-1, 30-2, and 30-3, through which the view multirays 20-1, 20-2, and 20-3 generated respectively are sent to the 3D scene to sample volume voxels in the scene. In this example figure, the volume is not shown. Instead, a 3D scene object 121-1 composed of one of the volumes is shown (as an ellipsoid). In the example, only multirays 20-1 and 20-2 intersect with voxels associated with the scene object. Furthermore, one individual view ray of each multiray 20-1, 20-2 is associated with a volume containing voxels of scene object 121-1. The final projected image 30 contains 2D projected values ​​for all pixels of image 30.

[0024] The volume ray marching processor 240 of the MVR system 200 processes each view ray 21, 22, 23, and 24 within its associated volume individually using volume ray marching. Conveniently, ray marching can be performed within the local coordinate system of each volume for performance optimization. This saves computationally expensive matrix multiplication after each ray marching step. The ray marching step for each view ray is performed only within the associated volume to sample each volume voxel. In particular, in the case of overlapping volumes (e.g., V1 overlaps with V2, and V1 overlaps with V3), the corresponding view rays perform sampling only within their associated volumes. In one embodiment, the first ray marching step for a particular view ray may occur over multiple ray marching step lengths, where the particular view ray first intersects with the associated volume. In the example in Figure 1, the first ray marching step for view ray 21 is indicated by the left ray marching dot on view ray 21. The first ray marching step of viewray 22 is indicated by the left ray marching dot on viewray 22 and is followed similarly. Note that in regions with overlapping volumes, the ray marching steps for each individual viewray are synchronized in that they are executed at the same distance with respect to the viewpoint position 10. In the example, the first ray marching step of viewray 22 in V2 occurs at the same distance with respect to the viewpoint as the sixth view marching step of viewray 21 in V1. The second ray marching step of viewray 22 occurs at the same distance with respect to the viewpoint as the seventh view marching step of viewray 21, and so on. Similarly, the first ray marching step of viewray 23 occurs at the same distance with respect to the viewpoint as the sixteenth view marching step of viewray 21, and so on.

[0025] In one embodiment, at each ray marching step, the scalar value of a corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel via a predefined transfer function. Optionally, the transfer function may also map potentially interpolated scalar values ​​sampled at the corresponding location to further parameters, such as luminosity or reflectivity. The ray marching processor 240 may provide one or more transfer functions TF241 for mapping within each volume. In the example, viewray 21 is used to process only the voxels of V1. A first transfer function TF1 may be used for voxel value mapping within V1. Viewray 22 is used to process only the voxels of V2. A second transfer function TF2 may be used for voxel value mapping within V2. In other words, in overlapping volume regions, voxel values ​​of different volumes mapped at the same distance from the viewpoint typically have different mapped voxel values, and individual viewrays can calculate such volume-specific mapped voxel values ​​for their associated volumes using their respective transfer functions at each ray marching step.

[0026] In an alternative embodiment, at each ray marching step, the scalar value of the corresponding voxel (or interpolation location within the volume) is mapped only to the transparency value via a predefined transfer function, and the color value is determined by applying an illuminance model or a combination of different illuminance models. In this regard, the system determines the color and intensity of light reflected in the direction of the viewpoint position by the surface of a scene object, taking into account different light sources and surface material properties. Further details can be found in the Wikipedia article "Bidirectional reflectance distribution function" available at https: / / en.wikipedia.org / wiki / Bidirectional_reflectance_distribution_function, or in the overview paper "An overview of BRDF models" by Montes, Rosana, and Carlos Urena, University of Grenada, Technical Report LSI-2012-001(2012).

[0027] The characteristics to be taken into consideration are determined by the selected illuminance model(s). A classic illuminance model is the Blinn-Fond reflection model (see Blinn, James F. "Models of light reflection for computer synthesized pictures," Proceedings of the 4th annual conference on Computer graphics and interactive techniques 1977). Another commonly used illuminance model is the Lambert model (see Lambert, Johann Heinrich. Photometria sive de mensura et gradibus luminis, colorum et umbrae. sumptibus vidvae, E. Klett, typis CP Detleffsen, 1760).

[0028] The result of this alternative embodiment is that the color value of a voxel is determined by the position and color of the light source, as well as the orientation of the scene object surface relative to the light source. Surfaces facing the light source are colored by the light source, while the rear surface remains dark. For example, objects can be clearly identified in a relatively simple CT scan.

[0029] In practical embodiments, the transfer function approach, when combined with the illumination model approach, can result in improved spatial perception for the observer of the projected image.

[0030] The concept of transfer functions is explained in more detail here using examples from Figures 4A to 4D. In Figure 4A, the raw CT slices (images) 410 provided by the CT scanning technique have grayscale pixel values. Consequently, the voxels of the CT volume, interpreted by joining multiple layers of the raw CT slices, also have corresponding grayscale scalar values ​​(raw data). In the CT volume, the brightness of the voxels corresponds to the X-ray absorption of the tissue at each body location in Hunsfield units (HU). To generate a realistic 3D rendering of the volume, a transfer function 400a is defined that maps grayscale HU values ​​401 to color and transparency (or opacity) values. In the CT example, the transfer function 400a maps the HU values ​​corresponding to air (black) and soft tissue (gray) to transparency value 402 "transparent", and the HU values ​​corresponding to bone (bright) to color value 403 "white" (or "beige" as an alternative). Direct rendering results 420, based on raw voxel values ​​and transfer function 400a, are shown as parts of a human skeleton.

[0031] Figure 4D uses transfer function 400d (similar to transfer function 400a) to generate rendering result 450 directly from the corresponding CT scan raw data volume. Figures 4B and 4C show direct rendering results 430 and 440 generated from the same CT scan raw data volume using different transfer functions 400b and 400c, respectively. In the figures, the direct rendering results are shown in grayscale, but in reality, transfer functions 400b and 400c map various scalar (grayscale) values ​​to color values ​​corresponding to red of different hues. Defining different transfer functions to map voxel scalar values ​​to corresponding transparency / color values ​​allows for setting different rendering focuses regarding what kinds of object properties are rendered.

[0032] The projection image updater module 250 of the MVR system updates the value of a specific pixel in the projection image 30 by combining the color and transparency values ​​of each voxel in the individual viewray (i.e., the mapped RGBA values ​​of each voxel) (1400). In this regard, updating each projection image pixel value with the value of a specific voxel intersecting a particular viewray is performed after updating with the values ​​of all intersecting voxels among the voxels closer to the viewpoint than that particular voxel. That is, individual viewrays are allowed to proceed rapidly until they encounter an opaque voxel in their associated volume. However, the processing of opaque voxels must be performed in synchronization with all rays processing the same location in the 3D scene to obtain the correct result for a particular pixel value. The generation (1200), individual processing (1300), and update (1400) steps are performed by each module of the MVR system 200 for all pixels in the projection image 30 (1500). For example, the step may be iterated within a loop (1450) that checks after each iteration whether there are still pixels in the projected image that have not yet been updated. If the projected image is not yet complete, further loop iterations are started for the pixels that have not yet been updated. Once the projected image is complete, method 1000 terminates. Alternatively, multiple pixels of the projected image may be processed in parallel, provided that each parallel processing is supported by the MVR system 200. As an example, the processing of individual pixels may be implemented as a shader program executed by the graphics processing unit. The updating of each projected image pixel value may be conveniently implemented such that the combined value is intermediately stored in a (local) variable during the processing of the projected image pixel. In other words, intermediately storing the combined value in a (local) variable during the processing of the projected image pixel, and updating each projected image pixel only at the end of the process, can improve performance in some cases. Once the processing of the view multiray 20 is complete, the final value of that variable is written to the projected image 30 as the respective projected image pixel value.Storing the updated projection image pixel values ​​in local variables during the ray marching process is computationally more efficient than writing the current combined values ​​to the projection image after each ray marching step.

[0033] Figure 6 illustrates direct volume rendering in more detail with an example. In direct volume rendering (i.e., calculating the projected image directly from the volume data), the MVR system does not know where the visible tissue surface (e.g., scene object SO6) is located within each volume. Therefore, a simple ray-surface collision calculation cannot be performed. The direct rendering approach can only determine the collision of a view ray with the bounding box BB6 of each volume (e.g., CT volume). Thus, multiple view rays VR6* (one for each pixel of the projected image) are transmitted from the viewpoint position 10 through the 3D scene. Ray marching is performed with each view ray within the volume defined by the bounding box BB6. The length ls of the ray marching step (i.e., the distance between two ray marching steps) is approximately the size of the voxel. For example, the length of the ray marching step can be set to be less than or equal to the length of the diagonal of the voxel. In these small steps along the viewray, raw voxel data is read at each ray marching step position (indicated by black and white dots), and the respective transfer function can be applied to determine the color and transparency (or alternatively, opacity, where: transparency value = 1 - opacity value) for each voxel. Typically, the color and transparency values ​​of a voxel are represented in the so-called RGBA format, in which the RGB channels represent color values ​​and the alpha value in the A channel represents opacity (0% = completely transparent, 100% = completely opaque).

[0034] Typically, voxels near the bounding box BB6 are transparent (e.g., air). Therefore, ray marching can simply continue until the system finds a visible voxel. Typically, when a view ray intersects a visible voxel (e.g., bone or soft tissue), the projection image updater begins accumulating the color and transparency of the visible voxel in subsequent visible voxels. In other words, the color and transparency values ​​of visible voxels along the view ray are combined with the color values ​​of each pixel in the projection image. This accumulation continues until the opacity channel (alpha value in RGBA) of the current pixel reaches a threshold close to 100%. For example, the opacity threshold can be predefined in the range between 90% and 99.9%. As a result, anything the view ray might encounter from that point onward will not be visible anyway, as it is hidden by previously intersected voxels. That is, the ray marching process can be stopped at this point. This action is often called early ray termination. In the example in Figure 6, the black dots represent ray marching steps where the view ray intersects an invisible voxel. The white dots indicate ray marching steps where a view ray intersects with visible voxels belonging to scene object S06. For each white dot, the aforementioned accumulation (combination) of mapped voxel values ​​is performed. In the example, all view rays penetrating scene object S06 continue the ray marching step after the object; that is, early ray termination did not work, or none of the exemplified rays reached the opacity threshold for early ray termination.

[0035] A detailed example is described below, with reference to Figure 1A, without any intention to limit the scope of the claims in any way. In particular, when joining mapped voxel values ​​within a region of overlapping volumes, the joining is always performed in an order in which the voxel closest to the viewpoint is joined first. Conveniently, individual viewrays are processed in "lockstep"; that is, processing of viewrays (especially within overlapping volume regions) proceeds at exactly the same speed and in the same direction.

[0036] First, when view multi-ray 20 is generated, view rays 21-24 are initialized. Then, the MVR system determines the position for the first ray marching step for each (individual) view ray within its respective associated volume. In the example, the first ray marching step is: -In V1, it occurs at a distance of 8 steps (position '8') from viewpoint position 10. -In V2, this occurs at a viewpoint position of 10 to 13 steps (position '13'), -In V3, this occurs at a distance of 23 steps from viewpoint position 10 (position '23'), -In V4, this occurs at viewpoint position 10 to 29 steps (position '39'). It is assumed that the following first ray marching step positions within each volume may be stored for each view ray as the number of step lengths from the viewpoint. The minimum value of these first ray marching step positions is the view multi-ray distance from the viewpoint (called the multi-ray position). That is, in the example, the system knows that all view rays have a first ray marching step position that is at least 8 step lengths from the viewpoint. The multi-ray position is then incremented by "1" after each ray marching step is completed. Using this information, the MVR system can at any time determine whether the mapped voxel values ​​on a particular view ray can be joined without risk, or whether there is a risk that mapped voxel values ​​closer to the viewpoint may be opaque and hide all voxels further away from the viewpoint, or whether there may be opaque color values ​​that need to be joined first to obtain the correct final pixel color, and therefore the system needs to wait further before joining. Also, in the case of semi-transparent pixels, the order in which voxel values ​​are joined is important to achieve the correct pixel color.

[0037] The volume ray marching processor iterates over all view rays 21-24 (i.e., all volumes V1-V4 rendered to the projected image) in each ray marching step. The order of these iterations is irrelevant. The following assumes an iteration order of V2 → V1 → V4 → V3. In the examples described below, where multiple iterations are described under a single heading, it should be understood that in each iteration, all volumes are processed in the aforementioned iteration order before the system executes the next iteration.

[0038] In an optional embodiment, the MVR system may store a distance map that can be calculated during the volume reading process. During this reading process and in a given one or more transfer functions, the system may calculate the distance to the nearest visible voxel in each volume for each voxel in the volume and store the calculated distance in the distance map. During volume ray marching, the ray marching processor of the MVR system can obtain from the distance map the distance to the next visible voxel at each location in each volume to determine whether the next ray marching step can be performed in multiple step lengths (ray marching jumps). Once such a distance is less than or equal to the ray marching step length, the scalar data of each voxel is read and processed by the corresponding transfer function. The distance map allows skipping transparent voxels in the volume during ray marching, thereby accelerating the ray marching process.

[0039] First iteration V2: Viewray 22 begins its first ray marching step at position "13". Upon reading the distance map, the system recognizes that the visible voxels are close and a ray marching jump cannot be performed. After reading the scalar voxel values ​​and applying the transfer function, the ray marching processor determines that the voxels are transparent and therefore no concatenation of the mapped color / transparency values ​​is necessary. As a result, Viewray 22 moves to the next ray marching step at position "14".

[0040] V1: For viewray 21, the situation is similar to viewray 22. Viewray 22 moves to the next ray marching step at position "9".

[0041] V4: For Viewray 24, at the edge of V4, the system derives from the distance map that all voxels within a radius of 2 steps are transparent. Therefore, the position of Viewray 24 can be safely increased from "39" to "41".

[0042] V3: For viewray 23, the situation is similar to viewrays 22 and 21, and the position can be incremented by 1 up to position "24".

[0043] After iterating over all viewrays, the multi-ray position is incremented by 1 for the next iteration, from "8" to "9". After the first iteration, the ray marching step positions for all viewrays for the second iteration are already within their respective scene objects, so further ray marching jumps are not allowed because their respective distance values ​​in the distance map are "0" (as long as each viewray is inside the scene object).

[0044] Second iteration Starting again from V2, at the current position "14" of viewray 22, the ray marching processor reads the first visible voxel of V2 belonging to scene object 121. However, the current position "14" of viewray 22 is further from the viewpoint than the current multi-ray position "9". Therefore, each mapped voxel value at the current position "14" should not yet be merged for each pixel in the projected image. Viewray 22 at the current position must wait until the multi-ray position also reaches "14".

[0045] V1: Viewray 21 has reached a visible voxel at its current position "9" which belongs to scene object 111. Each mapped voxel value is determined using its respective transfer function and is combined with its respective projected image pixel because there are no other viewrays whose current position is closer to the viewpoint position. The current position of viewray 21 is incremented to position "10".

[0046] V4: Viewray 24's current position "41" is still intersecting with a transparent voxel (→ no merge of mapped values). The distance map indicates that no further ray marching jumps are possible. Therefore, the current position is increased to position "42".

[0047] V3: Viewray 23 has reached a visible voxel at its current position "24," which belongs to scene object 131. Each mapped voxel value is determined using its respective transfer function. However, as with viewray 22, each mapped voxel value at the current position "24" must not yet be combined with the respective pixel in the projected image. Viewray 23 at its current position must wait until the multi-ray position also reaches "24." That is, volumes V1 and V2, which are closer to the viewpoint position, must be processed up to position "24" before the mapped voxel values ​​of viewray 23 can be combined with the projected image pixel values.

[0048] After all viewrays have been processed within the second iteration, the multi-ray position is incremented to "10".

[0049] Third, fourth, and fifth repetitions V2: At its current position "14", Viewray 22 needs to wait.

[0050] V1: Viewray 21 has reached three additional visible voxels at its current positions "11", "12", and "13", which belong to scene object 111. Each mapped voxel value is determined at each position using its respective transfer function and, since there are no other viewrays closer to the viewpoint position than the current position, is merged with each projected image pixel after each ray marching step. The current position of viewray 21 is increased to position "14" in the fifth iteration. Position "14" already overlaps with V2 and could allow for merging, but since the voxels intersecting viewray 22 were transparent, no merging with the V2 voxel value occurs.

[0051] V4: At its current location "42", Viewray 24 needs to wait.

[0052] V3: At its current position "24", Viewray 23 needs to wait.

[0053] After all viewrays have been processed in the third, fourth, and fifth iterations, the multi-ray position is "14".

[0054] Iterations 6 through 11 In the sixth iteration, viewray 21 has reached the current position of viewray 22. Both viewrays now have the same distance from the viewpoint position. Thus, for V2 and V1, their respective mapped voxel values ​​are determined at ray marching step positions "14" to "19" (the last position of viewray 22 intersecting with scene object 121) and combined with the projected pixel values ​​in their respective images. In the eleventh iteration, the current positions of viewrays 21 and 22 are incremented to position "20". Position "20" is already outside of V2. That is, the ray marching processor no longer considers viewray 22 for further processing.

[0055] V4 and V3: ViewRay 24 and 23 are still on standby.

[0056] After all viewrays have been processed in the 11th iteration, the multi-ray position is incremented to "20".

[0057] Iterations 12 through 14 V1: At positions "20" to "22", only the viewray 21 intersects with the visible pixels of its associated volume (scene object 111). Therefore, each mapped voxel value is combined with its respective image projection pixel. The current position of viewray 21 is incremented to position "23" in the 14th iteration.

[0058] V4 and V3: ViewRay 24 and 23 are still on standby.

[0059] After all viewrays have been processed in the 14th iteration, the multi-ray position is incremented to "23".

[0060] The 15th repetition V1: Viewray 21 is currently still intersecting with a visible voxel of scene object 111 at position "23". At position "23", V1 overlaps with V3, but the voxel in V2 at position "23" is invisible, and since viewray 23 has already moved to position 24 (while the multiray position is still position 23), only the mapped voxel value of viewray 21 is combined with the pixel value. The current position of viewray 21 is incremented to "24".

[0061] Viewray 23 and 24 are still on standby.

[0062] The multi-ray position is incremented to "24".

[0063] Iterations 16 and 17 V1: In the 16th and 17th iterations, the viewray 21 reaches positions "24" and "25" where it still intersects with the visible voxels of V1. Thus, each mapped voxel value of the scene object 111 is combined with its respective projected image pixel. The current position of the viewray 21 is set to "26" at the end of the 17th iteration.

[0064] V4: ViewRay 24 is still on standby.

[0065] V3: The current position of viewray 23 is equal to the multi-ray position "24" at the start of the 16th iteration. Thus, the mapped voxel values ​​of viewray 23 can now be merged with their respective projected image pixels, as other voxels closer to the viewpoint position can no longer obscure the visible voxels of scene object 131. The current position of viewray 23 is incremented to "25", and the same process continues for the visible voxels at this position. The current position of viewray 23 is set to "26" at the end of the 17th iteration.

[0066] The multi-ray position is also set to "26" at the end of the 17th iteration.

[0067] Iterations 18 through 21 V1: From position "26" onward, all remaining voxels inside V1 intersected by viewray 21 (to the right of scene object 111) are invisible. Therefore, the merging of mapped voxel values ​​is no longer necessary. The current position at the end of the 21st iteration is incremented to "29", which is outside of V1. Thus, viewray 21 is no longer considered for further ray marching after the 21st iteration. The example illustrates a situation using a ray marching jump. In this spatial skip embodiment, a ray marching step that could occur at position "28" is already skipped.

[0068] V4: ViewRay 24 is still on standby.

[0069] V3: The corresponding voxels on viewray 23 are the visible voxels of scene object 131. Thus, each mapped voxel value is combined with its respective projected image pixel. The current position of viewray 23 at the end of the 21st iteration is set to "29".

[0070] The multi-ray position at the end of the 21st iteration is set to "29".

[0071] 22nd iteration V4: ViewRay 24 is still on standby.

[0072] V3: Viewray 23 is located in the last visible voxel of scene object 131. Therefore, the mapped voxel values ​​are combined and the current position is incremented to "30".

[0073] The multi-ray position is set to "30".

[0074] The 23rd repetition V4: ViewRay 24 is still on standby.

[0075] V3: At position "30", viewray 23 intersects with the last (invisible) voxel of V3. Therefore, there is no value to combine with the projected image pixel. The current position is incremented to "31", which is outside of V3. Thus, V3 is no longer considered by the ray marching processor.

[0076] The multi-ray position is set to "30". Alternatively, when leaving the volume, the multi-ray position can be updated with the minimum view-ray position of any volume that has not yet been processed, which in this example is the current position "42" at view-ray 24.

[0077] 24th Iteration V4: V4 is currently the only volume remaining for ray marching. Multi-ray position "30" is still closer to the viewpoint than the current position "42" of viewray 24, but the ray marching processor can now continue processing viewray 24 as it has finished processing all volumes closer to the viewpoint. Therefore, the current multi-ray position will henceforth correspond to the current position of viewray 24. The voxel in V4 at position "42" is the first visible voxel of scene object 141. Therefore, the merge is performed and the current position of viewray 24 is incremented to "43".

[0078] Repetitions 25-29 V4: At positions "43" to "47", viewray 24 intersects with visible voxels belonging to scene object 141. Each mapped voxel value is combined with the projected image pixel after each ray marching step of viewray 24. The current position is incremented to "48" at the end of the 29th iteration.

[0079] At position "48", the last voxel in volume V4 is already an invisible voxel outside scene object 141, and no merging occurs. Therefore, the ray marching process for the entire multi-ray 20 is successfully completed, and all mapped voxel values ​​of volumes V1-V4 are merged with their respective projected image pixels in the correct order to ensure that voxels far from the viewpoint do not erroneously affect the projected image pixel values, even if they are hidden by voxels closer to the viewpoint. Nevertheless, the processing of individual view rays enables efficient processing of 3D scenes with multiple volumes, particularly in scenes with overlapping volumes. Needless to say, the multi-volume rendering approach disclosed herein works similarly for non-overlapping volumes (e.g., two adjacent volumes). In this case, the volumes can be rendered sequentially, starting with the volume furthest from the viewpoint, using standard alpha blending to obtain the final image. This works because, at each point in a 3D scene, only one volume exists to influence the color of the view ray passing through that part of the 3D scene. However, with overlapping volumes that have translucent overlapping scene objects, the simple per-volume approach no longer works. Instead, the aforementioned multi-ray approach can be used to render all volumes of the 3D scene at once, thus solving this problem. Furthermore, decoupling individual view rays to associated volumes allows for even faster rendering of some more complex cases where the view ray may need to slow down in different areas of the scene. Typical reasons why a ray slows down (i.e., cannot move forward as far as possible by using spatial skipping, for example) are noisy areas within the volume (due to, for example, low-quality images or a large number of small structures) or when the view ray passes close to a visible voxel structure without actually passing through anything visible.

[0080] Particularly in medical applications, multiple volumes with 3D data from different sources can be superimposed on each other in a 3D scene. For example, a CT volume and an MRI volume can be superimposed in a 3D scene so that the volumes overlap. This is typically the case when each scan targets the same part of the patient's body (e.g., the patient's head or chest). CT generally produces high-resolution images with no geometric distortion, which is well-suited for visualizing bone. However, some other tissues or pathologies are not well visible using CT data. To fill this gap, an MRI volume can be generated in addition to the CT volume. For example, an MRI volume can capture a tumor and align it with the CT volume so that it appears in the correct position relative to the bone. In a further use case, pre- and post-operative CTs can be rendered side-by-side to evaluate surgical outcomes. In yet another use case, a single CT volume can be divided into multiple smaller volumes, each containing one or more scene objects from the original CT volume. Scene objects in one smaller volume can then be moved individually relative to scene objects in other smaller volumes. This can be used, for example, to examine the correction of a fracture by rearranging the bone (i.e., by rearranging a small volume containing the bone fragment as a scene object).

[0081] Standard spatial skipping techniques (e.g., Euclidean distance maps, minimax octrees) can be used to accelerate view rays within the free areas of a volume. Spatial skipping is applied to individual view rays. This means that view rays for different associated volumes (for the same projected image pixels) may become out of sync, with some view rays potentially being further ahead than others. How this problem is addressed has been previously described by ensuring that visible voxels closer to the viewpoint are processed before visible voxels further away. With spatial skipping, performance optimization can be achieved by using additional information to skip as many invisible areas of each volume as possible (e.g., invisible voxels typically representing air). Spatial skipping techniques are known in the art. However, in the approach disclosed herein, spatial skipping is applied separately to each individual view ray of the view multiray.

[0082] One possible approach for spatial skipping is to perform some pre-calculation on the 3D data while it is being loaded into the scene for the first time. For each single voxel within a given volume, the distance to the nearest opaque voxel can be pre-calculated. In other words, this system determines the radius of the largest sphere that can be placed in a location containing only invisible voxels. This data structure is called a distance map. Figure 7 shows two visible scene objects 710 and 720 within volume 700. The camera, located at the center of sphere 711, represents the viewpoint position. Viewray 701 represents a ray that passes through the volume, corresponding to a single pixel in the projected image. As mentioned above, in one embodiment of the ray marching process, the distance map can be used at each ray marching step to determine whether larger ray marching jumps are possible to skip empty areas of volume 700 while ensuring that no visible voxels are skipped. The first ray marching step is positioned at the center of sphere 711 at the viewing distance in front of the camera. The value stored in the pre-calculated distance map at this position is the length of dashed line r1, which is the radius of the largest sphere that can be placed at this position without containing any visible voxels. The visible voxel closest to the center of sphere 711 is a portion of scene object 720, located at the intersection of dashed line r1 and sphere 711. Thus, in the first ray marching step, a jump equal to the length of r1 can be made along viewray 701, ending at the center of sphere 712. In step 2, the value read from the distance map is the length of dashed line r2, which corresponds to the distance between the center of sphere 712 and the closest visible voxel (a portion of scene object 710). Thus, in step 2, a jump equal to the length of r2 can be made. This process is repeated until a value is read from the distance map, and it does not guarantee a spatial skip jump. This is typically a distance smaller than or equal to the ray marching step size. In the example shown, the process is repeated for sphere 713 with radius r3, sphere 714 with radius r4, and three additional smaller spheres.At the last example position along viewray 701, the distance to the nearest visible voxel (part of scene object 720) is too small to perform a spatial jump. Therefore, at this point, the ray marching process returns to the default embodiment without spatial skipping, as described in the previous section. Note that when spatial skipping is used, the distance map is always read before returning to the default embodiment. Therefore, spatial jumps can occur within separate segments along the viewray. For example, a viewray can pass through a translucent scene object, in which case spatial skipping can be used both before and after passing through the scene object, or it cannot be used while passing through it.

[0083] Figure 8 is a schematic diagram showing examples of a general-purpose computer device 900 and a general-purpose mobile computer device 950, which may be used in conjunction with the technology described herein. The computing device 900 is associated with an MVR system 200 (see Figure 2) in an exemplary embodiment. The computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, and smartphones, and other similar computing devices, such as virtual reality glasses. The computing device 950 may be used to display projected images to the user directly as a result of multi-volume rendering. In the case of a user using a virtual reality display device, two projected images are rendered that reflect different viewing angles of the user's eyes. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0084] The computing device 900 includes a processor 902, memory 904, storage device 906, a high-speed interface 908 connected to memory 904 and a high-speed expansion port 910, and a low-speed bus 914 and a low-speed interface 912 connected to storage device 906. Each of the components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and can be mounted on a common motherboard or in other ways as needed. The processor 902 processes instructions for execution within the computing device 900, including instructions stored in memory 904 or on storage device 906, and can display graphical information to a GUI on an external input / output device, such as a display 916 connected to the high-speed interface 908. In other embodiments, multiple processors and / or multiple buses may be used together with multiple memories and memory types as needed. Also, multiple computing devices 900 may be connected to each device that provides the required portion of the operation (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0085] The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is a volatile memory device or a plurality of devices. In another embodiment, the memory 904 is a non-volatile memory device or a plurality of devices. The memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0086] The storage device 906 can provide large-capacity storage to the computing device 900. In one embodiment, the storage device 906 may be or include a computer-readable medium such as a floppy disk drive, hard disk drive, optical disk drive, or tape drive, flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied within the information carrier. A computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium such as memory 904, storage device 906, or memory on the processor 902.

[0087] The high-speed controller 908 manages operations that consume a large amount of bandwidth for the computing device 900, while the low-speed controller 912 manages operations that consume less bandwidth. Such function assignments are merely examples. In one embodiment, the high-speed controller 908 is coupled to memory 904, a display 916 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 910 that can accept various expansion cards (not shown). In another embodiment, the low-speed controller 912 is coupled to storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Zigbee, WLAN, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, pointing device, or scanner, or to a network device such as a switch or router, for example, through a network adapter.

[0088] The computing device 900 can be implemented in several different forms, as shown in the figure. For example, it can be implemented as a standard server 920, or multiple times within a group of such servers. It can also be implemented as part of a rack server system 924. In addition, it can be implemented within a personal computer, such as a laptop computer 922. Alternatively, components from the computing device 900 can be combined with other components in a mobile device (not shown), such as device 950. Each such device may contain one or more computing devices 900, 950, and the entire system may consist of multiple computing devices 900, 950 communicating with each other.

[0089] The computing device 950 includes, among other components, a processor 952, memory 964, input / output devices such as a display 954, a communication interface 966, and a transceiver 968. The device 950 may also include storage devices, such as a microdrive or other devices, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other ways as needed.

[0090] The processor 952 can execute instructions within the computing device 950, including instructions stored in memory 964. The processor may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor may be provided for coordinating other components of the device 950, such as the user interface, applications run by the device 950, and control of wireless communication by the device 950.

[0091] The processor 952 may communicate with the user through a control interface 958 and a display interface 956 coupled to the display 954. The display 954 may be, for example, a TFT LCD (thin-film transistor liquid crystal display) or an OLED (organic light-emitting diode) display, or other suitable display technology. The display interface 956 may include suitable circuitry for driving the display 954 to present graphics and other information to the user. The control interface 958 may receive commands from the user and translate them for submission to the processor 952. In addition, an external interface 962 may be provided in communication with the processor 952 to enable near-field communication of the device 950 with other devices. The external interface 962 may be provided for wired communication in some embodiments and for wireless communication in other embodiments, and multiple interfaces may also be used.

[0092] Memory 964 stores information within the computing device 950. Memory 964 can be implemented as one or more computer-readable media, volatile memory devices, or non-volatile memory devices. An expansion memory 984 is also provided and can be connected to the device 950 through an expansion interface 982, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such an expansion memory 984 can provide additional storage capacity to the device 950 or store applications or other information in the device 950. Specifically, the expansion memory 984 may include instructions to perform or supplement the aforementioned processing, and may also include security information. Therefore, for example, the expansion memory 984 can function as a security module for the device 950 and may be programmed with instructions that permit the secure use of the device 950. In addition, a secure application may be provided via the SIMM card along with additional information, such as identification information placed on the SIMM card in a hack-proof manner.

[0093] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product is tangibly embodied within an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 964, extended memory 984, or memory on processor 952, which can be received, for example, via transceiver 968 or external interface 962.

[0094] The device 950 may communicate wirelessly through a communication interface 966, which may include digital signal processing circuits if necessary. The communication interface 966 may provide communication based on various modes or protocols, including, among others, GSM voice telephone, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communication may occur, for example, through a wireless transceiver 968. In addition, short-range communication may occur, for example, using Bluetooth, WiFi, Zigbee, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 980 may provide the device 950 with additional navigation and location-related wireless data, which may be used by applications running on the device 950, if necessary.

[0095] The device 950 can also communicate audibly using an audio codec 960 that can receive voice information from the user and convert it into usable digital information. The audio codec 960 can also generate audible sound for the user, for example, through a speaker in the handset of the device 950. Such sound may include voice from a voice call, recorded voice (e.g., voice messages, music files, etc.), and voice generated by an application running on the device 950.

[0096] The computing device 950 can be implemented in several different forms, as shown in the figure. For example, it can be implemented as a mobile phone 980. It can also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile device.

[0097] Various embodiments of the systems and technologies described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits data and instructions to the storage system, at least one input device, and at least one output device.

[0098] These computer programs (also known as programs, software, software applications, or code) include machine instructions to a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, as well as in assembly / machine languages. In this specification, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives the machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0099] To provide user interaction, the systems and technologies described herein can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0100] The systems and technologies described herein can be implemented on a computing device that includes backend components (e.g., as data servers), middleware components (e.g., as application servers), or frontend components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0101] Computing devices can include clients and servers. Clients and servers are generally geographically separated and typically communicate with each other through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other.

[0102] Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the scope of the invention.

[0103] Furthermore, the logical flow shown in the diagram does not require a specific order or sequence to achieve the desired result. In addition, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system.

Claims

1. A computer implementation method (1000) for direct multi-volume rendering, wherein the method is Receiving 3D data (1100) specifying a three-dimensional scene (100) containing multiple volumes (110, 120, 130, 140), wherein each volume contains multiple voxels (pv_n), and each voxel in a particular volume has a scalar value which is the result of an imaging procedure for measuring the physical characteristics of the imaging subject. At the viewpoint position (10), a view multiray (20) is generated (1200) in the direction of a specific pixel of the projected image (30), wherein the projected image is located between the viewpoint position (10) and the scene (100), the specific pixel has a value representing the projection of all scene voxels that intersect the view multiray and the projected image (30) afterwards, the view multiray (20) represents a plurality of individual view rays (21, 22, 23, 24) with the same starting point and the same direction, and each individual view ray is associated with one of the volumes (110, 120, 130, 140) in the scene (100), Using volume ray marching, the process involves iterating over all individual view rays (21, 22, 23, 24) within the associated volume to process them individually (1300) in each ray marching step, wherein the ray marching step for each view ray is performed only within the associated volume, and in each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. The value of a specific pixel in the projected image (30) is updated after each ray marching step by combining the color and transparency values ​​of each voxel of the individual viewrays (1400), wherein updating with the value of a specific voxel intersecting with a specific viewray is performed after updating with the values ​​of all intersecting voxels among the voxels closer to the viewpoint than the specific voxel, so that the processing of opaque voxels of overlapping volumes at the same position is performed synchronously for each individual viewray. The process involves performing generation (1200), processing (1300), and updating (1400) steps on pixels of the projection image (30) that have not yet been updated (1500), Methods that include...

2. The method according to claim 1, wherein volumetric ray marching uses spatial skipping in such a way that individual processing of a particular view ray skips ray marching steps within an associated volume that are associated with completely transparent voxels.

3. The method according to claim 1, wherein the imaging procedure is an image acquisition or image generation process that results in a volumetric image.

4. The method according to claim 1, wherein the imaging procedure is selected from computed tomography, magnetic resonance imaging, optical coherence tomography, positron emission tomography, single-photon emission computed tomography, and 3D ultrasound imaging.

5. The method according to claim 1, wherein the physical properties are selected from X-ray irradiation absorption, proton number, light transmittance, positron emission of the injected radiopharmaceutical, and acoustic impedance.

6. The method according to claim 1, wherein two or more of the aforementioned volumes are aligned with respect to each other, and the origin of the local coordinate system of the first volume is known within the second volume, and vice versa.

7. The method according to claim 1, wherein at the start of the ray marching process for each viewray, each viewray is converted from a global coordinate system (xg, yg, zg) to the local coordinate system (xn, yn, Zn) of its associated volume (110, 120, 130, 140).

8. The method according to claim 1, wherein each processing (1300) of a first viewray (21) uses a first predefined transfer function for mapping the scalar values ​​of the voxels of each associated volume (110) to their respective color and transparency values, and each processing (1300) of a second viewray (22) uses a second predefined transfer function for mapping the scalar values ​​of the voxels of each associated volume (120) to their respective color and transparency values.

9. A computer program product for direct multi-volume rendering, wherein the computer program product includes computer-readable instructions that, when loaded into the memory of a computing device and executed by at least one processor of the computing device, cause the computing device to perform the method according to any one of claims 1 to 8.

10. A computer system (200) for direct multi-volume rendering, wherein the system is An interface (210) adapted to receive 3D data specifying a three-dimensional scene (100) containing multiple volumes (290), wherein each volume contains multiple voxels (pv_n), and each voxel in a particular volume has a scalar value which is the result of an imaging procedure for measuring the physical characteristics of the imaging subject. A multiray generator (230) configured to generate a view multiray (20) at a viewpoint position (10) in the direction of a specific pixel of a projected image (30), wherein the projected image is located between the viewpoint position (10) and the scene (100), the specific pixel has a value representing the projection of all scene voxels that intersect the view multiray and the projected image (30) afterwards, the view multiray (20) represents a plurality of individual view rays (21, 22, 23, 24) with the same starting point and the same direction, and each individual view ray is associated with one of the volumes (110, 120, 130, 140) in the scene (100), A volume ray marching processor (240) is configured to use volume ray marching by iterating over all individual view rays to process each view ray (21, 22, 23, 24) within its associated volume individually in each ray marching step, wherein the ray marching step for each view ray is performed only within the associated volume, and in each ray marching step, the scalar value of the corresponding voxel is mapped to the color and transparency values ​​of the corresponding voxel. A projection image updater (250) configured to update the value of a specific pixel in the projection image (30) by combining the color and transparency values ​​of each voxel of the individual viewray after each ray marching step, wherein updating with the value of a specific voxel intersecting with a specific viewray is performed after updating with the values ​​of all intersecting voxels among the voxels closer to the viewpoint than the specific voxel, and the processing of opaque voxels of overlapping volumes at the same location is performed synchronously for each of the individual viewrays, A system that includes this.

11. The system according to claim 10, wherein the volume ray marching processor (240) uses spatial skipping in such a way that the individual processing of a particular view ray skips ray marching steps within an associated volume that are associated with completely transparent voxels.

12. The system according to claim 10, further comprising a volume alignment module (220) configured to align one or more of the plurality of volumes with respect to each other, wherein the origin of the local coordinate system of the first volume is known in the second volume, and vice versa.

13. The system according to claim 10, wherein the volume ray marching processor (240) further includes a coordinate system converter (242) that, at the start of the ray marching process for each view ray, converts each view ray from a global coordinate system (xg, yg, zg) to the local coordinate system (xn, yn, Zn) of the associated volume (110, 120, 130, 140) for volume ray marching in the associated volume.

14. The system according to claim 10, wherein the volume ray marching processor uses a first predefined transfer function for mapping the scalar values ​​of the voxels of each associated volume (110) to their respective color and transparency values ​​when processing (1300) a first view ray (21) individually, and a second predefined transfer function for mapping the scalar values ​​of the voxels of each associated volume (120) to their respective color and transparency values ​​when processing (1300) a second view ray (22) individually, the system according to claim 10.

15. The system according to claim 10, wherein 3D data for a specific volume further includes a distance map for each voxel of the specific volume that stores the distance to the next opaque voxel in the volume, and ray marching on a specific viewray performs a ray marching jump that includes a plurality of distances between two ray marching steps according to the distance map, so that the next ray marching step is closer to the next opaque voxel.

Images