Light field volume rendering system and method

Abstract

A system and method for volumetric rendering of light fields, wherein the light field data undergoes a layering scheme, thereby introducing a subdivision of hogels into subsets. Each subset of subvolumes corresponding to a layer volume corresponds to a subregion of the layer. The new data partitioning is combined with efficient local memory caching techniques, a global light down-sampling strategy to reduce memory bandwidth requirements, and a volumetric rendering algorithm to generate a rendered light field image. A reduction in the total number of samples required can be obtained while still maintaining the quality of the resulting image. A method is also provided for aligning memory accesses with ray computations to maximize access coherency. Real-time layering scene decomposition can be combined with surface rendering methods to create a hybrid real-time rendering method that supports rendering scenes containing overlapping volumes and surfaces.

Description

[0001] Claiming priority

[0002] This application claims priority to U.S. Patent Application Serial No. 63 / 015,929, filed April 27, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to light field volume rendering for rendering images and light field data, which can be combined with rendering, data compression and decompression systems, and methods for providing interactive multidimensional content on light field displays. Background Technology

[0004] Volumetric rendering of light field techniques for visualizing volumetric data on light field displays is desirable. Volumetric rendering of light fields offers the opportunity to achieve new levels of compelling, immersive experiences. Volumetric rendering is particularly useful in medical imaging visualization, seismic visualization, fluid dynamics, and industrial inspection. Specifically, in the medical field, the ability to view three-dimensional (3D) images derived from scans such as magnetic resonance imaging (MRI) allows for in-depth understanding of human anatomy and contributes to effective diagnosis. Furthermore, when this visualization capability can be performed at real-time interactive speeds, larger and more detailed datasets (which can also incorporate temporal elements) can be used to perform more detailed, efficient, and accurate diagnoses.

[0005] In real-time computer graphics, 3D scenes are typically represented as a set of surfaces using, for example, polygons, points, and splines. Real-time surface rendering techniques known in the art typically only synthesize images of the scene based on light reflections. One representation capable of modeling beyond surface reflections involves combining volumetric representation with volumetric rendering techniques. However, this typically requires numerous samples of both light and volume data. Since volumetric rendering is computationally intensive, accelerated methods that maintain image quality while using computationally less expensive methods are needed for volumetric rendering of light fields.

[0006] It is said that usable light field displays require at least billions of pixels to achieve acceptable quality. Therefore, light field rendering requires orders of magnitude more pixels or rays compared to traditional 2D image rendering. Consequently, for light field volumetric rendering, it can be observed that even more rays are needed. Considering the inherent intersection / overlap between many rays in a light field, it can be observed that many identical samples are repeatedly loaded in the rays of light field volumetric rendering. However, the consistency scheme utilizing this inherent reuse is not so obvious.

[0007] The ray casting method described by Stegmaier et al. appears to be the preferred approach because it allows for increased sampling control and real-time operation on modern GPUs. A dynamic random access memory (DRAM) bandwidth bottleneck is typically observed in modern GPU implementations of volumetric rendering because the ray casting process requires numerous samplings of the volumetric dataset. The rendering algorithm accesses voxels within the volumetric data multiple times. If all accesses involve reading from DRAM, memory bandwidth and latency can become significant performance bottlenecks.

[0008] There is still a need for a light field volume rendering system and method that can provide good resolution at an acceptable working speed.

[0009] The purpose of providing this background information is to inform the applicant of known information that may be relevant to the present invention. It is not necessary to acknowledge, nor should it be construed, that any of the above information constitutes prior art opposing the present invention. Summary of the Invention

[0010] This invention generally relates to a method for rendering light field volumetric data from 3D image data. The invention describes an algorithm that requires fewer volumetric data samples, directly contributing to reducing bandwidth-based bottlenecks associated with 3D volumetric rendering and providing a faster and less computationally intensive rendering process. Furthermore, another object of the invention is to describe a method that illustrates how to effectively utilize caching for light field volumetric rendering to further alleviate slow memory-based bottlenecks during rendering computation.

[0011] According to one aspect, a method for performing light field volumetric rendering of a scene in a computer system is provided, comprising: dividing a three-dimensional description of the scene including volume data into multiple layers, each layer having an associated light field and at least one slab volume, each slab volume having an associated hogel subset such that all rays associated with the hogel subset intersect the slab volume; volumetrically rendering the light field associated with each layer using the volume data included in each layer, such that volumetric rendering calculations are performed on the rays in the hogel subset to provide the rendered light field; caching the volume data intersecting the corresponding slab volume and storing the volume data in a local storage cache memory; and blending the rendered light field associated with each layer into a single output light field image.

[0012] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0013] In an implementation of this method, only a portion of the volume data is cached.

[0014] In another embodiment, the method further includes volumetric rendering calculations performed synchronously by the rays in the hogel subset as they initially enter the slab.

[0015] In another embodiment of the method, at least one layer has more than one slab volume.

[0016] In another embodiment of the method, a reconstruction calculation is performed on a ray that intersects the slab volume by selectively accessing a set of volume elements from the local storage cache memory, the ray intersecting the slab volume associated with a resampled point along the ray path.

[0017] In another embodiment of the method, the size of the hogel subset is determined by the overlap of light samples in the hogels within the subset.

[0018] In another implementation of the method, the volume rendering calculations corresponding to each hogel subset and the corresponding slab volume are synchronized.

[0019] In another implementation, the method is used in a real-time rendering system.

[0020] On the other hand, a computer-implemented light field image rendering method is provided, comprising the following steps: dividing a three-dimensional description of a scene including surface data elements and volume data into multiple layers, each layer having an associated light field and sampling scheme; further dividing at least one of the multiple layers into multiple segments, each segment having an associated light field and sampling scheme, wherein the position of each segment is determined according to the geometry of at least a portion of an object represented in the scene; for each layer and each segment, according to the sampling scheme, rendering a set of pixels including additional pixel information using surface data included in each layer and each segment to generate a sampled light field; reconstructing the sampled light field of each layer and segment using the set of pixels; performing volume rendering on the light field associated with each layer using volume data included in the layer; and blending the reconstructed sampled light field and the volume rendered light field associated with each layer and segment into a single output light field image.

[0021] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0022] In an implementation of this method, the sampling scheme includes a binary matrix associated with each layer and a mapping function that maps each layer.

[0023] In another embodiment of the method, each light field associated with each layer consists of one or more hogel subsets, wherein each hogel subset and its associated layer define a slab volume as a sub-volume of the layer, and wherein the method further includes: synchronizing volume rendering calculations associated with the rays associated with each layer; caching volume data intersecting with each corresponding slab volume; and volume rendering of the rays in each hogel subset when the rays intersect with the slab volume.

[0024] In another implementation of the method, only a portion of the volume data is cached.

[0025] In another embodiment of the method, at least one layer has more than one slab volume.

[0026] In another embodiment of the method, the size of the hogel subset is determined by the overlap of light samples in the hogels within the subset.

[0027] In another implementation of the method, all volume rendering calculations corresponding to the hogel subset and the corresponding slab volumes are synchronized when they intersect.

[0028] In another implementation, the method is used in a real-time rendering system.

[0029] On the other hand, a method for performing light field volumetric rendering of a scene in a computer system is provided, comprising: dividing a three-dimensional scene including volumetric data elements into multiple layers, each layer having an associated light field and sampling scheme, wherein each light field consists of one or more hogel subsets, and each hogel subset and layer, as well as the associated light field, defines a slab volume including a sub-volume of the layer; performing volumetric rendering of the light field associated with each layer using the volumetric data included in the layer according to the sampling scheme to provide multiple rendered light fields; and upsampling and blending the rendered light fields associated with each layer into a single output light field image.

[0030] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0031] In an implementation of this method, the sampling scheme includes a binary matrix associated with each layer and a mapping function that maps each layer.

[0032] In another embodiment of the method, volume data intersecting with the slab volume is cached.

[0033] In another embodiment of the method, only a portion of the volume data intersecting with the slab volume is cached.

[0034] In another embodiment of the method, at least one layer has more than one slab volume.

[0035] In another embodiment of the method, the size of the hogel subset is determined by the overlap of light samples in the hogels within the subset.

[0036] In another implementation of the method, the method is used in a real-time rendering system.

[0037] On the other hand, a system for performing accelerated volumetric rendering of a scene in a computer system is provided, the system comprising: a system memory for storing volumetric data; and a processor device operatively coupled to the system memory, configured to: divide a three-dimensional description of a scene including volumetric data into layers, each layer having an associated light field and at least one slab volume, each slab volume having an associated hogel subset such that all rays associated with the hogel subset intersect the slab volume; volumetrically render the light field associated with each layer using the volumetric data included in each layer, such that volumetric rendering calculations are performed on the rays in the hogel subset and the rays intersecting the slab volume to provide a rendered light field; cache the volumetric light field data intersecting the corresponding slab volume and store the volumetric light field data in a local storage cache memory; and blend the rendered light field associated with each layer into a single output light field image.

[0038] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0039] In the implementation of the system, only a portion of the volume data is cached.

[0040] In another embodiment, the system memory also includes volumetric rendering calculations configured to synchronize light-related calculations within the hogel subset.

[0041] In the implementation of the system, at least one layer has more than one slab volume.

[0042] In the implementation of the system, a reconstruction calculation is performed on the rays intersecting the slab volume by selectively accessing a set of volume elements from the local storage cache memory.

[0043] In the implementation of the system, the size of the hogel subset is the overlap of light samples in the hogel subset.

[0044] In the implementation of the system, when a ray associated with a hogel subset intersects with the slab volume, the volume rendering calculation corresponding to the hogel subset is synchronized.

[0045] In the implementation scheme, the system is used for a real-time rendering system.

[0046] On the other hand, a system for performing accelerated volumetric rendering of a scene in a computer system is provided, the system comprising: a system memory for storing volumetric data; and a processor device operatively coupled to the system memory, configured to: divide a three-dimensional description of a scene, including surface data elements and volumetric data, into a plurality of layers, each layer having an associated light field and sampling scheme; further divide at least one of the plurality of layers into a plurality of segments, each segment having an associated light field and sampling scheme, wherein the location of each segment is determined according to the geometry of at least a portion of an object represented in the scene; for each layer and each segment, according to the sampling scheme, using surface data included in each layer and each segment, rendering a set of pixels including additional pixel information to generate a sampled light field; reconstructing the sampled light field of each layer and segment using the set of pixels; performing volumetric rendering of the light field associated with each layer using volumetric data included in the layer; and blending the reconstructed sampled light field and the volumetric rendered light field associated with each layer and segment into a single output light field image.

[0047] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0048] In the implementation of the system, the sampling scheme includes a binary matrix associated with each layer and a mapping function that maps each layer.

[0049] In another implementation of the system, each light field associated with a layer consists of one or more hogel subsets, wherein each hogel subset and its associated layer define a blank volume called a sub-volume of the layer, and further includes: synchronizing ray-related volume rendering calculations; caching volume data that intersects with the corresponding blank volume; and performing volume rendering calculations on all rays in the hogel subset that intersect with the blank volume.

[0050] In another implementation of the system, only a portion of the volume data is cached.

[0051] In another embodiment of the system, at least one layer has more than one slab volume.

[0052] In another implementation of the system, the size of the hogel subset is determined by the overlap of light samples in the hogels within the subset.

[0053] In another implementation of the system, the volume rendering calculation corresponding to each hogel subset is synchronized when the ray associated with the hogel subset intersects the slab volume.

[0054] In another implementation, the system is used for a real-time rendering system.

[0055] On the other hand, a system for performing accelerated volumetric rendering of a scene in a computer system is provided, the system comprising: a system memory for storing volumetric data; and a processor device operatively coupled to the system memory, configured to: divide a three-dimensional descriptive scene including volumetric data elements into layers, each layer having an associated light field and sampling scheme, wherein each light field consists of one or more hogel subsets and each hogel subset and layer defines a slab volume including a sub-volume of the layer; perform volumetric rendering of the light field associated with each layer using the volumetric data included in the layer according to the sampling scheme; and upsample and blend the rendered light field associated with each layer into a single output light field image.

[0056] The implementation scheme may include one or more of the following features, which may be combined in any arrangement or combination.

[0057] In the implementation of the system, the sampling scheme includes a binary matrix associated with each layer and a mapping function that maps each layer.

[0058] In another implementation of the system, the volume data of a subset of hogels that intersects with the slab volume is cached.

[0059] In another implementation of the system, only the volume data of a subset of hogels that intersects with the slab volume is cached.

[0060] In another embodiment of the system, at least one layer has more than one slab volume.

[0061] In another implementation of the system, the size of the hogel subset is determined by the overlap of light samples in the hogels within the subset. Attached Figure Description

[0062] These and other features of the invention will become more apparent in the following detailed description, with reference to the accompanying drawings.

[0063] Figure 1A : This is a top-down schematic diagram of a single hogel within the inner and outer frustum volumes of a light field display.

[0064] Figure 1B : This is a top-to-bottom schematic diagram of the inner and outer frustum volumes of the light field display.

[0065] Figure 1C : This is a top-down schematic diagram of the single-layer outer truncated volume of the light field display.

[0066] Figure 2 : This is a schematic diagram (block diagram) of the implementation scheme of the light field volume rendering system / method.

[0067] Figure 3A This is a top-down schematic diagram of the inner frustum volume of the light field display, showing the light path of a single hogel.

[0068] Figure 3B This is a top-down schematic diagram of the outer frustum volume of the light field display, showing the light path of a single hogel.

[0069] Figure 3C This is a top-down schematic diagram of the inner and outer frustum volumes of the light field display, showing the light path of a single hogel.

[0070] Figure 3D : This is a top-down schematic diagram of the outer frustum volume of the light field display, showing an alternative implementation of the light path of a single hogel.

[0071] Figure 3E This is a top-down schematic diagram of the inner frustum volume of a light field display, showing an alternative implementation of the light path of a single hogel.

[0072] Figure 3F : This is a top-down schematic diagram of the inner and outer frustum volumes of a light field display, showing an alternative implementation of the light path of a single hogel.

[0073] Figure 4 : This schematically illustrates an exemplary layered scene decomposition of the display's image data (two layered schemes with ten layers), associated with the inner frustum and outer frustum light field regions, respectively.

[0074] Figure 5 : A flowchart showing the light field volume rendering method.

[0075] Figure 6 : This is a top-to-bottom schematic diagram of the inner and outer frustum volumes of the light field display.

[0076] Figure 7 : A flowchart showing an alternative implementation scheme for the light field volume rendering method.

[0077] Figure 8 : A flowchart showing an alternative implementation scheme for the light field volume rendering method. Detailed Implementation

[0078] definition

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0080] When used in this text with the term “comprising,” the word “a” or “an” may mean “one,” but it is also consistent with the meanings of “one or more,” “at least one,” and “one or more.”

[0081] As used herein, the terms “comprising,” “having,” “including,” and “containing,” and their grammatical variations, are inclusive or open-ended and do not exclude additional, unreferenced elements and / or method steps. When used herein in conjunction with compositions, apparatus, articles, systems, uses, or methods, the term “substantially constitutes…” indicates the possibility of additional elements and / or method steps, but these additions do not materially affect the composition, apparatus, article, system, method, or manner of use. When used herein in conjunction with compositions, apparatus, articles, systems, uses, or methods, the presence of additional elements and / or method steps is not included. Compositions, apparatus, articles, systems, uses, or methods described herein that include certain elements and / or steps may also be substantially constituted by such elements and / or steps in some embodiments, and by such elements or steps in other embodiments, whether or not these embodiments are specifically mentioned.

[0082] As used herein, the term “approximately” refers to a change of approximately + / - 10% from a given value. It should be understood that such a change is always included in any given value provided herein, unless otherwise specifically mentioned.

[0083] Unless otherwise stated herein, the memorization of ranges in this document is intended to pass the range and individual values ​​within the range to the same bit values ​​as the numbers used to represent the range.

[0084] The use of any example or exemplary language, such as “for example,” “exemplary embodiment,” “illustrative embodiment,” and “if,” is intended to illustrate or represent aspects, implementations, variations, elements, or features relevant to the invention, and not to limit the scope of the invention.

[0085] As used herein, the terms “connection” and “linked” refer to any direct or indirect physical association between elements or features of the invention. Thus, these terms can be understood to mean elements or features that partially or completely contain, connect, couple, arrange, join together, communicate, operate together, etc., even if there are other elements or features between elements or features described as connected.

[0086] As used in this article, the term "pixel" refers to the light source and light-emitting mechanism used to create a display.

[0087] As used in this article, the term "light field" at its basic level refers to a function describing the amount of unobstructed light passing through each point in space in every direction. Therefore, a light field represents radiance as a function of the position and direction of light in free space. Light fields can be synthesized through various rendering processes or captured from a light field camera or an array of light field cameras. In a broader sense, the term "light field" can be described as an array or subset of hogels.

[0088] As used herein, the term "light field display" is a device that reconstructs a light field from a finite number of light field radiation samples from the input to the device. Irradiance sampling represents the red, green, and blue (RGB) color components. For the reconstruction in a light field display, the light field can also be understood as a mapping from four-dimensional space to a single RGB color. These four dimensions include the display's vertical and horizontal dimensions (x, y) and two dimensions describing the light field's directional components (u, v). The light field is defined as a function:

[0089] LF: (x, y, u, v) → (r, g, b)

[0090] For a fixed x f y f LF(x) f y f (x, u, v) represents a two-dimensional (2D) image called an "elemental image". An elemental image is derived from a fixed x-axis. f y f A positional light field orientation image. When multiple element images are connected side-by-side, the resulting image is called the "overall image." The overall image can be understood as the entire light field required by a light field display.

[0091] As used herein, the term "voxel" refers to a single sample or data point on a regularly spaced 3D grid consisting of individual data points. A voxel is a single volume element corresponding to a location in 3D data space and has one or more associated data values. As used herein, the term "scene description" refers to a geometric description of a 3D scene, which can be a potential source of a renderable light field image or video. This geometric description can be represented by, but is not limited to, points, quadrilaterals, and polygons.

[0092] As used herein, the term "scene description" refers to the geometric description of a 3D scene, which may be a potential source of a renderable light field image or video. This geometric description can be represented by, but is not limited to, points, quadrilaterals, and polygons.

[0093] As used herein, the term "additional pixel information" refers to information included in the scene description. Additional pixel information includes, but is not limited to, color, depth, surface coordinates, normals, material values, transparency values, and other possible scene information.

[0094] As used herein, the term "display surface" refers to the set of points and directions defined by the physical spacing of individual light field hogel elements. A display surface can be planar, as with traditional 3D displays, or it can be a non-planar surface at any location on the display surface, locally, globally, or throughout. For example, a display surface can be formed on a non-planar curved surface, so the set of points will lie on a curved display surface. Any other conceivable geometry of a desired display surface can be used. In an abstract mathematical sense, a light field can be defined and represented on any geometric surface and does not necessarily correspond to a physical display surface with actual physical energy emission capabilities.

[0095] As used in this article, the term "elemental image" refers to a fixed position x. f y f Two-dimensional (2D) image LF(x) f y f (u, v). The element image is from a fixed x. f y f A light field orientation image of the location.

[0096] As used in this article, the term "overall image" refers to multiple element images connected side-by-side, hence the resulting image is called the "overall image." The overall image can be understood as the entire light field required by a light field display.

[0097] As used in this article, the term "layer" refers to any two parallel or non-parallel boundaries, where the width is consistent or variable, and parallel or non-parallel to the display surface.

[0098] It is foreseeable that any embodiment of the compositions, devices, articles, methods and uses disclosed herein can be implemented by those skilled in the art, or implemented by making such variations or equivalents without departing from the scope and spirit of the invention.

[0099] This paper describes a system and method for volumetric rendering of light fields, wherein light field data undergoes a layering scheme, thereby introducing the partitioning of hogels into subsets. Volumetric rendering is highly useful in medical data visualization, industrial inspection, airport security, various computational science simulations such as computational fluid dynamics (CFD), wave equation simulations, and other applications. In the system and method of this invention, each subset of sub-volumes corresponding to a layer volume corresponds to a sub-region of the layer. Data partitioning is combined with efficient local memory caching techniques, using an all-light downsampling strategy to reduce memory bandwidth requirements, and the volumetric rendering algorithm is capable of generating rendered light field images. The system and method reduce the total number of samples required by reducing redundancy and making efficient use of sampled data caching, while still maintaining the quality of the generated image. Furthermore, the method provides a way to keep sequential memory access consistent with ray computation to maximize access consistency. The system and method described in this invention can be combined with real-time layered scene decomposition surface rendering methods to create hybrid real-time rendering methods that support scene rendering that includes stacked volumes and surfaces.

[0100] Volumetric rendering allows defining an entire scalar field in 3D space, rather than the common surface rendering of representations in most computer graphics. It can also be extended to display higher vector fields in 3D space. Real-time volumetric rendering allows interactive exploration of datasets or manipulation of time-varying volumetric data. One slicing-based approach, which slices the volume using planes aligned with the view, then texture-maps and composites it in the framebuffer, demonstrates real-time GPU-based performance because they are rendered before and after. In practice, memory caching schemes help reduce the performance impact of these redundant accesses, as coherent rays are typically computed in parallel, and there is redundancy between rays and within ray threads themselves. Even if all redundancy could be perfectly utilized, it must be acknowledged that, in the "best or worst case" scenario, each sample needs to be loaded directly from the voxel mesh from DRAM at least once, thus requiring a certain level of latency and memory bandwidth if rendering is to achieve real-time rates (e.g., 30Hz).

[0101] Figure 1A Describing another approach, as is evident from the prior art, involves defining inner and outer clipping planes 14, and allowing light to travel from the outer plane to the inner plane (and vice versa). For each hogel in the light field display, such as Figure 1AAs shown, based on the hogel field of view θ16 and hogel center position 58 on display plane 10, there exists a frustum region associated with the hogel. Simply put, each hogel must be rendered twice, for both the inner and outer hogels, and then composited. This results in two rays per direction represented by the display, thus billions. Another approach, quite evident in the prior art, is to define a near clipping plane 12 and a far clipping plane 14, where rays travel from the far clipping plane 14 to the near clipping plane 12 and vice versa. This results in one ray per pixel, with many samples per ray, thus many memory accesses are likely to be individual DRAM accesses. In practice, using existing GPU-based light field rendering methods, some cached coherence (inter-ray coherence) and intra-ray coherence can be utilized when rendering rays within a single hogel, as described by Hamilton et al. (2007). However, when ray calculations are constructed in a conventional manner, rays typically originating from individual hogels accessing the same voxel do not utilize the cache, leading to an increase in DRAM accesses. Accelerating light field volume rendering requires two times the following: (1) a method to reduce the total number of samples required while maintaining the quality of the resulting image; and (2) a method to command memory accesses to be consistent with ray calculations to maximize access consistency, thereby using caching to reduce the total number of DRAM accesses required.

[0102] Figure 1B This is a top-to-bottom schematic diagram of the inner frustum volume 18 and the outer frustum volume 20 of the light field display. Figure 1B An exemplary layered scene decomposition of image data associated with the light field regions of the inner frustum volume 18 and outer frustum volume 20 of the display is schematically illustrated. The inner and outer frustum volume layers extend from the display plane 10. Although the inner and outer frustum volumes 18 and 20 are illustrated as mirror images of each other, they can have different numbers of layers, different sizes of layers, or different depths of layers, and can be rendered using different rendering techniques. Rendering techniques include, but are not limited to, tilt rendering techniques and perspective rendering techniques.

[0103] Voxel mesh 22 Figure 1C As shown. The minimum requirement for a useful light field display is at least 1 billion pixels. This paper summarizes existing, well-known light field volumetric rendering methods. Simply put, each hogel must be rendered twice, for both the inner and outer hogels, and then composited, similar to Halle's proposed "double frustum rendering." Then, the number of rays in each direction represented by the display is two, thus amounting to billions.

[0104] Figure 2The flowchart of the method is shown, in which the inner frustum ray calculation 46 and the outer frustum ray calculation 48 are performed simultaneously to render the inner and outer frustum images respectively, and then the images are synthesized into a single output light field image 52 using the volume rendering integral discretization equation 50.

[0105] like Figure 3A -F illustrates how layers within a single frustum (e.g., inner frustum volume 18) are used in the rendering method. In volumetric rendering, each computation thread associated with a single ray computes a single instance of the volumetric rendering integral. The whole is discretized as an accumulation computation performed sequentially from front to back along ray path 32. Therefore, to render the inner frustum volume 18 for each pixel in each hogel, it can be assumed that the ray starts at intersection 38 at the display plane 10 of the inner frustum volume 18, as shown, and continues to intersection 36 at the far clipping plane 14 of the inner frustum volume 18. Figure 3B The same light path 32 in the outer frustum volume 20 is shown, moving from the intersection 38 on the display plane 10 to the intersection 34 with the near-cut plane 12. Figure 3C Will Figure 3A and Figure 3B Combined into a separate diagram, this illustrates the ray path 32 of a single pixel starting from intersection 38 on display plane 10, which traces to intersection 36 on the far clipping plane 14 of the inner frustum volume 18. The ray path 32 of the same pixel in the outer frustum volume 20 moves from intersection 38 on display plane 10, tracing to intersection 34 with the near clipping plane 12. Therefore, Figure 3D The inner frustum ray path 32 of an alternative embodiment of a single pixel in a hogel is shown, wherein the ray path 32 is traced from the intersection 38 of the display plane 10, as shown, and continues to the intersection 36 of the far clipping plane 14. Figure 3E The ray path 32 of the outer frustum volume 20 of an alternative embodiment of a single pixel in hogel is shown, wherein the ray path 32 is traced from the intersection 34 of the near clipping plane 12, as shown, and continues to the intersection 38 of the display plane 10. Figure 3F Will Figure 3D and Figure 3E Combining these into a separate diagram to illustrate the alternative ray path 32 of a single pixel traveling in a single direction, tracing from intersection 34 with the near clipping plane 12 to intersection 38 at the display plane 10 of the outer frustum volume 20, then continuing into the inner frustum volume 18 until reaching intersection 36 on the far clipping plane 14. Figure 3B Conversely, in order to render the volume of the outer frustum of 20, such as Figure 3EAs shown, it can be assumed that the light ray begins at the near clipping plane 12 and ends at the display plane 10. For any light ray that begins at the near clipping plane 12 within the outer frustum volume 20, it continues with the corresponding light ray that begins at the display plane 10 and continues to the far clipping plane 14 within the inner frustum volume, such as... Figure 3F As shown. The outer and inner frustum light segments can be calculated in parallel and then discretized through volumetric rendering integration. Figure 2 The simplified approach shown has been modified to a single final calculation.

[0106] like Figure 4 As shown, an exemplary layered scene decomposition of image data (two layering schemes with ten layers) is associated with the light field regions of the inner frustum volume 18 and the outer frustum volume 20 of the display, respectively. The inner frustum volume 18 and the outer frustum volume 20 are defined by the display plane 10. The far clipping plane 14 is the farthest boundary of the inner frustum volume 18, and the near clipping plane 12 is the farthest edge of the outer frustum volume 20. The inner frustum volume 18 is divided into multiple inner frustum layers 40, according to the disclosed layered scene decomposition technique. According to the layered scene decomposition technique, the outer frustum volume 20 is divided into multiple outer frustum layers 42. It should be noted that, for illustrative purposes only, layers 40, 42 are represented as having uniform width. As used herein, the term "layer" refers to any two parallel or non-parallel boundaries, with consistent or variable widths, parallel or non-parallel to the display plane.

[0107] Other techniques in this field for light field volumetric rendering have proposed using view composition to reduce the number of views that need to be rendered, which could allow for more efficient rendering systems. These methods are based on supervised learning, which requires large training datasets. Furthermore, it is unclear how the behavior of the algorithms generalizes to examples outside the training dataset, which could lead to severe artifacts.

[0108] Light field volume rendering can be defined as creating a light field image using a volume rendering model. For ease of discussion, assume that the rendered image is a light field image LF(x, y, u, v). The volume data function is defined as v(x, y, z), usually represented as a voxel mesh 22, but can be represented by a point-based representation or a tetrahedral representation (Hamilton (2013)). The volume mesh can be arbitrarily positioned, rotated, or manipulated relative to a virtual light field camera. When presenting finite volume data, assume that there is a far clipping plane 14 in the inner frustum, such as Figure 1A As shown, this represents the deepest boundary plane of the layer. Similarly, within the outer frustum, there is a near-cutting plane 12, representing the layer boundary within the frustum that is farthest from the display plane 10. Assume the outer frustum is near the cutting plane 12, C... n And the inner flat truncated body far cutting plane 14C fThe concept of volumetric rendering. These are the key inputs for rendering, thus knowing where to start and stop the volumetric rendering ray casting process. Another required input is the transfer function, which can be expressed as TF(density)|→(color, α). This can also be a function that is only more complex than density. A defining characteristic is that it must be used to calculate a(color, α) for each point in the volume during ray integral calculation. To achieve real-time volumetric rendering, the volumetric rendering integral can be calculated using the pre- and post-composition equations (or pre- and post-composition equations) without incurring a large computational burden.

[0109] This invention generally relates to volumetric rendering of light field data and methods and systems related to accelerating volumetric rendering techniques. It is commonly observed that modern graphics processing unit (GPU) implementations for volumetric rendering suffer from DRAM memory bandwidth bottlenecks because the ray casting process requires numerous samplings of the volumetric dataset. Rendering algorithms access voxels in the volumetric data multiple times. If all accesses involve reading from DRAM, memory bandwidth and latency become significant issues. In practice, memory caching schemes help mitigate the performance impact of these redundant accesses because coherent rays are typically computed in parallel, with redundancy existing between rays and within the ray thread computation itself. Even if all redundancy could be perfectly utilized, it must be acknowledged that, in the "best-worst-case" scenario, each sample must be loaded directly from the voxel mesh in DRAM at least once, thus introducing latency and memory bandwidth requirements.

[0110] Figure 5 A flowchart illustrating an embodiment of the disclosed method is provided. The method includes: dividing a 3D description of a scene including volume data into multiple layers, each layer having an associated light field and at least one slab volume, each slab volume having an associated hogel subset such that all rays associated with the hogel subset intersect the slab volume 60. Additionally, volume rendering is performed on the light field associated with each layer using the volume data included in the layers, such that for rays intersecting the slab volume, volume rendering calculation is performed on all rays in the hogel subset 62. The volume data intersecting the corresponding slab volume is cached and stored in local storage cache memory 64. The method further includes blending the rendered light field associated with each layer into a single output light field image 66.

[0111] Figure 7A flowchart illustrating an embodiment of the disclosed method is provided. The method includes dividing a 3D description of a scene, including surface data elements and volume data, into multiple layers, each layer having an associated lighting field and sampling scheme 68. The method further includes dividing at least one of the multiple layers into multiple segments, each segment having an associated lighting field and sampling scheme, wherein the location of each segment is determined based on the geometry of at least a portion of an object represented in the scene 70. For each layer and each segment, the method includes rendering a set of pixels, including additional pixel information, to generate a sampled lighting field 72, using surface data included in each layer and each segment, according to the sampling scheme. The sampled lighting field of each layer and segment is reconstructed using this set of pixels 74. The lighting field associated with each layer is volumetrically rendered using the volume data included in the layer 76, and the reconstructed sampled lighting field and the volumetrically rendered lighting field associated with each layer and segment are blended into a single output lighting field image 78.

[0112] Figure 8 A flowchart illustrating an embodiment of the disclosed method is provided, comprising dividing a 3D scene including volumetric data elements into multiple layers, each layer having an associated light field and sampling scheme, wherein each light field consists of one or more hogel subsets, and each hogel subset and its associated layer define a slab volume 80 referred to as a sub-volume of the layer. The method further includes volumetric rendering 82 of the light field associated with each layer using the volumetric data included in the layer, according to the sampling scheme. The rendered light field associated with each layer is upsampled and blended into a single output light field image 84.

[0113] Various features of the invention will become apparent from the following detailed description and illustrations. The design factors, construction, and use of the light field volumetric rendering techniques disclosed herein are described with reference to various examples of representative embodiments, which are not intended to limit the scope of the invention described and claimed herein. Those skilled in the art will recognize that, according to the teachings of the invention, other undisclosed variations, examples, and embodiments may exist without departing from the scope and spirit of the invention.

[0114] Layered Scene Decomposition (LSD) CODEC System and Method

[0115] The Layered Scene Decomposition Encoding and Decoding (CODEC) employs a strategy that utilizes known sampling, rendering, and view composition methods to generate light field displays. This strategy is then combined with a novel layered scene decomposition strategy disclosed herein, including its derivation, implementation, and application.

[0116] 3D display

[0117] As previously known in the art, conventional displays consist of spatial pixels that are substantially uniformly distributed and organized into a two-dimensional array, allowing for ideally uniform sampling. In contrast, three-dimensional displays require both spatial and angular sampling. While the spatial sampling of a typical three-dimensional display remains consistent, the angular sampling is not necessarily considered consistent in terms of the display's footprint in angular space. For a review of various optical field parameterizations for angular ray distribution, see U.S. Patent No. 6,549,308.

[0118] Angular samples, also known as the direction components of the light field, can be parameterized in various ways, such as the planar parameterization taught by Gortler et al. in "The Lumigraph". When the light field function is discretized according to position, the light field can be understood as a regularly spaced array of a planar parameterized pinhole projector, as taught by Chai in "Plenoptic Sampling". For a fixed x f y f Element image LF(x) f y f (u, v) represents a two-dimensional image, which can be understood as an image projected using a pinhole projector parameterized with arbitrary light rays. For a light field display, a continuous elemental image is represented by a finite number of light field radiation samples. For an idealized planar parameterized pinhole projector, the finite number of samples is mapped onto the image plane as a regularly spaced array (the regular intervals in the plane do not correspond to the regular intervals in the corresponding angular direction space).

[0119] For a typical 3D light field display, the set of points and directions is defined by the physical spacing between the planar display plane and its individual light field center elements. However, it is well known that displays can be formed on curved surfaces; therefore, the set of points will lie on the curved display surface, or any other desired display surface geometry conceivable. In an abstract mathematical sense, a light field can be defined and represented on any geometric surface and does not necessarily correspond to a physical display surface with actual physical energy emission capabilities. The concept of surface light fields in the literature, such as Chen et al., illustrates this situation.

[0120] Considering planar parameterization is not intended to limit the scope or spirit of this invention, as the directional components of the light field can be parameterized by a variety of other arbitrary parameterizations. For example, lens distortion or other optical effects in a physically manifested pinhole projector can be modeled as planar parameterized distortions. Furthermore, display components can be defined by warp functions, as taught by Clark et al. in "A transformation method for the reconstruction of functions from nonuniformly spaced samples".

[0121] The warping function α(u, v) defines the parameterization of the warping plane of the pinhole projector, thereby producing an arbitrary alternating angular distribution of directional rays in the light field. The angular distribution of rays propagating from the light field pinhole projector is determined by the focal length f of the pinhole projector and the corresponding two-dimensional warping function α(u, v).

[0122] An automated stereoscopic light field display that projects light fields for one or more users is defined as:

[0123] D = (M x M y N u N v f, α, D LP )

[0124] Among them (M) x M y ) are the horizontal and vertical dimensions of the display's spatial resolution, and (N) u N v ( ) refers to the horizontal and vertical dimensions of the display's angular resolution components. The display is an idealized array of light field projectors with a pitch D. LP The focal length f and the warping function α are used to define the ray direction distribution of the light field projected onto the display.

[0125] Driven light field display D = (M x M y N u N v f, α, D LP The light field LF(x, y, u, v) requires M in the x direction. x Light field radiation sampling, M in the y direction y Light field radiation sampling, and N in the u direction u and N v Light field radiation sampling. Although D is defined as a single warp function α, if there are significant microlens variations in a real pinhole projector, resulting in a significant change in the angular ray distribution from one microlens to another, then each light field plane parameterized pinhole projector in an idealized light field pinhole projector array can have a unique warp function α.

[0126] Light field display rendering

[0127] Surface rendering

[0128] In "Fast computer graphics rendering for full parallax spatial displays," Halle et al. provided a method for rendering an inner frustum volume 18 located on the display (e.g., Figure 6The method for objects within the volume 20 of the outer truncated body (as shown). Figure 6 The display shows a light field display representing objects within a volume region defined by two separate frustums, where the inner frustum volume 18 is located behind the display plane 10 (i.e., inside the display), and the outer frustum volume 20 is located in front of the display plane (i.e., outside the display). As shown, various objects (illustrated as prisms and circles) are located at different depths of the display plane 10.

[0129] Halle et al. taught a frustum rendering technique in which the inner frustum volume 18 and the outer frustum volume 20 are rendered as two different light fields. Through a depth merging process, the inner frustum volume LF is... O (x, y, u, v)18 and the volume of the outer truncated body LF P (x, y, u, v)20 is recombined into a single light field LF(x, y, u, v).

[0130] This technique uses a pinhole camera to render the model to generate single-element images of the light field. Each element image (i.e., each rendered planar parametric pinhole projector image) requires two cameras: one to capture the inner frustum volume 18 and another to capture the outer frustum volume 20. Halle et al. taught the use of a standard orthophoto camera and its conjugate pseudomirror camera to render pinhole projected images in the light field sampling region. For the pinhole camera C, the corresponding conjugate camera is denoted as C0. * .

[0131] To capture elemental images in a light field display using a projector parameterized with a warp function α, a general pinhole camera reparameterized based on an ideal plane-parameterized pinhole camera is used. As taught by Gortler et al., a pinhole camera C with a focal length f has parameterized rays created by two parallel planes. The pinhole camera C captures image I. C (u, v), where (u, v) are the coordinates in the ray parameterization plane. Generalized pinhole camera C α Based on a planar parametric camera using a two-dimensional, continuous, reversible time-warping function, as taught by Clark et al. Using the warping function α(u, v), the reciprocal is given. Therefore, C a The image is I Cα =I C (α(u, v)).

[0132] Given a generalized pinhole camera C a Forming a conjugate generalized camera To complete the rendering of the double-truncated body. From M x ×M y The raster-generated view of the universal pinhole camera pair will be rendered to render the light field displayed in the light field.

[0133] Therefore, rendering is necessary to generate a given light field display D = (M x M y N u N v f, α, D LP The set of all universal pinhole camera pairs whose light field LF(x, y, u, v) is defined as:

[0134]

[0135] A group of orthogonal cameras (O={(C α (x, y)|1≤x≤M x , 1≤y≤M y}) Captures light field images corresponding to the volume 18 of the inner frustum, and a set of conjugate generalized cameras Capture an image corresponding to the outer frustum volume 20. As described above, the inner frustum volume 18 and the outer frustum volume 20 combine to form a single light field.

[0136] Layered scene decomposition and sampling scheme

[0137] The sampling gap taught by Graziosi et al. and the all-optical sampling theory taught by Zwicker et al. provide complementary optical field sampling strategies: Graziosi et al. increase downsampling for distant objects (ΔEI), while Zwicker et al. increase downsampling for nearby objects (N). res Downsampling of the scene. However, when downsampling a single light field representing the scene, the combination of these strategies does not guarantee compression. Therefore, this invention divides the multidimensional scene into multiple layers. This division into multiple (data) layers is referred to as hierarchical scene decomposition in this paper. Wherein, K1 and K2 are natural numbers, and L is defined as (K1, K2, L... O L P ),like Figure 4 As shown, the inner and outer frustum volumes of the 3D display are divided. The inner frustum volume 18 is divided into a set of K1 inner frustum layers 40, in which... Each layer in the inner frustum body layer 40 is defined by a pair of boundary planes, the distance between which the boundary planes and the display surface 10 is... and distance Parallel to the display surface. The outer frustum volume 20 is divided into a group of K2 outer frustum layers 42, in which... Figure 1C A single outer frustum layer 42 is shown. Each outer frustum layer 42 is defined by a pair of boundary planes, the distance between which the boundary planes and the display surface is shown. and distance Parallel to the display surface. For example... Figure 1CAs shown, the outer frustum layer 42 is constrained by a pair of boundary planes 24 and 26, including a first boundary plane 26 and a second boundary plane 24. In an alternative embodiment, the inner frustum volume 18 and the outer frustum volume 20 can be divided by different layering schemes.

[0138] Due to scene constraints on the planar boundary regions of the layers, each layered scene decomposition layer has an associated light field (also referred to here as a "light field layer"). Consider a layer with an inner frustum of 40... Or has an outer truncated body layer 42 Light field display D = (M x M y N u N v f, a, D LP Perform a hierarchical scene decomposition L = (K1, K2, L) O L P Inner flat truncated body light field It consists of a set of universal pinhole cameras O={C a (x, y)|1≤x≤M x , 1≤y≤M y The equation is generated by constraining it to image only for the space at a distance d from the light field display surface, where Therefore, for a given x, y, and C a For the inner truncated body layer 40 (x, y) ∈ C, calculate... Similarly, the light field of the outer truncated body A set of universal pinhole cameras The equation is constrained to image only spatially at a distance d from the light field display surface, where... Therefore, for a given x, y, and C a Calculate the outer truncated body layer 42 of (x, y) ∈ F.

[0139] We can further define the light field sets of the inner and outer frustum regions associated with the hierarchical scene decomposition L. Assume a hierarchical scene decomposition L = (K1, K2, L...) O L P The light field display D = (M) x M y N u N v f, a, D LP The light field set of the inner flat truncated volume region is defined as follows: The light field set of the outer truncated cone region is defined as

[0140] By definition, a layered scene decomposition generates a light field for each layer. For any layered scene decomposition, an orthographic camera generates an inner frustum volume 18 light field, while a pseudo-camera generates an outer frustum volume 20 light field. If these generalized pinhole cameras capture a scene consisting only of opaque planes, then each point in the light field has an associated depth value, indicating the distance from the generalized pinhole camera plane to the corresponding point in the spatial image. Given a light field... or hour, Depth map is formally defined as and Depth map is formally defined as The depth map D does not intersect with the surface corresponding to the light rays from the relevant generalized pinhole camera. m =∝. Across these domains, and In other words, the depth map associated with the light field of the layered scene decomposition layer is constrained by the depth boundary of the layer itself.

[0141] The merge operation can recombine the decomposed layers of a layered scene into inner and outer frustum volumes of 18, 20, or LF. O and LF P The light fields of the inner and outer frustum volumes 18 and 20 are merged using the * operator. m Merge them. For example, given two arbitrary light fields LF1(x, y, u, v) and LF2(x, y, u, v), where i = argmin j∈{1，2} D m [LF j ](x, y, u, v), * m Defined as:

[0142] LF1(x, y, u, v)* m LF2(x, y, u, v) = LF i (x, y, u, v)

[0143] Therefore, by merging the light fields associated with the inner and outer truncated body layers 40 and 42, the light from set O can be obtained. LF and P LF China resumes LF O (x, y, u, v) and LF P (x, y, u, v). For example:

[0144]

[0145]

[0146] This provides a layered scene decomposition operation and a reverse operation that merges data to invert the decomposition. Understandably, performing layered scene decomposition using K layers creates K times more light fields than a single light field. The values ​​of the layered scene decomposition are in the layer-induced light fields; these light field layers are better suited for downsampling than the original total light field or the inner frustum volume 18 or outer frustum volume 20 light field because the total data size required for multiple downsampled layered scene decomposition light field layers with appropriate sampling schemes is significantly smaller than the size of the original light field.

[0147] Those skilled in the art will recognize that various sampling schemes can successfully sample the light field. The sampling scheme S provided is not intended to limit or depart from the scope and spirit of the invention, as other sampling schemes can be used, such as specifying a single sampling rate for each element image in the light field of a layered scene decomposition layer. Relatively simple sampling schemes can provide efficient CODEC and greater sampling control; therefore, this invention provides a simple sampling scheme to illustrate the invention without limiting or departing from its scope and spirit.

[0148] The light field sampling scheme provided by the present invention represents a light field encoding method. Given a display D = (M x M y N u N v f, α, D LP ) and hierarchical scene decomposition L = (K1, K2, L o L P The present invention provides a method for M. x ×M y binary matrix M S [l i ](with L c or L P any layer l i A sampling scheme S associated with L, and a mapping function R(l) i ), used to divide each layer l i Mapping to R(l) i )=(n x n y ). (x m y m M at ) s [l i The binary ({0, 1}) entries in the sampled image indicate whether the element image is included in the sampling scheme. (1) indicates that it includes And (0) means not included R(l i )=(n x n y Indicates resolution nx ×n y For light field The elements in the image are sampled.

[0149] This invention also provides a hierarchical scene decomposition light field coding method based on all-optical sampling theory. The following describes the inner frustum volume L of the hierarchical scene decomposition L. c 18 is relevant, but the volume L of the outer truncated head is... P 20 can be encoded in a similar way.

[0150] For each l i ∈L c Corresponding light field The depth map is limited to d within the range Based on the above sampling scheme, this invention uses the following equation to create sampling scheme S, in order to guide the creation of...

[0151]

[0152] In other words, ΔEI guides the M associated with each hierarchical scene decomposition layer. i The distance between "1" entries in the matrix. The following formula sets the image of each element in the layer. Resolution:

[0153]

[0154] This sampling scheme uses ΔEI and N res The sampling rate used to drive individual layered scene decomposition layers can be considered as a layered all-light sampling theoretical sampling scheme (also referred to herein as the "all-light sampling scheme"). This all-light sampling scheme is based on a display using the all-light sampling theoretical identifier function α(t) = t. This layer-by-layer sampling scheme provides lossless compression for front-parallel plane scene objects, where objects within a layer do not occlude each other.

[0155] The assumption that only frontally parallel planar scene objects are limited and do not represent typical scenes; intra-layer occlusion is inevitable, especially for larger layered scene decomposition layers. To capture and encode the entire potential scene without introducing significant perceptible artifacts, in addition to the all-light sampling scheme of this invention, the system can also utilize information.

[0156] Volume rendering

[0157] Levoy (1988) first proposed a direct volumetric rendering method to generate images of 3D volume datasets without explicitly extracting geometric surfaces from the data. Kniss et al. proposed that, although the dataset is interpreted as a continuous function in space, it is represented by a uniform array of 3D samples for practical purposes. In graphics memory, volumetric data is stored as a stack of 2D texture slices or as a single 3D texture object. The term voxel represents a single "volume element," similar to how the terms pixel represent an "image element" and texel represent a "texture element." Each voxel corresponds to a location in the data space and has one or more associated data values. The values ​​at intermediate locations are obtained by interpolating the data of adjacent volume elements. This process, called reconstruction, plays a crucial role in volumetric rendering and processing applications.

[0158] The role of optical models is to describe how light interacts with particles within a volume. More complex models consider light scattering effects by taking into account illumination (local) and volumetric shading. Optical parameters are specified directly from data values ​​or calculated by applying one or more transfer functions to the data to classify features within the data.

[0159] The core of volumetric rendering calculations is calculating the volumetric rendering integral along the view ray. While theoretically this is expressed in the art as a continuum-based integral calculation, in practice it is typically based on discrete and approximate calculations known as compositing (Ikits et al. 2004). Compositing can be performed along the ray in either front-to-back or front-to-front order. It is also possible for some ray segments to be composited within themselves in a certain order, and then composited with adjacent ray segments, while still maintaining the mathematical equivalence of the scheme.

[0160] In this invention, we refer to any calculation performed during or around the volume rendering integral calculation of light as a “volume rendering calculation.” This can include, but is not limited to, lighting calculations, gradient reconstruction that supports lighting calculations, composition operator calculations, transfer function calculations, or any combination of these potentially required calculations.

[0161] Generally, in this invention, any operation that combines two colors into a third color is referred to as "blending". The synthesis described above can be viewed as a form of blending that occurs in each sampling step of the volume rendering integral calculation.

[0162] Interactive direct volumetric rendering is essential for interactively viewing time-varying 4D volumetric data, as progressive rendering may not be suitable for the specific use cases proposed by Martin. Example use cases for interactive direct volumetric rendering include, but are not limited to, rendering of data based on static voxels, rendering without artifacts during rotation, and rendering of data based on time-varying voxels (e.g., 4D MRI or ultrasound, CFD, wave, meteorology, visual effects (OpenVDB), and other physics simulations).

[0163] Volumetric rendering is inherently a parallel process because many rays are being cast, and theoretically their computations can be performed independently without referencing each other. However, as mentioned earlier, even among independent parallel threads, memory accesses often overlap. To achieve the highest possible effective memory throughput, and thus execute the entire computation process more efficiently from a time perspective, it makes sense to coordinate memory accesses. This allows organizing independent threads with overlapping memory accesses to best utilize this overlap. Typically, cached memory is used to take advantage of this overlap.

[0164] Prior work by Hamilton (2007, 2013) and U.S. Patent No. 8,564,617 revealed how volumetric rendering computations of 2D images can be coordinated to allow individual threads to take advantage of this overlap. Modern microprocessors utilize faster but smaller cache memory, helping to automatically utilize access redundancy in the relatively slower but larger DRAM memory. However, the mechanisms these caches use to decide which data to cache are often based on different location principles. These are often good predictors of redundancy and lead to performance enhancements.

[0165] However, in many computations, redundancy patterns are not modeled by these local heuristics for automatic optimization. Several processor technologies allow for the implementation of user-programmable caches and dedicated computing processors in the form of custom application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Modern FPGAs contain large on-chip SRAMs that can be used for custom on-chip programmable circuitry. NVIDIA's GPUs and their Computing Unified Device Architecture (CUDA) application programming interface (API) allow a group of parallel threads to share user-programmable cache memory during execution. Previously, Cell processors had similar user-programmable caches (Hamilton et al., 2007) shared by multiple parallel processing elements.

[0166] Simple light field volume rendering algorithm

[0167] In 2D scenarios and given modern GPU hardware, a very common choice for implementing real-time interactive volumetric rendering is to simply use a ray-based approach. Each ray can be computed independently in parallel using the massive parallelism of modern GPUs. For each ray, this involves a simple intersection calculation with the boundary volumes of the volumetric data, followed by ray propagation at a predefined sampling rate. A simple approach to interactive volumetric rendering of images for a light field display is to use the same 2D approach for each hogel in the display. One method is to render the inner (orthogonal) and outer (artifact) frustum images separately, then synthesize them into a single image using a volumetric rendering integral discretization equation. One drawback of this approach is the need to buffer the inner and outer frustum computations separately, which can require significant memory storage and therefore memory bandwidth, as light field images require a large number of rays compared to traditional 2D images.

[0168] For each hogel in the light field display, such as Figure 1A As shown, based on the hogel field of view θ16 and hogel center position 58 on display plane 10, there exists a frustum region associated with the hogel. Simply put, each hogel must be rendered twice, for both the inner and outer hogels, and then composited. This results in two rays per direction represented by the display, thus billions. Another approach, quite obvious in the prior art, is to define a near clipping plane 12 and a far clipping plane 14, where rays travel from the far clipping plane 14 to the near clipping plane 12 and vice versa. This results in one ray per pixel, with multiple samples per pixel, thus many memory accesses that can all be individual DRAM accesses. In practice, some cache coherence can be utilized, but often much cache coherence is not. Accelerating ray volume rendering requires twice the speed; the first is to reduce the total number of samples required while still maintaining the quality of the resulting image; the second is to sort memory accesses aligned with ray calculations to maximize access coherence. This allows the use of caching to reduce the total amount of DRAM accesses required.

[0169] The process can be simplified by calculating the path of light rays from the inside to the outside frustum (and vice versa), all of which is performed in a single, continuous, parallel execution thread. As described herein, a near clipping plane 12 is defined in the outer frustum and a far clipping plane 14 is defined in the inner frustum, and light rays are traced from one clipping to another by displaying plane 10 at the position of its corresponding hogel.

[0170] The problem with the above approach is that numerous ray projection calculations must be performed for typical and future light field displays. For each ray, volumetric data needs to be sampled multiple times. While the voxel access patterns for these parallel rays are straightforward to implement and often overlap, they still result in redundant DRAM memory accesses to the voxels. In short, this leads to a performance-challenging approach, and scaling to larger displays presents a challenge.

[0171] Accelerate light field volume rendering

[0172] Consider the monitor D = (M x M y N u N v , α, D lp ) and hierarchical scene decomposition L = (K1, K2, L O L P ) and the light field LF(x, y, u, v) driving the display D. Let f denote the focal length of the pinhole projector associated with the hogel of the display D.

[0173] Consider located at L c or L P Layer l in p (These are either inner or outer frustum sections). Consider l p Some d min (l p ) and d min (l p ), representing the boundary plane of the layer. According to the definition of D, the display and its corresponding light field are defined by M x × M. y The hogels are composed of a two-dimensional array. These hogels can be considered as H... i，j (1<i <M x and 1 < j < M y The index of layer l. p The sector partition P can be defined relative to the function ΔEI, and as a hoglé set, it can be partitioned at most into ΔEI(d min (l p )) 2 Size subsets, thus subsets are at most based on ΔEId min (l p (i, j) are formed in the large and small subintervals of ).

[0174] Partitions can be created such that the size of the subset ΔEI(d) min (l p )) 2 They are all the same, that is, if ΔEId min (l p The average score is M x and M yIn practice, this may not happen. In such cases, the sizes of the subsets can be inconsistent, or they can be created such that the length of the sub-intervals they are based on is less than the size ΔEId. min (l p ).

[0175] Figure 1C Visually showing the outer flat truncated body layer l p A 42-sector partition, where layer l p It can also be an inner frustum layer, an outer frustum layer 42, or a combination thereof. A subset of hogels is created relative to ΔEI 30, and its size is such that it can capture or utilize the overlap of ray samples at a given layer depth between hogels within the subset to the maximum extent. In practice, the subset size may need to be adjusted to align with practical constraints such as cache size or other aspects of the underlying architecture.

[0176] Consider the display D, the layered scene decomposition L, and the light field LF. Consider any layer l of L in the voxel mesh 22. p 42. Sector partitioning l relative to ΔEI30 p Consider P. Consider the Hoglund H of P. s Subset. H s A set of hogels defines the sub-frustum of the main display's dual frustum, and the field of view is defined by the sum of all hogels. For example... Figure 1C As shown, layer l p The slab 28 of 42 is defined relative to subset H. s A volume region, which consists of layer l p 42 and H i The intersection point of the hogel-induced sub-frustum is defined. An important observation is that any ray associated with a hogel within a given subset of hogels interacts with the slab 28 relative to layer l. p 42 intersect. All samples occurring within the layer along these rays occur within this slab 28. Therefore, slab 28 can serve as a volume unit for user-programmable cache behavior, rather than a fixed-function cache.

[0177] To accurately determine the voxels to be cached when computing associated ray samples, it is necessary to define the concept of how the slab 28 intersects with the voxel data. As a practical example, consider a voxel dataset V defined over some discrete domain. (x，y，z) Given layer l p 42 slab S i 28 and the given reconstruction neighborhood size required to reconstruct the sample from the voxel data (for discussion, assuming uniformity KxKxK, but it can be non-uniform). Then the slab S i 28 and layer l pThe intersection of 42 is defined as the set of voxels of V contained in the region of slab 28, plus those voxels located near the strictly contained voxels kxKxK in slab 28.

[0178] like Figure 1C As shown, for a depth of d min and d max layer l p A subset P of the sector partitions P of the layered light field of depth i*f. s A frustum region can be defined as the sum of the intersections of the volumes enclosed by layer L and the frustum volumes associated with all hogels of a given sector. This frustum region can be called P relative to layer L. s Slab 28.

[0179] As described in this article, a method combining parallel light field volumetric rendering calculations with caching is employed. Theoretically, for ray-casting volumetric rendering methods that render light field images, each ray needs to have its volumetric rendering integral calculated along its path at a selected sampling rate, which may be uniform or variable along the ray path. The final output of this ray calculation is the color value of the pixel element of the rendered light field image associated with the ray. It is these multiple ray calculations that constitute the entire desired light field image. Theoretically, each ray pixel calculation can be performed completely independently of other calculations and will produce the same rendering output.

[0180] However, it is generally advantageous to execute these theoretically independent parallel computing threads in a more coordinated manner, which allows threads accessing the same slow memory locations to make repeated accesses using faster cached memory, thus mitigating the memory throughput and latency penalties associated with repeated access to slow memory.

[0181] The key to the proposed acceleration method is the construction and coordination of independent ray computing threads so that threads with overlapping memory access requirements can execute concurrently and can utilize a shared fast cache to accelerate the entire ray computing process.

[0182] A key observation is that, given a layered scene decomposition, all rays will pass through each layer at some point. Each layer defines a hogel subset, which further subdivides the layer into slab volumes. Therefore, we can further observe that when any ray passes through a layer, it also intersects with at least a single slab volume. We can further observe that all rays corresponding to a hogel subset will intersect with the slab volume associated with that hogel subset.

[0183] The key to the method disclosed in this paper is to coordinate ray calculations so that all rays associated with the hogel subset intersect the same slab volume, and to perform any partial ray calculations on each ray segment within the slab volume in a synchronous manner.

[0184] In computer science, process synchronization refers to multiple processes connecting or shaking hands at a certain point to reach an agreement or execute a sequence of actions.

[0185] The key to this method is that all rays corresponding to a hogel subset are synchronized and intersect with the corresponding slab 28, as they need to share a common, programmable local storage cache for greater efficiency. It is similar to the method described earlier, except that the hogel subset must be scheduled and synchronized around the local storage cache. Within the hogel subset group, all voxels intersecting with slab 28 should be preloaded into local storage memory before the ray calculation step when all rays initially enter slab 28 in the synchronized hogel. Once all rays are in slab 28, they will perform reconstruction calculations associated with resampling points along the ray path by simply accessing voxel (or other) volume elements in the local storage cache memory. Rays travel or “stop” along the ray path at discrete points, referred to here as resampling points.

[0186] One implementation of this process involves designating a single computation thread associated with a ray intersecting a single slab volume as the "master" and other threads as "slaves," because the master thread initiates slow memory accesses to initialize fast cache memory for itself and the slave threads in subsequent computations. Slave threads must synchronize with the master thread because they must wait until the required values ​​are preloaded from slow memory into cache memory before they can access those values ​​and then perform subsequent ray calculations. In the code example, we refer to the master ray as the "master ray" and the slave threads as "secondary rays." A possible implementation of this method is described using the following pseudocode:

[0187] for every layer L in(insert right symbols for layer list),in the order from near clip to far clip

[0188] calculate the hogel partition relative to the layer L,call it P.

[0189] Produce a list of subsets of hogels.

[0190] for each subset of hogels

[0191] for every ray-pixel R in this subset of hogels

[0192] determine if R is a primary ray for its hogel subset

[0193] determine the slab of L relative to this hogel subset

[0194] if (R is primary ray)

[0195] load all voxels that intersect the slab of L into local store("volumedata cache")

[0196] for every sample along ray R while in layer L

[0197] density=sample volume data cache to reconstruct density at samplelocation

[0198] accumulate alpha and color using previous values(from previous layerif necessary)and transfer function evaluated with density and store in alpha_accum and color_accum,respectively.

[0199] Write color_accum to the light field pixel for R

[0200] Many parameters can be changed to affect performance and / or suitability for implementation on a particular architecture. The chosen layer scheme will significantly determine the size of the blank 28. A larger blank 28 size will intersect with more voxels, thus requiring a larger preloading stage before loading the user-programmable voxel cache. For example, for a display with a large FoV located in a wide layer far from the display, a relatively very large blank 28 size will result, potentially intersecting with many voxel data points.

[0201] There are two obvious ways to solve this problem. One can choose a sector partition with a smaller hogel subset (less than d). min (l p ) 2 This would reduce the width of the slab 28 formed with each layer. This could reduce the overall theoretical ability of the method to utilize overlap, thereby increasing the required total DRAM bandwidth, but depending on the circumstances, this might be an acceptable trade-off. Another possibility is to choose narrower layers, as this would inevitably reduce the other width dimensions of the resulting slab 28.

[0202] Consider slab 28 relative to a certain layer and certain sector partitions. It can be seen that maintaining the same layering scheme results in the geometry of slab 28 remaining unchanged as the directional resolution of the display scales. It is easy to see that for any hogel in this subset of hogels, when the corresponding rays intersect slab 28, they only require slab 28 to be aligned with V... (x，y，z) A set of voxels at the intersection of datasets.

[0203] Therefore, regardless of the orientation resolution, the number of voxels loaded into the cache relative to a single slab 28 remains constant. As the orientation resolution increases, and all other things remain constant, the proportion of voxel accesses from the cache increases, while the memory bandwidth required by DRAM remains unchanged. This is a good scaling characteristic, and this optimization method can show advantages in terms of higher quality compared to the traditional brute-force approach.

[0204] Plenoptic downsampling strategy to reduce memory bandwidth requirements

[0205] Another approach to improving light field volumetric rendering performance and reducing the direct access required by DRAM is to reduce the total number of reconstruction samples required along the ray path, thereby limiting the total number of voxel accesses. It can be observed that in the published work on layered scene decomposition CODEC based on surface scenes, layers can be downsampled according to their distance from the display plane 10. This is the principle behind the all-light sampling theory described by Zwicker et al. In practice, it has been observed that in many cases, this downsampling results in very small or non-substantial (ultimately imperceptible) changes in the resulting light field image when viewed on a light field display.

[0206] Specifically, for a given display configuration of a hogel with focal length f, in order to represent a layer with maximum depth d, the required directional resolution is:

[0207]

[0208] Within a given sector P, any ray associated with Hogel intersects slab 28 (relative to layer L). All samples appearing along these rays within layer L also appear within slab 28.

[0209] It can be observed that, in worse cases, a ray must be calculated for each pixel. For a given slab 28, the set of sampled rays associated with the fan-shaped subset can actually be reduced or increased in number depending on the maximum layer depth. When fewer rays are used, a single ray represents multiple pixels, rather than a one-to-one correspondence. The purpose of this is to ensure that downsampling does not affect or substantially impact the quality of the resulting light field image.

[0210] Previously, in the context of layered scene decomposition, each layer was associated with a sampling scheme that indicated which pixels to include (or exclude) in the light field image associated with each layer. We can use similar conventions in volumetric rendering methods, also based on layered scene decomposition, to precisely specify the downsampling scheme. We propose further using a convention where, in the context of rendering the light field associated with a given layer using a ray casting procedural (as in the volumetric rendering proposed in this paper), rays corresponding to pixels included in the sampling are called "active," while rays excluded by the sampling scheme are called "dormant."

[0211] One object of this invention is to sample the imaging volume along the ray projection accumulation process using the same ray sampling principle. In other words, a single ray is initially projected, then branches into multiple rays, thus recursively branching into even more rays during the volume accumulation process. This branching process can be determined by the sampling principle. For example, one possible implementation is that for each focal length of depth, the directional resolution required to capture detail at that depth must increase by 1. Therefore, theoretically, the rays branch along each focal length increment according to this principle.

[0212] The advantage of this method is that fewer volume samples are required to calculate the volume rendering integral because the light accumulation process is close to the display. When the light source approaches the display's maximum depth of field, the sampling rate is close to that of the original method, where each hogel pixel is associated with a single ray throughout the entire light path. This reduction in the number of required samples can be quantified and analyzed mathematically.

[0213] Compared to the original scheme, the required sample score for the proposed scheme was calculated, which is the total number of hogel pixels accumulated at a constant rate across all depths 10 from the display plane. Without sacrificing generality, analysis of individual hogels is considered, as all cases are applicable to a multiplication factor based on the number of hogels.

[0214] Let D iIndicate the directional resolution of the light field (assuming D). r *D, hogel).

[0215] Based on the concept of all-optical sampling, which assumes an ideal planar parametric light distribution, and the requirement to fully sample the object at a distance of i*f, where i is an integer and f is the focal length of the Hogel lens, the directional resolution must be i. Therefore, one needs to sample the object at a depth of i*f. 2 Light.

[0216] Considering that light propagates only within a single frustum, assume that for volumetric rendering, there exists a ray casting process where light travels from the display surface to the maximum depth of field of the display. In depth D... r *f. In each layer of width f, when the layer is at depth i*f, at least i is required. 2 Light.

[0217] Assume that the volume is sampled by light at a rate M of f for each distance traveled from the display surface. Therefore, the expression describes the number of samples associated with each hogel throughout the entire process:

[0218]

[0219] Under simple sampling, where all D of each hogel r *D r A pixel is associated with a ray throughout its entire path, and assuming the same sampling rate M, the number of samples will be per f-width layer. Therefore, in layer D, the total number of samples is

[0220] Therefore, the sample ratio under the proposed sampling scheme and the original sampling scheme can be calculated:

[0221]

[0222] Notice:

[0223]

[0224] For lower values ​​D r For example, D r =32, displaying Ratio samples The rating is 0.36, very close to the limit value of 1 / 3. Therefore, it can be said that for a useful value D... r The number of samples is reduced to about one-third. In practice, this would lead to a higher-performance algorithm, since standard, simple methods have limited bandwidth.

[0225] Below, we describe a possible approach to implementing this strategy using pseudocode. Our proposed implementation involves marking a ray as "active" or "dormant" within a given layer and a given hogel, based on whether the ray is included in the sampling scheme.

[0226] The advantage of this method is that when light rays are dormant in a specific layer, we can essentially avoid performing volumetric rendering calculations on the light rays in that specific layer. This means that sampling of volume data and other volumetric rendering calculations, such as gradient reconstruction calculations, lighting calculations, or transfer function calculations based on volume data, can be avoided.

[0227] However, even when a light ray is in a dormant state, it must still be assigned a color and an α value in order to perform a hybrid computation for the same light rays associated with adjacent layers, where the same light rays may or may not be in a similar dormant state. In this invention, we refer to the process of assigning a color to dormant light rays as "upsampling." We suggest that a preferred method for achieving this operation is to use interpolation with adjacent light rays, which may involve nearest-neighbor interpolation methods, linear interpolation, or potentially edge-adaptive interpolation techniques.

[0228] The possible implementations of this method are described using pseudocode:

[0229]

[0230]

[0231] This can also be combined with the previously described method of calculating cached volumetric slab elements and synchronously corresponding volumetric rendering rays, based on specified downsampling for each layer. A method that simultaneously utilizes downsampling and layer-based caching is used to derive a method that generates a rendered light field image as output. The pseudocode is as follows:

[0232]

[0233]

[0234] Layer-based compression analysis

[0235] Creating a real-time rendering and delivery system requires predictable compression ratios and a downsampling standard (not indicating an achievable compression ratio). The following provides a compression analysis of the layered scene decomposition coding strategy of this invention.

[0236] As mentioned earlier, downsampling the optical field solely based on all-optical sampling theory cannot guarantee compression ratio. This invention provides a downsampled optical field coding strategy that allows for low-latency, real-time optical field CODEC. In one implementation, a complementary sampling scheme based on all-optical sampling theory is used, employing ΔEI and N... resThis drives the sampling rate of each layer of the layered scene decomposition. Layered scene decomposition represents the entire 3D scene as multiple light fields, expanding the scene representation as a factor of the number of layers. The invention further envisions that, when the layer depth is appropriately selected, compression ratio can be guaranteed when combined with downsampling based on all-optical sampling theory.

[0237] For a given hierarchical scene decomposition layer l i Corresponding light field The limited depth range of this layer provides a guaranteed compression ratio for its light field. The compression ratio achievable by downsampling a scene entirely contained within a single layer can be explained by the following theorem:

[0238] Theorem 1

[0239] Consider an isotropic directional resolution N = N u =N v The display D = (M x M y N u N v f, a, D LP ), hierarchical scene decomposition L and related sampling scheme S = (M s (R). Assuming a corresponding light field. Layered scene decomposition layer l i , making d min (l i ) < Z DOF (D), and The selected entries are therefore the distance between the "1" entries is set to ΔEI(d). min (l i )) and R(l i )=Nr es (d max (l i )). Compared to the layered scene decomposition layer l i The compression ratio associated with S is Evidence 1

[0240] Consider a layered scene decomposition layer within the maximum depth of field of the display, where for 0 < c, d ≤ Z. DOF , and therefore, and as well as Therefore, ΔEI(d) min (l i ))=N / c and N res (d max (l i ))=N / d.

[0241] Based on this sub-sampling rate, the system needs (N / c) th The image is element-wise, therefore the compression ratio provided is 1:(N / c). 2 Element image subsampling provides 1:d 2 Compression ratio. Therefore, the overall compression ratio is 1:(N / c) 2 *1: d 2 =1:N 2 (d / c) 2 Compression factor term Determine the compression ratio.

[0242] There may be alternative cases, where d min (l i ) = Z DOF and (d) max (l i It can extend to any depth. Given ΔEI(Z) DOF ) = N and N res Obtain all depths d≥Z DOF The maximum possible value N. Based on this subsampling rate, the system needs to... th Elemental images, thus providing a 1:N ratio for the light field. 2 Compression ratio. When representing planar objects with parallel frontal views, add a compression ratio exceeding Z. DOF The additional layered scene decomposition adds redundant representation capabilities. Therefore, when creating the core coded representation, the maximum depth of field in the layer can be used to optimally decompose the entire scene.

[0243] Given a compression calculation expression for downsampling layer decomposition of a layered scene, it is possible to determine how the compression factor varies with layer parameters. For layers of fixed width, or for certain w, d max (l i )-d min (l i ) = w, when d max (l i )-d min (l i When the distance to the display plane is closest to 10, C f-terms are minimized. Therefore, layered scene decomposition layers located near display plane 10 require narrower widths to achieve the same compression ratio as layers farther from display plane 10. This compression ratio analysis can be extended from display plane 10 to depth Z. DOF The scene is divided into multiple adjacent front plane layers in space.

[0244] Theorem 2

[0245] Consider an isotropic directional resolution N = N u =N v The display D = (Mx M y N u N v f, a, D LP ), hierarchical scene decomposition L and related sampling scheme S = (M s R). Let S LF =M x M y N u N v This indicates the number of image pixels in the light field. The compression ratio of a hierarchical scene decomposition representation can be defined as:

[0246]

[0247] Evidence 2

[0248] For a given layered scene decomposition layer, downsample according to the compression ratio:

[0249]

[0250] To calculate the compression ratio, we calculate and sum the size of each layer in the compressed form, and then divide the total size of the compressed layers by the size of the light field. Consider the summation, where the size of the compressed layer set is:

[0251]

[0252] Therefore, the compression ratio of the combined layer is:

[0253]

[0254] In a system where the layered scene decomposition layer has a variable width, where d min (i) and d max (i) means i th The compression ratio of the layered scene decomposition representation, considering the front and rear boundary depths of the layer, is:

[0255]

[0256] The sum of constant layered scene decomposition layers It decreases monotonically and tends to 1.

[0257] Therefore, the layered scene decomposition layers located near the display plane 10 achieve a lower compression ratio compared to layers of the same width located further away from the display plane 10. To maximize efficiency, narrower layered scene decomposition layers are located closer to the display plane 10, while wider layered scene decomposition layers are located further away from the display plane; this placement maintains a uniform compression ratio throughout the scene.

[0258] Number and size of layered scene decomposition layers

[0259] To determine the number and size of layers required for layered scene decomposition, a light field display with an identity function α(t) = t is provided as an example. Consideration of this identity function is not intended to limit the scope or spirit of the invention, as other functions may be used. Those skilled in the art will recognize that although the display D = (M x M y N u N v f, a, D LP The identifier α is defined by a single identifier function α, but each light field plane parameterized pinhole projector in a plane parameterized pinhole projector array can have a unique identifier function a.

[0260] To represent the front plane surface without loss of quality (assuming no occlusion), it has a depth Z... DOF The single-layer scene decomposition layer representation of the front boundary from Z DOF To systems extending to infinity. Lossless compression can be defined as a class of data compression algorithms that allow for the perfect reconstruction of the original data from the compressed data. To generate the core representation, layered scene decomposition layers located beyond the deepest layer at the maximum depth of field of the light field display are not considered, as these layers do not provide additional representation from the perspective of the core representation; this applies to the inner and outer frustum volumetric layer sets.

[0261] Within the region from display plane 10 to the maximum depth of field of the display (for the inner and outer frustum volumetric layers), the layered scene decomposition layers use maximum and minimum distance depths that are integer multiples of the light field display f-value. Narrower layered scene decomposition layers provide improved compression per layer, resulting in better overall scene compression. However, the more layers in the decomposition, the greater the processing required for decoding, as more layers must be reconstructed and merged. Therefore, this invention teaches a layer distribution scheme with different layer depths. In one embodiment, the narrower-width layered scene decomposition layers (and, by association, the light field represented by said layers) are located close to display plane 10, and the layer width (i.e., the depth difference between the front and rear layer boundaries) increases exponentially with increasing distance from display plane 10.

[0262] CODEC Encoder / Encoding

[0263] The encoding according to the invention is designed to support the generation of real-time interactive content (e.g., for gaming or simulation environments) and existing cubes captured by a light field universal pinhole camera or camera array.

[0264] For the light field display D, the layered scene decomposition L, and the sampling scheme S, the system encoder generates element images associated with the light field corresponding to each layered scene decomposition layer included in the sampling scheme. Each element image corresponds to a generic pinhole camera. The element images are sampled at the resolution specified by the sampling scheme, and each element image contains a depth map.

[0265] Achieving rendering performance to drive real-time interactive content to multi-dimensional displays with extremely high resolution and size is a significant challenge that cannot be overcome by relying solely on any one of the techniques described in this article, but rather by applying hybrid or combined rendering methods.

[0266] Given an identifier function α, a set of generic pinhole cameras specified by the coding scheme of a given layered scene decomposition layer can be rendered using a standard graphics viewport rendering system. This rendering method results in a large number of draw calls, especially for layered scene decomposition layers whose sampling schemes include a large number of base images. Therefore, in systems that use layered scene decomposition for realistic, automated stereoscopic light field displays, this rendering method alone cannot provide real-time performance.

[0267] Rendering techniques using standard graphics rendering calls restrict the rendering of the planar parametric representation (identification function α) of a general-purpose pinhole camera to perspective transformations. Hardware-optimized rasterization provides the performance required for high-quality real-time rendering on traditional 2D displays. These accelerated hardware features are based on planar parametric representation. Alternatively, parallel oblique projection can be used to render the planar parametric representation of a general-purpose pinhole camera using a standard rasterization graphics pipeline.

[0268] This invention considers the application of rasterization to present a universal pinhole camera view by converting a set of triangles into pixels on a display screen. When rendering a large number of views, each triangle must be rasterized in each view; oblique rendering reduces the number of rendering steps required for each layered scene decomposition layer and can adapt to any arbitrary identification function α. ​​The system uses a parallel oblique projection based on the angle specified by the identification function α. ​​Once the data is rendered, the system performs a “slice and dice” block transformation (see U.S. Patent Nos. 6,549,308 and 7,436,537) to regroup the stored data from its angle groups into basic image groups. When rendering a large number of angles, using only the “slice and dice” method is inefficient for real-time interactive content that requires many separate oblique rendering draw calls.

[0269] Ray tracing rendering systems can also accommodate arbitrary identification functions α. In ray tracing, specifying arbitrary angles does not require significantly higher performance than accepting planar parameterization. However, for real-time interactive content that requires rendering systems using the latest accelerated GPUs, rasterization offers more reliable performance scalability than ray tracing rendering systems.

[0270] This invention provides several hybrid rendering methods for effectively encoding light fields. In one embodiment, the encoding scheme renders layered scene decomposition layers located near the display plane 10, where more images require fewer angle samples, while layers located far from the display plane 10 have fewer images and more angle samples. In related embodiments, perspective rendering, tilt rendering, and ray tracing are combined to render layered scene decomposition layers; these rendering techniques can be implemented in a variety of interleaved rendering methods.

[0271] According to a general illustrative embodiment of the invention, one or more light fields are encoded by a GPU rendering a two-dimensional pinhole camera array. The rendered representation is created by computing pixels in a sampling scheme applied to each layer of the scene decomposition layer. Pixel shaders execute the encoding algorithm. Typical GPUs are optimized to generate up to 2 to 4 pinhole camera views per scene in a single transport frame. The invention requires rendering hundreds or thousands of pinhole camera views simultaneously, therefore employing multiple rendering techniques to render the data more efficiently.

[0272] In one optimized approach, a general-purpose pinhole camera in a layered scene decomposition layer located far from the display plane 10 is rendered using standard graphics pipeline viewport operations, referred to as perspective rendering. A general-purpose pinhole camera in a layered scene decomposition layer located near the display plane 10 is rendered using "slice and dice" block transformations. The combination of these methods provides efficient rendering for a layered all-light sampling theoretical sampling scheme. This invention provides layered scene decomposition layers, where layers farther from the display plane 10 contain fewer high-resolution element images, while layers closer to the display plane 10 contain more low-resolution element images. Using perspective rendering to render fewer element images in layers far from the display plane 10 is efficient because the method requires only one draw call per element image. However, in some cases, perspective rendering becomes inefficient for layers located near the display plane 10 because these layers contain more element images and require more draw calls. Since the element images in layers closer to the display plane 10 correspond to relatively fewer angles, tilt rendering can efficiently render these element images and reduce the number of draw calls. In one implementation, a method is provided for determining where a system should use perspective rendering, skew rendering, or ray tracing to render layered scene decomposition layers. A thresholding algorithm is applied to evaluate each layered scene decomposition layer by comparing the number of element images to be rendered (i.e., the number of perspective rendering draw calls) with the element image size required for a specific layer depth (i.e., the number of skew rendering draw calls). The system implements a rendering method (technique) that requires the fewest rendering draw calls.

[0273] When standard graphics calls are not available, the system can implement ray tracing instead of perspective or tilt rendering. Therefore, in another implementation, an alternative rendering method uses ray tracing to render layers located near the display plane 10, or a portion of layers located close to the display plane 10.

[0274] In a ray tracing rendering system, each pixel in a layered scene decomposition layer is associated with a ray defined by a light field. Each ray is projected, and its intersection with the layered scene decomposition is calculated according to standard ray tracing methods. Ray tracing is highly advantageous when rendering does not conform to the standard planar parameterized identifier function α expected by the standard GPU rendering pipeline because it can accommodate arbitrary ray angles that present challenges to traditional GPU rendering.

[0275] Those skilled in the art will recognize that there are various rendering methods and combinations thereof that can successfully encode layered scene decomposition element images. Other rendering methods can provide efficiency in different environments, depending on the underlying computational architecture of the system, the sampling scheme used, and the identification function α of the light field display.

[0276] CODEC Decoder / Decoder

[0277] The decoding according to the present invention aims to utilize an encoding strategy (sampling and rendering). The core representation is a set of layered light fields from the decomposition of a downsampled layered scene, which are decoded to reconstruct the light field LF. o and LF F Consider a hierarchical scene decomposition L = (K1, K2, L...) O L P ) and related sampling scheme S=(M s The display D = (M) of R) x M y N u N v f, α, D LP ). By deconstructing LF from samples sampled according to the sampling scheme S. O and LF F Light field reconstruction light field LF O and LF F The element image is decoded. Pixel alignment is performed so that the inner and outer frustum volume layers closest to the display plane 10 are viewed first, then the view moves to inner and outer frustum volume layers farther away from the display plane 10 until a non-empty pixel is located, and data from the non-empty pixel is transferred to an empty pixel near the display plane 10. In another embodiment, a particular embodiment may limit the viewing of the inner frustum volume 18 or outer frustum volume 20 of the light field display, thus requiring decoding of the LF. O or LF F one of the.

[0278] In one implementation, the decoding method is represented by the following pseudocode:

[0279] Core Layered Decoding:

[0280] for each l i ∈L o :

[0281] ReconLF

[0282] (front-back vs. back-front)

[0283] Similar process to rebuild LF F Each layer of the scene decomposition is reconstructed from a finite number of samples defined by a given sampling scheme S. Each inner frustum volume layer or outer frustum volume layer is merged to reproduce the LF. c or LF F .

[0284] ReconLF can be performed in various forms, with different computational and post-CODEC image quality characteristics. ReconLF can be defined as a function, for example, given the light field associated with a layer sampled according to a given sampling scheme S, and the corresponding depth map of the light field, it will reconstruct the entire sampled light field. The input to ReconLF is the given sampling scheme S and the corresponding downsampled depth map. Defined Subset of data. Depth-based image rendering (DIBR), described by Graziosi et al., can reconstruct the input light field. DIBR can be classified as a projection rendering method. In contrast to reprojection techniques, ray casting methods, such as screen-space ray casting taught by Widmer et al., can reconstruct the light field. Ray casting is more flexible than reprojection but increases computational resource requirements.

[0285] In the DIBR method, the element image specified in the sampling scheme S is used as a reference "view" to synthesize the missing element image from the light field. As described by Vincent Jantet in "Layered Depth Images for Multi-View Coding" and Graziosi et al., when a system uses DIBR for reconstruction, the method typically includes forward warping, merging, and back projection.

[0286] The application of backprojection techniques avoids the generation of cracks and sampling artifacts in the synthesized view (such as an element image). Backprojection assumes that a depth map or disparity map of the element image is synthesized together with the necessary reference image required to reconstruct the target image; this synthesis is typically performed through a forward warping process. Using the disparity value of each pixel in the target image, the system warps the pixels to their corresponding positions in the reference image; typically, these reference image positions are not aligned on an integer pixel grid, so values ​​from neighboring pixels must be interpolated. Implementations of backprojection known in the art use simple linear interpolation. However, linear interpolation can be problematic. If the warped reference image position is located on or near the edge boundary of the object, the interpolated values ​​can exhibit noticeable artifacts because information from the edge boundary is included in the interpolation operation. The synthesized image is generated with “blurred” edges.

[0287] This invention provides a back-projection technique for interpolation sub-steps, producing high-quality composite images without blurred edges. The invention introduces Edge Adaptive Interpolation (EAI), in which the system incorporates depth map information to identify the pixels required for the interpolation operation to calculate the color of distorted pixels in the reference image. EAI is a non-linear interpolation procedure that adjusts and preserves edges during low-pass filtering operations. Consider a target image I... t (x, y), Reference image I r (x, y) and depth map D m (I t ) and D m (I r The display D = (M) x M y N u N v f, α, D LP This invention utilizes depth map D. m (I t The relative positions of the pinhole camera parameters (f, α, etc.) and the display array of the planar parameterized pinhole projector will be used for each I. t The pixel integer (x, y) is twisted into I r real number (x) w y w ) position. Within (x) w y w In the case where the coordinates are not integers, it must be based on I. r Reconstructed values ​​of integer samples.

[0288] Linear interpolation methods known in the art reconstruct I from the four nearest integer coordinates in a 2×2 pixel neighborhood. r (x w y wAlternative reconstruction methods use larger neighborhoods (e.g., 3×3 pixel neighborhoods) to produce similar results with varying reconstruction quality (see Marschner et al., “An evaluation of reconstruction filters for volume rendering”). These linear interpolation methods are unaware of the fundamental geometry of the signal. Blurred edge images result when reconstruction uses pixel neighborhoods belonging to different objects (separated by edges in the image). Incorrectly including the colors of other objects produces ghosting artifacts. This invention provides a method by using depth maps D... m (I r This reconstruction problem can be solved by using a method that predicts the existence of edges created when multiple objects overlap, and then weights or omits pixel neighborhoods.

[0289] For target image I t Any fixed coordinate (x, y) in (x, y) r y r ), d t Define location depth:

[0290] d t =D m [I r (x r y r )]

[0291] Target image coordinates (x) r y r ) will be distorted to the reference image coordinates (x w y w ).

[0292] For close to (x) w y w The set N is the neighborhood of size m of a point. S ={(x i y i The weight of each neighborhood is defined as: |1≤i≤m}.

[0293] w i =f(d t D m [I r ](x i y i )]

[0294] Where w i It is the depth and the neighbor depth (x) of the corresponding index i. w y w The function of t. The following equation represents a given threshold t. e effective w i :

[0295] w i ={1|d t -D m [I r ](x i y i )|<t e 0|d t -D m [I r ](x i y i )|≥t e

[0296] Threshold t e It is the feature size parameter. The weight function determines how to reconstruct I. r (x r y r ):

[0297] I r (x r y r ) = Recon(w1I r (x1, y1), (w2I) r (x2, y2), ... (w) m I r (x m y m ))

[0298] The Recon function can be a simple modified linear interpolation, where w i The weights are merged with the standard weighting procedure and re-standardized to maintain a total weight of 1.

[0299] This invention also provides a performance-optimized decoding method for reconstructing hierarchical scene decomposition. Consider a hierarchical scene decomposition L = (K1, K2, L...) O L P ) and related sampling scheme S=(M s The display D = (M) of R) x M y N u N v f, a, D LP ). By deconstructing LF from samples sampled according to the sampling scheme S. C and LF P Light field reconstruction light field LF C and LF F The element image is decoded. As mentioned above, a specific implementation can limit the viewing of the inner frustum volume 18 or the outer frustum volume 20 of the light field display, thus requiring LF... C or LFF One of them is used for decoding.

[0300] The missing pixels can be reconstructed by decoding the element image specified by the sampling scheme S. The ReconLF method for a specific layer does not include inherent constraints regarding the order of missing pixels in the reconstructed image of missing elements. The objective of this invention is to reconstruct missing pixels using a method that maximizes throughput; for effective light field displays, sufficiently large light fields require very high data throughput to deliver content at interactive frame rates, thus necessitating improved data transmission for reconstruction.

[0301] Indicative implementation plan

[0302] This article describes a method that demonstrates how to leverage memory access redundancy to render light fields. A concrete example of implementing this method using a parallel processing framework is provided.

[0303] Consider a layered scene decomposition L associated with a light field display D. Based on the previously published form of the CODEC, this should be a set of layers and a sampling scheme S. Given the display D, it is well known that the maximum depth of field of the display is maxDoF(D). It is proposed to choose a layering scheme in such a way that there are layers at each depth such that they are integer multiples of the display's focal length f. For each layer at every "multiple f" depth, the value of the function ΔEI can be calculated. It is known that ΔEI(i*f) = i.

[0304] This invention describes a method in which layers within a single frustum (e.g., inner frustum volume 18) are used for a rendering approach. In volumetric rendering, each computation thread associated with a single ray computes a single instance of the volumetric rendering integral. The integral is discretized as an accumulated computation performed sequentially from front to back along the ray path. Thus, for rendering the inner frustum volume 18, it is assumed that for each pixel in each hogel, the ray begins at the display plane 10 of the inner frustum volume 18 and continues to the far clipping plane 14 of the inner frustum volume 18. Conversely, for rendering the outer frustum, the ray begins at the near clipping plane 12 and ends at the display plane 10. For any ray that begins at the near clipping plane in the outer frustum, it continues with the corresponding ray that begins at the display plane 10 and continues to the far clipping plane 14 in the inner frustum volume 18. The outer and inner frustum ray segments can be computed in parallel separately and then modified into a single final computation through direct volumetric rendering integral discretization.

[0305] Alternatively, ray calculations can be performed in a back-to-forehead order within the frustum. Furthermore, back-to-back calculations can be performed within the frustum, followed by another back-to-back calculation in a different plane, and vice versa. In these calculations, the ordering is fluid, limited only by the mathematical properties of integral calculations.

[0306] Combination of layered scene decomposition surface rendering and layered scene decomposition volumetric rendering

[0307] U.S. Patent No. 10,432,944 describes a method based on hierarchical scene decomposition for rendering surface-based representations at real-time rates. The advantage of this method is that it enables a system that can generate the large number of pixels required for light fields in real time using existing techniques. It is proposed to combine the second stage (decoding stage) of the above process with the light field volumetric rendering method described herein. As a result, a real-time light field rendering system that simultaneously supports volumetric and surface representations can be realized by leveraging the cache-efficient approach embodied in our novel surface decoding and volumetric rendering methods.

[0308] In this invention, it is assumed that a layered scene decomposition (LSD) exists for surface rendering. A widely used decoding method for surface-based layered scene decomposition (LSD) CODEC is described below:

[0309] A layered decoding method based on surface rendering is proposed, namely, rendering based on polygonal surfaces, which can be naturally combined with volumetric rendering methods. The basic idea is to reconstruct each layer in the above process and then merge them with adjacent layers (through m... * (Operator). The composition equations for volumetric rendering must include a merge operator. Therefore, in this more general blending approach, the layer merge operator becomes more general and complex because it performs a more general function, sometimes acting as a merge operator as before, and sometimes as a volumetric rendering ray accumulation function. It is suggested that this new operator be called the blending operator.

[0310] This operator is represented as * c .

[0311]

[0312]

[0313] In practice, this proposal may have potential problems. The surface layer could be quite large (e.g., the power of the two schemes proposed in U.S. Patent No. 10,244,230), creating a very large slab 28 that intersects with a large subset of voxels, thus requiring too much user-programmable cache for practicality. One suggestion to address this issue is to subdivide the surface layer into smaller layers to break down the volumetric rendering portion of the process into more manageable blocks. This is a practical implementation-level detail driven by architectural constraints and is likely determined simply by the amount of user-programmable cache available.

[0314] In practice, for this process to function as part of a real-time rendering system, it must be parallelized for efficient execution. The row-column approach below is considered a good way to implement surface reconstruction CODEC in practice because breaking down the reconstruction into a series of one-dimensional operations reduces the pressure on cache size requirements by trading off the cost of dividing the process into multiple stages and causing buffering latency. In many practical situations, this is an acceptable trade-off.

[0315] Dimensional decomposition and light field reconstruction

[0316] Pass 1:

[0317]

[0318] Pass 2:

[0319]

[0320]

[0321] All publications, patents, and patent applications mentioned in this specification represent the skill level of a person skilled in the art to which this invention pertains and are incorporated herein by reference. Any reference to prior art in this specification is not, and should not be construed as, an admission or suggestion that such prior art constitutes common knowledge.

[0322] As the invention has been described thus, it will be apparent that it can vary in many ways. Such variations should not be considered a departure from the scope of the invention, and all such modifications that are obvious to those skilled in the art should be included within the scope of the following claims.

[0323] References

[0324] AGUS,M.,GOBBETTI,E.,GUITIAN,JAI,MARTON,F.,and PINTORE,G. "GPUAccelerated Direct Volume Rendering on an Interactive Light Field Display".Visual Computing Group.Pula,Italy.Eurographics.2008.

[0325] CHAI,JIN-XIANG,XIN TONG,SHING-CHOW CHAN,AND HEUNG-YEUNG SHUM."Plenoptic Sampling"

[0326] CLARK,J.,PALMER,M.,and LAWRENCE,P.“A Transformation Method for theReconstruction of Functions from Nonuniformly Spaced Samples.IEEETransactions on Acoustics,Speech,and Signal Processing.VOL.ASSP-33.No,4.October 1985.

[0327] GANTER,D.,ALAIN,M.,HARDMAN,D.,SMOLIC,A.,and MANZKE,M.“Light-Field DVRon GPU for Streaming Time-Varying Data”.GV2.Trinity College.Dublin,Ireland.Pacific Graphics.2018.

[0328] GORTLER,STEVEN J.,RADEK GRZESZCZUK,RICHARD SZELISKI,AND MICHAELF.COHEN."The Lumigraph"43-52.

[0329] HALLE,M.,and KROPP,A.“Fast Computer Graphics Rendering for FullParallax Spatial Displays”.

[0330] HAMILTON,M.,ZHAN,L.,MAYNARD,A.,and BORDING,P.R.“Three DimensionalCache Coherency for Multicore Volume Rendering”.Proceedings of theSeventeenth Annual IEEE Newfoundland Electrical and Computer EngineeringConference(NECEC).2007.

[0331] HAMILTON,M.“Real-Time Time-Warped Multiscale Signal Processing forScientific Visualization”.PhD Dissertation,Department of Computing Science,University of Alberta.2013.

[0332] IKITS,M.,KNISS J.,LEFOHN A.,and HANSEN C.Chapter 39,Volume RenderingTechniques.Edited by Randima Fernando.Addison Wesley.

[0333] JANTET,VINCENT."Layered Depth Images for Multi-View Coding"Multimedia.pp.1-135.Universite Rennes 1,2012.English.

[0334] LEVOY,MARC,AND PAT HANRAHAN."Light Field Rendering"SIGGRAPH.pp.1-12.

[0335] LOCHMANN,G.,REINERT,B.,BUCHACHER,A.,and RITSCHEL,T.“Real-time Novel-view Synthesis for Volume Rendering Using a Piece-analytic Representation”.Vision,Modeling,and Visualization.2016.

[0336] MARTIN,S.,BRUTON,S.,GANTER,D.,and MANZKE,M.“Using a Depth Heuristicfor Light Field Volume Rendering”.Trinity College.Dublin,Ireland.2012.

[0337] MARTIN,S.“View Synthesis in Light Field Volume Rendering UsingConvolutional Neural Networks”.Trinity College.Dublin,Ireland.August 2018.

[0338] STEGMAIER,S.,STRENGERT,M.,KLEIN,T.,and ERTL,T.“A Simple and FlexibleVolume Rendering Framework for Graphics-Hardware-based Raycasting”.Institutefor Visualization and Interactive Systems.University of Stuttgart.VolumeGraphics.2005.

[0339] ZWICKER,M.,W.MATUSIK,F.DURAND,H.PFISTER."Antialiasing forAutomultiscopic 3D Displays"Eurographics Symposium on Rendering.2006.

Claims

1. A method for rendering light field volumetrics of a scene in a computer system, comprising: The 3D scene description, including volume data, is divided into multiple layers, each layer having an associated light field and at least one plate, each plate having an associated hogel subset such that all rays associated with the hogel subset intersect the plate; Reconstruction calculations are performed on rays that intersect the plate body by selectively accessing a set of volume elements from the local storage cache memory, the rays intersecting the plate body with resampled points along the ray path; Volume rendering is performed on the light field associated with each layer using the volume data included in each layer, such that volume rendering calculations are performed on the rays in the hogel subset to provide the rendered light field; The volume data intersecting with the corresponding board body is cached, and the volume data is stored in the local storage cache memory; and The rendered light field associated with each layer is blended into a single output light field image.

2. The method of claim 1, wherein only a portion of the volume data is cached.

3. The method of claim 1 or 2, further comprising volumetric rendering calculations performed synchronously by the rays in the hogel subset as they initially enter the plate.

4. The method of claim 1 or 2, wherein at least one layer has more than one plate.

5. The method of claim 1 or 2, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

6. The method of claim 1 or 2, wherein the volume rendering calculation for the ray in each hogel subset is synchronous when the ray in the hogel subset intersects with the corresponding plate.

7. The method of claim 1 or 2, wherein the method is used in a real-time rendering system.

8. A method for rendering a light field image, comprising the following steps: The 3D scene description, which includes surface data elements and volume data, is divided into multiple layers, each with an associated light field and sampling scheme. Further, at least one of the plurality of layers is divided into a plurality of sub-segments, each sub-segment having an associated light field and sampling scheme, wherein the position of each sub-segment is determined according to the geometry of at least a portion of the object represented in the scene; For each layer and each sub-segment, based on the sampling scheme, a set of pixels including additional pixel information is rendered to generate the sampled light field using the surface data included in each layer and each sub-segment; Use this set of pixels to reconstruct the sampled light field for each layer and sub-segment; Volumetric rendering is performed on the light field associated with each layer using the volume data included in the layer; and The reconstructed sampled light field and the volumetric rendered light field associated with each layer and sub-segment are blended into a single output light field image.

9. The method of claim 8, wherein the sampling scheme includes a binary matrix associated with each layer and a mapping function mapping each layer.

10. The method of claim 8 or 9, wherein each optical field associated with each layer consists of one or more hogel subsets, wherein each hogel subset and its associated layer define a plate as a sub-volume of the layer, and wherein the method further comprises: Synchronize volumetric rendering calculations related to the light rays associated with each layer; Cache the volume data that intersects with each corresponding board body; and When the ray intersects the plate, the volume renders the ray in each hogel subset.

11. The method of claim 10, wherein only a portion of the volume data is cached.

12. The method of claim 10, wherein at least one layer has more than one plate.

13. The method of claim 10, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

14. The method of claim 10, wherein the rays in the hogel subset are synchronized when all volume rendering calculations corresponding to the hogel subset intersect with the corresponding plate.

15. The method of claim 8 or 9, wherein the method is used in a real-time rendering system.

16. A method for rendering light field volumetrics of a scene in a computer system, comprising: The 3D scene, which includes volume data elements, is divided into multiple layers, each with an associated light field and sampling scheme. Each light field consists of one or more hogel subsets, and each hogel subset, layer, and associated light field defines a plate containing the sub-volume of the layer. According to the sampling scheme, the light field associated with each layer is volumetrically rendered using the volume data included in the layer to provide multiple rendered light fields. The sampling scheme includes a binary matrix associated with each layer and a mapping function that maps each layer. and The rendered light field associated with each layer is upsampled and blended into a single output light field image.

17. The method of claim 16, wherein volume data intersecting with the plate is cached.

18. The method of claim 17, wherein only a portion of the volume data intersecting with the plate is cached.

19. The method of any one of claims 16-18, wherein at least one layer has more than one plate.

20. The method of any one of claims 16-18, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

21. The method of any one of claims 16-18, wherein the method is used in a real-time rendering system.

22. A system for performing accelerated volumetric rendering of a scene in a computer system, the system comprising: System memory used to store volume data; The processor device operatively coupled to the system memory is configured to: The 3D scene description, including volume data, is divided into layers, each layer having an associated light field and at least one plate, each plate having an associated hogel subset such that all rays associated with the hogel subset intersect the plate; Reconstruction calculations are performed on the rays intersecting the plate by selectively accessing a set of volume elements from the local storage cache memory; Volume rendering is performed using the volume data included in each layer, which is associated with the light field of each layer, such that volume rendering calculations are performed on the rays in the hogel subset and the rays that intersect the plate to provide the rendered light field. The volumetric light field data that intersects with the corresponding plate is cached, and the volumetric light field data is stored in the local storage cache memory; and The rendered light field associated with each layer is blended into a single output light field image.

23. The system of claim 22, wherein only a portion of the volume data is cached.

24. The system of claim 22 or 23, wherein the system memory further comprises being configured to synchronize ray-related volumetric rendering calculations in the hogel subset.

25. The system of claim 22 or 23, wherein at least one layer has more than one plate.

26. The system of claim 22 or 23, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

27. The system of claim 22 or 23, wherein volume rendering calculations corresponding to the hogel subset are synchronized when a ray associated with the hogel subset intersects the plate.

28. The system of claim 22 or 23, used in a real-time rendering system.

29. A system for performing accelerated volumetric rendering of a scene in a computer system, the system comprising: System memory used to store volume data; The processor device operatively coupled to the system memory is configured to: The 3D scene description, which includes surface data elements and volume data, is divided into multiple layers, each with an associated light field and sampling scheme. Further, at least one of the plurality of layers is divided into a plurality of segments, each segment having an associated light field and sampling scheme, wherein the position of each segment is determined according to the geometry of at least a portion of the object represented in the scene; For each layer and each sub-segment, based on the sampling scheme, a set of pixels including additional pixel information is rendered to generate the sampled light field using the surface data included in each layer and each sub-segment; Use this set of pixels to reconstruct the sampled light field for each layer and sub-segment; Volumetric rendering is performed on the light field associated with each layer using the volume data included in the layer; and The reconstructed sampled light field and the volumetric rendered light field associated with each layer and sub-segment are blended into a single output light field image.

30. The system of claim 29, wherein the sampling scheme includes a binary matrix associated with each layer and a mapping function mapping each layer.

31. The system of claim 29 or 30, wherein each optical field associated with a layer consists of one or more hogel subsets, wherein each hogel subset and its associated layer define a plate body called a sub-volume of the layer, and further comprising: Synchronize volumetric rendering calculations related to lighting; Cache the volume data that intersects with the corresponding board body; and Volumetric rendering calculations are performed on all rays in the hogel subset when they intersect the plate.

32. The system of claim 29 or 30, wherein only a portion of the volume data is cached.

33. The system of claim 29 or 30, wherein at least one layer has more than one plate.

34. The system of claim 31, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

35. The system of claim 34, wherein volume rendering calculations corresponding to each hogel subset are synchronized when a ray associated with a hogel subset intersects the plate.

36. The system of claim 29 or 30, used in a real-time rendering system.

37. A system for performing accelerated volumetric rendering of a scene in a computer system, the system comprising: System memory used to store volume data; The processor device operatively coupled to the system memory is configured to: The 3D description scene, which includes volume data elements, is divided into layers. Each layer has an associated light field and sampling scheme, where each light field consists of one or more hogel subsets and each hogel subset and layer defines a plate containing the sub-volume of the layer. According to the sampling scheme, volumetric rendering is performed on the light field associated with each layer using the volume data included in the layer, the sampling scheme including a binary matrix associated with each layer and a mapping function mapping each layer; and The rendered light field associated with each layer is upsampled and blended into a single output light field image.

38. The system of claim 37, wherein volume data of a subset of hogels intersecting the plate is cached.

39. The system of claim 38, wherein only volume data of a subset of hogels intersecting with the plate is cached.

40. The system of any one of claims 37-39, wherein at least one layer has more than one plate.

41. The system of any one of claims 37-39, wherein the size of the hogel subset is adjusted to utilize the overlap of light samples in the hogels within the subset.

Images