Light field offset rendering

Abstract

A method for light field rendering of 3D scene data. The rendering method includes capturing a light field image at a retraction plane parallel to the display plane, decoding the light field camera to produce a holographic element camera for each holographic element at the retraction plane to produce an integrated image, and applying a pixel remapping technique to pixels of the integrated image for display on the light field display. The 3D scene is captured and remapped to adjust the viewing angle, thereby creating an illusion that the image is taken at the display plane, thereby allowing the captured image to be displayed in the inner and outer frustums of the light field display, thereby creating an immersive experience.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 17 / 663,943, filed on May 18, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to a light field rendering method using a light field offset scheme and a pixel remapping technique. The present disclosure also relates to rendering a light field using image rasterization. Background Art

[0004] Light field displays recreate the experience of viewing the real world through a window by replicating a light field that describes all the light associated with a particular scene. Creating an image for a standard display from a three-dimensional (3D) scene is called rendering or light field rendering. One rendering method, ray tracing, is done by modeling the light in the scene. The ray tracing model can then be used in a variety of rendering algorithms to generate a digital image from the light model. Another rendering method, called rasterization, takes an image described in a vector graphics format or shape and converts it into a raster image, which is a series of pixels, points, or lines that, when displayed together, create an image represented by the shape. Ray tracing is best suited for light field rendering, but many features of content creation tools do not support ray tracing. Therefore, rasterization methods that sacrifice fast rendering times can be used as offline renderers to generate high-quality content for light field displays.

[0005] Existing 3D display technologies can be divided into two categories, binocular stereo and autostereo. Binocular stereo displays use special glasses to facilitate the left and right eyes to view two slightly different images, thus creating depth cues. Autostereo displays allow viewers to see different images with their naked eyes, depending on where they view the display from. This can be likened to a person looking through a window: looking out the left, they see a completely different image than when they look out the right. Traditional 2D light field displays do not provide the proper depth cues for the brain to interpret the images seen on them as objects in the real world, at least because the displayed images are not stereoscopic and both eyes see the same image.

[0006] An automatic stereoscopic display technology, multi-view Figure 3D, approximates the light field associated with the scene. A light field is defined as a function that describes the amount of light that flows in each direction through a point in space without occluding objects, and contains information about all images that can be seen from all possible combinations of viewing positions and angles relative to the display surface. A real light field contains an infinite, continuous number of rays. In practice, to limit the amount of data required to describe the scene, the rendered light field is discretized by selecting a finite set of "views" that are used to estimate the actual light field at any given location on the 3D display.

[0007] In the example of light field rendering, Do et al. (Do, Luat and Sveta Zinger. "Quality improving techniques for free-viewpoint DIBR", Stereoscopic Displays and Applications XXI., Vol. 7524. International Society for Optics and Photonics, 2010) described a rendering algorithm based on deep image warping between two reference views of an existing camera. When experimenting with synthetic data, it was observed that the rendering quality was highly dependent on the complexity of the scene, and the method was performed using compressed video from surrounding cameras. The overall system quality is determined by the rendering quality rather than the encoding, so rendering times are long and require high-level computing power.

[0008] In another example of light field rendering, Li et al. (Li, Hui et al. "Optimized layered method for real-time interactive holographic display based on ray-tracing technique" Optical Engineering 59.10 (2020): 102408) described an optimized layered rendering method for real-time interactive holographic display based on ray tracing technology to overcome the challenges of light field rendering and achieve real-time interaction of three-dimensional scenes. The experiment demonstrated the reconstructed holographic image with real depth cues, and the optically reconstructed image can be interactive in real time, but the ray tracing-based method requires high computing power.

[0009] In another example, U.S. Patent Application Publication No. US20220012860 by Zhang et al. describes a method and apparatus for synthesizing a six-degree-of-freedom view from a sparse red, green, and blue (RGB) depth input. The example apparatus includes at least one memory, instructions in the apparatus, and a processor circuit for executing instructions to reproject a first depth view and a second depth view to a target camera position to obtain a first reprojected view and a second reprojected view, combining the first reprojected view and the second reprojected view into a hybrid view, including RGB depth information lost due to at least one of occlusion or non-occlusion, and generating a six-degree-of-freedom synthetic view of the hybrid view data, the synthetic view including the lost RGB depth information. The method requires at least two views and reprojects each view separately, and then mixes the views to generate a synthetic view.

[0010] Based on the directionality of pixels in light field displays (LFDs), light field rendering using ray tracing is an organic approach and has been thoroughly studied. It is the fastest method described for light field rendering and can be used for real-time rendering, enabling interactive light field experiences such as video games. However, many existing tools used by content creators do not support ray tracing to the degree required for high-quality light field rendering. It is desirable for designers to quickly transition to generating content for LFDs, as integration with existing architectures is critical to the development and adoption of new technologies. Therefore, in applications where quality takes precedence over speed, such as offline rendering, there remains a need for a high-performance light field rendering method with rasterization and reduced rendering times. A rendering method using rasterization is needed because commercially available real-time rendering software is designed to render images at 30 frames per second or faster, including but not limited to Unreal Engine 3. Maya, and Unity, which can be configured to render light field images as well as traditional 2D images.

[0011] The purpose of providing this background information is to make available known information believed by the applicant to be relevant to the present invention. It is not necessarily intended, nor should it be construed, that any of the foregoing information constitutes prior art against the present invention. Summary of the invention

[0012] One object of the present invention is to provide a light field rendering method, which uses a light field offset scheme and a pixel remapping technique to render a 3D light field. Another object of the present invention is to provide a method for light field mapping and rendering using image rasterization.

[0013] According to one aspect, a computer-implemented light field rendering method is provided, comprising: defining a display plane for a light field, the light field comprising an inner frustum volume defined by the display plane and a far clipping plane and an outer frustum volume defined by the display plane and a near clipping plane; defining a retraction plane parallel to the display plane and having an integral offset distance from the display plane, the retraction plane comprising a plurality of light field cameras spaced apart by sample gaps; capturing a view of the 3D scene at each of the plurality of light field cameras as a source image; decoding each source image to generate a plurality of holographic meta-cameras on the retraction plane, each holographic meta-camera providing an elemental image; generating an integral image comprising a plurality of pixels from the elemental image at the retraction plane; and performing a pixel remapping technique on individual pixels in the integral image to create a rendered light field image on the display plane.

[0014] In one embodiment of the method, the portion of the 3D scene captured at the retraction plane comprises image information from the inner frustum volume and the outer frustum volume.

[0015] In another embodiment of the method, the captured 3D scene includes all image information within the outer frustum volume.

[0016] In another embodiment of the method, the integrated offset distance is calculated based on the focal length, the directional resolution of the light field camera and the offset integer N.

[0017] In another embodiment of the method, the offset integer N≥1.

[0018] In another embodiment of the method, the retraction plane is located at the near clipping plane.

[0019] In another embodiment, the method further comprises: displaying the rendered light field image on a light field display.

[0020] In another embodiment of the method, the retraction plane surface area is greater than the display plane surface area.

[0021] In another embodiment of the method, the optical properties of the light field camera are orientation, lens spacing, directional resolution, and field of view.

[0022] In another embodiment, the method further comprises generating a plurality of integral images at a plurality of retraction planes.

[0023] In another embodiment, the method further comprises: synthesizing the plurality of integral images to create a synthesized rendered light field image at the display plane.

[0024] In another embodiment of the method, the synthesis incorporates transparency data.

[0025] In another embodiment of the method, each light field camera is one of a digital single-lens reflex camera (DSLR) camera, a pinhole camera, a plenoptic camera, a compact camera, and a mirrorless camera.

[0026] In another embodiment of the method, each light field camera is a computer generated camera.

[0027] In another embodiment of the method, the pixel remapping technique causes the pixel to have its holographic element index (H x ,H y ) changes from the retraction plane to the display plane.

[0028] In another embodiment of the method, the retraction plane is located outside the outer frustoconical volume.

[0029] In another aspect, a computer-implemented method for displaying a light field image is provided, comprising: capturing, using a light field camera, a first light field at a retraction plane relative to a light field display plane, the first light field comprising an array of retraction plane holographic elements, each holographic element having a plurality of pixels; assigning a holographic element index (H) to each pixel in each retraction plane holographic element by selecting a single pixel from each retraction plane holographic element by applying a pixel remapping technique x , H y ) and pixel index (P x , P y ) to indicate its position in the light field display; using a synthesis function to load each pixel from the light field into the retraction plane (LF r ) and stores each pixel in the display plane (LF d ) at a light field at a location; and generating a light field image comprising remapped pixels at the display plane.

[0030] In another embodiment of the method, one pixel from said retraction plane produces one pixel at said display plane.

[0031] In another embodiment of the method, a pixel remapping technique is applied to change the hologram index (H) of each pixel in each retracted plane hologram. x , H y ), and the pixel index (P x , P y ) remains unchanged.

[0032] In another embodiment of the method, the pixel remapping technique is a function of a directional resolution and an offset parameter N of the light field display.

[0033] In another embodiment of the method, the pixel remapping technique is based on the equation

[0034] In another embodiment of the method, the offset integer N≧1.

[0035] In another embodiment of the method, the retraction plane is composed of a sufficient number of holographic elements to provide a number of pixels on the display plane to achieve a required directional resolution for a light field display.

[0036] In another embodiment of the method, the size of the display plane is defined by the directional resolution and the spatial resolution of the light field display.

[0037] In another embodiment of the method, the light field camera is a reflective (DSLR) camera, a pinhole camera, a plenoptic camera, a compact camera, or a mirrorless camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] These and other features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0039] Figure 1A An illustration of a rasterization method that uses a frustum projection to flatten a 3D scene into a 2D image.

[0040] Figure 1B is an illustration of a front view of a flattened 2D image.

[0041] Figure 2 is an illustration of an example ray tracing method for solving the visibility problem.

[0042] Figure 3 is an illustration of an example digital representation of a ground truth image.

[0043] Figure 4 is an illustration of an example method for rendering a light field showing an inner frustum and an outer frustum.

[0044] Figure 5 is an illustration of a frustum view of a 4x4 holographic metaspatial resolution light field.

[0045] Fig. 6A Shown is the holographic meta-camera translation in the 3D view.

[0046] Figure 6B Shown is the holographic meta-camera translation in the grid view.

[0047] Fig. 7A The light field camera at the display plane and the forward vector of the light field camera are shown.

[0048] Figure 7B A light field camera is shown retracted an offset distance from the display plane.

[0049] Figure 8 The relationship between pixels in the light field at the display plane and pixels in the light field at an offset distance is shown.

[0050] Fig. 9 An array of light field cameras is shown at a display plane, showing captured volumes and uncaptured volumes.

[0051] Fig.10 A light field capture method is shown that offsets the light field camera array to the retraction plane, allowing the capture of the inner frustum volume.

[0052] Fig.11 An embodiment of the disclosed method is shown with an offset parameter of N=1.

[0053] Fig.12 An embodiment of the disclosed method is shown with an offset parameter of N=2.

[0054] Fig.13 Shown are indexed holograms and pixels within a light field display.

[0055] Fig.14 It is a flowchart of the light field rendering method.

[0056] Fig.15 is a flow chart of a light field rendering method including a pixel remapping technique.

[0057] Fig.16 The figure is a flow chart of a light field rendering method according to the present disclosure. DETAILED DESCRIPTION

[0058] definition

[0059] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0060] When used herein in conjunction with the term "comprising," the word "a" or "an" may mean "one," but it is also consistent with the meaning of "one or more," "at least one," and "one or more."

[0061] As used herein, the terms "comprising," "having," "including," and "containing," and grammatical variations thereof, are inclusive or open-ended and do not exclude additional, unlisted elements and / or method steps. When used herein in conjunction with a composition, device, article, system, use, or method, the term "consisting essentially of" indicates that additional elements and / or method steps may be present, but that such additions do not materially affect the manner in which the composition, device, article, system, method, or use functions. Compositions, devices, articles, systems, uses, or methods described herein that include certain elements and / or steps may also consist essentially of those elements and / or steps in certain embodiments and consist of those elements and / or steps in other embodiments, whether or not such embodiments are specifically mentioned.

[0062] As used herein, the term "about" refers to a variation of approximately + / - 10% from a given value. It should be understood that such variation is always included in any given value provided herein, whether or not specifically mentioned.

[0063] Unless otherwise indicated herein, the recitation of ranges herein is intended to convey the range and individual values ​​within the range to the same place value as the number used to express the range.

[0064] The use of any examples or exemplary language, such as "such as," "exemplary embodiment," "illustrative embodiment," and "for example," is intended to illustrate or represent aspects, embodiments, variations, elements or features related to the present invention and is not intended to limit the scope of the present invention.

[0065] As used herein, the terms "connected" and "connected" refer to any direct or indirect physical association between elements or features of the present disclosure. Thus, these terms can be understood to mean elements or features that are partially or completely contained within, attached, coupled, arranged on, connected together, in communication, in operational association, etc., even if there are other elements or features between the connected elements or features.

[0066] As used herein, the term "pixel" refers to the light source and light emitting mechanism used to create a display.

[0067] As used herein, the term "light field" refers, at a basic level, to a function that describes the amount of light that flows in every direction through a point in space without obstruction. A light field contains information about all images that can be seen from all possible viewing positions and angle combinations, with light flowing from all directions through a point in space without obstructing objects, suitable for a particular display format. Thus, a light field represents radiance as a function of the position and direction of light in free space. Light fields can be generated synthetically by various rendering processes, or they can be captured from a light field camera or array of light field cameras. In a broad sense, the term "light field" can be described as an array or subset of holographic elements.

[0068] As used herein, the term "hologram" is a holographic element that is a cluster of traditional pixels with directional control. An array of holograms can generate a light field. Pixels describe the spatial resolution of a two-dimensional display, while holograms describe the spatial resolution of a three-dimensional display.

[0069] As used herein, the term "light field display" is a device that reconstructs a light field from a finite number of light field radiation samples input to the device. The radiation samples represent the color components red, green, and blue (RGB). For reconstruction in a light field display, the light field can also be understood as a mapping from a four-dimensional space to a single RGB color. The four dimensions include the vertical and horizontal dimensions of the display (x, y) and two dimensions that describe the directional components of the light field (u, v). The light field is defined as the following function:

[0070] LF:(x,y,u,v)→(r,g,b)

[0071] For a fixed x f ,y f , LF(x f ,y f ,u,v) represents a two-dimensional (2D) image, called an "element image". The element image is obtained from a fixed x f ,y f The directional image of the light field starting at position . When multiple element images are connected side by side, the resulting image is called an "integral image". The integral image can be understood as the entire light field required for a light field display.

[0072] As used herein, the term "display plane" refers to the set of points and directions defined by the physical spacing of a flat display and its individual light field holographic elements, as in a conventional 3D display. In an abstract mathematical sense, a light field can be defined and represented on any geometric surface, and may not necessarily correspond to a physical display surface with actual physical energy emission capabilities. The inner and outer frustum volumes are the light field regions extending from above and below, behind and in front of the display plane, respectively. The inner and outer frustum volumes can have different numbers of layers, different volumes, different depths, and can be rendered using different rendering techniques.

[0073] As used herein, the term "voxel" refers to a single sample or data point on a regularly spaced three-dimensional grid consisting of a single data. A voxel is a single volume element corresponding to a location in a three-dimensional data space and has one or more data values ​​associated with it.

[0074] As used herein, the term "scene description" refers to a geometric description of a three-dimensional scene that may be a potential source from which a light field image or light field video may be rendered. The geometric description may be represented by points, lines, quadrilaterals, textures, parametric surfaces, and polygons, but is not limited thereto.

[0075] As used herein, the term "extra pixel information" refers to information included in a scene description. The extra pixel information includes, but is not limited to, color, depth, surface coordinates, normals, material values, transparency values, and other possible scene information.

[0076] As used herein, the term "source image" refers to a light field image captured at a single location in a camera array.

[0077] As used herein, the term "element image" refers to a fixed position x f ,y f The two-dimensional (2D) image LF(x f ,y f ,u,v). The element image is obtained from a fixed x f ,y f Orientation image of the light field at the location.

[0078] As used herein, the term "integral image" refers to multiple element images connected side by side, so the resulting image is called an "integral image". The integral image can be understood as the entire light field required for a light field display. A rendered light field image is a rendering or mapping of the integral image that can be displayed on a display plane.

[0079] It is contemplated that any embodiment of the compositions, apparatuses, articles, methods and uses disclosed herein may be implemented by one skilled in the art either as such or by making such modifications or equivalents without departing from the scope of the invention.

[0080] Various features of the present invention will become apparent from the following detailed description and the illustrations in the accompanying drawings. The design factors, construction, and use of the light field volume rendering technology disclosed herein are described with reference to various examples representing implementations, which are not intended to limit the scope of the invention described and claimed herein. Those skilled in the art to which the present invention relates will recognize that there may be other variations, examples, and implementations of the present invention not disclosed herein, which can be practiced according to the teachings of the present disclosure without departing from the scope of the present invention.

[0081] A light field rendering method is described herein that uses a light field offset scheme and a pixel remapping technique to capture inner frustum light field data and outer frustum light field data from a 3D scene. In the method, a camera array is located on a retraction plane that is an integer offset distance from the inner frustum volume from the display plane, inside or outside the outer frustum volume, to capture the scene. Capturing the scene using the camera array at the retraction plane enables the capture of the entire inner frustum volume of the light field and part or all of the outer frustum volume. The resulting light field is moved back to the display plane using a pixel remapping function to display the light field as if the camera array was originally located at the display plane. Therefore, the light field captured by the camera array contains more of the 3D scene than the 3D scene captured when the camera array is located at the display plane. Specifically, the method reduces or eliminates blind volumes in the inner frustum and the outer frustum, thereby enabling a more complete capture and rendering of the 3D scene. The captured 3D scene can then be presented on a 3D light field display.

[0082] This technology can be implemented in game engines (such as Unreal ) and is used to capture features of light field image data (such as advanced lighting and physically based rendering materials). ), the light field rendering method described so far also reduces light field rendering time and processing requirements by using rasterization techniques rather than relatively computationally intensive ray tracing techniques. Overall, capturing a dual-frustum light field through rasterization using a light field offset scheme is a viable technique that can be used for 3D scene rendering, particularly offline rendering. The placement and number of light field cameras in the retracted camera array can also provide a sufficiently large area to capture the entire expected light field volume expected frustum, thereby significantly reducing the size of the 3D source data to be rendered, while also providing a more complete 3D scene rendering.

[0083] The present disclosure proposes and details the concept of light field rendering using a retraction plane. The retraction plane function currently described defines the geometry of the retraction plane, and the sample gap between cameras in a camera array at the retraction plane specifies the light field camera spacing in the camera array at the retraction plane. The sample gap between light field cameras can also be understood as the distance between light field cameras located at the retraction plane, where the intermediate space between the cameras is occupied by a virtual holographic meta-camera created using pixel remapping to generate an integral image of a 3D scene at the retraction plane.

[0084] Rendering is the process of converting a 3D scene into a 2D image. With the exception of some artistic rendering techniques, the goal of rendering is generally to create realistic images that are indistinguishable from reality, a common theme in the field of computer graphics. Before a renderer (the program that does the rendering) can convert a 3D scene into an image, the geometry must be defined in a format that the computer can interpret. Geometry is represented using a combination of points in 3D space, each defined as a vector with an x ​​component, a y component, and a z component. Most commonly, geometry is represented using triangles, which are the simplest way to define a plane and are used as the building blocks of all polygons. To generate a 2D image from a 3D scene for use in a light field display (where a 2D image displays a 3D scene), the geometry in the scene must be "flattened" onto a canvas on a display plane that represents the screen. There are two common methods used to create 2D images from geometry: rasterization and ray tracing. Once a 2D image that accurately reflects how the eye sees a 3D scene has been produced, the appearance of the objects in the scene must also be recreated. The appearance of objects in the scene (such as color, reflectivity, and texture) depends on how the materials that make up the object interact with light. These interactions are modeled using mathematical models based on the laws of physics.

[0085] By using a retraction plane to capture a 3D scene at an offset distance from a display plane, an inner frustum and an outer frustum of a light field can be captured. The present method results in less data required to render the 3D scene as a 2D image, resulting in faster rendering, less data transmitted to the display surface for image display, and less data decoded overall. The light field is composed of an array of holographic elements, each of which can be, for example, 0.1-25 mm in diameter. To capture an image of each holographic element in a light field display, 100 to 100,000 images need to be captured from a physical camera. Using an array of light field cameras, where the light field cameras are located at a sample gap of a retraction plane to capture a 3D scene at an offset distance from the display plane, both full light field capture and efficient light field rendering for 3D display can be achieved.

[0086] Figure 1Ais an illustration of a rasterization method for flattening a 3D scene into a 2D image using a frustum projection. As shown, a cube 12 representing a 3D scene 22 is flattened using a frustum projection by tracing a number of rays 20 between each vertex 10 of an object in the 3D scene 22 and the eye of a viewer or a viewer at a viewing position 16, and then finding the intersection of those rays 20 with a display plane 18 representing a display screen. The intersection points on the display plane can then be connected to create a 2D image 14. If the cube 12 were not transparent, the output would not accurately reflect how the eye at the viewing position 16 would interpret the 3D scene 22, because the front face of the cube 12 would hide all other faces and edges. Calculating which parts of the 3D scene 22 are visible, the so-called "visibility problem," is one of the fundamental concepts of light field rendering. One of the most common methods of solving the visibility problem using rasterization is the Z-buffer algorithm, which is a depth buffer used to render non-transparent objects in a 3D scene to correct for polygons that occlude other polygons. The Z-buffer algorithm, also known as the depth buffer method, is an image space method based on pixels to be drawn in 2D, which compares the surface depth of each pixel location on the projection plane, where the z-axis is represented as depth. When multiple points or vertices 10 in the 3D scene geometry are projected to the same pixel in the display plane, they are added to a list called the depth buffer. The depth buffer is then sorted in increasing order of distance from the display plane 18. The first point in the depth buffer is closest to the display plane 18 and will be visible through the corresponding pixel of the light field display or screen at the display plane 18.

[0087] Figure 1B A front view of a flat 2D image 14 created by the intersection of each ray traced between the object's vertex and the viewer's eye at the viewing position of the display surface or screen is shown. Figure 1A In the projection embodiment shown in , each point or vertex 10 of each object in the 3D scene 22 is projected onto a flat canvas representing the screen at the display plane 18. The process starts with a known vertex 10 in the 3D scene 22 and finds its position when projected onto the screen at the display plane 18. Performing this series of steps in reverse is the basis of ray tracing.

[0088] Figure 2is an illustration of an example ray tracing method for solving the visibility problem. As shown, each pixel on the display plane 18 is first converted to a point 24 on the display plane 18. Then, the ray 20 is traced from the viewer's eye at the viewing position 16, through the point 24 and into the 3D scene 22, as shown. The point where the ray 20 intersects the scene geometry 26 will be the point visible through the pixel on the display plane 18. Like most algorithms designed to simulate the real world, the concept of ray tracing is easy to understand, but it is resource and computationally intensive. Specifically, ray tracing requires calculating the intersection points 24 of many rays 20 with different types of geometry, and a lot of research has been done to optimize this calculation. Rasterization remains a faster solution to solve the visibility problem. In other words, to solve the visibility problem, rasterization actually "projects" the geometry onto the display plane 18 using perspective projection, from a 3D representation to a 2D representation of the geometry.

[0089] Typical rendering generates realistic 2D images from a 3D scene to be viewed on a conventional 2D display. Similarly, at least one realistic 3D image is generated from a 3D scene to create 3D content that can be viewed on a 3D light field display (LFD). Light field rendering generates a light field from a 3D scene that can be loaded and viewed on an LFD. Light field rendering uses the same rendering techniques as conventional displays, but is applied differently to support multiple 3D views of a 3D scene. Based on the directionality of pixels in an LFD, light field rendering using ray tracing may be an organic approach and has been thoroughly studied for real-time rendering to enable interactive light field experiences like video games. However, many existing software tools used by content creators do not support ray tracing and are insufficient for high-quality light field rendering. Enabling designers to quickly transition to generating content for LFDs is desirable, as integration with existing architectures is critical to the development and adoption of new technologies. Therefore, in applications such as offline renderers where quality is prioritized over speed, light field rendering using rasterization may be a viable and superior solution if ray tracing is not fully supported. One of the most widely used creation tools is Unreal Unreal Engine 4 (UE4), which provides a platform for 3D creation in many technical fields, such as industrial and architectural visualization, games, visual effects, and filmmaking. UE4 supports ray tracing for visual effects such as reflections and refractions, but the main scene rendering is performed using rasterization. UE4's rendering pipeline can be modified to use ray tracing, however, this breaks many of the advanced features that designers use to create high-quality content on the UE4 platform. To obtain real-time light field rendering, quality can be sacrificed in exchange for speed. For offline rendering that prioritizes quality over speed, a non-intrusive rasterization approach that has a high degree of interoperability with engine features may be preferable to a faster ray tracing approach that provides minimal support for advanced engine features. Therefore, this paper describes a new method for rendering dual-frustum light fields using frustum projection as an alternative to the improved engine for offline export of light fields.

[0090] Figure 3 A digital representation of a ground truth, uncompressed light field is shown. A digital image for a 2D display is simply a grid of pixels, where the height and width of the grid are called the image resolution. A similar notation is used for LFDs. The image displayed on the LFD (hereinafter referred to as the light field) is represented by a holographic element grid 40 comprising a plurality of holographic elements 30, where the holographic element width 38 (SR x ) and holographic element height 36 (SR y ) determines the spatial resolution of the display. Each hologram 30 consists of a grid of directional pixels 28 that generate or emit light that conveys information about the color, intensity and direction of the light at that point on the hologram grid 40. The individual hologram width 32 and hologram height 34 of each hologram 30 in the hologram grid 40 determines the directional resolution of the LFD, and the image formed by each hologram is called an "element image." Therefore, each individual hologram 30 represents a single element image. In the example shown, the spatial resolution of this LFD will be 8x8 (holograms in the hologram grid 40) and the directional resolution will be 4x4 (directional pixels in each hologram 30).

[0091] Figure 4is an illustration of an example method for rendering a light field. The illustrated method requires capturing a volume of a scene that can be viewed from a plane representing a screen (referred to herein as display plane 18). The volume of the scene is defined by two "frustums" or truncated pyramids, as shown in the figure, an outer frustum volume 42 close to or near the viewer or human eye at the viewing position 16, and an inner frustum volume 44 located far from or away from the viewer or viewing position 16. Recall that perspective projection is used to "flatten" a 3D scene into a 2D light field that can be displayed on the display plane 18. This process is also called frustum projection because the frustum defines the volume of the 3D scene that the human eye sees at the viewing position 16. The inner frustum volume 44 extends from the display plane 18 into the scene, while the outer frustum volume 42 extends out from the scene. Objects in the inner frustum volume 44 will appear to be "inside" the display at the display plane 18 to the eye at the viewing position 16, while objects in the outer frustum volume 42 will appear to protrude outside the display at the display plane 18. The distance that the frustum extends in and out relative to the display plane 18 is defined by the far clipping plane 48 and the near clipping plane 46, respectively, and has practical limits. As the eye of the viewer at the viewing position 16 moves in a horizontal direction parallel to the display plane 18, the eye of the viewer at the viewing position 16 will perceive the movement of the light field, thereby providing an accurate perspective view of the light field from the new position. The shift in perspective, referred to herein as motion parallax, demonstrates the natural relationship between elements in the light field and how that relationship changes as the viewing angle changes. Motion parallax simulates a physical world scene, presenting the light field as if the viewer at the viewing position 16 were present in the physical world scene. Allowing the eyes of the viewer at the viewing position 16 to perceive the light field naturally, as if they existed in the physical world scene, provides many applications for 3D light field displays that are not possible with two-dimensional displays.

[0092] Lightfield displays discretize light to approximate a light field, and the number of rays used, or directional resolution, and the density of how the rays are packed affect the depth of the display. The density of the rays can be measured using the hologram pitch, which is the distance between the centers of two adjacent holograms. The maximum depth at which an object appears blurry can be calculated using the directional resolution (NxN), hologram pitch (HP), and field of view (F), as shown below:

[0093]

[0094] Rendering a light field requires capturing each source image from individual views and stitching them together to form the entire light field. To do this, first, a packing algorithm takes as input the LFD configuration and a set of rendering parameters, and then generates a data structure containing all the information needed to render each view, which must then be iterated over and captured for each view. The light field resolution describes the total number of pixels in the light field and is calculated as the product of the directional resolution and the spatial resolution using the following formula.

[0095] Light field resolution = Directional.X*Directional.Y*Spatial.X*Spatial.Y

[0096] in:

[0097] Directional.X is the directional resolution along the x-axis of the light field

[0098] Directional.Y is the directional resolution along the y-axis of the light field

[0099] Spatial.X is the spatial resolution along the x-axis of the light field

[0100] Spatial.Y is the spatial resolution along the y-axis of the light field

[0101] If the light field resolution exceeds the maximum supported frame size, the light field can be divided into subframes, where each subframe contains a portion of the total view. All views in the subframes are rendered and then exported to storage. Finally, the subframes are composited using the compositing function to generate the complete light field.

[0102] Figure 5 is an illustration of a frustum view of a 4x4 spatial resolution light field. For demonstration purposes, a front view of a holographic metagrid 50A, a side view of a holographic metagrid 50B, a top view of a holographic metagrid 50C, and an isometric view of a holographic metagrid 50D are shown, but it should be noted that each view is preferably captured consecutively rather than in parallel. The combination of each frustum view forms a light field frustum. The general method of rendering a light field using a frustum projection on a display plane does not allow for capturing objects in the outer frustum. Specifically, when the camera is placed at the display plane, the camera cannot capture any light field information behind the camera in the outer frustum volume. Traditionally, capturing light field data in both the inner and outer frustum volumes is a tedious task. In accordance with the present disclosure, the method describes a light field offset scheme, referred to herein as light field offset, for capturing a light field at a retraction plane and then moving the light field back to the display plane after all frustum views have been captured. The method can be easily implemented by moving the image capture to a position away from the inner frustum volume by a certain offset distance from the display plane. This method enables 3D light field scene capture of the outer frustum volume and more complete 3D light field scene capture of the inner frustum volume. Furthermore, performing this light field offset using rendering does not require engine modifications.

[0103] Fig. 6A shows the holographic meta-camera translation in the 3D view, Figure 6B Figure 2 shows the holographic meta-camera translation in the grid view. Fig. 6A and Figure 6BAs shown, the packing algorithm defines a light field camera 52 at the center of the display plane 18 to assist in rendering calculations. For illustrative purposes, only a single light field camera 52 is shown, but it is understood that multiple light field cameras in the camera array can also be used to capture the 3D scene in more detail at multiple locations. For each light field camera view, the position of the viewport of the associated holographic meta camera is defined by an x-direction offset 56 and a y-direction offset 58 from the light field camera 52. The light field camera 52 can be a physical camera or a computer-generated camera, and the holographic meta camera 54 can be understood as a virtual representation of the camera view, with an x, y offset relative to the light field camera 52 on the camera plane. The view information also contains the position in the subframe where the rendered element image is written to be displayed on the display plane of the LFD. The holographic meta camera 54 is defined and configured to represent the viewport of the element image at the holographic meta camera position. Using these offsets, the positions of multiple holographic meta cameras 54 shifted in the x and y directions around the display plane 18 can be achieved, so that each view in the subframe is rendered onto the subframe texture at the x, y position specified in the rendering information. Fig. 6A An isometric view of the x-direction offset 56 and the y-direction offset 58 (also referred to as holographic metacamera translation) from the light field camera 52 to each holographic metacamera 54 at the display plane 18 is shown. Figure 6B An elevation view of the x-direction offset 56 and y-direction offset 58 from the light field camera 52 to the example holographic metacamera 54 at the display plane 18 of a light field having a 4x4 spatial resolution is shown.

[0104] Fig. 7A A light field camera 52 and a forward vector 60 of the light field camera are shown located at the display plane 18. When the light field camera 52 is located at the display plane 18, the camera receives light field information only in the inner frustum volume 44.

[0105] Figure 7B The light field camera 52 is shown retracted an offset distance from the display plane 18 and located at the retraction plane 64. The light field camera 52 can be retracted to the retraction plane 64 at a distance or offset distance 62 from the display plane 18 to capture objects in the outer frustum volume 42 instead of capturing the scene from the display plane 18. It is worth noting that the outer frustum volume 42 is shown as a block of space for illustration purposes only. To apply this shift, the light field camera 52 is translated backward along the trajectory of its forward vector 60. Fig. 6AIn contrast to the camera translation method described in / 6B, the light field camera 52 is retracted from the display plane 18 to the retraction plane 64 before defining and configuring one or more holographic meta-cameras 54. The holographic meta-camera 54 is generated by applying a decoding scheme to the light field camera 52, wherein the decoding scheme decodes a source image at the light field camera and offsets the decoded source image containing the elemental image by translating the light field camera 52 to the holographic meta-camera 54 to create a virtual or synthetic image at the location of the holographic meta-camera 54. When the x-offset and y-offset from the light field camera 52 are applied to the holographic meta-camera 54, as shown in FIG. Fig. 6A 6B, the holographic meta-camera 54 is translated on a retraction plane 64, which is located behind the display plane, in which case the retraction plane 64 is also located at the near clipping plane 46. In other words, for this embodiment, the near clipping plane 46 is the retraction plane 64, but it should be noted that this is not necessarily the case. The retraction plane 64 can also be located anywhere along the trajectory of the forward vector 60 within the outer frustum volume 42 of the light field display, at an integer distance from the display plane 18.

[0106] After the light field camera 52 captures the frustum view from the retraction plane 64 and generates the light field, a pixel remapping function is used to move the light field from the retraction plane 64 back to the display plane 18. For all integer values ​​of the directional resolution NxN and the offset factor α (alpha), the offset distance D 62 can be calculated as:

[0107] D = α * N * focal length

[0108] The same pixel remapping can be performed for each light field camera 52 and each holographic element camera 54 to create an integrated light field image at the display plane 18. As can be seen, each pixel in the display plane can simply be offset by an integer number of holographic elements to move the light field toward or away from the camera.

[0109] It is worth noting that the disclosed rendering method can be used for computer-generated light field images. In this embodiment, the computer-generated light field image includes one or more element images, thereby eliminating the need to decode the source image.

[0110] Figure 8 18 and the retraction plane 64 at an offset distance 62 (D). For simplicity, the diagram is two-dimensional (along the xz plane) and focuses on a single holographic element 30 at the display plane 18, but it is important to note that the relationship is the same in the zy plane and for each holographic element. The relationship is based on the pixel index (P o), which is the index of the pixel within the hologram 30. Box 66 shows a view of the hologram at the display plane 18, while box 68 shows a view of the hologram at the retraction plane at an offset distance 62 (D). Angle theta, θ is defined as FOV / 2 (F / 2), and angle phi, Φ is the angle between the light ray corresponding to any pixel 70 in the hologram 30 and the centerline 72 of the hologram 30 that is perpendicular to the display plane 18. As shown, F / 2 (angle theta, θ) in the outer frustum volume represented by box 68 combined with F / 2 (angle theta, θ) in the inner frustum within box 66 equals the full field of view of the light field display. Light rays represented by one pixel 70 are projected from the display plane 18 back to the retraction plane 64, which is parallel to the display plane 18 and retracted by the offset distance 62. Using similar triangular geometry, tan functions, and formulas relating lens pitch to focal length, it can be seen that each pixel 70 is offset by an integer multiple of the lens pitch according to the pixel index when the offset distance is defined as: D = α * N * focal length. The lens pitch is the distance between adjacent holograms, so an offset of N times the lens pitch will simply offset the pixel by N holograms. The direction of the offset depends on whether the ray is in the front or back half of the hologram θ30.

[0111] The offset distance D 62 is a function of the focal length of the display and the integer N, given by the following formula:

[0112]

[0113] The lens spacing is defined as:

[0114] Solving equation 3 for D yields: D = LP × d

[0115] Substituting d into Equation 2 yields:

[0116] The result is: D = FL × N

[0117] Pixel indexing is a well-known step in image processing that describes how to index an image to identify all the pixels in the image. The notation (P x ,P y ) is how computer software identifies each pixel. Once an image is indexed, the computer software can manipulate the image information to change the image. In one example, the computer software can render, transmit, store, change color, change transparency, etc. The indexing method for 2D images can be applied to light field images by providing indices for holographic elements and pixels. Symbol LF[H x ,H y ,P x ,P y ] is the 3D identifier of each holographic element and pixel in the light field image.

[0118] Dual frustum light field rendering using frustum projection and light field offset has been implemented as a UE4 plugin. The light field frustum can be placed in the scene and configured as needed before rendering. Once the render is triggered, the technique described in this article is used to capture the light field. After capturing the light field from a retracted position using frustum projection, the entire light field is saved in memory. Then, a second light field buffer is created to perform the remapping. This results in a very large memory overhead that can be solved using a "scanning" method, which applies the remapping on the fly, rather than after the image is fully captured.

[0119] Based on the relationship between the pixels in the light field at the display plane 18 and the pixels in the light field at the retraction plane 64, a simple mapping can be applied to the captured light field to move it toward the camera. The remapping algorithm iterates over each pixel in each holographic element in the light field image buffer and remaps the pixels according to their pixel index P. o Each pixel is offset by "N" holograms. To offset a pixel by one full hologram, the spatial index is increased by one. After calculating the offset, the pixel is stored at the target location determined in the remap buffer. After remapping all pixels, the remapped light field is exported.

[0120] One of the most important engine features that the disclosed rendering techniques must support is lighting. Game engines, such as UE4, have three general types of lighting: static lighting, dynamic lighting, and fixed lighting. Each type of lighting is used by designers in different scenes, and all types should be fully supported. Physically based rendering (PBR) uses procedures to reproduce the interaction of materials (such as metals, glass, and mirrors) with light. UE4 makes extensive use of PBR materials to create realistic scenes, and the disclosed rendering method correctly supports PBR. In order to test this support using the currently described methods and systems, scenes with various PBR materials were rendered, and all tested PBR materials were successfully captured. PBR material tests include: roughness, metal, specular, and translucency.

[0121] To further advance the disclosed rendering method, a light field camera can be decoded into a holographic element camera array using multi-view depth image based rendering. This technique can be described as rendering a synthetic image using multiple views. 3D image warping can be further used to synthesize virtual views. Depth image based rendering is a decoding method that converts an image with image information (including color and depth) into index-based geometry and then performs known 3D rendering methods. Specifically, depth image based rendering uses a 3D scene and its associated depth information, as well as an index for each pixel in the scene. The depth information provides the depth for each pixel, and the index provides the position of the pixel in 3D space. Depth image based rendering completes the rendering operation by projecting these points onto an image plane (which can be a retraction plane or a display plane), resulting in a holographic element camera array with the same optical properties as the hologram and a geometrically correct element image for each holographic element camera. After decoding, each hologram has a holographic element camera with an element image, and the holographic element camera array together constitutes the integral image to be emitted by the holographic element array. The integral image is a light field image and can now be displayed on a light field display.

[0122] Reprojecting a color plus depth image is a decoding method that uses a light field camera and its forward vector direction. Using the depth information and index of a pixel, specifically the x index and y index, the coordinates (x, y, z) of the pixel in 3D space can be calculated. The origin and forward direction of the hologram can be used to convert this pixel in 3D space into the depth, x index, and y index of a single hologram and its associated element image. Using the decoded x index and y index, the pixel can be written to an element image on an image plane, including but not limited to a retraction plane or a display plane. The reprojecting color plus depth image decoding method is repeated until all holograms are filled with pixels written to each hologram element image. In general, the hologram camera array constitutes an integral image. After decoding the camera array on the retraction plane, the integral image on the display plane can be constructed by reindexing the decoded integral image on the retraction plane. The integral image is a light field, which can then be displayed on a light field display.

[0123] As methods for decoding light field images advance, fewer light field cameras are needed to capture a scene that is decoded to generate a cohesive light field. One advantage of using fewer light field cameras is that less data needs to be rendered, and the data is transmitted at a faster rendering rate, resulting in a higher frames per second (fps) rate. Therefore, it is highly desirable to limit the number of light field cameras required to capture a 3D description of a scene. In known light field image capture, light field cameras are aligned on a display plane. A light field camera can be a physical camera, such as a pinhole camera, a digital single lens and reflex camera (DSLR), a plenoptic camera, a compact camera, a mirrorless camera, or a computer-generated camera. Each light field camera captures a view from a specific point in space and can typically only capture the volume in front of the light field.

[0124] Fig. 9 A simplified diagram is shown depicting a light field capture technique wherein an array of light field cameras is located at a display plane, showing the volume of the light field captured by the light field cameras and the volume not captured, referred to herein as a "blind volume" or "blind volume region". As shown, an array of light field cameras 52a, 52b, 52c, 52d, 52e is located at the display plane 18, and the light field cameras 52a, 52b, 52c, 52d, 52e capture frustum volumes 74a, 74b, 74c, 74d, 74e, respectively, within the inner frustum volume 44 of the light field. Although a row of light field cameras 52a, 52b, 52c, 52d, 52e is shown, it will be appreciated that a plurality of light field cameras arranged in an array along an x, y plane are typically used to capture the light field. Only a two-dimensional representation is shown here to simplify the illustration. It should be noted that the size and shape of the frustum volume captured by each camera depends on the type of light field camera used to capture the light field source image, including its lens and optical properties. As shown, known light field image capture techniques do not capture images behind the display plane 18 in the frustum volume 42 outside the light field.

[0125] Physical cameras, when used as light field cameras, are also limited by the size and number required to generate a cohesive light field image. A light field consists of an array of holographic elements, each of which can be, for example, 0.1-25mm in diameter. To capture an image of each holographic element in a light field display, a range of 100 to 100,000 images need to be captured from a physical camera. A scene can be controlled using multiple physical light field cameras to capture this many images, but the controlled scene limits the type of content that can be captured. Specifically, it is a challenge to create an array of physical light field cameras that is small enough and patterned densely enough to capture a full light field image of an uncontrolled scene without any sparse sampling artifacts, where sparse sampling artifacts are areas of the image where the light field camera is unable to capture an image. Fig. 9A situation is shown in which the frustum volume 74a of one light field camera 52a in the camera array has limited or no overlap with the frustum volume 74b of the adjacent light field camera 52b, thereby creating a blind volume area 76a in which no image can be captured. As shown, in the camera array located at the display plane 18, there are multiple blind volume areas 76a, 76b, 76c, 76d in areas where the cameras in the array cannot capture the light field. Although the figure is shown in cross-section, it will be understood that the cameras in the camera array will generally be regularly spaced throughout the 2D display plane 18, and therefore the blind volumes will be distributed throughout the display plane 18. Since no image information can be transmitted to the pixels, these blind volume areas will cause sparse sampling artifacts that appear as black areas on the display, and therefore the pixels will be black or remain off.

[0126] Fig.10 A light field capture method is shown that offsets the light field cameras 52a, 52b in the camera array from the display plane 18 to the retraction plane 64, thereby allowing the entire inner frustum volume 44 to be captured by the light field camera array by increasing the overlap of the individual camera frustum volumes 74a, 74b, respectively. For simplicity, two light field cameras 52a, 52b are shown, however, it is understood that the camera array according to the present disclosure includes a plurality of cameras spaced apart in a 2D array in the x, y plane of the camera array, which plane is shown here as the retraction plane 64. To increase the overlap of the individual camera frustum volumes 74a, 74b, the retraction plane 64 is retracted within the outer frustum volume 42 of the light field by an integral offset distance 62 perpendicular to the display plane 18. The retraction plane 64 can be located, for example, outside or inside the near clip plane 46. At the retraction plane 64 is an array of light field cameras positioned so that each light field camera is the same offset distance 62 from the display plane 18, and the light field cameras are positioned relative to each other at the sample gap. As previously described, the number of light field cameras used to capture the 3D scene and render the light field can be reduced to the minimum number required based on the sample gap calculation. However, it will be appreciated that the camera array used to produce a complex and high quality light field display typically includes more than two cameras.

[0127] The disclosed method of capturing the outer frustum volume 42 includes moving the light field cameras 52a, 52b to the retraction plane 64 and rendering in the forward direction, as described herein. This method is contrasted with the less advantageous method of setting the near clip plane 46 and the far clip plane 48 to negative values ​​and placing the light field cameras 52a, 52b on the display plane 18. Unless each holographic element is rendered, the latter method does not capture the complete outer frustum 42 volume, which defeats the purpose of encoding. Retracting the light field cameras 52a, 52b to the retraction plane 64 allows the entire scene in the inner frustum volume 44 and the outer frustum volume 42 to be captured. Synthesizing the captured image to the display plane changes the image so that it appears as if the image was taken on the display plane and allows the image to present the inner frustum volume 42 and the outer frustum volume 44. Once synthesized, the focus of the scene is on the display plane 18, providing an immersive experience for the viewer.

[0128] The placement and number of light field cameras 52a, 52b on the retraction plane 64, as well as the number of these cameras used for any particular image capture, are designed based on the limitations and processing capabilities of the system. As previously described, the number of light field cameras 52 used to capture the 3D scene can be reduced to the minimum number required based on the sample gap calculation. The placement and number of light field cameras 52 in the camera array can provide a sufficiently large area to capture the entire expected light field volume expected frustum, that is, the inner frustum volume 44 and / or the outer frustum volume 42. The purpose of the disclosed rendering method is to calculate the number of light field cameras 52 required from the retraction plane 64. In this embodiment, the frustum expected to be captured is the inner frustum volume 44. Retracting the light field cameras 52a, 52b to the retraction plane 64 allows the individual camera frustum volumes 74a, 74b of the light field cameras 52a, 52b to capture the entire inner frustum volume 44, respectively, while the blind volume 76 only exists in the outer frustum volume 42. Note that although this embodiment shows two light field cameras 52a, 52b, any number of light field cameras may be used, and in any desired 2D orientation, such as square, rectangular, hexagonal, triangular, or any other reasonable orientation.

[0129] In one embodiment, in a three-dimensional embodiment, at an integral offset distance N=1, a plane of four light field cameras may be located on the retraction plane 64 to capture the inner frustum volume 44 of the light field of the LFD. In another embodiment, at an offset distance N=2, a plane of nine light field cameras 52a, 52b may be located on the retraction plane 64 to capture the outer frustum volume 42 of the light field display. Conventional methods of capturing light field images using physical light field cameras require capturing source images ranging from 100 to 100,000. In contrast, this method can create a light field image of the physical world using 4 source images (using a 2x2 light field camera array). It should be noted that the 2x2 light field camera array only represents the inner frustum volume 44, which is still considered a valid light field image. In order to capture the outer frustum volume 42, nine source images are required (i.e., at least 3x3 light field cameras in the camera array). The reduction in the number of light field cameras required to capture a 3D scene further reduces the amount of data that needs to be transmitted and processed, and improves the feasibility of capturing physical images of a light field display. In contrast, the disclosed method uses a 3x3 light field camera array to produce 9 source images, while traditional capture methods require at least 100 source images, so the data set that needs to be rendered is reduced, thereby improving the rendering efficiency of the same 3D scene by 91%.

[0130] The size of the retraction plane 64 results from the spatial and directional resolution of the light field display and the selected integral offset parameter N.

[0131] therefore:

[0132] Retraction plane = [SR x +(DR x *(N-2)), SR y +(DR y *(N-2)), DR x , DR y ]

[0133] in:

[0134] SR x is the spatial resolution or number of holograms in the display on the x-axis

[0135] SR y is the spatial resolution or number of holograms in the display on the y-axis

[0136] DR x is the directional resolution or number of pixels per hologram on the x-axis

[0137] DR y is the directional resolution or number of pixels per hologram on the y-axis

[0138] N is an integer offset or retraction number of the display plane

[0139] The offset distance 62 between the retraction plane 64 and the display plane 18 is equal to an integer multiple of the focal length multiplied by the directional resolution. Light field displays typically set the inner and outer frustum volumes 44, 42 to be equal to the focal length of the light field camera capturing the light field, which is equal to the focal length of the display multiplied by the directional resolution. At this volume, the pixel pitch will be equal to the hologram pitch, resulting in a high quality image, and the retraction plane 64 is located on the near clipping plane 46. As the inner and outer frustum volumes 44, 42 increase to be greater than the focal length of the light field camera multiplied by the directional resolution, the sampling rate limits the image quality at the near clipping plane 46 and the far clipping plane 48. A light field display does not have to conform to a specific size to prove useful. If the inner and outer frustum volumes 44, 42 of the LFD are not based on the focal length and directional resolution, the retraction plane 64 may not be located at the near clipping plane 46.

[0140] In the present approach, the retraction plane 64 is offset from the display plane 18 by a distance 62 determined by the properties of the light field display which are designed to accommodate and optimize the optical properties of the light field cameras 52a, 52b.

[0141] therefore:

[0142] The offset distance is = N(f*DR)

[0143] in:

[0144] f is the focal length of the display

[0145] DR is the directional resolution of the display plane

[0146] N is an integer offset from the display plane

[0147] Since the offset distance 62 is calculated based on the focal length and directional resolution of the light field cameras 52a, 52b, the directional resolution of the light field cameras will be the same as the directional resolution of the display plane 18. All light field cameras 52 in the camera array should have the same optical properties, including but not limited to orientation lens spacing, directional resolution, and field of view, so that the offset distance 62 of the camera array at the retraction plane 64 is consistent with the offset distance 62 of the camera array at the display plane, while keeping all inner frustum volumes 44 within the field of view of the light field cameras 52a, 52b.

[0148] At any particular distance, rays traced from camera pixels in the retraction plane 64 will intersect the display plane 18 such that the distance between pixels, or pixel pitch, at the display plane 18 is equal to the pixel pitch at the retraction plane 64. Therefore, the pixels at the display plane 18 and the pixels at the retraction plane 64 will have the same resolution and lens pitch, which is a required condition for implementing the pixel remapping function. The term "same resolution" refers to the directional resolution of the pixels on the display plane 18 in the LFD. The condition that the pixels on the display plane 18 have the same directional resolution as the pixels on the retraction plane 64 is required in any case because the pixel remapping function only changes the holographic element index (H x ,H y ), while the directional pixel index (P x ,P y ) does not change during remapping. Furthermore, it is required in all cases that the pixels on display plane 18 have the same lens pitch as the pixels on retraction plane 64, because the remapping function is achieved by offsetting the retraction plane 64 by a certain distance (offset distance 62), where the pixel pitch matches the lens pitch of each pixel in the display plane 18 and the retraction plane 64.

[0149] Challenges that arise when capturing and displaying physical light field images have previously inhibited the use of captured light field images of the physical world on light field displays. In particular, capturing the inner frustum volume 44 and the outer frustum volume 42 from a real-world light field is very challenging due to sparse sampling artifacts resulting from the lack of overlap of the frustum volumes of the light field camera. The present method and system solves this problem by offsetting the camera array to the retraction plane 64 and regenerating the light field image captured by the camera array using rendering, decoding, and re-indexing methods to generate a light field image at the display plane 18.

[0150] Fig.11 An embodiment of the disclosed method is shown, wherein the offset parameter is N=1. The retraction plane 64 can be offset from the display plane 18 by an offset distance 62 (N), where N is an arbitrary integer. The illustrated embodiment shows the retraction plane 64 with an offset of N=1, which is the minimum possible offset from the display plane 18. It will be appreciated that for N=0, the retraction plane 64 is located on the same plane as the display plane 18. When N=1, the retraction plane 64 is located on the near clipping plane 46, where the outer frustum volume 42 is determined by the focal length and directional resolution of the display. Otherwise, when N=1, the retraction plane 64 is not located on the near clipping plane 46. At other offset distances of N=2, N=3, and N>1, the retraction plane 64 is located outside the outer frustum volume 42 and the near clipping plane 46.

[0151] Once the multiple light field cameras 52 in the camera array capture the light field at the retraction plane 64, the depth image-based rendering completes the rendering operation by projecting the captured light field data onto the image plane. In the rendering method, based on the position of the hologram relative to the light field camera 52 in the camera array, multiple holographic element cameras 54 with the same optical properties as the hologram are generated, and each holographic element camera 54 has a geometrically correct element image. After decoding, a holographic element camera 54 with a related element image is generated, and the multiple element images generated by the multiple holographic element cameras 54 and the multiple element images generated by the multiple light field cameras 52 will together constitute an integral image of the light field. The integral image is a light field image that can be displayed on a light field display. As shown in the figure, each hologram at the retraction plane is rendered through 2D (x, y) geometric adjustment to provide an element image of each hologram at the display plane for emission in the LFD.

[0152] Reprojecting a color plus depth image is a decoding method using a light field camera 52 and its forward vector direction. Using the depth information and index of the pixel, particularly the x index and y index, the coordinates (x, y, z) of the pixel in 3D space can be calculated. The origin and forward direction of the hologram can be used to convert the pixel in 3D space into the depth, x index and y index of a single hologram and its associated element image. Using the decoded x index and y index, the pixel can be written to an element image on an image plane, including but not limited to the retraction plane 64 or the display plane 18. The pixel remapping function includes repeating the reprojecting color plus depth image decoding method until all holograms are filled with pixels of the element image written to each hologram.

[0153] One approach to addressing the physical limitations of conventional light field capture techniques is an additional advantage of the disclosed light field rendering method, which is to decode one or more light field cameras 52 by generating the holographic meta-cameras 54 in the holographic meta-camera array in the same plane as the light field camera array and synthesizing the holographic meta-camera array to produce a cohesive integral image. After capture, the 3D scene data can be processed to include image information, including but not limited to color, transparency, and depth information. Pixel indexing (also known as pixel remapping) provides a location for each pixel, allowing computer software to identify each individual pixel and the image information associated with each pixel. The identification of pixel locations and their image information allows computer software to modify and process the image so that the pixel information in the holographic meta-camera at the retraction plane 64 can be remapped to the pixel information in the hologram at the display plane 18. The holographic meta-camera 54 can be generated by a decoding method, such as rendering based on a depth image, reprojecting a color plus depth image, etc., to provide image information to all pixels in the LFD.

[0154] The decoding method, such as rendering based on the depth image, re-projecting the color plus depth image, etc., generates a holographic element camera 54 for each single holographic element, where each holographic element camera has an associated element image associated with the holographic element position in the light field. In other words, the decoding generates the element image from the source image captured by each light field camera. The decoding method can create a holographic element camera 54 that is small enough that, when patterned densely enough, the holographic element camera 54 creates an integral image while providing a very high density integral image for display on the LFD. According to the present disclosure, the input to the decoding in this context is a sparse array of light field cameras 52. The reason why the array is sparse is that a sample gap is used, which provides the distance or spacing between the light field cameras 52 in the camera array required to obtain the light field image, and the holographic element camera 54 is generated or synthesized using the gap between the light field cameras 52 located on the same x, y plane. In one embodiment, in order to create an integral light field using a sparse array of light field cameras 52, multiple light field cameras 52 are decoded into a complete array of holographic element cameras 54 in the same plane, and the position of each holographic element camera 54 is geometrically adjusted. In one embodiment, the decoding process first decodes the sparsely spaced light field camera array vertically. After this step, the height of the integral image matches the height of the decoded integral image. The second step is to decode the integral image horizontally. After the second step, the integral image is the decoded integral image with the correct height and width. With the sample gap, fewer light field cameras 52 need to be decoded to produce a full array of holographic meta-cameras 54 and provide a complete integral image on the camera plane.

[0155] For simplicity, the disclosed rendering method has been described with respect to a single light field camera 52 and a single holographic meta-camera 54. Naturally, it will be appreciated that a light field camera 52 array comprising a plurality of light field cameras 52 will generate a holographic meta-camera 54 array comprising a plurality of holographic meta-cameras 54. The number of light field cameras 52 required to capture 3D scene data is determined by a sample gap calculation, which is defined by the encoding and decoding scheme. The sample gap determines the maximum distance between two light field cameras 52 in the camera array that is sufficient to provide light field data between the light field cameras 52 and generate an array of holographic meta-cameras 54. Therefore:

[0156] Sample gap = l*DR

[0157] in:

[0158] l is the lens spacing of the light field camera

[0159] DR is the directional resolution of the display plane

[0160] After decoding, each hologram in the LFD will have a hologram camera 54, and the hologram camera 54 will share the same optical properties with its hologram. Each hologram camera 54 is generated to contain an element image, and its associated hologram will be remapped to emit the element image at the LFD. The group of hologram cameras 54 creates an integral image together with the light field camera 52, where the integral image is composed of multiple element images. Each element image is associated with a single hologram, and a set of holograms is required to create a light field at the LFD. A hologram can be defined as a light engine that produces light. Each hologram has a field of view, a 2D resolution, an (x, y, z) position in space (origin), and a forward direction. After decoding, the hologram camera spacing (i.e., the distance between the centers of two adjacent hologram cameras 54 on the retraction plane 64) is equal to the lens spacing of the light field display. The blind volume 76 is located only in the outer frustum volume 42. At this stage, the hologram camera 54 is synthesized with its hologram, which requires decoding to provide an element image. All holographic element cameras 54 are arranged on a uniform grid, are coplanar, and have the same position and optical properties as their holographic elements, including but not limited to orientation, lens spacing, directional resolution, and field of view. Once a holographic element camera 54 is synthesized with its holographic element, the prerequisites for implementing the pixel remapping technique are met.

[0161] Fig.12 An embodiment of the present disclosure is shown in which the retraction plane 64 is calculated using an offset parameter N=2 from the display plane 18. In the illustrated 3D embodiment, the frustum volume 74 is captured by nine light field cameras 52 arranged on an x, y grid on the retraction plane 64. Fig.12 Shown is a 2D illustration of a 3D embodiment in which three light field 52 cameras are shown, but it will be appreciated that multiple light field cameras 52 are required when capturing light field images, arranged in an array on the retraction plane 64. The number of light field cameras 52 required is calculated to ensure that the entire outer frustum volume 42 and inner frustum volume 44 of the light field display are adequately captured. Using an offset parameter N=2, the blind volume 76 is outside the near clipping plane 46, and the frustum volume 74 captured by the light field camera array at the retraction plane 64 is the complete inner frustum volume 44 and outer frustum volume 42 of the LFD. Therefore, when the offset distance is set to N=2, the inner frustum volume 44 and outer frustum volume 42 can be fully captured without the need for a blind volume 76. N=2 can be used instead of N=1 so that the complete volume will be captured with one layer. As shown Fig.11As shown, after decoding the light field camera 52, a holographic meta-camera 54 is generated, wherein the spacing of the holographic meta-cameras on the retraction plane 64 is equal to the lens spacing of the display. The holographic meta-camera 54 is synthesized with its hologram, thus satisfying the prerequisites for implementing the pixel remapping technique. Synthesis during the decoding stage is performed to generate a virtual image at each holographic meta-camera 54 to generate an element image, wherein each element image is associated with a single holographic meta-camera 54.

[0162] Fig.13 Indexing a single hologram 30 in a hologram array 40 including pixels 28 within a light field display is shown. In conventional rendering methods, a camera array would be located on a display plane of a light field display to capture an image of a 3D scene. According to the method of the present disclosure, a light field camera array may be located on a retraction plane at an offset distance perpendicular to the display plane. The pixel index is used to reference individual pixels 28 and holograms 30 contained in the hologram array 40 that includes the light field. Once each pixel in the light field is indexed, computer software can manipulate the image information and thereby manipulate the light field. For example, computer software can render light field image data captured by a light field camera, transmit light field image data captured by a light field camera, store light field image data captured by a light field camera, change the color of light field image data captured by a light field camera, change the transparency of light field image data captured by a light field camera, and the like. A light field has a spatial resolution and a directional resolution; each of these display properties has an x ​​component and a y component. Spatial resolution (SR x 38,SR y 36) refers to the number of holograms 30 in the hologram grid 40 used to construct the light field image. The spatial resolution x (SR x ) 38 represents the number of holographic elements 30 of the light field display along the horizontal direction (x axis). The spatial resolution y (SR y ) 36 represents the number of holograms 30 in the light field display along the vertical direction (y axis). x ,H y ) identifies the location of each hologram 30 within the spatial resolution of the light field display. The index provides an initial location for each pixel 28 and hologram 30, allowing computer software to process the image. The light field re-indexing method utilizes this indexing method to manipulate the image so that it appears in a plane parallel to the plane in which the image was captured. Directional resolution (DR x 32,DR y 34) refers to the number of pixels 28 comprising each single holographic element 30. The directional resolution x (DR x ) 32 represents the number of pixels 28 of the hologram 30 along the horizontal direction (ie, the x-axis). y ) 34 represents the number of pixels 28 of the hologram 30 along the vertical direction (ie, the y-axis).x ,P y ) identifies the position of each pixel 28 within the directional resolution of its hologram 30.

[0163] Using the holographic meta index (H x ,H y ) and pixel index (P x ,P y ), each pixel 28 can be indexed to indicate its position in the light field display.

[0164] therefore:

[0165] LF[H x , H y , P x , P y ]

[0166] in:

[0167] H x is the holographic meta index along the x-axis, where the leftmost column is 1

[0168] H y is the holographic element index along the y-axis, where the top row is 1

[0169] P x is the pixel index along the x-axis, where the leftmost column is 1

[0170] P y is the pixel index along the y-axis, where the top row is 1

[0171] For further explanation of pixel and hologram indexing, refer to indexed hologram 80. Identifying a particular pixel (e.g., pixel 78 within hologram 80) would be LF[1, 6, 2, 1], where H x =1,H y =6, P x =2,P y = 1, which provides the initial position for the pixels 78 and hologram 80, allowing the computer software to process the image. Pixel remapping technology uses the index to manipulate the image so that it appears in a plane parallel to the plane in which the image was captured.

[0172] Pixel indexing is a common practice, however, the pixel remapping technique described herein will result in a light field image shifting its perceived position while preserving motion parallax. Consider that the pixel remapping technique is essentially a hologram remapping technique, since holograms are made of pixels. However, known methods of shifting the image plane utilize interpolation or subsampling, which reduces image resolution, fails to preserve motion parallax, and increases computational complexity and bandwidth requirements for reading additional pixel data from memory. The present method of shifting the light field image plane utilizes pixel remapping techniques by shifting the perceived position of the light field image plane in the retraction plane (LF r ) and stores each pixel in the light field to the display plane (LF d ) at the light field, the holographic element index (H x , H y )Offset pixel index (P x , P y ).

[0173] therefore:

[0174]

[0175] in:

[0176] LF d is the light field at the display plane

[0177] LF r is the light field at the retraction plane

[0178] H x is the holographic meta index along the x-axis, where the leftmost column is 1

[0179] H y is the holographic element index along the y-axis, where the top row is 1

[0180] P x is the pixel index along the x-axis, where the leftmost column is 1

[0181] P y is the pixel index along the y-axis, where the top row is 1

[0182] DR x is the directional resolution of each holographic element on the x-axis, that is, the number of pixels

[0183] DR y is the directional resolution of each holographic element on the y-axis, that is, the number of pixels

[0184] N is the offset parameter; an integer representing the number of retractions

[0185] Applying the present pixel remapping technique allows computer software to select a pixel from the retracted planar hologram 30 and use the following equation LFr [H x +(DR x *N)-(N*P x ), H y +(DR y *N)-(N*P y ), P x , P y ] modifies its index position, thereby remapping the pixel to another holographic element at the display plane. This method is computationally straightforward for computer software, requiring only the computing software to read one pixel from the retraction plane to produce one pixel at the display plane. This minimizes mathematical operations, uses only integers, and requires only a few source images. In contrast, other methods may require thousands of source images, which increases the amount of data captured, rendered and / or transmitted. Other methods using a small number of source images may require floating point numbers, that is, not integer values, which increases computational complexity and therefore increases the time and hardware requirements for generating the light field. For example, a known method for moving an image plane is interpolation. To move an image plane by interpolation, each output pixel (e.g., a pixel at the display plane) will be read from four source pixels (e.g., a pixel at the retraction plane). In contrast, in the disclosed light field rendering method, each display plane pixel is read from only one retraction plane pixel. Therefore, the pixel remapping technique only requires one-quarter of the data required for interpolation. For commercially available memory devices, such as random access memory (RAM), double data rate (DDR) memory, synchronous dynamic random access memory (SDRAM), etc., it has been found that the method can be executed in one-quarter of the time compared to interpolation methods, resulting in higher frames per second (fps) frame rates. This reduction in time and data allows the light field to be transmitted and rendered in real time on commercially available systems while maintaining resolution and motion parallax, thereby creating high-quality light field images.

[0186] Preferably, the pixel remapping technique only causes pixels to change their holographic element index (H x , H y The pixel itself maintains its pixel index (P x , P y ). Therefore, the directional resolution of the light field display remains unchanged when the spatial resolution changes. The spatial resolution also changes because the holographic element index H x , H y Re-indexing requires peripheral (additional) holographic elements.

[0187] In the method, the display plane hologram receives each pixel from a different retract plane hologram. The retract plane must be composed of enough holograms to provide enough pixels to achieve the directional resolution required by the light field display on the display plane. The size of the display plane is defined by the directional resolution and spatial resolution of the light field display.

[0188] Display plane = [SR x , SR y , DR x , DR y ]

[0189] in:

[0190] SR x is the spatial resolution of the display on the x-axis, i.e. the number of holographic elements

[0191] SR y is the spatial resolution of the display on the y-axis, i.e. the number of holographic elements

[0192] DR x is the directional resolution of each holographic element on the x-axis, i.e. the number of pixels

[0193] DR y is the directional resolution of each holographic element on the y-axis, that is, the number of pixels

[0194] After the decoding process, the retraction plane consists of additional holograms (herein called peripheral holograms) that provide pixels for the display plane holograms. These peripheral holograms ensure that all display plane holograms have the required directional resolution. The number of peripheral holograms depends on the directional resolution of the display plane holograms and the offset parameter N.

[0195] therefore:

[0196] Peripheral hologram = [(DR x *N), (DR y *N)]

[0197] In addition, the peripheral holograms satisfy the requirement that each pixel in each hologram on the display plane is filled according to the pixel remapping technique. When capturing and decoding the light field image on the retraction plane, the area of ​​the display plane and subsequently the number of holograms on the display plane must be considered to create enough peripheral holograms to meet the requirements of the light field rendering method, thereby creating a high-quality light field image.

[0198] Compositing is used to combine multiple remapped planes to create a seamless display that is rendered in both the inner and outer frustum volumes. Compositing is an exact implementation of the light field rendering method that incorporates transparency information when combining multiple light fields into a single light field image on the display plane. Each remapped light field LFN The pixels are loaded into the Composite(LF d , LF N ), mixed with transparency data, and stored in the display plane LF d in a light field to be displayed as a single light field image.

[0199] therefore:

[0200] For each LF N :

[0201] For each holographic element:

[0202] For each pixel:

[0203] LF d [H x , H y , P x , P y ]

[0204] =Composite(LF d [H x , H y , P x , P y ],LF N [H x +DR x -P x , H y +DR y -P y , P x , P y ])

[0205] in:

[0206] LF d is the light field at the display plane

[0207] LF r is the light field at the retraction plane

[0208] H x is the holographic meta index along the x-axis, where the leftmost column is 1

[0209] H y is the holographic element index along the y-axis, where the top row is 1

[0210] P x is the pixel index along the x-axis, where the leftmost column is 1

[0211] P y is the pixel index along the y-axis, where the top row is 1

[0212] DRx is the directional resolution of each holographic element on the x-axis, i.e. the number of pixels

[0213] DR y is the directional resolution of each holographic element on the y-axis, that is, the number of pixels

[0214] Composite(LF d , LF N ) returns either remapped pixels or blended pixels, depending on the type of transparency data. Each remapped light field is composited to the display surface in ascending order of the offset parameter N value. In the absence of transparency data, Composite(LF d , LF N ) will write the remapped pixels to the display surface as follows:

[0215] Composite(LF d , LF N ) → Return to LF N

[0216] After compositing, the remapped pixels are stored on the display plane and ready for display. Without transparency data, compositing multiple light fields with different N values ​​onto a light field on the display plane will result in the last light field composited onto the display plane being the only visible light field image. Transparency data combines the different transparency and opacity levels of each element in the image so that when multiple light field images are composited together, opaque elements are visible when they are closer to the far clip plane than transparent elements. Transparency data is critical when compositing multiple light fields into a single light field image on the display plane to provide a 3D layered light field image. Commercially available physical light field cameras do not capture transparency data when capturing images because the camera's sensor captures the intensity of a color: red, blue, green. Source images from a physical light field camera can be processed using methods including, but not limited to, temporal median filtering, interactive foreground extraction, etc. to incorporate transparency data into the image before rendering. In the presence of transparency data, Composite(LF d , LF N ) will return a mixture of the light field at the retraction plane and the light field at the display plane.

[0217] Composce(LF d , LF N ) → Return to Blend(LF d , LF N )

[0218] The two light fields are then combined together for blending using the transparency data. After blending, the pixels can be stored on the display plane and prepared for display. In embodiments with multiple light fields, the composition will load the light fields in order starting with the light field closest to the far clip plane, which initially has the lowest N value and the shortest offset distance. Each rendered light field is composited to the display plane until all light fields are composited into a single light field on the display plane. Compositing using transparency data ensures that objects retain their proper depth, transparency, and blur once composited into a single light field image, further improving motion parallax of the light field. This ensures that objects in the light field closest to the near clip plane are fully visible at all angles within the display's field of view, i.e., no objects occlude their frontmost surfaces, objects farther from the near clip plane will be partially or fully covered, and the occlusion of each object depends on the occluding objects located between the object in question and the near clip plane and the transparency of the occluding objects. For example, an object located on the far clip plane may be fully visible if there are no occluding objects between the object in question and the near clip plane, or if any occluding objects are partially or fully transparent. The synthesis combines multiple light fields from the retraction planes and integrates transparency data and stores them into a single light field on the display plane to create a light field image containing an image of the physical world that is simultaneously presented in the inner and outer frustum volumes while maintaining motion parallax.

[0219] Fig.14 is a flow chart of an example light field rendering method. Before the method begins, the light field camera array needs to meet a series of prerequisites. The first step is to set up or decode the light field camera plane array 100. The camera plane array can be a physical camera, such as, for example, a digital single-lens reflex (DSLR) camera, a pinhole camera, a plenoptic camera, a compact camera, a mirrorless camera, etc., or it can be a computer-generated or virtual camera array. The decoding method (such as rendering based on a depth image, re-projecting a color plus depth image, etc.) generates a virtual holographic element array that displays an image in a holographic element where a physical camera does not exist or cannot exist due to size constraints. After decoding, the distance between the light field cameras on the retraction plane is equal to the sample gap, which is also the lens spacing of the display. The initial light field camera plane array is decoded, and a holographic element camera is generated in each holographic element, so that the final display plane camera array has [SR x , SR y , DR x , DR y ] shape. x is the spatial resolution of the display on the x-axis, SR y is the spatial resolution of the display on the y-axis, DR x DR is the directional resolution of the display on the x-axis. yis the directional resolution of the display on the y-axis. This sets the parameters of the final light field display after rendering. Step 102 asks: Does the retracted plane array of the light field camera satisfy [SR x +(DR x *N), SR y +(DR y *N), DR x , DR y ]? 102, where the offset parameter N represents the number of retractions. This ensures that the retraction plane contains enough holographic elements to provide the display plane with the full directional resolution (DR) required to create an integral image. x , DR y ) and spatial resolution (SR x , SR y ). If the answer to step 102 is no, the algorithm adjusts the camera to comply with 104 and returns to step 102. If yes, the algorithm continues to step 106. Step 106 asks: Does each hologram in the retraction plane have a camera? Each hologram on the retraction plane requires a hologram camera because the camera provides an image for its associated hologram. This step ensures that all holograms on the retraction plane have cameras, so each hologram has an image. If not, the algorithm adjusts the camera to comply with 108 and returns to step 106. If yes, the algorithm continues to step 110. Step 110 asks: Do all hologram cameras have the same position, orientation, lens spacing, directional resolution, and field of view as the holograms they are associated with? . Each hologram camera must have the same position as its associated hologram to ensure that the image provided by the camera for the hologram is correct for the position of the hologram. Subsequently, the hologram camera must have the same optical properties as its associated hologram; orientation, lens spacing, directional resolution, and field of view; to ensure that the camera's image shares the optical properties of the hologram. When the optical properties of the holographic meta-cameras and their respective holographic meta-cameras match, the rays from the pixels in the retraction plane will intersect on the display plane, allowing the method to remap the perceived position of the integral image to the display plane. If not, the algorithm adjusts the cameras to match 112 and returns to step 110. If yes, the algorithm continues to Fig.15 Step 114 in .

[0220] Fig.15 is a flow chart of an example light field rendering method including pixel remapping techniques. Once all prerequisites are met ( Fig.14100-112), the algorithm will continue by starting the pixel remapping technique 114 using hologram [1,1]. The first hologram to be remapped can be any hologram within the display. In this embodiment, for simplicity, hologram [1,1] is used. Once a hologram is selected, the pixel remapping technique will start 116 from pixel [1,1]. The first pixel within the hologram to be remapped can be any pixel. In this embodiment, for simplicity, pixel [1,1] is used. First, the LF of the retracted plane light field is r [H x +(DR x *N)-(N*P x ), H y +(DR y *N)-(N*P y ), P x , P y ] loads a pixel 118, where H x , H y represents a specific holographic element in the display, and P x , P y Represents a specific pixel within the hologram. This step remaps each pixel from the retraction plane to the display plane. x , H y ) is based on the display’s spatial resolution (SR x , SR y ). In this embodiment, for simplicity, the boundary 1≤H is selected x ≤SR x and 1≤H y ≤SR y , but can vary depending on how the hologram is indexed, i.e. the boundary can be 0≤H x ≤SR x -1 and 0 ≤ H y ≤SR y -1. Pixel (P x , P y ) is based on the directional resolution (DR) of the hologram. x , DR y ). In this embodiment, for simplicity, the boundary 1≤P is selected x ≤DR x and 1≤P y ≤DR y , but can vary depending on how the pixels are indexed, i.e. the boundary can be 0≤P x ≤DR x -1 and 0 ≤ P y ≤DR y -1. After the pixel is loaded from the retraction plane, the pixel is stored in the LF of the display plane light field.d [H x , H y , P x , P y ] in 120. Step 120 stores the remapped pixels in the display plane to allow it to be displayed as a light field image. A key advantage of this method is that the entire operation is loading from one address (in this embodiment, the retraction plane) and storing to another address (in this embodiment, the display plane). Compared to known methods, the disclosed rendering method does not require additional operations to be performed on the loaded data. Once loaded to the display plane, the data is an integral image and can be presented as a light field.

[0221] After each pixel is stored to the display plane light field in step 120, the next step 122 asks: Have all pixels in the hologram been remapped? If not, the algorithm repeats the light field reindexing method 124 for the next pixel, starting to load the next pixel in step 118. If yes, the algorithm proceeds to step 126. After all pixels within the hologram are stored to the display plane light field, the next step asks: Are all holograms on the display plane populated with remapped pixels? 126. If not, the algorithm repeats the remapping technique 128 for the next hologram, starting to load pixels in the next hologram in step 116. If yes, the algorithm proceeds to step 130. Once all pixels within all holograms have been remapped by loading and storing to their respective display plane holograms, all conditions of the method are satisfied and the light field display is able to display a rendered light field image 130.

[0222] Fig.16 is a flow chart of a light field rendering method process, illustrating a computer-implemented method, comprising:

[0223] a. defining a light field display having an inner frustum volume defined by a display plane and a far clipping plane and an outer frustum volume 140 defined by a display plane and a near clipping plane;

[0224] b. Positioning the light field camera array at a retraction plane at an offset distance from the display plane, wherein each light field camera has the same lens spacing, focal length, directional resolution, and field of view, and the offset distance is an integer multiple of the camera focal length multiplied by the directional resolution 142;

[0225] c. using each light field camera to capture the 3D scene at the retraction plane as a set of source images, each light field camera having one source image 144;

[0226] d. decoding the light field camera at the retraction plane to generate a plurality of holographic element cameras, wherein each holographic element camera has an associated holographic element, the associated holographic element comprising an element image 146 represented by a pixel array;

[0227] e. generating an integral image 148 comprising an elemental image at the retraction plane; and

[0228] f. Applying a pixel remapping technique to individual pixels in the integral image to create a rendered light field image 150 at the display plane.

[0229] All publications, patents and patent applications mentioned in this specification are indicative of the technical level of those skilled in the art to which the invention pertains and are incorporated herein by reference. Reference to any prior art in this specification is not, and should not be construed as, an admission or any form of suggestion that such prior art constitutes part of the common general knowledge.

[0230] The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the scope of the invention, and all such modifications obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

1. A computer-implemented light field rendering method, comprising: defining a display plane for a light field, the light field comprising an inner frustum volume bounded by the display plane and a far clipping plane and an outer frustum volume bounded by the display plane and a near clipping plane; defining a retraction plane parallel to the display plane and having an integral offset distance from the display plane, the retraction plane comprising a plurality of light field cameras spaced apart by a sample gap; capturing a view of the 3D scene as a source image at each of the plurality of light field cameras; decoding each source image to generate a plurality of holographic meta-cameras on the retraction plane, each holographic meta-camera providing an elemental image; generating an integral image comprising a plurality of pixels from an elemental image at the retraction plane; as well as A pixel remapping technique is performed on each pixel in the integral image to create a rendered light field image on the display plane. 2 . The method of claim 1 , wherein the portion of the 3D scene captured at the retraction plane includes image information from the inner frustum volume and the outer frustum volume.

3. The method according to claim 1 or 2, wherein the captured 3D scene includes all image information in the outer frustum volume.

4. The method according to any one of claims 1 to 3, wherein the integrated offset distance is calculated based on the focal length, the directional resolution of the light field camera and the offset integer N. The method according to claim 4 , wherein the offset integer N≧1. The method according to claim 1 , wherein the retraction plane is located at the near clipping plane.

7. The method according to any one of claims 1 to 6, further comprising: Display the rendered light field image on a light field display.

8. The method according to any one of claims 1-7, wherein the retraction plane surface area is larger than the display plane surface area.

9. The method according to any one of claims 1 to 8, wherein the optical properties of the light field camera are orientation, lens spacing, directional resolution and field of view.

10. The method according to any one of claims 1 to 9, further comprising: A plurality of integral images are generated at a plurality of retraction planes.

11. The method according to claim 10, further comprising: The plurality of integral images are composited to create a composite rendered light field image at the display plane. The method of claim 11 , wherein the synthesis incorporates transparency data.

13. The method of any one of claims 1-12, wherein each light field camera is one of a digital single-lens reflex (DSLR) camera, a pinhole camera, a plenoptic camera, a compact camera, and a mirrorless camera.

14. The method of any one of claims 1-13, wherein each light field camera is a computer generated camera.

15. The method according to any one of claims 1 to 14, wherein the pixel remapping technique causes the pixel to have its holographic element index (H x ,H y ) changes from the retraction plane to the display plane.

16. The method of any one of claims 1-15, wherein the retraction plane is located outside the outer frustum volume.

17. A computer-implemented method for displaying a light field image, comprising: capturing, using a light field camera, a first light field at a retraction plane relative to a light field display plane, the first light field comprising an array of retraction plane holographic elements, each holographic element having a plurality of pixels; By applying a pixel remapping technique to select a single pixel from each retracted plane hologram, each pixel in each retracted plane hologram is assigned a hologram index (H x ,H y ) and pixel index (P x ,P y ) to indicate its position in the light field display; Each pixel from the light field is loaded into the retraction plane (LF) using a composition function r ) and stores each pixel in the display plane (LF d ) at the light field; as well as A light field image including the remapped pixels is generated at the display plane.

18. The method of claim 17, wherein one pixel from the retraction plane produces one pixel at the display plane.

19. The method according to claim 17 or 18, wherein a pixel remapping technique is applied to change the holographic element index (H) of each pixel in each retracted plane holographic element. x ,H y ), and the pixel index (P x ,P y ) remain unchanged.

20. The method of any of claims 17-19, wherein the pixel remapping technique is a function of a directional resolution of the light field display and an offset parameter N.

21. The method of claim 19, wherein the pixel remapping technique is based on the equation The method of claim 20 , wherein the offset integer N≧1.

23. The method of any one of claims 17-22, wherein the retraction plane is composed of a sufficient number of holographic elements to provide a number of pixels on the display plane to achieve a directional resolution required for a light field display.

24. The method of any one of claims 17-23, wherein the size of the display plane is defined by a directional resolution and a spatial resolution of the light field display.

25. The method of any one of claims 17-24, wherein the light field camera is a reflective (DSLR) camera, a pinhole camera, a plenoptic camera, a compact camera, or a mirrorless camera.

Images