Wavefront sampling and modulation

Abstract

An energy modulation system that includes a first spatial energy modulator (SEM) located in a first energy propagation position and operable to spatially modulate energy propagated therethrough, and a second SEM located in a second energy propagation position and operable to spatially modulate energy propagated from the first SEM. The first SEM includes first and second modulation areas comprising respective first and second plurality of energy locations, wherein a first energy modulation is spatially applied to the energy propagated through the first and second plurality of energy locations. The second SEM includes first and second modulation regions, wherein a second energy modulation is spatially applied to the energy propagated therethrough. The first modulation region operable to modulate energy propagated from the first modulation area such that energy propagated through the first modulation area and the first modulation region comprises a collection of wavefronts, each wavefront having a complex amplitude modulation.

Description

SYSTEM AND METHODS FOR MODULATING AND PROPAGATING ENERGYWAVEFRONTSTECHNICAL FIELD

[0001] This disclosure is generally related to modulating energy waves, and more specifically for modulating and propagating energy waves with improved resolution.BACKGROUND

[0002] The dream of an interactive virtual world within a “holodeck” chamber as popularized by Gene Roddenberry’s Star Trek and originally envisioned by author Alexander Moszkowski in the early 1900s has been the inspiration for science fiction and technological innovation for nearly a century. However, no compelling implementation of this experience exists outside of literature, media, and the collective imagination of children and adults alike.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] FIG. 1A is a schematic diagram illustrating design parameters for an energy directing system;

[0004] FIG. IB illustrates an embodiment of an energy modulation system;

[0005] FIG. 1C illustrates an application of an energy modulation system according to an embodiment;

[0006] FIG. ID illustrates another application of the energy modulation system according to an embodiment;

[0007] FIGS. IE- IK illustrate a method for determining the amplitude and phase modulation frequency requirements at the energy modulation surface based on minimum visual acuity requirement for the object location according to an embodiment;

[0008] FIGS. IL- IM illustrates an application of a method for determining the amplitude and phase modulation frequency requirements at the energy modulation surface based on minimum visual acuity requirement for the object location according to an embodiment;

[0009] FIG. 2 is a schematic diagram illustrating gm energy system having an active device area with a mechanical envelope;

[0010] FIG. 3 is a schematic diagram illustrating an energy relay system;

[0011] FIG. 4 is a schematic diagram illustrating an embodiment of energy relay elements adhered together and fastened to a base structure;

[0012] FIG. 5A is a schematic diagram illustrating an example of a relayed image through multicore optical fibers;

[0013] FIG. 5B is a schematic diagram illustrating an example of a relayed image through an energy relay that exhibits the properties of the Transverse Anderson Localization principle;

[0014] FIG. 5C illustrates a schematic cutaway view in the transverse plane of a portion of an energy relay comprising a random distribution of two component materials;

[0015] FIG. 5D illustrates a schematic cutaway view in the transverse plane of a module of an energy relay comprising a non-random pattern of three component materials which define a single module;

[0016] FIG. 5E illustrates a schematic cutaway view in the transverse plane of a portion of a prefused energy relay comprising a random distribution of two component materials;

[0017] FIG. 5F illustrates a schematic cutaway view in the transverse plane of a portion of a prefused energy relay comprising a nonrandom distribution of three component materials which define multiple modules with similar orientations;

[0018] FIG. 5G illustrates a schematic cutaway view in the transverse plane of a portion of a prefused energy relay comprising a non-random pattern of three component materials which define multiple modules with varying orientations;

[0019] FIG. 5H illustrates a schematic cutaway view in the transverse plane of a portion of a fused energy relay comprising a random distribution of two component materials;

[0020] FIG. 51 illustrates a schematic cutaway view in the transverse plane of a portion of a fused energy relay comprising a non-random pattern of three component materials;

[0021] FIG. 6 is a schematic diagram showing rays propagated from an energy surface to a viewer;

[0022] FIG. 7A illustrates a perspective view of an energy waveguide system having a base structure, four energy devices, and four energy relay elements forming a seamless energy surface, in accordance with one embodiment of the present disclosure;

[0023] FIG. 7B illustrates an energy relay system according to one embodiment of the present disclosure;

[0024] FIG. 7C illustrates a top-down perspective view of an embodiment of an energy waveguide system according to one embodiment of the present disclosure;

[0025] FIG. 7D illustrates a front perspective view of the embodiment shown in FIG. 7C;

[0026] FIGS. 7E-7L illustrate various embodiments of an energy inhibiting element;

[0027] FIG. 8 is a flow chart illustrating an embodiment of a process for processing holographic sensory data;

[0028] FIG. 9 is a schematic diagram of a virtual environment constructed from sensory data;

[0029] FIG. 10 is a schematic diagram illustrating an embodiment of energy tracing;

[0030] FIG. 11 is a schematic diagram illustrating an embodiment of an energy directing device going through a tracing process;

[0031] FIG. 12 is a schematic diagram of a processing system for process holographic sensory data;

[0032] FIG. 13 is a block diagram illustrating an embodiment of a vectorization process;

[0033] FIG. 14 illustrates some aspects of modeling energy in a 3D environment;

[0034] FIG. 15 is a flow chart illustrating an embodiment of a process for rendering a 4D energyfield from a 3D environment;

[0035] FIG. 16 depicts a visual representation of a 3D environment;

[0036] FIG. 17 depicts one perspective of virtual pixels, a virtual aperture, and virtual viewing locations from a 3D environment;

[0037] FIG. 18 depicts one perspective of virtual pixels, virtual apertures, and a virtual viewing location from a 3D environment;

[0038] FIG. 19 depicts a visual representation of a 3D environment from the perspective of a virtual viewing location

[0039] FIG. 20 is a schematic diagram of inverse tracing through various objects in a digital volumetric representation;

[0040] FIG. 21 a schematic diagram illustrating an embodiment of a system for rendering a 4D energy field from a 3D environment;

[0041] FIG. 22A illustrates a laser modeled as a set of plane waves according to an embodiment;

[0042] FIG. 22B illustrates a laser modeled as a set of spherical waves according to an embodiment;

[0043] FIG. 22C illustrates a far-field single slit diffraction pattern arising from a single rectangular aperture illuminated by an energy wave from a monochromatic light source;

[0044] FIG. 23A is a schematic diagram illustrating a reference energy modulation system for modulating a plane wavefront according to an embodiment;

[0045] FIG. 23B is a schematic diagram illustrating a wavefront sample for the reference energy modulation system shown in FIG. 23A;

[0046] FIG. 23C is a schematic diagram illustrating a reference energy modulation system for modulating a plurality of plane wavefronts according to embodiment;

[0047] FIG. 23D is a schematic diagram illustrating a reference energy modulation system according to an embodiment;

[0048] FIG. 23E is a schematic diagram illustrating a reference energy modulation system for modulating a converging spherical wavefront according to an embodiment;

[0049] FIG. 23F is a schematic diagram illustrating a wavefront sample for the reference energy modulation system shown in FIG. 23E;

[0050] FIG. 23 G is a schematic diagram illustrating a reference energy modulation system according to an embodiment;

[0051] FIG. 23H is a schematic diagram illustrating a reference energy modulation system for modulating a diverging spherical wavefront according to an embodiment;

[0052] FIG. 23F is a schematic diagram illustrating a wavefront sample for the reference energy modulation system shown in FIG. 23H;

[0053] FIG. 23J is a schematic diagram illustrating a reference energy modulation system according to an embodiment;

[0054] FIG. 24A is a schematic diagram illustrating a reference energy modulation system for modulating a plane wavefront that takes into account the physical extent of a reference modulation area in accordance with an embodiment;

[0055] FIG. 24B is a schematic diagram a wavefront sample for the reference energy modulation system shown in FIG. 24A;

[0056] FIG. 24C is a schematic diagram illustrating a reference energy modulation system for modulating a converging spherical wavefront that takes into account the physical extent of a reference modulation area in accordance with an embodiment;

[0057] FIG. 24D is a schematic diagram a wavefront sample for the reference energy modulation system shown in FIG. 24C;

[0058] FIG. 24E is a schematic diagram illustrating a reference energy modulation system according to an embodiment;

[0059] FIG. 24F is a schematic diagram illustrating a reference energy modulation system for modulating a diverging spherical wavefront that takes into account the physical extent of a reference modulation area in accordance with an embodiment;

[0060] FIG. 24G is a schematic diagram a wavefront sample for the reference energy modulation system shown in FIG. 24F;

[0061] FIG. 24H is a schematic diagram illustrating a reference energy modulation system according to an embodiment;

[0062] FIG. 241 illustrates a comparison of the effective spherical wave and a theoretical plane wave;

[0063] FIG. 24J illustrates a portion of the effective spherical wave shown in Fig. 241 that includes contributions due to the physical extent of the modulation area;

[0064] FIG. 24K illustrates a portion of the effective spherical wave shown in Fig. 241 that includes contributions due to diffraction and the physical extent of the modulation area;

[0065] FIG. 24L illustrates a model for sampling effective spherical waves that make up a wavefront sample;

[0066] FIG. 240 illustrates a wavefront sample approximating an effective spherical wave according to an embodiment;

[0067] FIG. 24P illustrates a wavefront sample approximating an effective spherical wave according to an embodiment;

[0068] FIG. 24Q illustrates a wavefront sample approximating an aggregate of a plurality of spherical wave according to an embodiment;

[0069] FIG. 24R illustrates another wavefront sample approximating an effective spherical wave according to an embodiment;

[0070] FIG. 24S illustrates yet another wavefront sample approximating an aggregate of a plurality of spherical wave according to an embodiment;

[0071] FIG. 24T illustrates an embodiment of a wavefront sample based on effective spherical waves that are overlapping with and offset from each other according to an embodiment;

[0072] FIG. 24U illustrates an embodiment of an aggregated wavefront sample based on a plurality of effective spherical waves that are overlapping with and offset from each other in accordance with an embodiment;

[0073] FIG. 24V illustrates sampling of effective spherical waves from two perspective Tenderers;

[0074] FIG. 24W illustrates sampling of aggregated effective spherical waves from two perspective Tenderers;

[0075] FIG. 25A is a schematic diagram illustrating an embodiment of a reference energy modulation system for modulating plane wavefronts;

[0076] FIG. 25B is a schematic diagram illustrating the wavefront samples based on the model for the reference energy modulation system shown in FIG. 25A;

[0077] FIG. 26A is an illustration of a first example of a relative positioning reference surface;

[0078] FIG. 26B is an illustration of a second example of a relative position reference surface;

[0079] FIG. 27A is a schematic diagram illustrating a model of a reference energy modulation system having various reference modulation areas in a reference energy modulation surface;

[0080] FIG. 27B is a schematic diagram illustrating the wavefront samples corresponding to the reference energy modulation areas shown in FIG. 27A;

[0081] FIG. 28A is a schematic diagram illustrating a first example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27 A;

[0082] FIG. 28B is a schematic diagram illustrating a second example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27A;

[0083] FIG. 28C is a schematic diagram illustrating a third example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27 A;

[0084] FIG. 28D is a schematic diagram illustrating a fourth example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27A;

[0085] FIG. 28E is a schematic diagram illustrating a fifth example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27 A;

[0086] FIG. 28F is a schematic diagram illustrating a sixth example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27 A;

[0087] FIG. 28G is a schematic diagram illustrating a seventh example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27A;

[0088] FIG. 28H is a schematic diagram illustrating a eighth example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27A;

[0089] FIG. 281 is a schematic diagram illustrating a ninth example alignment of reference energy modulation elements and the reference energy modulation areas shown in FIG. 27 A;

[0090] FIG. 29A is a schematic diagram illustrating a first example distribution of the reference energy modulation areas;

[0091] FIG. 29B is a schematic diagram illustrating a second example distribution of the reference energy modulation areas;

[0092] FIG. 29C is a schematic diagram illustrating a third example distribution of the reference energy modulation areas;

[0093] FIG. 29D is a schematic diagram illustrating a fourth example distribution of the reference energy modulation areas;

[0094] FIG. 30A is a schematic diagram illustrating a model of a reference energy modulation system having a plurality of first reference modulation energy surfaces and second reference modulation energy surfaces;

[0095] FIG. 30B is a schematic diagram illustrating the wavefront samples based on the model for the reference energy modulation system shown in FIG. 20A; and

[0096] FIGS. 31-33 comprise schematic diagrams illustrating a reference energy modulation system with a plurality of reference energy modulation surfaces.

[0097] FIG. 34 illustrates an example of expansion of wavefronts of energy waves propagating over a distance according to an embodiment;

[0098] FIG. 35A is a schematic diagram illustrating a reference energy modulation system for modulating planar and spherical wavefronts according to an embodiment;

[0099] FIG. 35B is a schematic diagram illustrating a reference energy modulation system for modulating various wavefronts according to an embodiment;

[0100] FIG. 35C is a schematic diagram illustrating the propagation of energy waves modulated by the reference energy modulation system of FIG. 35B according to an embodiment;

[0101] FIGS. 36A-36B illustrate the modulation transfer function;

[0102] FIG. 37 shows a model of the energy intensity distribution of the energy sampled by an effective spherical wavefront sample according to an embodiment;

[0103] FIG. 38 shows a modeling of the energy intensity distribution of the energy sampled by effective spherical wavefront samples having a sample distance therebetween to allow MTF threshold of 100% at an object projection distance according to an embodiment;

[0104] FIG. 39 shows a modeling of the energy intensity distribution of the energy sampled by effective spherical wavefront samples having a sample distance therebetween to allow MTF threshold of 50% at an object projection distance according to an embodiment;

[0105] FIG. 40 shows a modeling of the energy intensity distribution of the energy sampled by effective spherical wavefront samples having a sample distance therebetween to allow MTF threshold of 9% at an object projection distance according to an embodiment;

[0106] FIG. 41 shows a modeling of the energy intensity distribution of the energy sampled by effective spherical wavefront samples through different energy propagation positions;

[0107] FIG. 43C is a graph illustrating data for the spot size as a function of resolved object projection distance and spot location according to an embodiment;

[0108] FIGS. 44A-44B illustrate examples of spacing between adjacent sampling regions as a function of the projection distance for a desired on-axis MTF according to embodiments;

[0109] FIG. 45A shows a modeling of the energy intensity distribution of the energy sampled by on- and off- axis effective spherical wavefront samples to allow MTF threshold of 70% at an object projection distance according to an embodiment;

[0110] FIG. 45B shows a modeling of the energy intensity distribution of the energy sampled by on- and off- axis effective spherical wavefront samples to allow MTF threshold of 50% at an object projection distance according to an embodiment;[oni] FIG. 45C shows a modeling of the energy intensity distribution of the energy sampled by on- and off- axis effective spherical wavefront samples to allow MTF threshold of 9% at an object projection distance according to an embodiment;

[0112] FIG. 45D shows a modeling of the energy intensity distribution of the energy sampled by on- and off- axis effective spherical wavefront samples to allow MTF threshold of 0% at an object projection distance according to an embodiment;

[0113] FIG. 47 is an illustration of an example modulation of inter-sampling region sample spacing to allow both MTF 100% and convergence of on-axis and off-axis effective spherical wavefront samples having respective on-axis and off-axis virtual effective origins in accordance with an embodiment;

[0114] FIG. 48 is an illustration of an example modulation of inter-sampling region sample spacing to allow both MTF 50% and convergence of on-axis and off-axis effective spherical wavefront samples having respective on-axis and off-axis effective virtual origins in accordance with an embodiment;

[0115] FIG. 49A shows effective spherical wavefront samples 4910 sampled through multiple sampling regions 4940 according to an embodiment;

[0116] FIG. 49B shows a wavefront sample that is oversampled with numerous effective spherical wavefront samples according to an embodiment;

[0117] FIG. 52 illustrates a model that accounts for relative illumination as a function of field angle in accordance with an embodiment;

[0118] FIG. 53 illustrates example plots of relative illumination and absolute numerical efficiency as a function of field angle for the model discussed with respect to FIG. 52; and

[0119] FIG. 54 illustrates an embodiment of a compensation for relative illumination.DETAILED DESCRIPTION

[0120] An embodiment of a Holodeck (collectively called “Holodeck Design Parameters”) provide sufficient energy stimulus to fool the human sensory receptors into believing that received energy impulses within a virtual, social and interactive environment are real, providing: 1) binocular disparity without external accessories, head-mounted eyewear, or other peripherals; 2) accurate motion parallax, occlusion and opacity throughout a viewing volume simultaneously for any number of viewers; 3) visual focus through synchronous convergence, accommodation and miosis of the eye for all perceived electromagnetic energy; and 4) converging energy wave propagation of sufficient density and resolution to exceed the human sensory “resolution” for vision, hearing, touch, taste, smell, and / or balance.

[0121] Based upon conventional technology to date, we are decades, if not centuries away from a technology capable of providing for all receptive fields in a compelling way as suggested by the Holodeck Design Parameters including the visual, auditory, somatosensory, gustatory, olfactory, and vestibular systems.

[0122] In this disclosure, the terms light field and holographic may be used interchangeably to define the energy propagation for stimulation of any sensory receptor response. While initial disclosures may refer to examples of electromagnetic energy and mechanical energy propagation through energy surfaces for holographic imagery and volumetric haptics, the principles of the present disclosure are equally applicable to all forms of sensory. Furthermore, the principles disclosed herein for energy propagation along propagation paths may be applicable to both energy emission and energy capture.

[0123] Many technologies exist today that are often unfortunately confused with holograms including lenticular printing, Pepper’s Ghost, glasses-free stereoscopic displays, horizontal parallax displays, head-mounted VR and AR displays (HMD), and other such illusions generalizedas “fauxlography.” These technologies may exhibit some of the desired properties of a true holographic display, however, lack the ability to stimulate the human visual sensory response in any way sufficient to address at least two of the four identified Holodeck Design Parameters.

[0124] These challenges have not been successfully implemented by conventional technology to produce a seamless energy surface sufficient for holographic energy propagation. There are various approaches to implementing volumetric and direction multiplexed light field displays including parallax barriers, hogels, voxels, diffractive optics, multi-view projection, holographic diffusers, rotational mirrors, multilayered displays, time sequential displays, head mounted display, etc., however, conventional approaches may involve a compromise on image quality, resolution, angular sampling density, size, cost, safety, frame rate, etc., ultimately resulting in an unviable technology.

[0125] To achieve the Holodeck Design Parameters for the visual, auditory, somatosensory systems, the human acuity of each of the respective systems is studied and understood to propagate energy waves to sufficiently fool the human sensory receptors. The visual system is capable of resolving to approximately 1 arc min, the auditory system may distinguish the difference in placement as little as three degrees, and the somatosensory system at the hands are capable of discerning points separated by 2 - 12mm. While there are various and conflicting ways to measure these acuities, these values are sufficient to understand the systems and methods to stimulate perception of energy propagation.

[0126] Of the noted sensory receptors, the human visual system is by far the most sensitive given that even a single photon can induce sensation. For this reason, much of this introduction will focus on visual energy wave propagation, and vastly lower resolution energy systems coupled within a disclosed energy waveguide surface may converge appropriate signals to induce holographic sensory perception. Unless otherwise noted, all disclosures apply to all energy and sensory domains.

[0127] When calculating for effective design parameters of the energy propagation for the visual system given a viewing volume and viewing distance, a desired energy surface may be designed to include many gigapixels of effective energy location density. For wide viewing volumes, or near field viewing, the design parameters of a desired energy surface may include hundreds of gigapixels or more of effective energy location density. By comparison, a desired energy source may be designed to have 1 to 250 effective megapixels of energy location density for ultrasonicpropagation of volumetric haptics or an array of 36 to 3,600 effective energy locations for acoustic propagation of holographic sound depending on input environmental variables. What is important to note is that with a disclosed bi-directional energy surface architecture, all components may be configured to form the appropriate structures for any energy domain to enable holographic propagation.

[0128] However, the main challenge to enable the Holodeck today involves available visual technologies and energy device limitations. Acoustic and ultrasonic devices are less challenging given the orders of magnitude difference in desired density based upon sensory acuity in the respective receptive field, although the complexity should not be underestimated. While holographic emulsion exists with resolutions exceeding the desired density to encode interference patterns in static imagery, state-of-the-art display devices are limited by resolution, data throughput and manufacturing feasibility. To date, no singular display device has been able to meaningfully produce a light field having near holographic resolution for visual acuity.

[0129] Production of a single silicon-based device capable of meeting the desired resolution for a compelling holographic display may not practical and may involve extremely complex fabrication processes beyond the current manufacturing capabilities. The limitation to tiling multiple existing display devices together involves the seams and gap formed by the physical size of packaging, electronics, enclosure, optics and a number of other challenges that inevitably result in an unviable technology from an imaging, cost and / or a size standpoint.

[0130] The embodiments disclosed herein may provide a real-world path to building the Holodeck.

[0131] Example embodiments will now be described hereinafter with reference to the accompanying drawings, which form a part hereof, and which illustrate example embodiments which may be practiced. As used in the disclosures and the appended claims, the terms "embodiment", "example embodiment", and "exemplary embodiment" do not necessarily refer to a single embodiment, although they may, and various example embodiments may be readily combined and interchanged, without departing from the scope or spirit of example embodiments. Furthermore, the terminology as used herein is for the purpose of describing example embodiments only and is not intended to be limitations. In this respect, as used herein, the term “in” may include “in” and “on”, and the terms “a,” “an” and “the” may include singular and plural references. Furthermore, as used herein, the term “by” may also mean “from”, depending on the context. Furthermore, as used herein, the term "if may also mean "when" or "upon," depending on thecontext. Furthermore, as used herein, the words “and / or” may refer to and encompass any and all possible combinations of one or more of the associated listed items.Energy Wave Propagation

[0132] Some forms of energy, such as electromagnetic energy and mechanical energy, can be modeled as energy waves. In some models, energy waves emitted by a point-source propagates in an oscillating energy wavefield. Energy wave oscillation may be characterized by cycles, in which a full cycle comprises 360 degrees, or 2n radians. A harmonic sinusoidal wave has a phase defined by Asin(2jix / ), where A is the wave amplitude (the maximum value of wave oscillation), x is the length of wave path from the origin, and X the wavelength of light. A wavefront may be understood to be an imaginary surface where the energy wave has the same phase. The geometric characteristics of an energy wave may be represented by light rays that are straight lines having the origin at the point-source and remain perpendicular to the wavefront. The light rays, however, do not represent the total energy propagating through the energy wavefield. The wavefront identifies the location of in-phase energy wave, and calculations determining the properties of the energy wave allow for qualitative and quantitative analyses of the energy wave propagation.

[0133] Wave equations define the analytical framework for the propagation of electromagnetic wave. Analytical solution to the wave equations describes a wavefield of a propagating wavefront anywhere in space. However, obtaining an analytical solution for an arbitrary object having arbitrary shape and size is complex and nearly impossible. Instead, a numerical solution of the wave equations may be determined computationally, but even the computational determination can be time consuming and impractical. As such, various approximation model can be derived and used to simplify the computational determination of the numerical solution of the wave equations.

[0134] In one approach, the solution to vectoral wave equations is approximated by making certain assumptions to reduce the wave equations to scalar components. These integral equations are scalar in nature and hence are also called as scalar diffraction formulas. These approximated solutions based on scalar diffraction can be referred to as the “scalar theory” and can simplify computational determination of a hologram with satisfactory resolution and accuracy in holographic display applications.

[0135] The starting point of the scalar diffraction theory are the Maxwell equations that describe an electromagnetic energy wave. When it is assumed that an electromagnetic energy wave ispropagating in a linear, uniform, isotropic, homogeneous, and non-dispersive material, the vector wave equation 1.1 below can be derived from the Maxwell equations to describe the electric field component:where e0is the permittivity of vacuum, eris a relative permittivity, and p0is the permeability of vacuum.

[0136] Since light behaves like a wave, the vector wave equation 1.1 can be written in the form of a scalar wave equation 1.2:where c is the speed of the wave in a dielectric media. Comparing the vector wave and scalar wave equations 1.1 and 1.2, the speed of light is can be written as:where cois the speed of light in vacuum and equal to l / ^ / p.oeo, and n is the refractive index and equal to

[0137] The component w(p, t) can be considered as a wave function defining a scalar field component at the given position p and time t in a material having the refractive index n. For a monochromatic wave, the scalar field function can be written as a complex amplitude according to equation 1.4:where co is the cyclic frequency of the light and is equal to 2TCV.

[0138] The function U(p) is referred to as a complex amplitude given by equation 1.5:wherein A(p) is a real value that can be understood as the amplitude and 0(p) can be understood as the phase of the complex amplitude. The complex amplitude U(p) is a three-dimensional function, and its two-dimensional distribution in a given plane is referred to as a wavefield.

[0139] Substituting equations 1.4 and 1.5 into the wave equation in equation 1.2, the wave equation can be rewritten in the same form as the Helmholtz equation:where A is the wavelength of light, and k is equal to 2JT / X and referred to as a wave number.

[0140] In the context of determining a wavefront, the object wave, the reference wave, and the signal wave can be modeled as plane waves or spherical waves based on the wave equations discussed above. Plane waves have planar wavefronts, have a constant frequency and amplitude, and extend indefinitely. However, for a given space, any complex wavefront can be modeled as one or more local plane wave. The solution to the wave equations above for a plane wave is:where <po is a constant defining the phase of the cosine function at time t = 0 and at r= 0, p is a position vector, and k defines a wave vector pointing in the direction of propagation with a length |fc| = in which Xo is the wavelength of the light in vacuum.

[0141] Based on equations 1.4 and 1.5, the complex amplitude of a plane wave described by equation 1.7 can be written as:

[0142] In Cartesian coordinates, the wavefield of a plane wave at Z=ZQ is:

[0143] Spherical waves are emitted from a single point source and have spherical wavefronts.The solution to the wave equations above for a plane wave is:where r is the radial distance in a spherical coordinate system and can be determined based on Cartesian coordinates, x, y, z as shown in equation 1.11:

[0144] Based on equations 1.4 and 1.5, the complex amplitude of a spherical wave described by equation 1.10 can be written as:

[0145] In Cartesian coordinates, the wavefield of a spherical wave at z=zo is:where

[0146] Based on equations 1.9 and 1.14, a wavefield can be sampled at various (x, y) positions at the z=zo plane for a plane wave or a spherical wave, respectively.

[0147] An energy wave in free space according to the assumptions of the scalar theory is described by the wave equations 1.2 and 1.6, and the plane and spherical waves discussed above are specific solutions to the wave equations 1.2 and 1.6. Boundary conditions can help solve the wave equations 1.2 and 1.6 to provide a more general solution for a wave that propagates in free space. When light is diffracted by an aperture, the boundary condition can be understood to be a binary function, in which the value of the function is unity inside the aperture and zero outside the aperture. In an embodiment, a wavefield defined by a first complex amplitude in a first plane can be used as the boundary condition for solving the wave equations 1.2 and 1.6 to obtain a diffracted wavefield defined by a second complex amplitude in a second plane at any location in a 3D space. In an embodiment, a computational implementation of this process may be referred to as numerical field propagation and include the steps of sampling a source wavefield (i.e., the first amplitude) and using it as a boundary condition to numerically solve the wave equations to simulate the destination / diffracted field (i.e., the second complex amplitude). In a cartesian coordinate system, the source field U(x, y; zo) is located in the source plane and is propagated by a distance d to the diffracted field U(x’, y z) located in the propagation plane. The spatial coordinates for a point in plane and a point in propagation plane and have coordinates (x’, y’, 0) and (x, y, z) respectively.

[0148] The propagation of wavefield can be modelled according to a number of diffraction formulations known in the art, including Fresnel-Kirchhoff diffraction and Rayleigh-Sommerfeld diffraction. Both Fresnel-Kirchhoff formulation and Rayleigh-Sommerfeld formulation have been derived and described extensively in the art, and thus detailed derivation will not be repeated here.

[0149] The Fresnel-Kirchhoff diffraction formulation relies on Green’s identities to express the disturbance at an arbitrary point P as a function of the values of the solution of the wave equation and its first order derivative at all points on an arbitrary surface which encloses P. Its most general form can be written as equation 1.15 below:where U is the complex amplitude of the disturbance at the surface, and 5 is the distance from P to the surface.

[0150] In the coordinate system discussed above, the Fresnel-Kirchhoff formulation can be written as equation 1.16 below:where r = [(x — x')2+ (y—yz)2+ z2]172, and % is the diffraction angle at point (x’, y’, 0) between the diffracted wave and the normal of the z=0 plane. The Fresnel-Kirchhoff formulation is an approximate scalar solution of the Helmholtz equation that is generally accurate unless very close to the aperture. The componen n equation 1.16 is also known as the obliquityfactor and can be approximated to be 1 when the diffraction angle is small.

[0151] Determining a solution for the integral in equation 1.16 can be challenging for most applications, and certain assumptions can be made for r in equation 1.16 to derive approximated diffraction models known in the art that are easier to solve either analytically or numerically.

[0152] For example, Fresnel approximation is a known approximation model that can simplify equation 1.16 by making several approximations. The first approximation is based on the assumption that the diffraction angle is small, and thus the component in equation 1.16,which is also known as the obliquity factor, can be approximated to 1. The assumption that the diffraction angle is small also allows the approximation of r in the denominator of to be z.However, in the exponent of th a small variation in r can significantly change the value of eikr. Thus, a more accurate approximation of r can be made by assuming (y—y')2truncating the expansion of according to equation1.17:

[0153] The Fresnel diffraction integral in equation 1.17 can be derived from the above assumptions:The Fresnel diffraction integral can be solved using various numerical techniques known in the art, including fast Fourier transforms.

[0154] The Fraunhofer approximation is another approximation that can simplify the Fresnel- Kirchhoff diffraction formula. As a diffraction pattern continuously evolves along the z-direction, at a distance far from the aperture, it eventually evolves into a final diffraction pattern that maintains itself as it continues to propagate (although it increases its size in proportion to distance). This far-away diffraction pattern in a far field is described by the Fraunhofer approximation. This is the limiting case of the Fresnel approximation when the field is observed at a distance far after the aperture.

[0155] Equation 1.18 can be expanded into equation 1.19 below,— K(xT / 2+v / 2l and in far field condition (z » k / 2), the exponential component ezz21 y 1in equation 1.19 can be approximated to be 1, resulting in the Fraunhofer diffraction integral as shown in equation 1.20 below:

[0156] The Fraunhofer integral can be interpreted as a two-dimensional (inverse) Fourier transform on the source wave field U (%', y 0) where kx / z and ky / z can be considered as spatial frequencies.

[0157] The above discussed Fresnel and Fraunhofer approximations can also be applied to Rayleigh-Sommerfeld formulation of diffraction. Compared to the Fresnel-Kirchhoff formulation, the Rayleigh- Sommerfeld formulation provides more rigorous solutions in equations 1.21 (first solution) and 1.22 (second solution) using two different boundary conditions.(1-21) and

[0158] In the coordinate system in FIG. 8, the first solution to the Rayleigh-Sommerfeld formulation can be written as equation 1.23 below:

[0159] Applying the same assumptions in the Fresnel approximation discussed above, (i.e., r ~ z for the non-exponential components and r ® z 4-in the exponential component),equation 1.23 can be simplified to equation 1.24:

[0160] From the Fresnel approximation in equation 1.24, the Franhofer approximation can be applied similarly to further simply equation 1.24 to

[0161] The Rayleigh Sommerfeld formulation is a more rigorous model than the Fresnel Kirchhoff formulation because of the former’s the mathematical consistency and the latter’s ability to reproduce closely the diffracted field right behind the aperture. However, the Rayleigh Sommerfeld formulation is limited by its assumption of a plane surface, and the Fresnel Kirchhoff formulation can handle surfaces of any shape, thereby allowing for more accurate propagations in optical applications.

[0162] The assumptions in the scalar theory discussed above may be not applicable in some applications. For example, in an embodiment, more accuracy may be needed in some applications where there are significant deviations from the assumptions of the scalar theory (i.e., monochromatic wave, a linear, uniform, isotropic, homogeneous, and non-dispersive material). A more rigorous approach may be used to model the propagation of a wavefront through a modulating 3D energy medium 602 without some or all of the assumptions made under the scalar theory.

[0163] More rigorous diffraction models can account for couplings between the various components of electric and the magnetic fields. In an embodiment, a more rigorous diffraction model suitable for implementation of the present disclosure may provide wave equations and boundary conditions that can be solved exactly without approximation. Analytical solutions for the more rigorous diffraction models may be more accurate for a wider scope of applications, but the mathematical complexity can make determining analytical solutions impractical. Instead, in an embodiment, numerical solutions can be determined for the more rigorous models in some implementations of the present disclosure.

[0164] In the coupled waved theory, a periodic grating structure and a diffracted field can be described in the form of Fourier series. The Fourier expression helps to reduce the mathematical complexity of the boundary problem.

[0165] It is to be appreciated that the above discussed coupled waves theory is provided herein as an example of a model for energy wavefield propagation without making the assumptions of scalar theory. A number of other modelling approaches not based on the assumptions of the scalar theory, such as the N-coupled wave theory and the Parallel Stacked Mirrors theory, are also well known in the art and will not be repeated here. One of ordinary skill in the art will appreciate that these models are also suitable to be implemented herein like the coupled wave theory to model the propagation of energy field in accordance with the principles of the present disclosure.Holographic System Considerations:Overview of Holographic Energy Propagation Resolution

[0166] In an embodiment, a holographic display is operable to propagate energy waves from a plurality of energy surface locations and to spatially apply phase and amplitude modulation to the energy propagated therethrough to allow for desired angular, color, and intensity information propagated within a holographic object volume. The disclosed energy surface provides opportunities for additional information to coexist and propagate through the same surface to induce other sensory system responses. Unlike a stereoscopic display, the viewed position of the converged energy propagation paths in space do not vary as the viewer moves around the viewing volume and any number of viewers may simultaneously see propagated objects in real- world space as if it was truly there. In some embodiments, the propagation of energy may be located in the same energy propagation path but in opposite directions. For example, energy emission and energy capture along an energy propagation path are both possible in some embodiments of the present disclosed.

[0167] FIG. 1 A is a schematic diagram illustrating variables relevant for an example approach of stimulation of sensory receptor response. These variables may include surface diagonal 101, surface width 102, surface height 103, a determined target seating distance 118, the target seating field of view field of view from the center of the display 104, the number of intermediate samples demonstrated here as samples between the eyes 105, the average adult inter-ocular separation 106,the average resolution of the human eye in arcmin 107, the horizontal field of view formed between the target viewer location and the surface width 108, the vertical field of view formed between the target viewer location and the surface height 109, the resultant horizontal waveguide element resolution, or total number of elements, across the surface 110, the resultant vertical waveguide element resolution, or total number of elements, across the surface 111, the sample distance based upon the inter-ocular spacing between the eyes and the number of intermediate samples for angular projection between the eyes 112, the angular sampling which may be based upon the sample distance and the target seating distance 113, the total resolution Horizontal per waveguide element derived from the angular sampling desired 114, the total resolution Vertical per waveguide element derived from the angular sampling desired 115, device Horizontal which is the count of the determined number of discreet energy sources desired 116, and device Vertical which is the count of the determined number of discreet energy sources desired 117.

[0168] An example approach to understanding the desired minimum resolution may be based upon the following criteria to ensure sufficient stimulation of visual (or other) sensory receptor response: surface size (e.g., 84" diagonal), surface aspect ratio (e.g., 16:9), seating distance (e.g., 128" from the display), seating field of view (e.g., 120 degrees or + / - 60 degrees about the center of the display), desired intermediate samples at a distance (e.g., one additional propagation path between the eyes), the average inter-ocular separation of an adult (approximately 65mm), and the average resolution of the human eye (approximately 1 arcmin). These example values should be considered placeholders depending on the specific application design parameters.

[0169] Further, each of the values attributed to the visual sensory receptors may be replaced with other systems to determine desired propagation path parameters. For other energy propagation embodiments, one may consider the auditory system’s angular sensitivity as low as three degrees, and the somatosensory system’s spatial resolution of the hands as small as 2 - 12mm.

[0170] While there are various and conflicting ways to measure these sensory acuities, these values are sufficient to understand the systems and methods to stimulate perception of virtual energy propagation. There are many ways to consider the design resolution, and the below proposed methodology combines pragmatic product considerations with the biological resolving limits of the sensory systems. As will be appreciated by one of ordinary skill in the art, the following overview is a simplification of any such system design, and should be considered for exemplary purposes only.

[0171] FIGS 1B-1M demonstrate another example approach for considering the design resolution of a holographic display. FIG. IB shows an energy modulation system 130 operable to modulate energy wavefronts across an energy modulation surface 132 with certain amplitude and phase modulation frequencies. In energy modulation system 130, the energy modulation surface 132 may have a plurality of modulation regions 136 through which energy may be propagated. The energy propagated through each modulation region 136 may comprise a plurality of modulated energy wavefronts. A first energy wavefront 140 propagated from a first modulation region of the energy modulation surface may be formed by the application of a first amplitude modulation and a first phase modulation to the energy across the first modulation region. In FIG. IB, the energy wavefront 140 is represented by a single geometrical ray 140 to schematically demonstrate a chief propagation path resulted from the first phase modulation applied. Other modulated energy wavefronts propagated from the first modulation region of the energy modulation surface may be formed by the application to respective amplitude modulation and respective phase modulation across the first modulation region. A plurality of energy wavefronts may be similarly formed and propagated from each of the other modulation regions of the energy modulation surface. As such the amplitude modulation frequency of the energy modulation system 130 across the energy modulation surface 132 is determined by the intervals 134 between the plurality of modulation regions 136 across a spatial dimension 142. For each modulation region 136, the phase modulation of the energy wavefronts propagated therethrough may comprise applying a phase modulation frequency across an aggregated angular range 144.

[0172] FIG. IB illustrates a parameterization of energy modulation based on phase and amplitude frequencies across a spatial dimension 142 and an angular range 144 for each region across the spatial dimension 142. It is appreciated that other types of energy modulation parameterization exist and may be used herein. The phase and amplitude frequencies for this parameterization may be determined based on the geometry in certain holographic environments, as detailed below.

[0173] In an example use case illustrated in FIG. 1C, the energy modulation system 130 of FIG. IB may operate to propagate modulated energy to form, at an object location 162, a holographic object that is resolvable from a viewing location 172. The design of such an energy modulation system 130 may be optimized based on a set of parameters related to this example envrionment, including a minimum acuity resolution for resolving the holographic object at the object location 162, and the relative positioning of the viewing location 172, the object location 162 in aholographic object volume, and the energy modulation surface location 152 of the energy modulation system 130. In an embodiment, the design of an energy modulation system 130 may be based on an optimization of the object location 162 at a near object location 164a in real space 154 between the viewing location 172 and the energy modulation surface location 152. In an embodiment, the design of an energy modulation system 130 may be based on an optimization of the objection location 162 at a far object location 164b in virtual space 153 behind the energy modulation surface location 152.

[0174] FIG. ID illustrates an example approach in which the design of the energy modulation system 130 is optimized for a minimum acuity resolution of R cycles per degree (CPD) for a near object location 164a at first distance DI from the viewing location 172. In an embodiment, the value of R may be in the range of 5 to 60 CPD. In an embodiment, the value of R may be in the range of 15 to 30 CPD. It is to be appreciated that the principles discussed herein may be applicable for optimizing values of R that may be higher or lower than the example ranges provided herein. In an example embodiment, the distance DI may be between 1 / 3 meter to 2 meters. In an embodiment optimized for relatively larger environments, DI may be between 1 meter to 100 meters. In an embodiment for smaller environments, DI may be between 1 / 10 meter to 2 / 3 meter. It is to be appreciated that the same principles and approach may be applied to other values of DI that may be higher or lower than the example ranges provided herein. Resolving, from the viewing location 172, the minimum resolution of the modulated energy propagated through the energy modulation surface 152 located at a distance D2 from the viewing location 172 may be considered as a desired design constraint that would in turn determine the design goals for amplitude and phase frequency that the configuration of the energy modulation system 130 would have to allow for.

[0175] FIGS. 1E-1L illustrate a method for determining the amplitude and phase modulation frequency requirements for an energy modulation system 130 to allow for the formation of a resolvable holographic object at an object location in accordance with an embodiment. As illustrated in FIG. IE, the goal is to form a holographic object at object location 162 that is resolvable by a viewer from the viewing location 172. The desired resolution of the holographic object may be determined based on a minimum acuity requirement. In an embodiment, a cycle per degree (CPD) acuity requirement R, may be used to define a desired angular resolution based on a point of reference at viewing location 172. One cycle, as illustrated by the dark and lightregions at the object location 162, is defined as transitioning from a dark region to a light region and to the next dark region. Alternatively, one cycle may also be defined as transitioning from a light region to a dark region and to the next light region. In an embodiment, the designed minimum acuity R CPD, as an angular resolution, is defined as N number of signals 226 across an angle of view 224 from the point of reference at viewing location 172. The angular resolution may be used to determine the spatial resolution requirement at the object location 162 as defined by a density measurement across a spatial dimension 214 across the object location 162. In an embodiment, the spatial dimension 214 may be defined by a width, length, or cross-sectional measurement. According to an embodiment, the density measurement refers to the number of black and white regions per unit distance in the spatial dimension 214 across the object location 162. In an embodiment, the angular resolution may be defined as the number of signals per unit angle (e.g., R = N / angle of view). The spatial resolution, on the other hand, may be defined as the number of signals per unit distance in the spatial dimension 214.

[0176] Next, as illustrated in FIG. IF, the spatial resolution at the object location 162 may be used to determine an angular resolution requirement (e.g., the phase modulation frequency requirement) for an energy modulation region 206 at the energy modulation surface location 152, with the modulation region 206 as a point of reference. For example, the density measurement across the spatial dimension 214 may be used to determine the number of signals 228 across a field of view 225 that is required to meet the minimum acuity requirement R. According to an embodiment, the distance DI between the viewing location 172 to the object location 162, distance D2 between the viewing location 172 and the energy modulation surface location 152, and the distance D3 between the energy modulation surface location 152 and the object location 162 may be used to determine the angular resolution requirement (e.g., the phase modulation frequency requirement) for an energy modulation region 206 at the energy modulation surface location 152. For example, the phase modulation frequency requirement for an energy modulation region 206 may be determined by the product of the angular resolution requirement (e.g., R CPD) at the viewing location 172 and the ratio of the distances D3 to DI (i.e.,D3Phase Frequency = — x CPD). The field of view 225 with the modulation region 206 as apoint of reference may be determined by the product of the angle of view 224, with the viewing location 172 as the point of reference, and the ratio of the distances DI to D3 (i.e.,Field of View = — X Angle of View). For example, in an embodiment, if distance DI is d and distance D3 is 0.5 , the resulting field of view would be twice as wide as the angle of view.

[0177] As for determining the spatial resolution requirement at the energy modulation surface 132 (e.g., the amplitude modulation frequency), one approach is to determine it based on the same angular resolution requirement defined by the visual acuity requirement (e.g., R CPD) at the viewing location 172 as illustrated by FIG. 1G. However, in doing so, the phase and amplitude modulations at the energy modulation surface 132 will not allow for enough converged signals at the object location 162 to satisfy the spatial resolution requirement associated therewith. As illustrated in FIG. 1H, when the phase modulation frequency (as determined based on the discussion with respect to FIG. IF) and amplitude modulation frequency (as determined based on the discussion with respect to FIG. 1G) are applied by the energy modulation surface (e.g., 132), the modulated energy signals generated therefrom are unable to form an array of convergence points that have enough spatial resolution at the object location 162. In FIG. 1H, when an eye is observing the object location 162 from the viewing location 172, it is able to focus only on points of convergence that have both an on-axis signal, as represented by the solid horizontal lines, and enough off-axis signals, as represented by the dashed diagonal lines, to satisfy the phase modulation frequency requirement determined based on the discussion with respect to FIG. IF. In other words, this means that the eye will not be able to focus on points where 1) there is only a convergence of off-axis signals without an on-axis signal, and 2) there is only a convergence of an on-axis signal but not enough off-axis signals. As shown in FIG. 1H, the spatial resolution (i.e. density) of the on-axis signals (solid lines) across the energy modulation surface location 152 is too sparse to allow for enough converged points at the dashed boxes even if there are infinite number of off-axis signals (dashed lines). The dashed boxes illustrate the spatial resolution requirement in the object location 162, which is not met because at least some 216 of the dashed boxes do not have a point of convergence between an on-axis signal and sufficient off-axis signals, because no on-axis signal propagates through those boxes 216. FIG. II illustrates the result of the sparse converging arrays generating insufficient spatial resolution at the object location 162 which, when perceived from the viewing location 172, results in a low-contrast blur.

[0178] Besides using the approach disclosed in FIGS. 1G-1I to determine the spatial resolution requirement at the energy modulation surface (i.e., the amplitude modulation frequency), another approach, as illustrated in FIG. 1 J, is to determine the difference in angular resolution at the objectlocation and at the energy surface location based on a same spatial resolution at these two locations. In FIG. 1 J, the spatial resolution at the energy modulation surface location 152 is quantifiable as an angular resolution over the same angle of view 224 that is defined using the viewing location 172 as a point of reference. This particular spatial resolution can be determined by the product of the angular resolution requirement (i.e., R CPD) and the ratio of the distances D2 and DI (i.e.,D2Amplitude Frequency = — X CPD), where D2 is the distance between the viewing location 172 and the energy modulation surface location 152, and DI is the distance between the viewing location 172 and the object location 162. According to an embodiment, the amplitude modulation interval 134 corresponds to the spatial resolution at the energy modulation surface location 152, which is set to be the same as the spatial resolution 234 at the object location 162.

[0179] As illustrated in FIG. IK, by setting the spatial resolution at the energy modulation surface location 152 to be the same as the spatial resolution at the object location 162, the spatial resolution of the on-axis signals, as represented by sold lines, is dense enough (i.e., better aligned with the number of dashed boxes compared to FIG. 1H) to allow for enough points of convergence with sufficient angular resolution (i.e., phase modulation frequency), as represented by dashed lines, to satisfy the spatial resolution requirement at the object location 162 dictated by the minimum acuity requirement.

[0180] FIGS. IL- IM illustrates a method for determining the amplitude and phase modulation frequency requirements at the energy modulation surface based on minimum visual acuity requirement for the object location, in accordance with an embodiment. Resolving, from the viewing location 172, the minimum resolution of the modulated energy propagated through the energy modulation surface location 152 located at a distance D2 from the viewing location 172 may be considered as a desired design constraint that would in turn determine the design goals for amplitude and phase modulation frequencies that the configuration of the energy amplitude modulation regions and energy modulation areas would have to allow for. Based on the geometry of DI and D2, the interval 134 of the energy amplitude modulation regions 136 may be designed to allow for an amplitude modulation frequency, at the energy modulation surface 132, that substantially satisfies the minimum resolution, and the interval of the energy modulation areas may be designed to allow for a phase frequency, at the energy modulation surface 132, that substantially satisfies the minimum resolution.

[0181] For example, to resolve, from the viewing location 172, a minimum acuity resolution of 15 CPD at the object location 162 that is 0.5m away, the modulated energy would have to be propagated through the energy modulation areas and the energy amplitude modulation regions of the energy modulation surface located 0.625m away and configured to allow an amplitude frequency of at least 18.75 CPD and a phase frequency of at least 3.75 CPD. Similarly, to resolve a minimum acuity resolution of 30 CPD at the same object location 162 from the same viewing location 172, the modulated energy would have to be propagated through the energy modulation areas and the energy amplitude modulation regions to allow an amplitude frequence of at least 37.5 CPD and a phase frequency of at least 7.5 CPD.

[0182] In an embodiment, the intersection of the angle of view from the viewing location 172 and the field of view from the energy modulation surface location 152 defines dimensions (i.e., width or length) of the object location 162. This, along with the CPD requirement at the object location, may be used to determine the amplitude and phase modulation frequencies and intervals to allow for the minimum CPD requirement.

[0183] Using the aforementioned example, given that an 18.75 CPD amplitude frequency at the energy modulation surface location 152 is desired in order to achieve a minimum visual acuity requirement of 15 CPD at the object location that is 0.5 meter away from the viewing location 172, an interval of an energy amplitude modulation region would be .293mm for a distance of 0.625 meter between the viewing location 172 and the energy modulation surface location 152. Given that a 3.75 CPD phase modulation frequency at the energy modulation surface location 152 is desired, the interval of the energy modulation areas would have to allow for 450 phase samples per energy amplitude sampling region for a 60 degrees field of view. To allow for the desired phase modulation frequency of 450 phase samples per energy amplitude modulation region and a desired amplitude modulation interval of .293mm, the energy modulation areas intervals would have to be around 650nm.

[0184] As illustrated in FIGS. 23A, 23C, and 23D, which will be described in greater details below, an embodiment of the energy modulation system 3000 may include a first energy modulation surface comprising one or more energy modulation areas 3004 and a second energy modulation surface comprising one or more of energy modulation regions 3002. Each one or more energy modulation areas 3004 may include one or more energy modulation locations 3006. Each energy modulation region 3002 may receive energy from a respective energy modulation area3004, such as from the one or more energy locations 3006 contained in the respective energy modulation area 3004. According to an embodiment, each energy modulation area 3004 may be configured to apply an amplitude modulation to the energy passing therethrough and have an area size and an interval therebetween. The energy modulation regions 3002 may be configured to apply a phase modulation to the energy passing therethrough and have an interval therebetween. In an embodiment, each energy modulation region 3002 may operate to apply the phase modulation to the energy propagated through the respective energy modulation area 3004 and / or the energy modulation locations 3006 contained therein.

[0185] In a simplified modeling approach, each modulation region 3002 may be modeled using an ideal optical model, such as a thin lens model, and each of a plurality of energy locations (e.g., 3006a-c) of each modulation area 3004 may be modeled as an ideal point source. In an embodiment, the phase modulation frequency of an energy modulation region is based on the number of energy modulation areas through which energy is propagated therethrough and the amplitude modulation frequency of the energy modulation region is based on the interval of the energy modulation regions at the second surface.

[0186] Even under the simplified ideal optical model in FIG. 23C, the modulation interval of the modulation areas has physical constraints due to pixelated modulation areas, thereby making it difficult to achieve the phase and amplitude modulation frequency requirements of wavefronts for forming a resolvable holographic object at the desired object location.Current Technology Limitations:Active Area, Device Electronics, Packaging, and the Mechanical Envelope

[0187] FIG. 2 illustrates a device 200 having an active area 220 with a certain mechanical form factor. The device 200 may include drivers 230 and electronics 240 for powering and interface to the active area 220, the active area having a dimension as shown by the x and y arrows. This device 200 does not take into account the cabling and mechanical structures to drive, power and cool components, and the mechanical footprint may be further minimized by introducing a flex cable into the device 200. The minimum footprint for such a device 200 may also be referred to as a mechanical envelope 210 having a dimension as shown by the M:x and M:y arrows. This device 200 is for illustration purposes only and custom electronics designs may further decrease the mechanical envelope overhead, but in almost all cases may not be the exact size of the active area of the device. In an embodiment, this device 200 illustrates the dependency of electronics as itrelates to active image area 220 for a micro OLED, DLP chip or LCD panel, or any other technology with the purpose of image illumination.

[0188] In some embodiments, it may also be possible to consider other projection technologies to aggregate multiple images onto a larger overall display. However, this may come at the cost of greater complexity for throw distance, minimum focus, optical quality, uniform field resolution, chromatic aberration, thermal properties, calibration, alignment, additional size or form factor. For most practical applications, hosting tens or hundreds of these projection sources 200 may result in a design that is much larger with less reliability.

[0189] Given the above resolution considerations, a plurality of devices similar to those shown in FIG. 2 may be desired. It should be noted that many devices comprise of various pixel structures that may or may not map to a regular grid. In the event that there are additional sub-pixels or locations within each full pixel, these may be exploited to generate additional resolution or angular density. Additional signal processing may be used to determine how to convert the phase and amplitude modulations into the correct pixel coordinates depending on the specified location of the pixel structure(s) and can be an explicit characteristic of each device that is known and calibrated. Further, other energy domains may involve a different handling of these ratios and device structures, and those skilled in the art will understand the direct intrinsic relationship between each of the desired frequency domains. This will be shown and discussed in more detail in subsequent disclosure.

[0190] The resulting calculation may be used to understand how many of these individual devices may be desired to produce a full resolution energy surface. In this case, a large number of devices may be desired to achieve the visual acuity threshold. In an aspect of the present disclosure, the challenge and novelty exists within the fabrication of a seamless energy surface from these available energy locations for sufficient sensory holographic propagation.Summary of Seamless Energy Surfaces:Configurations and Designs for Arrays of Energy Relays

[0191] In some embodiments, approaches are disclosed to address the challenge of generating high energy location density from an array of individual devices without seams due to the limitation of mechanical structure for the devices. In an embodiment, an energy propagating relay system may allow for an increase the effective size of the active device area to meet or exceed the mechanical dimensions to configure an array of relays and form a singular seamless energy surface.

[0192] FIG. 3 illustrates an embodiment of such an energy relay system 300. As shown, the relay system 300 may include a device 310 mounted to a mechanical envelope 320, with an energy relay element 330 propagating energy from the device 310. The relay element 330 may be configured to provide the ability to mitigate any gaps 340 that may be produced when multiple mechanical envelopes 320 of the device Eire placed into an array of multiple devices 310.

[0193] For example, if a device’s active area 310 is 20mm x 10mm and the mechanical envelope 320 is 40mm x 20mm, an energy relay element 330 may be designed with a magnification of 2:1 to produce a tapered form that is approximately 20mm x 10mm on a minified end (arrow A) and 40mm x 20mm on a magnified end (arrow B), providing the ability to align an array of these elements 330 together seamlessly without altering or colliding with the mechanical envelope 320 of each device 310. Mechanically, the relay elements 330 may be bonded or fused together to align and polish ensuring minimal seam gap 340 between devices 310. In one such embodiment, it is possible to achieve a seam gap 340 smaller than the visual acuity limit of the eye.

[0194] FIG. 4 illustrates an example of a base structure 400 having energy relay elements 410 formed together and securely fastened to an additional mechanical structure 430. The mechanical structure of the seamless energy surface 420 provides the ability to couple multiple energy relay elements 410, 450 in series to the same base structure through bonding or other mechanical processes to mount relay elements 410, 450. In some embodiments, each relay element 410 may be fused, bonded, adhered, pressure fit, aligned or otherwise attached together to form the resultant seamless energy surface 420. In some embodiments, a device 480 may be mounted to the rear of the relay element 410 and aligned passively or actively to ensure appropriate energy location alignment within the determined tolerance is maintained.

[0195] In an embodiment, the seamless energy surface comprises one or more energy locations and one or more energy relay element stacks comprise a first and second side and each energy relay element stack is arranged to form a singular seamless energy surface directing energy along propagation paths extending between one or more energy locations and the seamless energy surface, and where the separation between the edges of any two adjacent second sides of the terminal energy relay elements is less than the minimum perceptible contour as defined by the visual acuity of a human eye having better than 20 / 40 vision at a distance greater than the width of the singular seamless energy surface.

[0196] In an embodiment, each of the seamless energy surfaces comprise one or more energy relay elements each with one or more structures forming a first and second surface with a transverse and longitudinal orientation. The first relay surface has an area different than the second resulting in positive or negative magnification and configured with explicit surface contours for both the first and second surfaces passing energy through the second relay surface to substantially fill a + / - 10 degree angle with respect to the normal of the surface contour across the entire second relay surface.

[0197] In an embodiment, multiple energy domains may be configured within a single, or between multiple energy relays to direct one or more sensory holographic energy propagation paths including visual, acoustic, tactile or other energy domains.

[0198] In an embodiment, the seamless energy surface is configured with energy relays that comprise two or more first sides for each second side to both receive and emit one or more energy domains simultaneously to provide bi-directional energy propagation throughout the system.

[0199] In an embodiment, the energy relays are provided as loose coherent elements.Introduction to Component Engineered Structures: Transverse Energy Localization

[0200] The properties of energy relays may be optimized according to the principles disclosed herein for energy relay elements that induce transverse energy localization. In one embodiment, transverse energy localization may be affected by propagating energy along the longitudinal direction through a relay material that comprises component engineered structures that are transversely disordered but longitudinally consistent.

[0201] The propagation of energy through such relay material may be less impacted by total internal reflection than by multiple-scattering paths where wave interference can completely limit the propagation in the transverse orientation while continuing in the longitudinal orientation.

[0202] This configuration of relay material allows for the elimination of the cladding of traditional multi-core optical fiber materials. The cladding is to functionally eliminate the scatter of energy between fibers, but simultaneously act as a barrier to rays of energy thereby reducing transmission by at least the core to clad ratio (e.g., a core to clad ratio of 70:30 will transmit at best 70% of received energy transmission) and additionally forms a strong pixelated patterning in the propagated energy.

[0203] FIG. 5A illustrates an end view of an example of an energy relay 500 comprising a coreclad structure, wherein an image is relayed through multi-core optical fibers where pixilation and fiber noise may be exhibited due to the intrinsic properties of the optical fibers. With traditional multi-mode and multi-core optical fibers, relayed images may be intrinsically pixelated due to the properties of total internal reflection of the discrete array of cores where any cross-talk between cores will reduce the modulation transfer function and increase blurring. The resulting imagery produced with traditional multi-core optical fiber tends to have a residual fixed noise fiber pattern.

[0204] FIG. 5B illustrates an example of the same relayed image 550 through an energy relay comprising materials that exhibit the properties of transverse energy localization, where the relayed pattern has a greater density grain structures as compared to the fixed fiber pattern from FIG. 5A. In an embodiment, relays comprising randomized microscopic component engineered structures induce transverse energy localization and transport light more efficiently with higher propagation of resolvable resolution than commercially available multi-mode glass optical fibers.

[0205] There is an advantage to the transverse energy localization material properties in terms of both cost and weight, where a similar optical grade glass material, may cost and weigh upwards of 10 to 100-fold more than the cost for the same material generated within an embodiment, wherein disclosed systems and methods comprise randomized microscopic component engineered structures demonstrating significant opportunities to improve both cost and quality over other technologies known in the art.

[0206] While the Anderson localization principle was introduced in the 1950s, it wasn't until recent technological breakthroughs in materials and processes allowed the principle to be explored practically in optical transport. Transverse Anderson localization is the propagation of a wave transported through a transversely disordered but longitudinally invariant material without diffusion of the wave in the transverse plane.

[0207] Transverse Anderson localization has been observed through experimentation in which a fiber optic face plate is fabricated through drawing millions of individual strands of fiber with different refractive index (RI) that were mixed randomly and fused together. When an input beam is scanned across one of the surfaces of the face plate, the output beam on the opposite surface follows the transverse position of the input beam. Since Anderson localization exhibits in disordered mediums an absence of diffusion of waves, some of the fundamental physics are different when compared to optical fiber relays. This implies that the Anderson localizationphenomena in the random mixture of optical fibers with varying RI arises less by total internal reflection than by the randomization between multiple-scattering paths where wave interference can completely limit the propagation in the transverse orientation while continuing in the longitudinal path. Further to this concept, it is introduced herein that a non-random pattern of material wave propagation properties may be used in place of a randomized distribution in the transverse plane of an energy transport device. Such a non-random distribution may induce what is referred to herein as Ordered Energy Localization in a transverse plane of the device. This Ordered Energy Localization reduces the occurrence of localized grouping of similar material properties, which can arise due to the nature of random distributions, but which act to degrade the overall efficacy of energy transport through the device.

[0208] In an embodiment, it may be possible for Ordered Energy Localization materials to transport light with a contrast as high as, or better than, the highest quality commercially available multimode glass image fibers, as measured by an optical modulation transfer function (MTF). With multimode and multicore optical fibers, the relayed images are intrinsically pixelated due to the properties of total internal reflection of the discrete array of cores, where the loss of image transfer in regions between cores will reduce MTF and increase blurring. The resulting imagery produced with multicore optical fiber tends to have a residual fixed noise fiber pattern, as illustrated in FIG. 5A. By contrast, the same relayed image through an example material sample that exhibits Ordered Energy Localization, which is similar to that of the Transverse Anderson Localization principle, where the noise pattern appears much more like a grain structure than a fixed fiber pattern.

[0209] Another advantage to optical relays that exhibit the Ordered Energy localization phenomena is that Ordered Energy Localization materials can be fabricated from a polymer material, resulting in reduced cost and weight. A similar optical-grade material, generally made of glass or other similar materials, may cost more than a hundred times the cost of the same dimension of material generated with polymers. Further, the weight of the polymer relay optics can be 10- 100 times less. For the avoidance of doubt, any material that exhibits the Anderson localization property, or the Ordered Energy Localization property as described herein, may be included in this disclosure, even if it does not meet the above cost and weight suggestions. As one skilled in the art will understand, the above suggestion is a single embodiment that lends itself to significant commercial viabilities that similar glass products exclude. Of additional benefit is that for OrderedEnergy Localization to work, optical fiber cladding may not be needed, which for traditional multicore fiber optics is required to prevent the scatter of light between fibers, but simultaneously blocks a portion of the rays of light and thus reduces transmission by at least the core-to-clad ratio (e.g. a core-to-clad ratio of 70:30 will transmit at best 70% of received illumination). In certain embodiments, relaying energy through all or most of the materials of a relay may improve the efficiency of relaying energy through said material, since the need for extra energy controlling materials may be reduced or eliminated.

[0210] Another benefit is the ability to produce many smaller parts that can be bonded or fused without seams as the polymer material is composed of repeating units, and the merger of any two pieces is nearly the same as generating the component as a singular piece depending on the process to merge the two or more pieces together. For large scale applications, this is a significant benefit for the ability to manufacture without massive infrastructure or tooling costs, and it provides the ability to generate single pieces of material that would otherwise be impossible with other methods. Traditional plastic optical fibers have some of these benefits, but due to the cladding generally still involves a seam line of some distances.

[0211] The present disclosure includes engineered structure exhibiting the Ordered Energy Localization phenomena and the method of manufacturing same. The engineered structure of the present disclosure may be used to construct relays of electromagnetic energy, acoustic energy, or other types of energy using building blocks that may include one or more component engineered structures (“CES”). The term CES refers to a building block component with specific engineered properties (“EP”) that may include, but are not limited to, material type, size, shape, refractive index, center-of-mass, charge, weight, absorption, and magnetic moment, among other properties. The size scale of the CES may be on the order of wavelength of the energy wave being relayed, and can vary across the milli-scale, the micro-scale, or the nano-scale. The other EPs are also highly dependent on the wavelength of the energy wave.

[0212] Within the scope of the present disclosure, a particular arrangement of multiple CES may form a non-random pattern, which may be repeated in the transverse direction across a relay to effectively induce Ordered Energy Localization. A single instance of such a non-random pattern of CES is referred to herein as a module. A module may comprise two or more CES. A grouping of two or more modules within a relay is referred to herein as a structure.

[0213] Ordered Energy Localization is a general wave phenomenon that applies to the transport of electromagnetic waves, acoustic waves, quantum waves, energy waves, among others. The one or more component engineered structures may form an energy wave relay that exhibits Ordered Energy Localization each have a size that is on the order of the corresponding wavelength. Another parameter for the building blocks is the speed of the energy wave in the materials used for those building blocks, which includes refractive index for electromagnetic waves, and acoustic impedance for acoustic waves. For example, the building block sizes and refractive indices can vary to accommodate any frequency in the electromagnetic spectrum, from X-rays to radio waves, or to accommodate acoustic waves ranging from ultra-low frequencies just above 0 Hz to ultrasonic frequencies of approximately 20 MHz.

[0214] For this reason, discussions in this disclosure about optical relays can be generalized to not only the full electromagnetic spectrum, but to acoustical energy and other types of energy. For this reason, the use of the terms energy source, energy surface, and energy relay will be used in the present disclosure, even if an embodiment may be discussed with respect to one particular form of energy such as the visible electromagnetic spectrum. One of ordinary skill in the art would understand the principles of the present disclosure as discussed with respect to one form of energy would apply the same for embodiments implemented for other forms of energy.

[0215] For the avoidance of doubt, the material quantities, process, types, refractive index, and the like are merely exemplary and any optical material that exhibits the Ordered Energy Localization property is included herein. Further, any use of ordered materials and processes is included herein.

[0216] It should be noted that the principles of optical design noted in this disclosure apply generally to all forms of energy relays, and the design implementations chosen for specific products, markets, form factors, mounting, etc. may or may not need to address these geometries but for the purposes of simplicity, any approach disclosed is inclusive of all potential energy relay materials.

[0217] In one embodiment, for the relay of visible electromagnetic energy, the transverse size of the CES should be on the order of 1 micron. The materials used for the CES can be any optical material that exhibits the optical qualities desired to include, but not limited to, glass, plastic, resin, air pockets, and the like. The index of refraction of the materials used are higher than 1, and if two CES types are chosen, the difference in refractive index becomes a key design parameter. The aspect ratio of the material may be chosen to be elongated, in order to assist wave propagation ina longitudinal direction.

[0218] In embodiments, energy from other energy domains may be relayed using one or more CES. For example, acoustic energy or haptic energy, which may be mechanical vibrational forms of energy, may be relayed. Appropriate CES may be chosen based on transport efficiency in these alternate energy domains. For example, air may be selected as a CES material type in relaying acoustic or haptic energy. In embodiments, empty space or a vacuum may be selected as a CES in order to relay certain forms of electromagnetic energy. Furthermore, two different CES may share a common material type, but may differ in another engineered property, such as shape.

[0219] The formation of a CES may be completed as a destructive process that takes formed materials and cuts the pieces into a desired shaped formation or any other method known in the art, or additive, where the CES may be grown, printed, formed, melted, or produced in any other method known in the art. Additive and destructive processes may be combined for further control over fabrication. These CES are constructed to a specified structure size and shape.

[0220] In one embodiment, for electromagnetic energy relays, it may be possible to use optical grade bonding agents, epoxies, or other known optical materials that may start as a liquid and form an optical grade solid structure through various means including but not limited to UV, heat, time, among other processing parameters. In another embodiment, the bonding agent is not cured or is made of index matching oils for flexible applications. Bonding agent may be applied to solid structures and non-curing oils or optical liquids. These materials may exhibit certain refractive index (RI) properties. The bonding agent needs to match the RI of either CES material type 1 or CES material type 2. In one embodiment, the RI of this optical bonding agent is 1.59, the same as PS (polystyrene). In a second embodiment, the RI of this optical bonding agent is 1.49, the same as PMMA (poly methyl methacrylate). In another embodiment, the RI of this optical bonding agent is 1.64, the same as a thermoplastic polyester (TP) material.

[0221] In one embodiment, for energy waves, the bonding agent may be mixed into a blend of CES material type 1 and CES material type 2 in order to effectively cancel out the RI of the material that the bonding agent RI matches. The bonding agent may be thoroughly intermixed, with enough time given to achieve escape of air voids, desired distributions of materials, and development of viscous properties. Additional constant agitation may be implemented to ensure the appropriate mixture of the materials to counteract any separation that may occur due to various densities of materials or other material properties.

[0222] It may be required to perform this process in a vacuum or in a chamber to evacuate any air bubbles that may form. An additional methodology may be to introduce vibration during the curing process.

[0223] An alternate method provides for three or more CES with additional form characteristics and EPs.

[0224] In one embodiment, for electromagnetic energy relays, an additional method provides for only a single CES to be used with only the bonding agent, where the RI of the CES and the bonding agent differ.

[0225] An additional method provides for any number of CESs and includes the intentional introduction of air bubbles.

[0226] In one embodiment, for electromagnetic energy relays, a method provides for multiple bonding agents with independent desired RIs, and a process to intermix the zero, one, or more CESs as they cure either separately or together to allow for the formation of a completely intermixed structure. Two or more separate curing methodologies may be leveraged to allow for the ability to cure and intermix at different intervals with different tooling and procedural methodologies. In one embodiment, a UV cure epoxy with a RI of 1.49 is intermixed with a heat cure second epoxy with a RI of 1.59 where constant agitation of the materials is provisioned with alternating heat and UV treatments with only sufficient duration to begin to see the formation of solid structures from within the larger mixture, but not long enough for any large particles to form, until such time that no agitation can be continued once the curing process has nearly completed, whereupon the curing processes are implemented simultaneously to completely bond the materials together. In a second embodiment, CES with a RI of 1.49 are added. In a third embodiment, CES with both a RI of 1.49 and 1.59 both added.

[0227] In another embodiment, for electromagnetic energy relays, glass and plastic materials are intermixed based upon their respective RI properties.

[0228] In an additional embodiment, the cured mixture is formed in a mold and after curing is cut and polished. In another embodiment, the materials leveraged will re-liquefy with heat and are cured in a first shape and then pulled into a second shape to include, but not limited to, tapers or bends.

[0229] It should be appreciated that there exist a number of well-known conventional methods used to weld polymeric materials together. Many of these techniques are described in ISO 472(“Plastics- Vocabulary”, International Organization for Standardization, Switzerland 1999) which is herein incorporated by reference in its entirety, and which describes processes for uniting softened surfaces of material including thermal, mechanical (e.g. vibration welding, ultrasonic welding, etc.), electromagnetic, and chemical (solvent) welding methods.

[0230] FIG. 5C illustrates a cutaway view in the transverse plane of a portion 900 of a pre-fused energy relay comprising a randomized distribution of particles comprising two component materials, component engineered structure (“CES”) 901 and CES 903. In an embodiment, particles comprising either CES 901 or CES 903 may possess different material properties, such as different refractive indices, and may induce an Anderson Localization effect in energy transported therethrough, localizing energy in the transverse plane of the material. In an embodiment, particles comprising either CES 901 or CES 903 may extend into and out of the plane of the illustration in a longitudinal direction, thereby allowing energy propagation along the longitudinal direction with decreased scattering effects compared to traditional optical fiber energy relays due to the localization of energy in the transverse plane of the material.

[0231] FIG. 5D illustrates a cutaway view in the transverse plane of module 1001 of a pre-fused energy relay comprising a non-random pattern of particles, each particle comprising one of three component materials, CES 1003, CES 1004, or CES 1007. Particles comprising one of CES’s 1003, 1005, or 1007 may possess different material properties, such as different refractive indices, which may induce an energy localization effect in the transverse plane of the module. The pattern of particles comprising one of CES’s 1003, 1005, or 1007 may be contained within a module boundary 1009, which defines the particular pattern that particles comprising one of CES’s 1003, 1005, or 1007 are arranged in. Similar to FIG. 5C, particles comprising one of CES’s 1003, 1005, or 1007 may extend in a longitudinal direction into and out of the plane of the illustration to allow energy propagation along the longitudinal direction with decreased scattering effects compared to traditional optical fiber energy relays due to the localization of energy in the transverse plane of the material.

[0232] Particles comprising one of CES’s 901 or 903 from FIG. 5C and particles comprising one of CES’s 1003, 1005, or 1007 from FIG. 5D may be long, thin rods of respective material which extend in a longitudinal direction normal to the plane of the illustration and are arranged in the particular patterns shown in FIG. 5C and FIG. 5D respectively. Although small gaps may exist between individual particles of CES due to the circular cross-sectional shape of the particlesshown in FIG. 5C and FIG. 5D, these gaps would effectively be eliminated upon fusing, as the CES materials would gain some fluidity during the fusing process and “melt” together to fill in any gaps. While the cross-sectional shapes illustrated in FIG. 5C and FIG. 5D are circular, this should not be considered limiting of the scope of this disclosure, and one skilled in the art should recognize that any shape or geometry of pre-fused material may be utilized in accordance with the principles disclosed herein. For example, in an embodiment, the individual particles of CES have a hexagonal rather than circular cross section, which may allow for smaller gaps between particles prior to fusing.

[0233] FIG. 5E illustrates a cutaway view in the transverse plane of a portion 1100 of a pre-fused energy relay comprising a random distribution of particles comprising component materials CES 1101 and CES 1103. The portion 1100 may have a plurality of sub-portions, such as sub-portions 1106, 1108 and 1110 each comprising a randomized distribution of particles comprising CES 1101 and 1103. The random distribution of particles comprising CES 1101 and CES 1103 may, after fusing of the relay, induce a Transverse Anderson Localization effect in energy relayed in a longitudinal direction extending out of the plane of the illustration through portion 1100.

[0234] FIG. 5H illustrates a cutaway view in the transverse plane of a portion 1301 of a fused energy relay comprising a random distribution of particles comprising component materials CES 1303 and CES 1305. Portion 1301 may represent a possible fused form of portion 1100 from FIG. 11. In the context of the present disclosure, when adjacent particles of similar CES aggregate together upon fusing, this is referred to as an aggregated particle (“AP”). An example of an AP of CES 1303 can be seen at 1309, which may represent the fused form of several unfused CES 1303 particles (shown in FIG. 5H). As illustrated in FIG. 5H, the boundaries between each continuous particle of similar CES, as well as the boundaries between modules with similar CES border particles, are eliminated upon fusing, while new boundaries are formed between APs of different CES.

[0235] According to the Anderson Localization principle, a randomized distribution of materials with different energy wave propagation properties distributed in the transverse direction of a material will localize energy within that direction, inhibiting energy scattering and reducing interference which may degrade the transport efficiency of the material. In the context of transporting electromagnetic energy, for example, through increasing the amount of variance in refractive index in the transverse direction by randomly distributing materials with differingrefractive indices, it becomes possible to localize the electromagnetic energy in the transverse direction.

[0236] However, as discussed previously, due to the nature of randomized distributions, there exists the possibility that undesirable arrangements of materials may inadvertently form, which may limit the realization of energy localization effects within the material. For example, AP 1307 of FIG. 5H could potentially form after fusing the randomized distribution of particles shown in the corresponding location in FIG. 5E. When designing a material for transporting electromagnetic energy, for example, a design consideration is the transverse size of pre-fused particles of CES. In order to prevent energy from scattering in the transverse direction, one may select a particle size such that upon fusing, the resultant average AP size is substantially on the order of the wavelength of the electromagnetic energy the material is intended to transport. However, while the average AP size can be designed for, one skilled in the art would recognize that a random distribution of particles will result in a variety of unpredictable sizes of AP, some being smaller than the intended wavelength and some being larger than the intended wavelength.

[0237] In FIG. 5H, AP 1307 extends across the entire length of portion 1301 and represents an AP of a size much larger than average. This may imply that the size of AP 1307 is also much larger than the wavelength of energy that portion 1301 is intended to transport in the longitudinal direction. Consequently, energy propagation through AP 1307 in the longitudinal direction may experience scattering effects in the transverse plane, reducing the Anderson Localization effect and resulting in interference patterns within energy propagating through AP 1307 and a reduction in the overall energy transport efficiency of portion 1301.

[0238] It should be understood that, according to the principles disclosed herein and due to the nature of randomized distributions, a sub-portion within portion 1100, such as sub-portion 1108 for example, may be of arbitrary significance, since there is no defined distribution pattern. However, it should be apparent to one skilled in the art that in a given randomized distribution, there exists the possibility that one may identify distinct sub-portions that comprise the same or substantially similar patterns of distribution. This occurrence may not significantly inhibit the overall induced Transverse Anderson Localization effect, and the random patterns described herein should not be seen as limited to exclude such cases.

[0122] The non-random, Ordered Energy Localization pattern design considerations disclosed herein represent an alternative to a randomized distribution of component materials, allowingenergy relay materials to exhibit energy localization effects in the transverse direction while avoiding the potentially limiting deviant cases inherent to randomized distributions. FIG. 5F illustrates a cutaway view in the transverse plane of a portion 1203 of a pre-fused energy relay comprising a non-random pattern (a distribution configured to relay energy via Ordered Energy Localization) of three component materials CES 1205, CES 1207, or CES 1209, which define multiple modules with similar orientations. Particles of these three CES materials are arranged in repeating modules, such as module 1211 and module 1213, which share substantially invariant distributions of said particles. While portion 1203 contains six modules as illustrated in FIG. 5F, the number of modules in a given energy relay can be any number and may be chosen based on the desired design parameters. Additionally, the size of the modules, the number of particles per module, the size of the individual particles within a module, the distribution pattern of particles within a module, the number of different types of modules, and the inclusion of extra-modular or interstitial materials may all be design parameters to be given consideration and fall within the scope of the present disclosure.

[0123] Similarly, the number of different CESs included within each module need not be three as illustrated in FIG. 5F but may preferably be any number suited to the desired design parameters. Furthermore, the different characteristic properties possessed by each CES may be variable in order to satisfy the desired design parameters, and differences should not be limited only to refractive index. For example, two different CESs may possess substantially the same refractive index but may differ in their melting point temperatures.

[0124] In order to minimize the scattering of energy transported through the portion 1203 of the energy relay illustrated in FIG. 5F, and to promote transverse energy localization, the non-random pattern of the modules that comprise portion 1203 may satisfy the Ordered Energy Localization distribution characteristics described above. In the context of the present disclosure, contiguous particles may be particles that are substantially adjacent to one another in the transverse plane. The particles may be illustrated to be touching one another, or there may be an empty space illustrated between the adjacent particles. One skilled in the art will appreciate that small gaps between adjacent illustrated particles are either inadvertent artistic artifacts or are meant to illustrate the minute mechanical variations which can arise in real- world arrangement of materials. Furthermore, this disclosure also includes arrangements of CES particles in substantially non-random paterns, but contain exceptions due to manufacturing variations or intentional variation by design.

[0125] Ordered Energy Localization paterns of CES particles may allow for greater localization of energy, reduce scatering of energy in a transverse direction through a relay material, and consequently allow for higher efficiency of energy transport through the material relative to other embodiments. FIG. 5G illustrates a cutaway view in the transverse plane of a portion 1250 of a pre-fused energy relay comprising a non-random patern of particles of three component materials, CES 1205, CES 1207, and CES 1209, wherein the particles define multiple modules with varying orientations. Modules 1258 and 1260 of portion 1250 comprise a non-random patern of materials similar to that of modules 1211 and 1213 of FIG. 5F. However, the patern of materials in module 1260 is rotated relative to that of module 1258. Several other modules of portion 1250 also exhibit a rotated patern of distribution. It is important to note that despite this rotational arrangement, each module within portion 1250 possesses the Ordered Energy Localization distribution described above, since the actual patern of particle distribution within each module remains the same regardless of how much rotation is imposed upon it.

[0126] FIG. 51 illustrates a cutaway view in the transverse plane of a portion 1401 of a fused energy relay comprising a non-random patern of particles of three component materials, CES 1403, CES 1405, and CES 1407. Portion 1401 may represent a possible fused form of portion 1203 from FIG. 5F. By arranging CES particles in an Ordered Energy Localization distribution, the relay shown in FIG. 51 may realize more efficient transportation of energy in a longitudinal direction through the relay relative to the randomized distribution shown in FIG. 5H. By selecting CES particles with a diameter roughly Vi of the wavelength of energy to be transported through the material and arranging them in a pre-fuse Ordered Energy Localization distribution shown in FIG. 5F, the size of the resultant APs after fusing seen in FIG. 51 may have a transverse dimension between !4 and 2 times the wavelength of intended energy. By substantially limiting transverse AP dimensions to within this range, energy transported in a longitudinal direction through the material may allow for Ordered Energy Localization and reduce scatering and interference effects. In an embodiment, a transverse dimension of AP’s in a relay material may preferably be between !4 and 8 times the wavelength of energy intended to be transported in a longitudinal direction through the APs.

[0127] As seen in FIG. 51, and in contrast with FIG. 5H, there is notable consistency of size acrossall APs, which may result from exerting control over how pre-fused CES particles are arranged. Specifically, controlling the pattern of particle arrangement may reduce or eliminate the formation of larger APs with larger energy scattering and interference patterns, representing an improvement over randomized distributions of CES particles in energy relays.

[0128] It should be noted that across different fields and throughout many disciplines, the concept of “randomness,” and indeed the notions of what is and is not random are not always clear. There are several important points to consider in the context of the present disclosure when discussing random and non-random patterns, arrangements, distributions, et cetera, which are discussed below. However, it should be appreciated that the disclosures herein are by no means the only way to conceptualize and / or systematize the concepts of randomness or non-randomness. Many alternate and equally valid conceptualizations exist, and the scope of the present disclosure should not be seen as limited to exclude any approach contemplated by one skilled in the art in the present context.

[0129] Complete spatial randomness (“CSR”), which is well-known in the art and is described in Smith, T.E., (2016) Notebook on Spatial Data Analysis [online](http: / / www.seas.upenn.edu / ~ese502 / #notebook), which is herein incorporated by reference, is a concept used to describe a distribution of points within a space (in this case, within a 2D plane) which are located in a completely random fashion. There are two common characteristics used to describe CSR: The spatial Laplace principle, and the assumption of statistical independence.

[0130] The spatial Laplace principle, which is an application of the more general Laplace principle to the domain of spatial probability, essentially states that, unless there is information to indicate otherwise, the chance of a particular event, which may be thought of as the chance of a point being located in a particular location, is equally as likely for each location within a space. That is to say, each location within a region has an equal likelihood of containing a point, and therefore, the probability of finding a point is the same across each location within the region. A further implication of this is that the probability of finding a point within a particular sub-region is proportional to the ratio of the area of that sub-region to the area of the entire reference region.

[0131] A second characteristic of CSR is the assumption of spatial independence. This principle assumes that the locations of other data points within a region have no influence or effect on the probability of finding a data point at a particular location. In other words, the data points areassumed to be independent of one another, and the state of the “surrounding areas”, so to speak, do not affect the probability of finding a data point at a location within a reference region.

[0132] The concept of CSR is useful as a contrasting example of a non-random pattern of materials, such as some embodiments of CES materials described herein. An Anderson material is described elsewhere in this disclosure as being a random distribution of energy propagation materials in a transverse plane of an energy relay. Keeping in mind the CSR characteristics described above, it is possible to apply these concepts to some of the embodiments of the Anderson materials described herein in order to determine whether the “randomness” of those Anderson material distributions complies with CSR. Assuming embodiments of an energy relay comprising first and second materials, since a CES of either the first or second material may occupy roughly the same area in the transverse plane of the embodiments (meaning they are roughly the same size in the transverse dimension), and further since the first and second CES may be assumed to be provided in equal amounts in the embodiments, we can assume that for any particular location along the transverse plane of the energy relay embodiments, there is an equally likely chance of there being either a first CES or a second CES, in accordance with spatial Laplace principle as applied in this context. Alternatively, if the relay materials are provided in differing amounts in other energy relay embodiments, or possess a differing transverse size from one another, we would likewise expect that the probability of finding either material be in proportion to the ratio of materials provided or to their relative sizes, in keeping with the spatial Laplace principle.

[0133] Next, because both the first and second materials of Anderson energy relay embodiments are arranged in a random manner (either by thorough mechanical mixing, or other means), and further evidenced by the fact that the “arrangement” of the materials may occur simultaneously and arise spontaneously as they are randomized, we can assert that the identities of neighboring CES materials will have substantially no effect on the identity of a particular CES material, and vice versa, for these embodiments. That is, the identities of CES materials within these embodiments are independent of one another. Therefore, the Anderson material embodiments described herein may be said to satisfy the described CSR characteristics. Of course, as discussed above, the nature of external factors and “real-world” confounding factors may affect the compliance of embodiments of Anderson energy relay materials with strict CSR definitions, but one of ordinary skill in the art would appreciate that these Anderson material embodiments substantially fall within reasonable tolerance of such definitions.

[0134] By contrast, an analysis of some of the Ordered Energy Localization relay material embodiments as disclosed herein highlights particular departures from their counterpart Anderson material embodiments (and from CSR). Unlike an Anderson material, a CES material identity within an Ordered Energy Localization relay embodiment may be highly correlated with the identities of its neighbors. The very pattern of the arrangement of CES materials within certain Ordered Energy Localization relay embodiments is designed to, among other things, influence how similar materials are arranged spatially relative to one another in order to control the effective size of the APs formed by such materials upon fusing. In other words, one of the goals of some embodiments which arrange materials in an Ordered Energy Localization distribution is to affect the ultimate cross-sectional area (or size), in the transverse dimension, of any region comprising a single material (an AP). This may limit the effects of transverse energy scattering and interference within said regions as energy is relayed along a longitudinal direction. Therefore, some degree of specificity and / or selectivity is exercised when energy relay materials are first “arranged” in an Ordered Energy Localization distribution embodiment, which may disallow for a particular CES identity to be “independent” of the identity of other CES, particularly those materials immediately surrounding it. On the contrary, in certain embodiments materials are specifically chosen according to a non-random pattern, with the identity of any one particular CES being determined based on a continuation of the pattern and in knowing what portion of the pattern (and thus, what materials) are already arranged. It follows that these certain Ordered Energy Localization distribution energy relay embodiments cannot comply with CSR criteria. Thus, the pattern or arrangement of two or more CES or energy relay materials may be described in the present disclosure as “non-random” or “substantially non-random, “and one of ordinary skill in the art should appreciate that the general concept or characteristics of CSR as describe above may be considered, among other things, to distinguish non-random or substantially non-random pattern from random pattern. For example, in an embodiment, materials that do not substantially comply with the general concept or characteristics of CSR as described, may be considered an Ordered Energy Localization material distribution. In this disclosure, the term ‘ordered’ may be recited to describe a distribution of component engineered structure materials for relays that transmits energy through the principle of Ordered Energy Localization. The term ‘ordered energy relay’, ‘ordered relay’, ‘ordered distribution’, ‘non-random pattern’, etc., describe an energy relay inwhich energy is transmited at least partially through this same principle of Ordered Energy Localization described herein.

[0135] Of course, the CSR concept is provided herein as an example guideline to consider, and one of ordinary skill in the art may consider other principles known in the art to distinguish nonrandom paterns from random paterns. For example, it is to be appreciated that, like a human signature, a non-random patern may be considered as a non-random signal that includes noise. Non-random paterns may be substantially the same even when they are not identical due to the inclusion of noise. A plethora of conventional techniques exist in the art of patern recognition and comparison that may be used to separate noise and non-random signals and correlate the later. By way of example, U.S. Patent No. 7,016,516 to Rhoades, which is incorporated by reference herein, describes a method of identifying randomness (noise, smoothness, snowiness, etc.), and correlating non-random signals to determine whether signatures are authentic. Rhodes notes that computation of a signal's randomness is well understood by artisans in this field, and one example technique is to take the derivative of the signal at each sample point, square these values, and then sum over the entire signal. Rhodes further notes that a variety of other well-known techniques can alternatively be used. Conventional pattern recognition filters and algorithms may be used to identify the same non-random paterns. Examples are provided in U.S. Patent Nos. 5,465,308 and 7,054,850, all of which are incorporated by reference herein. Other techniques of patern recognition and comparison will not be repeated here, but it is to be appreciated that one of ordinary skill in the art would easily apply existing techniques to determine whether an energy relay comprises a plurality of repeating modules each comprising at least first and second materials being arranged in a substantially non-random patern, are in fact comprising the same substantially non-random patern.

[0136] Several different geometries for CES particles and material pre-forms have been illustrated thus far. One aspect of the present disclosure is that any arrangement or geometry of materials may be leveraged, so long as they comprise a non-random patern as previously discussed. However, the pre-fused relay material geometry may have a significant impact on the efficiency of the localization and energy propagation properties of the materials. In an embodiment, certain geometries, known as convex uniform tilings, may provide advantageous distributions of relay materials by arranging the materials in efficient configurations.

[0137] In general, a tiling or tessellation is an arrangement of geometric shapes where there issubstantially no overlap between the shapes and there are no gaps between the shapes. A tessellation can arranged on a 2-dimensional surface using planar shapes, or in 3-dimensions using volumetric structures. Furthermore, there exist subtypes within the domain of tiling. A regular tiling, for example, is a tessellation wherein each tile is the same shape. There are many nonregular tilings comprising a set of two or more shapes configured to tessellate with one another according. There are also non-periodic tilings which have no repeating pattern, as well as aperiodic tilings which use a set of repeating tile shapes that cannot form a repeating pattern, such as a Penrose tiling. All subtypes of tiling fall within the scope of the present disclosure. The shapes of the tiles, in two-dimensional embodiments, may be polygonal, convex, concave, curved, irregular, etc. Additionally, it should be apparent to one of ordinary skill in the art that while the definition of a tiling precludes there being gaps or space between tiles, there are real-world circumstances that sometimes cause deviation from strict definition, and that the existence of minor gaps or spaces between particular tiles should not be seen as a departure from a particular tiling or tessellation pattern.Considerations for 4D Plenoptic Functions:Selective Propagation of Energy through Holographic Waveguide Arrays

[0138] At an overarching level, a challenge of realizing a holographic display is the ability to produce an arbitrary energy wave for propagation through an energy field. One mathematical technique for describing the propagation of energy waves through an energy field is the angular spectrum method; the treatment of this is well-documented, so only a brief overview of the treatment is presented here. Considering a monochromatic field of angular frequency co, the complex field can be represented by Ewhere r is a spatial coordinate (e.g. x, y, z) and t is a time coordinate. The angular spectrum method considers the field E at a point along an axis z and in a plane x, y perpendicular to the chosen axis. In this plane, the two- dimensional Fourier transform of the complex field E(r) = E(x,y,z) is given bywhere x, y are the Cartesian transverse coordinates and kx, kyare their reciprocal coordinates in spatial frequency space. Assuming propagation through a homogenous, isotropic, linear and source-free medium, the field satisfies the vector Helmholtz equation (V2+ k2)E(f) = 0 . Accordingly, the Fourier spectrum of E in an arbitrary image plane located at constant z iscalculated by multiplying the Fourier spectrum in the object plane at z = 0, E(kx, ky; 0), by a propagating factor elkzZ. As a result, the angular spectrum representation of the field E(f) at an arbitrary position along the axis z is given by

[0139] The angular spectrum method effectively expands a complex wave field and is a technique for modeling the propagation of a wave field that expands a complex wave field into a summation of an infinite number of plane waves of the same frequency and different respective directions. The Fourier transform of a wavefront in a region on a certain plane in 3D space derives the angular spectrum. The components of the angular spectrum in spatial frequency space correspond to the respective directions of wave propagation of constituent plane waves forming the complex wave field. As discussed earlier, a light ray may be considered to be equivalent to a normal vector wavefront in an isotropic medium. Accordingly, the angular spectrum sets out the respective directions of plane waves, each of which can be interpreted as a respective light ray that passes through the same region in a direction of propagation of the wavefront. Thus, the intensity of the components of the angular spectrum may be considered to be the intensity of corresponding light rays.

[0140] It will be understood that, under this principle, an arbitrary wave field can be approximately represented, or synthesized, by a plurality of geometric rays in accordance with the above treatment. The complete set of rays passing through any point in space is described by a vector function know as a light field, or the plenoptic function, which will be familiar to one of skill in the art. In the limit of defining rays at an infinite number of positions, the plenoptic function is equivalent to a holographic representation of a space. As will be known to those skilled in the art, the plenoptic function be a 4D, 5D, or higher dimensional function. One particular 4D parameterization of the light field is the two-plane parametrization, in which rays are defined as an oriented line connecting a position on a first plane (u, v) to a second plane (5, f). An equivalent parameterization is the angular parameterization, in which rays are defined by a position on the u, v plane and by angles 0, <|> which can be thought of as azimuthal and zenith angles respectively.

[0141] The ability to produce a light field of arbitrary fidelity proves challenging. In particular, light field representations may struggle to reproduce arbitrarily small curvature of the wavefront, and also may not capture sufficient relative phase information between rays propagating in different directions. One approach to remedy the phase information is through phase-addedstereograms, which uses optical path length information to add phase information to the rays. However, producing arbitrarily small rays to reproduce the curvature of the wavefront runs up against limitations from diffraction, which produces broadening.

[0142] The conversion between a light field to a wavefront may be realized by sampling the light field throughout a sampling plane, and Fourier transforming the image of rays that path through a point in the sampling plane. Since phase information is not typically available in such a sampling, the phase of each ray may be assigned using techniques such as phase added stereogram or random / pseudo-random sequences. The rays may also be obtained by inverse conversion from wavefronts to rays, and the phase information in the angular spectrum can be retained. The Fourier spectrum obtained for all the sampling locations in the sampling plane can be aggregated together to obtain the wavefront on the sampling plane.

[0143] Fresnel and liquid lens provide examples of equivalence between phase and optics with diffractive and / or refractive computations. Fresnel zone plates focus incident light on a diffraction basis that is contributed by annual zones. Two types of Fresnel zone plates are commonly developed: amplitude-type and phase-type. Phase-type Fresnel zone plates (or phase Fresnel lenses) feature zones with phase modulation that form a lens-like wavefront. The radius of the m- th zone (rm) iswhere X is the wavelength and f is the effective focal length of the phase Fresnel lenses. Within a radius from rmto rm+l, the adjusted range of the optical phase is 2TL By combining refraction and diffraction, the refractive-diffractive-type phase Fresnel lens (the so called kinoform lens or blazed-phase Fresnel zone plate) could help enhance the focusing properties of LC lenses. To achieve precise phase modulation within a zone, it is possible to further divided the zone into subzones and used several levels of phase control to mimic the continuous phase profile of a lens. The radius of a kinoform lens can be expressed as:where 1 is the subzone index and L is the total number of phase levels within each zone (e.g. L = 8 in FIG. 4(c)). From the viewpoint of the lens profile, a higher number of phase levels provides a more precise phase profile. This leads to a better diffraction efficiency (r|). The diffraction efficiency isT] = sin c2(1 / L)

[0144] The diffraction efficiencies are 40.5%, 81.1%, and 95.0% for L = 2, 4, and 8, respectively. When L = 2, the phase Fresnel lenses become the typical binary-phase zone plate.

[0145] As discussed above and herein throughout, an energy modulation system may be implemented as a light field display system that generally includes an energy source (e.g., illumination source) and a seamless energy surface configured with sufficient energy location density to satisfy the holographic resolution requirements discussed above. A plurality of relay elements may be used to relay energy from the energy devices to the seamless energy surface. Once energy has been delivered to the seamless energy surface with the requisite energy location density, the energy can be propagated in accordance with a complex amplitude function, which in an embodiment, can be parameterized as a 4D plenoptic function through a disclosed energy waveguide system. As will be appreciated by one of ordinary skill in the art, a 4D plenoptic function is well known in the art and will not be elaborated further herein.

[0146] The energy waveguide system selectively propagates energy through a plurality of energy locations along the seamless energy surface representing the spatial coordinate of the 4D plenoptic function with a structure configured to alter an angular direction of the energy waves passing through representing the angular component of the 4D plenoptic function, wherein the energy waves propagated may converge in space in accordance with a plurality of propagation paths directed by the 4D plenoptic function.

[0147] Reference is now made to FIG. 6 illustrating an example of light field energy surface in 4D image space in accordance with a 4D plenoptic function. The figure shows ray traces of an energy surface 600 to a viewer 620 in describing how the rays of energy converge in space 630 from various positions within the viewing volume. As shown, each waveguide element 610 defines four dimensions of information describing energy propagation 640 through the energy surface 600. Two spatial dimensions (herein referred to as x and y) are the physical plurality of energy locations that can be viewed in image space, and the angular components theta and phi (herein referred to as u and v), which is viewed in virtual space when projected through the energy waveguide array. In general, and in accordance with a 4D plenoptic function, the plurality of waveguides (e.g., lenslets) are able to direct an energy location from the x, y dimension to a unique location in virtual space, along a direction defined by the u, v angular component, in forming the holographic or light field system described herein.

[0148] However, one skilled in the art will understand that a significant challenge to light field and holographic display technologies arises from uncontrolled propagation of energy due designs that have not accurately accounted for any of diffraction, scatter, diffusion, angular direction, calibration, focus, collimation, curvature, uniformity, element cross-talk, as well as a multitude of other parameters that contribute to decreased effective resolution as well as an inability to accurately converge energy with sufficient fidelity.

[0149] In an embodiment, an approach to selective energy propagation for addressing challenges associated with holographic display may include energy inhibiting elements and substantially filling waveguide apertures with near-collimated energy into an environment defined by a 4D plenoptic function.

[0150] In an embodiment, an array of energy waveguides may define a plurality of energy propagation paths for each waveguide element configured to extend through and substantially fill the waveguide element’s effective aperture in unique directions defined by a prescribed 4D function to a plurality of energy locations along a seamless energy surface inhibited by one or more elements positioned to limit propagation of each energy location to only pass through a single waveguide element.

[0151] In an embodiment, multiple energy domains may be configured within a single, or between multiple energy waveguides to direct one or more sensory holographic energy propagations including visual, acoustic, tactile or other energy domains.

[0152] In an embodiment, the energy waveguides and seamless energy surface are configured to both receive and emit one or more energy domains to provide bi-directional energy propagation throughout the system.

[0153] In an embodiment, the energy waveguides are configured to propagate non-linear or nonregular distributions of energy, including non-transmitting void regions, leveraging digitally encoded, diffractive, refractive, reflective, grin, holographic, Fresnel, or the like waveguide configurations for any seamless energy surface orientation including wall, table, floor, ceiling, room, or other geometry based environments. In an additional embodiment, an energy waveguide element may be configured to produce various geometries that provide any surface profile and / or tabletop viewing allowing users to view holographic imagery from all around the energy surface in a 360-degree configuration.

[0154] In an embodiment, the energy waveguide array elements may be reflective surfaces and the arrangement of the elements may be hexagonal, square, irregular, semi-regular, curved, non- planar, spherical, cylindrical, tilted regular, tilted irregular, spatially varying and / or multi-layered.

[0155] For any component within the seamless energy surface, waveguide, or relay components may include, but not limited to, optical fiber, silicon, glass, polymer, optical relays, diffractive, holographic, refractive, or reflective elements, optical face plates, energy combiners, beam splitters, prisms, polarization elements, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar materials exhibiting Anderson localization or total internal reflection.Realizing the Holodeck:Aggregation of Bi-directional Seamless Energy Surface Systems To Stimulate Human Sensory Receptors Within Holographic Environments

[0156] It is possible to construct large-scale environments of seamless energy surface systems by tiling, fusing, bonding, attaching, and / or stitching multiple seamless energy surfaces together forming arbitrary sizes, shapes, contours or form-factors including entire rooms. Each energy surface system may comprise an assembly having a base structure, energy surface, relays, waveguide, devices, and electronics, collectively configured for bi-directional holographic energy propagation, emission, reflection, or sensing.

[0157] In an embodiment, an environment of tiled seamless energy systems is aggregated to form large seamless planar or curved walls including installations comprising up to all surfaces in a given environment, and configured as any combination of seamless, discontinuous planar, faceted, curved, cylindrical, spherical, geometric, or non-regular geometries.

[0158] In an embodiment, aggregated tiles of planar surfaces form wall-sized systems for theatrical or venue-based holographic entertainment. In an embodiment, aggregated tiles of planar surfaces cover a room with four to six walls including both ceiling and floor for cave-based holographic installations. In an embodiment, aggregated tiles of curved surfaces produce a cylindrical seamless environment for immersive holographic installations. In an embodiment, aggregated tiles of seamless spherical surfaces form a holographic dome for immersive Holodeckbased experiences.

[0159] In an embodiment, aggregates tiles of seamless curved energy waveguides provide mechanical edges following the precise pattern along the boundary of energy inhibiting elements within the energy waveguide structure to bond, align, or fuse the adjacent tiled mechanical edges of the adjacent waveguide surfaces, resulting in a modular and seamless energy waveguide system.

[0160] In a further embodiment of an aggregated tiled environment, energy is propagated bidirectionally for multiple simultaneous energy domains. In an additional embodiment, the energy surface provides the ability to both display and capture simultaneously from the same energy surface with waveguides designed such that light field data may be projected by an illumination source through the waveguide and simultaneously received through the same energy surface. In an additional embodiment, additional depth sensing and active scanning technologies may be leveraged to allow for the interaction between the energy propagation and the viewer in correct world coordinates. In an additional embodiment, the energy surface and waveguide are operable to emit, reflect or converge frequencies to induce tactile sensation or volumetric haptic feedback. In some embodiments, any combination of bi-directional energy propagation and aggregated surfaces are possible.

[0161] In an embodiment, the system comprises an energy waveguide capable of bi-directional emission and sensing of energy through the energy surface with one or more energy devices independently paired with two-or-more-path energy combiners to pair at least two energy devices to the same portion of the seamless energy surface, or one or more energy devices are secured behind the energy surface, proximate to an additional component secured to the base structure, or to a location in front and outside of the FOV of the waveguide for off-axis direct or reflective projection or sensing, and the resulting energy surface provides for bi-directional transmission of energy allowing the waveguide to converge energy, a first device to emit energy and a second device to sense energy, and where the information is processed to perform computer vision related tasks including, but not limited to, 4D plenoptic eye and retinal tracking or sensing of interference within propagated energy patterns, depth estimation, proximity, motion tracking, image, color, or sound formation, or other energy frequency analysis. In an additional embodiment, the tracked positions actively calculate and modify positions of energy based upon the interference between the bi-directional captured data and projection information.

[0162] In some embodiments, a plurality of combinations of three energy devices comprising an ultrasonic sensor, a visible energy display, and an ultrasonic emitting device are configuredtogether for each of three first relay surfaces propagating energy combined into a single second energy relay surface with each of the three first surfaces comprising engineered properties specific to each device’s energy domain, and two engineered waveguide elements configured for ultrasonic and energy respectively to provide the ability to direct and converge each device’s energy independently and substantially unaffected by the other waveguide elements that are configured for a separate energy domain.

[0163] In some embodiments, disclosed is a calibration procedure to enable efficient manufacturing to remove system artifacts and produce a geometric mapping of the resultant energy surface for use with encoding / decoding technologies as well as dedicated integrated systems for the conversion of data into calibrated information appropriate for energy propagation based upon the calibrated configuration files.

[0164] In some embodiments, additional energy waveguides in series and one or more energy devices may be integrated into a system to produce opaque holographic pixels.

[0165] In some embodiments, additional waveguide elements may be integrated comprising energy inhibiting elements, beam-splitters, prisms, active parallax barriers or polarization technologies in order to provide spatial and / or angular resolutions greater than the diameter of the waveguide or for other super-resolution purposes.

[0166] In some embodiments, the disclosed energy system may also be configured as a wearable bi-directional device, such as virtual reality (VR) or augmented reality (AR). In other embodiments, the energy system may include adjustment optical element(s) that cause the displayed or received energy to be focused proximate to a determined plane in space for a viewer. In some embodiments, the waveguide array may be incorporated to holographic head-mounted-display. In other embodiments, the system may include multiple optical paths to allow for the viewer to see both the energy system and a real-world environment (e.g., transparent holographic display). In these instances, the system may be presented as near field in addition to other methods.

[0167] In some embodiments, the transmission of data comprises encoding processes with selectable or variable compression ratios that receive an arbitrary dataset of information and metadata; analyze said dataset and receive or assign material properties, vectors, surface IDs, new pixel data forming a more sparse dataset, and wherein the received data may comprise: 2D, stereoscopic, multi-view, metadata, light field, holographic, geometry, vectors or vectorized metadata, and an encoder / decoder may provide the ability to convert the data in real-time or off-line comprising image processing for: 2D; 2D plus depth, metadata or other vectorized information; stereoscopic, stereoscopic plus depth, metadata or other vectorized information; multi- view; multiview plus depth, metadata or other vectorized information; holographic; or light field content; through depth estimation algorithms, with or without depth metadata; and an inverse ray tracing methodology appropriately maps the resulting converted data produced by inverse ray tracing from the various 2D, stereoscopic, multi-view, volumetric, light field or holographic data into real world coordinates through a characterized 4D plenoptic function. In these embodiments, the total data transmission desired may be multiple orders of magnitudes less transmitted information than the raw light field dataset.Energy Directing Devices Suitable for Presenting Holographic Sensory Data

[0168] In an embodiment, the optomechanical display device may be capable of emitting and guiding light to form 2D, stereoscopic, multiview, plenoptic, 4D, volumetric, light field, holographic, or any other visual representation of light.

[0169] FIG. 7A is an example of a light field optomechanical system if configured with emissive display devices 77, optical relays 81, and a waveguide 74 that is realized as an array of refractive elements such as a micro lens array, where a visible image from one or more displays 77 may be optically relayed before being transmitted to the energy surface 75, where the array of refractive elements provides a mapping between each location on the energy surface 75 and the direction of projection of the light from that location, such that a 4D volumetric light field image may be projected.

[0170] In an embodiment, the waveguide may be operable to converge rays of light to induce both vergence and accommodation from an observer point of view.

[0171] In an embodiment, the waveguides and energy relays may be formed or polished with various surface geometries. In an embodiment, the energy relays include elements that induce transverse energy localization. In an embodiment, the energy relays are bidirectional and may both emit and / or project energy.

[0172] In one embodiment, an energy system includes a plurality of energy devices configured to direct energy according to a complex amplitude function, which in some embodiments, may be parameterized as a four-dimensional (4D) plenoptic function. In some embodiments, the pluralityof energy devices include illumination sources emitting image information, where the image information includes emissive, projection, or reflective display technologies, leveraging visible, IR, UV, coherent, laser, infrared, polarized or any other electromagnetic illumination source. In other embodiments, the plurality of energy devices include mechanical energy emitting devices configured to provide immersive audio or volumetric tactile sensation from an acoustic field.

[0173] In some embodiments, the energy system as configured above may further include a base structure (e.g., 72) such that the plurality of energy devices, the energy relay system, and the energy waveguide system may all be coupled to the base structure. In other embodiments, the plurality of energy devices, the energy relay system and the energy waveguide system may be coupled to the base structure with one or more mounting brackets 76.

[0174] In some embodiments, the plurality of energy devices include energy devices for capturing or sensing energy, including mechanical, chemical, transfer, thermal, electric, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In other embodiments, the plurality of energy devices include energy devices for propagating or emitting energy, including mechanical, chemical, transfer, thermal, electric, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In yet other embodiments, the plurality of energy devices include acoustic receiving devices configured to provide sensory feedback or audible controls

[0175] In one embodiment, the energy system further includes an energy relay system (e.g., 6110 as best shown in FIG. 7B) having one or more energy relay elements, where each of the one or more energy relay elements includes a first surface and a second surface, the second surface of the one or more energy relay elements being arranged to form a singular seamless energy surface of the energy relay system, and where a first plurality of energy propagation paths extend from the energy locations in the plurality of energy devices through the singular seamless energy surface of the energy relay system. This will be discussed in more detail below.

[0176] Reference is now made to FIG. 7B illustrating an energy relay system 6110, in an orthogonal view in accordance with one embodiment of the present disclosure. In one embodiment, the energy relay system 6110 may include two or more relay elements 6112, each relay element 6112 formed of one or more structures, each relay element 6112 having a first surface 6114, a second surface 6116, a transverse orientation (generally parallel to the surfaces 6114, 6116) and a longitudinal orientation (generally perpendicular to the surfaces 6114, 6116). In one embodiment,the surface area of the first surface 6114 may be different than the surface area of the second surface 6116. For example, the surface area of the first surface 6114 may be greater or lesser than the surface area of the second surface 6116. In another embodiment, the surface area of the first surface 114 may be the same as the surface area of the second surface 6116. Energy waves can pass from the first surface 6114 to the second surface 6116, or vice versa.

[0177] In one embodiment, the relay element 6112 of the energy relay system 6110 includes a sloped profile portion 6118 between the first surface 6114 and the second surface 6116. In operation, energy waves propagating between the first surface 6114 and the second surface 6116 may have a higher transport efficiency in the longitudinal orientation than in the transverse orientation, and energy waves passing through the relay element 6112 may result in spatial magnification or spatial de-magnification. In other words, energy waves passing through the relay element 6112 of the relay element device 6110 may experience increased magnification or decreased magnification. In some embodiments, the one or more structures for forming the energy relay element 6110 may include glass, carbon, optical fiber, optical film, plastic, polymer, or mixtures thereof.

[0178] In one embodiment, the energy waves passing through the first surface 6114 has a first resolution, while the energy waves passing through the second surface 6116 has a second resolution, and the second resolution is no less than about 50 % of the first resolution. In another embodiment, the energy waves, while having a uniform profile when presented to the first surface, may pass through the second surface radiating in every direction with an energy density in the forward direction that substantially fills a cone with an opening angle of + / - 10 degrees relative to the normal to the second surface, irrespective of location on the second relay surface.

[0179] In some embodiments, the first surface 6114 may be configured to receive energy from an energy wave source, the energy wave source including a mechanical envelope having a width different than the width of at least one of the first surface 6114 and the second surface 6116.

[0180] In each relay 6112, energy is transported between first and second surfaces which defines the longitudinal orientation, the first and second surfaces of each of the relays extends generally along a transverse orientation defined by the first and second directions, where the longitudinal orientation is substantially normal to the transverse orientation. In one embodiment, energy waves propagating through the plurality of relays have higher transport efficiency in the longitudinal orientation than in the transverse orientation due to randomized refractive index variability in thetransverse orientation coupled with minimal refractive index variation in the longitudinal orientation. In some embodiments where each relay is constructed of multicore fiber, the energy waves propagating within each relay element may travel in the longitudinal orientation determined by the alignment of fibers in this orientation.

[0181] In an embodiment, a separation between the edges of any two adjacent second sides of the terminal energy relay elements may be less than a minimum perceptible contour as defined by the visual acuity of a human eye having better than 20 / 40 vision at a distance from the seamless energy surface that is greater than the lesser of a height of the singular seamless energy surface or a width of the singular seamless energy surface.

[0182] In one embodiment, the plurality of energy relay elements in the stacked configuration may include a plurality of faceplates. In some embodiments, the plurality of faceplates may have different lengths or are loose coherent optical relays. In other embodiments, the plurality of elements may have sloped profile portions similar to that of FIG. 7B, where the sloped profile portions may be angled, linear, curved, tapered, faceted or aligned at a non-perpendicular angle relative to a normal axis of the relay element. In yet another embodiment, energy waves propagating through the plurality of relay elements have higher transport efficiency in the longitudinal orientation than in the transverse orientation due to randomized refractive index variability in the transverse orientation coupled with minimal refractive index variation in the longitudinal orientation. In embodiments where each energy relay is constructed of multicore fiber, the energy waves propagating within each relay element may travel in the longitudinal orientation determined by the alignment of fibers in this orientation.

[0183] In some embodiments, the one or more relay elements (e.g., 6112) includes fused or tiled mosaics, where any seams between adjacent fused or tiled mosaics are separated by or are less than the minimum perceptible contour as defined by the visual acuity of a human eye having better than 20 / 40 vision at a distance at or greater than the width or height of the singular seamless energy surface.

[0184] In other embodiments, the one or more relay elements (e.g., 6112) includes: optical fiber, silicon, glass, polymer, optical relays, diffractive elements, holographic optical elements, refractive elements, reflective elements, optical face plates, optical combiners, beam splitters, prisms, polarization components, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar materials having Anderson localization or total internalreflection properties for forming the singular seamless energy surface.

[0185] In yet other embodiments, the one or more relay elements (e.g., 6112) are configured to accommodate a shape of the singular seamless energy surface including planar, spherical, cylindrical, conical, faceted, tiled, regular, non-regular, or any other geometric shape for a specified application.

[0186] In another embodiment, the system further includes an energy waveguide system (e.g., 7100 as best shown in FIGS. 7C-7L) having an array of energy waveguides, where a second plurality of energy propagation paths extend from the singular seamless energy surface through the array of energy waveguides in directions determined by a 4D plenoptic function.

[0187] FIG. 7C illustrates a top-down perspective view of an embodiment of an energy waveguide system 7100 operable to define a plurality of energy propagation paths 7108. Energy waveguide system 7100 comprises an array of energy waveguides 7112 configured to direct energy therethrough along the plurality of energy propagation paths 7108. In an embodiment, the plurality of energy propagation paths 7108 extend through a plurality of energy locations 7118 on a first side of the array 7116 to a second side of the array 7114.

[0188] Referring to FIGS. 7C and 7L, in an embodiment, a first subset 7290 of the plurality of energy propagation paths 7108 extend through a first energy location 7122. The first energy waveguide 7104 is configured to direct energy along a first energy propagation path 7120 of the first subset 7290 of the plurality of energy propagation paths 7108. The first energy propagation path 7120 may be defined by a first chief ray 7138 formed between the first energy location 7122 and the first energy waveguide 7104. The first energy propagation path 7120 may comprise rays 7138A and 7138B, formed between the first energy location 7122 and the first energy waveguide 7104, which are directed by first energy waveguide 7104 along energy propagation paths 7120 A and 7120B, respectively. The first energy propagation path 7120 may extend from the first energy waveguide 7104 towards the second side of the array 7114. In an embodiment, energy directed along the first energy propagation path 7120 comprises one or more energy propagation paths between or including energy propagation paths 7120 A and 7120B, which are directed through the first energy waveguide 7104 in a direction that is substantially parallel to the angle propagated through the second side 7114 by the first chief ray 7138.

[0189] Embodiments may be configured such that energy directed along the first energy propagation path 7120 may exit the first energy waveguide 7104 in a direction that is substantiallyparallel to energy propagation paths 7120A and 7120B and to the first chief ray 7138. It may be assumed that sin energy propagation path extending through an energy waveguide element 7112 on the second side 7114 comprises a plurality of energy propagation paths of a substantially similar propagation direction.

[0190] FIG. 7D is a front view illustration of an embodiment of energy waveguide system 7100. The first energy propagation path 7120 may extend towards the second side of the array 7114 in a unique direction 7208 extending from the first energy waveguide 7104, which is determined at least by the first energy location 7122. The first energy waveguide 7104 may be defined by a spatial coordinate 7204, and the unique direction 7208 which is determined at least by first energy location 7122 may be defined by an angular coordinate 7206 defining the directions of the first energy propagation path 7120. The spatial coordinate 7204 and the angular coordinate 7206 may form a four-dimensional plenoptic coordinate set 7210 which defines the unique direction 7208 of the first energy propagation path 7120.

[0191] In an embodiment, energy directed along the first energy propagation path 7120 through the first energy waveguide 7104 substantially fills a first aperture 7134 of the first energy waveguide 7104 and propagates along one or more energy propagation paths which lie between energy propagation paths 7120 A and 7120B and are parallel to the direction of the first energy propagation path 7120. In an embodiment, the one or more energy propagation paths that substantially fill the first aperture 7134 may comprise greater than 50% of the first aperture 7134 diameter.

[0192] In a preferred embodiment, energy directed along the first energy propagation path 7120 through the first energy waveguide 7104 which substantially fills the first aperture 7134 may comprise between 50% to 80% of the first aperture 7134 diameter.

[0193] Turning back to FIGS. 7C and 7E-7L, in an embodiment, the energy waveguide system 7100 may further comprise an energy inhibiting element 7124 positioned to limit propagation of energy between the first side 7116 and the second side 7114 and to inhibit energy propagation between adjacent waveguides 7112. In an embodiment, the energy inhibiting element is configured to inhibit energy propagation along a portion of the first subset 7290 of the plurality of energy propagation paths 7108 that do not extend through the first aperture 7134. In an embodiment, the energy inhibiting element 7124 may be located on the first side 7116 between the array of energy waveguides 7112 and the plurality of energy locations 7118. In an embodiment, the energyinhibiting element 7124 may be located on the second side 7114 between the plurality of energy locations 7118 and the energy propagation paths 7108. In an embodiment, the energy inhibiting element 7124 may be located on the first side 7116 or the second side 7114 orthogonal to the array of energy waveguides 7112 or the plurality of energy locations 7118.

[0194] In an embodiment, energy directed along the first energy propagation path 7120 may converge with energy directed along a second energy propagation path 7126 through a second energy waveguide 7128. The first and second energy propagation paths may converge at a location 7130 on the second side 7114 of the array 7112. In an embodiment, a third and fourth energy propagation paths 7140, 7141 may also converge at a location 7132 on the first side 7116 of the array 7112. In an embodiment, a fifth and sixth energy propagation paths 7142, 7143 may also converge at a location 7136 between the first and second sides 7116, 7114 of the array 7112.

[0195] FIGS. 7E-7L are illustrations of various embodiments of energy inhibiting element 7124. For the avoidance of doubt, these embodiments are provided for exemplary purposes and in no way limiting to the scope of the combinations or implementations provided within the scope of this disclosure.

[0196] FIG. 7E illustrates an embodiment of the plurality of energy locations 7118 wherein an energy inhibiting element 7251 is placed adjacent to the surface of the energy locations 7118 and comprises a specified refractive, diffractive, reflective, or other energy altering property. The energy inhibiting element 7251 may be configured to limit the first subset of energy propagation paths 7290 to a smaller range of propagation paths 7253 by inhibiting propagation of energy along energy propagation paths 7252. In an embodiment, the energy inhibiting element is an energy relay with a numerical aperture less than 1.

[0197] FIG. 7F illustrates an embodiment of the plurality of energy locations 7118 wherein an energy inhibiting structure 7254 is placed orthogonal between regions of energy locations 7118, and wherein the energy inhibiting structure 7254 exhibits an absorptive property, and wherein the inhibiting energy structure 7254 has a defined height along an energy propagation path 7256 such that certain energy propagation paths 7255 are inhibited. In an embodiment, the energy inhibiting structure 7254 is hexagonal in shape. In an embodiment, the energy inhibiting structure 7254 is round in shape. In an embodiment, the energy inhibiting structure 7254 is non-uniform in shape or size along any orientation of the propagation path. In an embodiment, the energy inhibiting structure 7254 is embedded within another structure with additional properties.

[0198] FIG. 7G illustrates the plurality of energy locations 7118, wherein a first energy inhibiting structure 7257 is configured to substantially orient energy 7259 propagating therethrough into a first state. A second energy inhibiting structure 7258 is configured to allow energy 7259, which is substantially oriented in the first state, to propagate therethrough, and to limit propagation of energy 7260 oriented substantially dissimilarly to the first state. In an embodiment, the energy inhibiting element 7257, 7258 is an energy polarizing element pair. In an embodiment, the energy inhibiting element 7257,7 258 is an energy wave band pass element pair. In an embodiment, the energy inhibiting element 7257, 7258 is a diffractive waveguide pair.

[0199] FIG. 7H illustrates an embodiment of the plurality of energy locations 7118, wherein an energy inhibiting element 7261 is structured to alter energy propagation paths 7263 to a certain extent depending upon which of the plurality of energy locations 7118 the energy propagation paths 7263 extends through. Energy inhibiting element 7261 may alter energy propagation paths 7263 in a uniform or non-uniform way along energy propagation paths 7263 such that certain energy propagation paths 7262 are inhibited. An energy inhibiting structure 7254 is placed orthogonal between regions of energy locations 7118, and wherein the energy inhibiting structure 7254 exhibits an absorptive property, and wherein the inhibiting energy structure 7254 has a defined height along an energy propagation path 7263 such that certain energy propagation paths 7262 are inhibited. In an embodiment, an inhibiting element 7261 is a field lens. In an embodiment, an inhibiting element 7261 is a diffractive waveguide. In an embodiment, an inhibiting element 7261 is a curved waveguide surface.

[0200] FIG. 71 illustrates an embodiment of the plurality of energy locations 7118, wherein an energy inhibiting element 7264 provides an absorptive property to limit the propagation of energy 7266 while allowing other propagation paths '1 6'1 to pass.

[0201] FIG. 7J illustrates an embodiment of the plurality of energy locations 7118, and the plurality of energy waveguides 7112, wherein a first energy inhibiting structure 7268 is configured to substantially orient energy 7270 propagating therethrough into a first state. A second energy inhibiting structure 7271 is configured to allow energy 7270, which is substantially oriented in the first state, to propagate therethrough, and to limit propagation of energy 7269 oriented substantially dissimilarly to the first state. In order to further control energy propagation through a system, exemplified by the stray energy propagation 7272, energy inhibiting structures 7268, 7271 mayrequire a compound energy inhibiting element to ensure energy propagation maintains accurate propagation paths.

[0202] FIG. 7K illustrates an embodiment of the plurality of energy locations 7118, and wherein an energy inhibiting element 7116 provides an absorptive property to limit the propagation of energy along energy propagation path 7278 while allowing other energy along energy propagation path 7277 or 7279 to pass through a pair of energy waveguides 7112 for an effective aperture 7284 within the array of waveguides 7112. In an embodiment, energy inhibiting element 7276 comprises black chrome. In an embodiment, energy inhibiting element 7276 comprises an absorptive material. In an embodiment, energy inhibiting element 7276 comprises a transparent pixel array. In an embodiment, energy inhibiting element 7276 comprises an anodized material.

[0203] FIG. 7L illustrates an embodiment comprising a plurality of energy locations 7118, and a plurality of energy waveguides 7112, wherein a first energy inhibiting structure 7251 is placed adjacent to the surface of the energy locations 7118 and comprises a specified refractive, diffractive, reflective, or other energy altering property. The energy inhibiting structure 7251 may be configured to limit the first subset of energy propagation paths 7290 to a smaller range of propagation paths 7275 by inhibiting propagation of energy along energy propagation paths 7274. A second energy inhibiting structure 7261 is structured to alter energy propagation paths 7275 to a certain extent depending upon which of the plurality of energy locations 7118 the energy propagation paths 7275 extends through. Energy inhibiting structure 7261 may alter energy propagation paths 7275 in a uniform or non-uniform way such that certain energy propagation paths 7274 are inhibited. A third energy inhibiting structure 7254 is placed orthogonal between regions of energy locations 7118. The energy inhibiting structure 7254 exhibits an absorptive property and has a defined height along an energy propagation path 7275 such that certain energy propagation paths 7274 are inhibited. An energy inhibiting element 7276 provides an absorptive property to limit the propagation of energy 280 while allowing energy 7281 to pass through. A compound system of similar or dissimilar waveguide elements 7112 are positioned to substantially fill an effective waveguide element aperture 7285 with energy from the plurality of energy locations 7118 and to alter the propagation path 7273 of energy as defined by a particular system.

[0204] In an embodiment, the energy inhibiting structure 7124 may be located proximate the first energy location 7122 and generally extend towards the first energy waveguide 7104. In anembodiment, the energy inhibiting structure 7124 may be located proximate the first energy waveguide 7104 and generally extend towards the first energy location 7122.

[0205] In one embodiment, the energy system is configured to direct energy along the second plurality of energy propagation paths through the energy waveguide system to the singular seamless energy surface, and to direct energy along the first plurality of energy propagation paths from the singular seamless energy surface through the energy relay system to the plurality of energy devices.

[0206] In another embodiment, the energy system is configured to direct energy along the first plurality of energy propagation paths from the plurality of energy devices through the energy relay system to the singular seamless energy surface, and to direct energy along the second plurality of energy propagation paths from the singular seamless energy surface through the energy waveguide system.

[0207] In yet another embodiment, the singular seamless energy surface is operable to guide localized light transmission to within three or less wavelengths of visible light.Sensory Data Suitable For Holographic Displays

[0208] A 3D object can be described as a collection of modeling primitives such as points, lines, polygons, or surfaces. In an embodiment of the physical interference modeling, for example, a point-cloud based approach may assume the 3D scenes are a collection of point light sources, and spherical waves emitted from these point sources are computed and superposed in the hologram plane. This approach is easy to implement and flexible but is very time consuming for full-parallax CGHs because a large number of point sources are used to describe photo-realistic effects. In an embodiment of the physical interference modeling, a polygon-based approach may consider an object to be a collection of small planar surface light sources that have polygonal shapes, and the wave fields emitted by these polygonal sources are calculated and are superposed in the hologram plane.

[0209] In an embodiment, a silhouette method may be used to account for occlusion among separate objects in a scene. For an opaque object, the viewer should be able to see the near-side front surface of the object and a part of the background that does not overlap the object. If the viewpoint is moved, the visible background changes with the viewpoint. The silhouette methodprovides simple emulation of this physical phenomenon. In an embodiment, the background light can be expressed by the wave field placed at a position in which the cross section of the object takes the maximum. The part of the background field overlapping the object cannot pass through the object, and the field amplitude vanishes in the overlapping area. This may be expressed by multiplying the wave field by a binary mask function whose value is zero inside the silhouette of the object and unity outside the silhouette.

[0210] As discussed above, the geometric characteristics of an energy wave may be represented by light rays that are perpendicular to the wavefront of the energy wave. Also as discussed above, the angular spectrum method is a technique for modeling the propagation of a wave field that expands a complex wave field into a summation of infinite number of plane waves of the same frequency and different directions. The Fourier transform of a wavefront in a region on a certain plane in 3D space derives the angular spectrum. The spatial frequency of the angular spectrum corresponds to the direction of the wave propagation, which is parallel to the light ray that passes through the same region. Thus, the intensity of the angular spectrum may be considered to be the intensity of light rays. Under this principle, the conversion between a light field to a wavefront may be realized by sampling the light field throughout a sampling plane, and Fourier transforming the image of rays that path through a point in the sampling plane. The phase of each ray may be assigned using techniques such as phase added stereogram or random / pseudo-random sequences. The rays may also be obtained by inverse conversion from the complex amplitude of the wavefronts to rays, and the phase information in the angular spectrum can be retained. The Fourier spectrum obtained for all the sampling locations in the sampling plane can be aggregated together to obtain the wavefront on the sampling plane.

[0211] The geometric light ray representation of the complex amplitude function of an energy wave can be parameterized by a plenoptic 4D function through the surface from an energy directing surface provides for two spatial coordinates xbytfrom a first plane comprising energy locations and directed through a second coordinate along a second plane comprising waveguiding parameters ubvLdefining a vector of an energy propagation path ft(x;, ybubvt) . In consideration of a plurality of energy directing surfaces, the plenoptic 5D function provides for three spatial coordinates xlylztfrom a first coordinate comprising one or more energy locations and directed through a second coordinate along a plane comprising waveguiding parameters ubvLdefining a vector of an energy propagation path ft(xlylzbubvt) . For each of 4D or 5D, additional variablesfor time and colormay be considered and assumed to be inclusive of any of the plenoptic functions as necessary for an application even when not explicitly noted for simplicity of the function and discussion. For the avoidance of doubt, the reference to an energy directing surface is for exemplary purposes only and may comprise any additional point, location, direction, or plane in space for the localization of a 5D coordinate, and collectively referred to as an energy “directing surface”.

[0212] FIG. 8 is a flow chart diagram illustrating an embodiment of a process 800 for determining four dimensional (4D) plenoptic coordinates for content data. The process 800 may include a step 802 in which content data is received, which may include any signal perceptible by a visual, audio, textural, sensational, or olfactory sensor. FIG. 9 is a schematic diagram illustrating an embodiment of the content data, which may include at least one of the following: an object location, a material property (such as material properties 906, 907, and 908), a virtual light source 904, geometry 902 at non-object location, content out of the reference surface, a virtual camera position 914, a segmentation 910 of objects, background texture 912, and layered contents.

[0213] Referring to FIGS. 8 and 9, the process 800 may further include a step 804 in which locations of data points are determined with respect to a first surface 920 to creating a digital volumetric representation 922 of the content data. The first surface 920 may be used as a reference surface for defining the locations of data points in space. In an embodiment, the process 800 may further include a step 806 in which 4D plenoptic coordinates of the data points are determined at a second surface by tracing the locations of the data points in the volumetric representation to the second surface where a 4D function is applied. In an embodiment, the process 800 may further include a step 808 in which energy source location values are determined for 4D plenoptic coordinates that have a first point of convergence.

[0214] The content data received in step 802 may include N views, where N is one or more. A single view may be presented with or without a depth channel. Stereoscopic views may be presented with or without a depth channel. Multi-view imagery may be presented with or without a depth channel. Further, a 4D light field may be presented with or without a depth channel.

[0215] The tracing of step 806 may use prior knowledge of a calibrated geometry of an energy system, which may be stored in memory as a global model or an individually characterized system or some combination of the two methodologies.

[0216] In an embodiment, the mapping between the input data and the output energy source provides a methodology to accurately map between various bitrate sources. The tracing of step 806 provides the ability to infer the full volumetric 4D data set from the above listed partial samples. Depth information either needs to be provided or calculated from the available data. With the depth information known or calculated, the N view(s) may be inverse traced by triangulation of the samples from the known volumetric presentation based upon depth coordinate into the 4D space.

[0217] The triangulation may assume that each available energy source location in the N source content are representative of a energy source location for each energy waveguide in the event that a mapping between energy waveguide and energy source location format resolution are provided. In the event that the N source content resolution are lower, super-resolution or scaling algorithms may be implemented. In the event that the resolution of the N source image(s) are higher than the number of energy waveguides in the energy directing device, interpolation between super-sampled energy source locations may be performed to produce higher amount of energy source locations per energy waveguide in the resultant 4D inverse ray trace.

[0218] The above assumes distance information may be determined from the depth maps which may or may not be accurate depending on the form of depth information provided or calculated, and with the distance information known or assumed, the distance information in combination with the x-y energy source location coordinate and the (u,v) angular information as determined by the energy directing device properties may then be considered a 4D or 5D light field with limited imaging data samples. The imaging samples, based upon the distance information, are triangulated back to the appropriate energy source locations that may exist behind each energy waveguide respectively, and missing data may be generated in step 808 through the disclosures contained herein.

[0219] Referring to FIGS. 7C, 8, 9, 10 in an embodiment, the energy locations may be located in the first surface 920, and the second surface where a 4D function is applied may correspond to a waveguide system 7100 of an energy directing device, and energy is operable to be directed through the waveguide system according to the 4D plenoptic coordinates of the data points to form a detectable volumetric representation of the content data.

[0220] In an embodiment, the process 800 may further comprise a step 810, in which energy source location values are determined for 4D coordinates that have a first point of convergence.To provide an example implementation of the present disclosure, FIG. 10 illustrates an embodiment of an energy directing device 1000 going through a tracing process where content data in the form of an image 1002 is provided with a distance position 1004, which may be provided or calculated, within a determined minimum position 1006 and maximum position 1008 in reference to the energy locations 1010. In an embodiment, the energy locations 1010 may comprise an energy directing device surface 1012. The known geometry from the energy locations 1010 defined by the 4D plenoptic function allows for the triangulation of a point 1014 on the virtual surface of the image 1002 to be traced back along rays 1016 to specific energy locations 1018, each having a unique x-y coordinate. Missing samples may be computationally calculated based upon the available information contained within the dataset.

[0221] When additional N samples are provided, the same methodology is applied with the additional multi-perspective imaging data producing a richer set of inverse ray traced samples and provide superior holographic results. The depth information from a multiple N samples may be provided through a single depth map, or up to N, or greater than N depth maps with a known mapping between the source location (the N+X perspective) and the source depth map (the N+X depth map) to ensure appropriate inverse ray tracing is performed.

[0222] In the event that a singular depth map for the, for example, center N perspective is provided, the additional depth maps may be interpolated by calculating for the disparity between each of the adjacent views to accurately map the source and target location between the N and the N+X viewpoints. With this method, it is possible to inverse ray trace the appropriate view dependent mapping to the 4D light field such that the correct perspective(s) are projected to the appropriate waveguide coordinates and results in the viewer's ability to maintain the correct view dependencies in the associated viewpoints.

[0223] The encoder and decoders are robust and may interpret multiple data types to include, but not limited to, 2D / flat files, 2D with depth, stereoscopic, stereoscopic with single depth channel, stereoscopic with dual depth channel, N+X multi-view with no depth, N+X multi-view with N+Y depth, geometric or vector based scene files that may include textures, geometry, lighting, material properties and the like to reconstruct an environment, deep imaging files wherein multiple RGBAZ values may be provided for each x-y coordinate, 4D or 5D (4D plus depth) light fields, or provided as a N+X view plus N+Y delta channel dataset wherein the depth channel provides a lower bandwidth methodology for only rendering a certain amount of energy source location data asrequired for a determined energy directing device field of view. The processors are able to inverse ray trace at up to, or exceeding, real-time speeds, in order to provision the appropriate 4D light field to present to the viewer with and without world coordinate locations, with and without compensated minimum and maximum projected world locations and in consideration of the energy directing device intrinsic as characterized and / or designed.

[0224] In an embodiment, the process 800 may further comprise a step 812, in which a mapping between energy locations 7122 on a first side of the waveguide system 7100 and the angular directions of the energy propagation paths 7120 from the waveguide element 7100 on a second side of the waveguide system 7100 is applied. Doing so may allow a plurality of energy locations on the first side of the waveguide system 7100 corresponding to the 4D plenoptic coordinates of the data points to be determined.

[0225] FIG. 12 is a schematic diagram of a processing system 1200 comprising a data input / output interface 1201 in communication with a processing subsystem having a sensory data processor 1202, a vectorization engine 1204, and a tracing engine 1206. It is to be appreciated that the sensory data processor 1202, the vectorization engine 1204, and the tracing engine 1206 may be implement on one or more processors, whether individually or any combination thereof. Step 802 of the process 800 may input content data through the data input / output interface 1201 to the processing subsystem 1220. Step 804 may be performed by the sensory data processor 1202 to create a volumetric representation of the content data. Step 806 may be performed by the tracing engine to determine 4D plenoptic coordinates of the data points at a second surface.

[0226] In an embodiment, applying the mapping may comprise calibrating for a distortion in the waveguide system 7100, which may further comprise calibrating for at least one distortion selected from a group consisting of: a spatial distortion, angular distortion, intensity distortion, and color distortion.

[0227] In an embodiment, the energy directing device may further comprise a relay system 61 lOon the first side of the waveguide system 7100, the relay system having a first surface 6116 adjacent to the waveguide system 7100, and the energy locations 7112 on the first side of the waveguide system may be positioned adjacent to a second surface 6114 of the relay system 6110.

[0228] In an embodiment, applying the mapping may include calibrating for a distortion in the waveguide system 7100. In an embodiment, applying the mapping may include calibrating both for a distortion in the relay system 6110 and distortion in the waveguide system 7100. In anembodiment, the distortion to be calibrated may include at least one distortion selected from a group consisting of: a spatial distortion, angular distortion, intensity distortion, and color distortion.

[0229] In an embodiment, a portion of the method may be carried out in real time, or the method may be entirely carried out in real time, or at least two portions of the method may be carried out in different time periods.Conversion from 2D to Light Field Parameterization of Energy Wavefronts

[0230] In an embodiment, content data may comprise data points in a two dimensional (2D) space, and determining locations of step 704 may comprise applying a depth map to the data points in a two dimensional space.

[0231] There are several methods to convert two-dimensional or flat imagery into light field parameterization data for energy wavefronts. These include the estimation of depth information through depth from motion analysis, a provided depth channel through manual or rendered means, or the manual creation of disparity, depth, occlusion, geometry and / or any other methodology known as standard for visual effects content creation to reproduce the full light field through regeneration of the entire environment through manual and automated processes.

[0232] In a first embodiment, a system that includes a real-time or offline processor to perform estimation of depth from available energy source location information is possible. This may be performed at the energy directing device, as a set top box or as an offline process. Additional computation for missing volumetric data may be performed leveraging temporal information and / or state of the art texture synthesis or other technologies known in the art.

[0233] In a second embodiment, depth information is provided as an image stream and may be embedded into the image format. Similarly, additional computation may be performed for missing volumetric data.

[0234] In a third embodiment, an artist or a process is leveraged to generate the missing environmental information which may include a process to isolate or segment each object in a scene, track said objects over time manually, semi- automatically or automatically, place objects into space leveraging disparity space, energy directing device space, optical space or world coordinates, synthesizing background and foreground missing information through visual effectsprocesses known in the art for reconstruction of backgrounds, transparencies, edge details, etc. to regenerate the environment. For the avoidance of doubt, the implemented processes may be any, none or all of the listed embodiments for the reconstruction of these environments. The generated environmental information should include as much of the missing information as possible as determined by the energy directing device angles of view, and these angles of view may be known by the artist to ensure that appropriate occlusion and view dependent information is generated appropriately.

[0235] Additionally, the surface model for each object in the scene may be generated, either as a partial model or as a completely built model and textures from the image data are projected onto the surfaces of the geometry to provide appropriate shape for the following inverse ray tracing.

[0236] Additionally, material properties may be calculated or manually introduced to ensure that view dependent lighting may be introduced with virtual illumination sources to further increase the accuracy of the regeneration of the 4D light field.

[0237] Further, the addition of CG or synthetic content may be introduced to augment the existing converted materials. The addition of volumetric data may also be incorporated. The inter-mixing of N+X content may be introduced as well to provide a seamless blend between CG, 2D, stereoscopic, multiview and / or 4D media into a single composite.

[0238] The resultant 2D to light field converted content may be retained as a geometric scene file including geometry, textures, lighting, materials, etc. as indicated in the CG scene itself, rendered as N+X views with N+D depth channels, rendered as a 4D or 5D (4D + depth) light field, a deep image which is a format that allows for multiple RGBAZ samples per x-y energy source location coordinate with or without a limitation of stacking of Z samples per x-y coordinate, or provided as a N+X view plus N+Y delta channel dataset wherein the depth channel provides a lower bandwidth methodology for only rendering a certain amount of energy source location data as required for a determined energy directing device field of view. Tools may be provided to allow for the generation of all, some or one of these respective output formats.Conversion of Stereoscopic and Multi-view to Light Field Parameterization of Energy Wavefronts

[0239] The process from above leveraging single view content may be applied to stereoscopic and multi-view materials. The estimation of depth information is obtained through depth from motion analysis, as well as from stereoscopic, multi-view and / or disparity analysis, a provided depth channel or provided depth channels through manual or rendered means, or the manual creation of disparity, depth, occlusion, geometry and / or any other methodology known as standard for visual effects content creation to reproduce the full light field parameterization of energy wavefronts through regeneration of the entire environment through manual and automated processes and leveraging the appropriate data to further retain the view dependent content as available in the provided imaging materials.

[0240] In an embodiment, the content data received in step 102 may comprise data points in a three dimensional (3D) space, and determining locations may comprise adjusting the data points in the 3D space.

[0241] In an embodiment, adjusting the data points in the 3D space may include applying a depth map to the data points in the 3D space, adding new data points, reconstructing occluded data points, or any combination thereof.

[0242] The significant advantage to this approach exists in that the accuracy of stereoscopic disparity estimation is far greater than from motion parallax or other similar 2D estimation processes alone. Further the image quality of the resultant converted 4D light field is more accurate due to the availability of some of the view dependent conditions, including but not limited to illumination, transparencies, materials, occlusion, etc.

[0243] The ability to retain the explicit angular dependencies of the multi-view image data relies on the ability to calculate the surface normals in relation to the center viewpoint camera, or some other defined center point. With these normals and disparity or depth information known, it is possible to interpolate between viewpoints based upon energy directing device angle of view, which is then either directly applied to the inverse ray tracing, or synthesized as a portion of the texture synthesis during the inverse ray tracing.

[0244] For brevity, all of the previously disclosed methodologies for the reconstruction of 2D to light field parameterization of energy wavefronts may be applied to the reconstruction of stereoscopic or multi-view datasets.Generation of NxN RGB Images from 4D or 5D Light Field Parameterization of EnergyWavefronts

[0245] By leveraging a 4D or 5D light field parameterization of energy wavefronts, it is possible to generate NxN or any value of up to NxN number of RGB multi-view images. This process is accommodated by considering each bottom left coordinate under each waveguide, assuming a square grid, the 0,0 position, and the top right position as the N,N position. The grid is only exemplary and any other mapping methodology may be leveraged. For each 0,0 to N,N position, it is possible to form full resolution images from the light field with the widest possible depth of field based upon the capture system leveraged wherein each waveguide in the array is considered a single energy source location and each coordinate under each waveguide is a single energy source location of the larger energy source location array for each complete image from 0,0 to N,N respectively. This may be repeated for a 5D light field for the depth information as well. In this fashion, it is possible to easily translate between the 4D or 5D light field to any subset of the dataset that is desired for various distribution reasons to include 2D, stereoscopic, multi-view, point cloud, CG scene file, or any other desired combination of data that may be derived from the 4D or 5D light field. For non-regular or square packed 4D or 5D structures, further interpolation is required to align energy source locations to a regular grid, or a linear mapping between energy source locations and non-square packed structures may be implemented wherein the resultant images may not appear rectilinear and may also contain energy source location artifacts.

[0246] FIG. 11 exemplifies the methodology to convert from a 4D or 5D light field into multiple viewpoints by arranging the energy locations 1102 from underneath of each energy waveguide element 1104 according to energy waveguide element position and energy location coordinate respectively. This provides the ability to seamlessly transfer between light field and smaller datasets seamlessly.N+X RGB and N+Y Depth Datasets

[0247] The ideal dataset format that provides the highest quality with the balance of data transmission size includes the use of N+X RGB and N+Y Depth + vectorized channels wherein N+X RGB information contains N RGB images that may represent a certain resolution and format, and X that may represent a different resolution and format for RGB data to include lowerresolutions, delta information and the like, and N+Y Depth + vectorized channels that contains N depth + vectorized channels that may represent a certain resolution and format and Y that may represent a different resolution and format for depth + vector data to include lower resolutions, delta information and the like.

[0248] The number of N+X views may be generated on a regular grid, from a radius around a center point with or without a center view, from multiple radii around a center point with or without a center view, or any methodology to determine the mapping of the number of views and the associated packing or perspective locations. The configuration for the perspectives may be contained in the metadata of the file, or the depth + vectorized channels provided may include a direct mapping to world coordinates such that the imaging data aligns to the same coordinate in XYZ space without other necessary metadata.4D Disk inversion and Energy directing device Compatibility Processing

[0249] For any data captured with a plenoptic or light field 4D or 5D system, including potentially those captured with virtual rigs with optical simulation of a 4D or 5D light field system, the resultant fly's eye perspectives contain discs that represent the uv vectors for the light field. However, these coordinates assume energy focusing elements that may not exist in an energy directing device. In the proposed energy directing device solution, the focusing elements may be the viewer's eye, and the mapping between the capture system and the mapping between the original capture methodology and the viewed energy directing device are no longer correct.

[0250] To invert this and correct for the additionally missing energy directing element in the system when compared to the capture system, it is possible to individually flip each disc independently, wherein the x-y location of each (u,v) coordinate is retargeted based upon the center point of each waveguide respectively. In this fashion, the inversion of the image that forms as a result of the main waveguide is inverted and allows for the light field energy directing device to project the rays in the correct x-y-u-v orientation.

[0251] A further embodiment of this may implement a hardware modification wherein leveraging an energy waveguide array provides a direct inversion of every presented energy waveguide energy source location. For light field energy directing devices, this is advantageous to have a direct mapping between a potential capture system and energy directing device. This may furtherbe advantageous an embodiment comprising HMD systems or volumetric opacity energy directing devices such that a group of energy waveguides in the overall array may be eliminated by removing the necessity to relay additional times for accurate x-y-u-v coordinates.

[0252] Further, not all light fields are identical. They may be captured with differing NAs, FOVs, N values, optical prescriptions, etc. The intrinsics and extrinsics of the input light field data may be understood and convert to the energy directing device characteristics. This may be performed by embodiments contained within this disclosure for universal parameterization of holographic and light field data.Universal Parameterization of Holographic Sensory Data Transport through Inverse Energy Tracing and Vectorization of Sensory Properties for an Energy Directing System

[0253] In an embodiment of parameterizing the complex amplitude function of energy wavefronts, a plenoptic 4D function through the surface from an energy directing surface provides for two spatial coordinates xbytfrom a first plane comprising energy locations and directed through a second coordinate along a second plane comprising waveguiding parameters ubdefining a vector of an energy propagation path . In consideration of a plurality of energydirecting surfaces, the plenoptic 5D function provides for three spatial coordinatesfrom a first coordinate comprising one or more energy locations and directed through a second coordinate along a plane comprising waveguiding parameters defining a vector of an energypropagation path . For each of 4D or 5D, additional variables for time and colormaY be considered and assumed to be inclusive of any of the plenoptic functions asnecessary for an application even when not explicitly noted for simplicity of the function and discussion. For the avoidance of doubt, the reference to an energy directing surface is for exemplary purposes only and may comprise any additional point, location, direction, or plane in space for the localization of a 5D coordinate, and collectively referred to as an energy “directing surface”.

[0254] Along a first vector of an energy propagation path, a plurality of intersection points comprising convergence of energies may occur together with additional energy propagation paths. At this intersection point, a 3D point or depth parameter forms at location XbYbZtamong the plurality of energy propagation paths with the 4D or 5D functions, wherein the 3D point ofconvergence X YbZLamong the plurality of energy propagation paths, where for each xbytor xbYi>zi coordinate contained within the energy directing surface or surfaces, there is only a single ubvtpropagation path that forms between a first coordinate and the converging 3D point. The 4D function fz(.xby u v or 5D function fzxi,yt>zi>ui>vi) collectively define all 4D(,y(, or 5D xlylZi coordinates and commensurate ubpropagation paths that exist for each converging point at XbYbZ

[0255] At a first 5D coordinate resulting from the convergence of energies along a plurality of energy propagation paths through the energy directing surface X YbZt, the coordinate may represent a point within a larger object, volume, particle or localized energy parameter, wherein converging energies at additional coordinates proximate to the first 5D coordinate may exhibit additional vectorized properties for sensory energies within an environment or holographic dataset. These vectorized properties may comprise information for each 5D coordinate, for each energy location coordinate within the 4D dataset, for regions within either of the 4D or 5D datasets, or other sub-sets of coordinates comprising the energy surface.

[0256] In an embodiment, the universal parameterization of 4D and 5D holographic sensory energy properties for propagation of visual, auditory, somatosensory, gustatory, olfactory, vestibular or other desired energies for sensory system response for raster and vector 2D, 3D, 4D and 5D datasets are disclosed, wherein the 2D data may comprise a single angular sample, 3D data may comprise two or more angular samples in a single dimension, 4D data may comprise a plurality of angular samples in two dimensions, or 5D data may comprise a plurality of angular samples in three or more dimensions, in reference to the second coordinate of the second plane of the 4D energy directing surface.Embodiments of received sample data may comprise any of:1). 2D or monoscopic, flat, point cloud, uv-mapped geometry, intrinsic geometry, deep images, layered images, CAD files (intrinsic), single point sampling, single camera capture, single projector projection, volumetric (monoscopic single sample points with vectors in a volume), sources of 3 Degrees of Freedom (DoF; raster with monoscopic x, y, z rotation about a single point), sources of non-light field 6 DoF (raster + vectors from monoscopic samples), volumetric energy directing device (monoscopic samples in a volume), sources of Pepper’s Ghost (single point projection), sources of 2D AR HMD (monoscopic singleor multiple focus planes; layered monoscopic), sources of 2D VR HMD (monoscopic single or multiple focus planes; layered monoscopic), or any other representation of two- dimensional raster or vector information;2). 3D or stereoscopic, triscopic (single baseline), multiview (ID), ID multi-sample, ID multi-perspective, horizontal or vertical only parallax, ID projection array, two point sampling, ID point sampling, horizontal or vertical array, bullet time, sources of 3 DoF (raster; stereoscopic x, y, z rotation about a single point), sources of 3 DoF (3D raster within stereoscopic x, y, z rotation about a single point), sources of non-light field 6 DoF (3D raster + vectors from stereoscopic samples), sources of ID volumetric energy directing device (ID parallax contained samples), sources of autostereoscopic data, sources of horizontal multiview energy directing device, sources of 3D AR HMD (stereoscopic single or multiple focus plane; layered stereoscopic), sources of 3D VR HMD (stereoscopic single or multiple focus planes; layered stereoscopic), or any other representation of three- dimensional raster or vector information;3). 4D or plenoptic (5D), multiscopic, integral image, light field (4D), holographic (4D), 2D multiview, 2D multi-sample, 2D multi-perspective, 2D parallax, horizontal and vertical parallax, 2D projection array, 2D point sampling, motion capture stage (along a surface), planar array, witness camera array, rendered or raytraced geometric representations (4D representations), extrinsic geometry (4D representation), sources of light field 6 DoF (4D raster within planar light field samples), sources of free-viewpoint 6 DoF (4D raster + vectors from 4D light field samples), sources of 4D volumetric energy directing device (2D parallax contained samples), sources of light field energy directing device (4D sampling), sources of light field HMD (near field 4D sampling), sources of holographic energy directing device (4D sampling), or any other representation of four-dimensional raster or vector information;4). 5D or plenoptic + depth, light field + depth, holographic (5D sampling, 4D + depth), arbitrary multiview (along all x, y and z axis), multi-sample (along all xyz), multiperspective (along all xyz), volumetric parallax (along all xyz), projection array (along all xyz), point sampling (along all xyz), motion capture stage (along all xyz), witness camera array (arbitrary xyz configurations), rendered or raytraced geometric representations (5D representations), cubic or volumetric rendering (along all xyz), extrinsic geometry (5Drepresentation), sources of light field 6 DoF (5D raster within volumetric light field samples), sources of free-viewpoint 6 DoF (5D raster + vectors from 5D light field samples), sources of 5D volumetric energy directing device (multiplanar 4D sampling), sources of 5D light field energy directing device (5D sampling, 4D + multiple planes), sources of 5D light field HMD (near field 5D sampling, 4D + multiple planes), sources of holographic energy directing device (5D sampling, 4D + multiple planes), or any other representation of five-dimensional raster or vector information.

[0257] At each of the second coordinates, the provided data may comprise a sub-set or a super-set of either raster or vector samples and wherein samples may represent and include additional vectorized information to enable transformation into increased sampling density through interpretation or processing of the sub-set or super-set of raster or vector samples.

[0258] For each of 2D, 3D, 4D or 5D provided datasets, the information is converted through vectorized information, manual identification, computer vision analysis, automated processing, or other means to transform the provided samples from the original dataset into a 5D coordinate system. For each of 2D, 3D, 4D or 5D provided datasets, the information may comprise multiple samples or layers of samples as well as additional vectorized properties in respect to the originating angular sampling component for each provided dataset in reference to the second coordinate of the second plane of the 4D energy directing surface, or may comprise a combination of contributing samples for any of 2D, 3D, 4D or 5D additional provided datasets.

[0259] Each of the provided samples comprise intrinsic energy for each desired coordinate, wherein the intrinsic energy may include additional extrinsic energy attributes, where the intrinsic energy represents value at a given 5D coordinate in the absence of other external samples, properties or environmental conditions. In the electromagnetic spectrum, this may be referred to as the albedo as the dimensionless measurement for reflectance corresponding to a white body that reflects all incident radiation, but explicitly extended to each desired sensory energy wherein the range of dimensionless values is commensurate to the specified sensory energy. Within the visual sensory systems, this range is approximately 400nm to 700um, and in the auditory sensory systems, this range is approximately 20Hz to 20kHz.

[0260] Over the past several decades, vast technological improvements enabling the reproduction of human senses artificially leveraging sophisticated pattern recognition of detected sensation,aromas and flavors through electronic means. For other systems that may exist outside of the electromagnetic spectrum, these dimensionless values may be characterized in the same way based upon sensed acuity response. While holographic sensory energy technologies are newly emerging, disclosed within this embodiment comprises a system, method and format for the stimulation of all human senses in a virtual environment to articulate the universal construct for various sensory parameters whereby provisioning for the appropriate data handling, transmission, storage, vectorization, translation to, from and between any sensory energy parameter or device desired for complete immersion of the constructed virtual environment and embodiments of energy propagation for holographic sensory technologies will be disclosed in future applications. It is additionally the intent of this disclosure to enable other analogue devices, including novelties like the classic “smell-o-vision,” or contemporary versions like FeelReal’s smelling VR headset, to leverage the parameterized values provided for within the vectorization of the dataset herein.

[0261] In an embodiment, the somatosensory system may be defined based upon the components that define sensitivity including mechanoreceptors for textures with a pressure sensitivity range in the skin that may be normalized between 50Hz to 300Hz, thermoreceptors with a temperature sensitivity range in the skin that may be normalized between 0°c to 50°c (although this range may be much wider range with upper and lower bounds defined by the extremes of temperature) or surface deformability defining the range of viscoelastic behaviors of a material measure both viscous and elastic characteristics when undergoing deformations between stress and strain over time which provides for a multiplicity of physics including variables for time, strain, modulus, among other dynamics, and for the purposes of this disclosure is simplified to a dimensionless normalized scale with a value of 0 for unmovable solids such as granite, and 1 for low viscosity liquids such as water. Those skilled in the art will understand that the actual vectors provided will comprise the necessary physics to appropriately define the viscoelasticity of the material, and normalized for exemplary purposes only.

[0262] Finally, state of the art advances in artificial electronic sensing including gustatory and olfactory devices demonstrate a viable path to further vectorizing the sensory parameters disclosed for the Holodeck design parameters, as well as enable the electronic reproduction of artificial taste and smell through a holographic waveguiding means as described herein. Artificial electronic taste and smell receptors have made considerable progress through emerging nanodevices, wherein frequency-based artificial taste receptors using an enzymatic biosensor to sample the intensity ofchemical stimulus through the encoding and conversion to frequency based pulses to both repeatedly and accurately detect taste as frequencies of the sampled chemical compositions through a pattern recognition system resulting in the detection of the tastes that compose the human palate. It is believed that the technology may be extended to all types of detectable tastes and similar advances in artificial olfactory system have demonstrated digital interfaces for stimulating ones smell receptors using weak electrical pulses targeting the nasal conchae with ongoing studies to further parameterize the patterns contained within frequencies of particular olfactory responses through variation in electrical signals.

[0263] With the path established for the arbitrary generation of frequencies and complex electronic patterns to represent olfactory, gustatory and other sensory system, in one embodiment, the acuity response for taste may be vectorized to comprise a normalized scale for each of electronically controlled parameters along a scale from 0 to 1 to represent the minimum and maximum gustatory response to saturate the average human’s 2,000 - 8,000 taste buds, potentially comprising but not limited to vectors for sourness, saltiness, bitter (spiciness), sweetness, and savoriness (umami) wherein the vector and the spatial coordinate of the vectorized signals may inform the production for the complex olfactory implementations.

[0264] In another embodiment, the acuity response for smell may be further vectorized to comprise a normalized scale for each of electronically controlled parameters along a scale from 0 to 1 to represent the minimum and maximum olfactory response to saturate the average human’s 10 cm2of olfactory epithelium, for each of the highly complex olfactory spaces potentially comprising but not limited to vectors for fragrant, fruity, citrus, woody (resinous), chemical, sweet, mint (peppermint), toasted (nutty), pungent and decayed wherein the vector and the spatial coordinate of the vectorized signals may inform the production for the complex olfactory implementations.

[0265] Each of these vectors may provide the normalized values representing these patterns for taste, smell or other sensory domains, converted to a wave, amplitude, magnitude or other attribute as required for the appropriate application of the provided vectorized values. While the sense of smell and taste are two of the most highly debased senses within the sensory system, with parameterized values to vectorize complex amalgamations, it is additionally possible in an embodiment to provide for user based interactive control over the sensitivity of any such sensory19energy to provide for customization of individualization of each of visual, auditory, somatosensory, gustatory, olfactory, vestibular or other desired sensory system responses.

[0266] In an embodiment, each of the represented sensory albedo energy values of the sample may additionally comprise extrinsic energy attributes baked into the single sample value representing the additive result of each provided sample respective of other external samples, properties or environmental conditions. In this configuration, the compound sample value may or may not exhibit latent attributes of other energies from other samples in a physically based or simulated environment. The most efficient and pure methodology to transmit the parameterized and reconstructed holographic dataset is based upon the singular intrinsic sample information providing for simplified and lower bandwidth frequency information, although this is not always possible to receive outside of entirely synthetic environments, particularly for physically based imaging or acoustic systems. In any real-world environment, there is always some amount of extrinsic contribution to the resultant sample information. Certain systems like the Light Stage, or other systems known in the art to facilitate the estimation of reflectance, shape, texture, and motion capture leverage some form of structured illumination and one or more imaging devices which provide for the direct or indirect analysis of the albedo, depth information, surface normal and bidirectional scattering distribution surface properties.

[0267] The bidirectional scattering distribution function (BSDF) is a generalized superset of the bidirectional transmittance distribution function (BTDF), the bidirectional texture function (BTF), and the bidirectional reflectance distribution function (BRDF), which are often represented by the generalized functionr(w£,ivr), collectively act as a model to parameterize and identify surface properties in computer graphics and vision algorithms known in the art. The function describes how visible light is reflected, transmitted or otherwise interacts with a surface given an incoming incident direction W and outgoing reflected or transmitted direction wrfor an energy propagation path, where the surface normal is perpendicular to the tangent of the object surface and the function describes the ratio of reflected radiance exiting along the outgoing path wrto the irradiance incident on the surface along the incoming path wt, wherein each of witwrmay comprise a 4D function to define a parameterized azimuth and zenith angle for each of the incoming light path and the exiting light path.

[0268] The functions may further be articulated for a first location xtof energystriking a surface, and exit after material properties internally scatter the energy to a second location xrofenergy 2rto account for visible wavelength effects like iridescence, luminescence, subsurface scattering, non-local scattering effects, specularity, shadowing, masking, interreflections, or the like, resultant output energy based upon material properties of a surface, the input energies and locations, the output energies and locations across the surface of an object, volume, or point.

[0269] Therefore, the generalized properties to describe how energy is transported between any two energy rays that strike a surface, to include wavelength or frequency dependency and spatially varying material properties or surfaces, may be represented as a 10D function, and specifiedivr) for each or any of the available or provided samples within a dataset to account for input energy, the impact of a vectorized surface profile, and the output reflected, refracted, specular, transmitted, scattered, diffused, or other material property result from any energy domain given the generalization of the function fr.

[0270] In consideration now of the energy directing surface, the plenoptic 4D function provides for two spatial coordinates xl,ylfrom a first plane comprising energy locations and directed through a second coordinate along a second plane comprising waveguiding parameters ubVi defining a vector of an energy propagation path . In consideration of a plurality ofenergy directing surfaces, the plenoptic 5D function provides for three spatial coordinates xbybztfrom a first coordinate comprising one or more energy locations and directed through a second coordinate along a plane comprising waveguiding parameters ubvtdefining a vector of an energy propagation path. For each of 4D or 5D, additional variables for time and color may be considered and assumed to be inclusive of any of the plenoptic functions as necessary for an application even when not explicitly noted for simplicity of the function and discussion.

[0271] Along a first vector of an energy propagation path, a plurality of intersection points comprising convergence of energies may occur together with additional energy propagation paths. At this intersection point, a 3D point or depth parameter forms at location among theplurality of energy propagation paths with the 4D or 5D plenoptic functions, wherein the 3D point of convergence XbYbZtamong the plurality of energy propagation paths, where for eachx(, ybztcoordinate contained within the energy directing 4D surface or 5D surfaces, there is only a single ubpropagation path angle that forms between a first coordinate and the converging 3D point. The 4D function or 5D function ollectively define all4D xlylor 5D xlylz;coordinates and commensurate ulvlpropagation paths that exist for each converging point at XlYlZl

[0272] At a converging coordinate XlYlZla surface is formed and the surface may comprise a point, volume, object or other embodiment comprising a 3D position of converging energy propagation paths. The provided samples for each surface location may comprise one or more surface properties, vectors, materials, characterizations, or other identifying property Vtto characterize or otherwise process the resulting energy, as well as one or more input energy sources striking a given point proximate to the surface location wherein the reflectance function now comprises a generalized vector for the various properties of the surface and represented as an 1 ID universal object parameterization function fr(λixiwiλr, xr, wr, Vi)

[0273] The 11D universal holographic parameterization function fr(λixiwiλr, xr, wr, Vi) defines the resultant values for a given environment and vectorized object properties and the 4D function fi(xlylulVi) defines the energy propagation paths from an energy directing device surface, may therefore be further generalized as an 15D universal holographic parameterization function fr(λixiwiλr, xr, wr(xlylulvl),where the transmitted direction wrdefines and equals the propagation path of ubvbwhereby defining the spatial coordinate xbyLand for each transmitted direction wrthere may be only one fL(xlylulvl) set of values to satisfy wr= ubvbThose skilled in the art will appreciate the various transforms and mathematical constructs in addition to the rendering requirements associated with the disclosed universal parameterization of 4D and 5D holographic sensory energy properties.

[0274] With the complete 15D function describing the vectorization of all sensory energy properties to coincide with surfaces formed from converging points in space, multiple orders of magnitude of required data have been fundamentally eliminated provisioning for a viable path to enabling the transmission of truly holographic datasets.

[0275] The vectorized properties strive to provide accurate physics for each of the sensory domains for properties that may be synthetically programmed, captured, or computationally assessed, whereinmay prescribe attributes for each surface, volume or 3D coordinate XlYlZlvectorized properties about an object for a given sample within a provided dataset for general system metadata or for each or any sensory energy domain, comprising:1.) system metadata may provide for any of the sensory energy specific attributes or system wide references for surface properties for each sample including normals, depthinformation, environmental properties, multiple angular samples for a given 3D coordinate, procedural textures, geometry, point clouds, deep image data, static frames, temporal frames, video data, surface IDs, surface passes, coordinate maps, virtual camera coordinates, virtual illumination and visible energy information, environment maps, scene information outside of the field of the visual sensory sample information, curves, vertices, temporal information, networked data, databases, object recognition, energy devices, external data feeds, sensors for system modifications and interactivity, system status, voice recognition, olfactory detection, auditory detection, facial recognition, somatosensory recognition, gustatory recognition, UI, UX, user profiles, flow and motion vectors, layers, regions, transparency, segments, animation, sequence information, procedural information, displacement maps, or any other scene data that is necessary to provide sufficient data for the appropriate processing of each sample;2.) visual sensory energy may provide surface properties to define the appropriate rendering of visible or non-visible electromagnetic energy, iridescence, luminescence, subsurface scattering, non-local scattering effects, specularity, shadowing, absorbance, transmission, masking, interreflections, albedo, transparency, physics, dynamics, reflection, refraction, diffraction, optical effects, atmospheric effects, frequency, modulation, surface profiles, textures, displacement maps, physics and dynamics to specifically interrelate to other sensory energies and respond based upon provisioned energies (e.g. vibrations of sound altering reflectance properties or tactile material deformation causing surface deformations), layers, regions, transparency, segments, curves, animation, sequence information, procedural information, size of material, environmental conditions, room dynamics, or other related material properties for a surface, environment, room, object, point, volume or the like;3.) auditory sensory energy: vectors related to the placement of localized sound fields, magnitude, amplitude, mass, material propagation parameters, absorbance, transmission, material properties informing acoustic reflectance, diffusion, transmission, augmentation, masking, scattering, localization, frequency dependence or modulation, pitch, tone, viscosity, smoothness, texture, modulus, any other parameters that determine the propagation of acoustic waves within the object, surface, medium or otherwise, physics and dynamics to specifically interrelated to other sensory energies and respond based uponprovisioned energies (e.g. temperature changing the sound of a material), layers, regions, transparency, segments, curves, animation, sequence information, procedural information, size of material, environmental conditions, room dynamics, or other related material properties for a surface, environment, room, object, point, volume or the like;4.) somatosensory energy vectors related to the mechanoreceptors for textures, pressure, thermoreceptors, temperature, surface deformability parameters and vectors defining the range of viscoelastic behaviors of a material measure both viscous and elastic characteristics when undergoing deformations between stress and strain over time which provides for a multiplicity of physics including variables for time, strain, modulus, among other dynamics, layers, regions, transparency, segments, curves, animation, sequence information, procedural information, size of material, environmental conditions, room dynamics, or other related material properties for a surface, environment, room, object, point, volume or other somatosensory parameters;5.) gustatory sensory energy vectors for fragrant, fruity, citrus, woody (resinous), chemical, sweet, mint (peppermint), toasted (nutty), pungent and decayed wherein the vector and the spatial coordinate of the vectorized signals may inform the production for the complex olfactory implementations and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, regions, transparency, segments, curves, animation, sequence information, procedural information, size of material, environmental conditions, room dynamics, or other related material properties for a surface, environment, room, object, point, volume or other gustatory sensory parameters;6.) olfactory sensory energy vectors for sourness, saltiness, bitter (spiciness), sweetness, and savoriness (umami) wherein the vector and the spatial coordinate of the vectorized signals may inform the production for the complex olfactory implementations and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, regions, transparency, segments, curves, animation, sequence information, procedural information, size of material, environmental conditions, room dynamics, or other related material properties for a surface, environment, room, object, point, volume or other olfactory parameters;7.) or other interrelated sensory dynamics based upon physical, synthetic, transmitted, or computational interdependencies from any other sensory sample dataset, sensory systemvectors as needed, designed, or required and any additional sensory properties where parameterization of a particular characteristic is beneficial for the reconstruction, storage, processing or transmission of generalized holographic constructed data.

[0276] With the received dataset comprising 2D data having a single angular sample, 3D data having two or more angular samples in a single dimension, 4D data having a plurality of angular samples in two dimensions, or 5D data having a plurality of angular samples in three or more dimensions.

[0277] For all provided source materials, each source material may undergo additional processes to appropriately prepare for efficient vectorization of the holographic dataset. For any provided source materials that exhibit lower spatial or angular resolution that the energy directing surface, a transformation process may be required in order to accurately convert the originating source to a 4D or 5D dataset.

[0278] For appropriate preparation, in an embodiment, provided 2D or 3D source materials comprise photographic capture from a standard imaging system. Within this sequence of images are rasterized reflections, refractions, transparent elements and other similar examples of material property interaction with physically based illumination.

[0279] In the event that the content is prepared by simply identifying surface IDs for the surfaces with the already rasterized material properties, the effective data may be sufficient for converging into a 4D coordinate system, however, any additional rendering applied to these surfaces will exhibit a double image for the physics of both the photographic, as well as the parameterized synthetic rendered reflectance properties. The ideal source dataset for efficient holographic transmission comprises an albedo representation of the sample source information, plus vectorized material properties for each of the specified energy domains with metadata forming an objectbased volumetric sampling of the albedo multi-view samples, and wherein all material properties provide for accurate surface identification and rendering as well as the localization or projection of other sensory energies accurately based upon the specified vectorized surface properties.

[0280] In an embodiment, manual, semi-automated, computer vision, or automated processes are provisioned to algorithmically or manually assess the content within the source sample dataset, and wherein a manual or algorithmic analysis is performed whereby segmentation and other object isolation methodologies known in the art are performed to identify the regions that includeundesired physically rasterized effects. In an embodiment, a person is photographed in front of a background wherein the material properties of the person include reflections from the environment, and the background objects are occluded by the photographed person. After these regions have been identified as undesirable, a process may be leveraged to 1) isolate the objects in question; 2) separate all object elements into the core components to account for occlusion, transparency, edges, or other element; 3) through image analysis, temporal analysis, energy analysis, with the facilitation of machine learning, computer vision, extra hardware and energy devices that additionally captured information about the scene, objects and / or environment, or through completely manual means, the object elements are provisioned such that any surface that should exhibit a material property has any such baked-in material properties removed through computer vision, algorithms, processors, or manual visual effects wherein the manual processes are generally known in the art for methods to perform wire removals, paint fix, clean plates, image restoration, alpha matte creation, occlusion filling, object recreation, image projection, motion tracking, camera tracking, rotoscope, optical flow, and the like for the purpose of regenerating the intrinsic material property in the absence of the extrinsic material properties thereby preparing the content for the most efficient transmission and propagation for said dataset; 4) An additional process of the above involves the manual or computer assisted identification of depth or 3D coordinate values for each of the desired samples; and 5) Further within this embodiment is the identification of the associated material properties, each of which represent a point, region of data, surface, object or other representation of a material such that the data may easily be further rendered within the energy directing device’s display drivers or within any additional system capable of either encoding and decoding the parameterized dataset.

[0281] In an embodiment, the dataset from the above comprises 3D multi-view samples that are prepared with albedo visual energy samples, each of which having multiple layers of RGBA information, a collection of vectorized material properties to associate each segmented material with a surface ID and series of surface parameters to closely reconstruct the original source dataset prior to the removal of the extrinsic image data, and wherein an acoustic dataset is provisioned with vectorized material properties associated with the material properties of the visual energy system as well as multiple sound channels each having identified frequency, modulation, spatial placement and other sound localization properties, and wherein a somatosensory sensory energy dataset is provided for a subset of the surfaces contained within the visual energy dataset, toadditionally comprise viscoelastic and temperature vectorized material properties, both of which are correlated to the other vectorized datasets.

[0282] From any provided dataset, each provided sample from the visual energy dataset is assessed for a relative depth position in relation to the energy directing device surface, and wherein each of the samples for any of the visual energy samples are placed into a 3D coordinate system, and wherein the energy propagation path length for each of the provided samples is assessed in relation to the function that correlates each 3D coordinate in relation to the plurality of coexisting converging energy propagation paths that intersection a first 3D point at location XbYbZ] among the plurality of energy propagation paths within the 4D or 5D plenoptic functions, where for each X](yi or X[, ybZi coordinate contained within the energy directing 4D surface or 5D surfaces, there is only a single ubV] propagation path angle that forms between a first coordinate and the converging 3D point. The 4D function fz(x1(ybubvj or 5D function fz(xbybz1(ubV]) collectively define all 4D xbybor 5D xbybZi coordinates contained within the energy directing device and commensurate ubV] propagation paths that exist for each converging point at X1(Y1(Z] and wherein the total number of samples per presented or available 4D xbybor 5D xbybztspatial coordinates is known after performing this analysis process, and wherein the total energy propagation path length between each 3D point at location XbY1(Z] to the 4D or 5D coordinate location is known, and wherein a weighted distribution based upon total available samples per 4D or 5D coordinate and minimum path length to the sampled 3D coordinate values from the available plurality of 3D coordinate data provides for a complete sampling of the 4D or 5D light field from an arbitrary dataset.

[0283] As a further embodiment of the above, after each of the samples for any of the 1) visual, acoustic, somatosensory, and any other provided energy samples are 2) placed into a 3D coordinate system based upon the provided dataset, additional processing, or additional vectorized properties, and before performing a coordinate analysis; 3) the 15D universal holographic parameterization function fr(Ai, xbwbAr, xr, wr(xby1(ubV[), Vj is assessed wherein 4) additional known environmental scene, geometry, metadata or the like is provided, each with independent vectorized material properties; 5) virtual illumination information is provided and the additional sensory energy metadata properties are assessed for any potential interference between the properties that may altering the rendering functions and; 6) the 15D parameterization function assesses for each provided 3D coordinate and commensurate vectorized material property to; 7) perform a renderingprocess through on-line, off-line, real-time, processor, ASIC, FPGA, cloud, or other form of rendering process to result in a new plurality of angularly varying material properties given the arbitrary provided dataset, and wherein 8) the rendering process is specific to each of the transmitted direction wrdefining and equal to each of the propagation paths ubVj , whereby defining the spatial coordinate xbyi, and for each transmitted direction wrthere may be only one fiCx^y^ Ui, Vi) set of values to satisfy wr= ubVj, and wherein 9) based upon the rendered results and resultant available new angularly varying material properties, for each of the 4D or 5D coordinates comprising the energy propagation path length for each of the provided samples are assessed in relation to the function that correlates each 3D coordinate in relation to the plurality of coexisting converging energy propagation paths that intersection a first 3D point at location XbY], Zj among the plurality of energy propagation paths within the 4D or 5D plenoptic functions, where for each xbyj or x1(ybz1coordinate contained within the energy directing 4D surface or 5D surfaces, there is only a single ubV] propagation path angle that forms between a first coordinate and the converging 3D point. The 4D function fz(xbybubVi) or 5D function fz(x1(ybz1(ubV]) collectively define all 4D xby>, or 5D xbybzj coordinates contained within the energy directing device and commensurate ubV] propagation paths that exist for each converging point at XbYbL\ and wherein the total number of samples per presented or available 4D xbybor 5D xbybz\ spatial coordinates is known after performing this analysis process, and wherein the total energy propagation path length between each 3D point at location XbYbZ] to the 4D or 5D coordinate location is known, and wherein a weighted distribution based upon total available samples per 4D or 5D coordinate and minimum path length to the sampled 3D coordinate values from the available plurality of 3D coordinate data provides for a complete sampling of the 4D or 5D light field for all provided sensory energies from an arbitrary dataset.

[0284] An additional embodiment of the above system wherein the rendering additionally accounts for a bidirectional energy directing surface such that sensed electromagnetic energy representing the illumination of the real-world environment, or the absorbance of certain acoustic frequencies within the environment may result in the dynamic or off-line update to the rendering process or other sensed interactive real-world element is assessed, and wherein the illumination and acoustic or other sources are adjusted to accommodate for the modification in environmental conditions.

[0285] Turning back to FIG. 8, in view of the principles disclosed above, in an embodiment of process 800, the received content data may further comprise vectorized material property data, and wherein the process 800 further comprises a step 830, in which digital volumetric representation of the content data is associated with the vectorized material property data; and wherein, in step 804, determining energy source location values is based on at least the vectorized material property data associated with the volumetric representation of the content data.

[0286] Referring to FIGS. 9 and 13, in an embodiment, a vectorization process 1300 may include a step 1302 in which first content data is received and a step 1304 in which identifying a surface 915 in the content data. In an embodiment, identifying the surface 915 may comprise using segmentation data in the content data. The vectorization process 1300 may further include a step 1306 in which a surface identification of the surface 915 is determined and a step 1308 in which material property data of the surface 915 is determined. In an embodiment, determining the material property data may comprise manual determination, or using a predetermined process. After steps 1306 and 1308, the vectorization process 1300 may further include a step 1310 in which the surface identification is associated with the material property data of surface 915. The vectorization process 1300 may further include a step of 1312 in which the vectors of the material property data is created. The vectorization process 1300 may further include step 1314 in which vectorized material property data is generated based on the created vectors.

[0287] In an embodiment, the process 1300 may optionally include a step 1316 in which material property data is removed from the first content data and replaced by the vectorized material property data is generated in step 1314. In an embodiment the vectorized material property data is generated in step 1314 may used in process 800 as discussed above to determine 4D plenoptic coordinates for the energy directing devices of the present disclosure as discussed above.

[0288] The process 1300 may be carried out using any processing system of the present disclosure, including processing system 1200. In an embodiment, content data may be received in step 1302 through the data input / output interface 1201, and steps 1304 thru 1314 of the vectorization process 1300 may be carried out using the vectorization engine 1204. Additionally, the vectorized material property data generated in step 1314 may be used by the sensory data processor 1202 and tracing engine 1206 for processing according to the steps of process 800 as discussed above. Steps 808 and 812 may be performed by the tracing engine to determine 4D coordinates for holographic presentation. Step 810 may be performed by the sensory data processor 1202. The output of theprocessing subsystem may be provided to a compression engine 1210, from which compressed data may be stored in a memory 1208 or provided to the data input out interface 1201 for transmission to an energy directing system either connected locally or remotely to the system 1200. Data may also be stored in memory 1208 until a later time to be retrieved.Holographic Content Generation with Al Model

[0289] In some embodiments, a data processor operable to render instruction for operating the energy modulation systems of the present disclosure to propagate energy for forming a holographic representation, such as the sensory data processor 1202 or any other sensory data processor of the present disclosure, may incorporate an artificial intelligence (Al) model to create holographic content. The Al model may include supervised or unsupervised learning algorithms including but not limited to regression models, neural networks, classifiers, or any other Al algorithm. The Al model may be used to determine viewer preferences based on viewer information recorded by an energy modulation system of the present disclosure, which may include information on a viewer’s behavior.

[0290] The Al model may access information from a data store to create holographic content. For example, the Al model may access viewer information from a viewer profile or profiles in the data store or may receive viewer information from the various components of an energy modulation system. To illustrate, the Al model may determine a viewer enjoys seeing holographic content including animals. The Al model may determine the preference based on a group of viewer’s positive reactions or responses to previously viewed holographic content including animals. That is, the Al model may create holographic content personalized to a set of viewers according to the learned preferences of those viewers. So, for example, the Al model may incorporate animals in the holographic content viewed by a group of viewers. The Al model may also store the learned preferences of each viewer in the viewer profile store of the data store. In some examples, the Al model may create holographic content for an individual viewer rather than a group of viewers.

[0291] One example of an Al model that can be used to identify characteristics of viewers, identify reactions, and / or generate holographic content based on the identified information is a convolutional neural network model with layers of nodes, in which values at nodes of a current layer are a transformation of values at nodes of a previous layer. A transformation in the model isdetermined through a set of weights and parameters connecting the current layer and the previous layer. For example, Al model may include five layers of nodes: layers A, B, C, D, and E. The transformation from layer A to layer B is given by a function Wl, the transformation from layer B to layer C is given by W2, the transformation from layer C to layer D is given by W3, and the transformation from layer D to layer E is given by W4. In some examples, the transformation can also be determined through a set of weights and parameters used to transform between previous layers in the model. For example, the transformation W4 from layer D to layer E can be based on parameters used to accomplish the transformation Wl from layer A to B.

[0292] The input to the model can be an image taken by a tracking system encoded onto the convolutional layer A and the output of the model is holographic content decoded from the output layer E. Alternatively or additionally, the output may be a determined characteristic of a viewer in the image. In this example, the Al model identifies latent information in the image representing viewer characteristics in the identification layer C. The Al model reduces the dimensionality of the convolutional layer A to that of the identification layer C to identify any characteristics, actions, responses, etc. in the image. In some examples, the Al model then increases the dimensionality of the identification layer C to generate holographic content.

[0293] The image from the tracking system is encoded to a convolutional layer A. Images input in the convolutional layer A can be related to various characteristics and / or reaction information, etc. in the identification layer C. Relevance information between these elements can be retrieved by applying a set of transformations between the corresponding layers. That is, a convolutional layer A of an Al model represents an encoded image, and identification layer C of the model represents a smiling viewer. Smiling viewers in a given image may be identified by applying the transformations Wi and W2 to the pixel values of the image in the space of convolutional layer A. The weights and parameters for the transformations may indicate relationships between information contained in the image and the identification of a smiling viewer. For example, the weights and parameters can be a quantization of shapes, colors, sizes, etc. included in information representing a smiling viewer in an image. The weights and parameters may be based on historical data (e.g., previously tracked viewers).

[0294] Smiling viewers in the image are identified in the identification layer C. The identification layer C represents identified smiling viewers based on the latent information about smiling viewers in the image.

[0295] Identified smiling viewers in an image can be used to generate holographic content. To generate holographic content, the Al model starts at the identification layer C and applies the transformations W2 and W3 to the value of the given identified smiling viewers in the identification layer C. The transformations result in a set of nodes in the output layer E. The weights and parameters for the transformations may indicate relationships between an identified smiling viewers and specific holographic content and / or preferences. In some cases, the holographic content is directly output from the nodes of the output layer E, while in other cases the content generation system decodes the nodes of the output layer E into a holographic content. For example, if the output is a set of identified characteristics, the characteristics can be used to generate holographic content.

[0296] Additionally, the Al model can include layers known as intermediate layers. Intermediate layers are those that do not correspond to an image, identifying characteristics / reactions, etc., or generating holographic content. For example, in the given example, layer B is an intermediate layer between the convolutional layer A and the identification layer C. Layer D is an intermediate layer between the identification layer C and the output layer E. Hidden layers are latent representations of different aspects of identification that are not observed in the data, but may govern the relationships between the elements of an image when identifying characteristics and generating holographic content. For example, a node in the hidden layer may have strong connections (e.g., large weight values) to input values and identification values that share the commonality of “laughing people smile.” As another example, another node in the hidden layer may have strong connections to input values and identification values that share the commonality of “scared people scream.” Of course, any number of linkages are present in a neural network. Additionally, each intermediate layer is a combination of functions such as, for example, residual blocks, convolutional layers, pooling operations, skip connections, concatenations, etc. Any number of intermediate layers B can function to reduce the convolutional layer to the identification layer and any number of intermediate layers D can function to increase the identification layer to the output layer.

[0297] In one embodiment, the Al model includes deterministic methods that have been trained with reinforcement learning (thereby creating a reinforcement learning model).

[0298] Reinforcement learning is a machine learning system in which a machine learns ‘what to do’— how to map situations to actions— so as to maximize a numerical reward signal. The learneris not told which actions to take (e.g., generating prescribed holographic content), but instead discovers which actions yield the most reward (e.g., increasing the quality of holographic content by making more people cheer) by trying them. In some cases, actions may affect not only the immediate reward but also the next situation and, through that, all subsequent rewards. These two characteristics— trial-and-error search and delayed reward— are two distinguishing features of reinforcement learning.

[0299] Reinforcement learning is defined not by characterizing learning methods, but by characterizing a learning problem. Basically, a reinforcement learning system captures those important aspects of the problem facing a learning agent interacting with its environment to achieve a goal. That is, in the example of generating a scene for a holographic character, the reinforcement learning system captures information about viewers. Such an agent senses the state of the environment and takes actions that affect the state to achieve a goal or goals (e.g., creating a scene for which the viewers will cheer). In its most basic form, the formulation of reinforcement learning includes three aspects for the learner: sensation, action, and goal. The learner senses the state of the environment with sensors of a tracking system, displays holographic content to the viewers in the environment, and achieves a goal that is a measure of the viewer’s reception of that scene.

[0300] One of the challenges that arises in reinforcement learning is the trade-off between exploration and exploitation. To increase the reward in the system, a reinforcement learning agent prefers actions that it has tried in the past and found to be effective in producing reward. However, to discover actions that produce reward, the learning agent selects actions that it has not selected before. The agent ‘exploits’ information that it already knows in order to obtain a reward, but it also ‘explores’ information in order to make better action selections in the future. The learning agent tries a variety of actions and progressively favors those that appear to be best while still attempting new actions. On a stochastic task, each action is generally tried many times to gain a reliable estimate to its expected reward. For example, if the processor creates holographic content that the processor knows will result in a viewer laughing after a long period of time, the processor may change the holographic content such that the time until a viewer laughs decreases.

[0301] Further, reinforcement learning considers the whole problem of a goal-directed agent interacting with an uncertain environment. Reinforcement learning agents have explicit goals, can sense aspects of their environments, and can choose actions to receive high rewards (i.e., a roaringcrowd). Moreover, agents generally operate despite significant uncertainty about the environment they face. When reinforcement learning involves planning, the system addresses the interplay between planning and real-time action selection, as well as the question of how environmental elements are acquired and improved. For reinforcement learning to make progress, important sub problems have to be isolated and studied, the sub problems playing clear roles in complete, interactive, goal-seeking agents.

[0302] The reinforcement learning problem is a framing of a machine learning problem where interactions are processed and actions are carried out to achieve a goal. The learner and decisionmaker is called the agent (e.g., a data processor). The thing it interacts with, comprising everything outside the agent, is called the environment. These two interact continually, the agent selecting actions (e.g., creating holographic content) and the environment responding to those actions and presenting new situations to the agent. The environment also gives rise to rewards, special numerical values that the agent tries to maximize over time. In one context, the rewards act to maximize viewer positive reactions to holographic content. A complete specification of an environment defines a task which is one instance of the reinforcement learning problem.

[0303] To provide more context, an agent and environment interact at each of a sequence of discrete time steps, i.e. t = 0, 1, 2, 3, etc. At each time step t the agent receives some representation of the environment's state st. The states st are within S, where S is the set of possible states. Based on the state st and the time step t, the agent selects an action at (e.g., making the holographic character do the splits). The action at is within A(st), where A(st) is the set of possible actions. One time state later, in part as a consequence of its action, the agent receives a numerical reward rt+1. The states rt+1 are within R, where R is the set of possible rewards. Once the agent receives the reward, the agent selects in a new state st+1.

[0304] At each time step, the agent implements a mapping from states to probabilities of selecting each possible action. This mapping is called the agent's policy and is denoted 7it where 7rt(s,a) is the probability that at = a if st = s. Reinforcement learning methods can dictate how the agent changes its policy as a result of the states and rewards resulting from agent actions. The agent's goal is to maximize the total amount of reward it receives over time.

[0305] This reinforcement learning framework is flexible and can be applied to many different problems in many different ways (e.g. generating holographic content). The framework proposes that whatever the details of the sensory, memory, and control apparatus, any problem (or objective)of learning goal-directed behavior can be reduced to three signals passing back and forth between an agent and its environment: one signal to represent the choices made by the agent (the actions), one signal to represent the basis on which the choices are made (the states), and one signal to define the agent's goal (the rewards).

[0306] Of course, the Al model can include any number of machine learning algorithms. Some other Al models that can be employed are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. Whatever the case, generally, an input for a machine learning model can help create holographic content in response. Similarly, the Al model may direct the rendering of holographic content.RENDERING A 4D ENERGY FIELD FROM A 3D ENVIRONMENT

[0307] Further contemplated in the present disclosure are embodiments of an inverse tracing process that allows for rendering digital volumetric representations to a format compatible with the energy directing device 1000 with inverse mapping that provides for an inverse path to sample the digital volumetric representation of content data which can be presented as a 3D environment.

[0308] Rendering a 4D energy field from a 3D environment poses many challenges. For explanatory purposes, some of these challenges are described in context of a light field. But, the disclosures of this application are applicable to other forms of energy. And, these descriptions do not limit application of these principles to other energy forms.

[0309] Energy in a 3D environment interacts with objects in a 3D environment in a variety of different ways depending on the physical properties of the objects in the 3D environment that may model how energy is generated, reflected, absorbed, transmitted or otherwise impacted by these objects. FIG. 14 illustrates how some interactions between energy and objects throughout a 3D environment impact the perception of a scene from that environment. Illustrated is an observation point 1402 of a virtual observer in a 3D environment 1400 with a field of view 1404. Energy rays 1406 can reach the observation point directly from an energy source 1408 or by bouncing off reflective objects 1410A, 1410B, 1410C that affect the attributes of the energy when it arrives at the observation point 1402. Various attributes of energy in the 3D environment 1400 may be considered, including of visual, auditory, somatosensory, gustatory, olfactory, vestibular or other desired energies as described throughout the present disclosure. Note, that FIG. 14 only partiallydepicts some energy paths towards an observation point. The sources for some of the energy paths and other reflections may be omitted from the figure, as will be appreciated. In some cases, a single energy path 1406 A, reflects off multiple objects 1410A,1410C — some of which 1410C may be out of the observation point’s 1402 field of view 1404 — on its way to the observation point 1402. On the way to observation point 1402, energy 1406 can also pass through partially transmissive objects 1412, or refractive objects (not shown) that alter how energy is sensed when it reaches the observation point 1402. Thus, accurate rendering of a scene from a 3D environment may take some accounting for the multitudes of interactions that occur both in and out of the field of view 1404 from the observation point 1402 of a virtual viewer in the 3D environment. This can be further complicated in a holographic system because holographic systems may require simultaneous projection of a large number of unique energy convergences. FIG. 14 is presented for illustrative purposes and oversimplifies some possible aspects for ease of explanation. It will be appreciated that a 3D environment may account for many additional physical properties that impact the propagation of energy in the 3D environment, and FIG. 14 does not limit the embodiments of this disclosure.

[0310] Prior art techniques to render 4D energy-field data from a 3D environment include double frustrum rendering, oblique rendering, and multiview rendering. Double frustrum rendering is a multi-pass rendering method that fails to account for data outside the field of view of the observation points and requires computationally complex post-hoc integration of data from the multiple rendering passes. Oblique rendering also requires complex post-hoc computation, requires thousands to tens of thousands of rendering passes, and makes unrealistic assumptions about physical energy-projection systems like perfect projection optics, and no aperture. Oblique rendering also fails to integrate simply into existing workflows. And, finally, multi-view rendering, does not allow for simultaneous rendering out of the field of view and in the field of view, and presents some of the other problems presented by double frustum rendering and oblique rendering.

[0311] Embodiments of the present application allow a system and method for single-pass rendering of a 3D environment into a data set for 4D energy-projection. The embodiments of this disclosure produce noticeably smoother and more cohesive energy-field projections at least because the single-pass rendering allowed by the present disclosure avoids computationally- challenging post-hoc integration efforts. And, because the methods and systems of this disclosure account for both in-screen and off-screen effects. In embodiments, the system and method mayinclude wavelength / energy considerations, and display-specific calibration (or may include any other functions to move seamlessly between various systems). This may additionally include pixel processing with displacement maps to compute other 4D coordinates if and when it is advantageous.

[0312] FIG. 15 is a flow chart illustrating an embodiment of a process for rendering a 4D energyfield from a 3D environment. With reference to FIG. 15 and 16, the process may include a first step 1510 for providing a scene 1600 in a 3D environment described by a plurality of energy-data points 1602 located throughout the scene 1600. A visual representation of a scene 1600 in a 3D environment is depicted in FIG. 16. As can be appreciated, energy-data points 1602 may be configured in different ways in different embodiments of a 3D environment. The energy-data points 1602 may comprise a value, or collection of values that describes various physical properties that determine the virtual appearance of the scene 1600 and how elements in the scene 1602 may interact. Some examples of this are color, reflectance, material properties, and energy directions. Other properties that may be considered in scene 1600 may include visual, auditory, somatosensory, gustatory, olfactory, vestibular or other desired energies. In some embodiments the energy-data points may store a multidimensional function. The function may comprise values that correspond to properties in the physical world that describe how energy may be reflected, refracted, transmitted, or otherwise affected as energy intersects with the energy-data points 1602. And, in some embodiments, the energy-data points 1602 may also comprise intensity and directional information that allow the energy-data points to collectively model propagation of energy in the scene 1600. It should also be noted that the energy-data points 1602 may contain energy values for other types of energy in addition to, or instead of, information related to electromagnetic energy in the visible spectrum. There are no limits to the type of information that can be stored as a value in an energy-data point 1602.

[0313] It will be appreciated that there are various ways to configure or arrange energy data points 1602 in a 3D environment. The density of energy data points 1602 in a 3D environment 1602 may vary from embodiment to embodiment. Or, in some embodiments, the density of energy data points 1602 may vary from location to location in a 3D environment 1600, or object to object 1600 in a 3D environment.

[0314] In some embodiments of 3D environments 1600, the energy-data points 1602 may correspond to locations in a virtual or physical space. And, the values of the energy-data points1602 may change as different objects occupy the space where the energy-data points are mapped. The values of the energy-data points 1602 may also change as light propagates through an embodiment of a 3D environment. And, again, it should be noted that while FIG. 16 depicts a scene in an electromagnetic 3D environment, this disclosure is not limited to energy in that domain. Embodiments of the 3D environment may describe other types of energy in addition to electromagnetic energy in the visual spectrum, such mechanical energy in the form of ultrasound waves which may be used to create tactile surfaces. Other energies described may include visual, auditory, somatosensory, gustatory, olfactory, vestibular or other energies.

[0315] Again with reference to FIG. 15 and FIG. 16, in embodiments the process for rendering a 4D energy-field from a 3D environment may comprise a second step 1520 comprising locating a plurality of virtual pixels 1604 on a virtual pixel plane 1606 in the scene 1600 wherein each pixel has a known unique 4D coordinate that comprises a 2D angular coordinate and a 2D spatial coordinate wherein the 2D angular coordinate describes an angular correlation between each virtual pixel 1604 and a virtual viewing location of a plurality of virtual viewing locations 1608 located on a virtual viewing plane 1610 (as shown by the rectangular box in dashed lines labeled as 1610 in FIG. 16) in the scene 1600 and wherein the 2D spatial coordinate identifies the location of a virtual aperture of a plurality of virtual apertures 1612 disposed on a virtual display plane 1614 in the scene 1600. In some embodiments, the virtual viewing locations 1608 may be located on two or more virtual viewing planes allowing rendering for coplanar devices.

[0316] It will be appreciated that the term “virtual pixel” does not limit the embodiments of this disclosure to rendering 3D environments comprising light fields. The virtual pixels may comprise energy locations in 3D environments comprising any form of detectable energy field including, but not limited to haptic and acoustical fields. Other energy fields may include visual, auditory, somatosensory, gustatory, olfactory, vestibular or other desired energy fields. The plurality of virtual pixels 1604 may be arranged to form a two-dimensional array of virtual pixels. In some embodiments, the plurality of virtual pixels 1604 may be arranged in different ways. For example embodiments allow the virtual pixels to be arranged to form a curved surface, or any other shape. Similarly the plurality of virtual viewing locations 1608 are allowed to be arranged to form different shapes in different embodiments. In some embodiments, the virtual pixel plane 1606 and the virtual display plane 1614 are parallel. In some embodiments the virtual pixel plane 1606 andthe virtual viewing plane 1610 are parallel. In some embodiments, the virtual pixel plane 1606, the virtual display plane 1614, and the virtual viewing plane 1610 are parallel.

[0317] The location of the plurality of virtual pixels 1604 and the plurality of virtual apertures 1612 relative to the plurality of virtual viewing location 1608 in the scene 1602 may correspond to the location of a plurality of energy sources or locations, and a plurality of apertures in a physical 4D energy-projection system relative to the intended viewing volume for that 4D energy - proj ection system. And, in some embodiments, the number and locations of the plurality of virtual pixels 1604, plurality of virtual apertures 1612, and the plurality of virtual viewing locations 1608 may be varied to render data for different 4D energy-projection systems depending on the number of energy sources or locations, field of view, number of apertures, waveguides or other features of the physical 4D energy projection system. In some embodiments, an algorithm may be used to calibrate defined numbers and locations of the plurality of virtual pixels 1604, plurality of virtual apertures 1612, and the plurality of virtual viewing locations 1608 to any type of 4D energyprojection system. It will also be appreciated that while FIG. 16 only depicts a small number of virtual pixels 1604, virtual apertures, 1612, and virtual viewing locations 1608, embodiments may comprise tens, hundreds, thousands or more of each of these elements. Furthermore, the virtual display plane 1614 may correspond to the location of an energy-projection surface or screen in a physical 4D energy-projection system.

[0318] Referring again to FIG. 15 and FIG. 16, in embodiments the process for rendering a 4D energy-field from a 3D environment may comprise a third step 1530 comprising sampling energy data points of the plurality of energy-data points 1602 in the scene 1600 from the virtual viewing plane 1610 along a plurality of rays 1616.Each ray of the plurality of rays 1616 intersects one virtual viewing location of the plurality of virtual viewing locations 1608 and one virtual pixel of the plurality of virtual pixels 1606 at an angle determined by the 2D angular coordinate of the one virtual pixel of the plurality of virtual pixels 1604. Each ray of the plurality of rays 1616 intersects one virtual aperture of the plurality of virtual apertures 1612, the virtual aperture determined the 2D spatial coordinate of the ray. Together, the 2D spatial coordinate and the 2D angular coordinate form a single 4D energy field coordinate. For example, a first ray 1618 of the plurality of rays 1616 intersects with a first virtual pixel 1620 of the plurality of pixels 1604 and a first virtual viewing location 1622 of the plurality of virtual viewing locations 1608 at an angle determined by the 2D angular coordinate of the known unique 4D coordinate of the first virtual pixel 1620. Thefirst ray 1618 also intersects a first aperture 1624 of the plurality of virtual apertures 1612 that is identified by the 2D spatial coordinate of the first aperture.

[0319] FIG. 17 further illustrate the relationships described by the known unique 4D coordinate of the virtual pixels in some embodiments of the method or system for rendering a 4D energy field from a 3D environment of this disclosure by partially depicting a scene 1700 in a 3D environment. It will be appreciated that many more virtual pixels, virtual apertures, and virtual viewing locations may be located in a scene 1700. To render data for a 4D energy field, some embodiments may comprise a very large number of virtual pixels and virtual viewing locations to provide an energy field that is perceived as seamless. The 2D angular coordinate for one of the virtual pixels 1702, 1704, 1706 describes the angular correlation between the pixels 1702, 1704, 1706 and virtual viewing locations L 1 , L2, L3. For example, the 2D angular coordinate (U1 , VI ) for the first virtual pixel 1702 describes the angular correlation between pixel 1702 and virtual viewing location L3. The 2D angular coordinate (U2, VI) for the second virtual pixel 1704 describes the angular correlation between pixel 1704 and virtual viewing location L2. And, the 2D angular coordinate (U3, VI) for the third virtual pixel 1706 describes the angular correlation between pixel 1706 and virtual viewing location LI. In some embodiments, the angular coordinate may correspond to a 2D angular coordinate in a 4D energy-projection system that defines the unique direction of a first propagation path as described elsewhere in this disclosure. Some embodiments, like the one depicted in FIG. 16 and FIG. 17, require only one ray intersecting with any one virtual pixel of the plurality of virtual pixels for a specific energy type.

[0320] FIG. 17 also depicts a 2D spatial coordinate (X1,Y1) that identifies the location of one virtual aperture 1708 of the plurality of virtual apertures. The 2D spatial coordinate may correspond to a 2D spatial coordinate in a 4D light field coordinate for a physical 4D energy directing system as disclosed elsewhere in this disclosure, such as the embodiment discussed herein with respect to FIGS. 7C and 7D. Together, the 2D angular coordinate and the 2D spatial coordinate define a unique 4D coordinate. The unique 4D coordinates may correspond to 4D light field coordinate for a physical 4D energy-projection system. In embodiments like the one depicted in FIG. 17 the first virtual pixel 1702 has a 4D coordinate (XI, Yl, Ul, VI), the second virtual pixel 1704 has a 4D coordinate (XI, Yl, U2, VI), and the third virtual pixel 1706 has a 4D coordinate (XI, Yl, U3, VI). In some embodiments, like the ones shown in FIG. 16 and FIG. 17, multiple rays may converge at each virtual aperture 1708 of the plurality of virtual apertures. Thecoordinates for the first 1702, second 1704 and third 1706 virtual pixels share the same spatial coordinate so the rays 1710 of those virtual pixels converge at the same virtual aperture 1708. But, these rays intersect with different virtual viewing locations LI, L2, and L3 because of the 2D angular coordinate differ among the virtual pixels.

[0321] In some embodiments of the systems and methods of this disclosure at least one ray 1710 of the plurality of rays intersects each virtual viewing location LI, L2, L3 of the plurality of virtual viewing locations.

[0322] Multiple rays 1802 may also converge at a single viewing location L2, in some embodiments, as depicted in FIG. 18, which further illustrates the relationships described by the known unique 4D coordinates. In some of those embodiments, only one ray of the rays 1802 that converge at one virtual viewing location L2 can intersect with the same virtual aperture. As depicted in FIG. 18, one of the converging rays 1802 intersects with (XI, Yl), one intersects with (X2, Yl) and one intersects with (X3, Yl). FIG. 17 and 18 are for illustrative purposes and do not limit the embodiments of this disclosure. As will be appreciated, even though FIG 16, FIG. 17, and FIG. 18 depict a small number of virtual pixels, virtual apertures, rays and virtual viewing locations different embodiments may comprise tens, hundreds, thousands, or other numbers of these elements. It should be appreciated that virtual pixels may be located anywhere on a virtual plane. And, virtual locations may be located anywhere on the virtual viewing plane. Also, each of these figures are stylized representations that may vary from embodiments. And the disclosures of this application may be applied to any type of energy; they are not limited to electromagnetic energy in the visible spectrum.

[0323] Returning to FIG. 16, in embodiments, the plurality of rays 1616 extend beyond the virtual pixel plane 1606. In some embodiments the plurality of rays 1616 may extend indefinitely beyond the virtual pixel plane 1606. This allows sampling of energy-data points from objects in front 1628A, and objects such as the photograph 1628B behind the virtual pixel plane 1606.

[0324] In some embodiments the energy data points 1602 along the plurality of rays 1616 are sampled by tracing the plurality of rays 1616 from the virtual viewing plane 1610 through the virtual display plane 1614, through the virtual pixel plane 1606 and beyond. This allows the method and systems of this disclosure to sample energy data points that capture information from objects such as the table 1628A in front of the virtual display plane 1610 and objects such as the photograph 1628B behind the virtual display plane 1610 in a single pass. This also allows thesystems and methods of this disclosure to accurately account for multitudes of energy interactions, as described in more detail in other parts of this disclosure that occur outside and inside the field of view from an observation point in a single pass. This eliminates the need for difficult, burdensome, and imperfect post-hoc computations to try to integrate multiple rendering passes.

[0325] FIG. 19 depicts a scene from a 3D environment from a virtual viewing location 1906 that further illustrates how embodiments of the systems and methods of this disclosure allow for the multitudes of various energy interactions to be accounted for in a single pass. Embodiments of the systems and methods of this disclosure allow data points 1902 to be sampled along a plurality of rays 1904 from a virtual viewing location 1906. It will be appreciated that FIG. 19 is provided for illustrative purposes and does not limit the embodiments of this disclosure. For clarity, FIG. 19 does not depict the virtual pixels or the virtual apertures. And, the size of the viewing location 1906 may vary from embodiment to embodiment. For the purposes of this illustration the virtual viewing location 1906 has a field of view 1908. The size of the field of view may vary from embodiment to embodiment and, in cases, may depend on the type of physical 4D energyprojecting system. And, as will be appreciated, the size of the field of view 1908 can be determined or varied for different purposes. Sampling energy data points 1902 along the plurality of rays 1904 account for objects, like example objects 1910A, 1910B, in the field view 1908, and energy reflected by objects inside the field of view 1908. In this example, energy from a person (not shown) outside the field of view 1908 is reflected off a mirror 1910B in the field of view 1908 creating a reflection 1912 in the mirror 1910B that is sampled by the rays 1904. FIG. 19 is provided for illustrative purposes and does not limit the embodiments of this disclosure. And, although FIG. 19 depicts a 3D light environment, embodiments of the systems and methods of this disclosure are applicable other sensory and energy domains.

[0326] Referring once again to FIG. 15, in embodiments the process for rendering a 4D energyfield from a 3D environment may comprise a fourth step 1540 comprising correlating the energy data points sampled along each ray to an energy value for the one virtual pixel of the plurality of virtual pixels.

[0327] This step 1540 allows embodiments of the systems and methods of this disclosure to account for varying properties of the objects in the path of the ray. For example, in some embodiments, an energy value for a virtual pixel (not shown in FIG. 19) is correlated to the energy data points 1902 sampled along the lone ray that intersects the virtual pixel (not shown in FIG.19). For example, ray 1916 intersects with a partially reflective object 1920 and a partially transmissive object 1918 that is located between the partially reflective object 1920 and virtual viewing location 1906. To render data for this scene 1900, the system or method of this disclosure accounts for how the partially transmissive object 1918 affects light reflected from the partially reflective 1920 object through the partially transmissive object 1918 to the virtual viewing location 1906. For example, the partially reflective object 1920 may have a spectral radiance function, which when illuminated by the lighting source (not shown), will distribute energy in different directions, which will be attenuated by the partially transmissive object 1918 before reaching the virtual viewing location 1906. Embodiments may also allow energy value for different data point to be weighted based at least on the transmissive or reflected properties of the other energy data points along the same ray. The systems and methods of the present disclosure allow an energy value for the virtual pixel (not shown in FIG. 19) to be correlated to energy data points along the length of the ray 1916 to account for these types of varying properties. As will be appreciated, a variety energy data points may be modeled to account any number of different properties in different embodiments some that may be specifically tailored for different energy domains. Determining energy values can be very difficult for multi-pass rendering systems because they must attempt to integrate the renders and all the information necessary to model the multitudes of possible interactions along the three-dimensional interfaces of each render to arrive at an energy value. The systems and methods of the present disclosure allow those burdensome and inaccurate measures to be avoided.

[0328] The process for rendering a 4D energy-field from a 3D environment may comprise a fifth step 1550 that comprises rendering the energy values of the one virtual pixels of the plurality of virtual pixels and the known unique 4D coordinates of the one virtual pixels of the plurality of virtual pixels into a data set having a format operable for instructing an energy device to output a 4D energy field. In some embodiments, the systems and methods of this disclosure may allow the data set to comprise any data set described, or referenced elsewhere in this disclosure including 4D, 5D, 10D, 11D, or 15D functions. It will be appreciated that other types of data sets may be rendered for different embodiments of 3D environments that account for different kinds of data. In some embodiments, rendering may calibrate the datasets for different types of 4D energy projecting systems. Different types of calibration may be applied to render data for 4D energy directing systems with different architectures, elements, or numbers of elements. For example,calibration may be required for systems with different fields of view or with different numbers or locations of energy sources or other components. In some embodiments of the systems and methods of this disclosure the energy-data points 1902 comprise a value describing at least one of the following: an energy frequency, an energy intensity, an energy transparency, an energy refractivity, an energy reflectivity.

[0329] Some embodiments of the method for rendering a four-dimensional energy field from a three-dimensional environment of this disclosure may allow the method to be calibrated for energy devices that project a 4D energy field. As will be appreciated, the location of the physical energy sources and energy propagation paths in a 4D projection system may differ from the theoretical locations of the virtual pixels and the theoretical angular correlation between virtual pixels and virtual viewing locations. This may be due to small variations from energy device to energy device or even waveguide to waveguide of an energy device. For example slight imperfections in a waveguide may cause energy propagation paths to slightly deviate from their expected direction. And, an energy location may vary slightly from a corresponding virtual location in a 3D environment, which also may cause deviations in the direction of energy propagation Methods for calibration are disclosed in co-owned US Patent 11156771 titled “Method of Calibration for Holographic Energy Directing Systems,” which is incorporated by reference herein .

[0330] FIG. 20 is a schematic diagram illustrating inverse tracing in a three-dimensional environment. The method demonstrated in FIG. 20 may include providing a scene 21000 in a 3D environment described by a plurality of energy-data points located throughout the scene. In an embodiment, providing the scene 21000 may be implemented according to the embodiments described above. Also demonstrated in FIG. 20 is locating a plurality of virtual pixels 21012 on a virtual pixel plane 21004 in the scene 21000 wherein each virtual pixel of the plurality of virtual pixels 21012 has a known unique 4D coordinate that comprises a 2D angular coordinate and a 2D spatial coordinate wherein the 2D angular coordinate of each virtual pixel of the plurality of virtual pixels 21012 describes an angular correlation between the virtual pixel and a virtual viewing location of a plurality of virtual viewing locations 21008 located on a virtual viewing plane 21006 in the scene 21000 and wherein the 2D spatial coordinate of each virtual pixel identifies the location of a virtual aperture of a plurality of virtual apertures 21010 located on a virtual display 21019 plane in the scene 21000. In an embodiment, locating the plurality of virtual pixels 21012 may be implemented according to the embodiments described above.

[0331] Further demonstrated in FIG. 20 is sampling energy data points of the plurality of energydata points in the scene 21000 along a plurality of rays 21020 from the virtual viewing plane 21006 wherein each ray of the plurality or rays 21020 intersects one virtual viewing location of the plurality of virtual viewing locations 21008 and one virtual pixel of the plurality of virtual pixels 21012 at an angle determined by the 2D angular coordinate of the one virtual pixel intersected by the ray and wherein each ray intersects one virtual aperture determined by the 2D spatial coordinate of the one virtual pixel intersected by the ray. In an embodiment, sampling data points may be implemented according to the embodiments described above.

[0332] Further demonstrated in FIG. 20 is correlating the energy data points sampled along each ray of the plurality of rays 21020 to an energy value for the one virtual pixel of the plurality of virtual pixels 21012 intersected by the ray. In an embodiment, correlating the energy data points sampled may be implemented according to the embodiments described above. The energy value of the one virtual pixel of each ray and the known unique 4D coordinates of the one virtual pixel of each ray may be rendered into a data set having a format operable for instructing an energy device to output a 4D energy field. In an embodiment, rendering may be implemented according to the embodiments described above.

[0333] As depicted in FIG. 20 the plurality rays 21012 may be inversely traced from the virtual viewing plane through the virtual pixel plane 21004 and beyond to capture information about the energy data points behind the virtual pixel plane 21004. This allows the method for rendering 4D energy fields from a 3D environment to account for energy reflectance behind the virtual pixel plane even if the reflectance comes from distant objects 21013.

[0334] FIG. 20 also demonstrates how embodiments of this method may account for a partially transmissive object 21015. As depicted, energy data is sampled through the partially transmissive object 21015 past the virtual pixel display 21004 where it will sample data reflected from objects 21013. In some embodiments, energy values can then be correlated according to the information sampled by weighting the energy data 21014 of the objects 21013 and the partially transmissive object 21015 to arrive at an energy data values for virtual pixels of the array of virtual pixels.

[0335] It is to be appreciated that in an embodiment, the inverse tracing and mapping process of FIGS. 19 and 20 may allow for a single process step of inverse raytracing to the desired 4D display specific volumetric energy representation (e.g., image). This may include wavelength / energy considerations, and display specific calibration (or may include any other functions to moveseamlessly between various systems). This may additionally include pixel processing with displacement maps to compute other 4D coordinates if and when advantageous.

[0336] In some embodiments of the systems and methods of this disclosure one ray of the plurality of rays intersects each virtual pixel of the plurality of virtual pixels.

[0337] In some embodiments of the systems and methods of this disclosure the virtual display plane corresponds to a waveguide system of an energy directing device, and energy is operable to be directed through the waveguide system according to the data set to form a detectable volumetric representation of at least a portion of the scene.

[0338] In some embodiments of the systems and methods of this disclosure the plurality of virtual pixels correspond to a plurality of energy locations on a first side of the waveguide system.

[0339] In some embodiments of the systems and methods of this disclosure the data set further comprises vectorized material property data.

[0340] In some embodiments of the systems and methods of this disclosure at least a portion of the method is carried out in real time.

[0341] In some embodiments of the systems and methods of this disclosure the method is entirely carried out in real time.

[0342] In some embodiments of the system and method of this disclosure at least two portions of the method are carried out in different time periods.

[0343] In some embodiments of the system and method of this disclosure at least one ray of the plurality of rays intersects each virtual aperture of the plurality of virtual apertures.

[0344] In some embodiments of the system and method of this disclosure the data set describes at least one signal perceptible by a visual, audio, textural, sensational, or smell sensor.

[0345] In some embodiments of the system and method of this disclosure the energy data points sampled along each ray of the plurality of rays are simultaneously correlated to energy values.

[0346] In some embodiments of the system and method of this disclosure the data set is stored in a binary file format.In some embodiments of the system and method of this disclosure energy data points of the plurality of energy-data points are sampled along each ray of the plurality of rays from the virtual viewing plane through the one virtual aperture of the plurality of virtual apertures and beyond the one virtual pixel of the plurality of virtual pixels.

[0347] In some embodiments of the system and method of this disclosure the steps may be repeated indefinitely. In some embodiments this may allow a scene to be dynamically captures the scene changes throughout time.

[0348] In embodiments the method for rendering a 4D energy field from a 3D environment of this disclosure further comprises storing information about the virtual viewing locations and the 4D coordinates of the virtual viewing locations into a ray file that can provide instructions for sampling data along any rays that intersect with the virtual viewing locations from those virtual viewing locations. The ray file is literally just the mapping for what samples are rendered on the virtual viewing plane, and it may contain a list of 3D virtual viewing plane coordinates and the associated 2D (u,v) angular coordinates. As an example, some embodiments, the ray file may be used with CG scene information such as scene geometry, color, reflections, etc. to effectively sample a synthetic 4D light field. It will be appreciated that the information stored in the ray file may vary depending on the type of energy device that will be utilized to output the 4D energy field because the number of pixels and the location of allowable observation points may vary from energy device to energy device. In some embodiments, the ray file may provide a means to instruct a system how to sample a 3D environment for a given energy device. In other embodiments, the ray file is an ideal mapping for a particular energy field projection device design, and does not reflect relatively small device-to-device differences, requiring any sampling of a scene using the ray file to undergo a further device-dependent correction at a later time. The following discussion refers to FIG. 17 and 18, but, it will be appreciated that the methods described herein apply to embodiments with many more virtual viewing locations, virtual pixels, virtual apertures, and rays. In an embodiment, a 3D spatial coordinate may identify the location of each virtual viewing location L2 and the 2D angular coordinate of the one virtual pixel 1702, 1704, 1706 intersected by every ray 1710. In some embodiments, the ray file associates the 3D spatial coordinate of each virtual viewing location L2 with the 2D angular coordinate of every virtual pixel 1702, 1704, 1706 intersected by every ray that intersects the virtual viewing location. For example, L2 in FIG. 18 is intersected by rays 1802. And, the ray file may associate the angular coordinate of virtual pixels 1803, 1805, and 1807 L2 as well as a 3D spatial coordinate that identifies the location of L2 in the scene. In embodiments, the ray file provides an instruction for each ray 1802 for sampling energy data points of the plurality of energy data points 1802 along the ray from the one virtual viewing location L2 intersected by the ray wherein the instruction is determined at least by the 3D spatial coordinate ofthe one virtual viewing location L2 intersected by the ray and the 2D angular coordinate of the one virtual pixel intersected by the ray. For example, ray file may provide an instruction for ray 1809 determined at least by the 3D spatial coordinate of L2 and the 2D angular coordinate of 1803. As will be appreciated, this method may be used for a variety of applications other than energy fields including rendering data for 2D, stereoscopic, virtual reality, augmented reality, and multi-view display systems.

[0349] In some embodiments of the method for rendering a four-dimensional energy field from a three-dimensional environment of this disclosure rendering the energy value may also further comprise calibrating the energy data for the energy device. As will be appreciated, the theoretical locations for the virtual pixels and theoretical angular correlation between virtual pixels and virtual viewing locations may differ from the physical implementation. This may be due to small variations from energy device to energy device or even waveguide to waveguide of an energy device. For example slight imperfections in a waveguide may energy propagation paths to slightly deviate from their expected direction. And, an energy location may vary slightly from a corresponding virtual location in a 3D environment which also may cause a slight deviation in the direction of an energy propagation. In some embodiments, these deviations may be accounted for by calibrating the energy data for an energy device or specific type of energy device. It will be appreciated that other types of calibrations may also be appropriate.

[0350] The process depicted in FIG. 15 may be carried out using any processing system of the present disclosure or appropriate system known in the art, including processing system 2200 shown in FIG. 21. In embodiments, the processing system may comprise a rendering engine 2210 and a sensory data engine 2220. The sensory data engine 2220 may carry out step 1510 to provide a scene in a 3D environment, in some embodiments. And, the sensory data engine 2220 may also carry out the step 1520 to locate a plurality of virtual pixels on a virtual pixel plane. The rendering engine 2210, of some embodiments, may carry out the step 1530 to sample energy data points. The rendering engine 2210 may also carry out the step 1540 to correlate the energy data points sampled along each ray to an energy value. And, the rendering engine 2210 may also carry out the step to 1550 to render energy values. In some embodiments, system may also comprise a memory 2230 to store data sets.

[0351] It will be appreciated that some forms of energy data may have far fewer virtual viewing pixels and virtual viewing location necessary for rendering a 4D energy field. For example, insome embodiments, each virtual aperture is intersected by two rays. In such embodiments, the plurality of virtual viewing locations comprises two virtual viewing locations. But, such embodiments may be operable for generating a data set operable for instructing an energy device to output a stereoscopic image. It will be appreciated that stereoscopic displays direct a first image to a first eye of a viewer and a second image to a second eye. Consequently, in some embodiments each aperture may only comprise two virtual pixels and there may only be two virtual viewing locations - one for each eye. Such embodiments may allow a data sets with a format operable for instructing an energy device to output a virtual reality image. And, such embodiments may allow data sets with a format operable for instructing an energy device to output a augmented reality image.

[0352] It will also be appreciated that in other embodiments each virtual aperture may be intersected by additional rays corresponding to additional viewing locations or views. Such embodiments may allow data sets with a format operable for a multi-view display wherein the number of views corresponds to the number of rays that intersects each virtual pixel. This principle can be illustrated by FIG. 17 that show three viewing locations LI, L2, L3, and three rays 1710 intersecting one aperture XI, Yl, wherein each ray intersects one virtual pixel 1702, 1704, and 1706. Each viewing location may correspond to one view of a multi- view system. It will be appreciated that some embodiments will many additional viewing locations.Wavefront Sampling

[0353] As discussed above, a wavefront of an energy wave may be understood to be an imaginary surface where the energy wave has the same phase. An energy wave may comprise planar or spherical wavefronts, and may be approximated by a laser. In FIG. 22A, the energy waves are illustrated as propagation paths extending from a laser. A portion of the outputted energy waves 2211 may be used to approximate a plane wave over a distance. In the model shown in FIG. 22B, the outputted energy waves from the laser are shown as spherical wavefronts propagating from a point source. Such a spherical wave would create a pattern when passed through micro-optics, rather than scattering light beyond the focal length as a true plane wave would.

[0354] FIGS. 23 A- J and 24A-H provide illustrations that help to demonstrate a rendering approach based on sampling a wavefront in accordance with the principles disclosed in the presentdisclosure. In an embodiment, a reference energy modulation system 3000 as shown in FIG. 23A comprises a first reference energy modulation surface 3004 located in a first energy propagation position 2324 and a second reference energy modulation surface 3002 located in a second energy propagation position 2322. In an embodiment, the first reference energy modulation surface 3004 is operable to apply a first energy modulation to energy propagated therethrough and the second reference energy modulation surface 3002 is operable to apply a second energy modulation to energy from the first reference energy modulation surface and propagated therethrough. The first and second reference energy modulation surfaces 3004 and 3002 may cooperate to spatially modulate the phase and / or amplitude of an energy wavefront according to an energy modulation function, which may be a phase function, an amplitude function, or a complex amplitude function. In an embodiment, the first reference energy modulation surface 3004 may be configured to spatially modulate energy according to at least a phase function, and the second reference energy modulation surface 3002 may be configured to spatially modulate energy according to at least an amplitude function. In another embodiment, the first reference energy modulation surface 3004 may be configured to spatially modulate energy according to at least an amplitude function, and the second reference energy modulation surface 3002 may be configured to spatially modulate energy according to at least a phase function. In an embodiment, either one of the spatially modulating phase function and amplitude function may be a temporally static function while the other may be a temporally dynamic function. For example, in an embodiment, the second reference energy modulation surface 3002 may be configured to comprise a static optical element while the first reference energy modulation surface 3004 may be configured to comprise an addressable optical element, or vice versa. In an embodiment, both of the spatially modulating phase function and amplitude function may be a temporally dynamic function. It is to be appreciated that while the principles of the present disclosure may be illustrated with a particular example configuration of the first and second reference energy modulation surfaces 3004 and 3002, one skilled in the art would readily understand that the same principles would be applicable to all embodiments and configurations of the of the first and second reference energy modulation surfaces 3004 and 3002 contemplated in the present disclosure.

[0355] In an embodiment, the first reference energy modulation surface 3004 is operable to provide spatially modulated energy and propagate such energy therefrom according to one or more energy wave propagation properties of the reference energy modulation system 3000 through thesecond reference energy modulation surface 3002. The second reference energy modulation surface 3002, according to an embodiment, is operable to provide spatially modulated energy and propagated such energy therethrough. In an embodiment illustrated in FIG. 23 C, the first reference energy modulation surface 3004 may include a plurality of reference modulation areas 3006a, 3006b, and 3006c to provide spatially modulated energy from different areas of the first reference energy modulation surface 3004 and propagate such energy through the second reference energy modulation surface 3002 to and from different directions. While only three reference modulation areas 3006a-c are shown in FIG. 23C, one skilled in the art will readily appreciate that the first reference energy modulation surface 3004 may comprise any number of reference modulation areas.

[0356] FIG. 23D illustrates an embodiment of a reference energy modulation system 3000, which may be an aggregate of a plurality of the reference energy modulation system of FIG. 23C. As illustrated in FIG. 23D, the first reference energy modulation surface 3004 may comprise a plurality of reference modulation areas 3006 and the second reference energy modulation surface 3002 may comprise a plurality of reference modulation regions 3002a-c. Each of the plurality of reference modulation regions 3002a-c of the second reference energy modulation surface 3002 may correspond to a respective number of reference modulation areas 3006. The number of reference modulation areas 3006 corresponding to each of the reference modulation regions 3002a- c need not be the same. According to an embodiment, the first reference energy modulation surface 3004 is located at a first energy propagation position 2324 and the second reference energy modulation surface 3002 is located at a second energy propagation position 2322. The first and second energy propagation positions 2324, 2322 may correspond to respective planes in space. While only a fixed number of reference modulation areas 3006 and reference modulation regions 3002a-c are illustrated in FIG. 23D, one skilled in the art will readily appreciate that the first reference energy modulation surface 3004 may comprise any number of reference modulation areas and the second reference energy modulation surface 3002 may comprise any number of reference modulation regions.

[0357] According to an embodiment, a first energy modulation is operable to be spatially applied to energy propagated through the plurality of modulation areas 3006 of the first reference energy modulation surface 3004, and a second energy modulation is operable to be spatially applied to the energy propagated from the first reference energy modulation surface 3004 and through theIllplurality of reference modulation regions 3002a-c of the second reference energy modulation surface 3002.

[0358] In an embodiment, a reference energy modulation element may be located in a reference modulation area 3006 of the first reference energy modulation surface 3004. The reference energy modulation element may include a pixel-based addressable element of a spatial light modulator (“SLM”) or anon-pixel based addressable element. Examples of SLMs include various electrically addressed spatial light modulator (“EASLM”) and optically-addressed spatial light modulator (“OASLM”). In an embodiment, the reference modulation element in the reference modulation area 3006 may be transmissive, reflective, diffractive, refractive, or any combination thereof. In an embodiment, the reference modulation element in the reference modulation area 3006 may be configured to modulate and propagate either coherent or incoherent energy. Examples of pixelbased addressable elements include liquid crystal (LC), liquid crystal on silicon (LCOS), organic light emitting diodes (OLED), and microLED (pLED), all of which are well known in the art and need not be described in detailed here. For a pixel-based addressable element, a reference modulation element in the reference modulation area 3006 may model a pixel, a plurality of pixels, or a portion of a pixel. Examples of non-pixel based addressable element include various colloids, metamaterials, nanoparticles, and other programmable materials as described in the commonly owned U.S. Patent App. No. 17 / 990,258, all of which is hereby incorporated by reference. The wave propagation properties of the reference energy modulation system 3000 may comprise one or more properties selected from a group consisting of scattering, diffractive, reflective, and refractive properties.

[0359] In a modelling of the reference energy modulation system 3000 of FIG. 23 A, a reference modulation area 3006 in the first reference energy modulation surface 3004 may be assumed, in an embodiment, to be a point source from which modulated energy is propagated through an aperture defined in the second reference energy modulation surface 3002. In some embodiments, the aperture may comprise a modulation region. This model may further assume that the modulated energy propagated through the reference modulation area 3006 in the first reference energy modulation surface 3004 fills the aperture in the second reference energy modulation surface 3002 and is propagated through the aperture according to an ideal thin lens. FIG. 23A shows that energy modulated by an on-axis point source area 3006 of the first reference energy modulation surface 3004 is propagated through an ideal thin lens in the second reference energy modulation surface3002 in a first substantially parallel direction perpendicular to the second reference energy modulation surface 3002 as a planar wavefront 3009. For such a modelling of the reference energy modulation system 3000, as shown in FIG. 23B, a rendering wavefront sample 3030 that is sufficient to determine the rendering data for the reference modulation element in the reference modulation area 3006 may comprise one set of parallel sampling paths 3008 that span the aperture in the second reference energy modulation surface 3002. In an embodiment, the set of sampling paths 3008 extends through a sampling region 3044. The sampling region 3044 may be aligned or located at an energy sampling position 2332 that is based on the second energy propagation position 2322 of the second reference energy modulation surface 3002 of the reference energy modulation system 3000.

[0360] In contrast to FIG. 23A, an embodiment of the reference energy modulation system 3000 shown FIG. 23E shows that energy modulated by an on-axis point source area 3006 of the first reference energy modulation surface 3004 is propagated through an ideal thin lens in the second reference energy modulation surface 3002 as a spherical wavefront 3011 to a point of convergence 3005 rather than as a planar wavefront 3009. In an embodiment, the distance 2350 between the first energy propagation position 2324 and the second energy modulation position 2322 is greater than the focal length of the second reference energy modulation surface 3002. In an embodiment, the distance 2350 is greater than the focal length of the reference modulation regions of the second reference energy modulation surface 3002.

[0361] For the modeling of the reference energy modulation system 3000 of FIG. 23E, a rendering wavefront sample 3030 that is sufficient to determine the rendering data for the reference modulation element in the reference modulation area 3006 may comprise one set of sampling paths 3008 that span the aperture in the second reference energy modulation surface 3002 as shown in FIG. 23F. In an embodiment, the rendering wavefront sample 3030 effectively comprise a spherical wavefront that converges in real space and / or diverges in virtual space. In an embodiment, the point of convergence 3005 for the set of sampling paths 3008 is located in real space or at an off-screen position. In an embodiment, the set of sampling paths 3008 extends through a sampling region 3044. The sampling region 3044 may be aligned or located at an energy sampling position 2332 that is based on the second energy propagation position 2322 of the second reference energy modulation surface 3002 of the reference energy modulation system 3000.

[0362] FIG. 23 G illustrates an embodiment of the reference energy modulation system 3000 in which the first reference energy modulation surface 3004 includes a plurality of reference modulation areas (e.g., 3006a-c) to provide spatially modulated energy from different areas of the first reference energy modulation surface 3004. In an embodiment, the energy from the plurality of reference modulation areas (e.g., 3006a-c) is propagated through the second reference energy modulation surface 3002 to respective points of convergence (e.g., 3005a-c).

[0363] FIG. 23H shows another embodiment of the reference energy modulation system 3000 in which energy modulated by an on-axis point source area 3006 of the first reference energy modulation surface 3004 is propagated through an ideal thin lens in the second reference energy modulation surface 3002 along diverging energy propagation paths as a spherical wavefront 3011. In an embodiment, the distance 2352 between the first energy propagation position 2324 and the second energy modulation position 2322 is less than the focal length of the second reference energy modulation surface 3002. In an embodiment, the distance 2352 is less than the focal length of the reference modulation regions of the second reference energy modulation surface 3002.

[0364] For the modeling of the reference energy modulation system 3000 of FIG....

Claims

What is claimed is:

1. An energy modulation system comprising: a first spatial energy modulator (SEM) located in a first energy propagation position and operable to spatially modulate energy propagated therethrough, the first SEM comprising: a first modulation area comprising a first plurality of energy locations; and a second modulation area comprising a second plurality of energy locations; wherein a first energy modulation is spatially applied to the energy propagated through the first and second plurality of energy locations in the first and second modulation areas; and a second SEM located in a second energy propagation position and operable to spatially modulate energy propagated from the first SEM, the second SEM comprising: a first modulation region; and a second modulation region; wherein a second energy modulation is spatially applied to the energy propagated through the first and second modulation regions; and wherein the first modulation region is operable to modulate the energy propagated from the first modulation area of the first SEM such that the energy propagated through the first modulation area of the first SEM and through the first modulation region of the second SEM comprises a first collection of wavefronts, each wavefront of the first collection having a complex amplitude modulation.

2. The energy modulation system of claim 1 , wherein the first energy modulation comprises a first amplitude modulation applied to the first modulation area and a second amplitude modulation applied to the second modulation area.

3. The energy modulation system of claim 1 , wherein the first energy modulation comprises a first phase modulation applied to the first modulation area and a second phase modulation applied to the second modulation area.

4. The energy modulation system of claim 1 , wherein the first SEM further comprises:a third modulation area comprising a third plurality of energy locations; and a fourth modulation area comprising a fourth plurality of energy locations, wherein the first energy modulation is spatially applied to the energy propagated through the third and fourth plurality of energy locations in the third and fourth modulation areas.

5. The energy modulation system of claim 4, wherein the first energy modulation further comprises a third amplitude modulation applied to the third modulation area and a fourth amplitude modulation applied to the fourth modulation area.

6. The energy modulation system of claim 4, wherein the first energy modulation further comprises a third phase modulation applied to the third modulation area and a fourth phase modulation applied to the fourth modulation area.

7. The energy modulation system of claim 4, wherein the second energy modulation applied by the first modulation region to the energy from the first modulation area of the first SEM comprises a first summation of point spread functions that describe, at a third energy propagation position, a summation of impulse responses to the energy propagated from the first plurality of energy locations in the first modulation area of the first SEM in the first energy propagation position.

8. The energy modulation system of claim 7, wherein the first summation of impulse responses comprises an intensity distribution of the first collection of wavefronts at the third energy propagation position.

9. The energy modulation system of claim 8, wherein the intensity distribution of the first collection of wavefronts comprises a sum of an intensity of each wavefront.

10. The energy modulation system of claim 8, wherein the intensity distribution of the first collection of wavefronts comprises a sum of square of an intensity of each wavefront.

11. The energy modulation system of claim 7, wherein the second modulation region is operable to modulate the energy propagated from the second modulation area of the first SEM such that the energy propagated through the second modulation area of the first SEM and through the second modulation region of the second SEM comprises a second collection of wavefronts, each wavefront of the second collection having a complex amplitude modulation.

12. The energy modulation system of claim 11 , wherein the modulation applied by the second modulation region to the energy from the second modulation area of the first SEM comprises a second summation of point spread functions that describe, at the third energy propagation position, a summation of impulse responses to the energy propagated from the second plurality of energy locations in the second modulation area of the first SEM in the first energy propagation position.

13. The energy modulation system of claim 12, wherein the summation of impulse responses of the second summation of point spread functions comprises an intensity distribution of the second collection of wavefronts at the third energy propagation position.

14. The energy modulation system of claim 1 , wherein the intensity distribution of the second collection of wavefronts comprises a sum of an intensity of each wavefront.

15. The energy modulation system of claim 13, wherein the intensity distribution of the second collection of wavefronts comprises a sum of square of an intensity of each wavefront.

16. The energy modulation system of claim 12, wherein the first modulation region is further operable to modulate the energy propagated from the third modulation area of the first SEM such that the energy propagated through the third modulation area of the first SEM and through the first modulation region of the second SEM comprises a third collection of wavefronts, each wavefront of the third collection having a complex amplitude modulation.

17. The energy modulation system of claim 16, wherein the second energy modulation applied by the first modulation region to the energy from the third modulation area of the firstSEM comprises a third summation of point spread functions that describe, at the third energy propagation position, a summation of impulse responses to the energy propagated from the third plurality of energy locations in the third modulation area of the first SEM in the first energy propagation position.

18. The energy modulation system of claim 17, wherein the summation of impulse responses of the third summation of point spread functions comprises an intensity distribution of the third collection of wavefronts at the third energy propagation position.

19. The energy modulation system of claim 18, wherein the intensity distribution of the third collection of wavefronts comprises a sum of an intensity of each wavefront.

20. The energy modulation system of claim 18, wherein the intensity distribution of the third collection of wavefronts comprises a sum of square of an intensity of each wavefront.

21. The energy modulation system of claim 17, wherein the second modulation region is further operable to modulate the energy propagated from the fourth modulation area of the first SEM such that the energy propagated through the fourth modulation area of the first SEM and through the second modulation region of the second SEM comprises a fourth collection of wavefronts, each wavefront of the fourth collection having a complex amplitude modulation.

22. The energy modulation system of claim 21 , wherein the second energy modulation applied by the second modulation region to the energy from the fourth modulation area of the first SEM comprises a fourth summation of point spread functions that describe, at the third energy propagation position, a summation of impulse responses to the energy propagated from the fourth plurality of energy locations in the fourth modulation area of the first SEM in the first energy propagation position.

23. The energy modulation system of claim 22, wherein the summation of impulse responses of the fourth summation of point spread functions comprises an intensity distribution of the fourth collection of wavefronts at the third energy propagation position.

24. The energy modulation system of claim 23, wherein the intensity distribution of the fourth collection of wavefronts comprises a sum of an intensity of each wavefront.

25. The energy modulation system of claim 23, wherein the intensity distribution of the fourth collection of wavefronts comprises a sum of square of an intensity of each wavefront.

26. The energy modulation system of claim 1 , further comprising a third SEM located in a third energy propagation position, the third SEM comprising a plurality of modulation regions, and a third energy modulation is operable to be spatially applied to the energy propagated through the plurality of modulation regions of the third SEM.

27. The energy modulation system of claim 26, wherein the first energy modulation for the first modulation area, the second energy modulation for the first modulation region of the second SEM, and the third energy modulation for a first modulation region of the plurality of modulation regions of the third SEM define a first compounded complex amplitude function, and wherein energy modulated by the first compounded complex amplitude function comprises a first sub-sampled collection of wavefronts.

28. The energy modulation system of claim 27, wherein the first energy modulation for the first modulation area, the second energy modulation for the first modulation region of the second SEM, and the third energy modulation for a second modulation region of the plurality of modulation regions of the third SEM define a second compounded complex amplitude function, and wherein energy modulated by the second compounded complex amplitude function comprises a second sub-sampled collection of wavefronts.

29. The energy modulation system of claim 28, wherein the first and second collections of wavefronts each comprise a collection of planar wavefronts or a collection of spherical wavefronts.

30. The energy modulation system of claim 28, wherein the first and second sub-sampled collections of wavefronts are operable to be approximated by first and second effective spherical wavefronts.

31. The energy modulation system of claim 30, wherein the first and second effective spherical wavefront each comprise a first real effective origin and a first virtual effective origin.

32. The energy modulation system of claim 31 , wherein the first modulation region of the third SEM comprises an optical element having a focal length, and wherein the third energy modulation for the first modulation region of the third SEM comprises a phase function accounting for the focal length of the optical element.

33. The energy modulation system of claim 32, wherein the first virtual effective origin of the first effective spherical wavefront is located at the focal length of the optical element.

34. The energy modulation system of claim 28, wherein the plurality of modulation regions of the third SEM each comprise a diffractive waveguide element, and wherein the third energy modulation for the first modulation region of the third SEM comprises a phase function operable to increase an angular extent of energy propagated through the first modulation region of the third SEM.

35. The energy modulation system of claim 34, wherein the energy comprising the first collection of wavefronts propagated through the first modulation region of the second SEM is operable to be approximated by a first effective spherical wavefront comprising a first real effective origin and a first virtual effective origin, and wherein the third SEM is located in a position proximate to the position of the first real effective origin of the first effective spherical wavefront.

36. The energy modulation system of claim 35, wherein the first effective spherical wavefront has a first angular extent defined around the first virtual effective origin of the first effective spherical wavefront, and the phase function of the third energy modulation for the firstmodulation region of the third SEM operable to increase the angular extent of energy propagated through the first modulation region of the third SEM by a second angular extent, and further wherein the first real effective origin is located at a first distance from the first modulation region of the second SEM, and the first modulation region of the third SEM is located at a second distance from the first modulation region of the second SEM.

37. The energy modulation system of claim 36, wherein the first and second angular extents are the same, and the first and second distances are the same.

38. The energy modulation system of claim 36, wherein the first and second angular extents are different, and the first and second distances are different.

39. The energy modulation system of claim 36, wherein ratio of the first angular extent to second angular extents is the same as the ratio of the second distance to the first distance.

40. A rendering system for rendering instructions for a reference energy modulation system to propagate modulated energy for forming a holographic representation of an energy surface, the reference energy modulation system comprising a first reference energy modulation surface located in a first energy propagation position and a second reference energy modulation surface located in a second energy propagation position, the first reference energy modulation surface comprising a plurality of reference modulation areas including first and second reference modulation areas, the second reference energy modulation surface comprising a plurality of reference modulation regions including first and second reference modulation regions, wherein a first energy modulation is operable to be spatially applied to energy propagated through the plurality of reference modulation areas of the first reference energy modulation surface, and wherein a second energy modulation is operable to be spatially applied to the energy propagated from the first reference energy modulation surface and through the plurality of reference modulation regions of the second reference energy modulation surface, the rendering system comprising: a processor configured to:determine the first energy modulation based on a sampling of the energy surface, the sampling of the energy surface comprises a plurality of wavefront samples; wherein the first energy modulation for the first and second reference modulation areas of the first reference energy modulation surface is based on first and second wavefront samples of the plurality of wavefront samples, respectively, the first and second wavefront samples operable to define a first portion of the holographic representation of the energy surface; wherein the first wavefront sample comprises a first plurality of sets of sampling paths extending through a first sampling region of a plurality of sampling regions and the second wavefront sample comprises a second plurality of sets of sampling paths extending through a second sampling region of the plurality of sampling regions; wherein the first plurality of sets of sampling paths of the first wavefront sample is determined basing, at least in part, on a location of the first reference modulation area along the first energy propagation position, a location of a first reference modulation region of the second reference energy modulation surface along the second energy propagation position, and the second energy modulation for the first reference modulation region; and wherein the second plurality of sets of sampling paths of the second wavefront sample is determined basing, at least in part, on a location of the second reference modulation area along the first energy propagation position, a location of a second reference modulation region of the second reference energy modulation surface along the second energy propagation position, and the second energy modulation for the second reference modulation region.

41. The system of claim 40, wherein the first and second plurality of sets of sampling paths further extend through a relative positioning reference surface of the reference energy modulation system.

42. The system of claim 40, wherein the plurality of sampling regions is aligned in an energy sampling position based on the second energy propagation position of the second reference energy modulation surface of the reference energy modulation system.

43. The system of claim 42, wherein the first energy modulation determined for the first reference modulation area and the second energy modulation for the first reference modulation region define a first complex amplitude function; wherein energy modulated by the first complex amplitude function comprises a first collection of wavefronts; and wherein the location of the first sampling region along the energy sampling position is based on a location within the first reference modulation region of the second reference energy modulation surface, and sampling along the first plurality of sets of sampling paths of the first wavefront sample comprises a representation of the energy comprising the first collection of wavefronts.

44. The system of claim 43, wherein the first energy modulation determined for the second reference modulation area and the second energy modulation for the second reference modulation region define a second complex amplitude function; wherein energy modulated by the second complex amplitude function comprises a second collection of wavefronts; and wherein the location of the second sampling region along the energy sampling position is based on a location within the second reference modulation region of the second reference energy modulation surface, and sampling along the second plurality of sets of sampling paths of the second wavefront sample comprises a representation of the energy comprising the second collection of wavefronts.

45. The system of claim 40, wherein the reference energy modulation system further comprises a third reference energy modulation surface located in a third energy propagation position, and wherein the plurality of sampling regions is aligned in an energy sampling position that is based on the third energy propagation position of the third reference energy modulation surface of the reference energy modulation system.

46. The system of claim 45, wherein the third reference energy modulation surface of the reference energy modulation system comprises a plurality of reference modulation regions, and a third energy modulation is operable to be spatially applied to the energy propagated through the plurality of reference modulation regions of the third reference energy modulation surface; wherein the first energy modulation determined for the first reference modulation area, the second energy modulation for the first reference modulation region of the second reference energy modulation surface, and the third energy modulation for a first reference modulation region of the plurality of reference modulation regions of the third reference energy modulation surface define a first compounded complex amplitude function; wherein energy modulated by the first compounded complex amplitude function comprises a first collection of wavefronts; and wherein a location of the first sampling region along the energy sampling position is based on a location of the first reference modulation region of the third reference energy modulation surface, and sampling along the first plurality of sets of sampling paths comprises a representation of the energy comprising the first collection of wavefronts.

47. The system of claim 46, wherein the first energy modulation determined for the first reference modulation area, the second energy modulation for the first reference modulation region of the second reference energy modulation surface, and the third energy modulation for a second reference modulation region of the plurality of reference modulation regions of the third reference energy modulation surface define a second compounded complex amplitude function; wherein energy modulated by the second compounded complex amplitude function comprises a second collection of wavefronts; and wherein a location of the second sampling region along the energy sampling position is based on a location of the second reference modulation region of the third reference energy modulation surface, and sampling along the second plurality of sets of sampling paths comprises a representation of the energy comprising the second collection of wavefronts.

48. The system of claim 47, wherein the first and second collections of wavefronts each comprises a collection of planar wavefronts or a collection of spherical wavefronts.

49. The system of claim 48, wherein the first plurality of sets of sampling paths of the first wavefront sample comprises a first set of sampling paths, a second set of sampling paths, and at least one additional set of sampling paths, and wherein sampling along the first set of sampling paths comprises a representation of the energy comprising a first plane wave having a first planar wavefront, sampling along the second set of sampling paths comprises a representation of the energy comprising a second plane wave having a second planar wavefront, and sampling along the at least one additional set of sampling paths comprises a representation of the energy comprising at least one additional plane wave having at least one additional planar wavefront.

50. The system of claim 48, wherein the first plurality of sets of sampling paths of the first wavefront sample comprises a first set of sampling paths, a second set of sampling paths, and at least one additional set of sampling paths, and wherein sampling along the first set of sampling paths comprises a representation of the energy comprising a first spherical wave having a first spherical wavefront, sampling along the second set of sampling paths comprises a representation of the energy comprising a second spherical wave having a second spherical wavefront, and sampling along the at least one additional set of sampling paths comprises a representation of the energy comprising at least one additional spherical wave having at least one additional spherical wavefront.

51. The system of claim 48, wherein the first wavefront sample comprises a first effective spherical wavefront sample, and wherein the collection of planar wavefronts or the collection of spherical wavefronts are operable to be sampled by the first effective spherical wavefront sample accounting for at least one of physical extent, diffraction, and wavefront errors.

52. The system of claim 51, wherein the first effective spherical wavefront sample comprises a first real effective origin, a first virtual effective origin, or both.

53. The system of claim 52, wherein first reference modulation region of the third reference energy modulation surface comprises an optical element having a focal length, wherein the third energy modulation for the first reference modulation region of the third reference energy modulation surface comprises a phase function accounting for the focal length of the optical element.

54. The system of claim 53, wherein the first virtual effective origin of the first effective spherical wavefront sample is located at the focal length of the optical element.

55. The system of claim 44, wherein the first and second collections of wavefronts each comprises a collection of planar wavefronts or a collection of spherical wavefronts.

56. The system of claim 55, wherein the first plurality of sets of sampling paths of the first wavefront sample comprises a first set of sampling paths, a second set of sampling paths, and at least one additional set of sampling paths, and wherein sampling along the first set of sampling paths comprises a representation of the energy comprising a first plane wave having a first planar wavefront, sampling along the second set of sampling paths comprises a representation of the energy comprising a second plane wave having a second planar wavefront, and sampling along the at least one additional set of sampling paths comprises a representation of the energy comprising at least one additional plane wave having at least one additional planar wavefront.

57. The system of claim 55, wherein the first plurality of sets of sampling paths of the first wavefront sample comprises a first set of sampling paths, a second set of sampling paths, and at least one additional set of sampling paths, and wherein sampling along the first set of sampling paths comprises a representation of the energy comprising a first spherical wave having a first spherical wavefront, sampling along the second set of sampling paths comprises a representation of the energy comprising a second spherical wave having a second spherical wavefront, and sampling along the at least one additional set of sampling paths comprises a representation of theenergy comprising at least one additional spherical wave having at least one additional spherical wavefront.

58. The system of claim 55, wherein the first wavefront sample comprises a first effective spherical wavefront sample, and wherein the collection of planar wavefronts or the collection of spherical wavefronts are operable to be sampled by the first effective spherical wavefront sample accounting for at least one of physical extent, diffraction, and wavefront errors.

59. The system of claim 58, wherein the first effective spherical wavefront sample comprises a real effective origin, a virtual effective origin, or both.

60. The system of claim 59, wherein the first effective spherical wavefront is sampled from the real effective origin.

61. The system of claim 59, wherein the first effective spherical wavefront is sampled from the virtual effective origin.

62. The system of claim 59, wherein the first effective spherical wavefront is sampled from both the real effective origin and the virtual effective origin.

63. The system of claim 59, wherein the first effective spherical wavefront sample is sampled by a perspective Tenderer as wavefronts converge on the real effective origin having a real sample clipping plane located thereof.

64. The system of claim 59, wherein the first effective spherical wavefront is sampled by a perspective Tenderer as wavefronts converge on the virtual effective origin having a virtual sample clipping plane located thereof.

65. The system of claim 59, wherein the first effective spherical wavefront is sampled by two perspective Tenderers as wavefronts converging on the real and the virtual effective origins and having real and virtual sample clipping plane located thereof.

66. The system of claim 55, wherein each set of the first plurality of sets of sampling paths of the first wavefront sample comprises substantially parallel sampling paths that collectively are operable to sample a representation of a respective plane wave propagating along a propagation angle with respect to a normal of a relative positioning reference surface, each of the plane waves having a respective reference location in the relative positioning reference surface and a radial extent.

67. The system of claim 66, wherein the first wavefront sample has a first aggregated angular sampling range at the first sampling region, the first aggregated angular sampling range being defined by the propagation angles of the plane waves.

68. The system of claim 67, wherein the processor is to associate, in a reference data set, the first energy modulation to a first set of parameters for the reference energy modulation system, the first set of parameters at least partially determining the first sampling region and the first aggregated angular sampling range of the first wavefront sample.

69. The system of claim 68, wherein the first set of parameters at least partially determining the first sampling region and the first aggregated angular sampling range of the first wavefront sample comprises the location of the first reference modulation area along the first energypropagation position and the location of a first reference modulation region of the second reference energy modulation surface along the second energy propagation position.

70. The system of claim 69, wherein the second wavefront sample has a second aggregated angular sampling range at the second sampling region, the second aggregated angular sampling range being defined by the propagation angles of the plane waves.

71. The system of claim 70, wherein the processor is to associate, in the reference data set, the first energy modulation to a second set of parameters for the reference energy modulation system, the second set of parameters at least partially determining the second sampling region and the second aggregated angular sampling range of the second wavefront sample.

72. The system of claim 71, wherein the second set of parameters at least partially determining the second sampling region and the second aggregated angular sampling range of the second wavefront sample comprises the location of the second reference modulation area along the first energy propagation position and the location of a second reference modulation region of the second reference energy modulation surface along the second energy propagation position.

73. The system of claim 72, wherein first energy modulation is associated with the first and second set of parameters for the reference energy modulation system in a look up table format.

74. The system of claim 73, wherein the first and second sets of parameters for the reference energy modulation system comprise parameters for first and second reference modulation areas in the first reference energy modulation surface, respectively.

75. The system of claim 74, wherein the parameters for the first and second reference modulation areas comprise first and second reference modulation area sizes, respectively.

76. The system of claim 75, wherein the first and second reference modulation area sizes are the same.

77. The system of claim 75, wherein the first and second reference modulation area sizes are different.

78. The system of claim 75, wherein the first and second reference modulation area sizes are greater than, equal to, or smaller than a size of a reference modulation element in the first reference energy modulation surface.

79. The system of claim 74, wherein the parameters for the first and second reference modulation areas comprise first and second reference modulation area locations, respectively.

80. The system of claim 79, wherein the first and second reference modulation area locations are correlated to a center of first and second reference modulation elements in the first reference energy modulation surface, respectively.

81. The system of claim 79, wherein the first and second reference modulation area locations are separated by a first sample interval therebetween, the first sample interval being greater than, equal to, or smaller than a size of a reference modulation element in the first reference energy modulation surface.

82. The system of claim 81, wherein the parameters for the first and second reference modulation areas further comprise first and second reference modulation area sizes, respectively, the first and second reference modulation area sizes being greater than, equal to, or smaller thanthe size of the reference modulation element in the first reference energy modulation surface.

83. The system of claim 79, wherein the plurality of reference modulation areas of the first reference energy modulation surface further includes at least one additional reference modulation area and the plurality of reference modulation regions of the second reference energy modulation surface further includes at least one additional reference modulation region.

84. The system of claim 83, wherein the first energy modulation for the at least one additional reference modulation area of the first reference energy modulation surface is based on at least one additional wavefront sample of the plurality of wavefront samples, wherein the at least one additional wavefront sample and at least one of the first or the second wavefront samples are operable to define a second portion of the holographic representation of the energy surface.

85. The system of claim 84, wherein the at least one additional wavefront sample comprises at least one additional plurality of sets of sampling paths extending through at least one additional sampling region of the plurality of sampling regions, wherein the at least one additional plurality of sets of sampling paths of the at least one additional wavefront sample is determined basing, at least in part, on a location of the at least one additional reference modulation area along the first energy propagation position, a location of the at least one additional reference modulation region of the second reference energy modulation surface along the second energy propagation position, and the second energy modulation for the at least one additional reference modulation region.

86. The system of claim 85, wherein the at least one additional wavefront sample has at least one additional aggregated angular sampling range at the at least one additional sampling region, the at least one additional aggregated angular sampling range being defined by the propagation angles of the plane waves.

87. The system of claim 86, wherein the processor is to associate, in the reference data set, the first energy modulation to at least one additional set of parameters for the reference energy modulation system, the at least one additional set of parameters at least partially determining the at least one additional sampling region and the at least one additional aggregated angular sampling range of the at least one additional wavefront sample.

88. The system of claim 87, wherein the at least one additional set of parameters at least partially determining the at least one additional sampling region and the at least one additional aggregated angular sampling range of the at least one additional wavefront sample comprises the location of the at least one additional reference modulation area along the first energy propagation position and the location of a at least one additional reference modulation region of the second reference energy modulation surface along the second energy propagation position.

89. The system of claim 88, wherein the at least one additional set of parameters for the reference energy modulation system comprise parameters for the at least one additional reference modulation area in the first reference energy modulation surface.

90. The system of claim 89, wherein the at least one additional set of parameters for the at least one additional reference modulation area comprises at least one additional reference modulation area location.

91. The system of claim 90, wherein the at least one additional reference modulation area location is operable to be correlated to a center of a reference modulation element of the first reference energy modulation surface.

92. The system of claim 90, wherein the first reference modulation area location, the second reference modulation area location, and the at least one additional reference modulation arealocation have sample intervals therebetween.

93. The system of claim 92, wherein the sample intervals are substantially the same.

94. The system of claim 92, wherein the sample intervals are substantially different.

95. The system of claim 90, wherein the parameters for the first reference modulation area, the second reference modulation area, and the at least one additional reference modulation area further comprise a first reference modulation area size, a second reference modulation area size, and at least one additional reference modulation area size, respectively.

96. The system of claim 95, wherein the first reference modulation area location, the second reference modulation area location, the at least one additional reference modulation area location, the first reference modulation area size, the second reference modulation area size, and the at least one additional reference modulation area size define a sample distribution pattern in the first reference energy modulation surface.

97. The system of claim 96, wherein the sample distribution pattern comprises a random pattern.

98. The system of claim 96, wherein the sample distribution pattern comprises a non-random pattern.

99. The system of claim 98, wherein the sample distribution pattern comprises substantially a periodic pattern.

100. The system of claim 98, wherein the sample distribution pattern comprises substantially an aperiodic pattern.

101. The system of claim 98, wherein the sample distribution pattern comprises substantially a regular pattern.

102. The system of claim 98, wherein the sample distribution pattern comprises substantially an irregular pattern.

103. The system of claim 40, wherein the first and second sampling regions overlap.

104. The system of claim 40, wherein the first and second sampling regions are offset from each other.

105. The system of claim 40, wherein the first and second sampling regions partially overlap.

106. The system of claim 70, wherein the processor is further configured to render a rasterized data set from the reference data set, the rasterized data set comprising instruction for operating modulating elements of an optical system that is different from the reference energy modulation system.

107. The system of claim 106, a spatial frequency of the first energy modulation in the reference data set is altered in the rasterized data set.

108. The system of claim 106, a temporal frequency of the first energy modulation in the reference data set is altered in the rasterized data set.

109. The system of claim 106, spatial and temporal frequencies of the first energy modulation in the reference data set are altered in the rasterized data set.

110. The system of claim 55, wherein the first plurality of sets of sampling paths comprises a first set of sampling paths operable to sample a representation of the energy comprising a first plane wave having a first propagation angle and a first reference location in a relative positioning reference surface, the first reference location being defined by an intersection of a referencesampling path of the first set of sampling paths and the relative positioning reference surface.

111. The system of claim 110, wherein the reference sampling path corresponds to a path that passes through a center portion of a corresponding reference modulation area in the first reference energy modulation surface.

112. The system of claim 110, wherein the first set of sampling paths have a first radial extent from the first reference location in the relative positioning reference surface.

113. The system of claim 110, wherein the first plurality of sets of sampling paths comprises a second set of sampling paths operable to sample a representation of the energy comprising a second plane wave having a second propagation angle and a second reference location in the relative positioning reference surface, the second reference location being defined by an intersection of a reference sampling path of the second set of sampling path and the relative positioning reference surface.

114. The system of claim 113, wherein the first and second reference locations are different.

115. The system of claim 113, wherein the first and second propagation angles are different.

116. The system of claim 41, wherein the relative positioning reference surface is offset from the first reference energy modulation surface.

117. The system of claim 41, wherein the relative positioning reference surface passes through an optical center of the reference energy modulation system.

118. The system of claim 41, wherein the relative positioning reference surface coincides with the first reference energy modulation surface or the second reference energy modulation surface.

119. The system of claim 59, wherein the first plurality of sets of sampling paths of the first wavefront sample is determined basing, additionally, on a maximum acceptance angle, whereinthe maximum acceptance angle is based, at least in part, on a viewer location with respect to the location of the first reference modulation region of the second reference energy modulation surface and a size of a viewer aperture.

120. The system of claim 119, wherein the size of the viewer aperture is based, at least in part, on a viewer pupil size.

121. The system of claim 119, wherein the maximum acceptance angle is based, additionally, on a size of the first sampling region.

122. The system of claim 121, wherein the size of the first sampling region is based, at least in part, on a viewer acuity spot size.

123. The system of claim 122, wherein the viewer acuity spot size is determined based, at least in part, on a viewer visual acuity.

124. The system of claim 121, wherein the size of the first sampling region is based, at least in part, on a size of the first reference modulation region of the second reference energy modulation surface.

125. A processer for determining instructions for a reference energy modulation system to propagate modulated energy for forming a holographic representation of an energy surface, the reference energy modulation system comprising a first reference energy modulation surface located in a first energy propagation position and a second reference energy modulation surface located in a second energy propagation position, the first reference energy modulation surface comprising a plurality of reference modulation areas including first and second reference modulation areas, the second reference energy modulation surface comprising a plurality of reference modulation regions including first and second reference modulation regions, wherein a first energy modulation is operable to be spatially applied to energy propagated through the plurality of reference modulation areas of the first reference energy modulation surface, and wherein a second energy modulation is operable to be spatially applied to the energy propagatedfrom the first reference energy modulation surface and through the plurality of reference modulation regions of the second reference energy modulation surface, the processor configured to: determine a first spatial modulation frequency limit for applying the first energy modulation; wherein an intensity contrast of the modulated energy to be propagated from the reference energy modulation system is a function of a spatial modulation frequency of the reference energy modulation system and an energy propagation distance from the reference energy modulation system, and wherein the first spatial modulation frequency limit is determined based on a first contrast threshold for a first energy propagation distance.

126. The processor of claim 125, wherein the processor is further configured to: determine a sampling of the energy surface, the sampling of the energy surface comprises a plurality of wavefront samples; wherein the first energy modulation for the first and second reference modulation areas of the first reference energy modulation surface is based on first and second wavefront samples of the plurality of wavefront samples, respectively, the first and second wavefront samples operable to define a first portion of the holographic representation of the energy surface; the first wavefront sample comprises sampling paths extending through a first sampling region of a plurality of sampling regions and the second wavefront sample comprises sampling paths extending through a second sampling region of the plurality of sampling regions; wherein the plurality of sampling regions is aligned in an energy sampling position determined based on the second energy propagation position of the second reference energy modulation surface of the reference energy modulation system; wherein the sampling paths of the first wavefront sample is determined basing, at least in part, on a location of the first reference modulation area along the first energy propagation position, a location of a first reference modulation region of the second reference energy modulation surface along the second energy propagation position, and the second energy modulation for the first reference modulation region; wherein the sampling paths of the second wavefront sample is determined basing, at least in part, on a location of the second reference modulation area along the first energy propagationposition, a location of a second reference modulation region of the second reference energy modulation surface along the second energy propagation position, and the second energy modulation for the second reference modulation region.

127. The processor of claim 126, wherein the first spatial modulation frequency limit is based on an intra-modulation-region contrast threshold and an inter-modulation-regions contrast threshold, and wherein the processor is configured to determine first and second subsets of the plurality of reference modulation areas and first and second subsets of the plurality of reference modulation regions; wherein modulated energy propagated from the modulation areas in the first subset of modulation areas through a same reference modulation region in the first subset of the plurality of reference modulation regions can have a contrast greater than the intra-modulation-region contrast threshold; wherein modulated energy propagated from the modulation areas in the first subset of modulation areas through different reference modulation regions in the first subset of the plurality of reference modulation regions can have a contrast greater than the inter-modulation- regions contras threshold; wherein the reference modulation areas in the first subset of the first plurality of reference modulation areas are not in the second subset of the first plurality of reference modulation areas; and wherein the instruction determined by the processor comprises limiting propagation of energy through the second subset of the plurality of reference modulation areas.

128. The processor of claim 127, wherein the processor is configured to determine the plurality of wavefront samples for both the first and second subsets of the plurality of reference modulation areas, and the instruction determined by the processor comprises selecting only the wavefront samples for the first subsets of the plurality of reference modulation areas to define the holographic representation of the energy surface.

129. The processor of claim 127, wherein the processor is configured to determine the plurality of wavefront samples for only the first subset of the plurality of reference modulationareas, and the instruction determined by the processor comprises defining the holographic representation of the energy surface with the plurality of wavefront samples determined by the processor.

130. The processor of claim 127, wherein the instruction determined by the processor further comprises limiting propagation of energy through the second subset of the plurality of reference modulation region.

131. The processor of claim 125, wherein the first energy modulation for the first reference modulation area and the second energy modulation for the first reference modulation region defines a first complex amplitude function, and the first energy modulation for the second reference modulation area and the second energy modulation for the second reference modulation region defines a second complex amplitude function.

132. The processor of claim 131, wherein energy modulated by the first and second complex amplitude functions comprises representations of first and second collections of wavefronts that are operable to form the first portion of the holographic representation of the energy surface.

133. The processor of claim 132, wherein the first portion of the holographic representation of the energy surface is formed at the first energy propagation distance.

134. The processor of claim 132, wherein the first and second collections of wavefronts are operable to be approximated by first and second effective spherical wavefronts each comprising both real and virtual effective origins.

135. The processor of claim 134, wherein the first effective spherical wavefront comprises a beam waist located in real space between the first reference modulation region and the real effective origin and a beam waist located in virtual space between the first reference region and the virtual effective origin.

136. The processor of claim 1 5, wherein the second effective spherical wavefront comprises a beam waist located in real space between the second reference modulation region and the real effective origin and a beam waist located in virtual space between the second reference region and the virtual effective origin.

137. The processor of claim 136, wherein for a given contrast threshold for the function, the function comprise a peak spatial modulation frequency limit at an energy propagation distance to one of the beam waists.

138. The processor of claim 125, wherein the function comprises a modulation transfer function.

139. The processor of claim 126, wherein the first and second wavefront samples comprise first and second effective spherical wavefront samples, respectively.

140. The processor of claim 126, wherein the first and second wavefront samples comprise first and second pluralities of sets of sampling paths, respectively.

141. The processor of claim 140, wherein each set of the first and second plurality of sets of sampling paths defines a respective planar wavefront.

142. The processor of claim 140, wherein each set of the first and second plurality of sets of sampling paths defines a respective spherical wavefront.

Images