System and method of holographic sensory data generation, manipulation, and transport

Abstract

To solve the problem of no existence of an indoor interactive virtual world seen within a "holodeck".SOLUTION: A method determines four dimensional (4D) plenoptic coordinates for content data by: receiving content data; determining locations of data points with respect to creating a digital volumetric representation of the content data, a first surface being a reference surface; determining 4D plenoptic coordinates of the data points at a second surface by tracing location data points in the volumetric representation to the second surface where a 4D function is applied; and determining energy source location values for 4D plenoptic coordinates that have a first point of convergence.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present disclosure generally relates to the generation of holographic content including sensory information, and more specifically, to the generation of holographic content from non-holographic information.

Background Art

[0002] The dream of an interactive virtual world within the "holodeck" quarters, popularized to the general public by Gene Roddenberry's Star Trek and originally planned by the writer Alexander Moszkowski in the early 1900s, has been an inspiration for science fiction and technological innovation for nearly a century. However, the epoch-making realization of this experience has been entirely outside the realm of literature, media, and the overall imagination of children as well as adults alike.

Summary of the Invention

[0003] In one embodiment, a method for determining 4D (four-dimensional) plenoptic coordinates for content data may include receiving the content data, determining the positions of data points with respect to a first surface to generate a digital three-dimensional representation of the content data, where the first surface is a reference surface, generating, tracking the position data points within the three-dimensional representation up to a second surface to which a 4D function is applied, determining the 4D plenoptic coordinates of the data points at the second surface, and determining an energy source position value for the 4D plenoptic coordinates having a first convergence point.

[0004] In one embodiment, a method for determining 4D (four-dimensional) plenoptic coordinates for content data may include receiving the content data, determining the position of a data point relative to a reference point position, vectorizing the data point by generating a vector of the data point based on the reference point position, generating a digital three-dimensional representation of the content data based on the vectorized data point, determining the position of the data point relative to a first surface, where the first surface is a reference surface, and determining the 4D plenoptic coordinates of the data point on a second surface by tracking the position data point within the three-dimensional representation to a second surface to which a 4D function is applied.

[0005] In one embodiment, a method for vectorization may include receiving first content data, identifying a surface within the content data, determining a surface identification of the surface, determining material property data of the surface, associating the surface identification with the material property data of the surface, generating a vector of the material property data, and generating vectorized material property data based on the generated vector.

[0006] In one embodiment, a system for determining four-dimensional (4D) plenoptic coordinates for content data can comprise an input / output interface operable to receive the content data and a processing subsystem in communication with the input / output interface and including a sensory data processor, a vectorization engine, and a tracing engine. The sensory data processor is operable to determine the positions of data points within the content data relative to a first surface and generate a digital three-dimensional representation of the content data, where the first surface is a reference surface. The tracing engine is operable to determine the 4D plenoptic coordinates of data points on a second surface by tracking the positions of data points within the digital three-dimensional representation up to the second surface to which a 4D function is applied. The tracing engine is operable to determine an energy source position value for 4D plenoptic coordinates having a first convergence point.

[0007] In one embodiment, a system for determining four-dimensional (4D) plenoptic coordinates for content data can comprise an input / output interface operable to receive the content data and a processing subsystem in communication with the input / output interface and including a sensory data processor, a vectorization engine, and a tracing engine. The sensory data processor is operable to determine the positions of data points within the content data relative to a reference point position. The vectorization engine is operable to vectorize the data points based on the reference point position. The sensory data processor is further operable to determine the positions of the data points relative to a first surface based on the vectorized data points and generate a digital three-dimensional representation of the content data, where the first surface is a reference surface. The tracing engine is operable to determine the 4D plenoptic coordinates of data points on a second surface by tracking the positions of data points within the three-dimensional representation up to the second surface to which a 4D function is applied.

[0008] In one embodiment, a system for vectorization can include an input / output interface operable to receive content data, and a processing subsystem in communication with the input / output interface and including a vectorization engine, the vectorization engine being operable to identify surfaces within the content data, determine a surface identification for the surfaces, determine material property data for the surfaces, and associate the surface identification with the material property data for the surfaces, the vectorization engine being further operable to generate a vector of the material property data and generate vectorized material property data based on the generated vector.

[0009] In one embodiment, a system for determining four-dimensional (4D) plenoptic coordinates for content data can include an input / output interface operable to receive the content data, a processing subsystem in communication with the input / output interface and including a sensory data processor, a vectorization engine, and a tracing engine, a compression engine in communication with the processing subsystem and the input / output interface, and an optional memory in communication with the compression module, the input / output interface, and the processing subsystem. The sensory data processor is operable to determine the positions of data points within the content data relative to a reference point position. The vectorization engine is operable to vectorize the data points based on the reference point position. The sensory data processor is further operable to determine the positions of the data points relative to a first surface based on the vectorized data points to generate a digital volumetric representation of the content data, where the first surface is a reference surface. The tracing engine is operable to determine the 4D plenoptic coordinates of the data points on a second surface by tracking the positions of the data points within the volumetric representation to a second surface to which a 4D function is applied. The compression engine is operable to receive data from the processing subsystem, compress the data, and store the compressed data in the optional memory or transmit the compressed data to the input / output interface. The optional memory is configured to receive data from the input / output interface, the processing subsystem, and the compression engine, store the data, and transmit the stored data to any of the input / output interface, the processing subsystem, or the compression engine.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 7J

Figure 7K

Figure 7L

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

DETAILED DESCRIPTION OF THE INVENTION

[0011] One embodiment of a Holodeck (collectively referred to as "Holodeck design parameters") provides sufficient energy stimuli to deceive human sensory receptors and convince the energy impulses received within a virtual, social, and interactive environment that they are real. It provides 1) binocular disparity without external accessories, head-mounted eyewear, or other peripheral devices, 2) accurate motion parallax, occlusion, and opacity across the entire viewing volume simultaneously for any number of viewers, 3) visual focus via synchronous convergence, accommodation, and miosis for all perceived light rays, and 4) convergent energy wave propagation with sufficient density and resolution to exceed human sensory "resolution" for vision, hearing, touch, taste, smell, and / or balance.

[0012] Based on prior art to date, as suggested by Holodeck Design Parameters including the visual, auditory, somatosensory, gustatory, olfactory, and vestibular systems, we are decades, if not centuries, away from a technology that can provide all receptive fields in an epoch-making way.

[0013] In the present disclosure, the terms light field and holographic may be used interchangeably to define the energy propagation for the stimulation of any sensory receptor response. The initial disclosure may refer to examples of energy and mechanical energy propagation through an energy surface for holographic images and volumetric haptics, but all forms of sensory receptors are envisioned within the present disclosure. Further, the principles disclosed herein for energy propagation along a propagation path may be applicable to both energy emission and energy capture.

[0014] Today, many technologies exist that are often confused with holograms, including, unfortunately, lenticular printing, Pepper's Ghost, autostereoscopic displays, horizontal parallax displays, head-mounted VR and AR displays (HMDs), and other such illusions generalized as "faux holography." These technologies may exhibit some of the desired characteristics of a true holographic display, but lack the ability to stimulate the human visual sensory response in any way sufficient to address at least two of the four identified holodeck design parameters.

[0015] To generate an energy surface that is sufficiently seamless for holographic energy propagation, these challenging endeavors have not been successfully implemented by conventional techniques. However, there are various approaches for implementing volumetric and directional multiplexed light field displays, including parallax barriers, hologels, voxels, diffractive optical elements, multi-view projection, holographic diffusers, rotating mirrors, multilayer displays, time-sequential displays, head-mounted displays, etc., but conventional approaches may require compromises regarding image quality, resolution, angular sampling density, size, cost, safety, frame rate, etc., and may ultimately result in unfeasible technologies.

[0016] To achieve the hold deck design parameters for the visual, auditory, and somatosensory systems, the sensitivity of each human in each system is studied, and it is understood that energy waves are propagated to sufficiently deceive the human sensory receptors. The visual system can resolve up to approximately one minute, the auditory system can distinguish a placement difference of only three degrees, and the somatosensory system of the hand can identify points separated by 2 - 12 mm. There are various opposing methods to measure these sensitivities, but these values are sufficient to understand the systems and methods for stimulating the perception of energy propagation.

[0017] Among the well-known sensory receptors, the human visual system is far more sensitive if it can even induce a sensation with a single photon. For this reason, much of this introduction focuses on visual energy wave propagation, and an extremely low-resolution energy system coupled within the disclosed energy waveguide surface can converge appropriate signals and induce a holographic sensory perception. Unless otherwise noted, all disclosures apply to all energy and sensory regions.

[0018] When calculating the effective design parameters of energy propagation for a visual system given a visual volume and viewing distance, the desired energy surface can be designed to include many gigapixels of effective energy position density. For a large visual volume, or near-field observation, the design parameters of the desired energy surface can include an effective energy position density of hundreds of gigapixels or more. In comparison, the desired energy source can be designed to have an energy position density of 1 - 250 effective gigapixels for ultrasonic propagation in volume haptics, or an array of 36 - 3,600 effective energy positions for acoustic propagation in holographic acoustics, depending on the input environmental variables. An important thing to note is that in the disclosed bidirectional energy surface architecture, all components can be configured to form an appropriate structure for any energy region to enable holographic propagation.

[0019] However, today, the main challenges for enabling a Holodeck encompass the limitations of available visual technologies and energy devices. Acoustic and ultrasonic devices are not as difficult, provided there are several orders of magnitude difference in the desired density based on the sensory acuity in their respective receptive fields, but their complexity should not be underestimated. Holographic emulsions exist with resolutions exceeding the desired density and encode interference patterns within static images, whereas state-of-the-art display devices are limited by resolution, data yield, and manufacturability. To date, even exceptional display devices have not been able to significantly generate a bright field with a holographic resolution nearly comparable to that of vision.

[0020] Manufacturing a single silicon-based device capable of meeting the desired resolution for a revolutionary bright-field display is not realistic and can result in an extremely complex manufacturing process that exceeds current manufacturing capabilities. Constraints on tiling together multiple existing display devices lead to many other issues that are inevitably infeasible technologies from the perspectives of packaging, electronics circuitry, housing, seams and gaps formed by optical components, and imaging, cost, and / or size.

[0021] The embodiments disclosed herein can provide a practical path for constructing a Holodeck.

[0022] Hereinafter, embodiments will be described with reference to the accompanying drawings in this specification. The accompanying drawings form a part of this specification and show embodiments that can be implemented. As used in this disclosure and the appended claims, the terms "embodiment," "embodiment example," and "exemplary embodiment" do not necessarily refer to a single embodiment, but they may be a single embodiment, and various embodiment examples can be easily combined and exchanged without departing from the scope or spirit of the embodiment examples. Furthermore, the technical terms used in this specification are for the purpose of only explaining the embodiment examples and are not intended to be limiting. In this regard, as used in this specification, the term "in" can include "in" and "on," and the terms "a," "an," and "the" can include singular and plural references. Furthermore, as used in this specification, the term "by" can also mean "from" depending on the context. Furthermore, as used in this specification, the term "if" can also mean "when" or "on" depending on the context. Furthermore, as used in this specification, the word "and / or" refers to any and all possible combinations of one or more of the relatedly listed items and can include them.

[0023] [Consideration of Holographic System] [Overview of Light Field Energy Propagation Resolution] Light fields and holographic displays are the result of multiple projections, where the energy surface position provides information on the angles, colors, and intensities propagated within the viewing volume. The disclosed energy surfaces provide opportunities for additional information to coexist and propagate through the same surface, inducing other sensory system responses. Unlike stereoscopic displays, the perceived position of the convergent energy propagation paths within the space does not change as the viewer moves around the viewing volume, and multiple viewers can simultaneously see an object propagated within the real-world space as if the object were really there. In some embodiments, the propagation of energy can be arranged within the same energy propagation path, but can also be arranged in opposite directions. For example, both energy emission and energy capture along an energy propagation path are possible in some embodiments of the present disclosure.

[0024] Figure 1 is a schematic diagram illustrating variables related to the stimulation of sensory receptor responses. These variables include surface diagonal 101, surface width 102, surface height 103, determined target seating distance 118, target seating field of view from the center of the display 104, number of intermediate samples shown herein as a sample between the two eyes 105, average separation distance between the pupils of an adult 106, average resolution of the human eye in angular minutes 107, horizontal field of view formed between the target viewer position and the surface width 108, vertical field of view formed between the target viewer position and the surface height 109, resulting overall surface horizontal waveguide element resolution, or total number of elements 110, resulting overall surface vertical waveguide element resolution, or total number of elements 111, pupil-to-pupil spacing between the two eyes, and sample distance based on the number of intermediate samples for the angular projection between the two eyes 112, angular sampling can be based on the sample distance and the target seating distance 113, total horizontal resolution per waveguide element derived from the desired angular sampling 114, total vertical resolution per waveguide element derived from the desired angular sampling 115, device horizontal is the count of the determined number of careful energy sources desired 116, and device vertical is the count of the determined number of careful energy sources desired 117.

[0025] The method for understanding the desired minimum resolution can be based on the following criteria to ensure sufficient stimulation of visual (or other) sensory receptor responses, namely, surface size (e.g., 84-inch diagonal), surface aspect ratio (e.g., 16:9), seating distance (e.g., 128 inches from the display), seating viewing field (e.g., 120 degrees or ±60 degrees with respect to the center of the display), the desired number of intermediate samples at a certain distance (e.g., one additional propagation path between the two eyes), the average separation distance between adult pupils (approximately 65 mm), and the average resolution of the human eye (approximately 1 arc minute). The values of these examples should be considered as placeholders according to specific application design parameters.

[0026] Furthermore, each value resulting from the visual sensory receptor can be replaced with other systems, and the desired propagation path parameters can be determined. In the case of other embodiments of energy propagation, the angular sensitivity of the auditory system can be considered to be as low as 3 degrees, and the spatial resolution of the somatosensory system of the hand can be considered to be as small as 2 - 12 mm.

[0027] There are various, opposing methods for measuring these perceptual sensitivities, but these values are sufficient to understand the systems and methods that stimulate the perception of virtual energy propagation. There are many methods for considering the design resolution, but the principle system proposed below combines practical product consideration with the biological resolution limit of the sensory system. As will be understood by those skilled in the art, the following overview simplifies any such system design and should be considered for illustrative purposes only.

[0028] Once the resolution limit of the sensory system is understood, given the following, the total energy waveguide element density can be calculated such that the receiving sensory system cannot distinguish between a single energy waveguide element and an adjacent element. [Equation 1] JPEG2025111410000002.jpg1578[Equation 2] JPEG2025111410000003.jpg21105[Equation 3] JPEG2025111410000004.jpg21104[Number 4] JPEG2025111410000005.jpg21132[Number 5] JPEG2025111410000006.jpg17123[Number 6] JPEG2025111410000007.jpg19119[Number 7] JPEG2025111410000008.jpg16118

[0029] As a result of the above calculations, a field of view of approximately 32×18° is obtained, and approximately 1920×1080 (rounded to the nearest format) energy waveguide elements are desired. Also, variables can be constrained so that the field of view is compatible with both (u,v) and provides a more regular spatial sampling of the energy positions (e.g., pixel aspect ratio). Assume that the angular sampling of the system is defined between two points at an optimized distance given as follows, the target volume position of the view volume, and the additional propagation energy path. [Number 8] JPEG2025111410000009.jpg18159[Number 9] JPEG2025111410000010.jpg13115

[0030] In this case, the interpupillary distance is used to calculate the sample distance, but any scale can be used to account for the appropriate number of samples as a given distance. Considering the above variables, approximately one ray per 0.57° may be desired, and the overall resolution of the system per independent sensory system can be calculated considering the following. [Number 10] JPEG2025111410000011.jpg14116[Number 11] JPEG2025111410000012.jpg784[Number 12] JPEG2025111410000013.jpg984

[0031] Using the above scenario, given the size of the energy surface and the angular resolution addressed to the vision system, the resulting energy surface can desirably include an energy resolution position of approximately 400k x 225k pixels, or a holographic propagation density of 90 gigapixels. These given variables are for illustrative purposes only, and many other sensory and energy metrology considerations should be examined in the case of optimizing the holographic propagation of energy. In additional embodiments, an energy resolution position of 1 gigapixel can be determined based on the input variables. In additional embodiments, an energy resolution position of 1,000 gigapixels can be determined based on the input variables.

[0032] [Limitations of current technology] [Active area, device electronics circuitry, packaging, and mechanical envelope] Figure 2 shows an apparatus 200 having an active area 220 with a particular mechanical form factor. The apparatus 200 includes a driver 230 and electronics circuitry 240 for power supply and can be connected to the active area 220, which has dimensions as indicated by the x and y arrows. This apparatus 200 does not take into account the cable wiring and mechanical structure for driving the power and cooling components, and the mechanical mounting area can be further minimized by introducing flexible cables into the apparatus 200. Also, the minimum mounting area of such an apparatus 200 can also be referred to as a mechanical envelope 210 having dimensions as indicated by the M:x and M:y arrows. This apparatus 200 is for illustrative purposes only, and application-specific electronics circuitry design can further reduce the mechanical envelope overhead, but in almost all cases, it cannot be the exact size of the active area of the apparatus. In one embodiment, this apparatus 200 shows a dependency on the electronics circuitry because it is associated with the active image area 220 for a micro-OLED, DLP chip, or LCD panel, or any other technology having the purpose of image illumination.

[0033] In some embodiments, it may also be possible to consider other projection techniques in order to aggregate multiple images onto a larger overall display. However, this can result in the cost of more complex projection distances, minimum foci, optical quality, uniform field resolution, chromatic aberration, thermal characteristics, calibration, alignment, additional size, or form factor. For the most practical applications, hosting dozens or hundreds of these projection sources 200 can result in a less reliable and larger-scale design.

[0034] For illustrative purposes only, assuming an energy device with an energy position density of 3840×2160 sites, the number of individual energy devices (e.g., device 100) desired for the given energy surface below can be calculated. [Equation 13] JPEG2025111410000014.jpg1386[Equation 14] JPEG2025111410000015.jpg1385

[0035] Assuming the above resolution considerations, approximately 105×105 devices similar to those shown in FIG. 2 may be desired. Note that numerous devices may be composed of various pixel structures that may or may not be mapped in a regular grid. If there are additional sub-pixels or positions within each complete pixel, these can be utilized to generate additional resolution or angular density. Using additional signal processing, a method can be determined to transform the light field to the correct (u,v) coordinates according to the specified positions of the pixel structures, and the known, calibrated explicit characteristics of each device can be obtained. Further, other energy regions may require different handling of these ratios and device structures, and those skilled in the art will understand the direct inherent relationships between each of the desired frequency regions. This will be shown and considered in more detail in the following disclosure.

[0036] Using the resulting calculations, one can understand how many of these individual devices may be desired to generate a maximum resolution energy surface. In this case, to achieve the vision threshold, approximately 105 x 105 devices, or approximately 11,080 devices, may be desired. There are challenges and novelty in creating a seamless energy surface from these available energy positions for sufficient sensory holographic propagation.

[0037] [Overview of Seamless Energy Surface] [Configuration and Design of Array Energy Relay] In some embodiments, an approach is disclosed for addressing the problem of generating high energy position intensity from an array of individual devices without joints due to mechanical structure constraints of each device. In one embodiment, an energy propagation relay system allows increasing the effective size of the active device area to meet or exceed mechanical dimensions, enabling the construction of an array of relays and forming a single seamless energy surface.

[0038] Figure 3 shows an embodiment of such an energy relay system 300. As shown in the figure, the relay system 300 may include a device 310 mounted on a mechanical envelope 320, and an energy relay element 330 propagates energy from the device 310. The relay element 330 can be configured to provide a function of reducing any gaps 340 that may occur when a plurality of mechanical envelopes 320 of the devices are arranged in an array of a plurality of devices 310.

[0039] For example, when the active region 310 of the device is 20 mm × 10 mm and the mechanical envelope 320 is 40 mm × 20 mm, the energy relay element 330 is designed at a magnification of 2:1 to create a tapered shape that is approximately 20 mm × 10 mm at the reduced end (arrow A) and 40 mm × 20 mm at the enlarged end (arrow B), providing the ability to seamlessly align an array of these elements 330 together without changing or colliding the mechanical envelopes 320 of each device 310. Mechanically, the relay elements 330 can be joined or fused together, aligned, and polished while ensuring a minimum seam gap 340 between each device 310. In such an embodiment, it becomes possible to achieve a seam gap 340 that is smaller than the limit of human visual acuity.

[0040] FIG. 4 shows an example of a base structure 400 having an energy relay element 410 formed together and securely fixed to an additional mechanical structure 430. The mechanical structure of the seamless energy surface 420 provides the function of coupling a plurality of energy relay elements 410, 450 in series to the same base structure through a bonding or other mechanical process for mounting the relay elements 410, 450. In some embodiments, each relay element 410 can be fused, joined, adhered, press-fitted, aligned, or otherwise attached together to form the resulting seamless energy surface 420. In some embodiments, the device 480 can be passively or actively aligned to ensure that it is mounted at the rear of the relay element 410 and centered at an appropriate energy position within a determined tolerance range.

[0041] In one embodiment, the seamless energy surface includes one or more energy positions, and one or more energy relay element stacks include first and second sides. Each energy relay element stack is arranged to form a single seamless energy surface that directs energy along a propagation path that extends between one or more energy positions and the seamless energy surface, where the end-to-end distance between any two adjacent second sides of the terminal energy relay elements is less than the minimum distinguishable contour as defined by human vision better than 20 / 40 at a distance greater than the width of the single seamless energy surface.

[0042] In one embodiment, each of the seamless energy surfaces includes one or more energy relay elements each having one or more structures that form first and second surfaces in the transverse and longitudinal directions. The first relay surface has a region different from the second relay surface that results in a positive or negative magnification, and is configured with an apparent surface contour for both the first and second relay surfaces that substantially satisfies an angle of ±10 degrees with respect to the normal of the surface contour across the entire second relay surface, such that energy passes through the second relay surface.

[0043] In one embodiment, the plurality of energy regions can be configured within a single energy relay or between multiple energy relays to direct one or more sensory holographic energy propagation paths including visual, auditory, tactile, or other energy regions.

[0044] In one embodiment, the seamless energy surface is composed of energy relays that include more than two first sides with respect to each second side, such that the seamless energy surface simultaneously receives and emits one or more energy regions to provide bidirectional energy propagation across the system.

[0045] In one embodiment, the energy relay is provided as a soft coherent element.

[0046] [General Introduction to Component Processing Structures: Disclosed Progress Regarding Lateral Anderson Localized Energy Relay] The characteristics of the energy relay can be significantly optimized according to the principles disclosed herein for energy relay elements that induce lateral Anderson localization. Lateral Anderson localization is the propagation of light rays that are disordered in the lateral direction but coherent in the longitudinal direction through a material.

[0047] This means that the influence of the material causing the Anderson localization phenomenon is less likely to be affected by total internal reflection than by the randomization between multiple scattering paths such that wave interference can completely limit lateral propagation while allowing longitudinal propagation to continue.

[0048] An additional important advantage is the removal of the cladding of conventional multi-core optical fiber materials. This cladding functionally removes the scattering of energy between fibers, but at the same time functions as a barrier to light energy, thereby reducing transmission by at least the core-to-cladding ratio (e.g., at a 70:30 core-to-cladding ratio, up to 70% of the received energy transmission can be transmitted), and further forming pixelated patterning within the propagated energy.

[0049] FIG. 5A shows an end view of an example of such a non-Anderson localized energy relay 500, where the image is relayed through a multi-core optical fiber where pixelation and fiber noise can appear due to the inherent characteristics of the optical fiber. Using conventional multi-mode and multi-core optical fibers, the relayed image is inherently prone to pixelation due to the total internal reflection characteristics of discrete array cores, where any inter-core crosstalk can degrade the modulation transfer function and increase contour blur. The resulting image generated using conventional multi-core optical fibers tends to have a residual fixed noise fiber pattern similar to that shown in FIG. 3.

[0050] Figure 5B shows an example of the same relay image 550 passing through an energy relay that includes a material exhibiting the characteristics of transverse Anderson localization, where the relay pattern has a greater density of particle structures compared to the fixed fiber pattern of Figure 5A. In one embodiment, a relay that includes a randomized micro-component processing structure induces transverse Anderson localization and more efficiently transports light with a resolvable resolution of propagation higher than that of a commercially available multimode glass optical fiber.

[0051] There are significant advantages in the transverse Anderson localization material characteristics with respect to both cost and weight, where a similar optical grade glass material can cost and weigh 10 to 100 times or more than the cost of the same material produced within one embodiment, and where the disclosed systems and methods present an important opportunity to improve both cost and quality over other techniques known in the art by including a randomized micro-component processing structure.

[0052] In one embodiment, a relay element exhibiting transverse Anderson localization can include a plurality of at least two different component processing structures in each of three orthogonal planes arranged in a one-dimensional lattice, the plurality of structures forming a randomized distribution of matter wave propagation characteristics in the transverse plane within the one-dimensional lattice and an equivalent channel of matter wave propagation characteristics in the longitudinal plane within the one-dimensional lattice, where the localized energy wave propagating through the energy relay has a higher transport efficiency longitudinally with respect to the transverse direction.

[0053] In one embodiment, a plurality of energy regions are configured within a single transverse Anderson localization energy relay or between a plurality of transverse Anderson localization energy relays and can be directed to one or more sensory holographic energy propagation paths that include visual, auditory, tactile, or other energy regions.

[0054] In one embodiment, the seamless energy surface is composed of a transverse Anderson localized energy relay that includes two or more first side surfaces for each second side surface so as to simultaneously receive and emit one or more energy regions and provide bidirectional energy propagation across the system.

[0055] In one embodiment, the transverse Anderson localized energy relay is configured as a loose coherent element or a flexible energy relay element.

[0056] [Considerations Regarding the 4D Prenoptic Function] [Selective Propagation of Energy Through a Holographic Waveguide Array] As discussed above and throughout this specification, a light field display system generally includes an energy source (e.g., an illumination source) and a seamless energy surface configured with a sufficient energy position density as clearly shown in the above discussion. Multiple relay elements can be used to relay energy from the energy device to the seamless energy surface. Once the energy is delivered to the seamless energy surface having the required energy position density, the energy can be propagated according to the 4D prenoptic function through the disclosed energy waveguide system. As will be understood by those skilled in the art, the 4D prenoptic function is known in the art and will not be further detailed herein.

[0057] The energy waveguide system selectively propagates energy passing through a plurality of energy positions along the seamless energy surface representing the spatial coordinates of the 4D prenoptic function, together with a structure configured to change the angular direction of the energy wave passing through by representing the angular component of the 4D prenoptic function, where the propagated energy wave can converge in space according to a plurality of propagation paths directed by the 4D prenoptic function.

[0058] Here, refer to FIG. 6 which shows an example of a light field energy surface in a 4D image space according to a 4D plenoptic function. This figure shows a ray tracing of the energy surface 600 to the viewer 620 when explaining how the energy rays converge in the space 630 from various positions within the viewing volume. As shown in the figure, each waveguide element 610 defines four-dimensional information that explains the energy propagation 640 passing through the energy surface 600. Two spatial dimensions (referred to herein as x and y) are a plurality of physical energy positions observable within the image space, and angular components θ and φ (referred to herein as u and v), which are observed in the virtual space when projected through the energy waveguide array. Usually, and according to the 4D plenoptic function, a plurality of waveguides (e.g., small lenses) can direct the energy position from the x, y dimensions to a specific position within the virtual space along the direction defined by the u, v angular components when forming the holographic or light field system described herein.

[0059] However, those skilled in the art will understand that an important issue for light field and holographic display technologies is that diffraction, scattering, diffusion, angular direction, calibration, focus, collimation, curvature, uniformity, element crosstalk, as well as decreasing effective resolution and many other parameters that contribute to the inability to accurately converge energy faithfully, can cause uncontrolled energy propagation due to designs that do not accurately consider any of these.

[0060] In one embodiment, an approach to selective energy propagation to address issues related to holographic displays may include substantially filling the apertures of the waveguides with energy that is substantially parallelized within an environment defined by an energy suppression element and a 4D plenoptic function.

[0061] In one embodiment, the array of energy waveguides defines a plurality of energy propagation paths for each waveguide element configured to substantially fill the effective aperture, through the effective aperture of the waveguide element, in a unique direction defined by a predetermined 4D function, towards a plurality of energy positions along a seamless energy surface suppressed by one or more elements positioned to limit the propagation at each energy position to pass through only a single waveguide element.

[0062] In one embodiment, the plurality of energy regions can be configured within a single energy waveguide or between a plurality of energy waveguides to direct one or more sensory holographic energy propagations including visual, auditory, tactile, or other energy regions.

[0063] In one embodiment, the energy waveguide and the seamless energy surface are configured to perform both reception and emission of one or more energy regions to provide bi-directional energy propagation across the system.

[0064] In one embodiment, the energy waveguide is configured to propagate non-linear or irregularly distributed energy and includes waveguide configurations for any seamless energy surface orientation including walls, tables, floors, ceilings, rooms, or other shape-based environments including non-transmitting void regions, leveraging digitally encoded, diffraction, refraction, reflection, GRIN, holographic, Fresnel, etc. In additional embodiments, the energy waveguide elements can be configured in a 360-degree configuration to generate various shapes providing any surface profile and / or desktop viewing enabling a user to view holographic images from anywhere around the energy surface.

[0065] In one embodiment, the energy waveguide array element can be a reflective surface, and the arrangement of the elements can be hexagonal, square, irregular, semi-regular, curved, non-planar, spherical, cylindrical, inclined regular, inclined irregular, spatially varying, and / or multi-layered.

[0066] For any component within a seamless energy surface, the components of the waveguide or relay can include, but are not limited to, optical fiber relays, silicon relays, glass relays, polymer relays, optical relays, diffractive elements, holographic elements, refractive or reflective elements, optical faceplates, energy couplers, beam splitters, prisms, polarization elements, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar materials exhibiting Anderson localization or total internal reflection.

[0067] [Realization of a Holodeck: [Aggregation of a Bidirectional Seamless Energy Surface System for Stimulating Human Sensory Receptors within a Holographic Environment] A large-scale environment of the seamless energy surface system can be constructed by tiling, fusing, adhering, pasting, and / or stitching together a plurality of seamless energy surfaces to form any size, shape, contour, or form factor, including an entire room. Each energy surface system can comprise an assembly having a base structure, an energy surface, relays, waveguides, devices, and electronic circuitry, collectively configured for bidirectional holographic energy propagation, emission, reflection, or sensing.

[0068] In one embodiment, the environment of the tiled seamless energy system is aggregated to form a large seamless planar or curved wall that includes a device covering all surfaces within a given environment, configured as any combination of seamless, discontinuous planar, faceted, curved, cylindrical, spherical, geometric, or irregular shapes.

[0069] In one embodiment, a set of flat surface tiles forms a wall-sized system for theater or venue-based holographic entertainment. In one embodiment, a set of flat surface tiles covers a room having four to six walls, including both the ceiling and floor for a cave-based holographic device. In one embodiment, a set of curved surface tiles creates a cylindrical seamless environment for an immersive holographic device. In one embodiment, a set of seamless spherical tiles forms a holographic dome for an immersive holodeck-based experience.

[0070] In one embodiment, a set of seamless curved energy waveguides provides a mechanical edge according to an accurate pattern along the boundary of energy suppression elements within an energy waveguide structure, and adheres, aligns, or fuses adjacent tile-shaped mechanical edges of adjacent waveguide surfaces to result in a modular seamless energy waveguide system.

[0071] In further embodiments of the set tile-shaped environment, energy is propagated bidirectionally with respect to a plurality of simultaneous energy regions. In additional embodiments, the energy surface provides the ability to simultaneously display and capture from the same energy surface using waveguides designed such that light field data can be projected through the waveguides by an illumination source and simultaneously received through the same energy surface. In additional embodiments, additional depth sensing and active scanning techniques can be utilized to enable interaction between energy propagation and a viewer in correct world coordinates. In additional embodiments, the energy surface and waveguides are operable to emit, reflect, or converge frequencies to induce tactile or volumetric tactile feedback. In some embodiments, any combination of bidirectional energy propagation and set surfaces is possible.

[0072] In one embodiment, the system comprises an energy waveguide capable of bidirectional emission and detection of energy through an energy surface having an energy coupler for two or more paths and one or more energy devices paired independently to pair at least two energy devices for the same portion of a seamless energy surface, or one or more energy devices are fixed in proximity to an additional component fixed to a base structure behind the energy surface, or in proximity to a position in front of and external to the FOV of a waveguide for off-axis direct or reflected projection or sensing, and the resulting energy surface enables the waveguide to converge energy, a first device to emit energy, and a second device to detect energy, where the information is processed to perform computer vision-related tasks including, but not limited to, 4D plenoptic tracking or detection of interference in the propagated energy pattern by the eye and retina, depth prediction, proximity, motion tracking, image, color or sound formation, or other energy frequency analysis. In a further embodiment, the tracked position actively calculates and corrects the position of the energy based on interference between the bidirectional capture data and the projection information.

[0073] In some embodiments, multiple combinations of three energy devices, including ultrasonic sensors, visible energy displays, and ultrasonic emitters, are configured together for each of three first relay surfaces that propagate energy coupled to a single second energy relay surface, each of the three first surfaces including designed characteristics specific to the energy region of each device, and two designed waveguide elements are configured for ultrasonic and energy, respectively, to provide the ability to direct and converge the energy of each device independently and substantially unaffected by other waveguide elements configured for another energy region.

[0074] In some embodiments, a calibration procedure that enables efficient manufacturing, and a dedicated integrated system for converting data into calibrated information suitable for energy propagation based on the calibrated configuration file, are disclosed for removing system artifacts and generating a geometric mapping of the resulting energy surface for use with encoding / decoding techniques.

[0075] In some embodiments, additional energy waveguides in series, and one or more energy devices can be integrated into the system to generate opaque holographic pixels.

[0076] In some embodiments, additional waveguide elements, including energy suppression elements, beam splitters, prisms, active parallax barriers, or polarization techniques, can be integrated to provide a spatial resolution and / or angular resolution greater than the diameter of the waveguide, or for other super-resolution purposes.

[0077] In some embodiments, the disclosed energy system can also be configured as a wearable bidirectional device such as virtual reality (VR) or augmented reality (AR). In other embodiments, the energy system can include an adjustment optical element that focuses the displayed or received energy proximate to a determined plane within the space for the viewer. In some embodiments, the waveguide array can be incorporated into a holographic head-mounted display. In other embodiments, the system can include multiple optical paths to enable the viewer to see both the energy system and the real-world environment (e.g., a transparent holographic display). In these examples, the system can be provided as a near-field in addition to other methods.

[0078] In some embodiments, the transmission of data involves receiving any dataset of information and metadata, analyzing this dataset, and receiving or assigning new pixel data that forms a sparser dataset of material properties, vectors, surface IDs, using an encodable process with a selectable or variable compression ratio. The received data can include 2D, volumetric, multi-view, metadata, light field, holographic, shape, vector, or vectorized metadata. The encoder / decoder can provide the ability to convert data in real-time or offline through a depth estimation algorithm with or without depth metadata, including image processing for 2D, 2D plus depth, metadata, or other vectorized information, volumetric, volumetric plus depth, metadata, or other vectorized information, multi-view, multi-view plus depth, metadata, or other vectorized information, holographic, or light field content. The backlight ray tracing methodology appropriately maps the resulting transformed data, generated by backlight ray tracing from various 2D, volumetric, multi-view, volume, light field, or holographic data, to the real world through a characterized 4D plenoptic function. In these embodiments, the desired total data transmission can have significantly fewer digits of information transmitted than the raw light field dataset.

[0079] [Energy-Directed Device Suitable for Presenting Holographic Sensory Data] In one embodiment, the optomechanical display device can emit and direct light to form 2D, volumetric, multi-view, plenoptic, 4D, volume, light field, holographic, or any other visual representation of light.

[0080] FIG. 7A is an example of a light field optical mechanical system configured using a waveguide realized as an array of refractive elements such as a light emitting display device, an optical relay, and a microlens array. A visible image from one or more displays can be optically relayed and then transmitted to an energy surface. The array of refractive elements provides a mapping between each position on the energy surface and the projection direction of light from that position such that a 4D volumetric light field image can be projected.

[0081] In one embodiment, the waveguide can be operable to converge light rays so as to cause both convergence and depth of field adjustment from the observer's perspective.

[0082] In one embodiment, the waveguide and the energy relay can be formed or polished with various surface shapes. In one embodiment, the energy relay includes an element that induces lateral Anderson localization. In one embodiment, the energy relay is bidirectional and can emit and / or project energy.

[0083] In one embodiment, an energy system configured to direct energy according to a four-dimensional (4D) plenoptic function includes a plurality of energy devices. In some embodiments, the plurality of energy devices include an illumination source that emits image information, which includes light emitting, projection, or reflective display technologies that utilize visible, IR, UV, coherent, laser, infrared, polarized, or any other electromagnetic illumination source. In other embodiments, the plurality of energy devices include a mechanical energy emission device configured to provide immersive audio or volumetric haptics from a sound field.

[0084] In some embodiments, the energy system configured as described above may further include a base structure (e.g., 72) such that a plurality of energy devices, an energy relay system, and an energy waveguide system can all be coupled to the base structure. In other embodiments, the plurality of energy devices, the energy relay system, and the energy waveguide system can be coupled to the base structure using one or more mounting brackets.

[0085] In some embodiments, the plurality of energy devices includes energy devices for capturing or detecting energy, including mechanical, chemical, transmission, thermal, electrical, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In other embodiments, the plurality of energy devices includes energy devices for propagating or releasing energy, including mechanical, chemical, transmission, thermal, electrical, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In still other embodiments, the plurality of energy devices includes acoustic receiving devices configured to provide sensory feedback or audible control.

[0086] In one embodiment, the energy system further includes an energy relay system (e.g., 6110, best shown in FIG. 7B) having one or more energy relay elements, each of the one or more energy relay elements including a first surface and a second surface, the second surface of the one or more energy relay elements being arranged to form a single seamless energy surface of the energy relay system, and a first plurality of energy propagation paths extending from energy locations within the plurality of energy devices through the single seamless energy surface of the energy relay system. This will be considered in more detail below.

[0087] Referring now to FIG. 7B, which shows an energy relay system 6110 according to an embodiment of the present disclosure in an orthogonal view. In one embodiment, the energy relay system 6110 can include two or more relay elements 6112, each relay element 6112 being formed of one or more structures, and each relay element 6112 having a first surface 6114, a second surface 6116, a lateral direction (generally parallel to the surfaces 6114, 6116) and a longitudinal direction (generally perpendicular to the surfaces 6114, 6116). In one embodiment, the surface area of the first surface 6114 can be different from the surface area of the second surface 6116. For example, the surface area of the first surface 6114 can be larger or smaller than the surface area of the second surface 6116. In another embodiment, the surface area of the first surface 114 can be the same as the surface area of the second surface 6116. Energy waves can pass from the first surface 6114 to the second surface 6116 or vice versa.

[0088] In one embodiment, the relay element 6112 of the energy relay system 6110 includes an inclined contour portion 6118 between the first surface 6114 and the second surface 6116. During operation, the energy wave propagating between the first surface 6114 and the second surface 6116 can have a higher transport efficiency in the longitudinal direction than in the lateral direction, and the energy wave passing through the relay element 6112 can result in a spatial expansion rate or a spatial contraction rate. In other words, the energy wave passing through the relay element 6112 of the relay element device 6110 may experience an increased expansion rate or a decreased contraction rate. In some embodiments, the one or more structures for forming the energy relay element 6110 can include glass, carbon, optical fibers, optical films, plastics, polymers, or mixtures thereof.

[0089] In one embodiment, the energy wave passing through the first surface 6114 has a first resolution, the energy wave passing through the second surface 6116 has a second resolution, and the second resolution is about 50% or more of the first resolution. In another embodiment, the energy wave has a uniform profile when provided to the first surface, but radiates in all directions with a forward energy density that substantially fills a cone having an opening angle of ±10 degrees with respect to the normal of the second surface, regardless of the position of the second relay surface, and can pass through the second surface.

[0090] In some embodiments, the first surface 6114 can be configured to receive energy from an energy wave source, and the energy wave source includes a mechanical envelope having a width different from the width of at least one of the first surface 6114 and the second surface 6116.

[0091] In each relay 6112, energy is transported between the first and second surfaces that define the longitudinal direction, and the first and second surfaces of each relay generally extend along the transverse direction defined by the first and second directions, and the longitudinal direction is substantially perpendicular to the transverse direction. In one embodiment, the energy wave propagating through a plurality of relays has a higher transport efficiency in the longitudinal direction than in the transverse direction due to the minimum refractive index variation in the longitudinal direction combined with the random refractive index variation in the transverse direction. In some embodiments where each relay is constructed of a multi-core fiber, the energy wave propagating within each relay element can travel in the longitudinal direction determined by the arrangement of the fibers in this orientation.

[0092] In one embodiment, the distance between the ends of any two adjacent second side surfaces of the terminal energy relay element can be smaller than the minimum distinguishable contour defined by the visual acuity of a human having a visual acuity better than 20 / 40 at a distance from the seamless energy surface, which is larger than the smaller of the height of a single seamless energy surface or the width of a single seamless energy surface.

[0093] In one embodiment, the plurality of energy relay elements in a stacked configuration can include a plurality of faceplates. In some embodiments, the plurality of faceplates may have different lengths or be a soft coherent optical relay. In other embodiments, the plurality of elements can have an inclined contour portion similar to that of FIG. 7B, and the inclined contour portion can be arranged at an inclination, straight line, curve, taper, chamfer, or non-right angle with respect to the normal axis of the relay element. In yet another embodiment, the energy wave propagating through the plurality of relay elements has a higher transport efficiency in the longitudinal direction than in the lateral direction due to the longitudinal minimum refractive index variation combined with the lateral random refractive index variation. In embodiments where each energy relay is constructed of a multi-core fiber, the energy wave propagating within each relay element can travel in the longitudinal direction determined by the arrangement of the fibers in this direction.

[0094] In some embodiments, one or more relay elements (e.g., 6112) include a fused or tiled mosaic, and any seam between adjacent fused or tiled mosaics is separated by a minimum recognizable contour as defined by the human visual acuity having a value better than 20 / 40 at a width or height of a single seamless energy surface or at a distance greater than that, or less than the minimum recognizable contour.

[0095] In other embodiments, one or more relay elements (e.g., 6112) include optical fibers, silicon, glass, polymers, optical relays, diffractive elements, holographic optical elements, refractive elements, reflective elements, optical faceplates, optical couplers, beam splitters, prisms, polarization components, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar material having Anderson localization or total reflection characteristics for forming a single seamless energy surface.

[0096] In yet other embodiments, one or more relay elements (e.g., 6112) are configured to conform to the shape of a single seamless energy surface, including planar, spherical, cylindrical, conical, faceted, tiled, regular, irregular, or any other geometric shape for a given application.

[0097] In another embodiment, the system further includes an energy waveguide system (e.g., 7100, best shown in FIGS. 7C - 7L) having an array of energy waveguides, and the second plurality of energy propagation paths extend in directions determined by a 4D plenoptic function from a single seamless energy surface through the array of energy waveguides.

[0098] FIG. 7C shows a top - down perspective view of one embodiment of an energy waveguide system 7100 operable to define a plurality of energy propagation paths 7108. The energy waveguide system 7100 includes an array of energy waveguides 7112 configured to direct energy to pass through the array of energy waveguides 7112 along the plurality of energy propagation paths 7108. In one embodiment, the plurality of energy propagation paths 7108 extend from a plurality of energy positions 7118 on a first side 7116 of the array to a second side 7114 of the array.

[0099] Referring to FIGS. 7C and 7L, in one embodiment, a first subset 7290 of a plurality of energy propagation paths 7108 extends through a first energy position 7122. The first energy waveguide 7104 is configured to direct energy along a first energy propagation path 7120 of the first subset 7290 of the plurality of energy propagation paths 7108. The first energy propagation path 7120 can be defined by a first principal ray 7138 formed between the first energy position 7122 and the first energy waveguide 7104. The first energy propagation path 7120 is formed between the first energy position 7122 and the first energy waveguide 7104 and can include light rays 7138A and 7138B directed by the first energy waveguide 7104 along energy propagation paths 7120A and 7120B, respectively. The first energy propagation path 7120 can extend from the first energy waveguide 7104 toward a second side 7114 of the array. In one embodiment, the energy directed along the first energy propagation path 7120 includes one or more energy propagation paths between or including energy propagation paths 7120A and 7120B that are directed through the first energy waveguide 7104 in a direction substantially parallel to the angle at which the first principal ray 7138 propagates through the second side 7114.

[0100] The embodiment can be configured such that the energy directed along the first energy propagation path 7120 can exit the first energy waveguide 7104 in a direction substantially parallel to the first principal ray 7138 and along energy propagation paths 7120A and 7120B. It can be assumed that the energy propagation paths extending through the energy waveguide element 7112 on the second side 7114 include a plurality of energy propagation paths in substantially the same propagation direction.

[0101] FIG. 7D is a front view of an embodiment of an energy waveguide system 7100. A first energy propagation path 7120 may extend toward a second side 7114 of the array in a unique direction 7208 extending from a first energy waveguide 7104 that is determined at least by a first energy position 7122. The first energy waveguide 7104 may be defined by spatial coordinates 7204, and the unique direction 7208 determined at least by the first energy position 7122 may be defined by angular coordinates 7206 that define the direction of the first energy propagation path 7120. The spatial coordinates 7204 and the angular coordinates 7206 can form a four-dimensional plenoptic coordinate set 7210 that defines the unique direction 7208 of the first energy propagation path 7120.

[0102] In one embodiment, the energy directed along the first energy propagation path 7120 through the first energy waveguide 7104 substantially fills a first aperture 7134 of the first energy waveguide 7104 and propagates along one or more energy propagation paths that are located between the energy propagation paths 7120A and 7120B and parallel to the direction of the first energy propagation path 7120. In an embodiment, the one or more energy propagation paths that substantially fill the first aperture 7134 may include more than 50% of the diameter of the first aperture 7134.

[0103] In a preferred embodiment, the energy directed along the first energy propagation path 7120 through the first energy waveguide 7104 that substantially fills the first aperture 7134 may include from 50% to 80% of the diameter of the first aperture 7134.

[0104] Referring again to FIGS. 7C and 7E-7L, in one embodiment, the energy waveguide system 7100 may further include an "energy suppression element 7124" positioned to restrict the propagation of energy between a first side 7116 and a second side 7114 and to suppress energy propagation between adjacent waveguides 7112. In one embodiment, the energy suppression element is configured to suppress energy propagation along a first subset 7290 of a plurality of energy propagation paths 7108 that do not penetrate a first opening 7134. In one embodiment, the energy suppression element 7124 may be disposed on the first side 7116 between an array of energy waveguides 7112 and a plurality of energy locations 7118. In one embodiment, the energy suppression element 7124 may be disposed on the second side 7114 between a plurality of energy locations 7118 and the energy propagation path 7108. In one embodiment, the energy suppression element 7124 can be disposed orthogonally to an array of energy waveguides 7112 or a plurality of energy locations 7118 on the first side 7116 or the second side 7114.

[0105] In one embodiment, energy directed along a first energy propagation path 7120 may converge with energy directed along a second energy propagation path 7126 through a second energy waveguide 7128. The first and second energy propagation paths may converge at a location 7130 on the second side 7114 of the array 7112. In one embodiment, third and fourth energy propagation paths 7140, 7141 may also converge at a location 7132 on the first side 7116 of the array 7112. In one embodiment, fifth and sixth energy propagation paths 7142, 7143 may also converge at a location 7136 between the first and second sides 7116, 7114 of the array 7112.

[0106] FIGS. 7E-7L show various embodiments of the energy suppression element 7124. For the sake of clarity, it is noted that these embodiments are provided for illustrative purposes and in no way limit the scope of the combinations or implementations provided within the scope of this disclosure.

[0107] Figure 7E shows an embodiment of a plurality of energy positions 7118 where the energy suppression element 7251 is disposed adjacent to the surface of the energy position 7118 and includes specific refraction, diffraction, reflection, or other energy modification characteristics. The energy suppression element 7251 may be configured to limit a first subset of the energy propagation paths 7290 to a narrower range of propagation paths 7253 by suppressing the propagation of energy along the energy propagation path 7252. In one embodiment, the energy suppression element is an energy relay having a numerical aperture of less than 1.

[0108] Figure 7F shows an embodiment of a plurality of energy positions 7118 where the energy suppression structure 7254 is disposed orthogonally between regions of the energy position 7118, the energy suppression structure 7254 exhibits absorption characteristics, and the energy suppression structure 7254 has a height defined along the energy propagation path 7256 such that a specific energy propagation path 7255 is suppressed. In one embodiment, the energy suppression structure 7254 is hexagonal in shape. In one embodiment, the energy suppression structure 7254 is circular. In one embodiment, the energy suppression structure 7254 has a non-uniform shape or size along any orientation of the propagation path. In one embodiment, the energy suppression structure 7254 is embedded within another structure having additional characteristics.

[0109] Figure 7G shows a plurality of energy positions 7118 where the first energy suppression structure 7257 is configured to substantially orient the energy 7259 propagating therethrough to a first state. The second energy suppression structure 7258 is configured to allow the energy 7259 substantially oriented in the first state to propagate therethrough and to limit the propagation of the energy 7260 oriented substantially differently from the first state. In one embodiment, the energy suppression elements 7257, 7258 are a pair of energy polarization elements. In one embodiment, the energy suppression elements 7257, 7258 are a pair of energy bandpass elements. In one embodiment, the energy suppression elements 7257, 7258 are a pair of diffractive waveguides.

[0110] FIG. 7H shows an embodiment of a plurality of energy positions 7118 in which an energy suppression element 7261 is configured to change the energy propagation path 7263 to some extent depending on which of the plurality of energy positions 7118 the energy propagation path 7263 extends through. The energy suppression element 7261 can change the energy propagation path 7263 uniformly or non-uniformly along the energy propagation path 7263 such that a particular energy propagation path 7262 is suppressed. The energy suppression structure 7254 is disposed orthogonally between regions of the energy position 7118. The energy suppression structure 7254 exhibits absorption characteristics and has a height defined along the energy propagation path 7263 such that a particular energy propagation path 7262 is suppressed. In one embodiment, the suppression element 7261 is a field lens. In one embodiment, the suppression element 7261 is a diffractive waveguide. In one embodiment, the suppression element 7261 is a curved waveguide surface.

[0111] FIG. 7I shows an embodiment of a plurality of energy positions 7118 in which an energy suppression element 7264 provides absorption characteristics to limit the propagation of energy 7266 while allowing other propagation paths 7267 to pass through.

[0112] FIG. 7J shows an embodiment of a plurality of energy positions 7118 and a plurality of energy waveguides 7112 in which a first energy suppression structure 7268 is configured to orient energy 7270 propagating through it to a substantially first state. A second energy suppression structure 7271 is configured to allow energy 7270 oriented in a substantially first state to propagate through it and to limit the propagation of energy 7269 oriented in a state substantially different from the first state. To further control the energy propagation through the system, as exemplified by stray energy propagation 7272, the energy suppression structures 7268, 7271 may require composite energy suppression elements to ensure that the energy propagation maintains an accurate propagation path.

[0113] FIG. 7K shows one embodiment of a plurality of energy positions 7118 where the energy suppression element 7276 provides absorption characteristics to limit the propagation of energy along the energy propagation path 7278 while allowing other energy along the energy propagation path 7277 to pass through a pair of energy waveguides 7112 for the effective aperture 7284 within the array of waveguides 7112. In one embodiment, the energy suppression element 7276 includes black chrome. In one embodiment, the energy suppression element 7276 includes an absorptive material. In one embodiment, the energy suppression element 7276 includes a transparent pixel array. In one embodiment, the energy suppression element 7276 includes an anodized material.

[0114] FIG. 7L shows an embodiment including a plurality of energy positions 7118 and a plurality of energy waveguides 7112, where a first energy suppression structure 7251 is disposed adjacent to the surface of an energy position 7118 and includes specific refraction, diffraction, reflection, or other energy modification characteristics. The energy suppression structure 7251 can be configured to limit a first subset of the energy propagation paths 7290 to a narrower range of propagation paths 7275 by suppressing the propagation of energy along the energy propagation path 7274. A second energy suppression structure 7261 is configured to modify the energy propagation path 7275 to some extent depending on which of the plurality of energy positions 7118 the energy propagation path 7275 extends through. The energy suppression structure 7261 can modify the energy propagation path 7275 uniformly or non-uniformly such that a specific energy propagation path 7274 is suppressed. A third energy suppression structure 7254 is disposed orthogonally between regions of the energy position 7118. The energy suppression structure 7254 exhibits absorption characteristics and has a height defined along the energy propagation path 7275 such that a specific energy propagation path 7274 is suppressed. The energy suppression element 7276 provides absorption characteristics to allow the passage of energy 7281 while restricting the propagation of energy 280. A composite system of similar or different waveguide elements 7112 is positioned to substantially fill the effective aperture 7285 of the waveguide element with energy from the plurality of energy positions 7118 and to modify the energy propagation path 7273 defined by a specific system.

[0115] In one embodiment, the energy suppression structure 7124 is disposed proximate to the first energy position 7122 and can generally extend toward the first energy waveguide 7104. In one embodiment, the energy suppression structure 7124 is disposed proximate to the first energy waveguide 7104 and can generally extend toward the first energy position 7122.

[0116] In one embodiment, the energy system is configured to direct energy along a second plurality of energy propagation paths through an energy waveguide system to a single seamless energy surface, and to direct energy along a first plurality of energy propagation paths from the single seamless energy surface through an energy relay system to a plurality of energy devices.

[0117] In another embodiment, the energy system is configured to direct energy along a first plurality of energy propagation paths from a plurality of energy devices through an energy relay system to a single seamless energy surface, and to direct energy along a second plurality of energy propagation paths from the single seamless energy surface through an energy waveguide system.

[0118] In yet another embodiment, the single seamless energy surface is operable to restrict local light transmission to visible light of three or fewer wavelengths.

[0119] [Sensory data suitable for holographic display] The plenoptic 4D function through the surface from the energy directing surface includes an energy position and is directed through a second coordinate along a second surface including guiding parameters u l (x l , y l , u l , v l ) that define the vector f of the energy propagation path, providing two spatial coordinates x l , y l from the first surface. Considering a plurality of energy directing surfaces, the plenoptic 5D function includes one or more energy positions and guiding parameters u l , y l that define the vector f of the energy propagation path, providing two spatial coordinates x l (x l , y l , z l , u l , v l ) that define the vector f of the energy propagation path, providing two spatial coordinates x l , v lThree spatial coordinates x l 、 y l 、 z l are provided, directed through a second coordinate along a plane containing the first coordinate. For each of 4D or 5D, additional variables f l (λ l 、 t l ) for time and color, even if not explicitly noted for the simplification and consideration of functions, can be considered and presumed to include any of the plenoptic functions. To clarify to avoid ambiguity, the reference to the energy-directed surface is for illustrative purposes only and can include any additional points, positions, directions, or planes within the space for the localization of 5D coordinates, and is generically referred to as the "energy-directed surface".

[0120] FIG. 8 is a flowchart diagram showing an embodiment of a process 800 for determining four-dimensional (4D) plenoptic coordinates for content data. Process 800 can include step 802 in which content data that can include any signal perceptible by a visual, auditory, tactile, sensory, or olfactory sensor is received. FIG. 9 is a schematic diagram showing an embodiment of content data that can include at least one of the position of an object, material properties (such as material properties 906, 907, and 908), virtual light source 904, shape 902 at a non-object position, content from a reference surface, virtual camera position 914, segmentation 910 of the object, background texture 912, and hierarchical content.

[0121] Referring to FIGS. 8 and 9, process 800 can further include step 804 where the positions of the data points are determined with respect to a first surface 920 for the generation of a digital three-dimensional representation 922 of the content data. The first surface 920 can be used as a reference plane for defining the positions of the data points in space. In one embodiment, process 800 can further include step 806 where the 4D plenoptic coordinates of the data points are determined at a second surface by tracking the positions of the data points in the three-dimensional representation to a second surface to which a 4D function is applied. In one embodiment, process 800 can further include step 808 where the energy source position values are determined for the 4D plenoptic coordinates having a first convergence point.

[0122] The content data received at step 802 can include N views, where N is greater than or equal to 1. A single view can be presented with or without using a depth channel. A stereoscopic view can be presented with or without using a depth channel. A multi-view image can be presented with or without using a depth channel. Further, a 4D light field can be presented with or without using a depth channel.

[0123] In the tracking of step 806, prior knowledge of the calibrated shape of the energy system, which can be stored in memory as a global model or an individually characterized system or some combination of these two methodologies, can be used.

[0124] In one embodiment, the mapping between the input data and the output energy source provides a methodology for accurately mapping between various bit rate sources. The tracking of step 806 provides the ability to infer a complete 4D volume data set from the partial samples enumerated above. Depth information can be provided or needs to be calculated from the available data. Using the known or calculated depth information, the N views can be backtracked by triangulation of samples from a known three-dimensional representation based on the depth coordinates into the 4D space.

[0125] In triangulation, when a mapping between an energy waveguide and an energy source position format resolution is provided, it can be assumed that each available energy source position within the N source contents represents the energy source position for each energy waveguide. When the resolution of the N source contents is lower, a super-resolution or scaling algorithm can be implemented. When the resolution of the N source images is higher than the number of energy waveguides within the energy directing device, interpolation between the super-sampled energy source positions can be performed to generate more energy source positions per energy waveguide in the resulting 4D ray back-tracing.

[0126] Above, it is assumed that the distance information can be determined from a depth map that may or may not be accurate depending on the form of the provided or calculated depth information, and given known or assumed distance information, the distance information combined with the x-y energy source position coordinates and the (u, v) angular information determined by the characteristics of the energy directing device can be regarded as a 4D or 5D light field with limited imaging data samples. The imaging samples can be triangulated back to appropriate energy source positions that may each exist behind each energy waveguide based on the distance information, and through the disclosure included herein, missing data can be generated at step 808.

[0127] Referring to FIGS. 7C, 8, 9, and 10, in one embodiment, the energy position may be located within a first surface 920, the second surface to which the 4D function is applied may correspond to the waveguide system 7100 of the energy directing device, and the energy is operable to be directed through the waveguide system according to the 4D plenoptic coordinates of the data points to form a detectable three-dimensional representation of the content data.

[0128] In one embodiment, process 800 can further include step 810 in which an energy source position value is determined for 4D coordinates having a first convergence point. To provide an exemplary implementation of the present disclosure, FIG. 11 shows an embodiment of an energy-directed device 1000 passing through a tracking process in which content data in the form of an image 1002 is provided or calculated within a range of a minimum position 1006 and a maximum position 1008 determined in relation to an energy position 1010, along with a distance position 1004. In one embodiment, the energy position 1010 can include the surface of the energy-directed device. The known shape from the energy position 1010 defined by the 4D plenoptic function allows triangulation of points 1014 on the virtual surface of the image 1002 along rays 1016 back to specific energy positions 1018, each having unique x-y coordinates. Missing samples can be computed by a computer based on the available information contained within the dataset.

[0129] When an additional N samples are provided, the same methodology is applied using additional multi-view imaging data that generates a richer set of back-ray tracing samples, providing excellent holographic results. Depth information from multiple N samples is provided through a single depth map having a known mapping between a source position (N + X viewpoints) and a source depth map (N + X depth maps), or through a number of depth maps up to or exceeding N, ensuring that appropriate back-ray tracing is performed.

[0130] For example, when a single depth map for the central N viewpoints is provided, additional depth maps can be interpolated by calculating the differences between each of the adjacent views, accurately mapping the positions of the source and target between the N viewpoints and the N + X viewpoints. Using this method, the correct viewpoints can be back-ray traced to a 4D light field for a visualizer's ability to project to appropriate waveguide coordinates and maintain the correct field-of-view dependence at the associated viewpoints.

[0131] The symbolizer and decoder are robust and can reconstruct the environment including 2D / flat files, 2D with depth, stereoscopic, stereoscopic with a single depth channel, stereoscopic with dual depth channels, N+X multi-view without depth, N+X multi-view with N+Y depth, texture, shape, lighting, material properties, etc., a shape or vector-based scene file, multiple RGBAZ values can be provided for each x-y coordinate, for a 4D or 5D (4D plus depth) light field, or a lower bandwidth methodology only for rendering the specific amount of energy source position data required for the field of view of a determined energy-directed device, a deep imaging file that can be provided as an N+X view plus N+Y delta channel data set including a plurality of data types including but not limited to these. With or without including world coordinate positions, with or without including compensated minimum and maximum projected world positions, and considering a uniquely characterized and / or designed energy-directed device, the processor can backtrace light at real-time speed or faster to provision an appropriate 4D light field to be presented to the viewer.

[0132] In one embodiment, process 800 can further include step 812 where a mapping is applied between the energy position 7122 on the first side of waveguide system 7100 and the angular direction of the energy propagation path 7120 from waveguide element 7100 on the second side of waveguide system 7100. By doing so, it becomes possible to determine a plurality of energy positions on the first side of waveguide system 7100 corresponding to the 4D plenoptic coordinates of the data points.

[0133] FIG. 12 is a schematic diagram of a processing system 1200 including a data input / output interface 1201 that communicates with a processing subsystem having a sensory data processor 1202, a vectorization engine 1204, and a tracing engine 1206. It should be understood that the sensory data processor 1202, the vectorization engine 1204, and the tracing engine 1206 can be implemented on one or more processors, either individually or in any combination thereof. Step 802 of process 800 can input content data to the processing subsystem 1220 via the data input / output interface 1201. Step 804 is executed by the sensory data processor 1202 to generate a three-dimensional representation of the content data (step 806).

[0134] In one embodiment, applying the mapping may include calibrating for distortions within the waveguide system 7100, and may further include calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion.

[0135] In one embodiment, the energy directing device can further include a relay system 6110 on a first side of the waveguide system 7100. This relay system has a first surface 6116 adjacent to the waveguide system 7100, and the energy position 7112 on the first side of the waveguide system can be positioned adjacent to a second surface 6114 of the relay system 6110.

[0136] In one embodiment, applying the mapping may include calibrating for distortions within the waveguide system 7100. In one embodiment, applying the mapping may include calibrating for both distortions within the relay system 6110 and within the waveguide system 7100. In one embodiment, the distortion to be calibrated may include at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion.

[0137] In one embodiment, part of this method can be executed in real time, or this method can be executed completely in real time, or at least two parts of this method can be executed in different periods.

[0138] [Conversion from 2D to light field] In one embodiment, the content data can include data points in a two-dimensional (2D) space, and determining the position in step 704 can include applying a depth map to the data points in the 2D space.

[0139] There are several methods for converting a 2D or planar image into light field data. These include the estimation of depth information through depth from motion analysis, depth channels provided by manual or rendered means, or the manual generation of parallax, depth, occlusion, shape, and / or the regeneration of a complete light field through the regeneration of the entire environment by manual and automated processes, and / or any other methodology known as a standard for visual effect content generation.

[0140] In a first embodiment, a system including a real-time or offline processor for performing depth estimation from available energy source position information is possible. This can be executed as a set-top box with an energy-directed device or as an offline process. Additional calculations for missing volume data can be performed using time information and / or state-of-the-art texture synthesis or other techniques known in the art.

[0141] In a second embodiment, the depth information is provided as an image stream and can be embedded in the image format. Similarly, additional calculations can be performed for missing volume measurement data.

[0142] In the third embodiment, missing environmental information is generated using an artist or process, which includes separating or segmenting each object in the scene, manually, semi-automatically, or automatically tracking the objects over time, and using parallax space, energy-directed device space, optical space, or world coordinates to synthesize the missing information of the background and foreground by a known visual effect process that regenerates the background, transparency, edge details, etc. to place the objects in the space. To clarify to avoid doubt, the processes implemented can be any, none, or all of the embodiments listed for the reconstruction of these environments. The generated environmental information should contain as much missing information as possible determined by the field of view angles of the energy-directed devices, and the artist will know to ensure that appropriate occlusion and field-of-view-dependent information is accurately generated.

[0143] Furthermore, a surface model of each object in the scene can be generated as either a partial model or a fully constructed model, and textures from the image data are projected onto the surfaces of the shapes to provide appropriate shapes for subsequent backlighting ray tracing.

[0144] Furthermore, it can be ensured that the accuracy of the 4D light field regeneration can be further enhanced by calculating or manually introducing material properties and introducing view-dependent lighting using virtual light sources.

[0145] Furthermore, the addition of CG or synthetic content can be introduced to enhance existing conversion materials. The addition of volume data can also be incorporated. The mutual mixing of N+X content can also be introduced to provide a seamless blend between CG, 2D, stereoscopic, multi-view, and / or 4D media within a single composite image.

[0146] The resulting 2D to light field converted content can be retained as a shape scene file containing shapes, textures, lighting, materials, etc., as shown in the CG scene itself, can be rendered as an N+X view with an N+D depth channel, can be rendered as a 4D or 5D (4D+depth) light field, a deep image in a format that allows multiple RGBAZ samples per x-y energy source position coordinate with or without a limit on the stacking of Z samples per x-y coordinate, or provided as an N+X view plus an N+Y delta channel dataset where the depth channel provides a lower bandwidth methodology for rendering only the specific amount of energy source position data required for the field of view of a determined energy directing device. Tools may be provided to enable all, some, or one of these respective output formats.

[0147] [Conversion from Stereo and Multi-View to Light Field] The processes from above that utilize single-view content can be applied to stereoscopic and multi-view materials. Depth information can be estimated through motion analysis and from one or multiple depth channels provided through stereoscopic, multi-view and / or disparity analysis, manual or rendered means, or through manual generation of disparity, depth, occlusion, shape, and / or regeneration of the entire environment through manual or automated processes, and through depth from any other methodology known as a standard for visual effect content generation that utilizes appropriate data to regenerate a full light field.

[0148] In one embodiment, the content data received at step 102 can include data points in a three-dimensional (3D) space, and the determination of position can include adjusting the data points in the 3D space.

[0149] In one embodiment, adjusting the data points in the 3D space may include applying a depth map to the data points in the 3D space, adding new data points, reconstructing occluded data points, or any combination thereof.

[0150] An important advantage of this approach is that the accuracy of stereoscopic disparity estimation is much higher than that of motion parallax or other similar 2D estimation processes alone. Furthermore, the image quality of the resulting transformed 4D light field is more accurate due to the availability of some of the view-dependent conditions including, but not limited to, illumination, transparency, material, occlusion, etc.

[0151] The ability to maintain the distinct angular dependence of the multi-view image data depends on the ability to calculate surface normals with respect to a central viewpoint camera, or some other defined central point. If these normals and disparity or depth information are known, it is possible to interpolate between viewpoints based on the angular field of view of the energy-directed device, and the resulting interpolation is then either applied directly to the back ray tracing or is synthesized as part of texture synthesis during back ray tracing.

[0152] For simplicity, all of the previously disclosed methodologies for the reconstruction from 2D to light field images can be applied to the reconstruction of stereo or multi-view datasets.

[0153] [Generation of N×N RGB Images from 4D or 5D Light Fields] By utilizing a 4D or 5D light field, it is possible to generate an RGB multi-view image of a number of N×N, or any value up to N×N. This process is adapted by considering each lower left coordinate under each waveguide, assuming a square grid, a 0,0 position, and an upper right position as N,N. This grid is merely illustrative, and other mapping methodologies can also be utilized. For each position from 0,0 to N,N, it is possible to form a full-resolution image from the light field with the widest possible depth of field based on the capture system utilized, and each waveguide within the array is considered a single energy source position, and each coordinate under each waveguide is a single energy source position within a larger energy source position array for each complete image from 0,0 to N,N. This can be repeated for the depth information of the 5D light field. In this way, for any subset of a data set desired to include any other desired combination of data derivable from 2D, stereo, multi-view, point cloud, CG scene file, or 4D or 5D light fields, it is possible to easily switch between 4D or 5D light fields. For irregular or square-packed 4D or 5D structures, additional interpolation is required to align the energy source positions to a regular grid, or a linear mapping between the energy source positions and a non-square-packed structure can be implemented, and the resulting image may not appear linear and may include artifacts of the energy source positions.

[0154] Figure 11 illustrates a methodology for converting from a 4D or 5D light field to multiple viewpoints by placing energy position 1102 under each energy waveguide element 1104 according to each of the energy waveguide element positions and energy position coordinates. This provides the ability to seamlessly transfer between the light field and a smaller data set.

[0155] [N+X RGB and N+Y depth data set] The ideal dataset format that provides the highest quality with a balance of data transmission size involves the use of N+X RGB and N+Y depth+vectorized channels, where the N+X RGB information includes N RGB images that can represent a specific resolution and format, X that can represent different resolutions and formats for RGB data to include lower resolution, delta information, etc., the N depth+vectorized channels that can represent a specific resolution and format, and the N+Y depth+vectorized channels that include Y that can represent different resolutions and formats for depth+vector data to include lower resolution, delta information, etc.

[0156] The number of N+X views can be generated from any methodology for determining the mapping of the number of views and the associated packing or line-of-sight positions from one radius around the center point with or without a center view, multiple radii around the center point with or without a center view, or on a regular grid. The configuration for the viewpoints can be included in the metadata of the file, or the provided depth+vectorized channels can include a direct mapping to world coordinates such that the imaging data aligns to the same coordinates in XYZ space without other required metadata.

[0157] [4D Disk Inversion and Compatibility Processing of Energy-Directing Devices] For any data captured using a plenoptic or light field 4D or 5D system, which may include those captured with a virtual rig using the optical simulation of a 4D or 5D light field system, the resulting fly-eye perspective includes a disk representing the uv vectors of the light field. However, these coordinates assume an energy focusing element that may not exist in the energy-directing device. In the proposed solution for the energy-directing device, the focusing element can be the viewer's eye, and the mapping between the capture systems and the mapping between the original capture method and the observed energy-directing device is no longer correct.

[0158] To correct for additional missing energy - directing elements within the system when this is inverted and compared to the capture system, each disk can be individually flipped independently, and the x - y position of each (u, v) coordinate is retargeted based on the center point of each waveguide. In this way, the inversion of the image formed as a result of the main waveguide is reversed, enabling the light - field energy - directing device to project the light rays in the correct x - y - u - v directions.

[0159] A further embodiment of this can implement a hardware modification where leveraging an energy waveguide array provides a direct inversion of the energy - source positions of all presented energy waveguides. For a light - field energy - directing device, this has the advantage of having a direct mapping between the expected capture system and the energy - directing device. This can be further advantageous for embodiments including an HMD system or a volumetric opaque energy - directing device, as it enables the removal of a group of energy waveguides across the array by eliminating the need for additional mid - relays for accurate x - y - u - v coordinates.

[0160] Furthermore, all light fields are not the same. They can be captured using different NAs, FOVs, N values, optical prescriptions, etc. The intrinsic and extrinsic nature of the input light - field data can be understood and converted into the characteristics of the energy - directing device. This can be performed by embodiments included within the present disclosure for the universal parameterization of holographic data and light - field data.

[0161] [Universal Parameterization of Holographic Sensory Data Transport by Inverse Energy Tracking and Vectorization of Sensory Characteristics for an Energy - Directing System] The plenoptic 4D function through the surface from the energy - directing surface includes the energy position and the vector f of the energy propagation path l (x l 、y l 、u l 、v lThe waveguide parameter u that defines l , v l Directed through the second coordinate along a second surface that includes, from the first surface, two spatial coordinates x l , y l are provided. Considering multiple energy-directed surfaces, the plenoptic 5D function includes one or more energy positions and the vector f of the energy propagation path l (x l , y l , z l , u l , v l ) The waveguide parameter u that defines l , v l Directed through the second coordinate along a surface that includes, from the first coordinate, three spatial coordinates x l , y l , z l are provided. For each of the 4D or 5D, additional variables f for time and color l (λ l , t l ) are considered, and for the sake of simplicity of the function and description, even if not explicitly noted, any of the plenoptic functions can be considered to be included if necessary for the application. To clarify to avoid ambiguity, the reference to the energy-directed surface is for illustrative purposes only and can include any additional points, positions, directions, or planes in space for the localization of 5D coordinates, and is collectively referred to as the "energy-directed surface".

[0162] Along the first vector of the energy propagation path, multiple intersections including the convergence of energy can occur with additional energy propagation paths. At this intersection, a 3D point or depth parameter is formed at the position X l , Y l , Z l between multiple energy propagation paths having a 4D or 5D function, where each x l , y l or x l , y l , z lFor the coordinates, a single u formed between the first coordinate and the 3D convergence point l , v l The 3D convergence points X l , Y l , Z l . The 4D function f z (x l , y l , u l , v l ) or the 5D function f z (x l , y l , z l , u l , v l ) collectively defines all 4D x l , y l or 5D x l , y l , z l coordinates, and the corresponding u l , Y l , Z l existing at X l , v l propagation paths for each convergence point.

[0163] At the first 5D coordinates resulting from the convergence of energy along multiple energy propagation paths passing through the energy-directed surfaces X l , Y l , Z l , this coordinate can represent a point within a larger object, volume, particle, or local energy parameter, and the energy converging at additional coordinates close to the first 5D coordinate can exhibit additional vectorized characteristics with respect to the sensory energy within the environment or holographic dataset. These vectorized characteristics can include information regarding each 5D coordinate, each energy position coordinate within the 4D dataset, a region within either the 4D or 5D dataset, or other subsets of coordinates including the energy surface.

[0164] In one embodiment, a universal parameterization of 4D and 5D holographic sensory energy characteristics for the propagation of visual energy, auditory energy, somatosensory energy, gustatory energy, olfactory energy, vestibular energy, or any other desired energy for the response of the sensory system is disclosed for 2D, 3D, 4D, and 5D data sets of raster and vector, where with respect to the second coordinates of the second surface of the 4D energy-directed surface, the 2D data can include a single angular sample, the 3D data can include two or more angular samples in a single dimension, the 4D data can include a plurality of angular samples in two dimensions, or the 5D data can include a plurality of angular samples in three or more dimensions. Embodiments of the received sample data can include any of the following: 1) 2D or monoscopic, flat, point cloud, UV mapping geometry, internal geometry, depth image, layered image, CAD file (proprietary), single point sampling, single camera capture, single projector projection, volumetric (monoscopic single sample points with vectors within the volume), 3 degrees of freedom source (DoF, raster with monoscopic x, y, z rotation about a single point), non-light field 6 DoF source (raster + vectors from monoscopic samples), volumetric energy-directed device (monoscopic samples within the volume), Pepper's ghost source (single point projection), 2D AR HMD source (monoscopic single or multiple focal planes, layered monoscopic), 2D VR HMD source (monoscopic single or multiple focal planes, layered monoscopic), or any other representation of two-dimensional raster or vector information. 2) 3D or stereoscopic, triscopic (single baseline), multi-view (1D), 1D multi-sample, 1D multi-viewpoint, horizontal or vertical only parallax, 1D projection array, two-point sampling, 1D point sampling, horizontal or vertical array, bullet time, 3DoF source (raster, stereoscopic x, y, z rotation centered on a single point), 3DoF source (3D raster within stereoscopic x, y, z rotation centered on a single point), non-light field 6 DoF source (vector from 3D raster + stereoscopic samples), 1D volume energy-directed device source (1D parallax-containing samples), autostereoscopic data source, horizontal multi-viewpoint energy-directed device source, 3D AR HMD source (stereoscopic single or multiple focal planes, layered stereoscopic), 3D VR HMD source (stereoscopic single focal plane or multiple focal planes, layered stereoscopic), or any other representation of three-dimensional raster or vector information. 3) 4D or plenoptic (5D), multi-scopic, integral image, light field (4D), holographic (4D), 2D multi-view, 2D multi-sample, 2D multi-viewpoint, 2D parallax, horizontal and vertical parallax, 2D projection array, 2D point sampling, motion capture stage (along the surface), planar array, eyewitness camera array, shape representation by rendering or ray tracing (4D representation), external geometry (4D representation), light field 6 DoF source (4D raster within planar light field samples), free viewpoint 6 DoF source (vector from 4D raster + 4D light field samples), 4D volume energy-directed device source (2D parallax-containing samples), light field energy-directed device source (4D sampling), light field HMD source (near-field 4D sampling), holographic energy-directed device source (4D sampling), or any other representation of four-dimensional raster or vector information. 4) 5D or plenoptic + depth, light field + depth, holographic (5D sampling, 4D + depth), any multi-view (along all of the x, y, and z axes), multi-sample (along all of xyz), multi-viewpoint (along all of xyz), volumetric parallax (along all of xyz), projection array (along all of xyz), point sampling (along all of xyz), motion capture stage (along all of xyz), eyewitness camera array (any xyz configuration), shape representation by rendering or ray tracing (5D representation), cube or volume rendering (along all of xyz), external geometry (5D representation), light field 6 DoF source (5D raster within a volumetric light field sample), free viewpoint 6 DoF source (vector from 5D raster + 5D light field sample), 5D volumetric energy directing device source (multi-planar 4D sampling), 5D light field energy directing device source (5D sampling, 4D + multiple planes), 5D light field HMD source (near-field 5D sampling, 4D + multiple planes), holographic energy directing device source (5D sampling, 4D + multiple planes), or any other representation of 5D raster or vector information.

[0165] At each of the second coordinates, the data provided can include any subset or superset of either raster samples or vector samples, where the samples represent additional vectorized information that enables conversion to an increased sampling density through the interpretation or processing of a subset or superset of raster samples or vector samples and can be included.

[0166] For each of the provided 2D, 3D, 4D, or 5D data sets, the information is transformed by vectorized information, manual identification, computer vision analysis, automated processing, or other means of converting the provided samples from the original data set into a 5D coordinate system. For each of the provided 2D, 3D, 4D, or 5D data sets, the information can include multiple samples or layers of samples regarding the original angular sampling components for each provided data set, and additional vectorized characteristics, with reference to the second coordinates of the second plane of the 4D energy-oriented surface, or can include a combination of contributing samples for any of the additional provided 2D, 3D, 4D, or 5D data sets.

[0167] Each of the provided samples contains an intrinsic energy for each desired coordinate, which can include additional extrinsic energy attributes, and the intrinsic energy represents the value at a given 5D coordinate in the absence of other external samples, characteristics, or environmental conditions. In the electromagnetic spectrum, this corresponds to a white body that reflects all incident radiation and may be referred to as albedo, a dimensionless measure of reflectivity that clearly spans each desired sensory energy for which a range of dimensionless values corresponds to a particular sensory energy. Within the visual system, this range is approximately 400 nm to 700 μm, and within the auditory system, this range is approximately 20 Hz to 20 kHz.

[0168] Over the past few decades, there have been tremendous technological improvements that artificially utilize electronic means to recognize high-level patterns of detected sensations, scents, and flavors, enabling the reproduction of human sensations. In other systems that may exist outside the electromagnetic spectrum, these dimensionless values can be characterized in the same way based on the perceived sharpness response. Although holographic sensory energy technology has newly emerged, the embodiments disclosed herein include systems, methods, and formats for stimulating all human senses in a virtual environment, clarifying the universal construction for various sensory parameters, and by doing so, providing for the proper processing, transmission, storage, vectorization of data, and transfer to, from, or between any sensory energy parameters or devices desired for the complete immersion of the constructed virtual environment. Embodiments of energy propagation for holographic sensory technology will be disclosed in future applications. Additionally, it is the intention of this disclosure to enable other analog devices, including novel products like classical "smellovision" or modern versions like the VR headset with a scent from FeelReal, to utilize the parameterized values provided within the vectorization of the dataset of this specification.

[0169] In one embodiment, the somatosensory system has a texture mechanical receptor with a pressure sensitivity range in the skin that can be normalized from 50 Hz to 300 Hz, a thermoreceptor with a temperature sensitivity range in the skin that can be normalized from 0 °C to 50 °C (where this range can be a much wider range with upper and lower limits defined by excessive temperatures), or many physical properties including variables for time, strain, modulus of elasticity among other kinetic properties, and for the purposes of the present disclosure, is simplified to a dimensionless normalization scale with a value of 0 for immovable solids such as granite and 1 for low-viscosity fluids such as water, and includes surface deformability that defines the range of viscoelastic behavior of both viscous and elastic material measurements when undergoing deformation between stress and strain over time, and can be defined based on components that define sensitivity. Those skilled in the art will understand that the actual vectors provided include the physical properties necessary to properly define the viscoelasticity of the material and are normalized only for purposes of illustration.

[0170] Finally, the advancement of the state of the art in artificial electronic sensing, including taste and olfactory devices, further vectors the sensory parameters disclosed for holodeck design parameters, showing a feasible path enabling the electronic reproduction of artificial taste and smell via holographic waveguide means as described herein. Artificial electronic taste and olfactory receptors have made significant progress through newly emerging nanodevices, through a pattern recognition system that results in the detection of tastes that constitute human taste, and through encoding and conversion to frequency-based pulses that repeatedly and accurately detect taste as the frequency of the sampled chemical composition to sample the intensity of chemical stimuli using enzyme biosensors. This technology is considered extensible to all types of detectable tastes, and similar progress in artificial olfactory systems, along with current research that further parameterizes the patterns contained within the frequency of a specific olfactory response by changes in electronic signals, has demonstrated a digital interface for stimulating the human olfactory receptor body using weak electrical pulses targeted at the nasal conchae.

[0171] A pathway is established for arbitrarily generating frequencies and complex electronic patterns representing smell, taste, and other sensory systems. In one embodiment, it can include vectors for sour, salty, bitter (spiciness), sweet, and deliciousness (umami), but is not limited thereto. The sharpness response of taste can be vectorized to include a normalized scale for each parameter electronically controlled along a scale of 0 to 1 representing the minimum and maximum taste responses to saturate the average human's 2,000 - 8,000 taste buds. The vectors of the vectorized signals and the spatial coordinates can provide information for fabrication for the realization of complex smells.

[0172] In another embodiment, for each of the very complex olfactory spaces, which can include vectors for good smell, fruity smell, citrus smell, woody (resinous) smell, chemical smell, sweet smell, mint (peppermint) smell, toasted (nutty) smell, pungent smell, and putrid smell, but is not limited thereto, the sharpness response of smell can be further vectorized to include a normalized scale for each parameter electronically controlled along a scale of 0 to 1 representing the minimum and maximum olfactory responses to saturate the average human's olfactory epithelium of 10 cm. The vectors of the vectorized signals and the spatial coordinates can provide information for fabrication for the realization of complex smells. 2 For each of these vectors, for the appropriate application of the provided vectorized values, normalized values representing these patterns can be provided for taste, smell, or other sensory regions, which are converted to waves, amplitudes, magnitudes, or other attributes as needed. Smell and taste are two of the most degraded senses within the sensory system, but with parameterized values for vectorizing complex blends, in embodiments, it is further possible to provide user-based interactive control of the sensitivity of any such sensory energy to provide customization of the individuation of each of the visual, auditory, somatosensory, taste, smell, vestibular, or other desired sensory system responses.

[0173] For each of these vectors, for the appropriate application of the provided vectorized values, normalized values representing these patterns can be provided for taste, smell, or other sensory regions, which are converted to waves, amplitudes, magnitudes, or other attributes as needed. Smell and taste are two of the most degraded senses within the sensory system, but with parameterized values for vectorizing complex blends, in embodiments, it is further possible to provide user-based interactive control of the sensitivity of any such sensory energy to provide customization of the individuation of each of the visual, auditory, somatosensory, taste, smell, vestibular, or other desired sensory system responses.

[0174] In one embodiment, each of the represented sensory albedo energy values of the samples may further include extrinsic energy attributes baked into a single sample value representing the additive result of each sample provided for each of the other external samples, characteristics, or environmental conditions. In this configuration, the composite sample value may or may not indicate potential attributes of other energies from other samples in a physics-based or simulated environment. The most efficient and pure way to transmit a parameterized and reconstructed holographic data set is based on a single unique sample information that provides simplified and lower bandwidth frequency information, but this method may not necessarily be possible to receive outside of a complete synthetic environment, especially in physics-based imaging or acoustic systems. In any real-world environment, there is always some degree of external contribution to the resulting sample information. Certain systems such as light stages, or other systems known in the art that facilitate the estimation of reflectance, shape, texture, and motion capture, utilize some form of structured illumination and one or more imaging devices that provide direct or indirect analysis of surface characteristics such as albedo, depth information, surface normals, and bidirectional scattering distributions.

[0175] The bidirectional scattering distribution function (BSDF) is a generalized superset of the bidirectional transmittance distribution function (BTDF), bidirectional texture function (BTF), and bidirectional reflectance distribution function (BRDF) that collectively act as models for parameterizing and identifying surface characteristics in computer graphics and vision algorithms known in the art. This function represents how visible light is reflected, transmitted, or otherwise interacts with the surface when the surface normal is perpendicular to the tangent of the target surface and the incident direction w r (w i , w r ) of the incident light and the reflected or transmitted direction w i of the outgoing light are given for the energy propagation path, and the function represents the reflected radiance exiting along the outgoing path w r for the incident path w r i ​Represents the ratio to the irradiance incident on the surface along, w i , w r Each of which may include a 4D function that defines the parameterized azimuth and zenith angles of each of the incident and exit paths.

[0176] The energy λ hitting the surface i at the first position x i , and the material properties scatter this energy internally to the second position x r of energy λ r To further clarify these functions for the exit after internal scattering, the output energy, input energy and position, and the effects of visible wavelengths such as output energy and position over the entire surface of the object, volume or point, obtained based on the material properties of the surface, such as iridescence, luminescence, subsurface scattering, non-local scattering effects, specular reflection, shadowing, masking, mutual reflection, etc., can be explained.

[0177] Therefore, to account for wavelength or frequency dependence, and spatially varying material properties or surfaces, the generalized property for explaining how energy is transported between any two energy lines hitting the surface can be represented as a 10D function, and for each or any of the available or provided samples in the dataset as f r (λ i , x i , w i , λ r , x r , w r ), specifying the input energy, the effect of the vectorized surface shape, and the output of the results of reflection, refraction, specular, transmission, scattering, diffusion, or other material properties from any energy region, can be explained when the generalized function f r is given.

[0178] Considering an energy-directed surface here, the plenoptic 4D function includes the energy position and the vector of the energy propagation path f l (x l , y l , u l , vl The waveguide parameter u that defines l , v l Two spatial coordinates x are directed through the second coordinate along a second plane that includes l , y l are provided. Considering multiple energy-directed surfaces, the plenoptic 5D function includes one or more energy positions, and the vector f of the energy propagation path l (x l , y l , z l , u l , v l The waveguide parameter u that defines l , v l Three spatial coordinates x are directed through the second coordinate along a plane that includes l , y l , z l are provided. For each of 4D or 5D, additional variables f for time and color l (λ l , t l ) can be considered to be included in any of the plenoptic functions if necessary for the application, even if not explicitly noted to simplify the function and description.

[0179] Along the first vector of the energy propagation path, multiple intersections including the convergence of energy can occur with additional energy propagation paths. At this intersection, a 3D point or depth parameter is formed at the position X l , Y l , Z l between multiple energy propagation paths where only a single u l , v l or x l , y l , z l propagation path angle is present between the first coordinate and the 3D convergence point for each x l , v l coordinate within the energy-directed 4D surface or 5D surface, and the 3D convergence point X l , Yl , Z l 4D function f z (x l , y l , u l , v l ) or 5D function f z (x l , y l ,z l , u l , v l ) is for all 4Dx l , y l or 5Dx l , y l , z l coordinates, and X for each convergence point l , Y l , Z l The corresponding u l , v l Collectively define the propagation path.

[0180] Convergence coordinate X l , Y l , Z l A surface is formed at the surface, which may include points, volumes, objects, or other representations that contain the 3D locations of the focused energy propagation paths. The samples provided for each surface location are used to generate one or more surface features, vectors, materials, characteristics, or other distinguishing properties V for characterizing or otherwise processing the resulting energy. i , as well as one or more input energy sources impinging at specific points proximate to the surface location, where the reflectance function is a 11D universal object parameterization function f for various properties of the surface. r (λ i , x i , w i , λ r , x r , w r , V i )

[0181] 11D universal holographic parameterization function f r (λ i , x i , wi 、 λ r 、 x r 、 w r 、 V i ) defines the resulting value for specific environments and vectorized characteristics, and the 4D function f l (x l 、 y l 、 u l 、 v l ) defines the energy propagation path from the surface of the energy-directed device, and thus, the 15D universal holographic parameterization function f r (λ i 、 x i 、 w i 、 λ r 、 x r 、 w r (x l 、 y l 、 u l 、 v l )、 V i ) can be further generalized, where the transmitted direction w r is made equal to defining the propagation path of u l 、 v l , thereby defining the spatial coordinates x l 、 y l , and for each transmitted direction w r , w r = u l 、 v l satisfies, and there is only one set of values of f l (x l 、 y l 、 u l 、 v l ). One of ordinary skill in the art will understand various transformation and mathematical composition concepts in addition to the rendering requirements related to the disclosed universal parameterization of 4D and 5D holographic sensory energy characteristics.

[0182] Providing an executable path that enables the transmission of a truly holographic data set by substantially eliminating the required multi-digit data using a complete 15D function that shows all sensory energy characteristics being vectorized to coincide with the surface formed from the convergence points in space.

[0183] The vectorized characteristics aim to provide the exact physical characteristics of each of the sensory areas for characteristics that can be synthetically programmed, captured, or evaluated by a computer, V i For a particular sample within a dataset provided for general system metadata or for each or any sensory energy area, including the following, for each surface, volume, or 3D coordinate X l Y l Z l the attributes of the characteristics vectorized in can be specified. 1) System metadata can provide either sensory energy-specific attributes regarding the surface characteristics of each sample or a system-wide reference, including normal, depth information, environmental characteristics, multiple angle samples for a given 3D coordinate, procedural textures, geometric shapes, point clouds, deep image data, static frames, time frames, video data, surface IDs, surface paths, coordinate maps, virtual camera coordinates, virtual lighting and visible energy information, environmental maps, scene information outside the field of visual sample information, curves, vertices, time information, network data, databases, object recognition, energy devices, external data feeds, sensors for system changes and interactions, system status, voice recognition, olfactory detection, auditory detection, face recognition, somatosensory recognition, taste recognition, UI, UX, user profiles, flow and motion vectors, layers, regions, transparency, segments, animations, sequence information, procedural information, displacement maps, or any other scene data necessary to provide sufficient data for the proper processing of each sample. 2) Visual energy is visible or invisible electromagnetic energy, rainbow colors, luminescence, subsurface scattering, non-local scattering effects, specular reflection, shadowing, absorbance, transmittance, masking, inter-reflection, albedo, transparency, physical properties, dynamic properties, reflection, refraction, diffraction, optical effects, atmospheric effects, frequency, modulation, surface shape, texture, displacement map, physical and dynamic properties (e.g., sound vibrations that change reflectivity properties, or deformation of a tactile material that causes surface deformation) for interacting and responding with other sensory energies in particular based on the provided energy, layers, regions, transparency, segments, curves, animations, sequence information, procedural information, material size, environmental conditions, room dynamics, or surface properties for defining an appropriate rendering of other material properties related to surfaces, environments, rooms, objects, points, volumes, etc. 3) Auditory energy: Arrangement, size, amplitude, mass, material propagation parameters, absorbance, transmittance, material properties indicating acoustic reflectivity, diffusion, transmittance, enhancement, masking, scattering, localization, frequency dependence or modulation, pitch, tone, viscosity, smoothness, texture, elastic modulus, any other parameters determining the propagation of sound waves within an object, physical and dynamic properties (e.g., temperature that changes the sound of a material) for interacting and responding with other sensory energies in particular based on the provided energy, layers, regions, transparency, segments, curves, animations, sequence information, procedural information, material size, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, etc., vectors related thereto. 4) Provide many physical properties including variables for mechanical receptacles for texture, pressure, temperature receptacles, temperature, and time, strain, modulus of elasticity among other kinetic properties, and surface deformability parameters and vectors that define the range of viscoelastic behavior of both viscous and elastic material measurements when undergoing deformation between stress and strain over time, layers, regions, transparency, segments, curves, animations, sequence information, procedure information, material sizes, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other somatosensory parameters, and somatosensory energy vectors related to them. 5) Taste energy vectors for good smells, fruit odors, citrus odors, wood (resin) odors, chemical odors, sweet odors, mint (peppermint) odors, toasted (nutty) odors, pungent odors, and putrid odors, where the vectors and spatial coordinates of the vectorized signals can provide information for fabrication for the realization of complex olfaction, and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, regions, transparency, segments, curves, animations, sequence information, procedure information, material sizes, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other taste parameters. 6) Olfactory energy vectors for sourness, saltiness, bitterness (spiciness), sweetness, and deliciousness (umami), where the vectors and spatial coordinates of the vectorized signals can provide information for fabrication for the realization of complex olfaction, and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, regions, transparency, segments, curves, animations, sequence information, procedure information, material sizes, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other taste parameters. 7) Alternatively, physical, synthetic, transmitted, or computationally interdependent, required, designed, or demanded sensory vectors from any other sensory sample dataset, and parameterizations of specific features, as well as any additional sensory characteristics where the generalization of holographic configuration data reconstruction, storage, processing, or transmission would benefit from them, based on other mutually related sensory dynamics.

[0184] If there is a received dataset that includes 2D data with a single angle sample, 3D data with two or more angle samples in a single dimension, 4D data with multiple angle samples in two dimensions, or 5D data with multiple angle samples in three or more dimensions.

[0185] For all source materials provided, each source material can undergo an additional process to properly prepare for the efficient vectorization of the holographic dataset. For any provided sheath material that exhibits its energy-directed surface with lower spatial or angular resolution, a conversion process may be required to accurately convert the original source into a 4D or 5D dataset.

[0186] For proper preparation, in one embodiment, the 2D or 3D source material provided includes photo capture from a standard imaging system. This series of images includes raster reflections, refractions, transparent elements, and other similar examples of the interaction of material properties with physics-based lighting.

[0187] If the content is prepared simply by identifying the surface ID of a surface that already has rasterized material properties, the available data may be sufficient to converge in a 4D coordinate system, but any additional rendering applied to these surfaces will show double images due to the photographic properties and the physical properties of the parameterized, synthetically rendered reflection properties. The ideal source data set for efficient holographic transmission includes the albedo representation of the sample source information and the vectorized material properties for each of the specific energy regions that form the object-based volume sampling of the albedo multi-view samples, with all material properties providing accurate surface identification and rendering, as well as localization or projection of other sensory energies, based precisely on the specific vectorized surface properties.

[0188] In one embodiment, a manual, semi-automatic, computer vision, or automated process is provisioned to algorithmically or manually evaluate the content within a source sample dataset, and a manual or algorithmic analysis is performed, thereby performing segmentation and other object separation methods known in the art to identify regions that include an undesired physical rasterization effect. In one embodiment, a person is photographed in front of a background, the physical properties of the person include reflections from the environment, and the background objects are blocked by the photographed person.After these areas are identified as undesirable, a process can be utilized to 1) isolate the object in question, 2) separate all object elements to reveal occlusion, transparency, edges, or other elements, and 3) facilitate additional hardware and energy devices that capture additional information regarding machine learning, computer vision, and scenes, objects, and / or the environment, such that any surface indicative of material properties is removed by computer vision, algorithms, processors, or manual visual effects through image analysis, temporal analysis, energy analysis, or completely by manual means, such that object elements are provisioned to have any such baked-in material properties, where the manual process reproduces the intrinsic material properties when there are no extrinsic material properties to prepare the content for the most efficient transmission and propagation of the dataset, using methods generally known in the art for performing wire removal, paint fix, clean plate, image restoration, alpha matte creation, occlusion filling, object recreation, image projection, motion tracking, camera tracking, rotoscope, optical flow, etc., 4) the additional processes described above include manual or computer-assisted identification of depth or 3D coordinate values for each of the desired samples, and 5) further within the scope of this embodiment, there is an identification of the relevant material properties represented by each of the points, data regions, surfaces, objects, or other representations of materials such that the data can be further easily rendered within the display driver of the energy-directed device or within any additional system capable of encoding or decoding the parameterized dataset.

[0189] In one embodiment, the dataset from the above includes an albedo visual energy sample prepared using 3D multi-view samples, each having rgba information of multiple layers, a series of vectorized material properties for associating each segmented material with a surface ID, and a series of surface parameters for exactly reconstructing the original source dataset before removal of the exogenous image data. The acoustic dataset is provisioned with the material properties of the visual energy system and vectorized material properties associated with a plurality of acoustic channels, each having an identified frequency, modulation, spatial arrangement, and other sound source localization properties. The somatosensory energy dataset is provided for a subset of the surfaces included in the visual energy dataset such that both the viscoelastic and temperature vectorized material properties are further included, both of which are related to other vectorized datasets.

[0190] From any provided dataset, each provided sample from the visual energy dataset is evaluated for its relative depth position with respect to the surface of the energy directing device. For any of the visual energy samples, each of the samples is placed in a 3D coordinate system, and the energy propagation path length for each of the provided samples is at position X among a plurality of coexisting convergent energy propagation paths that intersect a first 3D point in a 4D or 5D plenoptic function l 、Y l 、Z l and evaluated with respect to a function that correlates each 3D coordinate with respect to a plurality of coexisting convergent energy propagation paths that intersect at the first 3D point. Here, for each x l 、y l or x l 、y l 、z l coordinates, only a single u l 、v l propagation path angle is formed between the first coordinate and the 3D convergence point. The 4D function f z (x l 、y l 、u l 、v l ) or the 5D function f z(x l 、y l 、z l 、u l 、v l ) represents all 4D x l 、y l 、 or 5D x l 、y l 、z l coordinates, and X l 、Y l 、Z l corresponding u l 、v l propagation paths are collectively defined for each convergence point in, and after this analysis process is executed, the total number of samples per 4D x l 、y l 、 or 5D x l 、y l 、z l spatial coordinate becomes known, and the total energy propagation path length between each 3D point in positions X l 、Y l 、Z l and the 4D or 5D coordinate positions becomes known, and the weighted distribution based on the total available samples per 4D or 5D coordinate, and the minimum path length to the 3D coordinate values sampled from the available multiple 3D coordinate data provide a complete sampling of the 4D or 5D light field from any dataset.

[0191] As a further embodiment described above, 1) each of the samples for any of vision, hearing, somatic sensation, and any other provided energy samples, 2) after being arranged in a 3D coordinate system based on the provided dataset, additional processing, or additional vectorized characteristics, and before performing coordinate analysis, 3) the 15D universal holographic parameterization function f r (λ i 、x i 、w i 、λ r 、x r 、w r (x l 、y l 、u l 、v l )、Vi ) is evaluated, 4) additional known environmental scenes, geometric shapes, metadata, etc. each having independent vectorized material properties are provided, 5) virtual lighting information is provided and any possible interference between additional sensory energy metadata properties that may change the rendering function is evaluated, 6) a 15D parameterization function is evaluated for each provided 3D coordinate and corresponding vectorized material property, 7) when any provided dataset is given, the rendering process is executed via an online, offline, real-time, ASIC, FPGA, cloud, or other form of rendering process to yield new multiple angularly varying material properties, where 8) this rendering process defines each of the propagation paths u l , v l and is specific to each of their equal transmission directions w r , thus defining the spatial coordinates x l , y l , and for each transmission direction w r , only one set of values of f l (x l , y l , u l , v l ) that satisfies w r = u l , v l can exist, 9) based on the rendering result and the resulting available new angularly varying material properties, for each of the 4D or 5D coordinates including the energy propagation path length for each provided sample, it is evaluated in relation to a function that associates each 3D coordinate with a plurality of coexisting converging energy propagation paths intersecting at a first 3D point at positions X l , Y l , Z l within the 4D or 5D plenoptic function, where for each x l , y l or x l , y l , z l coordinate included within the energy-directed 4D or 5D surface, only one u is formed between the first coordinate and the converging 3D pointl , v l There is only the propagation path angle. The 4D function f z (x l , y l , u l , v l ) or the 5D function f z (x l , y l , z l , u l , v l ) includes all 4D x l , y l , or 5D x l , y l , z l coordinates within the energy directing device, and for each convergence point X l , Y l , Z l where the corresponding u l , v l propagation paths are collectively defined. After this analysis process is executed, the total number of samples per 4D x l , y l , or 5D x l , y l , z l spatial coordinate is known, the total energy propagation path length between each 3D point at positions X l , Y l , Z l and the 4D or 5D coordinate positions is known, the weighted distribution based on the total available samples per 4D or 5D coordinate, and the minimum path length to the 3D coordinate values sampled from the available multiple 3D coordinate data provide a complete sampling of the 4D or 5D light field for all the provided sensory energy from any dataset.

[0192] The rendering further reveals a bidirectional energy-directed surface such that the perceived electromagnetic energy representing the lighting of the real-world environment, or the absorbance of a specific acoustic frequency within the environment, can provide dynamic or offline updates to the rendering process or other perceived interactive real-world elements, and lighting and acoustic or other sources are adjusted to adapt to changes in environmental conditions.

[0193] Referring again to FIG. 8 and considering the above principles, in one embodiment of process 800, the received content data can further include vectorized material property data, and process 800 further includes step 830 in which a digital three-dimensional representation of the content data is associated with the vectorized material property data, and in step 804, determining the energy source position value is based at least on the vectorized material property data associated with the volumetric representation of the content data.

[0194] Referring to FIGS. 9 and 13, in one embodiment, vectorization process 1300 can include step 1302 in which first content data is received and step 1304 in which surface 915 within the content data is identified. In one embodiment, identifying surface 915 can include using segmentation data within the content data. Vectorization process 1300 can further include step 1306 in which a surface identification of surface 915 is determined and step 1308 in which material property data for surface 915 is determined. In one embodiment, determining the material property data can include a manual determination or using a predetermined process. After steps 1306 and 1308, vectorization process 1300 can further include step 1310 in which the surface identification is associated with the material property data of surface 915. Vectorization process 1300 can further include step 1312 in which a vector of the material property data is generated. Vectorization process 1300 can further include step 1314 in which vectorized material property data is generated based on the generated vector.

[0195] In one embodiment, process 1300 can optionally include step 1316, in which the material property data is removed from the first content data and replaced with the vectorized material property data from step 1314. In one embodiment, the vectorized material property data from step 1314 can be used in process 800 as described above to determine the 4D plenoptic coordinates of the energy-directed device of the present disclosure as described above.

[0196] Process 1300 can be executed using any processing system of the present disclosure, including processing system 1200. In one embodiment, the content data can be received via data input / output interface 1201 at step 1302, and steps 1304 - 1314 of vectorization process 1300 can be executed using vectorization engine 1204. Further, the vectorized material property data generated at step 1314 can be used by sensory data processor 1202 and tracing engine 1206 to be processed according to the steps of process 800 as described above. Steps 808 and 812 can be executed by the tracing engine to determine 4D coordinates for holographic display. Step 810 can be executed by sensory data processor 1202. The output of the processing subsystem can be provided to compression engine 1210, from which the compressed data can be stored in memory or provided to data input / output interface 1201 to be transmitted to an energy-directed system connected locally or remotely to system 1210. The data can also be stored in memory 1208 until retrieved later.

[0197] A hologel is part of a light field hologram. The hologel in the figure can be rendered.

[0198] Although various embodiments have been described above in accordance with the principles disclosed herein, it should be understood that these are for illustrative purposes only and are not intended to be limiting. Accordingly, the scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the claims and their equivalents arising from the present disclosure. Further, while advantages and features have been presented in the described embodiments, these are not intended to limit the application of such issued claims to any process and structure that achieves any or all of the above advantages.

[0199] It will be understood that the main features of the present disclosure can be employed in various embodiments without departing from the scope of the present disclosure. One of ordinary skill in the art will recognize, or be able to ascertain using no more than routine experimentation, numerous equivalents to the specific procedures described herein. Such equivalents are within the scope of the present disclosure and are covered by the claims.

[0200] Furthermore, the section headings in this specification are provided for consistency with 37 C.F.R. § 1.77 or, alternatively, to provide a clue to the organization. These headings do not limit or characterize the invention described in any claim that may issue from this disclosure. Specifically, by way of example, a heading refers to the "Field of the Invention," but such a claim should not be limited by the words under this heading that describe the so-called field of technology. Further, the description of the technology in the "Background of the Invention" section should not be construed as an admission that such technology is prior art to any invention in this disclosure. The "Summary of the Invention" also should not be regarded as characterizing the invention described in the issued patent claims. Further, references to the "invention" in the singular in this disclosure should not be used in an argument that there is only one novel point in this disclosure. Multiple inventions may be presented that fall within the limitations of multiple claims arising from this disclosure, and such claims accordingly define the inventions and their equivalents that are thereby protected. In any event, the scope of such claims should be considered in light of their own merits in view of this disclosure, but should not be constrained by the headings set forth herein.

[0201] The use of the word "a" or "an" may mean "one" when used in combination with the term "comprising" in the claims and / or the specification, but is also consistent with the meanings of "one or more", "at least one", and "one or more than one". The use of the term "or" in the claims, while the present disclosure supports a definition that refers only to alternatives and "and / or", is not explicitly shown to mean only alternatives and is used to mean "and / or" unless the alternatives are mutually exclusive. Throughout this application, the term "about" is used to indicate that a value includes the inherent variability of error with respect to the device or method used to determine that value, or the variability that exists between research subjects. In general, but subject to the above discussion, numerical values in this specification modified by approximating terms such as "about" may vary from the stated value by at least ±1, 2, 3, 4, 5, 6, 7, 10, 12, or 15%.

[0202] As used in this specification and claims, the word "comprising" (and any form of comprising such as "comprise" and "comprises"), "having" (and any form of having such as "have" and "has"), "including" (and any form of including such as "includes" and "include"), or "containing" (and any form of containing such as "contains" and "contain") is inclusive or non-limiting and does not exclude additional unrecited elements or method steps.

[0203] Words of comparison, measurement, and timing, such as "at the time", "equivalent", "during", "complete", etc., should be understood to mean "substantially at the time", "substantially during", "substantially equivalent", "substantially complete", etc., where "substantially" means that such comparison, measurement, and timing are practicable to achieve the desired result, whether implicitly or explicitly stated. Terms regarding the relative position of elements, such as "near", "proximate", "adjacent to", etc., shall be taken to mean close enough to materially affect the interaction of the respective system elements. Other approximating words, likewise, when so modified, are not necessarily absolute or complete, but refer to conditions that are considered close enough for one of ordinary skill in the art to ensure that the condition exists as specified. The range within which the description may vary depends on how great a change may occur and further makes one of ordinary skill in the art recognize that the modified feature still has the features and capabilities required of the unmodified feature.

[0204] As used herein, the term "or combinations thereof" refers to all permutations and combinations of the items listed before that term. For example, "A, B, C, or combinations thereof" is intended to include at least one of A, B, C, AB, AC, BC, or ABC, and, if order is important in a particular context, also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB. Continuing with this example, clearly included are combinations that include repetitions of one or more of the items or terms, such as BB, AAA, AB, BBC, AAABCCCC, CBBAAA, CABABB, etc. One of ordinary skill in the art will understand that there are typically no restrictions on the number of items or terms in any combination, unless it is clear from the context otherwise.

[0205] All of the disclosed and claimed configurations and / or methods herein can be made and executed without undue experimentation in light of the present disclosure. Although the configurations and methods of the present disclosure are described with respect to preferred embodiments, it will be apparent to one of ordinary skill in the art that changes can be applied to the configurations and / or methods and to the steps or sequence of steps of the methods described herein without departing from the concept, spirit, and scope of the present disclosure. All such similar substitutes and modifications apparent to one of ordinary skill in the art are considered to be within the spirit, scope, and concept of the present disclosure as defined by the appended claims. [Item 1] A method for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, comprising: receiving content data; determining the position of a data point with respect to a first surface for generating a digital three-dimensional representation of the content data, wherein the first surface is a reference surface; tracking the position of each of the data points to the first surface through a plurality of positions, wherein the plurality of positions are on a second surface. Determining a set of 4D plenoptic coordinates for each data point based on a plurality of tracked intersection points on the first surface and the second surface; Determining a first set of energy source position values for a first set of 4D plenoptic coordinates having a first convergence point for a first data point; Determining a second set of energy source position values for a second set of 4D plenoptic coordinates having a second convergence point for a second data point different from the first data point, comprising: A method, wherein the first set and the second set of 4D plenoptic coordinates are determined based on different pluralities of tracked intersection points on the first surface and the second surface. [Item 2] The method according to item 1, wherein the content data includes signals perceptible by visual, auditory, tactile, sensory, or olfactory sensors. [Item 3] The method according to item 1 or 2, wherein the content data includes at least one of object position, material properties, virtual light source, content of the shape at non-object position, content from the reference surface, virtual camera position, segmentation of the object, and hierarchical content. [Item 4] The method according to any one of items 1 to 3, wherein the content data includes data points in a two-dimensional space (2D space), and the step of determining the position includes applying a depth map to the data points in the two-dimensional space. [Item 5] The method according to any one of items 1 to 4, wherein the content data includes data points in a three-dimensional space (3D space), and the step of determining the position includes adjusting the data points in the 3D space. [Item 6] The method according to item 5, wherein the step of adjusting includes applying a depth map to the data points in the 3D space. [Item 7] The method according to item 5 or 6, wherein the step of adjusting includes adding new data points. [Item 8] The method according to any one of Items 5 to 7, wherein the adjusting step includes reconstructing the masked data points. [Item 9] The method according to any one of Items 1 to 8, wherein the second surface corresponds to a waveguide system of the energy directing device, and the energy is operable to be directed through the waveguide system to form a detectable three-dimensional representation of the content data according to the 4D plenoptic coordinates of the data points. [Item 10] The method according to Item 9, further comprising applying a mapping between an energy position on a first side of the waveguide system and an angular direction of an energy propagation path from a waveguide element on a second side of the waveguide system, wherein a plurality of energy positions on the first side of the waveguide system corresponding to the 4D plenoptic coordinates of the data points are determined by applying the mapping. [Item 11] The method according to Item 10, wherein the step of applying the mapping includes calibrating for distortion within the waveguide system. [Item 12] The method according to Item 11, wherein the step of calibrating for distortion within the waveguide system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 13] The method according to any one of Items 9 to 12, wherein the energy directing device further comprises a relay system on a first side of the waveguide system, the relay system has a first surface adjacent to the waveguide system, and further, the energy positions on the first side of the waveguide system are positioned adjacent to a second surface of the relay system. [Item 14] The method according to Item 13, wherein the step of applying the mapping includes calibrating for distortion within the waveguide system. [Item 15] The method according to item 13 or 14, wherein the step of applying the mapping includes a step of calibrating for distortion in the relay system. [Item 16] The method according to item 15, wherein the step of applying the mapping includes a step of calibrating for distortion in the waveguide system. [Item 17] The method according to item 15 or 16, wherein the step of calibrating for distortion in the relay system includes a step of calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 18] The method according to any one of items 9 to 17, wherein the energy position is arranged within the first surface. [Item 19] The received content data further includes vectorized material property data, and the method further includes a step of associating the digital three-dimensional representation of the content data with the vectorized material property data, and determining the energy source position value is based at least on the vectorized material property data associated with the digital three-dimensional representation of the content data. The method according to any one of items 1 to 18. [Item 20] The method according to any one of items 1 to 19, wherein at least a part of the method is performed in real time. [Item 21] The method according to any one of items 1 to 20, wherein the method is executed completely in real time. [Item 22] The method according to any one of items 1 to 21, wherein at least two parts of the method are executed in different periods. [Item 23] A method for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, receiving content data; determining the position of a data point relative to a reference point position; Vectorizing the data points by generating vectors of the data points based on the reference point positions; Determining positions of the data points with respect to a first surface to generate a digital three-dimensional representation of the content data based on the vectorized data points, the first surface being a reference surface; Tracking the positions of each of the data points to the first surface through a plurality of positions on a second surface; Determining a set of 4D coordinates for each data point based on a plurality of tracked intersection points on the first and second surfaces. A method comprising: [Item 24] The method according to item 23, wherein the content data includes signals perceptible by visual, auditory, tactile, sensory, or olfactory sensors. [Item 25] The method according to item 23 or 24, wherein the content data includes at least one of object position, material properties, virtual light source, content of the shape at non-object positions, content from the reference surface, virtual camera position, segmentation of the object, and hierarchical content. [Item 26] The method according to any one of items 23 to 25, wherein the content data includes data points in a two-dimensional space (2D space), and the step of determining positions includes applying a depth map to the data points in the two-dimensional space. [Item 27] The method according to any one of items 23 to 26, wherein the content data includes data points in a three-dimensional space (3D space), and the step of determining positions includes adjusting the data points in the 3D space. [Item 28] The method according to item 27, wherein the adjusting step includes applying a depth map to the data points in the 3D space. [Item 29] The method according to item 27 or 28, wherein the adjusting step includes adding new data points. [Item 30] The method according to any one of Items 27 to 29, wherein adjusting includes reconstructing the obscured data points. [Item 31] The method according to any one of Items 23 to 30, wherein the second surface corresponds to a waveguide system of the energy directing device, and the energy is operable to be directed through the waveguide system to form a detectable three-dimensional representation of the content data according to the 4D plenoptic coordinates of the data points. [Item 32] The method according to Item 31, further including applying a mapping between an energy position on a first side of the waveguide system and an angular direction of an energy propagation path from a waveguide element on a second side of the waveguide system, and a plurality of energy positions on the first side of the waveguide system corresponding to the 4D plenoptic coordinates of the data points are determined by applying the mapping. [Item 33] The method according to Item 32, wherein applying the mapping includes calibrating for distortion in the waveguide system. [Item 34] The method according to Item 33, wherein calibrating for distortion in the waveguide system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 35] The method according to any one of Items 31 to 34, wherein the energy directing device further includes a relay system on a first side of the waveguide system, the relay system has a first surface adjacent to the waveguide system, and further the energy positions on the first side of the waveguide system are positioned adjacent to a second surface of the relay system. [Item 36] The method according to Item 35, wherein applying the mapping includes calibrating for distortion in the waveguide system. [Item 37] The method according to item 35 or 36, wherein the step of applying the mapping includes calibrating for distortion within the relay system. [Item 38] The method according to item 37, wherein the step of applying the mapping includes calibrating for distortion within the waveguide system. [Item 39] The method according to item 37 or 38, wherein the step of calibrating for the distortion within the relay system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 40] The method according to any one of items 31 to 39, wherein the energy position is arranged within the first surface. [Item 41] The method according to any one of items 23 to 40, wherein the received content data further includes vectorized material property data, the method further includes associating the digital three-dimensional representation of the content data with the vectorized material property data, and the step of determining the energy source position value is based at least on the vectorized material property data associated with the digital three-dimensional representation of the content data. [Item 42] Receiving first content data; Identifying a surface within the first content data; Determining material property data of the surface; Generating a vector of the material property data; Generating vectorized material property data based on the generated vector, wherein the vectorized material property data is defined by a set of four-dimensional plenoptic coordinates (4D plenoptic coordinates), or includes a format adaptable to the association with the surface identification of the surface independent of the vectorized material property data. A method for vectorization including the step of generating. [Item 43] The method according to item 42, wherein the step of identifying the surface includes using segmentation data within the first content data. [Item 44] The method according to item 42 or 43, wherein the step of determining the material property data includes manual determination. [Item 45] The method according to any one of items 42 to 44, wherein the step of determining the material property data includes using a predetermined process. [Item 46] Determining the position of the data points of the surface with respect to the reference surface for the step of generating a digital three-dimensional representation of the first content data, Associating the digital three-dimensional representation of the first content data with the vectorized material property data, Determining the 4D plenoptic coordinates of the set of data points of the surface on the 4D application surface by tracking the position of the data points within the digital three-dimensional representation to a second surface to which a 4D function is applied, Determining an energy source position value of the 4D coordinates having a first convergence point, wherein determining the energy source position value is based on at least the vectorized material property data associated with the digital three-dimensional representation of the first content data, the method according to any one of items 42 to 45, further comprising the step of: [Item 47] The method according to item 46, further comprising the step of removing the material property data. [Item 48] A system for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, An input / output interface configured to receive content data, A processing subsystem including a sensory data processor and a tracing engine, communicating with the input / output interface, The sensory data processor is configured to determine the position of data points of the content data with respect to the first surface and generate a digital three-dimensional representation of the content data, where the first surface is a reference surface, The tracing engine is configured to trace the position of each of the data points to the first surface through a plurality of positions, where the plurality of positions are on a second surface, The tracing engine is further configured to determine a set of 4D plenoptic coordinates for each data point based on a plurality of traced intersection points on the first surface and the second surface, determine an energy source position value for a first set of 4D plenoptic coordinates having a first convergence point for a first data point, and determine an energy source position value for a second set of 4D plenoptic coordinates having a second convergence point for a second data point different from the first data point, A system, wherein the first set and the second set of 4D plenoptic coordinates are determined based on different pluralities of traced intersection points on the first surface and the second surface. [Item 49] The system according to item 48, wherein the content data includes signals perceptible by visual, auditory, tactile, sensory, or olfactory sensors. [Item 50] The system according to item 48 or 49, wherein the content data includes at least one of object position, material properties, virtual light source, content of the shape at a non-object position, content from the reference surface, virtual camera position, segmentation of the object, and hierarchical content. [Item 51] The system according to any one of items 48 to 50, wherein the content data includes data points in a two-dimensional space (2D space), and determining the position includes applying a depth map to the data points in the two-dimensional space. [Item 52] The system according to any one of items 48 to 51, wherein the content data includes data points in a three-dimensional space (3D space), and determining a position includes adjusting the data points in the 3D space. [Item 53] The system according to item 52, wherein adjusting includes applying a depth map to the data points in the 3D space. [Item 54] The system according to item 52 or 53, wherein adjusting includes adding new data points. [Item 55] The system according to any one of items 52 to 54, wherein adjusting includes reconstructing occluded data points. [Item 56] The system according to any one of items 48 to 55, wherein the second surface corresponds to a waveguide system of an energy directing device, and energy is configured to be directed through the waveguide system to form a detectable three-dimensional representation of the content data according to the 4D plenoptic coordinates of the data points. [Item 57] The system according to item 56, wherein the tracing engine is further configured to apply a mapping between an energy position on a first side of the waveguide system and an angular direction of an energy propagation path from a waveguide element on a second side of the waveguide system, and the plurality of energy positions on the first side of the waveguide system corresponding to the 4D plenoptic coordinates of the data points are determined by applying the mapping. [Item 58] The system according to item 57, wherein applying the mapping includes calibrating for distortion in the waveguide system. [Item 59] The system according to item 58, wherein calibrating for distortion in the waveguide system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 60] The energy-directed device further comprises a relay system on a first side of the waveguide system, the relay system having a first surface adjacent to the waveguide system, and the energy position on the first side of the waveguide system being positioned adjacent to a second surface of the relay system, the system according to item 56. [Item 61] Applying the mapping includes calibrating for strain in the waveguide system, the system according to item 60. [Item 62] Applying the mapping includes calibrating for strain in the relay system, the system according to item 61. [Item 63] Calibrating for the strain in the relay system includes calibrating for at least one strain selected from the group consisting of spatial strain, angular strain, intensity strain, and color strain, the system according to item 62. [Item 64] The energy position is disposed within the first surface, the system according to any one of items 56 to 63. [Item 65] At least two functions of the system are executed during different periods, the system according to any one of items 48 to 64. [Item 66] A system for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, An input / output interface configured to receive content data, A processing subsystem that communicates with the input / output interface and includes a sensory data processor, a vectorization engine, and a tracing engine, The sensory data processor is configured to determine the position of data points in the content data relative to a reference point position, The vectorization engine is configured to vectorize the data points based on the reference point position, The above-mentioned sensory data processor is further configured to determine the position of the data points with respect to the first surface based on the vectorized data points, and generate a digital three-dimensional representation of the content data, where the first surface is a reference surface, The tracing engine is configured to trace the position of each of the data points up to the first surface through a plurality of positions, where the plurality of positions are on a second surface, The tracing engine is further configured to determine a set of 4D plenoptic coordinates for each data point based on a plurality of traced intersection points on the first surface and the second surface, determine an energy source position value for a first set of 4D plenoptic coordinates having a first convergence point for a first data point, and determine an energy source position value for a second set of 4D plenoptic coordinates having a second convergence point for a second data point different from the first data point, A system, wherein the first set and the second set of 4D plenoptic coordinates are determined based on different pluralities of traced intersection points on the first surface and the second surface. [Item 67] The system according to item 66, wherein the content data includes signals perceptible by visual, auditory, tactile, sensory, or olfactory sensors. [Item 68] The system according to item 66 or 67, wherein the content data includes at least one of object position, material properties, virtual light source, content of the shape at the non-object position, content from the reference surface, virtual camera position, segmentation of the object, and hierarchical content. [Item 69] The system according to any one of items 66 to 68, wherein the content data includes data points in a two-dimensional space (2D space), and determining the position includes applying a depth map to the data points in the two-dimensional space. [Item 70] The system according to any one of items 66 to 69, wherein the content data includes data points in a three-dimensional space (3D space), and determining a position includes adjusting the data points in the 3D space. [Item 71] The system according to item 70, wherein adjusting includes applying a depth map to the data points in the 3D space. [Item 72] The system according to item 70 or 71, wherein adjusting includes adding new data points. [Item 73] The system according to any one of items 70 to 72, wherein adjusting includes reconstructing occluded data points. [Item 74] The second surface corresponds to a waveguide system of an energy directing device, and energy is configured to be directed through the waveguide system to form a detectable three-dimensional representation of the content data according to the 4D plenoptic coordinates of the data points, according to any one of items 66 to 72. [Item 75] The system further includes applying a mapping between an energy position on a first side of the waveguide system and an angular direction of an energy propagation path from a waveguide element on a second side of the waveguide system, and a plurality of energy positions on the first side of the waveguide system corresponding to the 4D plenoptic coordinates of the data points are determined by applying the mapping, according to item 74. [Item 76] The system according to item 75, wherein applying the mapping includes calibrating for distortion in the waveguide system. [Item 77] The system according to item 76, wherein calibrating for the distortion in the waveguide system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion. [Item 78] The energy directing device further comprises a relay system on a first side of the waveguide system, the relay system having a first surface adjacent to the waveguide system, and further, an energy position on the first side of the waveguide system is positioned adjacent to a second surface of the relay system, the system according to any one of items 74 to 77. [Item 79] Applying the mapping includes calibrating for distortion in the waveguide system, the system according to item 78. [Item 80] Applying the mapping includes calibrating for distortion in the relay system, the system according to item 78 or 79. [Item 81] Calibrating for the distortion in the relay system includes calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and color distortion, the system according to item 80. [Item 82] The energy position is disposed within the first surface, the system according to any one of items 74 to 81. [Item 83] The received content data further includes vectorized material property data, and the processing subsystem associates the digital three-dimensional representation of the content data with the vectorized material property data, and is further configured to determine an energy source position value based on at least the vectorized material property data associated with the digital three-dimensional representation of the content data, the system according to any one of items 66 to 82. [Item 84] A system for vectorization, An input / output interface configured to receive content data, and A processing subsystem that communicates with the input / output interface and includes a vectorization engine, The vectorization engine is configured to identify a surface within the content data and determine material property data for the surface, The vectorization engine is further configured to generate a vector of the material property data and generate vectorized material property data based on the generated vector, The system, wherein the vectorized material property data includes a format adaptable for association with a surface identification of the surface defined by a set of four-dimensional plenoptic coordinates (4D plenoptic coordinates) of a single set that is otherwise independent of the vectorized material property data. [Item 85] The system according to item 84, wherein identifying the surface includes using segmentation data within the content data. [Item 86] The system according to item 84 or 85, wherein determining the material property data includes manual determination. [Item 87] The system according to any one of items 84 to 86, wherein determining the material property data includes using a predetermined process. [Item 88] The processing subsystem is A sensory data processor configured to determine positions of data points of the surface relative to a reference surface for generating a digital three-dimensional representation of first content data and associate the digital three-dimensional representation of the first content data with the material property data, And a tracing engine configured to determine a set of 4D plenoptic coordinates of the data points of the surface at a 4D application surface by tracing the positions of the data points within the digital three-dimensional representation to a second surface to which a 4D function is applied. The tracing engine is further configured to determine an energy source position value for a 4D coordinate having a first convergence point, and determining the energy source position value is based on at least the vectorized material property data associated with the digital three-dimensional representation of the first content data, the system according to any one of items 84 to 87. [Item 89] The system according to item 88, further comprising removing the material property data. [Item 90] A system for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, An input / output interface configured to receive content data, A processing subsystem comprising a sensory data processor, a vectorization engine, and a tracing engine, in communication with the input / output interface, A compression engine in communication with the processing subsystem and the input / output interface, Comprising the compression engine, the input / output interface, and an optional memory in communication with the processing subsystem, The sensory data processor is configured to determine the position of data points in the content data relative to a reference point position, The vectorization engine is configured to vectorize the data points based on the reference point position, The sensory data processor is further configured to determine the position of the data points relative to a first surface to generate a digital three-dimensional representation of the content data based on the vectorized data points, the first surface being a reference surface, The tracing engine is configured to trace the position of each of the data points to the first surface through a plurality of positions, the plurality of positions being on a second surface, The tracing engine determines, for each data point, a set of 4D plenoptic coordinates based on a plurality of traced intersection points on the first surface and the second surface, determines a first set of energy source position values for the first set of 4D plenoptic coordinates having a first convergence point for a first data point, and determines a second set of energy source position values for the second set of 4D plenoptic coordinates having a second convergence point for a second data point different from the first data point, and is further configured such that, the first set and the second set of 4D plenoptic coordinates are determined based on different pluralities of traced intersection points on the first surface and the second surface, the compression engine is configured to receive data from the processing subsystem, compress the data, and store the compressed data in the optional memory or transmit the compressed data to the input / output interface, the optional memory is configured to receive data from the input / output interface, the processing subsystem, and the compression engine, store the data, and transmit the stored data to any of the input / output interface, the processing subsystem, or the compression engine, a system.

Claims

【Claim 1】 A method for determining four-dimensional plenoptic coordinates (4D plenoptic coordinates) for content data, comprising: receiving content data; determining the position of a data point with respect to a first surface for generating a digital three-dimensional representation of the content data, wherein the first surface is a reference surface; tracking the position of each of the data points to the first surface through a plurality of positions, wherein the plurality of positions are on a second surface; determining a set of 4D plenoptic coordinates for each data point based on a plurality of tracked intersection points on the first surface and the second surface; determining a first set of energy source position values for a first set of 4D plenoptic coordinates having a first convergence point for a first data point; determining a second set of energy source position values for a second set of 4D plenoptic coordinates having a second convergence point for a second data point different from the first data point; and wherein the first set and the second set of 4D plenoptic coordinates are determined based on different pluralities of tracked intersection points on the first surface and the second surface.

Images