System and method for rendering data from a 3D environment

Abstract

Systems and methods for rendering data from a 3D environment are disclosed. The methods and systems of the present disclosure capture energy data from a 3D environment in a single rendering pass using inverse ray tracking from a view, thereby enabling more efficient and more accurate collection of data.

Description

[0001] This application is a divisional application of Chinese patent application 201980018331.9 (PCT application number: PCT / US2019 / 013554), whose international application date is January 14, 2019 and whose national phase entry date is September 11, 2020, and whose name is “System and method for rendering data from a 3D environment”. Technical Field

[0002] The present disclosure generally relates to rendering a three-dimensional environment into a data set to instruct an energy field projection device to output a four-dimensional energy field. Background Art

[0003] The dream of an interactive virtual world in a "holodeck" room, popularized by Gene Roddenberry's Star Trek, was originally conceived by writer Alexander Moszkowski in the early 20th century, and has been an inspiration for science fiction and technological innovation for nearly a century. However, no convincing implementation of this experience exists outside of literature, the media, and the collective imagination of children and adults alike. The present application teaches systems and methods for rendering information from a 3D environment into a format that allows a 4D energy field projection system to output a 4D energy field modeled on a scene from a 3D environment. Summary of the invention

[0004] In one embodiment, a method for rendering a three-dimensional (4D) energy field from a three-dimensional (3D) environment includes the steps of providing a scene in the 3D environment, the scene being described by a plurality of energy data points located throughout the scene; and locating a plurality of virtual pixels on a virtual pixel plane in the scene, wherein each virtual pixel has a known unique 4D coordinate, the coordinates comprising a 2D angular coordinate and a 2D spatial coordinate. In this embodiment, the 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and a virtual viewing position of a plurality of virtual viewing positions located on a virtual viewing plane in the scene, and wherein the 2D spatial coordinate of each virtual pixel identifies a position of a virtual aperture of a plurality of virtual apertures located on a virtual display plane in the scene. Next, energy data points from the plurality of energy data points in the scene are sampled along a plurality of rays from the virtual viewing plane, wherein each ray intersects a virtual viewing position and a virtual pixel, the intersection angle is determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray. In this embodiment, the method includes correlating the energy data points sampled along each ray with an energy value of the one virtual pixel intersected by the ray; and rendering the energy value of the one virtual pixel for each ray and the known unique 4D coordinates of the one virtual pixel for each ray as a data set, the format of the data set being usable to instruct an energy device to output a 4D energy field.

[0005] In an embodiment, at least one ray of the plurality of rays intersects each virtual viewing position of the plurality of virtual viewing positions. In an embodiment, one ray of the plurality of rays intersects each virtual pixel of the plurality of virtual pixels. In some embodiments, the 4D energy field comprises a light field, a haptic field, or a tactile field.

[0006] In one embodiment, the energy data points include values ​​describing at least one of: energy frequency, energy intensity, energy transparency, energy refractive index, energy reflectivity. In one embodiment, the 3D environment may be determined by applying a depth map to points in a two-dimensional space. In another embodiment, the virtual display plane corresponds to a waveguide system of an energy guiding device, and energy is operable to be guided through the waveguide system according to the data set to form a detectable 4D energy representation of at least a portion of the scene.

[0007] In an embodiment, the plurality of virtual pixels correspond to a plurality of energy locations on the first side of the waveguide system. In another embodiment, the data set further comprises vectorized material property data. In operation, at least a portion of the method may be performed in real time. In another technique, the method may be performed entirely in real time. In another embodiment, at least two portions of the method are performed in different time periods.

[0008] In some embodiments, the data set describes a signal perceptible by a visual, audio, texture, feel, or smell sensor. In embodiments, the energy data points sampled along each of the plurality of rays are simultaneously associated with energy values. In some cases, the data set is stored in a binary file format.

[0009] In an embodiment, each ray of the plurality of rays extends through the one of the plurality of virtual apertures to and beyond the one of the plurality of virtual pixels, and wherein the energy data points of the plurality of energy data points are sampled from the virtual viewing plane.

[0010] In one embodiment, the above steps may be repeated indefinitely. In another embodiment, the above steps may be repeated indefinitely to render a dynamic 4D energy field from a 3D environment.

[0011] In an embodiment, rendering the energy data further comprises calibrating the energy data for the energy device. In one embodiment, the ray file stores 3D spatial coordinates identifying the position of each virtual viewing position and the 2D angular coordinates of the one virtual pixel intersecting each ray. In another embodiment, the ray file associates the 3D spatial coordinates of each virtual viewing position with the 2D angular coordinates of each virtual pixel intersecting each ray intersecting the virtual viewing position. In yet another embodiment, the ray file provides instructions for each ray for sampling energy data points from the one virtual viewing position intersecting the ray along the ray, wherein the instructions are determined by at least the 3D spatial coordinates of the one virtual viewing position intersecting the ray and the 2D angular coordinates of the one virtual pixel intersecting the ray.

[0012] In one embodiment, a system for rendering a four-dimensional (4D) energy field from a dynamic three-dimensional (3D) environment includes: a processing subsystem having a sensory data engine and a rendering engine, wherein the sensory data engine provides a scene in a 3D environment, the scene being described by a plurality of energy data points located throughout the scene. In this embodiment, the sensory data engine locates a plurality of virtual pixels on a virtual pixel plane in the scene, wherein each virtual pixel has a known unique 4D coordinate, the coordinates comprising a 2D angular coordinate and a 2D spatial coordinate. The 2D angular coordinate of each virtual pixel describes, by the sensory data engine, an angular correlation between the virtual pixel and a virtual viewing position of a plurality of virtual viewing positions located on a virtual viewing plane in the scene, and wherein the 2D spatial coordinate of each virtual pixel identifies, by the sensory data engine, a position of a virtual aperture of a plurality of virtual apertures located on a virtual display plane in the scene.

[0013] In this embodiment, the rendering engine samples energy data points from the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, wherein each ray intersects a virtual viewing position and a virtual pixel, the intersection angle is determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray.

[0014] Still in this embodiment, the rendering engine correlates the energy data points sampled along each ray with the energy value of the one virtual pixel among the multiple virtual pixels, and the rendering engine renders the energy value of the one virtual pixel among the multiple virtual pixels and the known unique 4D coordinates of the one virtual pixel among the multiple virtual pixels as a data set, and the format of the data set can be used to indicate that the energy device outputs a 4D energy field.

[0015] In an embodiment, at least one of the plurality of rays intersects each of the plurality of virtual viewing positions. In another embodiment, one of the plurality of rays intersects each of the plurality of virtual pixels. In some embodiments, the 4D energy field may be a light field, a tactile field, or a haptic field.

[0016] In an embodiment, the energy data point comprises a value describing at least one of: energy frequency, energy intensity, energy transparency, energy refractive index, energy reflectivity. In operation, the 3D environment is determined by applying a depth map to a point in a two-dimensional space.

[0017] In one embodiment, the virtual display plane corresponds to a waveguide system of an energy guiding device, and energy is available to be guided through the waveguide system according to the data set to form a detectable 4D energy representation of at least a portion of the scene. In another embodiment, the plurality of virtual pixels corresponds to a plurality of energy locations on a first side of the waveguide system. In another embodiment, the data set further comprises vectorized material property data.

[0018] In operation, at least a portion of the system is performed in real time. In another operation, the system is performed completely in real time. In another embodiment, at least two portions of the system are performed in different time periods.

[0019] In one embodiment, the data set describes a signal perceivable by a visual, audio, texture, feel, or smell sensor. In another embodiment, the energy data points sampled along each of the plurality of rays are simultaneously associated with energy values. In yet another embodiment, the data set is stored in a binary file format.

[0020] In an embodiment, an energy data point of the plurality of energy data points is sampled by the rendering engine along each of the plurality of rays extending from the virtual viewing plane through the one of the plurality of virtual apertures to and beyond the one of the plurality of virtual pixels. In another embodiment, the system is infinitely operable to render a dynamic 4D energy field from a 3D environment. In yet another embodiment, the system further comprises a memory for storing the data set. In one embodiment, the rendering system calibrates the energy data for the energy device. In another embodiment, the system further comprises a memory for storing the plurality of energy data points.

[0021] In one embodiment, the ray file stores 3D spatial coordinates identifying the position of each virtual viewing position and the 2D angular coordinates of the one virtual pixel intersecting each ray. In another embodiment, the ray file associates the 3D spatial coordinates of each virtual viewing position with the 2D angular coordinates of each virtual pixel intersecting each ray intersecting the virtual viewing position. In yet another embodiment, the ray file provides, for each ray, instructions for sampling energy data points of the plurality of energy data points along the ray from the one virtual viewing position intersecting the ray, wherein the instructions are determined by at least the 3D spatial coordinates of the one virtual viewing position intersecting the ray and the 2D angular coordinates of the one virtual pixel intersecting the ray.

[0022] In one embodiment, a method for rendering energy data from a three-dimensional (3D) environment includes the steps of providing a scene in the 3D environment, the scene being described by a plurality of energy data points located throughout the scene; and locating a plurality of virtual pixels on a virtual pixel plane in the scene, wherein each virtual pixel has a known unique 4D coordinate, the coordinates comprising a 2D angular coordinate and a 2D spatial coordinate. The 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and a virtual viewing position of a plurality of virtual viewing positions located on a virtual viewing plane in the scene, and wherein the 2D spatial coordinate of each virtual pixel identifies a position of a virtual aperture of a plurality of virtual apertures located on a virtual display plane in the scene. The next step of the method includes sampling energy data points from the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, wherein each ray intersects a virtual viewing position and a virtual pixel at an intersection angle determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray. In one method, the next step includes correlating the energy data points sampled along each ray with an energy value of the one virtual pixel intersected by the ray; and rendering the energy value of the one virtual pixel for each ray and the known unique 4D coordinates of the one virtual pixel for each ray as a data set having a format that can be used to instruct an energy device to output energy data.

[0023] In one embodiment, each virtual aperture intersects two rays. In another embodiment, the plurality of virtual viewing positions includes two virtual viewing positions. In some embodiments, the format of the data set can be used to instruct the energy device to output a stereoscopic image, the format of the data set can be used to instruct the energy device to output a virtual reality image, or the format of the data set can be used to instruct the energy device to output an augmented reality image.

[0024] In one embodiment, each virtual aperture intersects a number of rays. In another embodiment, the format of the data set may be used to instruct the energy device to output images from a plurality of views, wherein the plurality of views correspond to the number of rays intersecting each virtual aperture. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram showing design parameters for an energy directing system;

[0026] Figure 2 is a schematic diagram showing an energy system having an active device area with a mechanical housing;

[0027] Figure 3 is a schematic diagram showing an energy relay system;

[0028] Figure 4 is a schematic diagram showing an embodiment of energy repeater elements bonded together and fastened to a base structure;

[0029] Figure 5A is a schematic diagram showing an example of a relay image through a multi-core optical fiber;

[0030] Figure 5B is a schematic diagram showing an example of a relayed image by an energy repeater that exhibits characteristics of the lateral Anderson localization principle;

[0031] Figure 6 is a schematic diagram showing rays propagating from the energy surface to the viewer;

[0032] Fig. 7A A perspective view showing an energy waveguide system having a base structure, four energy devices, and four energy repeater elements to form a seamless energy surface according to one embodiment of the present disclosure;

[0033] Figure 7B An energy relay system according to an embodiment of the present disclosure is shown;

[0034] Figure 7C A top-down perspective view showing an embodiment of an energy waveguide system according to one embodiment of the present disclosure;

[0035] Fig.7D Show Figure 7C A front perspective view of the embodiment shown in FIG.

[0036] Figure 7E-7L Various embodiments of energy suppression elements are shown;

[0037] Figure 8 is a flow chart illustrating an embodiment of a process for processing holographic sensing data;

[0038] Fig. 9 is a schematic diagram of a virtual environment constructed from sensory data;

[0039] Fig.10 is a schematic diagram illustrating an embodiment of energy tracking;

[0040] Fig.11 is a schematic diagram showing an embodiment of energy guiding device 1000 undergoing a tracking process;

[0041] Fig.12 is a schematic diagram of a processing system for processing holographic sensing data;

[0042] Fig.13is a block diagram illustrating an embodiment of a vectorization process;

[0043] Fig.14 Shows some aspects of energy modeling in a 3D environment;

[0044] Fig.15 is a flow chart illustrating an embodiment of a process for rendering a 4D energy field from a 3D environment;

[0045] Fig.16 Describe a visual representation of a 3D environment;

[0046] Fig.17 A perspective view depicting virtual pixels, virtual apertures, and virtual viewing positions from a 3D environment;

[0047] Fig.18 A perspective view depicting virtual pixels, virtual apertures, and virtual viewing positions from a 3D environment;

[0048] Fig.19 Depicting a visual representation of a 3D environment from the perspective of a virtual viewing position;

[0049] Fig. 20 is a flow chart illustrating an embodiment of a process for rendering energy data from a 3D environment;

[0050] Fig.21 is a schematic diagram of back-tracing of respective objects in a digital stereo representation; and

[0051] Fig. 22 is a schematic diagram illustrating an embodiment of a system for rendering a 4D energy field from a 3D environment. DETAILED DESCRIPTION

[0052] Embodiments of the holodeck (collectively, the "holodeck design parameters") provide sufficient to fool human sensory receptors into believing that energy pulses received within a virtual, social, and interactive environment are real energy stimuli, thereby providing: 1) binocular parallax without external accessories, head-mounted glasses, or other peripherals; 2) accurate motion parallax, occlusion, and opacity throughout the viewing volume for any number of viewers simultaneously; 3) visual focus through synchronized convergence, accommodation, and miosis of all perceived rays by the eyes; and 4) convergence of energy wave propagation with sufficient density and resolution to exceed the sensory "resolution" of human vision, hearing, touch, taste, smell, and / or balance.

[0053] Based on conventional technology to date, we are decades, if not centuries, away from being able to realize all receptive fields in a convincing manner as proposed in the design parameters of the holodeck (which encompasses vision, hearing, somatosensory, taste, olfactory, and vestibular systems).

[0054] In this disclosure, the terms light field and holographic are used interchangeably to define energy propagation for stimulating any sensory receptor response. Although the initial disclosure may involve examples of energy and mechanical energy propagation through holographic images and stereoscopic tactile energy surfaces, all forms of sensory receptors are contemplated in this disclosure. In addition, the principles of propagating energy along a propagation path disclosed herein can be applied to energy emission and energy capture.

[0055] Many technologies exist today that are often unfortunately confused with holograms, including lenticular printing, Pepper's Ghost, glasses-free stereoscopic displays, horizontal parallax displays, head-mounted VR and AR displays (HMDs), and other such illusions summarized as "fauxlography." These technologies may exhibit some of the desirable characteristics of true holographic displays, but they fail to stimulate human visual responses in any way sufficient to achieve at least two of the four identified holodeck design parameters.

[0056] Conventional technologies have not successfully implemented these challenges to produce seamless energy surfaces sufficient for holographic energy propagation. There are various approaches to implement stereoscopic and directionally multiplexed light field displays, including parallax barriers, hogels, voxels, diffractive optics, multi-view projection, holographic diffusers, rotating mirrors, multi-layer displays, time-sequential displays, head-mounted displays, etc., but conventional approaches may involve sacrifices in image quality, resolution, angular sampling density, size, cost, security, frame rate, etc., which ultimately make the technology infeasible.

[0057] In order to achieve the holodeck design parameters for the visual, auditory, and somatosensory systems, the human acuity of each of the corresponding systems needs to be studied and understood to propagate energy waves in order to adequately confuse human sensory receptors. The visual system is able to resolve to approximately 1 arc minute, the auditory system can distinguish positional differences as small as three degrees, and the somatosensory system of the hand is able to distinguish points separated by 2-12 mm. Although the ways of measuring these acuities vary and conflict, these values ​​are sufficient to understand the systems and methods of stimulating the perception of energy propagation.

[0058] Of the sensory receptors mentioned, the human visual system is by far the most sensitive, as even a single photon can induce a sensation. For this reason, much of this introduction will focus on visual energy wave propagation, and much lower resolution energy systems coupled within the disclosed energy waveguide surfaces can focus the appropriate signals to induce holographic sensory perception. Unless otherwise noted, all disclosures apply to all energy and sensory domains.

[0059] When calculating the effective design parameters for energy propagation of a visual system given a viewing volume and viewing distance, the desired energy surface can be designed to include billions of pixels with effective energy position density. For wide viewing volume or near-field inspection, the design parameters of the desired energy surface can include hundreds of billions of pixels or more with effective energy position density. In comparison, the desired energy source can be designed to have an energy position density of 1 to 250 effective megapixels for ultrasonic propagation of stereo touch, or an array of 36 to 3,600 effective energy positions for acoustic propagation of holographic sound, depending on the input environmental variables. It is important to note that with the disclosed bidirectional energy surface architecture, all components can be configured to form a structure suitable for any energy domain to achieve holographic propagation.

[0060] However, the main challenges to realizing holodecks currently relate to available visual technology and energy device limitations. Acoustic and ultrasonic devices are less challenging, given the order of magnitude difference in the desired density based on the sensory acuity in the corresponding receptive fields, but the complexity should not be underestimated. Although holographic emulsions with resolution exceeding the desired density exist to encode interference patterns in static images, prior art display devices are limited by resolution, data throughput, and manufacturing feasibility. To date, no display device has been able to meaningfully produce light fields with near-holographic resolution for visual acuity.

[0061] The production of a single silicon-based device capable of meeting the desired resolution for a convincing lightfield display may be impractical and may involve extremely complex manufacturing processes that are beyond current manufacturing capabilities. Limitations on tiling multiple existing display devices together involve gaps and spaces created by the physical dimensions of packaging, electronics, housings, optics, and several other challenges that inevitably render the technology infeasible from an imaging, cost, and / or size perspective.

[0062] Embodiments disclosed herein may provide a realistic path to building a holographic deck.

[0063] Example embodiments will now be described below with reference to the accompanying drawings, which form a part of the present invention and illustrate example embodiments that can be practiced. As used in the present disclosure and the appended claims, the terms "embodiment", "example embodiment" and "exemplary embodiment" do not necessarily refer to a single embodiment but may refer to a single embodiment, and various example embodiments may be easily combined and interchanged without departing from the scope or spirit of the example embodiments. In addition, the terms as used herein are only for the purpose of describing example embodiments and are not intended to be limiting. In this regard, as used herein, the term "in" may include "in" and "on", and the terms "one", "an" and "the" may include singular references and plural references. In addition, as used herein, the term "by" may also mean "according to" depending on the context. In addition, as used herein, the term "if" may also mean "when" or "when" depending on the context. In addition, as used herein, the word "and / or" may refer to and cover any and all possible combinations of one or more of the associated listed items.

[0064] Holographic system considerations:

[0065] Overview of light field energy propagation resolution

[0066] Light fields and holographic displays are the result of multiple projections, where the energy surface locations provide angle, color, and intensity information propagating within the viewing volume. The disclosed energy surfaces provide the opportunity for additional information to coexist and propagate through the same surface to elicit responses from other sensory systems. Unlike stereoscopic displays, the viewing location of the energy propagation path that converges in space does not change as the viewer moves around in the viewing volume, and any number of viewers can see the propagated object in real space at the same time, just as if it were really in real space. In some embodiments, the propagation of energy may be located in the same energy propagation path, but in opposite directions. For example, in some embodiments of the present disclosure, both energy emission and energy capture along the energy propagation path are possible.

[0067] Figure 1is a schematic diagram showing variables associated with the stimulation to which the sensory receptors respond. These variables may include the surface diagonal 101, the surface width 102, the surface height 103, the determined target seating distance 118, the target seating field of view from the center of the display 104, the number of intermediate samples shown herein as samples between eyes 105, the average adult interocular spacing 106, the average resolution of the human eye in arc minutes 107, the horizontal field of view formed between the target viewer position and the surface width 108, the vertical field of view formed between the target viewer position and the surface height 109, the resulting horizontal waveguide element resolution or the total number of elements on the surface 110, the resulting vertical waveguide element resolution or the total number of elements on the surface 111, the sample distance 112 based on the interocular spacing between the eyes and the angular projection between the eyes, the angular sampling 113 which may be based on the sample distance and the target seating distance, the total resolution per waveguide element derived from the desired angular sampling 114. The total resolution per waveguide element Vertical 115 derived from the desired angular sampling, the device Horizontal is a count of the determined number of desired precision energy sources 116, and the device Vertical is a count of the determined number of desired precision energy sources 117.

[0068] One approach to understanding the minimum resolution desired may be to ensure adequate stimulation of visual (or other) sensory receptor responses based on the following criteria: surface size (e.g., 8" diagonal), surface aspect ratio (e.g., 16:9), seating distance (e.g., 128" from the display), seating field of view (e.g., 120 degrees or + / - 60 degrees around the center of the display), desired intermediate samples at certain distances (e.g., one additional propagation path between the eyes), average interocular separation for adults (approximately 65 mm), and average resolution of the human eye (approximately 1 arc minute). These example values ​​should be considered placeholders depending on specific application design parameters.

[0069] Furthermore, each value attributed to a visual sensory receptor may be replaced by other systems to determine the desired propagation path parameters. For other energy propagation embodiments, angular sensitivity of the auditory system as low as three degrees, and spatial resolution of the somatosensory system of the hand as small as 2 to 12 mm may be considered.

[0070] Although the methods of measuring these sensory acuities vary and conflict, these values ​​are sufficient to understand the systems and methods of stimulating the perception of virtual energy propagation. There are many ways to consider design resolution, and the method proposed below combines practical product considerations with the biological resolution limits of sensory systems. Those skilled in the art will appreciate that the following overview is a simplification of any such system design and should be considered for exemplary purposes only.

[0071] With an understanding of the resolution limit of the sensory system, the total energy waveguide element density, such that the receiving sensory system cannot discern a single energy waveguide element from neighboring elements, can be calculated given the following equation:

[0072] ·

[0073] ·

[0074] ·

[0075] ·

[0076] ·

[0077] ·

[0078] ·

[0079] The above calculations yield a field of view of approximately 32×18°, yielding a desired number of approximately 1920×1080 (rounded to the nearest format) energy waveguide elements. The variables may also be constrained so that the field of view is uniform for both (u, v), providing a more regular spatial sampling of energy positions (e.g., pixel aspect ratio). The angular sampling of the system employs additional propagating energy paths between defined target view volume positions and two points at an optimized distance, given by:

[0080] ·

[0081] ·

[0082] In this case, the interocular distance is used to calculate the sample distance, but any metric may be used to take an appropriate number of samples as a given distance. Taking into account the above variables, approximately one ray every 0.57° may be required, and the total system resolution per independent sensory system may be determined given the following equation:

[0083] ·

[0084] Total resolution H = N * horizontal element resolution

[0085] Total resolution V = N * vertical element resolution

[0086] In the above scenario, given the size of the energy surface and the angular resolution addressed for the visual acuity system, the resulting energy surface may desirably contain energy resolution locations of approximately 400k x 225k pixels, or a holographic propagation density of 90 billion pixels. These variables are provided for exemplary purposes only, and optimization of holographic propagation of energy should take into account many other sensory and energy metrology considerations. In another embodiment, based on the input variables, energy resolution locations of 1 gigapixel may be desired. In another embodiment, based on the input variables, energy resolution locations of 1,000 gigapixels may be desired.

[0087] Current technical limitations:

[0088] Active area, device electronics, packaging and mechanical envelope

[0089] Figure 2 A device 200 is shown having an active area 220 with specific mechanical exterior dimensions. The device 200 may include a driver 230 and electronics 240 for powering and interfacing to the active area 220, the active area having dimensions as indicated by the x and y arrows. This device 200 does not take into account the routing and mechanical structures for driving, powering, and cooling components, and the mechanical footprint can be further minimized by introducing wiring into the device 200. The minimum footprint of this device 200 can also be referred to as the mechanical envelope 210, which has dimensions as indicated by the M:x and M:y arrows. This device 200 is for illustrative purposes only, and a self-limited electronics design can further reduce mechanical housing overhead, but may not be the exact dimensions of the active area of ​​the device in almost all cases. In an embodiment, this device 200 shows the dependence on electronics as it relates to an active image area 220 of a micro OLED, DLP chip, or LCD panel, or any other technology for the purpose of image illumination.

[0090] In some embodiments, other projection technologies that aggregate multiple images onto a larger overall display may also be considered. However, these technologies may come at the expense of greater complexity in throw distance, shortest focus, optical quality, uniform field resolution, chromatic aberration, thermal properties, calibration, alignment, additional size, or apparent size. For most practical applications, accommodating dozens or hundreds of these projection sources 200 may result in a much larger and less reliable design.

[0091] For exemplary purposes only, assuming an energy device with an energy position density of 3840×2160 sites, the number of individual energy devices (eg, device 100) desired for the energy surface may be determined given the following equation:

[0092] ·

[0093] ·

[0094] Given the above resolution considerations, one might want something like Figure 2 5 devices of the device shown in . It should be noted that many devices are composed of various pixel structures that may or may not be mapped to a regular grid. Where there are additional sub-pixels or positions within each full pixel, these can be exploited to produce additional resolution or angular density. Additional signal processing may be used to determine the manner in which the light field is converted to corrected (u, v) coordinates depending on the specified position of the pixel structure, and may be an obvious characteristic of each device that is known and calibrated. Furthermore, other energy domains may involve different processing of these ratios and device structures, and those skilled in the art will understand the direct intrinsic relationship between each desired frequency domain. This will be shown and discussed in more detail in the subsequent disclosure.

[0095] The resulting calculations can be used to understand how many of these individual devices might be needed to produce a full resolution energy surface. In this case, approximately 105×105 or approximately 11,080 devices might be needed to achieve the visual acuity threshold. There are challenges and novelties in making a seamless energy surface from these available energy positions for adequate sensory holographic propagation.

[0096] Seamless Energy Surface Overview:

[0097] Configuration and design of energy repeater arrays

[0098] In some embodiments, methods are disclosed that address the challenge of generating high energy position density from an array of individual devices without seams due to limitations of the mechanical structure of the devices. In embodiments, an energy propagation relay system can allow the effective size of the active device area to be increased to meet or exceed the mechanical dimensions used to configure the repeater array and form a single seamless energy surface.

[0099] Figure 3 An embodiment of such an energy relay system 300 is shown. As shown, the relay system 300 may include a device 310 mounted to a mechanical housing 320, wherein an energy relay element 330 propagates energy from the device 310. The relay element 330 may be configured to provide the ability to reduce any gaps 340 that may be created when multiple mechanical housings 320 of the device are placed into an array of multiple devices 310.

[0100] For example, if the active area 310 of the device is 20 mm x 10 mm, and the mechanical housing 320 is 40 mm x 20 mm, then the energy repeater element 330 may be designed with a 2:1 magnification ratio to produce a tapered form that is 20 mm x 10 mm on the reduced end (arrow A) and 40 mm x 20 mm on the magnified end (arrow B), thereby providing the ability to seamlessly align arrays of these elements 330 together without altering or interfering with the mechanical housing 320 of each device 310. The repeater elements 330 may be mechanically joined or fused together for alignment and polishing to ensure that the seam gap 340 between the devices 310 is minimized. In one such embodiment, it is possible to achieve a seam gap 340 that is less than the visual acuity limit of the eye.

[0101] Figure 4 An example of a base structure 400 is shown with energy repeater elements 410 formed together and securely fastened to an additional mechanical structure 430. The mechanical structure of the seamless energy surface 420 provides the ability to couple multiple energy repeater elements 410, 450 in series to the same base structure through a joining or other mechanical process of mounting the repeater elements 410, 450. In some embodiments, each repeater element 410 may be fused, joined, bonded, press-fitted, aligned, or otherwise attached together to form the resulting seamless energy surface 420. In some embodiments, a device 480 may be mounted onto the rear of the repeater element 410 and passively or actively aligned to ensure proper energy position alignment within determined tolerances is maintained.

[0102] In an embodiment, the seamless energy surface comprises one or more energy locations, and one or more energy repeater element stacks comprise a first and a second side, and each energy repeater element stack is arranged to form a single seamless display surface, thereby guiding energy along a propagation path extending between the one or more energy locations and the seamless energy surface, and wherein a spacing between edges of any two adjacent second sides of a terminal energy repeater element is less than a minimum perceptible outline, the minimum perceptible outline being defined by a human visual acuity having better than 20 / 40 vision at a distance greater than the width of the single seamless energy surface.

[0103] In an embodiment, each seamless energy surface comprises one or more energy repeater elements, each energy repeater element having one or more structures forming first and second surfaces with transverse and longitudinal orientations. The first repeater surface has an area different from the second repeater surface, thereby forming positive or negative magnification, and is configured with a well-defined surface profile for the first and second surfaces to transmit energy through the second repeater surface, thereby substantially filling an angle of + / - 10 degrees relative to a normal to the surface profile across the second repeater surface.

[0104] In an embodiment, multiple energy domains may be configured within a single energy repeater or between multiple energy repeaters to direct one or more sensory holographic energy propagation paths including visual, acoustic, tactile, or other energy domains.

[0105] In an embodiment, a seamless energy surface is configured with an energy repeater comprising two or more first sides for each second side to simultaneously receive and transmit one or more energy domains, thereby providing bidirectional energy propagation throughout the system.

[0106] In an embodiment, the energy repeater is provided as a loosely coherent element.

[0107] Brief introduction of component engineering structure:

[0108] Published progress on lateral Anderson localized energy repeaters

[0109] According to the principles disclosed herein for energy repeater elements that induce lateral Anderson localization, the properties of the energy repeater can be significantly optimized. Lateral Anderson localization is the propagation of radiation through a laterally disordered but longitudinally consistent material.

[0110] This means that the material effects that produce the Anderson localization phenomenon may be less affected by total internal reflection than by randomization among multiple scattering paths where wave interference may completely limit propagation in the transverse orientation while continuing in the longitudinal orientation.

[0111] The most significant additional benefit is the elimination of the cladding of conventional multi-core fiber materials. The cladding is intended to functionally eliminate energy scattering between fibers while simultaneously acting as a barrier to energy rays, thereby reducing transmission by at least the core to clad ratio (e.g., a 70:30 core to clad ratio will transmit at most 70% of the received energy transmission), and additionally forming a strong pixelated pattern in the propagated energy.

[0112] Figure 5A An end view of an example of such a non-Anderson localized energy repeater 500 is shown, where the image is relayed through a multi-core optical fiber, where pixelation and fiber noise may be present due to the intrinsic properties of the optical fiber. For conventional multimode and multi-core optical fibers, the relayed image may be pixelated in nature due to the properties of total internal reflection of the discrete array of cores, where any crosstalk between the cores will degrade the modulation transfer function and increase blur. The resulting image produced through conventional multi-core optical fibers will tend to have a similar Figure 3 The residuals of those shown are fixed noise fiber patterns.

[0113] Figure 5B An example of the same relay image 550 is shown through an energy repeater comprising a material exhibiting lateral Anderson localization properties, wherein compared to Figure 5A The fixed fiber pattern has a greater density of particle structures. In an embodiment, the repeater including the randomized micro-component engineered structure induces lateral Anderson localization and transmits light more efficiently with a higher resolvable resolution of propagation compared to commercially available multimode glass optical fibers.

[0114] The lateral Anderson localized material properties offer significant advantages in both cost and weight, where the cost and weight of similar optical grade glass materials may be 10 to 100 times higher than the cost of the same material produced within the embodiments, wherein the disclosed systems and methods include randomized microcomponent engineered structures that offer significant opportunities to improve cost and quality over other techniques known in the art.

[0115] In an embodiment, a repeater element exhibiting lateral Anderson localization may include a plurality of at least two different component engineered structures in each of three orthogonal planes arranged into a three-dimensional grating, and the plurality of structures form a randomized distribution of material wave propagation properties in the lateral planes within the three-dimensional grating and form channels with similar values ​​of material wave propagation properties in the longitudinal planes within the three-dimensional grating, wherein the localized energy waves propagating through the energy repeater have a higher transmission efficiency in the longitudinal orientation as compared to the lateral orientation.

[0116] In an embodiment, multiple energy domains may be configured within a single lateral Anderson localized energy repeater or between multiple lateral Anderson localized energy repeaters to direct one or more sensory holographic energy propagation paths including visual, acoustic, tactile, or other energy domains.

[0117] In an embodiment, the seamless energy surface is configured with a lateral Anderson localized energy repeater, which includes two or more first sides for each second side to simultaneously receive and transmit one or more energy domains, thereby providing bidirectional energy propagation throughout the system.

[0118] In an embodiment, the lateral Anderson localized energy repeater is configured as a loosely coherent or flexible energy repeater element.

[0119] Considerations for 4D plenoptic functions:

[0120] Selective propagation of energy through holographic waveguide arrays

[0121] As discussed above and herein, a light field display system generally includes an energy source (e.g., an illumination source) and a seamless energy surface configured with a sufficient energy location density, as described in the above discussion. A plurality of repeater elements may be used to relay energy from the energy device to the seamless energy surface. Once the energy is delivered to the seamless energy surface with the necessary energy location density, the energy may propagate through the disclosed energy waveguide system according to a 4D plenoptic function. As will be appreciated by one of ordinary skill in the art, 4D plenoptic functions are well known in the art and will not be further detailed herein.

[0122] The energy waveguide system selectively propagates energy through multiple energy positions along a seamless energy surface, wherein the energy positions represent spatial coordinates of a 4D plenoptic function, wherein the structure is configured to change the angular direction of the passing energy waves, wherein the angular direction represents the angular component of the 4D plenoptic function, wherein the propagated energy waves can converge in space according to multiple propagation paths guided by the 4D plenoptic function.

[0123] Reference Figure 6 , which shows an example of a light field energy surface in a 4D image space according to a 4D plenoptic function. This figure shows ray trajectories from the energy surface 600 to the viewer 620, which describes how energy rays converge in space 630 from various locations within the view volume. As shown, each waveguide element 610 defines four information dimensions that describe the propagation 640 of energy through the energy surface 600. Two of the spatial dimensions (referred to herein as x and y) are the physical multiple energy locations that can be viewed in image space, and the angular components θ and φ (referred to herein as u and v) observed in virtual space when projected through the energy waveguide array. In general, according to the 4D plenoptic function, when forming a holographic or light field system described herein, multiple waveguides (e.g., microlenses) are capable of guiding energy locations from the x, y dimensions to unique locations in virtual space along directions defined by the u, v angular components.

[0124] However, those skilled in the art will appreciate that significant challenges with light field and holographic display technologies arise from uncontrolled energy propagation because the design does not precisely account for any of the following: diffraction, scattering, diffusion, angular orientation, calibration, focus, collimation, curvature, uniformity, element crosstalk, and a host of other parameters that result in reduced effective resolution and the inability to accurately focus energy with sufficient fidelity.

[0125] In an embodiment, a method for selective energy propagation that addresses challenges associated with holographic displays may include an energy suppression element and substantially filling a waveguide aperture with nearly collimated energy entering an environment defined by a 4D plenoptic function.

[0126] In an embodiment, the energy waveguide array may define multiple energy propagation paths for each waveguide element, wherein the energy propagation paths are configured to extend through and substantially fill the effective aperture of the waveguide element in a unique direction defined by a prescribed 4D function to multiple energy locations along the seamless energy surface, wherein the energy locations are suppressed by one or more elements, and the one or more elements are positioned to limit the propagation of each energy location to only through a single waveguide element.

[0127] In embodiments, multiple energy domains may be configured within a single energy waveguide or between multiple energy waveguides to direct one or more sensory holographic energy propagations including visual, acoustic, tactile, or other energy domains.

[0128] In embodiments, the energy waveguide and seamless energy surface are configured to receive and transmit one or more energy domains to provide bi-directional energy propagation throughout the system.

[0129] In an embodiment, the energy waveguide is configured to be oriented to any seamless energy surface including a wall, table, floor, ceiling, room or other geometrically arranged environment, utilizing digitally encoded, diffractive, refractive, reflective, grin, holographic, Fresnel, etc. waveguide configurations to propagate non-linear or irregular distributions of energy, including non-transmitting void areas. In another embodiment, the energy waveguide elements may be configured to produce a variety of geometric arrangements that provide any surface contour and / or tabletop viewing, allowing the user to view holographic images from various locations of the energy surface in a 360 degree configuration.

[0130] In an embodiment, the energy waveguide array elements may be reflective surfaces, and the arrangement of the elements may be hexagonal, square, irregular, semi-regular, curved, non-planar, spherical, cylindrical, tilted regular, tilted irregular, spatially varying and / or multi-layered.

[0131] For any components within the seamless energy surface, the waveguide or repeater components may include, but are not limited to, optical fibers, silicon, glass, polymers, optical repeaters, diffractive, holographic, refractive or reflective elements, optical panels, energy combiners, beam splitters, prisms, polarizing elements, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar material exhibiting Anderson localization or total internal reflection.

[0132] Implementing the Holodeck:

[0133] Convergence of bidirectional seamless energy surface systems for stimulating human sensory receptors within holographic environments

[0134] It is possible to construct a large environment of seamless energy surface systems by tiling, fusing, joining, attaching and / or stitching together multiple seamless energy surfaces to form any size, shape, contour or apparent dimensions encompassing an entire room. Each energy surface system may include an assembly having a base structure, energy surface, repeaters, waveguides, devices and electronics that are collectively configured for bidirectional holographic energy propagation, emission, reflection or sensing.

[0135] In an embodiment, the environment of a tiled seamless energy system aggregates to form a large seamless planar or curved wall that contains facilities including up to all surfaces in a given environment and is configured as any combination of seamless, discontinuous planar, faceted, curved, cylindrical, spherical, geometric, or irregular geometric arrangements.

[0136] In an embodiment, aggregated tiles of planar surfaces form a wall-sized system for theatrical or location-based holographic entertainment. In an embodiment, for a cave-based holographic installation, aggregated tiles of planar surfaces cover a space with four to six walls, including a top and floor. In an embodiment, aggregated tiles of curved surfaces create a cylindrical seamless environment for an immersive holographic installation. In an embodiment, aggregated tiles of seamless spherical surfaces form a holographic dome for an immersive experience based on a holodeck.

[0137] In an embodiment, polymeric tiles of seamless curved energy waveguides provide mechanical edges that follow a precise pattern along the boundaries of energy suppression elements within the energy waveguide structure to join, align or fuse adjacent tiled mechanical edges of adjacent waveguide surfaces, thereby creating a modular seamless energy waveguide system.

[0138] In another embodiment of a polymerized tiling environment, energy is propagated bidirectionally for multiple simultaneous energy domains. In another embodiment, the energy surface provides the ability to display and capture simultaneously from the same energy surface, where the waveguide is designed so that light field data can be projected through the waveguide by an illumination source and simultaneously received by the same energy surface. In another embodiment, additional depth sensing and active scanning techniques can be utilized to enable interaction between energy propagation and the viewer in accurate world coordinates. In another embodiment, the energy surface and waveguide can be used to emit, reflect or converge frequencies to induce tactile or stereo tactile feedback. In some embodiments, any combination of bidirectional energy propagation and polymerized surfaces is possible.

[0139] In an embodiment, a system includes an energy waveguide capable of bidirectionally emitting and sensing energy through an energy surface, wherein one or more energy devices are independently paired with two or more path energy combiners to pair at least two energy devices to the same portion of a seamless energy surface, or one or more energy devices are fixed behind the energy surface and close to additional components fixed to a base structure, or close to a location in front of and outside the FOV of the waveguide for off-axis direct or reflected projection or sensing, and the resulting energy surface enables bidirectional transmission of energy, thereby allowing the waveguide to focus energy, allowing a first device to emit energy and allowing a second device to sense energy, and wherein information is processed to perform computer vision related tasks, including but not limited to 4D plenoptic eye movement and retinal tracking or sensing of interference within the propagated energy pattern, depth estimation, proximity, motion tracking, image, color or sound formation, or other energy frequency analysis. In another embodiment, the tracked position is actively calculated and the energy position is modified based on the interference between the bidirectionally captured data and the projected information.

[0140] In some embodiments, for each of the three first repeater surfaces that transmit energy combined into a single second energy repeater surface, multiple combinations of three energy devices including an ultrasonic sensor, a visible energy display, and an ultrasonic transmitting device are configured together, wherein each of the three first surfaces includes engineered properties specific to the energy domain of each device, and two engineered waveguide elements are configured for ultrasound and energy, respectively, to be able to independently guide and focus the energy of each device, and are substantially unaffected by other waveguide elements configured for separate energy domains.

[0141] In some embodiments, a calibration procedure is disclosed that enables efficient manufacturing to eliminate system artifacts and produce a geometric map of the resulting energy surface for use with encoding / decoding techniques, as well as a dedicated integrated system for converting the data into calibration information suitable for energy propagation based on a calibration profile.

[0142] In some embodiments, additional energy waveguides and one or more energy devices in series may be integrated into the system to produce opaque holographic pixels.

[0143] In some embodiments, additional waveguide elements including energy suppression elements, beam splitters, prisms, active parallax barriers, or polarization techniques may be integrated to provide spatial and / or angular resolution greater than the waveguide diameter or for other super-resolution purposes.

[0144] In some embodiments, the disclosed energy system may also be configured as a wearable bidirectional device, such as virtual reality (VR) or augmented reality (AR). In other embodiments, the energy system may include an adjustment optical element that focuses the displayed or received energy close to a plane determined in space for the viewer. In some embodiments, the waveguide array may be incorporated into a holographic head-mounted display. In other embodiments, the system may include multiple optical paths to allow the viewer to see the energy system and the real world environment (e.g., a transparent holographic display). In these cases, the system may appear as a near field, among other methods.

[0145] In some embodiments, the transmission of data includes an encoding process with a selectable or variable compression ratio, the encoding process receives an arbitrary data set of information and metadata; analyzes the data set and receives or assigns material properties, vectors, surface IDs, new pixel data, thereby forming a sparser data set, and wherein the received data may include: 2D, stereo, multi-view, metadata, light field, holographic, geometric arrangement, vector or vectorized metadata, and the encoder / decoder may provide the ability to convert data in real time or offline including image processing for: 2D; 2D + depth, metadata or other vectorized information; stereo, stereo + depth, metadata or other vectorized information; multi-view; multi-view + depth, metadata or other vectorized information; holographic; or light field content; with or without depth metadata through a depth estimation algorithm; and an inverse ray tracing method maps the resulting converted data generated by inverse ray tracing of various 2D, stereo, multi-view, stereo, light field or holographic data into real world coordinates in an appropriate manner by characterizing a 4D plenoptic function. In these embodiments, the total data transmission required may be several orders of magnitude less than the transmission information of the original light field data set.

[0146] Energy channeling device suitable for presenting holographic sensory data

[0147] In embodiments, the optomechanical display device is capable of emitting and directing light to form 2D, stereoscopic, multi-view, plenoptic, 4D, stereoscopic, light field, holographic, or any other visual representation of light.

[0148] Fig. 7A An example of a light field optomechanical system in which a emitting display device, an optical repeater, and a waveguide implemented as an array of refractive elements such as a microlens array are configured, wherein a visible image from one or more displays can be optically relayed and then transmitted to an energy surface, wherein the array of refractive elements provides a mapping between each position on the energy surface and the projection direction of light at said position, so that a 4D stereoscopic light field image can be projected.

[0149] In embodiments, waveguides may be used to converge rays to induce vergence and accommodation of a viewer's viewing angle.

[0150] In embodiments, the waveguides and energy repeaters may be formed or polished with various surface geometries. In embodiments, the energy repeaters include elements that induce lateral Anderson localization. In embodiments, the energy repeaters are bidirectional and may transmit and / or project energy.

[0151] In one embodiment, an energy system configured to direct energy according to a four-dimensional (4D) plenoptic function comprises a plurality of energy devices. In some embodiments, the plurality of energy devices comprises an illumination source that emits image information, wherein the image information comprises an emission, projection, or reflection display technique utilizing visible light, IR, UV, coherent, laser, infrared, polarized, or any other electromagnetic illumination source. In other embodiments, the plurality of energy devices comprises a mechanical energy emission device configured to provide immersive audio or stereo tactile sensations from a sound field.

[0152] In some embodiments, the energy system configured as above may further include a base structure (e.g., 72) such that the plurality of energy devices, the energy relay system, and the energy waveguide system may be coupled to the base structure. In other embodiments, the plurality of energy devices, the energy relay system, and the energy waveguide system may be coupled to the base structure via one or more mounting brackets.

[0153] In some embodiments, the plurality of energy devices include energy devices for capturing or sensing energy, the energy including mechanical energy, chemical energy, transfer energy, thermal energy, electrical energy, potential energy, kinetic energy, magnetic energy, gravitational energy, radiant energy, structured, unstructured or other forms of energy. In other embodiments, the plurality of energy devices include energy devices for propagating or emitting energy, the energy including mechanical energy, chemical energy, transfer energy, thermal energy, electrical energy, potential energy, kinetic energy, magnetic energy, gravitational energy, radiant energy, structured, unstructured or other forms of energy. In other embodiments, the plurality of energy devices include acoustic receiving devices configured to provide sensory feedback or audible control.

[0154] In one embodiment, the energy system further comprises an energy relay system (e.g., as best described in Figure 7B 6110 shown in ), the energy relay system has one or more energy relay elements, wherein each of the one or more energy relay elements includes a first surface and a second surface, the second surface of the one or more energy relay elements is arranged to form a single seamless energy surface of the energy relay system, and wherein a first plurality of energy propagation paths extend from energy locations in the plurality of energy devices through the single seamless energy surface of the energy relay system. This will be discussed in more detail below.

[0155] Reference now Figure 7B, which shows an energy relay system 6110 of an orthogonal view according to one embodiment of the present disclosure. In one embodiment, the energy relay system 6110 may include two or more repeater elements 6112, each repeater element 6112 formed by one or more structures, each repeater element 6112 having a first surface 6114, a second surface 6116, a transverse orientation (substantially parallel to the surfaces 6114, 6116), and a longitudinal orientation (substantially perpendicular to the surfaces 6114, 6116). In one embodiment, the surface area of ​​the first surface 6114 may be different from the surface area of ​​the second surface 6116. For example, the surface area of ​​the first surface 6114 may be greater than or less than the surface area of ​​the second surface 6116. In another embodiment, the surface area of ​​the first surface 114 may be the same as the surface area of ​​the second surface 6116. Energy waves may be transmitted from the first surface 6114 to the second surface 6116, or vice versa.

[0156] In one embodiment, the repeater element 6112 of the energy relay system 6110 includes an inclined contour portion 6118 between the first surface 6114 and the second surface 6116. In operation, the energy waves propagating between the first surface 6114 and the second surface 6116 may be transmitted more efficiently in the longitudinal orientation than in the transverse orientation, and the energy waves passing through the repeater element 6112 may cause spatial amplification or spatial reduction. In other words, the energy waves passing through the repeater element 6112 of the repeater element device 6110 may experience increased amplification or decreased amplification. In some embodiments, one or more structures used to form the energy repeater element 6110 may include glass, carbon, optical fiber, optical film, plastic, polymer, or a mixture thereof.

[0157] In one embodiment, the energy waves passing through the first surface 6114 have a first resolution, while the energy waves passing through the second surface 6116 have a second resolution, and the second resolution is not less than about 50% of the first resolution. In another embodiment, despite having a uniform characteristic curve when presented to the first surface, the energy waves can pass through the second surface, radiate in each direction, and have a certain energy density in the forward direction that substantially fills a cone with a + / - 10 degree opening angle relative to the normal of the second surface, regardless of the position on the second repeater surface.

[0158] In some embodiments, the first surface 6114 can be configured to receive energy from an energy wave source, wherein the energy wave source comprises a mechanical housing having a width that is different from a width of at least one of the first surface 6114 and the second surface 6116.

[0159] In each repeater 6112, energy is transmitted between a first surface and a second surface defining a longitudinal orientation, and the first and second surfaces of each repeater extend generally along a transverse orientation defined by first and second directions, wherein the longitudinal orientation is generally orthogonal to the transverse orientation. In one embodiment, energy waves propagating through the plurality of repeaters have a higher transmission efficiency in the longitudinal orientation than in the transverse orientation due to the random refractive index variability in the transverse orientation associated with the minimum refractive index variation in the longitudinal orientation. In some embodiments where each repeater is constructed of multi-core optical fibers, energy waves propagating within each repeater element may travel in a longitudinal orientation determined by aligning the optical fibers in this orientation.

[0160] In an embodiment, the spacing between any two adjacent second side edges of the terminal energy repeater element may be less than a minimum perceptible outline defined by a human visual acuity of better than 20 / 40 vision at a distance from the single seamless energy surface that is greater than the smaller of the height of the single seamless energy surface or the width of the single seamless energy surface.

[0161] In one embodiment, the plurality of energy repeater elements in a stacked configuration may include a plurality of panels. In some embodiments, the plurality of panels may have different lengths or may be loosely coherent optical repeaters. In other embodiments, the plurality of elements may have similar Figure 7B The invention relates to an energy repeater element having an inclined profile portion, wherein the inclined profile portion may be angled, linear, curved, tapered, faceted, or aligned at a non-perpendicular angle relative to the normal axis of the repeater element. In yet another embodiment, energy waves propagating through the plurality of repeaters have a higher transmission efficiency in the longitudinal orientation than in the transverse orientation due to the random refractive index variability in the transverse orientation associated with the minimum refractive index variation in the longitudinal orientation. In embodiments where each energy repeater is constructed from a multi-core optical fiber, energy waves propagating within each repeater element may travel in a longitudinal orientation determined by aligning the optical fiber in this orientation.

[0162] In some embodiments, the one or more repeater elements (e.g., 6112) comprise fused or tiled inlays wherein any seams between adjacent fused or tiled inlays are separated by or smaller than a minimum perceptible outline defined by human visual acuity of better than 20 / 40 vision at a distance at or greater than the width or height of the single seamless energy surface.

[0163] In other embodiments, the one or more repeater elements (e.g., 6112) include: optical fiber, silicon, glass, polymer, optical repeater, diffraction element, holographic optical element, refractive element, reflective element, optical panel, photosynthetic combiner, beam splitter, prism, polarization component, spatial light modulator, active pixel, liquid crystal cell, transparent display or any similar material with Anderson localization or total internal reflection properties used to form the single seamless energy surface.

[0164] In other embodiments, the one or more repeater elements (e.g., 6112) are configured to adapt to the shape of the single seamless energy surface, including planar, spherical, cylindrical, conical, faceted, tiled, regular, irregular, or any other geometric arrangement for a given application.

[0165] In another embodiment, the system further comprises an energy waveguide system having an energy waveguide array (e.g., Figures 7C to 7L 7100), best shown in FIG. 7, wherein a second plurality of energy propagation paths extend from said single seamless energy surface through said energy waveguide array in directions determined by a 4D plenoptic function.

[0166] Figure 7C A top-down perspective view of an embodiment of an energy waveguide system 7100 that can be used to define a plurality of energy propagation paths 7108 is shown. The energy waveguide system 7100 includes an array of energy waveguides 7112 configured to guide energy therethrough along the plurality of energy propagation paths 7108. In an embodiment, the plurality of energy propagation paths 7108 extend through a plurality of energy locations 7118 on a first side 7116 of the array to a second side 7114 of the array.

[0167] refer to Figure 7C and 7L, in an embodiment, a first subset 7290 of the plurality of energy propagation paths 7108 extends through a first energy location 7122. The first energy waveguide 7104 is configured to guide energy along a first energy propagation path 7120 of the first subset 7290 of the plurality of energy propagation paths 7108. The first energy propagation path 7120 may be defined by a first primary ray 7138 formed between the first energy location 7122 and the first energy waveguide 7104. The first energy propagation path 7120 may include rays 7138A and 7138B formed between the first energy location 7122 and the first energy waveguide 7104, which are guided by the first energy waveguide 7104 along energy propagation paths 7120A and 7120B, respectively. The first energy propagation path 7120 may extend from the first energy waveguide 7104 toward the array second side 7114. In an embodiment, the energy guided along the first energy propagation path 7120 includes one or more energy propagation paths between or including energy propagation paths 7120A and 7120B, which are guided through the first energy waveguide 7104 in a direction substantially parallel to the first principal ray 7138 propagating through the second side 7114.

[0168] Embodiments may be configured such that energy guided along the first energy propagation path 7120 may exit the first energy waveguide 7104 in a direction substantially parallel to the energy propagation paths 7120A and 7120B and parallel to the first chief ray 7138. It may be assumed that the energy propagation paths extending through the energy waveguide element 7112 on the second side 7114 include a plurality of energy propagation paths having substantially similar propagation directions.

[0169] Fig.7D 7100. The first energy propagation path 7120 may extend in a unique direction 7208 extending from the first energy waveguide 7104 toward the array second side 7114, the unique direction being determined by at least the first energy position 7122. The first energy waveguide 7104 may be defined by a spatial coordinate 7204, and the unique direction 7208 determined by at least the first energy position 7122 may be defined by an angular coordinate 7206, the angular coordinate defining the direction of the first energy propagation path 7120. The spatial coordinate 7204 and the angular coordinate 7206 may form a four-dimensional plenoptic coordinate set 7210 that defines the unique direction 7208 of the first energy propagation path 7120.

[0170] In an embodiment, energy guided through the first energy waveguide 7104 along the first energy propagation path 7120 substantially fills the first aperture 7134 of the first energy waveguide 7104 and propagates along one or more energy propagation paths that are between the energy propagation paths 7120A and 7120B and parallel to the direction of the first energy propagation path 7120. In an embodiment, the one or more energy propagation paths that substantially fill the first aperture 7134 may comprise greater than 50% of the first aperture 7134 diameter.

[0171] In a preferred embodiment, the energy directed through the first energy waveguide 7104 along the first energy propagation path 7120 to substantially fill the first aperture 7134 may include between 50% and 80% of the first aperture 7134 diameter.

[0172] Return to Figure 7C and 7E 7L, in an embodiment, the energy waveguide system 7100 may further include an energy suppression element 7124 positioned to limit energy propagation between the first side 7116 and the second side 7114 and to suppress energy propagation between adjacent waveguides 7112. In an embodiment, the energy suppression element is configured to suppress energy propagation along a portion of the first subset 7290 of the plurality of energy propagation paths 7108 that does not extend through the first aperture 7134. In an embodiment, the energy suppression element 7124 may be located on the first side 7116 between the array of energy waveguides 7112 and the plurality of energy locations 7118. In an embodiment, the energy suppression element 7124 may be located on the second side 7114 between the plurality of energy locations 7118 and the energy propagation paths 7108. In an embodiment, the energy suppression element 7124 may be located on the first side 7116 or the second side 7114 orthogonal to the array of energy waveguides 7112 or the plurality of energy locations 7118.

[0173] In an embodiment, energy directed along the first energy propagation path 7120 may converge with energy directed along the second energy propagation path 7126 through the second energy waveguide 7128. The first and second energy propagation paths may converge at a location 7130 on the second side 7114 of the array 7112. In an embodiment, the third energy propagation path 7140 and the fourth energy propagation path 7141 may also converge at a location 7132 on the first side 7116 of the array 7112. In an embodiment, the fifth energy propagation path 7142 and the sixth energy propagation path 7143 may also converge at a location 7136 between the first side 7116 and the second side 7114 of the array 7112.

[0174] Figures 7E to 7L7124 are illustrations of various embodiments of energy suppression elements 7124. For the avoidance of doubt, these embodiments are provided for illustrative purposes and in no way limit the scope of the combinations or embodiments provided within the scope of the present disclosure.

[0175] Fig. 7E An embodiment of multiple energy locations 7118 is shown, wherein an energy suppression element 7251 is placed adjacent to a surface of the energy location 7118 and includes a specified refractive, diffractive, reflective or other energy-altering property. The energy suppression element 7251 can be configured to restrict a first subset of energy propagation paths 7290 to a smaller range of propagation paths 7253 by suppressing energy propagation along energy propagation paths 7252. In an embodiment, the energy suppression element is an energy repeater having a numerical aperture of less than 1.

[0176] Figure 7F An embodiment of multiple energy locations 7118 is shown, wherein energy suppression structures 7254 are placed orthogonally between regions of energy locations 7118, and wherein energy suppression structures 7254 exhibit absorptive properties, and wherein energy suppression structures 7254 have defined heights along energy propagation paths 7256 such that certain energy propagation paths 7255 are suppressed. In an embodiment, energy suppression structures 7254 are hexagonal in shape. In an embodiment, energy suppression structures 7254 are circular in shape. In an embodiment, the shape or size of energy suppression structures 7254 is non-uniform along any orientation of a propagation path. In an embodiment, energy suppression structures 7254 are embedded within another structure having additional properties.

[0177] Figure 7G A plurality of energy locations 7118 are shown, wherein a first energy suppression structure 7257 is configured to direct energy 7259 propagating therethrough substantially into a first state. A second energy suppression structure 7258 is configured to allow energy 7259 oriented substantially into the first state to propagate therethrough, and to restrict propagation of energy 7260 oriented substantially in a manner not similar to the first state. In an embodiment, the energy suppression elements 7257, 7258 are a pair of energy polarizing elements. In an embodiment, the energy suppression elements 7257, 7258 are a pair of energy wavebandpass elements. In an embodiment, the energy suppression elements 7257, 7258 are a pair of diffractive waveguides.

[0178] Figure 7HAn embodiment of a plurality of energy locations 7118 is shown wherein an energy suppression element 7261 is configured to alter an energy propagation path 7263 to a certain range depending on which of the plurality of energy locations 7118 the energy propagation path 7263 extends through. The energy suppression element 7261 may alter the energy propagation path 7263 in a uniform or non-uniform manner along the energy propagation path 7263 such that certain energy propagation paths 7262 are suppressed. An energy suppression structure 7254 is placed orthogonally between regions of the energy locations 7118, and wherein the energy suppression structure 7254 exhibits absorptive properties, and wherein the energy suppression structure 7254 has a defined height along the energy propagation path 7263 such that certain energy propagation paths 7262 are suppressed. In an embodiment, the suppression element 7261 is a field lens. In an embodiment, the suppression element 7261 is a diffractive waveguide. In an embodiment, the suppression element 7261 is a curved waveguide surface.

[0179] Fig.7I An embodiment of multiple energy locations 7118 is shown where energy suppression elements 7264 provide absorptive properties to limit the propagation of energy 7266 while allowing other propagation paths 7267 to be transmitted.

[0180] Figure 7J An embodiment of multiple energy locations 7118 and multiple energy waveguides 7112 is shown, wherein a first energy suppression structure 7268 is configured to substantially direct energy 7270 propagating therethrough into a first state. A second energy suppression structure 7271 is configured to allow energy 7270 substantially directed into the first state to propagate therethrough, and to restrict propagation of energy 7269 that is substantially directed in a manner not similar to the first state. To further control energy propagation through the system, such as stray energy propagation 7272, the energy suppression structures 7268, 7271 may require combined energy suppression elements to ensure that energy propagation maintains an accurate propagation path.

[0181] Figure 7K An embodiment of multiple energy locations 7118 is shown, and wherein the energy suppression element 7276 provides absorptive properties to limit energy propagation along energy propagation path 7278, while allowing other energy along energy propagation path 7277 to pass through a pair of energy waveguides 7112 for an effective aperture 7284 within the waveguide array 7112. In an embodiment, the energy suppression element 7276 comprises black chrome. In an embodiment, the energy suppression element 7276 comprises an absorptive material. In an embodiment, the energy suppression element 7276 comprises a transparent pixel array. In an embodiment, the energy suppression element 7276 comprises an anodized material.

[0182] Figure 7LAn embodiment including a plurality of energy locations 7118 and a plurality of energy waveguides 7112 is shown, wherein a first energy suppression structure 7251 is placed adjacent to a surface of the energy locations 7118 and includes a specified refractive, diffractive, reflective or other energy modifying property. The energy suppression structure 7251 can be configured to restrict a first subset of energy propagation paths 7290 to a smaller range of propagation paths 7275 by suppressing energy propagation along the energy propagation paths 7274. A second energy suppression structure 7261 is configured to modify the energy propagation paths 7275 to a certain range depending on which energy locations in the plurality of energy locations 7118 the energy propagation paths 7275 extend through. The energy suppression structure 7261 can modify the energy propagation paths 7275 in a uniform or non-uniform manner such that certain energy propagation paths 7274 are suppressed. A third energy suppression structure 7254 is placed orthogonally between the regions of the energy locations 7118. The energy suppression structure 7254 exhibits absorptive properties and has a defined height along the energy propagation paths 7275 such that certain energy propagation paths 7274 are suppressed. Energy suppression element 7276 provides absorptive properties to limit the propagation of energy 280 while allowing energy 7281 to pass through. A combined system of similar or dissimilar waveguide elements 7112 is positioned to substantially fill the effective waveguide element aperture 7285 with energy from multiple energy locations 7118 and alter the propagation path 7273 of the energy as defined by the particular system.

[0183] In an embodiment, the energy suppression structure 7124 may be located proximate to the first energy waveguide 7122 and extend generally toward the first energy waveguide 7104. In an embodiment, the energy suppression structure 7124 may be located proximate to the first energy waveguide 7104 and extend generally toward the first energy location 7122.

[0184] In one embodiment, the energy system is configured to guide energy through the energy waveguide system to the single seamless energy surface along the second plurality of energy propagation paths, and to guide energy from the single seamless energy surface through the energy relay system to the plurality of energy devices along the first plurality of energy propagation paths.

[0185] In another embodiment, the energy system is configured to guide energy from the multiple energy devices through the energy relay system to the single seamless energy surface along the first multiple energy propagation paths, and to guide energy from the single seamless energy surface through the energy waveguide system along the second multiple energy propagation paths.

[0186] In yet another embodiment, the single seamless energy surface may be used to direct localized light transmission into three or fewer wavelengths of visible light.

[0187] Sensory data for holographic displays

[0188] The plenoptic 4D function from the energy guiding surface through the surface is from a first plane including the energy position and guided through a second coordinate along a path including the waveguide parameter u l ,v l The second plane provides two spatial coordinates x l ,y l , thus limiting the energy propagation path vector f l (x l ,y l ,u l ,v l ). Considering multiple energy guiding surfaces, the plenoptic 5D function starts from a first coordinate including one or more energy locations and guides through a second coordinate along a path including the waveguide parameter u l ,v l The plane provides three spatial coordinates x l ,y l ,z l , thus limiting the energy propagation path vector f l (x l ,y l ,z l ,u l ,v l ). For each of the 4D or 5D, additional variables f for time and color l (λ l ,t l ) Even if not explicitly stated for simplicity of function and discussion, it can be considered and assumed to include any of the necessary plenoptic functions for the application. For the avoidance of doubt, reference to energy guiding surfaces is for exemplary purposes only, and may include any other point, position, direction or plane in space for locating 5D coordinates, and collectively referred to as energy "guiding surfaces".

[0189] Figure 8 is a flow chart illustrating an embodiment of a process 800 for determining four-dimensional (4D) plenoptic coordinates of content data. The process 800 may include a step 802 of receiving content data, which may include any signal perceptible by a visual, audio, texture, feel, or olfactory sensor. Fig. 9 is a schematic diagram showing an embodiment of content data, which may include at least one of the following: object positions, material properties (e.g., material properties 906, 907, and 908), virtual light sources 904, geometric arrangements at non-object positions 902, content outside a reference surface, virtual camera positions 914, segmentation of objects 910, background textures 912, and layered content.

[0190] refer to Figure 8 and 9, process 800 may further include step 804, wherein the position of the data point is determined relative to the first surface 920 to create a digital stereoscopic representation 922 of the content data. The first surface 920 may be used as a reference surface for defining the position of the data point in space. In an embodiment, process 800 may further include step 806, wherein 4D plenoptic coordinates of the data point are determined at the second surface by tracking the position of the data point in the stereoscopic representation to the second surface to which the 4D function is applied. In an embodiment, process 800 may further include step 808, wherein an energy source position value is determined for the 4D plenoptic coordinates having the first convergence point.

[0191] The content data received in step 802 may include N views, where N is one or greater than one. A single view with or without a depth channel may be presented. A stereoscopic view with or without a depth channel may be presented. A multi-view image with or without a depth channel may be presented. In addition, a 4D light field with or without a depth channel may be presented.

[0192] The tracking of step 806 may use a priori knowledge of the calibration geometry of the energy system, which may be stored in memory as a global model or individually characterized systems, or some combination of both approaches.

[0193] In an embodiment, the mapping between input data and output energy sources provides a method for accurately mapping between various bit rate sources. The tracking of step 806 provides the ability to infer a complete stereoscopic 4D data set from the partial samples listed above. Depth information needs to be provided or calculated based on the available data. Using known or calculated depth information, N views can be inversely tracked by triangulating samples from a known stereo representation based on the depth coordinates into 4D space.

[0194] The triangulation may assume that each available energy source position in the N source contents represents the energy source position for each energy waveguide, provided a mapping between the energy waveguide and energy source position format resolution is provided. In the case where the resolution of the N source contents is lower, a super-resolution or scaling algorithm may be implemented. In the case where the resolution of the N source images is higher than the number of energy waveguides in the energy guiding device, interpolation between the super-sampled energy source positions may be performed to produce a higher number of energy source positions for each energy waveguide in the resulting 4D inverse ray trace.

[0195] The above assumes that distance information can be determined from a depth map, which may be accurate or inaccurate depending on the form of the depth information provided or calculated, and where the distance information is known or assumed, the combination of the distance information with the xy energy source position coordinates and (u, v) angle information as determined by the energy guiding device characteristics can be viewed as a 4D or 5D light field with limited imaging data samples. Based on the distance information, the imaging samples are triangulated back to the appropriate energy source positions that may exist behind each energy waveguide, respectively, and the missing data is generated in step 808 through the disclosure contained herein.

[0196] refer to Figure 7C , 8 , 9, 10, in an embodiment, the energy position can be located in the first surface 920, and the second surface to which the 4D function is applied can correspond to the waveguide system 7100 of the energy guiding device, and the energy can be operated to be guided through the waveguide system according to the 4D plenoptical coordinates of the data points to form a detectable stereoscopic representation of the content data.

[0197] In an embodiment, process 800 may further include step 810, wherein an energy source position value is determined for the 4D coordinates having the first convergence point. To provide example embodiments of the present disclosure, Fig.10 An embodiment of an energy guiding device 1000 is shown that undergoes a tracking process where content data in the form of an image 1002 has a provided or calculated distance position 1004 within a minimum position 1006 and a maximum position 1008 determined by a reference energy position 1010. In an embodiment, energy positions 1010 may include an energy guiding device surface. The known geometric arrangement of energy positions 1010 defined by a 4D plenoptic function allows triangulation of points 1014 on the virtual surface of image 1002 to be traced along rays 1016 to specific energy positions 1018 each having a unique xy coordinate. Missing samples may be calculated based on available information contained in the data set.

[0198] When additional N samples are provided, the same approach is applied with the additional multi-perspective imaging data, resulting in a richer set of inverse ray tracing samples and providing superior holographic results. The depth information from multiple (N) samples can be provided via a single depth map or up to N or more depth maps given a known mapping between source positions (N+X perspective maps) and source depth maps (N+X depth maps) to ensure that proper inverse ray tracing is performed.

[0199] Where a single depth map is provided, for example for the central N perspectives, additional depth maps can be interpolated by computing the disparity between each adjacent view to accurately map source and target positions between the N and N+X viewpoints. Using this approach, it is possible to inversely ray trace the mapping that depends on the appropriate view into the 4D light field, so that the correct perspective is projected onto the appropriate waveguide coordinates and the viewer is able to maintain the correct view dependencies in the associated viewpoints.

[0200] The encoder and decoder are powerful and can interpret a variety of data types, including but not limited to 2D / flat files, 2D with depth, stereo, stereo with a single depth channel, stereo with dual depth channels, N+X multi-view without depth, N+X multi-view with N+Y depths, geometry-based or vector-based scene files (which may include textures, geometry, lighting, material properties, etc. to reconstruct the environment), depth imaging files that can provide multiple RGBAZ values ​​for each xy coordinate, 4D or 5D (4D plus depth) light fields, or provided as N+X views plus N+Y delta channel data sets, where the depth channel provides a lower bandwidth method for only the amount of energy source position data required to render the determined energy-guiding device field of view. The processor is capable of inverse ray tracing at up to or exceeding real-time speeds in order to provide an appropriate 4D light field for presentation to the viewer, with or without world coordinate positions, with or without compensation for minimum and maximum projected world positions, and taking into account the nature of the energy-guiding device being characterized and / or designed.

[0201] In an embodiment, the process 800 may further include a step 812 in which a mapping between energy positions 7122 on the first side of the waveguide system 7100 and angular directions of energy propagation paths 7120 from the waveguide element 7100 on the second side of the waveguide system 7100 is applied. Doing so may allow for determination of a plurality of energy positions on the first side of the waveguide system 7100 corresponding to 4D plenoptic coordinates of the data points.

[0202] Fig.12 1 is a schematic diagram of a processing system 1200 including a data input / output interface 1201 in communication with a processing subsystem having a sense data processor 1202, a vectorization engine 1204, and a tracking engine 1206. It should be appreciated that the sense data processor 1202, the vectorization engine 1204, and the tracking engine 1206 may be implemented on one or more processors, either individually or in any combination thereof. Step 802 of process 800 may input content data to the processing subsystem 1220 via the data input / output interface 1201. The sense data processor 1202 may perform step 804 to create a stereoscopic representation of the content data. Step 806

[0203] In an embodiment, applying the mapping may include calibrating for distortions in the waveguide system 7100, which may further include calibrating for at least one distortion selected from the group consisting of: spatial distortion, angular distortion, intensity distortion, and color distortion.

[0204] In an embodiment, the energy guiding device may further include a relay system 6110 on a first side of the waveguide system 7100, wherein the relay system has a first surface 6116 adjacent to the waveguide system 7100, and the energy position 7112 on the first side of the waveguide system may be positioned adjacent to a second surface 6114 of the relay system 6110.

[0205] In an embodiment, applying the mapping may include calibrating for distortion in the waveguide system 7100. In an embodiment, applying the mapping may include calibrating for both distortion in the relay system 6110 and distortion in the waveguide system 7100. In an embodiment, the distortion to be calibrated may include at least one distortion selected from the group consisting of: spatial distortion, angular distortion, intensity distortion, and color distortion.

[0206] In an embodiment, a portion of the method may be performed in real time, or the method may be performed completely in real time, or at least two portions of the method may be performed in different time periods.

[0207] 2D to Light Field Conversion

[0208] In an embodiment, the content data may include data points in a two-dimensional (2D) space, and determining the position of step 704 may include applying a depth map to the data points in the two-dimensional space.

[0209] There are several methods to convert a two-dimensional or planar image into light field data. These methods include estimating depth information through depth through motion analysis, providing a depth channel through manual or rendering, or manually creating parallax, depth, occlusion, geometric arrangement and / or any other method known as standard for visual effects content creation to reproduce the entire light field by reproducing the entire environment through manual and automatic processes.

[0210] In a first embodiment, a system is possible that includes a real-time or offline processor to perform depth estimation based on available energy source location information. This can be performed at the energy directing device as a set-top box or as an offline process. Additional calculations for missing stereo data can be performed using temporal information and / or prior art texture synthesis or other techniques known in the art.

[0211] In a second embodiment, the depth information is provided as an image stream and may be embedded in the image format. Similarly, additional calculations may be performed for missing stereo data.

[0212] In a third embodiment, the missing environment information is generated using an artist or process that may include the following processes: isolating or segmenting each object in the scene, tracking the objects manually, semi-automatically, or automatically over time, placing the objects in space using parallax space, energy directing device space, optical space, or world coordinates, synthesizing background and foreground missing information through visual effects processes known in the art to reconstruct background, transparency, edge details, etc. to regenerate the environment. For the avoidance of doubt, the process implemented may be any, none, or all of the listed embodiments for reconstructing these environments. The generated environment information should contain as much missing information as possible, which is determined by the perspective of the energy directing device, and the artist can know these perspectives to ensure that the appropriate occlusion and view-dependent information is properly generated.

[0213] Additionally, surface models of each object in the scene can be generated, either as partial or fully constructed models, and textures from the image data projected onto the surfaces of the geometry to provide appropriate shapes for subsequent inverse ray tracing.

[0214] Additionally, material properties can be calculated or manually introduced to ensure that view-dependent lighting can be introduced together with the virtual illumination source to further improve the accuracy of the 4D light field reproduction.

[0215] Additionally, additions of CG or synthetic content may be introduced to augment existing conversion material. Additions of stereoscopic data may also be introduced. Intermixing of N+X content may also be introduced to provide seamless mixing between CG, 2D, stereo, multi-view and / or 4D media to form a single composite.

[0216] The resulting 2D to light field converted content may be retained as a geometric scene file, containing the geometry arrangement, textures, lighting, materials, etc. indicated in the CG scene itself, rendered as N+X views with N+D depth channels, rendered as a 4D or 5D (4D+depth) light field, a stacked depth image in a format that allows multiple RGBAZ samples for each xy energy source position coordinate with or without limited Z samples per xy coordinate, or provided as an N+X view plus N+Y delta channel data set, where the depth channel provides a lower bandwidth method for rendering only the amount of energy source position data required for the determined energy directing device field of view. Tools may be provided to allow generation of all, some, or one of these respective output formats.

[0217] Stereo and multi-view to light field conversion

[0218] The process from above utilizing single view content can be applied to stereo and multi-view material. An estimate of depth information is obtained via depth through motion analysis and stereo, multi-view and / or disparity analysis, through one or more depth channels provided manually or in a rendered manner or manually creating disparity, depth, occlusion, geometric arrangement and / or any other method known as a standard for visual effects content creation to reproduce the entire light field by reproducing the entire environment via manual and automated processes and utilizing appropriate data to further maintain view-dependent content available in the provided imaging material.

[0219] In an embodiment, the content data received in step 102 may include data points in a three-dimensional (3D) space, and determining the position may include adjusting the data points in the 3D space.

[0220] In an embodiment, adjusting the data points in the 3D space may include applying a depth map to the data points in the 3D space, adding new data points, reconstructing occluded data points, or any combination thereof.

[0221] The significant advantage of this method is that the accuracy of stereo disparity estimation is much higher than that of motion disparity alone or other similar 2D estimation processes. In addition, since some view-dependent conditions, including but not limited to lighting, transparency, material, occlusion, etc., can be obtained, the image quality of the resulting converted 4D light field is more accurate.

[0222] The ability to preserve explicit angular dependencies of multi-view image data relies on the ability to compute surface normals with respect to a central viewpoint camera or some other defined central point. Knowing these normals and disparity or depth information, it is possible to interpolate between viewpoints based on the perspective of the energy directing device and then apply them directly to inverse ray tracing or synthesize them as part of texture synthesis during inverse ray tracing.

[0223] For the sake of brevity, all previously published methods for 2D reconstruction into light field images can be applied to reconstruct stereo or multi-view datasets.

[0224] Generate N×N RGB image from 4D or 5D light field

[0225] By utilizing a 4D or 5D light field, it is possible to generate RGB multi-view images of any value of N×N or up to N×N number. This process is adapted by considering each lower left coordinate under each waveguide (assuming a square grid, 0,0 position and upper right position as N,N position). The grid is exemplary only and any other mapping method can be utilized. For each 0,0 to N,N position, it is possible to form a full resolution image from a light field with the maximum possible depth of field based on the capture system utilized, where each waveguide in the array is considered as a single energy source position and each coordinate under each waveguide is a single energy source position of a larger array of energy source positions for each complete image from 0,0 to N,N, respectively. This process can also be repeated for a 5D light field of depth information. In this way, it is possible to easily convert between 4D or 5D light fields to any subset of the data set that needs to contain 2D, stereo, multi-view, point cloud, CG scene file or any other desired combination of data that can be derived from a 4D or 5D light field for various distribution reasons. For 4D or 5D structures with irregular or square packaging, further interpolation is required to align the energy source positions to a regular grid, or a linear mapping between the energy source positions and the non-square packaging structures can be implemented, where the resulting image may not appear as a straight line and may also contain energy source position artifacts.

[0226] Fig.11 A method of converting a 4D or 5D light field into multiple viewpoints is illustrated by arranging energy locations 1102 from beneath each energy waveguide element 1104, respectively, based on energy waveguide element locations and energy location coordinates. This provides the ability to seamlessly convert between light fields and smaller data sets.

[0227] N+X RGB and N+Y Depth Datasets

[0228] An ideal dataset format that provides the highest quality while balancing data transfer size includes using N+XRGB and N+Y depth+vector channels, where the N+X RGB information contains: N RGB images, which can represent a specific resolution and format; and X, which can represent different resolutions and formats of RGB data to include lower resolutions, incremental information, etc.; and N+Y depth+vector channels containing N depth+vector channels, which can represent a specific resolution and format; and Y, which can represent different resolutions and formats of depth+vector channels to include lower resolutions, incremental information, etc.

[0229] The number N+X views may be generated on a regular grid, from a radius around a center point with or without a center view, multiple radii around a center point with or without a center view, or any method that determines a mapping of the number of views to associated fill or perspective positions. The configuration of the viewing angle may be contained in the metadata of the file, or the depth+vectorization pass provided may contain a direct mapping to world coordinates so that the imaging data is aligned to the same coordinates in XYZ space without further necessary metadata.

[0230] 4D disk reversal and energy guide device compatibility processing

[0231] For any data captured using a plenoptic or lightfield 4D or 5D system, including potentially data captured using a virtual rig with an optical simulation of a 4D or 5D lightfield system, the resulting fly's eye view contains a disk of uv vectors representing the lightfield. However, these coordinates assume an energy focusing element which may not exist in the energy guiding device. In the proposed energy guiding device solution, the focusing element may be the viewer's eye, and the mapping between the capture systems and between the original capture method and the viewed energy guiding device is no longer correct.

[0232] To reverse this and correct for the additional missing energy guiding elements in the system when compared to the capture system, it is possible to flip each disk individually, where the xy position of each (u,v) coordinate is relocated based on the center point of each waveguide respectively. In this way, the inversion of the image formed due to the main waveguide is reversed and allows the light field energy guiding device to project rays with the correct xyuv orientation.

[0233] Another embodiment of this case may implement a hardware modification where an array of energy waveguides is utilized to provide a direct inversion of the energy source position of each presented energy waveguide. For light field energy guiding devices, this is advantageous to have a direct mapping between the potential capture system and the energy guiding device. This is further advantageous for embodiments including HMD systems or stereoscopic opaque energy guiding devices, such that one set of energy waveguides in the entire array may be eliminated by removing the necessity to relay more time to obtain accurate xyuv coordinates.

[0234] Additionally, not all light fields are the same. They can be captured with different NA, FOV, N values, optical prescriptions, etc. The intrinsics and extrinsics of the input light field data can be understood and converted into properties of the energy guiding device. This can be performed by embodiments of the generic parameterization for holographic and light field data contained in the present disclosure.

[0235] Energy Guidance System uses inverse energy tracking and vectorization of sensory characteristics to holographic sensory data. delivery conduct General parameterization

[0236] The plenoptic 4D function from the energy guiding surface through the surface is from a first plane including the energy position and guided through a second coordinate along a path including the waveguide parameter u l ,v l The second plane provides two spatial coordinates x l ,y l , thus limiting the energy propagation path vector f l (x l ,y l ,u l ,v l ). Considering multiple energy guiding surfaces, the plenoptic 5D function starts from a first coordinate including one or more energy locations and guides through a second coordinate along a path including the waveguide parameter u l ,v l The plane provides three spatial coordinates x l ,y l ,Z l , thus limiting the energy propagation path vector f l (x l ,y l ,z l ,u l ,v l ). For each of the 4D or 5D, additional variables f for time and color l (λ l ,t l ) Even if not explicitly stated for simplicity of function and discussion, it can be considered and assumed to include any of the necessary plenoptic functions for the application. For the avoidance of doubt, reference to energy guiding surfaces is for exemplary purposes only, and may include any other point, position, direction or plane in space for locating 5D coordinates, and collectively referred to as energy "guiding surfaces".

[0237] Along the first vector of energy propagation paths, multiple intersections including energy convergence may occur with additional energy propagation paths. At this intersection, the 3D point or depth parameter is at a location X in the multiple energy propagation paths having a 4D or 5D function. l ,Y l ,Z l Formation, where the 3D convergence point X l ,Y l ,Z l In the plurality of energy propagation paths, for x contained in one or more energy guiding surfaces l ,y l or x l ,y l ,z l coordinates, there is only a single u l ,v lThe propagation path is formed between the first coordinate and the convergent 3D point. 4D function f Z (x l ,y l ,u l ,v l ) or 5D function f Z (x l ,y l ,z l ,u l ,v l ) Commonly define all 4Dx l ,y l or 5Dx l ,y l ,z l Coordinates and for each convergence point at X l ,Y l ,Z l The commensurate existence l ,v l Transmission path.

[0238] Due to the energy guiding surface X l ,Y l ,Z l The first 5D coordinate generated by the convergence of energy from multiple energy propagation paths of the 4D dataset may represent a point within a larger object, volume, particle, or localized energy parameter, wherein the converged energy at additional coordinates near the first 5D coordinate may exhibit additional vectorized characteristics of the perceived energy in the environment or holographic data set. These vectorized characteristics may include information about each 5D coordinate, about each energy position coordinate within the 4D dataset, about an area within either the 4D or 5D dataset, or other subsets of coordinates comprising an energy surface.

[0239] In an embodiment, a general parameterization of 4D and 5D holographic sensory energy characteristics for propagating visual, auditory, somatosensory, taste, olfactory, vestibular or other desired energy for sensory system responses for grating and vector 2D, 3D, 4D, and 5D data sets is disclosed, where the 2D data may include a single angular sample, the 3D data may include two or more angular samples in a single dimension, the 4D data may include multiple angular samples in two dimensions, or the 5D data may include multiple angular samples in three or more dimensions, with reference to a second coordinate of a second plane of the 4D energy guiding surface.

[0240] Embodiments of the received sample data may include any of the following:

[0241] 1). 2D or monoscopic, planar, point cloud, uv-mapped geometry, intrinsic geometry, depth image, layered image, CAD file (intrinsic), single point sampling, single camera capture, single projector projection, stereoscopic (monoscopic single sampling point with vector in volume), source with 3 degrees of freedom (DoF; grating with monoscopic x, y, z rotation around a single point), source with non-lightfield 6DoF (grating + vector from monoscopic sample), stereoscopic energy directing device (monoscopic sample in volume), source with Pepper's ghost (single point projection), source for 2D AR HMD (monoscopic single or multiple focus planes; layered monoscopic), source for 2D VR HMD (monoscopic single or multiple focus planes; layered monoscopic), or any other representation of 2D grating or vector information;

[0242] 2). 3D or Stereo, Triplex (single baseline), Multiview (1D), 1D multisample, 1D multiview, horizontal or vertical disparity only, 1D projection array, two point sampling, 1D point sampling, horizontal or vertical array, bullet time, source for 3DoF (grating; stereo x,y,z rotation around a single point), source for 3DoF (3D grating within stereo x,y,z rotation around a single point), source for non-lightfield 6DoF (3D grating + vectors from stereo samples), source for 1D stereo energy director (samples with 1D disparity), autostereo data source, source for horizontal multiview energy director, 3D AR HMD (stereo single or multiple focus planes; layered stereo), source for 3D VR HMD (stereo single or multiple focus planes; layered stereo), or any other representation of three-dimensional raster or vector information;

[0243] 3). 4D or plenoptic (5D), multi-view, integral image, light field (4D), holographic (4D), 2D multi-view, 2D multi-sample, 2D multi-angle, 2D parallax, horizontal and vertical parallax, 2D projection array, 2D point sampling, motion capture stage (along the surface), planar array, position camera array, rendered or ray traced geometric representation (4D representation), extrinsic geometric arrangement (4D representation), source of light field 6DoF (4D grating within planar light field sample), source of free viewpoint 6DoF (4D grating + vectors from 4D light field sample), source of 4D stereo energy guiding device (sample with 2D parallax), source of light field energy guiding device (4D sampling), source of light field HMD (near field 4D sampling), source of holographic energy guiding device (4D sampling), or any other representation of 4D grating or vector information;

[0244] 4) 5D or plenoptic + depth, light field + depth, holographic (5D sampling, 4D + depth), arbitrary multiple views (along all x, y and z axes), multi-sampling (along all xyz), multi-view (along all xyz), stereo disparity (along all xyz), projection array (along all xyz), point sampling (along all xyz), motion capture stage (along all xyz), position camera array (arbitrary xyz configuration), rendered or ray traced geometric representation (5D representation), cubic or stereo rendering (along all xyz), Non-intrinsic geometric arrangement (5D representation), source of light field 6DoF (5D grating within stereo light field samples), source of free viewpoint 6DoF (5D grating + vectors from 5D light field samples), source of 5D stereo energy guiding device (multi-plane 4D sampling), source of 5D light field energy guiding device (5D sampling, 4D + multiple planes), source of 5D light field HMD (near-field 5D sampling, 4D + multiple planes), source of holographic energy guiding device (5D sampling, 4D + multiple planes), or any other representation of five-dimensional grating or vector information.

[0245] At each second coordinate, the data provided may include a subset or superset of raster or vector samples, and wherein the samples may represent and contain other vectorization information to enable transformation to an increased sampling density by interpretation or processing of the subset or superset of raster or vector samples.

[0246] For each of the provided 2D, 3D, 4D or 5D data sets, the information is converted by vectorizing the information, manually identifying, computer vision analysis, automatically processing or otherwise to convert the provided samples from the original data set to a 5D coordinate system. For each of the provided 2D, 3D, 4D or 5D data sets, the information may include multiple samples or sample layers, and additional vectorized characteristics (with reference to the second coordinate of the second plane of the 4D energy guiding surface) regarding the starting angle sampling component of each provided data set, or may include a combination of constituent samples of any of the 2D, 3D, 4D or 5D additional provided data sets.

[0247] Each sample provided includes an eigenenergy for each desired coordinate, wherein the eigenenergy may include additional extrinsic energy properties, wherein the eigenenergy represents the value at a given 5D coordinate in the absence of other external samples, characteristics or environmental conditions. In the electromagnetic spectrum, this may be referred to as albedo, which is a dimensionless measure corresponding to the reflectivity of a white object, which reflects all incident radiation, but explicitly extended to each desired sensory energy, where the range of dimensionless values ​​is commensurate with the specified sensory energy. Within the visual sensory system, this range is approximately 400nm to 700um, and within the auditory sensory system, this range is approximately 20Hz to 20kHz.

[0248] Over the past few decades, tremendous technological advances have made it possible to artificially reproduce human senses using complex pattern recognition of sensations, aromas, and tastes detected by electronic means. For other systems that may exist outside the electromagnetic spectrum, these dimensionless values ​​can be characterized in the same way based on the sensed acuity response. Although holographic sensory energy technology is emerging, the content disclosed in this embodiment includes systems, methods, and formats for stimulating all human senses in a virtual environment to illustrate the general construction for various sensory parameters, thereby providing any sensory energy parameters or devices required for full immersion in the constructed virtual environment, appropriate data handling, transmission, storage, vectorization from the parameters or devices and between them, and embodiments of energy propagation for holographic sensory technology will be disclosed in future applications. In addition, the intention of this disclosure is to enable other simulation devices (including novelties (such as classic "smell-vision") or modern versions (such as FeelReal's olfactory VR headset)) to utilize the parameterized values ​​provided in the vectorization of the data set in this article.

[0249] In an embodiment, a somatosensory system may be defined based on components defining sensitivity, including: mechanoreceptors for texture, whose pressure sensitivity range in the skin may be normalized between 50 Hz and 300 Hz; thermoreceptors, whose temperature sensitivity range in the skin may be normalized between 0°C and 50°C (although this range may be much wider, with upper and lower limits defined by the extreme values ​​of temperature) or surface deformability thereof defining a range of viscoelastic behavior of the material, which measures both viscous and elastic properties when undergoing deformation between stress and strain over time, which provides a variety of physical quantities, including variables for time, strain, modulus, and other dynamics, and for the purposes of this disclosure is simplified to a dimensionless normalized scale, with a value of 0 for an immovable solid (such as granite) and a value of 1 for a low viscosity liquid such as water. Those skilled in the art will appreciate that the actual vectors provided will include the necessary physical quantities to appropriately define the viscoelasticity of the material and are normalized for exemplary purposes only.

[0250] Finally, recent advances in artificial electronic sensing, including taste and smell devices, demonstrate viable pathways to further vectorize the sensory parameters disclosed by the Holodeck design parameters and enable electronic reproduction of artificial tastes and odors by means of holographic waveguides, as described herein. Artificial electronic taste and odor receptors have made great strides through emerging nanodevices, where frequency-based artificial taste receptors use enzymatic biosensors to sample the intensity of chemical stimuli by encoding and converting to frequency-based pulses, thereby repeatedly and accurately detecting the taste as the frequency of the sampled chemical composition through a pattern recognition system, thereby detecting the taste that constitutes human taste. It is believed that the technology described can be extended to all types of detectable tastes, and similar advances in artificial olfactory systems have demonstrated digital interfaces for stimulating odor receptors using weak electrical pulses directed to the nasal concha, and research is underway to further parameterize the patterns contained within the frequency of specific olfactory responses through changes in the electrical signal.

[0251] By establishing pathways for arbitrarily generating frequencies and complex electronic patterns representing olfactory, taste, and other sensory systems, in one embodiment, the acuity response to taste can be vectorized to include a normalized scale for each electronic control parameter, with a minimum and maximum taste response represented along the scale from 0 to 1 to saturate the average 2,000-8,000 taste buds in a person, potentially including but not limited to vectors for sour, salty, bitter (spicy), sweet, and savory (umami), where the vectors and the spatial coordinates of the vectorized signals can provide information for the generation of a complex olfactory implementation.

[0252] In another embodiment, the acuity response to odors can be further vectorized to include a normalized scale for each electronic control parameter, with minimum and maximum olfactory responses along the scale from 0 to 1, such that the average 10 cm human 2 The olfactory epithelium is saturated, for each highly complex olfactory space, which may include but is not limited to vectors for aromatic, fruity, citrus, woody (resinous), chemical, sweet, minty (peppermint), roasted (nutty), spicy, and decayed, where the spatial coordinates of the vectors and the vectorized signals can provide information for the generation of complex olfactory implementations.

[0253] Each of these vectors may provide normalized values ​​representing these patterns for taste, smell, or other sensory domains, and be converted to waves, amplitudes, magnitudes, or other attributes as required for the appropriate application of the provided vectorized values. While smell and taste are two of the most highly degraded senses within the sensory system, with parameterized values ​​to vectorize complex fusions, in embodiments, it is additionally possible to provide user-based interactive control of the sensitivity of any such sensory energies to provide personalized customization of visual, auditory, somatosensory, taste, olfactory, vestibular, or other desired sensory system responses.

[0254] In an embodiment, each represented perceived albedo energy value of a sample may additionally include extrinsic energy attributes incorporated into the single sample value, which represents the cumulative result of each corresponding provided sample of other external samples, characteristics, or environmental conditions. In this configuration, the composite sample value may or may not exhibit the potential attributes of other energies from other samples in a physically based or simulated environment. The most efficient and purest method of transmitting parameterized and reconstructed holographic data sets is based on a single intrinsic sample information to provide simplified and lower bandwidth frequency information, but it is not always possible to receive outside of a fully synthetic environment, especially for physically based imaging or acoustic systems. In any real-world environment, there will always be a certain amount of extrinsic factors that affect the resulting sample information. Certain systems (such as light stages or other systems known in the art that facilitate estimation of reflectivity, shape, texture, and motion capture) utilize some form of structured lighting and one or more imaging devices to directly or indirectly analyze albedo, depth information, surface normals, and bidirectional scattering distribution surface characteristics.

[0255] The bidirectional scattering distribution function (BSDF) is a generalized superset of the bidirectional projection rate distribution function (BTDF), the bidirectional texture function (BTF), and the bidirectional reflectivity distribution function (BRDF), which are usually represented by the generalized function f r (w i ,w r ) represent, together serve as a model to parameterize and identify surface properties in computer graphics and vision algorithms known in the art. The function describes the energy propagation path given the incoming incident direction w i and the outgoing reflection or projection direction w r How visible light reflects, projects, or otherwise interacts with a surface in the case of , where the surface normal is perpendicular to the tangent to the object surface, and the function describes the direction of the light along the outgoing path w r The outgoing reflected radiation is the same as the incoming path w i The ratio of the irradiance incident on a surface, where each w i ,w r A 4D function may be included to define parameterized azimuth and apex angles for each of the incoming and outgoing light paths.

[0256] Can target x i Energy λ i The function is further elaborated in the first position of the impact surface, and then in the material properties the energy is scattered internally to x r Energy λ rThe output energy is based on the material properties of the surface, the input energy and position, and the output energy and position across the surface, volume or point of the object.

[0257] Therefore, generalized properties describing how energy is transferred between any two energy rays impinging on a surface (including wavelength or frequency dependence and spatially varying material properties or surfaces) can be expressed as 10D functions, and f r (λ i ,x i ,w i ,λ r ,x r ,w r ) is specified for each or any available or provided sample in the data set to take into account the input energy, the effect of the vectorized surface profile, and the reflection, refraction, specular reflection, projection, scattering, diffusion or other material property output results from any energy domain (generalization of a given function) f r .

[0258] Now consider the energy guiding surface, the plenoptic 4D function is guided from the first plane including the energy position and through the second coordinate along the waveguide parameter u l ,v l The second plane provides two spatial coordinates x l ,y l , thus limiting the energy propagation path vector f l (x l ,y l ,u l ,v l ). Considering multiple energy guiding surfaces, the plenoptic 5D function starts from a first coordinate including one or more energy locations and guides through a second coordinate along a path including the waveguide parameter u l ,v l The plane provides three spatial coordinates x l ,y l ,z l , thus limiting the energy propagation path vector f l (x l ,y l ,z l ,u l ,v l ). For each of the 4D or 5D, additional variables f for time and color l (λ l ,t l) Even if not explicitly stated for simplicity of function and discussion, it can be considered and assumed to include any of the necessary plenoptic functions for the application.

[0259] Along the first vector of energy propagation paths, multiple intersections including energy convergence may occur with additional energy propagation paths. At this intersection, the 3D point or depth parameter is at a location X in the multiple energy propagation paths with a 4D or 5D plenoptic function. l ,Y l ,Z l Formation, where the 3D convergence point X l ,Y l ,Z l In multiple energy propagation paths, for each x contained in the energy guiding 4D surface or 5D surface l ,y l or x l ,y l ,z l coordinates, there is only a single u l ,v l The propagation path angle formed between the first coordinate and the convergence 3D point. 4D function f Z (x l ,y l ,u l ,v l ) or 5D function f Z (x l ,y l ,z l ,u l ,v l ) Commonly define all 4Dx l ,y l or 5Dx l ,y l ,z l Coordinates and for each convergence point at X l ,Y l ,Z l The commensurate existence l ,v l Transmission path.

[0260] At the convergence coordinate X l ,Y l ,Z l At a location, a surface is formed, and the surface may include a point, a volume, an object, or another embodiment of a 3D location including a converging energy propagation path. The samples provided for each surface location may include one or more surface characteristics, vectors, materials, features, or other identifying characteristics V ito characterize or otherwise process the resulting energy and one or more input energy sources impinging a given point near a surface location, where the reflection function now includes generalized vectors for various properties of the surface and is represented as an 11D general object parameterized function f r (λ i ,x i ,w i ,λ r ,x r ,w r ,V i ).

[0261] 11D general holographic parameterization function f r (λ i ,x i ,w i ,λ r ,x r ,w r ,V i ) defines the resulting values ​​of the given environment and vectorized object properties, and the 4D function f l (x l ,y l ,u l ,v l ) defines the energy propagation path from the surface of the energy guiding device, and can therefore be further generalized to a 15D universal holographic parameterization function f r (λ i ,x i ,w i ,λ r ,x r ,w r (x l ,y l ,u l ,v l ),V i ), where the transmission direction w r Defined and equal to the propagation path u l ,v l , thus limiting the spatial coordinate x l ,y l , and for each transmission direction w r , there may be only one value set f l (x l ,y l ,u l ,v l ) satisfies w r =u l ,v l In addition to the rendering requirements associated with the disclosed general parameterization of 4D and 5D holographic sensory energy characteristics, those skilled in the art will understand the various transformations and mathematical constructions.

[0262] With a full 15D function describing the vectorization of all sensory energy properties to coincide with the surface formed by the convergence point in space, multiple orders of magnitude of required data are essentially eliminated, providing a viable path to achieving the transmission of a truly holographic data set.

[0263] Vectorized properties strive to provide accurate physical quantities for the properties of each sensory domain that can be programmed, captured synthetically, or evaluated computationally, where V i Each surface, volume or 3D coordinate X of an object for a given sample within a provided dataset may be specified with respect to general system metadata or each or any sensory energy domain. l ,Y l ,Z l Attributes of the vectorized feature, including:

[0264] 1.) System metadata may provide any sensory energy specific properties or system-wide references to surface properties for each sample, including normals, depth information, environmental properties, multiple angle samples for a given 3D coordinate, procedural textures, geometry arrangements, point clouds, depth image data, still frames, time frames, video data, surface IDs, surface passes, coordinate maps, virtual camera coordinates, virtual lighting and visible energy information, environment maps, scene information outside the domain of visual sensory sample information, curves, vertices, time information, networked data, databases, object recognition, energy devices, external data sources, sensors for system modification and interaction, system status, voice recognition, smell detection, hearing detection, facial recognition, somatosensory recognition, taste recognition, UI, UX, user profiles, flow and motion vectors, layers, regions, transparency, segments, animations, sequence information, procedural information, displacement maps, or any other scene data necessary to provide sufficient data for proper processing of each sample;

[0265] 2.) Visual sensory energies may provide surface properties to define appropriate visible or invisible electromagnetic energy, iridescence, luminescence, subsurface scattering, non-local scattering effects, specular reflections, shadows, absorptivity, projections, masking, interreflection, albedo, transparency, physics, dynamics, reflection, refraction, diffraction, optical effects, atmospheric effects, frequency, modulation, surface contours, textures, displacement maps, physics and dynamics to specifically relate to other sensory energies and respond based on the provided energy (e.g., sound vibrations that change reflectivity properties or tactile material deformations that cause surface deformation), layers, zones, transparency, segments, curves, animations, sequence information, procedural information, size of materials, environmental conditions, room dynamics, or other relevant material properties for a surface, environment, room, object, point, volume, etc.;

[0266] 3.) Auditory sensory energies: vectors, magnitudes, amplitudes, masses, material propagation parameters, absorptivity, projections, material properties informing acoustic reflectivity, diffusion, projection, amplification, masking, scattering, localization, frequency dependence or modulation, pitch, tone, viscosity, smoothness, texture, modulus, any other parameters that determine the propagation of sound waves within an object, surface, medium or other object, physics and dynamics related to the placement of a localized sound field to specifically relate to other sensory energies and respond based on the energy provided (e.g., changing the temperature of the sound of a material), layers, zones, transparency, segments, curves, animations, sequence information, procedural information, size of the material, environmental conditions, room dynamics or other relevant material properties for a surface, environment, room, object, point, volume, etc.;

[0267] 4.) somatosensory energy vectors associated with mechanoreceptors for textures, pressure, thermoreceptors, temperature, surface deformability parameters, and vectors defining the range of viscoelastic behavior of materials measuring both viscous and elastic properties when undergoing deformations between stress and strain over time, providing a variety of physical quantities including variables for time, strain, modulus and other dynamics, layers, regions, transparency, sections, curves, animations, sequence information, procedural information, size of materials, environmental conditions, room dynamics, or other relevant material properties or other somatosensory parameters for surfaces, environments, rooms, objects, points, volumes;

[0268] 5.) Taste sensation energy vectors for aromatic, fruity, citrus, woody (resinous), chemical, sweet, minty (peppermint), roasted (nutty), spicy, and decayed, wherein the vectors and spatial coordinates of the vectorized signals can provide information for the generation of complex olfactory implementations and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, zones, transparency, segments, curves, animations, sequence information, program information, size of materials, environmental conditions, room dynamics, or other relevant material properties for a surface, environment, room, object, point, volume, or other taste sensation parameters;

[0269] 6.) Olfactory sensory energy vectors for sour, salty, bitter (spicy), sweet, and savory (unsavory), wherein the spatial coordinates of the vectors and vectorized signals can provide information for the generation of complex olfactory implementations and further provide duration, magnitude, frequency, length, time, radius, modulation, layer, zone, transparency, segment, curve, animation, sequence information, program information, size of material, environmental conditions, room dynamics, or other relevant material properties for a surface, environment, room, object, point, volume, or other olfactory parameters;

[0270] 7.) or other related sensory dynamics based on physical, synthetic, transmission, or computational interdependencies from any other sensory sample data sets; required, designed, or desired sensory system vectors; and any additional sensory characteristics where parameterization of specific features is beneficial for the reconstruction, storage, processing, or transmission of generalized holographic construction data.

[0271] Where the received data set comprises 2D data having a single angular sample, 3D data has two or more angular samples in a single dimension, 4D data has multiple angular samples in two dimensions, or 5D data has multiple angular samples in three or more dimensions.

[0272] For all provided source materials, each source material may be subject to additional processing to properly prepare for efficient vectorization of the holographic dataset. For any provided source material that exhibits lower spatial or angular resolution than the energy guiding surface, a transformation process may be required in order to accurately convert the original source into a 4D or 5D dataset.

[0273] For proper preparation, in an embodiment, the 2D or 3D source material provided comprises a photographic capture from a standard imaging system. Within this image sequence are rasterized reflections, refractions, transparent elements, and other similar examples of material property interactions occurring with physically based lighting.

[0274] In cases where content is prepared by merely identifying surface IDs of surfaces with already rasterized material properties, valid data may be sufficient to converge to a 4D coordinate system, however any additional rendering applied to these surfaces will exhibit a double image of the physical quantities used for both the photograph and the parameterized synthetic rendered reflectivity properties. An ideal source data set for efficient holographic transmission includes an albedo representation of the sample source information, plus vectorized material properties for each specified energy domain, where the metadata forms an object-based stereo sampling of the albedo multi-view samples, and where all material properties accurately provide accurate surface identification and rendering based on the specified vectorized surface properties, as well as positioning or projection of other perceived energies.

[0275] In an embodiment, a manual, semi-automatic, computer vision, or automatic process is provided to algorithmically or manually evaluate the content in a source sample data set, and wherein a manual or algorithmic analysis is performed, thereby performing segmentation and other object isolation methods known in the art to identify areas containing undesirable physical rasterization effects. In an embodiment, a person is photographed in front of a background, wherein the material properties of the person include reflections from the environment, and background objects are occluded by the photographed person. After these areas are identified as undesirable areas, a process can be utilized to: 1) isolate the relevant objects; 2) separate all object elements into core components to account for occlusion, transparency, edges or other elements; 3) provide object elements through image analysis, temporal analysis, energy analysis, and with the help of machine learning, computer vision, additional hardware and energy devices that additionally capture information about the scene, objects and / or environment, or through a completely manual approach, so that any surface that should exhibit material properties has any such incorporated material properties removed through computer vision, algorithms, processors, or manual visual effects, wherein manual processes are typically used in the art to remove lines, paint repairs, clean panels, image repairs, etc. Methods for complex, alpha matte creation, occlusion filling, object reconstruction, image projection, motion tracking, camera tracking, rotating mirrors, optical flow, etc., for the purpose of reproducing intrinsic material properties in the absence of non-intrinsic material properties, thereby preparing the most efficient transmission and dissemination content for the data set; 4) the above additional processes involve manual or computer-assisted identification of the depth or 3D coordinate value of each required sample; and 5) in this embodiment, also includes the identification of associated material properties, each material property represents a point, data area, surface, object or other representation of the material, so that the data can be easily further rendered in a display driver of the energy-guiding device or any additional system capable of encoding and decoding parameterized data sets.

[0276] In an embodiment, the dataset from above includes 3D multi-view samples prepared with albedo visual energy samples, each sample having multiple layers of rgba information, a set of vectorized material properties to associate each segmented material with a surface ID and a series of surface parameters to closely reconstruct the original source dataset before removing extrinsic image data, and wherein the acoustic dataset is provided with vectorized material properties associated with the material properties of the visual energy system and multiple sound channels, each having an identified frequency, modulation, spatial placement and other sound localization characteristics, and wherein a somatosensory sensation energy dataset is provided for a subset of the surfaces contained in the visual energy dataset to additionally include viscoelastic and temperature vectorized material properties, both of which are related to the other vectorized datasets.

[0277] From any provided data set, evaluating the relative depth position of each provided sample in the visual energy data set relative to the surface of the energy guiding device, wherein each sample in any one of the visual energy samples is placed in a 3D coordinate system, and wherein relative to each 3D coordinate being associated with a position x in a plurality of energy propagation paths within the 4D or 5D plenoptic function l ,Y l ,Z l The energy propagation path length of each provided sample is evaluated by a function associated with a plurality of concurrent converging energy propagation paths intersecting the first 3D point, wherein for each x contained within the energy guiding 4D surface or 5D surface l ,y l or x l ,y l ,z l coordinates, there is only a single u l ,v l The propagation path angle formed between the first coordinate and the convergence 3D point. 4D function f Z (x l ,y l ,u l ,v l ) or 5D function f Z (x l ,y l ,z l ,u l ,v l ) together define all 4D x contained in the energy guiding device l ,y l or 5Dx l ,y l ,z l Coordinates and for X l ,Y l ,Z l Each convergence point at l ,v l propagation path, and wherein after performing this analysis process according to the presented or available 4D x l ,y l or 5D x l ,y l ,z l The total number of samples is known, and the number of samples at position X l ,Y l ,Z lThe total energy propagation path length between each 3D point at and the 4D or 5D coordinate position is known, and a weighted distribution based on the total available samples per 4D or 5D coordinate and the minimum path length of the 3D coordinate values ​​sampled from the available multiple 3D coordinate data provides a complete sampling of the 4D or 5D light field from an arbitrary data set.

[0278] As another embodiment of the above, after 1) placing each sample of any of the visual, acoustic, somatosensory and any other provided energy samples 2) in a 3D coordinate system based on the provided data set, additional processing or additional vectorization characteristics, and before performing coordinate analysis; 3) evaluating the 15D general holographic parameterization function f r (λ i ,x i ,w i ,λ r ,x r ,w r (x l ,y l ,u l ,v l ),V i ), wherein 4) additional known environment scenes, geometric arrangements, metadata, etc. are provided, each with independent vectorized material properties; 5) virtual lighting information is provided and additional perceived energy metadata properties are evaluated to evaluate any potential interference between the properties that may change the rendering function, and 6) the 15D parameterized function is evaluated for each provided 3D coordinate and corresponding vectorized material property to 7) the rendering process is performed by an online, offline, real-time, processor, ASIC, FPGA, cloud or other form of rendering process to generate new multiple angle-varying material properties given any provided data set, and wherein 8) the rendering process is specific to and equal to each propagation path u l , each transmission direction w of v1 r , thereby limiting the spatial coordinates x1,y l , and for each transmission direction w r , there may be only one value set f l (x l ,y l ,u l ,v l ) satisfies w r =u l ,v l , and wherein 9) based on the rendering results and the resulting available new angle-varying material properties, relative to making each 3D coordinate relative to a position x in a plurality of energy propagation paths within a 4D or 5D plenoptic function l ,Y l ,Z lThe evaluation of a function associated with a plurality of coexisting converging energy propagation paths intersecting the first 3D point at a comprises each 4D or 5D coordinate of the energy propagation path length for each provided sample, wherein for each x contained within the energy guiding 4D or 5D surface l ,y l or x l ,y l ,z l Coordinates, only a single u exists l ,v l The propagation path angle formed between the first coordinate and the convergence 3D point. 4D function f Z (x l ,y l ,u l ,v l ) or 5D function f Z (x l ,y l ,z l ,u l ,v l ) together define all 4D x contained in the energy guiding device l ,y l or 5D x l ,y l ,z l Coordinates and for each convergence point at X l ,Y l ,Z l The commensurate existence l ,v l propagation path, and wherein after performing this analysis process according to the presented or available 4D x l ,y l or 5D x l ,y l ,z l The total number of samples is known, and the number of samples at position X l ,Y l ,Z l The total energy propagation path length between each 3D point at and the 4D or 5D coordinate location is known, and a weighted distribution based on the total available samples per 4D or 5D coordinate and the minimum path length of the 3D coordinate values ​​sampled from the available multiple 3D coordinate data provides a complete sampling of the 4D or 5D light field for all provided sensory energies from an arbitrary data set.

[0279] Additional embodiments of the above system wherein the rendering further takes into account bi-directional energy directing surfaces such that sensed electromagnetic energy representative of lighting of the real-world environment, or absorption of certain sound frequencies in the environment can result in dynamic or offline updating of the rendering process, or evaluation of other sensed interactive real-world elements, and wherein lighting and sound sources or other sources are adjusted to accommodate modifications to environmental conditions.

[0280] Back to Figure 8 In view of the principles disclosed above, in an embodiment of process 800, the received content data may further include vectorized material property data, and wherein process 800 further includes step 830, wherein the digital stereoscopic representation of the content data is associated with the vectorized material property data; and wherein in step 804, the energy source position value is determined based at least on the vectorized material property data associated with the stereoscopic representation of the content data.

[0281] refer to Fig. 9 and 13 In an embodiment, the vectorization process 1300 may include steps 1302 and 1304, in which first content data is received, and in which surface 915 is identified in the content data. In an embodiment, identifying surface 915 may include using segmentation data in the content data. The vectorization process 1300 may further include steps 1306 and 1308, in which a surface identification of surface 915 is determined in step 1306, and in which material property data of surface 915 is determined in step 1308. In an embodiment, determining the material property data may include manually determining or using a predetermined process. After steps 1306 and 1308, the vectorization process 1300 may further include step 1310, in which the surface identification is associated with the material property data of surface 915. The vectorization process 1300 may further include step 1312, in which a vector of material property data is created. The vectorization process 1300 may further include step 1314, in which vectorized material property data is generated based on the created vector.

[0282] In an embodiment, process 1300 may optionally include step 1316, wherein the material property data is removed from the first content data and replaced by the vectorized material property data generated in step 1314. In an embodiment, the vectorized material property data generated in step 1314 may be used in process 800 as described above to determine 4D plenoptic coordinates of an energy guiding device of the present disclosure as described above.

[0283] Process 1300 may be performed using any processing system of the present disclosure including processing system 1200. In an embodiment, content data may be received via data input / output interface 1201 in step 1302, and steps 1304 to 1314 of vectorization process 1300 may be performed using vectorization engine 1204. Additionally, the vectorized material property data generated in step 1314 may be used by sensation data processor 1202 and tracking engine 1206 for processing according to the steps of process 800 as described above. The tracking engine may perform steps 808 and 812 to determine 4D coordinates for the holographic representation. Step 810 may be performed by sensation data processor 1202. The output of the processing subsystem may be provided to compression engine 1210, from which the compressed data may be stored in memory or provided to data input / output interface 1201 for transmission to an energy guidance system locally or remotely connected to system 1210. The data may also be stored in memory 1208 until retrieved at a later time.

[0284] Rendering 4D energy fields from 3D environments

[0285] Further contemplated in the present disclosure are embodiments of an inverse tracking process that allows the digital stereoscopic representation to be rendered into a format compatible with energy directing device 1000 through an inverse mapping that provides an inverse path for sampling the digital stereoscopic representation of content data that can be presented as a 3D environment.

[0286] Rendering 4D energy fields in a 3D environment presents many challenges. For explanatory purposes, some of these challenges are described in the context of light fields. However, the disclosure of this application is applicable to other forms of energy. Furthermore, these descriptions do not limit the application of these principles to other energy forms.

[0287] Energy in a 3D environment may interact with objects in the 3D environment in various ways depending on their physical properties, making it possible to simulate how these objects generate, reflect, absorb, project, or otherwise affect energy. Fig.14 1400. A virtual viewer's viewpoint 1402 is shown in a 3D environment 1400 having a field of view 1404. Energy rays 1406 may arrive at the viewpoint directly from an energy source 1408, or by bouncing off reflective objects 1410A, 1410B, 1410C, which may affect the properties of the energy as it arrives at the viewpoint 1402. Various properties of energy in the 3D environment 1400 may be considered, including visual, auditory, somatosensory, taste, olfactory, vestibular, or other desired energy, as described throughout this disclosure, including paragraph 203. Note that Fig.14Only some of the energy paths toward the observation point are partially depicted. As will be appreciated, some energy paths and sources of other reflections may be omitted from the figure. In some cases, a single energy path 1406A reflects off multiple objects 14010A, 1410C (some of which 1410C may not be within the field of view 1404 of the observation point 1402) on its way to the observation point 1402. On the way to the observation point 1402, the energy 1406 may also pass through partially projected objects 1412 or refracting objects (not shown) that change how the energy is sensed when it arrives at the observation point 1402. Thus, accurate rendering of a scene from a 3D environment can, to some extent, take into account the multiple interactions that occur within and outside the field of view 1404 from the observation point 1402 of a virtual viewer in the 3D environment. This can become even more complicated in a holographic system, which may need to project a large number of unique energy convergences simultaneously. The diagram is presented for illustrative purposes. Fig.14 , and some possible aspects are oversimplified for ease of explanation. It should be understood that a 3D environment can take into account many additional physical properties that affect energy propagation in a 3D environment, and Fig.14 The embodiments of the present disclosure are not limited.

[0288] Existing techniques for rendering 4D energy field data from a 3D environment include dual frustum rendering, oblique rendering, and multi-view rendering. Dual frustum rendering is a multi-pass rendering method that fails to account for data outside the field of view of the observation point and requires computationally complex post-integration of data from multiple rendering passes. Oblique rendering also requires complex post-computation, requires tens of thousands of rendering passes, and makes unrealistic assumptions about the physical energy projection system (such as perfect projection optics, no aperture). Oblique rendering is also not easily integrated into existing workflows. And, finally, multi-view rendering does not allow for simultaneous rendering outside and inside the field of view, and there are some other problems caused by dual frustum rendering and oblique rendering.

[0289] Embodiments of the present application allow for systems and methods for rendering a 3D environment into a dataset for 4D energy projection in a single pass. At least because the single-pass rendering allowed by the present disclosure avoids computationally challenging post-integration efforts, embodiments of the present disclosure produce significantly smoother and more cohesive energy field projections. Moreover, because the methods and systems of the present disclosure take into account on-screen and off-screen effects. In embodiments, the systems and methods may include wavelength / energy considerations and display-specific calibrations (or any other functions that may be included for seamless movement between various systems). If and when advantageous, this may additionally include pixel processing with a displacement map to calculate other 4D coordinates.

[0290] Fig.15 is a flow chart illustrating an embodiment of a process for rendering a 4D energy field from a 3D environment. Fig.15 and 16 , the process may include a first step 1510 for providing a scene 1600 in a 3D environment described by a plurality of energy data points 1602 throughout the scene 1600. Fig.16 1600 in a 3D environment. As can be appreciated, energy data points 1602 can be configured in different ways in different embodiments of the 3D environment. Energy data points 1602 may include values ​​or sets of values ​​describing various physical properties that determine the virtual appearance of scene 1600 and how elements in scene 1602 can interact. Some examples thereof are color, reflectivity, material properties, and energy direction. Other properties that may be considered in scene 1600 may include visual, auditory, somatosensory, taste, olfactory, vestibular, or other required energies. In some embodiments, energy data points may store multidimensional functions. The function may include values ​​corresponding to properties in the physical world that describe how energy is reflected, refracted, projected, or otherwise affected when energy intersects with energy data points 1602. Moreover, in some embodiments, energy data points 1602 may also include intensity and direction information that allows energy data points to collectively model energy propagation in scene 1600. It should also be noted that energy data point 1602 may contain energy values ​​for other types of energy in addition to or instead of information related to electromagnetic energy in the visible spectrum. There is no limitation on the type of information that may be stored as a value in energy data point 1602.

[0291] It should be appreciated that there are various ways to configure or arrange the energy data points 1602 in the 3D environment. The density of the energy data points 1602 in the 3D environment 1602 may vary from embodiment to embodiment. Alternatively, in some embodiments, the density of the energy data points 1602 may vary from location to location in the 3D environment 1600, or from object to object in the 3D environment.

[0292] In some embodiments of the 3D environment 1600, the energy data point 1602 may correspond to a location in a virtual or physical space. Furthermore, the value of the energy data point 1602 may change as different objects occupy the space to which the energy data point is mapped. The value of the energy data point 1602 may also change as light propagates through an embodiment of the 3D environment. Again, it should be noted that although Fig.16 A scene in an electromagnetic 3D environment is depicted, but the disclosure is not limited to energy in that domain. Embodiments of the 3D environment may describe other types of energy besides electromagnetic energy in the visible spectrum, such as mechanical energy in the form of ultrasound waves that may be used to create a tactile surface. The other energies described may include visual, auditory, somatosensory, taste, olfactory, vestibular, or other energies.

[0293] Reference again Fig.15and Fig.16 In an embodiment, a process for rendering a 4D energy field from a 3D environment may include a second step 1520, which includes locating a plurality of virtual pixels 1604 on a virtual pixel plane 1606 in a scene 1600, wherein each pixel has a known unique 4D coordinate including a 2D angular coordinate and a 2D spatial coordinate, wherein the 2D angular coordinate describes the relative position of each virtual pixel 1604 to a virtual viewing plane 1610 (e.g., Fig.16 1600, and wherein the 2D spatial coordinates identify the position of a virtual aperture in a plurality of virtual apertures 1612 disposed on a virtual display plane 1614 in the scene 1600. In some embodiments, the virtual viewing position 1608 may be located on two or more virtual viewing planes, thereby allowing for rendering of coplanar devices.

[0294] It should be understood that the term "virtual pixel" does not limit the embodiments of the present disclosure to rendering a 3D environment including a light field. A virtual pixel may include an energy position in a 3D environment, including any form of detectable energy field, including but not limited to a tactile field and an acoustic field. Other energy fields may include visual, auditory, somatosensory, taste, olfactory, vestibular or other required energy fields. A plurality of virtual pixels 1604 may be arranged to form a two-dimensional array of virtual pixels. In some embodiments, a plurality of virtual pixels 1604 may be arranged in different ways. For example, an embodiment allows virtual pixels to be arranged to form a curved surface or any other shape. Similarly, in different embodiments, a plurality of virtual viewing positions 1608 are allowed to be arranged to form different shapes. In some embodiments, the virtual pixel plane 1606 is parallel to the virtual display plane 1614. In some embodiments, the virtual pixel plane 1606 is parallel to the virtual viewing plane 1610. In some embodiments, the virtual pixel plane 1606, the virtual display plane 1614 and the virtual viewing plane 1610 are parallel.

[0295] The positions of the plurality of virtual pixels 1604 and the plurality of virtual apertures 1612 relative to the plurality of virtual viewing positions 1608 in the scene 1602 may correspond to one or more positions of the plurality of energy sources and the positions of the plurality of apertures in the physical 4D energy projection system relative to the intended view volume of the 4D energy projection system. Moreover, in some embodiments, the number and positions of the plurality of virtual pixels 1604, the plurality of virtual apertures 1612, and the plurality of virtual viewing positions 1608 may vary depending on the number of energy sources or positions, the field of view, the number of apertures, the waveguides, or other features of the physical 4D energy projection system to render data for different 4D energy projection systems. In some embodiments, an algorithm may be used to calibrate the defined number and positions of the plurality of virtual pixels 1604, the plurality of virtual apertures 1612, and the plurality of virtual viewing positions 1608 to any type of 4D energy projection system. It should also be appreciated that although Fig.16 Only a small number of virtual pixels 1604, virtual apertures 1612, and virtual viewing positions 1608 are depicted, but embodiments may include tens, hundreds, thousands, or more of each of these elements. In addition, the virtual display plane 1614 may correspond to the location of an energy projection surface or screen in a physical 4D energy projection system.

[0296] Reference again Fig.15 and Fig.16 In an embodiment, a process for rendering a 4D energy field from a 3D environment may include a third step 1530, which includes sampling energy data points of a plurality of energy data points 1602 in a scene 1600 along a plurality of rays 1616 from a virtual viewing plane 1610. Each ray in the plurality of rays 1616 intersects a virtual viewing position in a plurality of virtual viewing positions 1608 and a virtual pixel in a plurality of virtual pixels 1606, and the angle of intersection is determined by the 2D angular coordinate of the one virtual pixel in the plurality of virtual pixels 1604. Each ray in the plurality of rays 1616 intersects a virtual aperture in a plurality of virtual apertures 1612, and the virtual aperture determines a 2D spatial coordinate of the ray. The 2D spatial coordinate and the 2D angular coordinate together form a single 4D energy field coordinate. For example, first ray 1618 in plurality of rays 1616 intersects first virtual pixel 1620 in plurality of pixels 1604 and first virtual viewing position 1622 in plurality of virtual viewing positions 1608 at an angle determined by the 2D angular coordinates of the known unique 4D coordinates of first virtual pixel 1620. First ray 1618 also intersects first aperture 1624 in plurality of virtual apertures 1612, the first aperture being identified by the 2D spatial coordinates of the first aperture.

[0297] Fig.17The relationship described by the known unique 4D coordinates of the virtual pixels in some embodiments of the method or system for rendering a 4D energy field from the 3D environment of the present disclosure is further illustrated by partially depicting a scene 1700 in a 3D environment. It should be understood that more virtual pixels, virtual apertures, and virtual viewing positions can be located in the scene 1700. In order to render data for a 4D energy field, some embodiments may include a very large number of virtual pixels and virtual viewing positions to provide an energy field that is perceived as seamless. The 2D angular coordinates of one of the virtual pixels 1702, 1704, 1706 describe the angular correlation between the pixel 1702, 1704, 1706 and the virtual viewing position L1, L2, L3. For example, the 2D angular coordinates (U1, V1) for the first virtual pixel 1702 describe the angular correlation between the pixel 1702 and the virtual viewing position L3. The 2D angular coordinates (U2, V1) for the second virtual pixel 1704 describe the angular correlation between the pixel 1704 and the virtual viewing position L2. Moreover, the 2D angular coordinates (U3, V1) for the third virtual pixel 1706 describe the angular correlation between the pixel 1706 and the virtual viewing position L1. In some embodiments, the angular coordinates may correspond to 2D angular coordinates in a 4D energy projection system that defines a unique direction for the first propagation path, as described elsewhere in this disclosure. Fig.16 and Fig.17 As with the embodiments depicted in , some embodiments require only one ray to intersect any one of a plurality of virtual pixels for a particular energy type.

[0298] Fig.17 Also depicted is a 2D spatial coordinate (X1, Y1) identifying the location of one of the plurality of virtual apertures 1708. The 2D spatial coordinate may correspond to a 2D spatial coordinate in a 4D light field coordinate of a physical 4D energy guiding system as disclosed elsewhere in this disclosure, such as described herein with respect to Figure 7C and 7D The 2D angular coordinate and the 2D spatial coordinate together define a unique 4D coordinate. The unique 4D coordinate may correspond to the 4D light field coordinate of the physical 4D energy projection system. Fig.17 In the depicted embodiment, the first virtual pixel 1702 has 4D coordinates (X1, Y1, U1, V1), the second virtual pixel 1704 has 4D coordinates (X1, Y1, U2, V1), and the third virtual pixel 1706 has 4D coordinates (X1, Y1, U3, V1). Fig.16 and Fig.17In some embodiments shown in , multiple rays can converge at each virtual aperture 1708 in multiple virtual apertures. The coordinates of the first virtual pixel 1702, the second virtual pixel 1704, and the third virtual pixel 1706 share the same spatial coordinates, so the rays 1710 of these virtual pixels converge at the same virtual aperture 1708. However, because the 2D angular coordinates are different between virtual pixels, these rays intersect different virtual viewing positions L1, L2, and L3.

[0299] In some embodiments of the systems and methods of the present disclosure, at least one ray 1710 of the plurality of rays intersects each virtual viewing location L1 , L2 , L3 of the plurality of virtual viewing locations.

[0300] In some embodiments, Fig.18 As depicted in FIG. 1 , multiple rays 1802 may also converge at a single viewing position L2, which further illustrates the relationship described by a known unique 4D coordinate. In some of those embodiments, only one of the rays 1802 that converges at one virtual viewing position L2 may intersect the same virtual aperture. Fig.18 , one of the converging rays 1802 intersects (X1, Y1), one intersects (X2, Y1), and one intersects (X3, Y1). Fig.17 and 18 It is for illustrative purposes only and does not limit the embodiments of the present disclosure. As will be appreciated, even if Fig.16 , Fig.17 and Fig.18 A small number of virtual pixels, virtual apertures, rays, and virtual viewing positions are depicted, but different embodiments may include tens, hundreds, thousands, or other numbers of these elements. It should be understood that virtual pixels can be located anywhere on the virtual plane. Moreover, virtual positions can be located anywhere on the virtual viewing plane. Moreover, each of these figures is a stylized representation that can vary depending on the embodiment. Moreover, the disclosure of the present application can be applied to any type of energy; it is not limited to electromagnetic energy in the visible spectrum.

[0301] return Fig.16 In an embodiment, plurality of rays 1616 extend beyond virtual pixel plane 1606. In some embodiments, plurality of rays 1616 may extend indefinitely beyond virtual pixel plane 1606. This allows energy data points to be sampled from objects in front 1626 and objects behind virtual pixel plane 1606, such as photograph 1628.

[0302] In some embodiments, energy data points 1602 along multiple rays 1616 are sampled by tracing multiple rays 1616 from the virtual viewing plane 1610 through the virtual display plane 1614, the virtual pixel plane 1606, and beyond. This allows the disclosed methods and systems to sample energy data points in a single pass that capture information from objects such as the table 1628A in front of the virtual display plane 1610 and objects such as the picture 1628B behind the virtual display plane 1610. This also allows the disclosed systems and methods to accurately account for multiple energy interactions, as described in more detail in other parts of this disclosure, that occur both outside and inside the field of view from the observation point in a single pass. This eliminates the need for difficult, cumbersome, and imperfect post-calculations to try to integrate multiple rendering passes.

[0303] Fig.19 A scene from a 3D environment is depicted from a virtual viewing position 1906, which further illustrates how embodiments of the disclosed systems and methods allow for multiple energy interactions to be considered in a single pass. Embodiments of the disclosed systems and methods allow for data points 1902 to be sampled along multiple rays 1904 from a virtual viewing position 1906. It should be appreciated that providing Fig.19 It is for illustrative purposes only and does not limit the embodiments of the present disclosure. For the sake of clarity, Fig.19 Virtual pixels or virtual apertures are not depicted. Moreover, the size of the viewing position 1906 may vary depending on the embodiment. For purposes of this illustration, the virtual viewing position 1906 has a field of view 1908. The size of the field of view may vary depending on the embodiment, and in some cases, may depend on the type of physical 4D energy projection system. Moreover, as will be appreciated, the size of the field of view 1908 may be determined or varied for different purposes. Sampling energy data points 1902 along multiple rays 1904 takes into account objects in the field of view 1908 (e.g., example objects 1910A, 1910B) as well as energy reflected by objects within the field of view 1908. In this example, energy from a person (not shown) outside the field of view 1908 reflects from a reflector 1910B in the field of view 1908, thereby producing a reflection 1912 in the reflector 1910B that is sampled by the ray 1904. Provided Fig.19 It is for illustrative purposes only and does not limit the embodiments of the present disclosure. Fig.19 A 3D light environment is depicted, but embodiments of the disclosed systems and methods are applicable to other sensory and energy domains.

[0304] Reference again Fig.15 In an embodiment, a process for rendering a 4D energy field from a 3D environment may include a fourth step 1540, which includes correlating energy data points sampled along each ray with an energy value of a virtual pixel in a plurality of virtual pixels.

[0305] This step 1540 allows embodiments of the systems and methods of the present disclosure to take into account the changing characteristics of objects in the ray path. For example, in some embodiments, the virtual pixel ( Fig.19 ) and the energy value along the virtual pixel ( Fig.19 1906). For example, ray 1916 intersects partially reflective object 1920 and partially transmissive object 1918 located between partially reflective object 1920 and virtual viewing position 1906. In order to render data for this scene 1900, the system or method of the present disclosure considers how partially transmissive object 1918 affects light reflected from partially reflective object 1920 via partially transmissive object 1918 to virtual viewing position 1906. For example, partially reflective object 1920 may have a spectral radiance function that, when illuminated by a light source (not shown), will distribute energy along different directions that will be attenuated by partially transmissive object 1918 before reaching virtual viewing position 1906. Embodiments may also allow energy values ​​of different data points to be weighted based on at least the transmission or reflection characteristics of other energy data points along the same ray. The systems and methods of the present disclosure allow virtual pixels ( Fig.19 1916) are related to energy data points along the length of ray 1916 to account for these types of varying characteristics. As will be appreciated, in different embodiments, various energy data points may be modeled to account for any number of different characteristics, some of which may be specifically tailored for different energy domains. Determining energy values ​​can be very difficult for multi-pass rendering systems because they must attempt to integrate all of the information necessary to model the multiple possible interactions of the renderers and the three-dimensional interface along each renderer to derive the energy values. The systems and methods of the present disclosure allow those cumbersome and inaccurate measures to be avoided.

[0306] The process for rendering a 4D energy field from a 3D environment may include a fifth step 1550, which includes rendering the energy value of a virtual pixel in a plurality of virtual pixels and the known unique 4D coordinates of a virtual pixel in a plurality of virtual pixels as a data set, the format of the data set being usable to indicate that the energy device outputs a 4D energy field. In some embodiments, the systems and methods of the present disclosure may allow the data set to include any data set described or referenced elsewhere in the present disclosure, including 4D, 5D, 10D, 11D, or 15D functions. It should be understood that other types of data sets may be rendered for different embodiments of a 3D environment that considers different types of data. In some embodiments, rendering may be calibrated for different types of 4D energy projection system data sets. Different types of calibration may be applied to render data for 4D energy guidance systems with different architectures, elements, or numbers of elements. For example, calibration may be required for systems with different fields of view or with different numbers or positions of energy sources or other components. In some embodiments of the systems and methods of the present disclosure, the energy data point 1902 includes a value describing at least one of the following: energy frequency, energy intensity, energy transparency, energy refractive index, energy reflectivity.

[0307] Some embodiments of the disclosed method for rendering a four-dimensional energy field from a three-dimensional environment may allow the method to be calibrated for an energy device that projects a 4D energy field. As will be appreciated, the location of the physical energy source and energy propagation path in a 4D projection system may be different from the theoretical location of the virtual pixel and the theoretical angular correlation between the virtual pixel and the virtual viewing position. This may be due to slight variations between energy devices, or even slight variations between the waveguides of the energy devices. For example, a slight imperfection in a waveguide may cause the energy propagation path to deviate slightly from its intended direction. Moreover, the energy position may be slightly different from the corresponding virtual position in the 3D environment, which may also result in a deviation in the direction of energy propagation. A method for calibration is disclosed in co-owned U.S. application US16064300, entitled "Method of Calibration for Holographic Energy Directing Systems," which is incorporated herein by reference.

[0308] Fig.21 is a schematic diagram illustrating inverse tracking in a three-dimensional environment. Fig.21 The method demonstrated in may include providing a scene 21000 in a 3D environment, the scene being described by a plurality of energy data points located at various locations in the scene. In an embodiment, providing the scene 21000 may be implemented according to the embodiments described in paragraphs

[0244] to

[0245] . Fig.21Also demonstrated is positioning a plurality of virtual pixels 21012 on a virtual pixel plane 21004 in a scene 21000, wherein each of the plurality of virtual pixels 21012 has a known unique 4D coordinate, the coordinate comprising a 2D angular coordinate and a 2D spatial coordinate, wherein the 2D angular coordinate of each of the plurality of virtual pixels 21012 describes an angular correlation between the virtual pixel and a virtual viewing position in a plurality of virtual viewing positions 21008 located on a virtual viewing plane 21006 in the scene 21000, and wherein the 2D spatial coordinate of each virtual pixel identifies a position of a virtual aperture in a plurality of virtual apertures 21010 located on a virtual display plane 21019 in the scene 21000. In an embodiment, positioning the plurality of virtual pixels 21012 may be implemented according to the embodiments described in paragraphs

[0245] to

[0247] .

[0309] Fig.21 2 further illustrates sampling energy data points from a plurality of energy data points in a scene 21000 along a plurality of rays 21020 from a virtual viewing plane 21006, wherein each ray in the plurality of rays 21020 intersects a viewing position in a plurality of virtual viewing positions 21008 and a virtual pixel in a plurality of virtual pixels 21012, the intersection angle being determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray. In an embodiment, sampling data points may be implemented according to the embodiments described in paragraphs

[0228] to

[0234] .

[0310] Fig.21 21020 and an energy value of one virtual pixel in the plurality of virtual pixels 21012 intersected by the ray. In an embodiment, correlating the sampled energy data points may be implemented according to the embodiments described in paragraphs

[0256] to

[0257] . The energy value of one virtual pixel of each ray and the known unique 4D coordinates of one virtual pixel of each ray may be rendered as a data set, the format of which may be used to indicate that the energy device outputs a 4D energy field. In an embodiment, rendering may be implemented according to the embodiments described in paragraphs

[0258] .

[0311] like Fig.21 , multiple rays 21012 may be traced backwards from the virtual viewing plane through the virtual pixel plane 21004 and beyond to capture information about energy data points behind the virtual pixel plane 21004. This allows the method for rendering a 4D energy field from a 3D environment to account for energy reflections behind the virtual pixel plane, even if the reflections are from distant objects 21013.

[0312] Fig.21 Also demonstrated is how embodiments of this method may take into account partially transmissive object 21015. As depicted, energy data is sampled via partially transmissive object 21015 passing through virtual pixel display 21004 where it will sample data reflected from object 21013. In some embodiments, the energy values ​​may then be correlated based on the sampled information by weighting the energy data 21014 of object 21013 and partially transmissive object 21015 to arrive at energy data values ​​for virtual pixels of the virtual pixel array.

[0313] It should be understood that in embodiments, Fig. 20 and 21 The inverse tracing and mapping process can allow a single processing step of inverse ray tracing to a desired 4D display specific stereo energy representation (e.g., image). This can include wavelength / energy considerations as well as display specific calibration (or any other functionality that may include seamless movement between various systems). If and when advantageous, this can additionally include pixel processing with displacement maps to calculate other 4D coordinates.

[0314] In some embodiments of the systems and methods of the present disclosure, a ray of the plurality of rays intersects each virtual pixel of the plurality of virtual pixels.

[0315] In some embodiments of the systems and methods of the present disclosure, the virtual display plane corresponds to a waveguide system of the energy guiding device, and energy is operable to be guided through the waveguide system according to the data set to form a detectable stereoscopic representation of at least a portion of the scene.

[0316] In some embodiments of the systems and methods of the present disclosure, the plurality of virtual pixels corresponds to a plurality of energy locations on the first side of the waveguide system.

[0317] In some embodiments of the systems and methods of the present disclosure, the data set further includes vectorized material property data.

[0318] In some embodiments of the systems and methods of the present disclosure, at least a portion of the method is performed in real time.

[0319] In some embodiments of the systems and methods of the present disclosure, the methods are performed entirely in real time.

[0320] In some embodiments of the systems and methods of the present disclosure, at least two parts of the method are performed in different time periods.

[0321] In some embodiments of the systems and methods of the present disclosure, at least one ray of the plurality of rays intersects each virtual aperture of the plurality of virtual apertures.

[0322] In some embodiments of the systems and methods of the present disclosure, the data set describes at least one signal perceptible by a visual, audio, texture, feel, or smell sensor.

[0323] In some embodiments of the systems and methods of the present disclosure, energy data points sampled along each ray of the plurality of rays are simultaneously associated with energy values.

[0324] In some embodiments of the systems and methods of the present disclosure, the data set is stored in a binary file format.

[0325] In some embodiments of the systems and methods of the present disclosure, an energy data point of the plurality of energy data points is sampled along each ray of the plurality of rays from the virtual viewing plane through a virtual aperture of the plurality of virtual apertures and beyond a virtual pixel of the plurality of virtual pixels.

[0326] In some embodiments of the systems and methods of the present disclosure, the steps may be repeated indefinitely. In some embodiments, this may allow the scene to be dynamically captured to change the scene.

[0327] In an embodiment, the disclosed method for rendering a 4D energy field from a 3D environment further includes storing information about a virtual viewing position and the 4D coordinates of the virtual viewing position into a ray file, which may provide instructions for sampling data from those virtual viewing positions along any rays that intersect the virtual viewing position. A ray file is actually just a mapping of samples rendered on a virtual viewing plane, and it may contain a list of 3D virtual viewing plane coordinates and associated 2D (u, v) angular coordinates. As an example, in some embodiments, a ray file may be used with CG scene information such as scene geometry, color, reflections, etc. to effectively sample a synthetic 4D light field. It should be understood that the information stored in the ray file may vary depending on the type of energy device that will be used to output the 4D energy field, as the number of pixels and the location of the allowable observation points may vary with the energy device. In some embodiments, a ray file may provide a means for instructing the system how to sample a 3D environment for a given energy device. In other embodiments, the ray file is an ideal mapping of a particular energy field projection device design and does not reflect relatively small device-to-device differences, requiring any sampling of the scene using the ray file to be subject to correction dependent on another device at a later time. Fig.17 and 18, but it should be understood that the methods described herein are applicable to embodiments with more virtual viewing positions, virtual pixels, virtual apertures, and rays. In an embodiment, the 3D spatial coordinates may identify the position of each virtual viewing position L2 and the 2D angular coordinates of one virtual pixel 1702, 1704, 1706 intersected by each ray 1710. In some embodiments, the ray file associates the 3D spatial coordinates of each virtual viewing position L2 with the 2D angular coordinates of each virtual pixel 1702, 1704, 1706 intersected by each ray intersecting the virtual viewing position. For example, Fig.18 L2 in the scene intersects with ray 1802. Moreover, the ray file may associate the angular coordinates of virtual pixels 1803, 1805, and 1807 L2 with the 3D spatial coordinates identifying the location of L2 in the scene. In an embodiment, the ray file provides instructions for each ray 1802 for sampling energy data points in a plurality of energy data points 1802 along the ray from one virtual viewing position L2 intersected by the ray, wherein the instructions are determined by at least the 3D spatial coordinates of the one virtual viewing position L2 intersected by the ray and the 2D angular coordinates of the one virtual pixel intersected by the ray. For example, the ray file may provide instructions for ray 1809 determined by at least the 3D spatial coordinates of L2 and the 2D angular coordinates of 1803. As will be appreciated, this method may be used for a variety of applications other than energy fields, including rendering data for 2D, stereoscopic, virtual reality, augmented reality, and multi-view display systems.

[0328] In some embodiments of the disclosed method for rendering a four-dimensional energy field from a three-dimensional environment, rendering energy values ​​may further include calibrating energy data for energy devices. As will be appreciated, the theoretical position of virtual pixels and the theoretical angular correlation between virtual pixels and virtual viewing positions may be different from physical implementations. This may be due to slight variations between energy devices, or even slight variations between waveguides of energy devices. For example, a slight defect in a waveguide may cause the energy propagation path to deviate slightly from its expected direction. Moreover, the energy position may be slightly different from the corresponding virtual position in the 3D environment, which may also result in slight deviations in the direction of energy propagation. In some embodiments, these deviations can be resolved by calibrating the energy device or the energy data of a particular type of energy device. It should be appreciated that other types of calibration may also be appropriate.

[0329] Fig.15The process described in can be performed using any processing system of the present disclosure or an appropriate system known in the art (including processing system 2200). In an embodiment, the processing system may include a rendering engine 2210 and a sensory data engine 2220. In some embodiments, the sensory data engine 2220 may perform step 1510 to provide a scene in a 3D environment. Moreover, the sensory data engine 2220 may also perform step 1520 to position multiple virtual pixels on a virtual pixel plane. The rendering engine 2210 of some embodiments may perform step 1530 to sample energy data points. The rendering engine 2230 may also perform step 1540 to correlate the energy data points sampled along each ray with energy values. Moreover, the rendering engine 2210 may also perform step 1550 to render energy values. In some embodiments, the system may also include a memory 2240 to store data sets.

[0330] It will be appreciated that the method for rendering a 4D energy field from a 3D environment may be applied to render energy data for other applications. Such a method may include a first step 2210 of providing a scene in a 3D environment, the scene being described by a plurality of energy data points located throughout the scene. A second step 2220 may include locating a plurality of virtual pixels on a virtual pixel plane in the scene, wherein each virtual pixel has a known unique 4D coordinate, the coordinates comprising a 2D angular coordinate and a 2D spatial coordinate, wherein the 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and a virtual viewing position of a plurality of virtual viewing positions located on a virtual viewing plane in the scene, and wherein the 2D spatial coordinate of each virtual pixel identifies a position of a virtual aperture of a plurality of virtual apertures located on a virtual display plane in the scene. A third step 2230 may include sampling energy data points from the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, wherein each ray intersects a virtual viewing position and a virtual pixel, the intersection angle being determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray. A fourth step may include correlating the energy data points sampled along each ray with an energy value of the one virtual pixel intersected by the ray, and a fifth step 2250 may include rendering the energy value of the one virtual pixel for each ray and the known unique 4D coordinates of the one virtual pixel for each ray as a data set, the format of the data set being usable to instruct an energy device to output energy data.

[0331] It should be appreciated that some forms of energy data may have fewer virtual viewing pixels and virtual viewing positions required for rendering a 4D energy field. For example, in some embodiments, each virtual aperture intersects two rays. In such embodiments, the multiple virtual viewing positions include two virtual viewing positions. However, such embodiments may be used to generate a data set that can be used to instruct an energy device to output a stereoscopic image. It should be appreciated that a stereoscopic display directs a first image to a viewer's first eye and a second image to a second eye. Therefore, in some embodiments, each aperture may include only two virtual pixels, and there may be only two virtual viewing positions - one for each eye. Such embodiments may allow the format of the data set to be used to instruct an energy device to output a virtual reality image. Moreover, such embodiments may allow the format of the data set to be used to instruct an energy device to output an augmented reality image.

[0332] It should also be appreciated that in other embodiments, each virtual aperture may be intersected with additional rays corresponding to additional viewing positions or views. Such embodiments may allow the format of the data set to be used for multi-view display, where the number of views corresponds to the number of rays intersecting each virtual pixel. This principle may be explained by Fig.17 , the diagram shows three viewing positions L1, L2, L3 and three rays 1710 intersecting an aperture X1, Y1, wherein each ray intersects a virtual pixel 1702, 1704, and 1706. Each viewing position may correspond to a view of the multi-view system. It should be appreciated that some embodiments may include many additional viewing positions.

[0333] Although various embodiments according to the principles disclosed herein have been described above, it should be understood that they are presented by way of example only and are not limiting. Therefore, the breadth and scope of the present disclosure should not be limited by any of the exemplary embodiments described above, but should only be limited according to the claims issued by the present disclosure and their equivalents. In addition, the above-mentioned advantages and features are provided in the described embodiments, but the application of such issued claims should not be limited to the process and structure that achieves any or all of the above-mentioned advantages.

[0334] It should be understood that the main features of the present disclosure may be employed in various embodiments without departing from the scope of the present disclosure. Those skilled in the art will recognize or be able to determine many equivalents to the specific procedures described herein using no more than routine experimentation. Such equivalents are considered to be within the scope of the present disclosure and are covered by the claims.

[0335] In addition, the section titles herein are provided to be consistent with the recommendations of 37CFR 1.77, or to provide organizational cues in addition. These titles should not limit or characterize the subject matter set forth in any claims that may be issued from this disclosure. Specifically and as an example, although the title mentions "technical field", such claims should not be limited by the language describing the so-called technical field under this title. In addition, the description of the technology in the "background technology" section should not be understood as admitting that the technology is the prior art of any invention in this disclosure. "Summary of the invention" should also not be considered as a feature of the invention set forth in the issued claims. In addition, any reference to "invention" in the singular form in this disclosure should not be used to demonstrate that there is only a single novel point in this disclosure. Multiple inventions can be set forth according to the limitations of multiple claims issued from this disclosure, and such claims accordingly define the inventions and their equivalents protected thereby. In all cases, the scope of such claims should be considered on their own merits in view of this disclosure, and should not be constrained by the titles set forth herein.

[0336] The word "a" is used to refer to "one" when used in conjunction with the term "comprising" in the claims and / or the specification, and it also complies with the meaning of "one or more", "at least one" and "one or more than one". Unless it is clearly indicated that only alternatives are referred to or the alternatives are mutually exclusive, the term "or" is used in the claims to refer to "and / or", but the present disclosure supports definitions and "and / or" that only refer to alternatives. Throughout this application, the term "about" is used to indicate that a value includes the inherent variation of the error of the device or method used to determine the value, or the variation that exists between research subjects. In general but consistent with the above discussion, the numerical values ​​modified by approximate words such as "about" herein can vary at least ±1, 2, 3, 4, 5, 6, 7, 10, 12 or 15% around the stated value.

[0337] As used in this specification and claims, the words "comprise" (and any form of comprising, such as "comprises" and "comprises"), "having" (and any form of having, such as "have" and "has"), "including" (and any form of including, such as "includes" and "include"), or "contain" (and any form of containing, such as "contains" and "contain") are inclusive or open-ended, and do not exclude additional unrecited elements or method steps.

[0338] Comparison, measurement, and timing terms such as "at," "equivalent to," "during," "completely," etc., should be understood to mean "substantially at," "substantially equivalent to," "substantially during," "substantially completely," etc., where "substantially" means that such comparisons, measurements, and timing can be used to achieve the desired results implicitly or explicitly stated. Words such as "near," "close to," and "adjacent" related to the relative position of elements should mean close enough to have a substantial effect on the interaction of the corresponding system elements. Other approximate terms similarly refer to certain conditions that, when so modified, are understood not to be necessarily absolute or perfect but would be considered close enough to warrant specifying the conditions as presented by one skilled in the art. The degree to which the description can vary will depend on how large a variation can be made and still allow one of ordinary skill in the art to recognize the modified feature as still having the desired properties and capabilities of the unmodified feature.

[0339] As used herein, the term "or a combination thereof" refers to all permutations and combinations of the listed items preceding the term. For example, "A, B, C, or a combination thereof" is intended to include at least one of: A, B, C, AB, AC, BC, or ABC, and if order is important in a particular case, BA, CA, CB, CBA, BCA, ACB, BAC, or CAB. Continuing with this example, combinations containing repetitions of one or more items or clauses are expressly included, such as BB, AAA, AB, BBC, AAABCCCC, CBBAAA, CABABB, etc. Those skilled in the art will appreciate that, unless otherwise apparent from the context, there is generally no limit to the number of items or clauses in any combination.

[0340] All compositions and / or methods disclosed and claimed herein can be made and performed without undue experimentation in light of the present disclosure. Although the compositions and methods of the present disclosure have been described in terms of preferred embodiments, it will be apparent to those skilled in the art that changes may be made to the compositions and / or methods and in the steps or sequence of steps of the methods described herein without departing from the concept, spirit and scope of the present disclosure. All such similar substitutions and modifications apparent to those skilled in the art are considered to be within the spirit, scope and concept of the present disclosure as defined by the appended claims.

Claims

1. A method for rendering a four-dimensional (4D) energy field from a three-dimensional (3D) environment, the method comprising: Providing a scene in a 3D environment, the scene being described by a plurality of energy data points located throughout the scene; positioning a plurality of virtual pixels on a virtual pixel plane in the scene, wherein each virtual pixel has a known unique 4D coordinate comprising a 2D angular coordinate and a 2D spatial coordinate, wherein the 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and a virtual viewing position of a plurality of virtual viewing positions located on a virtual viewing plane in the scene, and wherein the 2D spatial coordinate of each virtual pixel identifies a position of a virtual aperture of a plurality of virtual apertures located on a virtual display plane in the scene; sampling energy data points from the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, wherein each ray intersects a virtual viewing position and a virtual pixel at an intersection angle determined by the 2D angular coordinates of the one virtual pixel intersected by the ray, and wherein each ray intersects a virtual aperture determined by the 2D spatial coordinates of the one virtual pixel intersected by the ray; Correlating the energy data points sampled along each ray with an energy value of the one virtual pixel intersecting the ray; as well as The energy value of the one virtual pixel of each ray and the known unique 4D coordinate of the one virtual pixel of each ray are rendered as a data set, and the format of the data set can be used to instruct the energy device to output a 4D energy field. 2 . The method of claim 1 , wherein at least one of the plurality of rays intersects each of the plurality of virtual viewing locations. 3 . The method of claim 1 , wherein a ray of the plurality of rays intersects each virtual pixel of the plurality of virtual pixels.

4. The method of claim 1, wherein the 4D energy field comprises a light field.

5. The method of claim 1, wherein the 4D energy field comprises a tactile field.

6. The method of claim 1, wherein the 4D energy field comprises a tactile field.

7. The method of claim 1, wherein the energy data points include values ​​describing at least one of: energy frequency, energy intensity, energy transparency, energy refractive index, energy reflectivity.

8. The method of claim 1, wherein the 3D environment is determined by applying a depth map to points in a two-dimensional space.

9. The method of claim 1 , wherein the virtual display plane corresponds to a waveguide system of an energy guiding device, and energy can be directed through the waveguide system based on the data set to form a detectable 4D energy representation of at least a portion of the scene.

10. The method of claim 9, wherein the plurality of virtual pixels correspond to a plurality of energy locations on a first side of the waveguide system.

11. The method of claim 1, wherein the data set further comprises vectorized material property data.

Images