System and method for rendering data from 3D environment
A bi-directional seamless energy surface system using energy waveguide arrays and transverse Anderson localized relays addresses the challenge of creating a Holodeck-like immersive experience, effectively stimulating multiple sensory receptors, overcoming resolution and manufacturing limitations.
Patent Information
- Application Number
- JP2024106572
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2018-01-14
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Current technologies have not successfully created a seamless energy surface capable of stimulating all human sensory receptors within a virtual environment, lacking the ability to produce a sufficiently immersive and interactive 4D energy field experience akin to the Holodeck concept.
The development of a bi-directional seamless energy surface system using energy waveguide arrays and transverse Anderson localized energy relays, combined with 4D plenoptic functions, to propagate energy waves that stimulate visual, auditory, tactile, and other sensory receptors, overcoming limitations in resolution and manufacturing constraints.
This system enables a highly immersive and interactive 4D energy field experience, effectively stimulating multiple human sensory receptors without the need for external accessories, providing a realistic and interactive virtual environment.
Smart Images

Figure 2024156664000005 
Figure 2024156664000006 
Figure 2024156664000007
Abstract
Description
[Technical field]
[0001] This disclosure generally relates to rendering a three-dimensional environment into a data set and instructing an energy field projector to output a four-dimensional energy field. [Background technology]
[0002] Popularized by Gene Roddenberry's Star Trek and originally envisioned by author Alexander Moszkowski in the early 1900s, the dream of an interactive virtual world inside a "Holodeck" room has been an inspiration for science fiction and technological innovation for nearly a century. However, groundbreaking realizations of this experience have never existed outside of the literature, media, and collective imaginations of children and adults alike. This application teaches systems and methods for rendering information from a 3D environment into a format that allows a 4D energy field projection system to output a modeled 4D energy field into a scene from the 3D environment. Summary of the Invention
[0003] In one embodiment, a method for rendering a four-dimensional (4D) energy field from a three-dimensional (3D) environment includes providing a scene of the 3D environment described by a plurality of energy data points disposed throughout the scene, and disposing a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having known unique 4D coordinates including a 2D angular coordinate and a 2D spatial coordinate. In this embodiment, the 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and one of a plurality of virtual viewpoints disposed in a virtual viewing plane in the scene, and the 2D spatial coordinate of each virtual pixel identifies a position of one of a plurality of virtual apertures disposed in a virtual display surface in the scene. Next, sample energy data points of a plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, where each ray intersects a virtual viewpoint and a virtual pixel at an angle determined by the 2D angular coordinate of a virtual pixel intersecting the ray, and where each ray intersects a virtual aperture determined by the 2D spatial coordinate of a virtual pixel intersecting the ray. In this embodiment, the method includes correlating the energy data points sampled along each ray with an energy value for a virtual pixel intersected by the ray, and rendering the energy value of the virtual pixel for each ray and the known unique 4D coordinate of the virtual pixel for each ray into a data set having a format operable to instruct an energy device to output a 4D energy field.
[0004] In one embodiment, at least one ray of the plurality of light rays intersects with each virtual viewpoint of the plurality of virtual viewpoints. In one embodiment, one ray of the plurality of light rays intersects with each virtual pixel of the plurality of virtual pixels. In some embodiments, the 4D energy field comprises a bright field, a haptic field, or a tactile field.
[0005] In one embodiment, the energy data points include values describing at least one of energy frequency, energy intensity, energy transparency, energy refraction, or energy reflectivity. In one embodiment, the 3D environment can be determined by applying a depth map to points in two-dimensional space. In another embodiment, the virtual display plane corresponds to a waveguide system of the energy directing device, and energy is operable to be directed through the waveguide system according to the data set to form a detectable 4D energy representation of at least a portion of the scene.
[0006] In one embodiment, the plurality of virtual pixels corresponds to a plurality of energy locations on the first side of the waveguide system. In another embodiment, the data set further comprises vectorized material property data. During operation, at least a portion of the method can be performed in real time. In another technique, the entire method can be performed in real time. In another embodiment, at least two portions of the method are performed in different time periods.
[0007] In some embodiments, the dataset describes signals perceptible by visual, audio, texture, sensational, or odor sensors. In one embodiment, the energy data points sampled along each ray of the multiple rays are simultaneously correlated with the energy value. In some cases, the dataset is stored in a binary file format.
[0008] In one embodiment, each ray of the multiple rays extends through one virtual aperture of the multiple virtual apertures to and beyond one virtual pixel of the multiple virtual pixels, and an energy data point of the multiple energy data points is sampled from the virtual viewing plane.
[0009] In one embodiment, the above steps may be repeated indefinitely. In another embodiment, the above steps may be repeated indefinitely to render a dynamic 4D energy field from a 3D environment.
[0010] In one embodiment, rendering the energy data further includes calibrating the energy data for the energy device. In one embodiment, the ray file stores 3D spatial coordinates identifying the location of each virtual viewpoint and 2D angular coordinates of one virtual pixel intersected by every ray. In another embodiment, the ray file associates the 3D spatial coordinates of each virtual viewpoint with the 2D angular coordinates of all virtual pixels intersected by every ray intersecting the virtual viewpoint. In yet another embodiment, the ray file provides instructions for each ray to sample energy data points of a plurality of energy data points along a ray from one virtual viewpoint intersecting the ray, the instructions being determined by at least the 3D spatial coordinates of one virtual viewpoint intersecting the ray and the 2D angular coordinates of one virtual pixel intersecting the ray.
[0011] In one embodiment, a system for rendering a four-dimensional (4D) energy field from a dynamic three-dimensional (3D) environment includes a process subsystem having a sensory data engine and a rendering engine, where the sensory data engine provides a scene of the 3D environment described by a plurality of energy data points disposed throughout the scene. In this embodiment, the sensory data engine disposes a plurality of virtual pixels in a virtual pixel plane of the scene, each virtual pixel having a known unique 4D coordinate including a 2D angular coordinate and a 2D spatial coordinate. The 2D angle of each virtual pixel coordinate describes an angular correlation between the virtual pixel and a virtual viewpoint of a plurality of virtual viewpoints disposed in a virtual viewing plane in the scene by the sensory data engine, while the 2D spatial coordinate of each virtual pixel identifies a position of one of a plurality of virtual apertures disposed in a virtual viewing plane in the scene by the sensory data engine.
[0012] In this embodiment, the rendering engine samples energy data points of a plurality of energy data points in the scene along a plurality of rays from a virtual viewing plane, where each ray intersects a virtual viewpoint and a virtual pixel at an angle determined by the 2D angular coordinates of a virtual pixel where the ray intersects, and where each ray intersects a virtual aperture determined by the 2D spatial coordinates of a virtual pixel where the ray intersects.
[0013] Further in this embodiment, the rendering engine correlates the energy data points sampled along each ray with an energy value for a virtual pixel of the plurality of virtual pixels, and the rendering engine renders the energy value of the virtual pixel of the plurality of virtual pixels and the known unique 4D coordinates of the virtual pixel of the plurality of virtual pixels into a dataset having a format operable to instruct the energy device to output a 4D energy field.
[0014] In one embodiment, at least one ray of the plurality of light rays intersects with each virtual viewpoint of the plurality of virtual viewpoints. In another embodiment, one ray of the plurality of light rays intersects with each virtual pixel of the plurality of virtual pixels. In some embodiments, the 4D energy field can be a bright field, a haptic field, or a tactile field.
[0015] In one embodiment, the energy data points include values describing at least one of: energy frequency, energy intensity, energy transparency, energy refraction, or energy reflectivity. During operation, the 3D environment is determined by applying a depth map to points in the two-dimensional space.
[0016] In one embodiment, the virtual display plane corresponds to a waveguide system of the energy directing device, and energy is directed through the waveguide system according to the data set to form a detectable 4D energy field representation of at least a portion of the scene. In another embodiment, the plurality of virtual pixels corresponds to a plurality of energy locations on a first side of the waveguide system. In another embodiment, the data set further includes vectorized material property data.
[0017] During operation, at least a portion of the system executes in real time. In another operation, the system executes completely in real time. In another embodiment, at least two portions of the system execute at different time periods.
[0018] In one embodiment, the data set describes signals perceptible by visual, audio, texture, sensational, or odor sensors. In another embodiment, energy data points sampled along each ray of a plurality of rays are simultaneously correlated with an energy value. In yet another embodiment, the data set is stored in a binary file format.
[0019] In one embodiment, an energy data point of the plurality of energy data points is sampled by the rendering engine along each ray of the plurality of rays, the ray extending from a virtual viewing plane through a virtual aperture of the plurality of virtual apertures to and beyond a virtual pixel of the plurality of virtual pixels. In another embodiment, the system may be operated indefinitely to render a dynamic 4D energy field from a dynamic 3D environment. In yet another embodiment, the system further includes a memory to store the data set. In one embodiment, the rendering system calibrates energy data for an energy device. In another embodiment, the system further includes a memory to store the plurality of energy data points.
[0020] In one embodiment, the ray file stores 3D spatial coordinates identifying the location of each virtual viewpoint and the 2D angular coordinates of one virtual pixel intersected by every ray. In another embodiment, the ray file associates the 3D spatial coordinates of each virtual viewpoint with the 2D angular coordinates of all virtual pixels intersected by all rays that intersect the virtual viewpoint. In yet another embodiment, the ray file provides instructions for each ray to sample energy data points of a plurality of energy data points along a ray from one virtual viewpoint that intersects the ray, the instructions being determined by at least the 3D spatial coordinates of one virtual viewpoint that intersects the ray and the 2D angular coordinates of one virtual pixel that intersects the ray.
[0021] In one embodiment, a method for rendering energy data from a three-dimensional (3D) environment includes providing a scene of the 3D environment described by a plurality of energy data points located throughout the scene, and arranging a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having a known unique 4D coordinate including a 2D angular coordinate and a 2D spatial coordinate. The 2D angular coordinate of each virtual pixel describes an angular correlation between the virtual pixel and one of a plurality of virtual viewpoints located in the virtual viewing plane of the scene, while the 2D spatial coordinate of each virtual pixel identifies a position of one of a plurality of virtual apertures located in a virtual display surface in the scene. The next step of the method is sampling the energy data points of the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, each ray intersecting one virtual viewpoint and one virtual pixel at an angle determined by the 2D angular coordinate of one virtual pixel intersecting the ray, and each ray intersecting one virtual aperture determined by the 2D spatial coordinate of one virtual pixel intersecting the ray. In one method, the next step includes correlating energy data points sampled along each ray with energy values for a virtual pixel intersected by the ray, and rendering the energy value of the virtual pixel for each ray and the known unique 4D coordinates of the virtual pixel for each ray into a dataset having a format operable to instruct an energy device to output the energy data.
[0022] In one embodiment, each virtual aperture intersects two rays. In another embodiment, the plurality of virtual viewpoints includes two virtual viewpoints. In some embodiments, the dataset has a format operable to instruct the energy device to output a stereoscopic image, the dataset has a format operable to instruct the energy device to output a virtual reality image, or the dataset has a format operable to instruct the energy device to output an augmented reality image.
[0023] In one embodiment, each virtual aperture intersects several rays, hi another embodiment, the data set has a format operable to instruct the energy device to output images from multiple views, the multiple views corresponding to the number of rays intersecting each virtual aperture. [Brief description of the drawings]
[0024] [Figure 1] FIG. 1 is a schematic diagram illustrating design parameters of an energy directing system. [Diagram 2] FIG. 1 is a schematic diagram illustrating an energy system having an active device area with a mechanical envelope. [Diagram 3] FIG. 1 is a schematic diagram showing an energy relay system. [Figure 4] FIG. 1 is a schematic diagram illustrating one embodiment of energy relay elements bonded together and secured to a base structure. [Figure 5A] FIG. 1 is a schematic diagram illustrating an example of an image relayed through a multicore optical fiber. [Figure 5B] FIGURE 14 is a schematic diagram showing an example of an image relayed via an energy relay illustrating the properties of the transverse Anderson localization principle. [Figure 6] FIG. 2 is a schematic diagram showing light rays propagating from an energy surface to a viewer. [Figure 7A] 1 illustrates a perspective view of an energy waveguide system having a base structure, four energy devices, and four energy relay elements forming a seamless energy surface according to one embodiment of the present disclosure. [Figure 7B] 1 illustrates an energy relay system according to one embodiment of the present disclosure. [Figure 7C] FIG. 1 illustrates a top-down perspective view of an embodiment of an energy waveguide system, according to an embodiment of the present disclosure. [Figure 7D] FIG. 7D is a front perspective view of the embodiment shown in FIG. 7C. [Figure 7E] 7E-7L show various embodiments of energy suppression elements. [Figure 7F] Same as above. [Figure 7G] Same as above. [Figure 7H] Same as above. [Figure 7I] Same as above. [Figure 7J] Same as above. [Figure 7K] Same as above. [Figure 7L] Same as above. [Figure 8] 1 is a flow chart illustrating an embodiment of a process for processing holographic sensory data. [Figure 9] FIG. 1 is a schematic diagram of a virtual environment constructed from sensory data. [Figure 10] FIG. 1 is a schematic diagram illustrating one embodiment of energy tracking. [Figure 11] FIG. 10 is a schematic diagram of an embodiment of an energy directing device 1000 during a tracking process. [Figure 12] FIG. 2 is a schematic diagram of a processing system for processing holographic sensory data. [Figure 13] FIG. 2 is a block diagram illustrating one embodiment of a vectorization process. [Figure 14] 1 illustrates several aspects of modeling the energy of a 3D environment. [Figure 15] 1 is a flow chart illustrating one embodiment of a process for rendering a 4D energy field from a 3D environment. [Figure 16] FIG. 16A shows a visual representation of the 3D environment. [Figure 17] It represents one perspective of a virtual pixel, a virtual aperture, and a virtual viewpoint from a 3D environment. [Figure 18] It represents one perspective of a virtual pixel, a virtual aperture, and a virtual viewpoint from a 3D environment. [Figure 19] FIG. 19A depicts a visual representation of a 3D environment from the perspective of a virtual viewpoint. [Figure 20] 1 is a flow chart illustrating an embodiment of a process for rendering energy data from a 3D environment. [Figure 21]FIG. 2 is a schematic diagram of reverse tracing through various objects in a digital volumetric representation. [Figure 22] FIG. 1 is a schematic diagram illustrating one embodiment of a system for rendering a 4D energy field from a 3D environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] One embodiment of the Holodeck (collectively referred to as the "Holodeck Design Parameters") provides sufficient energy stimulation to trick human sensory receptors into believing that energy impulses received within a virtual, social, and interactive environment are real, providing 1) binocular parallax without external accessories, head-mounted eyewear, or other peripherals; 2) precise motion parallax, occlusion, and opacity throughout the entire viewing volume for any number of viewers simultaneously; 3) visual focusing via synchronous convergence, eye accommodation, and miosis for all perceived light rays; and 4) focused energy wave propagation of sufficient density and resolution to exceed human sensory "resolution" for vision, hearing, touch, taste, smell, and / or balance.
[0026] Based on current prior art, we are decades, if not centuries, away from technology that can address all receptive fields in a groundbreaking way as suggested by the Holodeck Design Parameters, which include visual, auditory, somatosensory, gustatory, olfactory, and vestibular systems.
[0027] In this disclosure, the terms bright field and holographic may be used interchangeably to define energy propagation for the stimulation of any sensory receptor response. Although the initial disclosure may refer to examples of energy propagation through energy planes for holographic images and volumetric haptics, sensory receptors of all forms are contemplated within this disclosure. Furthermore, the principles disclosed herein for energy propagation along a propagation path may be applicable to both energy emission and energy capture.
[0028] Today, many technologies exist that are unfortunately often confused with holograms, including lenticular printing, Pepper's Ghost, autostereoscopic displays, horizontal parallax displays, head-mounted VR and AR displays (HMDs), and other such illusions generalized as "pseudo-holograms." While these technologies may exhibit some of the desired characteristics of true holographic displays, they lack the ability to stimulate the human visual sensory response in any way sufficient to address at least two of the four identified Holodeck design parameters.
[0029] These challenges are that no conventional technique has been successfully implemented to generate a sufficiently seamless energy surface for Holodeck energy propagation. However, while there are various approaches to implementing volumetric and directionally multiplexed light-field displays, including parallax barriers, hogels, voxels, diffractive optical elements, multi-view projection, holographic diffusers, rotating mirrors, multi-layer displays, time-sequential displays, head-mounted displays, etc., conventional approaches may require compromises regarding image quality, resolution, angular sampling density, size, cost, safety, frame rate, etc., and may ultimately render the technology unfeasible.
[0030] To arrive at the Holodeck design parameters for the visual, auditory and somatosensory systems, the human acuity of each of the systems is studied and understood to propagate energy waves sufficiently to fool the human sensory receptors. The visual system can resolve up to about one arc minute, the auditory system may distinguish placement differences of only 3 degrees, and the somatosensory system of the hand can discern points 2-12 mm apart. There are various conflicting ways to measure these acuities, but these values are sufficient to understand systems and methods for stimulating the perception of energy propagation.
[0031] Of the well-known sensory receptors, the human visual system is by far the most sensitive, with even single photons capable of inducing sensations. For this reason, much of this introduction focuses on visual energy wave propagation, and the extremely low resolution energy systems coupled into the disclosed energy waveguide surfaces can focus appropriate signals to induce holographic sensory perceptions. Unless otherwise noted, all disclosures apply to all energy and sensory domains.
[0032] When calculating the effective design parameters of energy propagation for a visual system given a viewing volume and viewing distance, the desired energy surface may be designed to include many gigapixels of effective energy location density. For large viewing volumes, or near-field viewing, the design parameters of the desired energy surface may include effective energy location densities of hundreds of gigapixels or more. In comparison, the desired energy source may be designed to have an energy location density of 1-250 effective gigapixels for ultrasonic propagation of volumetric haptics, or an array of 36-3,600 effective energy locations for acoustic propagation of holographic acoustics, depending on the input environmental variables. What is important to note is that using the disclosed bidirectional energy surface architecture, all components may be configured to form the appropriate structure for any energy region and enable holographic propagation.
[0033] However, today, the main challenge to enabling the Holodeck lies within the limitations of available visual technology and energy devices. Acoustic and ultrasonic devices are less challenging given the difference in the magnitude of the desired density based on sensory acuity in the respective receptive fields, but the complexity should not be underestimated. While holographic emulsions exist with resolutions that exceed the desired density and encode interference patterns within still images, state-of-the-art display devices are constrained by resolution, data throughput, and manufacturing feasibility. To date, even extraordinary display devices have not been able to meaningfully generate bright fields with holographic resolutions that come close to visual acuity.
[0034] Fabrication of a single silicon-based device capable of meeting the desired resolution for a highly realistic light field display would be impractical and would involve extremely complex manufacturing processes beyond current manufacturing capabilities. Existing constraints on tiling multiple display devices together involve seams and gaps created by the physical size of the packaging, electronics, housings, optics, and many other challenges that would inevitably result in an infeasible technology from an imaging, cost, and / or size standpoint.
[0035] The embodiments disclosed herein may provide a real-world path to building a Holodeck.
[0036] Example embodiments will now be described hereinafter with reference to the accompanying drawings, which form a part hereof and which illustrate example embodiments that may be practiced. As used in this disclosure and the appended claims, the terms "embodiment," "example embodiment," and "exemplary embodiment" do not necessarily refer to a single embodiment, but they may be a single embodiment, and various example embodiments may be readily combined and used interchangeably without departing from the scope or spirit of the example embodiments. Furthermore, the terminology used herein is for the purpose of describing the example embodiments only and is not intended to be limiting. In this regard, as used herein, the term "in" can include "in" and "on," and the terms "a," "an," and "the" can include singular and plural. Furthermore, as used herein, the term "by" can also mean "from," depending on the context. Furthermore, as used herein, the term "if" can also mean "when" or "on," depending on the context. Furthermore, as used herein, the word "and / or" may refer to and include any and all possible combinations of one or more of the associated listed items.
[0037] Holographic system Brightfield Energy Propagation Resolution Overview Light field and holographic displays are the result of multiple projections where the energy surface location provides angle, color, and intensity information propagated within the viewing volume. The disclosed energy surfaces provide the opportunity for additional information to coexist and propagate through the same surface, eliciting other sensory system responses. Unlike volumetric displays, the viewed location of the converged energy propagation paths in space does not change as the viewer moves around the viewing volume, and multiple viewers may simultaneously observe the propagated objects in real-world space as if the objects were truly present there. In some embodiments, the energy propagation may be located within the same energy propagation path, but in opposite directions. For example, energy emission and energy capture along the energy propagation path are both possible in some embodiments of the present disclosure.
[0038] 1 is a schematic diagram illustrating variables related to the stimulation of a sensory receptor response. These variables include surface diagonal 101, surface width 102, surface height 103, defined target seat distance 118, target seat field of view relative to the field of view from the center of the display 104, number of intermediate samples demonstrated here as interocular samples 105, average adult interocular separation 106, average resolution per angular minute of the human eye 107, horizontal field of view formed between the target viewer position and the surface width 108, vertical field of view formed between the target viewer position and the surface height 109, resulting horizontal waveguide element resolution, or total number of elements across the surface 110, The resulting vertical waveguide element resolution, or total number of elements across the surface 111, may include a sample distance 112 based on the interocular spacing between the eyes and the intermediate sample number for the angular projection between the eyes, the angular sampling may be based on the sample distance and the target seat distance 113, the total resolution Horizontal per waveguide element may be derived from the desired angular sampling 114, the total resolution Vertical per waveguide element may be derived from the desired angular sampling 115, the Device Horizontal is a count of a determined number of desired discreet energy sources 116, and the Device Vertical is a count of a determined number of desired discreet energy sources 117.
[0039] A method for understanding the desired minimum resolution may be based on the following criteria to ensure sufficient stimulation of visual (or other) sensory receptor responses: surface size (e.g., 84 inch diagonal), surface aspect ratio (e.g., 16:9), seating distance (e.g., 128 inch distance from the display), seating field of view (e.g., 120 degrees or + / - 60 degrees relative to the center of the display), desired intermediate samples at a distance (e.g., one additional propagation path between the eyes), average separation between the lenses of an adult (about 65 mm), and average resolution of the human eye (about 1 arc minute). These example values should be considered placeholders depending on the specific application design parameters.
[0040] Additionally, each of the values attributed to the visual sensory receptors may be substituted for other systems to determine the desired propagation path parameters. For other energy propagation embodiments, the angular sensitivity of the auditory system may be considered to be as low as 3 degrees, and the spatial resolution of the somatosensory system of the hand may be considered to be as low as 2-12 mm.
[0041] Although there are various and conflicting ways to measure these perception acuity, these values are sufficient to understand systems and methods that stimulate the perception of virtual energy propagation. While there are many ways to consider design resolution, the following proposed system of principles combines practical product considerations with the biological resolution limits of the sensory system. As will be appreciated by those skilled in the art, the following outline is a simplification of any such system design and should be considered for illustrative purposes only.
[0042] Once the resolution limits of the sensory system are understood, the total energy waveguide element density can be calculated such that the receiving sensory system cannot distinguish between a single energy waveguide element and an adjacent element, given:
number
[0043] The above calculations result in a field of view of approximately 32×18°, resulting in a desired energy waveguide element of approximately 1920×1080 (rounded to nearest format). Variables can also be constrained to provide a more regular spatial sampling of energy locations (e.g. pixel aspect ratio) such that the field of view is compatible for both (u,v). Assuming the angular sampling of the system is a target viewing volume location defined between two points at the optimized distance, and additional propagating energy paths, it is given by:
number
[0044] In this case, the interocular distance is used to calculate the sample distance, although any scale may be used to account for the appropriate number of samples for a given distance. Considering the above variables, approximately one ray per 0.57° is desired, and the overall system resolution per separate sensory system can be calculated and is given as follows:
number
[0045] Using the above scenario, given the energy surface size and angular resolution addressed for the vision system, the resulting energy surface may desirably include an energy resolution location of approximately 400k x 225k pixels, or a holographic propagation density of 90 gigapixels. These given variables are for illustrative purposes only, and many other sensory and energy metric considerations should be considered for optimizing the holographic propagation of energy. In additional embodiments, a 1 gigapixel energy resolution location may be sought based on the input variables. In additional embodiments, a 1,000 gigapixel energy resolution location may be desired based on the input variables.
[0046] Limitations of current technology Active area, device electronics, packaging, and mechanical envelope FIG. 2 shows a device 200 having an active area 220 with a certain mechanical form factor. The device 200 may include a driver 230 and electronics 240 for power supply, connected to the active area 220, which has dimensions as indicated by the x and y arrows. This device 200 does not take into account cabling and mechanical structures for driving power and cooling components, and furthermore, the mechanical footprint may be minimized by introducing flex cables into the device 200. Also, the minimum footprint for such a device 200 may be referred to as a mechanical envelope 210, with dimensions as indicated by the M:x and M:y arrows. This device 200 is for illustrative purposes only, and application specific electronics design may further reduce the mechanical envelope overhead, but in almost all cases may not be the exact size of the active area of the device. In one embodiment, this device 200 illustrates the dependency of the electronics circuitry associated with the active image area 220 on a micro OLED, DLP chip, or LCD panel, or any other technology having the purpose of image illumination.
[0047] In some embodiments, it may also be possible to consider other projection technologies to aggregate multiple images across a larger display. However, this may result in costs due to greater complications with respect to throw distance, minimum focus, optical quality, uniform field resolution, chromatic aberration, thermal properties, calibration, alignment, additional size, or form factor. For most practical applications, hosting tens or hundreds of these projection sources 200 may result in less reliable and more extensive designs.
[0048] For illustrative purposes only, assuming an energy device having an energy location density of 3840 x 2160 sites, the number of individual energy devices (e.g., device 100) desired for the energy surface can be calculated and is given as follows:
number
[0049] Given the above resolution considerations, approximately 105 x 105 devices similar to the energy device shown in Figure 2 may be desired. Note that many devices are composed of various pixel structures that may or may not be mapped to a regular grid. If there are additional sub-pixels or positions within each complete pixel, these may be exploited to generate additional resolution or angular density. Additional signal processing may be used to determine how to convert the bright field to the correct (u,v) coordinates according to the specified position of the pixel structure(s), which may be a known, calibrated, explicit characteristic of each device. Furthermore, other energy regions may require different handling of these ratios and device structures, and one skilled in the art will understand the direct inherent relationship between each of the desired frequency regions. This will be shown and discussed in more detail in the following disclosure.
[0050] The resulting calculations can be used to understand how many of these individual devices would be desired to create a full resolution energy surface. In this case, approximately 105 x 105, or approximately 11,080 devices, would be desired to achieve visual acuity threshold. Challenges and novelty exist in creating a seamless energy surface from these available energy locations for sufficient sensory holographic propagation.
[0051] Seamless Energy Surface Overview: Construction and design of an array of energy relays In some embodiments, an approach is disclosed to address the challenge of producing high energy location densities from an array of individual devices without the seams constrained by the mechanical structure of each device. In one embodiment, an energy propagation relay system allows for increasing the effective size of the active device area to meet or exceed the mechanical dimensions to make up an array of relays and form a single seamless energy surface.
[0052] 3 illustrates one embodiment of such an energy relay system 300. As shown, the relay system 300 may include a device 310 mounted to a mechanical envelope 320, with an energy relay element 330 transmitting energy from the device 310. The relay element 330 may be configured to provide the ability to reduce any gaps 340 that may occur when multiple mechanical envelopes 320 of a device are arranged in an array of multiple devices 310.
[0053] For example, if the active area 310 of the device is 20 mm x 10 mm and the mechanical envelope 320 is 40 mm x 20 mm, assuming that the array of these elements 330 can be seamlessly aligned together without changing or colliding the mechanical envelope 320 of each device 310, the energy relay elements 330 can be designed with a 2:1 scale factor to produce a tapered shape of approximately 20 mm x 10 mm on the small end (arrow A) and approximately 40 mm x 20 mm on the enlarged end (arrow B). Mechanically, the relay elements 330 can be bonded or fused together, aligned and polished while ensuring a minimum seam gap 340 between each device 310. In such an embodiment, it becomes possible to achieve a seam gap 340 smaller than the visual acuity limit of the eye.
[0054] FIG. 4 illustrates an example of a base structure 400 having energy relay elements 410 formed therewith and securely fastened to an additional mechanical structure 430. The mechanical structure of the seamless energy surface 420 provides the ability to couple multiple energy relay elements 410, 450 in series to the same base structure through bonding or other mechanical processes to mount the relay elements 410, 450. In some implementations, each relay element 410 may be fused, bonded, glued, pressure fitted, aligned, or otherwise attached together to form the resulting seamless energy surface 420. In some embodiments, the device 480 may be mounted to the rear of the relay elements 410 and passively or actively aligned to ensure proper alignment to the energy position within a defined tolerance.
[0055] In one embodiment, the seamless energy surface includes one or more energy locations and one or more energy relay element stacks include first and second sides, each energy relay element stack arranged to form a single seamless energy surface that directs energy along an expanding propagation path between the one or more energy locations and the seamless energy surface, where the distance between the ends of any two adjacent second sides of the terminating energy relay elements is less than a minimum perceptible outline as defined by human acuity better than 20 / 40 acuity at a distance greater than the width of the single seamless energy surface.
[0056] In one embodiment, each of the seamless energy surfaces includes one or more energy relay elements each having one or more structures forming a first and second surface in a transverse and longitudinal orientation, the first relay surface having a different area from the second relay surface resulting in a positive or negative magnification, and configured with a pronounced surface contour for both the first and second relay surfaces that passes energy through the second relay surface to substantially fill an angle of + / - 10 degrees relative to the normal of the surface contour across the entire second relay surface.
[0057] In one embodiment, multiple energy fields may be configured within a single energy relay, or among multiple energy relays, to direct one or more sensory Holodeck energy propagation pathways, including visual, auditory, tactile, or other energy fields.
[0058] In one embodiment, a seamless energy surface may be comprised of an energy relay that includes two or more first sides for each second side to both receive and emit one or more energy fields simultaneously to provide bidirectional energy propagation throughout the system.
[0059] In one embodiment, the energy relay is provided as a loosely coherent element.
[0060] Introducing component engineered structures: Disclosed progress in transverse Anderson localized energy relays Energy relay properties may be significantly optimized in accordance with the principles disclosed herein for energy relay elements that induce transverse Anderson localization, which is the propagation of light rays transported through a material that is laterally irregular but longitudinally coherent.
[0061] This may mean that the material effects that give rise to the Anderson localization phenomenon are less affected by total internal reflection than by randomization between multiple scattering paths, where wave interference may completely restrict the propagation of transverse orientations while continuing the propagation of longitudinal orientations.
[0062] An even more important advantage is the elimination of the cladding of conventional multicore optical fiber materials, which functionally eliminates scattering of energy between fibers, but at the same time acts as a barrier to the light energy, thereby reducing the transmission up to at least the core-to-cladding ratio (e.g., a 70:30 core-to-cladding ratio can transmit up to 70% of the received energy transmission), and also creates strong pixelated patterning in the transmitted energy.
[0063] FIG. 5A illustrates an end view of one such non-Anderson localized energy relay 500, where an image may be relayed through a multi-core optical fiber that may exhibit pixilation and fiber noise due to the inherent properties of optical fibers. With conventional multimode and multi-core optical fibers, the relayed image is inherently prone to pixelation due to the total internal reflection properties of the discrete array cores, where any inter-core crosstalk may degrade the modulation transfer function and increase edge blurring. The resulting images produced using conventional multi-core optical fibers tend to have residual fixed noise fiber patterns similar to those shown in FIG. 3.
[0064] Figure 5B illustrates an example of the same relay image 550 through an energy relay including a material exhibiting properties of transverse Anderson localization, where the relay pattern has a greater density of grain structure compared to the fixed fiber pattern from Figure 5 A. In one embodiment, a relay including a randomized microcomponent engineered structure induces transverse Anderson localization and more effectively transports light with a higher resolvable resolution of propagation than commercially available multimode glass optical fibers.
[0065] There are significant advantages to transverse Anderson localized material properties with respect to both cost and weight, where similar optical grade glass materials can cost and weigh 10-100 times more than the cost of the same material produced in one embodiment, where the disclosed systems and methods demonstrate significant opportunities to improve both cost and quality over other techniques known in the art, including randomized micro-component engineered structures.
[0066] In one embodiment, a relay element exhibiting transverse Anderson localization may include a plurality of at least two different component engineered structures in each of three orthogonal planes arranged in a three-dimensional lattice, the plurality of structures forming channels of randomized distribution of material wave propagation properties in transverse planes in the three-dimensional lattice, and similar values of material wave propagation properties in longitudinal planes in the three-dimensional lattice, where energy waves propagating through the energy relay have higher transport efficiency in the longitudinal orientation relative to the transverse orientation and are spatially localized in the transverse orientation.
[0067] In one embodiment, multiple energy fields may be configured within a single, or between multiple lateral Anderson Localized Energy Relays, and directed to one or more sensory Holodeck energy propagation pathways including visual, auditory, tactile, or other energy fields.
[0068] In one embodiment, the seamless energy surface is comprised of a Transverse Anderson Localized Energy Relay that includes two or more first sides for each second side such that one or more energy fields can both receive and emit simultaneously to provide bidirectional energy propagation throughout the system.
[0069] In one embodiment, the transverse Anderson localized energy relay is configured as a loose coherent element, or flexible energy relay element.
[0070] Considerations regarding 4D plenoptic functions: selective propagation of energy through holographic waveguide arrays As discussed above and throughout this specification, bright field display systems generally include an energy source (e.g., an illumination source) and a seamless energy surface configured with sufficient energy position density, as clearly indicated in the above discussion. A plurality of relay elements may be used to relay energy from the energy device to the seamless energy surface. Once the energy is delivered to the seamless energy surface with the required energy position density, the energy may be propagated according to a 4D plenoptic function through the disclosed energy waveguide system. As will be understood by those skilled in the art, 4D plenoptic functions are well known in the art and will not be further detailed herein.
[0071] The energy waveguide system, together with a structure configured to change the angular direction of the energy waves passing therethrough through representing the angular components of the 4D plenoptic function, selectively propagates energy through multiple energy locations along a seamless energy surface representing the spatial coordinates of the 4D plenoptic function, where the propagated energy waves may converge in space according to multiple propagation paths directed by the 4D plenoptic function.
[0072] Referring now to FIG. 6, an example of a brightfield energy plane in a 4D image space is illustrated according to a 4D plenoptic function. The figure shows a ray trace of the energy plane 600 to a viewer 620 in describing how rays of energy converge in space 630 from various locations in the viewing volume. As shown in the figure, each waveguide element 610 defines four-dimensional information describing energy propagation 640 through the energy plane 600. The two spatial dimensions (referred to herein as x and y) are the physical energy locations that may be observed in the image space, as well as the angular components θ and φ (referred to herein as u and v) that are observed in the virtual space when projected through the energy waveguide array. Typically, and according to a 4D plenoptic function, a plurality of waveguides (e.g., lenslets) may direct energy locations from the x, y dimensions to specific locations in the virtual space along directions defined by the u, v angular components in forming the holographic or brightfield systems described herein.
[0073] However, those skilled in the art will appreciate that a significant challenge to bright field and holographic display technologies results in uncontrolled energy propagation due to designs that do not properly account for diffraction, scattering, diffusion, angular orientation, calibration, focus, collimation, curvature, uniformity, element crosstalk, and any of the many other parameters that contribute to reduced effective resolution and the inability to accurately focus energy with great fidelity.
[0074] In one embodiment, an approach to selective energy propagation to address challenges associated with holographic displays may include an energy suppression element and a substantially filling waveguide aperture with approximately collimated energy into an environment defined by a 4D plenoptic function.
[0075] In one embodiment, the array of energy waveguides may define multiple energy propagation paths through which each waveguide element is configured to extend, substantially filling the effective aperture of the waveguide element in a unique direction defined by a predetermined 4D function for the multiple energy locations along a seamless energy surface constrained by one or more elements positioned to restrict propagation of each energy location to only pass through a single waveguide element.
[0076] In one embodiment, multiple energy fields may be configured within a single energy field or between multiple energy waveguides to direct one or more sensory Holodeck energy propagation including visual, auditory, tactile, or other energy fields.
[0077] In one embodiment, the energy waveguides and seamless energy surfaces are configured to both receive and emit one or more energy fields, providing bidirectional energy propagation throughout the system.
[0078] In one embodiment, the energy waveguide is configured to propagate a non-linear or non-regular energy distribution of energy that includes non-transmitting void regions and leverages digitally encoded, diffractive, refractive, reflective, grin, holographic, Fresnel, or similar waveguide configurations for any seamless energy orientation including walls, tables, floors, ceilings, rooms, or other geometrically based environments. In additional embodiments, the energy waveguide elements can be configured to generate a variety of shapes that provide any surface profile and / or tabletop field of view that allows a user to view the holographic image from all around the energy surface in a 360 degree configuration.
[0079] In one embodiment, the energy waveguide array elements may be reflective surfaces and the arrangement of those elements may be hexagonal, square, irregular, semi-regular, curved, non-planar, spherical, cylindrical, tapered regular, tapered irregular, spatially varying, and / or multi-layered.
[0080] For any component within a seamless energy surface, the waveguide or relay component may include, but is not limited to, optical fibers, silicon, glass, polymers, optical relays, diffractive, holographic, refractive, or reflective elements, optical faceplates, energy combiners, beam splitters, prisms, polarizing elements, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar material that exhibits Anderson localization or total internal reflection.
[0081] Realizing the Holodeck: Aggregation of Bidirectional Seamless Energy Plane Systems for Stimulating Human Sensory Receptors in a Holographic Environment Large-scale environments of seamless energy plane systems can be constructed by tiling, fusing, gluing, affixing, and / or stitching multiple seamless energy planes together to form any size, shape, contour or form factor, including entire rooms. Each energy plane system may include an assembly having a base structure, energy planes, relays, waveguides, devices, and electronics collectively configured for bidirectional Holodeck energy propagation, emission, reflection, or detection.
[0082] In one embodiment, the tiled seamless energy system environments are aggregated to form large seamless planar or curved walls that contain devices up to and including all surfaces within a given environment and are configured as any combination of seamless, discontinuous planar, faceted, curved, cylindrical, spherical, geometric, or irregular shapes.
[0083] In one embodiment, the planar surface aggregate tiles form a wall-sized system for theater or venue-based holographic entertainment. In one embodiment, the planar surface aggregate tiles cover a room with 4-6 walls, including both the ceiling and floor for a cave-based holographic installation. In one embodiment, the curved surface aggregate tiles create a cylindrical seamless environment for an immersive holographic installation. In one embodiment, the seamless spherical surface aggregate tiles create a holographic dome for an immersive Holodeck-based experience.
[0084] In one embodiment, aggregate tiles of seamless curved energy waveguides provide mechanical edges that follow a precise pattern along the boundaries of energy suppression elements within the energy waveguide structure, bonding, aligning, or fusing adjacent tiled mechanical edges of adjacent waveguide surfaces, resulting in a modular, seamless energy waveguide system.
[0085] In further embodiments of the aggregated tiled environment, energy is propagated bidirectionally to multiple simultaneous energy regions. In additional embodiments, the energy surface provides the ability to simultaneously display and capture from the same energy surface with a waveguide designed such that bright field data can be projected by an illumination source through the waveguide and simultaneously received through the same energy surface. In further embodiments, depth sensing and active scanning techniques may be further utilized to enable interaction between the energy propagation and the viewer in precise world coordinates. In additional embodiments, the energy surface and the waveguide are operable to radiate, reflect or focus frequencies to induce haptic or volumetric haptic feedback. In some embodiments, any combination of bidirectional energy propagation and aggregated surfaces is possible.
[0086] In one embodiment, the system comprises an energy waveguide that allows bidirectional emission and detection of energy through an energy surface with one or more energy devices separately paired using two or more path energy couplers to pair at least two energy devices to the same portion of a seamless energy surface, or one or more energy devices are fixed behind the energy surface and in closest proximity to additional components fixed to the base structure or to a location in front and outside the FOV of the waveguide for off-axis direct or reflected projection or detection, the resulting energy surface providing bidirectional energy transmission that allows the waveguide to focus the energy, a first device to emit the energy, and a second device to detect the energy, where the information is processed to perform computer vision related tasks including, but not limited to, 4D plenoptic eyes, and retinal tracking or detection of interference within the propagating energy pattern, depth estimation, approximation, motion tracking, image, color, or sound information, or other energy frequency analysis. In additional embodiments, the tracked position actively calculates and corrects the position of the energy based on interference between bidirectionally captured data and projection information.
[0087] In some embodiments, multiple combinations of three energy devices including an ultrasonic sensor, a visible energy display, and an ultrasonic emitting device are configured together for each of three first relay surfaces that propagate energy coupled to a single second energy relay surface, each of the three first surfaces including designed characteristics specific to the energy region of each device, and two designed waveguide elements configured for ultrasound and energy, respectively, to provide the ability to direct and focus the energy of each device independently and substantially unaffected by other waveguide elements configured for another energy region.
[0088] In some embodiments, a calibration procedure is disclosed that uses encoding / decoding techniques as well as a dedicated integrated system for converting data into calibration information suitable for energy propagation based on a calibrated configuration file, allowing efficient manufacturing, removal of system artifacts, and generation of a geometric mapping of the resulting energy surface.
[0089] In some embodiments, a series of additional energy waveguides and one or more energy devices may be integrated into one system to generate indistinct holographic pixels.
[0090] In some embodiments, additional waveguide elements may be integrated, including energy suppression elements, beam splitters, prisms, active parallax barriers, or polarization techniques, to provide spatial and / or angular resolution greater than the diameter of the waveguide, or for other super-resolution purposes.
[0091] In some embodiments, the disclosed energy system may also be configured as a wearable interactive device, such as a virtual reality (VR) or augmented reality (AR) device. In other embodiments, the energy system may include conditioning optics that focus the displayed or received energy closely to a determined plane in space for the viewer. In some embodiments, the waveguide array may be incorporated into a holographic head mounted display. In other embodiments, the system may include multiple optical paths that allow the viewer to see both the energy system and the real world environment (e.g., a transparent holographic display). In these examples, the system may present as near field in addition to other methods.
[0092] In some embodiments, the transmission of data includes an encoding process with selectable or variable compression ratios that receives any dataset of information and metadata, analyzes the dataset, and receives or assigns material properties, vectors, surface IDs, new pixel data forming a sparser dataset, where the received data may include 2D, stereo, multi-view, metadata, bright field, holographic, geometric shapes, vectors or vectorized metadata, and the encoder / decoder may provide the ability to transform real-time or offline data including image processing for 2D, 2D plus depth, metadata or other vectorized information, stereo, stereo plus depth, metadata or other vectorized information, multi-view, multi-view plus depth, metadata or other vectorized information, holographic, or bright field content via depth estimation algorithms with or without depth metadata, and an inverse ray tracing method appropriately maps the resulting transformed data generated by inverse ray tracing from the various 2D, stereo, multi-view, volumetric, bright field, or holographic data to real world coordinates via a characterized 4D plenoptic function. In these embodiments, the desired total data transmission can transmit information that is several orders of magnitude smaller than the raw brightfield data set.
[0093] Energy Directing Device Suitable for Presenting Holographic Sensory Data In one embodiment, the opto-mechanical display device can emit and direct light to form 2D, stereoscopic, multi-view, plenoptic, 4D, volumetric, bright field, holographic, or any other visual representation of light.
[0094] FIG. 7A is an example of a bright field opto-mechanical system when configured with emissive displays, optical relays, and waveguides realized as an array of refractive elements such as a microlens array, where a visible image from one or more displays can be optically relayed and then transmitted to an energy plane, with the array of refractive elements providing a mapping between each location on the energy plane and the projection direction of light from that location so that a 4D volumetric bright field image can be projected.
[0095] In one embodiment, the waveguide may be operable to converge light rays to induce both vergence and accommodation from an observer's perspective.
[0096] In one embodiment, the waveguide and energy relay can be formed or polished with a variety of surface geometries. In one embodiment, the energy relay includes an element that induces transverse Anderson localization. In one embodiment, the energy relay is bidirectional and can emit and / or project energy.
[0097] In one embodiment, the energy system configured to direct energy according to a four-dimensional (4D) plenoptic function includes a plurality of energy devices. In some embodiments, the plurality of energy devices includes an illumination source that emits image information, the image information including light-emitting, projected, or reflective display technology utilizing visible, IR, UV, coherent, laser, infrared, polarized, or any other electromagnetic illumination source. In other embodiments, the plurality of energy devices includes mechanical energy emitting devices configured to provide immersive audio or volumetric haptics from an acoustic field.
[0098] In some embodiments, the energy system configured as described above may further include a base structure (e.g., 72) such that the multiple energy devices, the energy relay system, and the energy waveguide system may all be coupled to the base structure. In other embodiments, the multiple energy devices, the energy relay system, and the energy waveguide system may be coupled to the base structure using one or more mounting brackets.
[0099] In some embodiments, the plurality of energy devices includes energy devices for capturing or sensing energy, including mechanical, chemical, conductive, thermal, electrical, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In other embodiments, the plurality of energy devices includes energy devices for propagating or emitting energy, including mechanical, chemical, conductive, thermal, electrical, potential, kinetic, magnetic, gravitational, radiant, energy, structured, unstructured, or other forms of energy. In yet other embodiments, the plurality of energy devices includes an acoustic receiving device configured to provide sensory feedback or audible control.
[0100] In one embodiment, the energy system further includes an energy relay system (e.g., 6110, best shown in FIG. 7B) having one or more energy relay elements, each of the one or more energy relay elements including a first surface and a second surface, the second surfaces of the one or more energy relay elements arranged to form a single seamless energy surface of the energy relay system, and the first plurality of energy propagation paths extending from energy locations in the plurality of energy devices through the single seamless energy surface of the energy relay system, as will be discussed in more detail below.
[0101] Reference is now made to FIG. 7B, which illustrates an orthogonal view of an energy relay system 6110, according to one embodiment of the present disclosure. In one embodiment, the energy relay system 6110 can include two or more relay elements 6112, each relay element 6112 formed of one or more structures, each relay element 6112 having a first surface 6114, a second surface 6116, a lateral direction (generally parallel to the surfaces 6114, 6116) and a longitudinal direction (generally perpendicular to the surfaces 6114, 6116). In one embodiment, the surface area of the first surface 6114 can be different from the surface area of the second surface 6116. For example, the surface area of the first surface 6114 can be larger or smaller than the surface area of the second surface 6116. In another embodiment, the surface area of the first surface 6114 can be the same as the surface area of the second surface 6116. An energy wave can pass from the first surface 6114 to the second surface 6116 or vice versa.
[0102] In one embodiment, the relay element 6112 of the energy relay system 6110 includes a sloped contour portion 6118 between the first surface 6114 and the second surface 6116. In operation, an energy wave propagating between the first surface 6114 and the second surface 6116 may have a higher transport efficiency in the longitudinal direction than in the lateral direction, and an energy wave passing through the relay element 6112 may result in a spatial expansion rate or a spatial contraction rate. In other words, an energy wave passing through the relay element 6112 of the relay element device 6110 may experience an increased expansion rate or a decreased contraction rate. In some embodiments, the one or more structures for forming the energy relay element 6110 may include glass, carbon, optical fiber, optical film, plastic, polymer, or a mixture thereof.
[0103] In one embodiment, the energy wave passing through the first surface 6114 has a first resolution and the energy wave passing through the second surface 6116 has a second resolution, the second resolution being about 50% or greater than the first resolution. In another embodiment, the energy wave has a uniform profile when provided to the first surface, but can pass through the second surface radiating in all directions with a forward directed energy density that substantially fills a cone having an opening angle of + / - 10 degrees relative to the normal to the second surface, regardless of the position of the second relay surface.
[0104] In some embodiments, the first surface 6114 can be configured to receive energy from an energy wave source, the energy wave source including a mechanical envelope having a width different from a width of at least one of the first surface 6114 and the second surface 6116.
[0105] In each relay 6112, energy is transported between first and second surfaces that define a longitudinal direction, with the first and second surfaces of each of the relays generally extending along a transverse direction defined by the first and second directions, with the longitudinal direction being substantially perpendicular to the transverse direction. In one embodiment, energy waves propagating through multiple relays have a higher transport efficiency in the longitudinal direction than in the transverse direction due to the random refractive index variations in the transverse direction coupled with minimal refractive index variations in the longitudinal direction. In some embodiments where each relay is constructed of multicore fiber, energy waves propagating within each relay element may travel in a longitudinal direction determined by the alignment of the fibers in this orientation.
[0106] In one embodiment, the spacing between the ends of any two adjacent second sides of the terminal energy relay element may be less than the minimum perceivable outline defined by human visual acuity better than 20 / 40 at a distance from the seamless energy surface that is greater than the smaller of the height of the single seamless energy surface or the width of the single seamless energy surface.
[0107] In one embodiment, the multiple energy relay elements in a stacked configuration can include multiple faceplates. In some embodiments, the multiple faceplates may have different lengths or are loosely coherent optical relays. In other embodiments, the multiple elements can have angled contours similar to those in FIG. 7B, where the angled contours can be slanted, straight, curved, tapered, faceted, or arranged non-orthogonally to the normal axis of the relay elements. In yet another embodiment, energy waves propagating through the multiple relay elements have a higher transport efficiency in the longitudinal direction than in the lateral direction due to random refractive index variations in the lateral direction coupled with minimal refractive index variations in the longitudinal direction. In an embodiment where each energy relay is constructed of multicore fiber, energy waves propagating within each relay element can travel in a longitudinal direction determined by the alignment of the fibers in this orientation.
[0108] In some embodiments, one or more relay elements (e.g., 6112) comprise a fused or tiled mosaic, and any seams between adjacent fused or tiled mosaics are separated by a minimum perceptible contour as defined by human visual acuity having a value better than 20 / 40 acuity at or greater than the width or height of a single seamless energy surface, or are smaller than the minimum perceptible outline.
[0109] In other embodiments, one or more relay elements (e.g., 6112) include optical fibers, silicon, glass, polymers, optical relays, diffractive elements, holographic optical elements, refractive elements, reflective elements, optical faceplates, optical combiners, beam splitters, prisms, polarizing components, spatial light modulators, active pixels, liquid crystal cells, transparent displays, or any similar material with Anderson localization or total internal reflection properties to form a single seamless energy surface.
[0110] In yet other embodiments, one or more relay elements (e.g., 6112) are configured to accommodate the shape of a single seamless energy surface, including planar, spherical, cylindrical, conical, faceted, tiled, regular, irregular, or any other geometric shape for a specified application.
[0111] In another embodiment, the system further includes an energy waveguide system (e.g., 7100 best shown in Figures 7C-7L) having an array of energy waveguides, and a second plurality of energy propagation paths extend from the single seamless energy surface through the array of energy waveguides in a direction determined by a 4D plenoptic function.
[0112] 7C shows a top-down perspective view of an embodiment of an energy waveguide system 7100 operable to define multiple energy propagation paths 7108. The energy waveguide system 7100 comprises an array of energy waveguides 7112 configured to direct energy through the array of energy waveguides 7112 along the multiple energy propagation paths 7108. In one embodiment, the multiple energy propagation paths 7108 extend through multiple energy locations 7118 on a first side 7116 of the array to a second side 7114 of the array.
[0113] 7C and 7L, in one embodiment, a first subset 7290 of the plurality of energy propagation paths 7108 extends through a first energy location 7122. The first energy waveguide 7104 is configured to direct energy along a first energy propagation path 7120 of the first subset 7290 of the plurality of energy propagation paths 7108. The first energy propagation path 7120 may be defined by a first chief ray 7138 formed between the first energy location 7122 and the first energy waveguide 7104. The first energy propagation path 7120 may include rays 7138A and 7138B formed between the first energy location 7122 and the first energy waveguide 7104 and directed by the first energy waveguide 7104 along energy propagation paths 7120A and 7120B, respectively. The first energy propagation path 7120 may extend from the first energy waveguide 7104 toward the second side 7114 of the array. In one embodiment, the energy directed along the first energy propagation path 7120 includes one or more energy propagation paths between or including energy propagation paths 7120A and 7120B that are directed through the first energy waveguide 7104 in a direction substantially parallel to the angle at which the first chief ray 7138 propagates through the second side 7114.
[0114] The embodiment may be configured such that energy directed along the first energy propagation path 7120 may exit the first energy waveguide 7104 in a direction substantially parallel to the energy propagation paths 7120A and 7120B and the first chief ray 7138. The energy propagation paths extending through the energy waveguide element 7112 on the second side 7114 may be assumed to include multiple energy propagation paths of substantially similar propagation directions.
[0115] 7D is a front view of one embodiment of the energy waveguide system 7100. The first energy propagation path 7120 may extend in a unique direction 7208 extending from the first energy waveguide 7104, determined by at least the first energy location 7122, toward the second side 7114 of the array. The first energy waveguide 7104 may be defined by spatial coordinates 7204, and the unique direction 7208, determined by at least the first energy location 7122, may be defined by an angular coordinate 7206 that defines the direction of the first energy propagation path 7120. The spatial coordinates 7204 and the angular coordinates 7206 may form a four-dimensional plenoptic coordinate set 7210 that defines the unique direction 7208 of the first energy propagation path 7120.
[0116] In one embodiment, energy directed along the first energy propagation path 7120 through the first energy waveguide 7104 substantially fills the first opening 7134 of the first energy waveguide 7104 and propagates along one or more energy propagation paths located between energy propagation paths 7120A and 7120B and parallel to the direction of the first energy propagation path 7120. In an embodiment, the one or more energy propagation paths that substantially fill the first opening 7134 may include more than 50% of a diameter of the first opening 7134.
[0117] In a preferred embodiment, the energy directed along the first energy propagation path7120 through the first energy waveguide7104 substantially fills the first opening7134, which may include 50% to 80% of the diameter of the first opening7134.
[0118] 7C and 7E-7L, in one embodiment, the energy waveguide system 7100 may further comprise an energy suppression element 7124 positioned to limit propagation of energy between the first side 7116 and the second side 7114 and suppress energy propagation between adjacent waveguides 7112. In one embodiment, the energy suppression element is configured to suppress energy propagation along a portion of the first subset 7290 of the plurality of energy propagation paths 7108 that does not penetrate the first opening 7134. In one embodiment, the energy suppression element 7124 may be disposed between the array of energy waveguides 7112 and the plurality of energy locations 7118 at the first side 7116. In one embodiment, the energy suppression element 7124 may be disposed between the plurality of energy locations 7118 and the energy propagation paths 7108 at the second side 7114. In one embodiment, the energy suppression element 7124 can be positioned orthogonal to the array or plurality of energy locations 7118 of the energy waveguide 7112 on the first side 7116 or the second side 7114 .
[0119] In one embodiment, energy directed along the first energy propagation path 7120 may converge with energy directed along the second energy propagation path 7126 through the second energy waveguide 7128. The first and second energy propagation paths may converge at a location 7130 on the second side 7114 of the array 7112. In one embodiment, the third and fourth energy propagation paths 7140, 7141 may also converge at a location 7132 on the first side 7116 of the array 7112. In one embodiment, the fifth and sixth energy propagation paths 7142, 7143 may also converge at a location 7136 between the first and second sides 7116, 7114 of the array 7112.
[0120] 7E-7L illustrate various embodiments of energy suppression element 7124. For the avoidance of doubt, these embodiments are provided for illustrative purposes and in no way limit the scope of combinations or implementations provided within the scope of the present disclosure.
[0121] 7E illustrates an embodiment of a plurality of energy locations 7118 in which energy suppression elements 7251 are disposed adjacent to a surface of the energy locations 7118 and include specific refraction, diffraction, reflection, or other energy altering properties. The energy suppression elements 7251 may be configured to restrict a first subset of energy propagation paths 7290 to a narrower range of propagation paths 7253 by suppressing propagation of energy along energy propagation paths 7252. In one embodiment, the energy suppression elements are energy relays having a numerical aperture less than one.
[0122] 7F illustrates one embodiment of a plurality of energy locations 7118, where the energy suppression structures 7254 are positioned orthogonally between regions of the energy locations 7118, where the energy suppression structures 7254 exhibit absorbing properties, and where the energy suppression structures 7254 have heights defined along the energy propagation paths 7256 such that a particular energy propagation path 7255 is suppressed. In one embodiment, the energy suppression structures 7254 are hexagonal in shape. In one embodiment, the energy suppression structures 7254 are circular. In one embodiment, the energy suppression structures 7254 are non-uniform in shape or size along any orientation of the propagation path. In one embodiment, the energy suppression structures 7254 are embedded within another structure having additional properties.
[0123] 7G illustrates a plurality of energy locations 7118, with a first energy suppression structure 7257 configured to orient energy 7259 propagating therethrough substantially in a first state. A second energy suppression structure 7258 is configured to allow energy 7259 oriented substantially in the first state to propagate therethrough and restrict propagation of energy 7260 oriented substantially differently from the first state. In one embodiment, the energy suppression elements 7257, 7258 are energy polarizing element pairs. In one embodiment, the energy suppression elements 7257, 7258 are energy wave bandpass element pairs. In one embodiment, the energy suppression elements 7257, 7258 are diffractive waveguide pairs.
[0124] 7H illustrates an embodiment of a plurality of energy locations 7118 in which an energy suppression element 7261 is configured to modify the energy propagation path 7263 to some extent depending on which of the plurality of energy locations 7118 the energy propagation path 7263 extends through. The energy suppression element 7261 can modify the energy propagation path 7263 uniformly or non-uniformly along the energy propagation path 7263 such that a particular energy propagation path 7262 is suppressed. The energy suppression structure 7254 is disposed orthogonally between the regions of the energy locations 7118, the energy suppression structure 7254 exhibits absorptive properties, and the energy suppression structure 7254 has a defined height along the energy propagation path 7263 such that a particular energy propagation path 7262 is suppressed. In one embodiment, the suppression element 7261 is a field lens. In one embodiment, the suppression element 7261 is a diffractive waveguide. In one embodiment, the suppression element 7261 is a curved waveguide surface.
[0125] FIG. 7I illustrates one embodiment of multiple energy locations 7118 where an energy suppression element 7264 provides absorption properties to limit the propagation of energy 7266 while allowing other propagation paths 7267 to pass.
[0126] 7J illustrates an embodiment of a plurality of energy locations 7118 and a plurality of energy waveguides 7112, where a first energy suppression structure 7268 is configured to orient energy 7270 propagating therethrough substantially in a first state. A second energy suppression structure 7271 is configured to allow energy 7270 oriented substantially in the first state to propagate therethrough and to limit the propagation of energy 7269 oriented substantially differently from the first state. To further control energy propagation through the system, exemplified by stray energy propagation 7272, the energy suppression structures 7268, 7271 may require composite energy suppression elements to ensure that energy propagation maintains a precise propagation path.
[0127] 7K illustrates one embodiment of a plurality of energy locations 7118 where the energy suppression element 7276 provides absorptive properties to restrict the propagation of energy along energy propagation path 7278 while allowing other energy along energy propagation path 7277 to pass through a pair of energy waveguides 7112 for effective opening 7284 in the array of waveguides 7112. In one embodiment, the energy suppression element 7276 comprises black chrome. In one embodiment, the energy suppression element 7276 comprises an absorptive material. In one embodiment, the energy suppression element 7276 comprises a transparent pixel array. In one embodiment, the energy suppression element 7276 comprises an anodized material.
[0128] FIG. 7L illustrates an embodiment in which a first energy suppression structure 7251 is disposed adjacent to a surface of the energy locations 7118 and includes a plurality of energy waveguides 7112 that include specific refraction, diffraction, reflection, or other energy modifying properties. The energy suppression structure 7251 may be configured to restrict a first subset of the energy propagation paths 7290 to a narrower range of propagation paths 7275 by suppressing energy propagation along the energy propagation paths 7274. The second energy suppression structure 7261 is configured to modify the energy propagation paths 7275 to some extent depending on which of the plurality of energy locations 7118 the energy propagation paths 7275 extend through. The energy suppression structure 7261 may modify the energy propagation paths 7275 uniformly or non-uniformly such that a specific energy propagation path 7274 is suppressed. The third energy suppression structure 7254 is disposed orthogonally between the regions of the energy locations 7118. The energy suppression structures 7254 exhibit absorptive properties and have heights defined along the energy propagation paths 7275 such that a particular energy propagation path 7274 is suppressed. The energy suppression elements 7276 provide absorptive properties to limit the propagation of energy 280 while allowing the passage of energy 7281. A composite system of similar or different waveguide elements 7112 is positioned to substantially fill the effective aperture 7285 of the waveguide element with energy from multiple energy locations 7118 and alter the propagation path 7273 of the energy defined by the particular system.
[0129] In one embodiment, the energy suppression structure 7124 may be positioned proximate to the first energy location 7122 and extend generally toward the first energy waveguide 7104. In one embodiment, the energy suppression structure 7124 may be positioned proximate to the first energy waveguide 7104 and extend generally toward the first energy location 7122.
[0130] In one embodiment, the energy system is configured to direct energy along the second plurality of energy propagation paths through an energy waveguide system to a single seamless energy surface, and to direct energy along the first plurality of energy propagation paths from the single seamless energy surface through an energy relay system to a plurality of energy devices.
[0131] In another embodiment, the energy system is configured to direct energy along a first plurality of energy propagation paths from a plurality of energy devices through an energy relay system to a single seamless energy surface, and to direct energy along a second plurality of energy propagation paths from the single seamless energy surface through an energy waveguide system.
[0132] In yet another embodiment, a single seamless energy surface is operable to restrict localized light transmission to no more than three wavelengths of visible light.
[0133] Sensory data suitable for holographic displays The 4D plenoptic function from an energy directing surface through a surface contains the energy position and the vector f of the energy propagation path. l (x l , y l , u l , v l ) is defined by the guided wave parameter u l , v l Two spatial coordinates x from the first surface, directed through a second coordinate along a second surface including l , y l Considering multiple energy directing surfaces, the plenoptic5D function contains one or more energy locations and provides a vector of energy propagation paths f l (x l , y l , z l , u l , v l ) is defined by the guided wave parameter u l , v lA three-dimensional coordinate system is defined from the first coordinate to three spatial coordinates x, oriented through the second coordinate along a plane that contains l , y l , z l For 4D or 5D respectively, additional variables f for time and color l (λ l , t l ), functions and may be considered to include any of the plenoptic functions if necessary for the application, even if not explicitly noted for ease of explanation. For the avoidance of doubt, references to energy directing surfaces are for illustrative purposes only and may include any additional points, positions, directions, or planes in space for 5D coordinate localization, collectively referred to as "energy directing surfaces."
[0134] Figure 8 is a flow chart diagram illustrating an embodiment of a process 800 for determining four-dimensional (4D) plenoptic coordinates for content data. Process 800 can include step 802 in which content data is received, which may include any signal perceptible by a visual, auditory, tactile, sensory, or olfactory sensor. Figure 9 is a schematic diagram illustrating an embodiment of content data that may include at least one of: object positions, material properties (such as material properties 906, 907, and 908), virtual light source 904, shape at non-object positions 902, content from reference surfaces, virtual camera position 914, object segmentation 910, background texture 912, and layered content.
[0135] 8 and 9, the process 800 may further include a step 804 in which the positions of the data points are determined relative to a first surface 920 for generation of a digital volumetric representation 922 of the content data. The first surface 920 may be used as a reference surface for defining the positions of the data points in space. In one embodiment, the process 800 may further include a step 806 in which 4D plenoptic coordinates of the data points are determined at the second surface by tracing the positions of the data points in the volumetric representation to a second surface where a 4D function is applied. In one embodiment, the process 800 may further include a step 808 in which the energy source position values are determined relative to the 4D plenoptic coordinates having the first convergence point.
[0136] The content data received in step 802 may include N views, where N is 1 or greater. A single view may be presented with or without a depth channel. A stereoscopic view may be presented with or without a depth channel. A multi-view image may be presented with or without a depth channel. Additionally, a 4D light field may be presented with or without a depth channel.
[0137] The tracking of step 806 may use prior knowledge of the calibrated geometry of the energy system, which may be stored in memory as a global model or an individually characterized system, or some combination of the two methodologies.
[0138] In one embodiment, the mapping between input data and output energy sources provides a methodology for accurate mapping between various bit rate sources. The tracking of step 806 provides the ability to infer a complete 4D volumetric data set from the partial samples listed above. Depth information must be provided or calculated from available data. With known or calculated depth information, the N views can be back tracked by triangulation of samples from a known stereoscopic representation based on depth coordinates into 4D space.
[0139] In triangulation, if a mapping between energy waveguides and energy source position format resolution is provided, each available energy source position in the N source content can be assumed to represent an energy source position for each energy waveguide. If the resolution of the N source content is lower, a super-resolution or scaling algorithm can be implemented. If the resolution of the N source images is higher than the number of energy waveguides in the energy directing device, an interpolation between the supersampled energy source positions can be performed to generate more energy source positions per energy waveguide in the resulting 4D inverse ray trace.
[0140] In the above, it is assumed that the distance information can be determined from a depth map, which may or may not be accurate depending on the form of depth information provided or calculated, and with known or assumed distance information, the distance information combined with the xy energy source position coordinates and the (u,v) angle information determined by the characteristics of the energy directing device can be considered as a 4D or 5D light field with limited imaging data samples. The imaging samples are triangulated back to the appropriate energy source positions that may be respectively behind each energy waveguide based on the distance information, and the missing data can be generated in step 808 through the disclosure contained herein.
[0141] Referring to Figures 7C, 8, 9, and 10, in one embodiment, the energy locations may be located within a first surface 920 and a second surface to which the 4D function is applied may correspond to a waveguide system 7100 of the energy directing device, where the energy is directed through the waveguide system according to the 4D plenoptic coordinates of the data points, operable to form a detectable stereoscopic representation of the content data.
[0142] In one embodiment, the process 800 may further include step 810, in which an energy source location value is determined relative to the 4D coordinates having the first convergence point. To provide an example implementation of the present disclosure, FIG. 10 illustrates one embodiment of an energy directing device 1000 undergoing a tracking process in which content data in the form of an image 1002 is provided along with distance locations 1004 that may be provided or calculated within a minimum location 1006 and a maximum location 1008 determined relative to an energy location 1010. In one embodiment, the energy location 1010 may include a surface of the energy directing device. The known shape from the energy location 1010 defined by the 4D plenoptic function allows for triangulation of points 1014 on the imaginary surface of the image 1002 back along rays 1016 to specific energy locations 1018, each having unique xy coordinates. The missing samples may be computationally calculated based on available information contained within the data set.
[0143] If an additional N samples are provided, the same methodology can be applied using additional multi-view imaging data generating a richer set of inverse ray trace samples, providing superior holographic results. Depth information from multiple N samples can be provided through a single depth map with known mapping between source positions (N+X viewpoints) and source depth maps (N+X depth maps), or through up to or more than N depth maps, ensuring that proper inverse ray tracing is performed.
[0144] For example, if a single depth map for the central N viewpoints is provided, additional depth maps can be interpolated by calculating the difference between each of the adjacent views to accurately map source and target locations between the N and N+X viewpoints. Using this method, the view-dependent mapping can be back-ray traced up to the 4D light field such that the correct viewpoint is projected into the appropriate waveguide coordinates resulting in the viewer's ability to maintain the correct view-dependence at the associated viewpoint.
[0145] The encoders and decoders are robust and capable of interpreting multiple data types including but not limited to 2D / flat files, 2D with depth, stereo, stereo with a single depth channel, stereo with dual depth channels, N+X multiview with no depth, N+X multiview with N+Y depth, geometry or vector based scene files that may reconstruct the environment including texture, geometry, lighting, material properties, etc., deep imaging files that may be provided as N+X view plus N+Y delta channel datasets where multiple RGBAZ values may be provided for each xy coordinate, 4D or 5D (4D plus depth) light field, or where the depth channel provides a lower bandwidth methodology for rendering only the specific amount of energy source position data required for the determined energy directing device's field of view. The processor may inverse ray trace at up to or exceeding real time speeds to provision the appropriate 4D light field to be presented to the viewer, with or without the world coordinate positions, with or without the compensated minimum and maximum projected world positions, and taking into account the unique energy directing device as characterized and / or designed.
[0146] In one embodiment, the process 800 may further include a step 812 in which a mapping is applied between the energy positions 7122 on the first side of the waveguide system 7100 and the angular orientation of the energy propagation paths 7120 from the waveguide elements 7100 on the second side of the waveguide system 7100. Doing so may enable determining a plurality of energy positions on the first side of the waveguide system 7100 that correspond to the 4D plenoptic coordinates of the data points.
[0147] FIG. 12 is a schematic diagram of a processing system 1200 comprising a data input / output interface 1201 in communication with a processing subsystem having a sensory data processor 1202, a vectorization engine 1204, and a tracing engine 1206. It should be understood that the sensory data processor 1202, the vectorization engine 1204, and the tracing engine 1206, either individually or in any combination thereof, may be implemented on one or more processors. Step 802 of process 800 may input content data to the processing subsystem 1220 via the data input / output interface 1201. Step 804 may be performed by the sensory data processor 1202 to generate a stereoscopic representation of the content data. Step 806
[0148] In one embodiment, applying the mapping may include calibrating for distortions in the waveguide system 7100, and may further include calibrating for at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and chromatic distortion.
[0149] In one embodiment, the energy directing device may further include a relay system 6110 on a first side of the waveguide system 7100, the relay system having a first surface 6116 adjacent to the waveguide system 7100, and an energy location 7112 on the first side of the waveguide system may be positioned adjacent to a second surface 6114 of the relay system 6110.
[0150] In one embodiment, applying the mapping may include calibrating for distortions in the waveguide system 7100. In one embodiment, applying the mapping may include calibrating for both distortions in the relay system 6110 and distortions in the waveguide system 7100. In one embodiment, the distortions calibrated for may include at least one distortion selected from the group consisting of spatial distortion, angular distortion, intensity distortion, and chromatic distortion.
[0151] In one embodiment, portions of the method may be performed in real time, or the method may be performed entirely in real time, or at least two portions of the method may be performed in different time periods.
[0152] 2D to Brightfield Conversion In one embodiment, the content data may include data points in two-dimensional (2D) space, and determining the location of step 704 may include applying a depth map to the data points in the two-dimensional space.
[0153] There are several methods for converting two-dimensional or planar images into lightfield data. These include estimating depth information through depth from motion analysis, depth channels provided by manual or rendered means, or any other methodology known as standard for visual effects content generation that recreates the complete lightfield through manual generation of disparity, depth, occlusion, shape, and / or recreating the entire environment through manual and automated processes.
[0154] In a first embodiment, the system can include a real-time or offline processor to perform depth estimation from available energy source location information. This can be performed in an energy directing device as a set-top box or as an offline process. Additional calculations for missing volumetric data can be performed utilizing temporal information and / or state-of-the-art texture synthesis or other techniques known in the art.
[0155] In a second embodiment, the depth information can be provided as an image stream and embedded in the image format. Similarly, additional calculations can be performed for missing volumetric data.
[0156] In a third embodiment, an artist or process is utilized to generate the missing environmental information, which may include a process of isolating or segmenting each object in the scene, manually, semi-automatically, or automatically tracking the objects over time, utilizing parallax space, energy directing device space, optical space, or world coordinates, and synthesizing the background and foreground missing information with known visual effects processes that regenerate backgrounds, transparency, edge details, etc. to recreate the environment. For the avoidance of doubt, the process implemented may be any, none, or all of the embodiments enumerated for the reconstruction of these environments. The artist would know that the generated environmental information should include as much of the missing information as possible as determined by the angles of view of the energy directing device, which will ensure that the appropriate occlusion and view-dependent information is accurately generated.
[0157] Additionally, a surface model can be generated for each object in the scene, either as a partial model or a fully constructed model, and textures from the image data can be projected onto the surface of the shape to provide an appropriate shape for subsequent reverse ray tracing.
[0158] Furthermore, material properties can be calculated or manually introduced to ensure that virtual illumination sources can be used to introduce field-dependent illumination to further improve the accuracy of the 4D bright-field reproduction.
[0159] Additionally, the addition of CG or synthetic content can be introduced to augment the existing transformation material. The addition of volumetric data can also be incorporated. Intermixing of N+X content can also be introduced to provide a seamless blend between CG, 2D, stereoscopic, multiview and / or 4D media within a single composite image.
[0160] The resulting 2D to brightfield converted content may be persisted as a geometric scene file containing the geometry, textures, lighting, materials, etc. as shown in the CG scene itself, may be rendered as an N+X view with an N+D depth channel, may be rendered as a 4D or 5D (4D+depth) brightfield, depth image that is a format allowing multiple RGBAZ samples per xy source position coordinate with or without the limitation of Z sample stacking per xy coordinate, or may be provided as an N+X view plus N+Y delta channel dataset where the depth channel provides a lower bandwidth methodology for rendering only the specific amount of source position data required for the determined energy directing device field of view. Tools may be provided to enable the generation of all, some, or one of each of these output formats.
[0161] Stereo and Multiview to Brightfield Conversion The process from above utilizing single-view content can be applied to stereoscopic and multiview materials. Estimation of depth information is obtained through depth from motion analysis, and from stereoscopic, multiview and / or disparity analysis, one depth channel or multiple depth channels provided through manual or rendered means, or manual generation of disparity, depth, occlusion, shape, and / or recreation of the entire environment by manual or automated processes, and any other methodology known as standard for visual effects content generation that utilizes appropriate data to recreate the complete brightfield.
[0162] In one embodiment, the content data received in step 102 may include data points in three-dimensional (3D) space, and determining the location may include adjusting the data points in the 3D space.
[0163] In one embodiment, adjusting the data points in the 3D space may include applying a depth map to the data points in the 3D space, adding new data points, reconstructing occluded data points, or any combination thereof.
[0164] A key advantage of this approach is that the accuracy of stereo disparity estimation is much higher than that of motion parallax or other similar 2D estimation processes alone. Furthermore, the image quality of the resulting transformed 4D light-field is more accurate due to the availability of several view-dependent conditions, including but not limited to illumination, transparency, materials, occlusion, etc.
[0165] The ability to preserve the explicit angular dependence of multi-view image data depends on the ability to compute surface normals with respect to a central viewpoint camera, or some other defined center point. With these normals and disparity or depth information known, it is possible to interpolate between viewpoints based on the angle of view of the energy directing device, and the resulting interpolations are then either applied directly to the back ray tracing or synthesized as part of texture synthesis during back ray tracing.
[0166] For simplicity, all of the previously disclosed methodologies for 2D to bright-field image reconstruction can be applied to the reconstruction of volumetric or multi-view datasets.
[0167] Generating N×N RGB Images from 4D or 5D Bright Fields By utilizing 4D or 5D bright fields, it is possible to generate a number of RGB multi-view images, N×N in number, or any number of values up to N×N. This process is adapted by assuming a square grid, a 0,0 location, and a top right location as the N,N location, and considering each bottom left coordinate below each waveguide. This grid is merely exemplary, and other mapping methodologies can be utilized. For each location 0,0 through N,N, a full resolution image can be formed from the bright field with the widest possible depth of field based on the capture system utilized, with each waveguide in the array considered as a single energy source location, and each coordinate below each waveguide being a single energy source location in a larger array of energy source locations for each complete image from 0,0 through N,N. This can be repeated for 5D bright field depth information. In this way, it is possible to easily switch between 4D or 5D brightfield for any subset of the data set desired to include 2D, stereo, multiview, point cloud, CG scene file, or any other desired combination of data that may be derived from 4D or 5D brightfield due to different distributions. For irregular or square-filled 4D or 5D structures, further interpolation may be required to align the energy source positions to a regular grid, or a linear mapping between the energy source positions and non-square-filled structures may be performed, and the resulting image may not look linear and may contain artifacts of the energy source positions.
[0168] 11 illustrates a methodology for converting from 4D or 5D brightfield to multiple viewpoints by locating energy locations 1102 from beneath each energy waveguide element 1104 according to the energy waveguide element location and energy location coordinates, respectively. This provides the ability to seamlessly transfer between brightfield and smaller data sets.
[0169] N+X RGB and N+Y Depth Dataset An ideal dataset format that provides the highest quality with a balance of data transmission size includes the use of N+X RGB and N+Y depth+vectorized channels, where the N+X RGB information includes N RGB images, which may represent a specific resolution and format, X, which may represent a different resolution and format for the RGB data to include lower resolution, delta information, etc., and N+Y depth+vectorized channels, including N depth+vectorized channels, which may represent a specific resolution and format, and Y, which may represent a different resolution and format for the depth+vector data to include lower resolution, delta information, etc.
[0170] The number of N+X views can be generated on a regular grid, from one radius around a central point with or without a central view, from multiple radii around a central point with or without a central view, or from any methodology for determining the mapping between the number of views and the associated packing or gaze positions. The configuration for the viewpoint can be included in the metadata of the file, or the depth+vectorization channel provided can include a direct mapping to world coordinates such that the imaging data aligns to the same coordinates in XYZ space without other required metadata.
[0171] 4D Disk Inversion and Energy Directing Apparatus Compatibility Handling For any data captured using a plenoptic or brightfield 4D or 5D system, which may include that captured with a virtual rig using an optical simulation of a 4D or 5D brightfield system, the resulting fly's eye perspective contains a disk representing the brightfield uv vector. However, these coordinates assume an energy focusing element that may not exist in the energy directing apparatus. In the proposed energy directing apparatus solution, the focusing element may be the viewer's eye, and the mapping between the capture systems, and between the original capture method and the observed energy directing apparatus, is no longer correct.
[0172] To reverse this and compensate for the additional missing energy directing elements in the system compared to the capture system, each disk can be flipped independently and the xy location of each (u,v) coordinate is retargeted based on the center point of each waveguide. In this way, the flip of the resulting image of the main waveguide is reversed, allowing the brightfield energy director to project light rays in the correct xyuv directions.
[0173] Further embodiments of this may implement hardware modifications that leverage the energy waveguide array to provide a direct inversion of the energy source location of all presented energy waveguides. For bright field energy directing devices, this has the advantage of having a direct mapping between the expected capture system and the energy directing device. This may be further advantageous for embodiments including HMD systems or volumetric opaque energy directing devices, as it allows the removal of a group of energy waveguides in the entire array by eliminating the need to relay additional times for precise xyuv coordinates.
[0174] Furthermore, not all brightfields are the same. They may be captured with different NAs, FOVs, N values, optical prescriptions, etc. The intrinsic and extrinsic properties of the input brightfield data may be understood and translated into properties of the energy directing device. This may be performed by embodiments contained within this disclosure for universal parameterization of holographic and brightfield data.
[0175] Universal parameterization of holographic sensory data transport by inverse energy tracing and vectorization of sensory properties for energy-directed systems. The 4D plenoptic function through the surface from an energy-directed surface contains the energy position and the vector of the energy propagation path f l (x l , y l , u l , v l ) is defined by the guided wave parameter u l , v lTwo spatial coordinates x from the first surface, directed through a second coordinate along a second surface including l , y l Considering multiple energy directing surfaces, the plenoptic5D function contains one or more energy locations and provides a vector of energy propagation paths f l (x l , y l , z l , u l , v l ) is defined by the guided wave parameter u l , v l A three-dimensional coordinate system is defined from the first coordinate to three spatial coordinates x, oriented through the second coordinate along a plane that contains l , y l , z l For 4D or 5D respectively, additional variables f for time and color l (λ l , t l ), functions and may be considered to include any of the plenoptic functions if necessary for the application, even if not explicitly noted for ease of explanation. For the avoidance of doubt, references to energy directing surfaces are for illustrative purposes only and may include any additional points, positions, directions, or planes in space for 5D coordinate localization, collectively referred to as "energy directing surfaces."
[0176] Along the first vector of the energy propagation path, multiple intersections involving energy convergence may occur with additional energy propagation paths. At the intersections, a 3D point or depth parameter is a position X between multiple energy propagation paths with a 4D or 5D function. l , Y l , Z l where each x contained within one or more energy directing surfaces l , y l or x l , y l , z l For a coordinate, a single u is formed between the first coordinate and the 3D convergence point. l , v l3D convergence point X between multiple energy propagation paths, where only the propagation paths exist l , Y l , Z l 4D function f z (x l , y l , u l , v l ) or 5D function f z (x l , y l , z l , u l , v l ) is available for all 4Dx l , y l or 5Dx l , y l , z l coordinates, and X l , Y l , Z l For each convergence point of l , v l Collectively define the propagation path.
[0177] Energy Directing Surface X l , Y l , Z l In a first 5D coordinate resulting from the convergence of energy along multiple energy propagation paths through, this coordinate may represent a point within a larger object, volume, particle, or local energy parameter, and the energy converging at additional coordinates proximate to the first 5D coordinate may exhibit additional vectorized properties relative to the sensed energy in the environment or holographic dataset. These vectorized properties may include information about each 5D coordinate, each energy position coordinate within a 4D dataset, an area within either the 4D or 5D dataset, or other subsets of coordinates that include an energy surface.
[0178] In one embodiment, a universal parameterization of 4D and 5D holographic sensory energy properties for propagation of visual energy, auditory energy, somatosensory energy, gustatory energy, olfactory energy, vestibular energy, or other desired energy for a sensory system response is disclosed for raster and vector 2D, 3D, 4D and 5D data sets, where with respect to a second coordinate of a second face of the 4D energy directing surface, the 2D data can include a single angular sample, the 3D data can include two or more angular samples in a single dimension, the 4D data can include multiple angular samples in two dimensions, or the 5D data can include multiple angular samples in three or more dimensions. Embodiments of received sample data may include any of the following: 1) 2D or monoscopic, flat, point cloud, UV mapping geometry, intrinsic geometry, depth image, layered image, CAD file (intrinsic), single point sampling, single camera capture, single projector projection, volumetric (monoscopic single sample point with vector in volume), 3 degree of freedom sources (raster with DoF, monoscopic x,y,z rotation around a single point), non-bright field 6 DoF sources (raster + vector from monoscopic samples), volumetric energy directing device (monoscopic samples in volume), Pepper's Ghost sources (single point projection), 2D AR HMD sources (monoscopic single or multiple focal planes, layered monoscopic), 2D VR HMD sources (monoscopic single or multiple focal planes, layered monoscopic), or any other representation of 2-dimensional raster or vector information.2) 3D or stereoscopic, triscopic (single baseline), multi-view (1D), 1D multi-sample, 1D multi-view, horizontal or vertical only parallax, 1D projected array, two-point sampling, 1D point sampling, horizontal or vertical array, bullet-time, source of 3DoF (raster, stereoscopic x,y,z rotation around a single point), source of 3DoF (3D raster in stereoscopic x,y,z rotation around a single point), source of non-brightfield 6 DoF (3D raster + vector from stereoscopic samples), source of 1D volumetric energy directing device (1D parallax containing samples), source of autostereoscopic data, source of horizontal multi-view energy directing device, source of 3D AR HMD (stereoscopic single or multiple focal planes, layered stereoscopic), source of 3D VR HMD (stereoscopic single or multiple focal planes, layered stereoscopic), or any other representation of three-dimensional raster or vector information. 3) 4D or plenoptic (5D), multiscopic, integral image, brightfield (4D), holographic (4D), 2D multiview, 2D multisample, 2D multiview, 2D parallax, horizontal and vertical parallax, 2D projected array, 2D point sampling, motion capture stage (along a surface), planar array, eyewitness camera array, rendered or ray-traced shape representation (4D representation), external geometry (4D representation), brightfield 6 DoF source (4D raster in planar brightfield samples), free-viewpoint 6 DoF source (4D raster + vector from 4D brightfield samples), 4D volumetric energy directing device source (2D parallax containing samples), brightfield energy directing device source (4D sampling), brightfield HMD source (near-field 4D sampling), Holodeck ray-traced device source (4D sampling), or any other representation of 4 dimensional raster or vector information.4) 5D or plenoptic + depth, brightfield + depth, holographic (5D sampling, 4D + depth), arbitrary multi-view (along all x, y, and z axes), multi-sample (along all xyz), multi-viewpoint (along all xyz), volumetric parallax (along all xyz), projected array (along all xyz), point sampling (along all xyz), motion capture stage (along all xyz), witness camera array (arbitrary xyz configuration), rendered or ray-traced shape representation (5D representation), cubic or volumetric rendering (along all xyz), extrinsic geometry (5D representation), brightfield 6 DoF source (5D raster in volumetric brightfield samples), free viewpoint 6 DoF sources (5D raster + vectors from 5D brightfield samples), 5D volumetric energy directing device sources (multiplanar 4D sampling), 5D brightfield energy directing device sources (5D sampling, 4D + multi-planar), 5D brightfield HMD sources (near-field 5D sampling, 4D + multi-planar), Holodeck energy directing device sources (5D sampling, 4D + multi-planar), or any other representation of 5 dimensional raster or vector information.
[0179] At each second coordinate, the data provided may include a subset or superset of either the raster samples or the vector samples, and the samples may represent or include additional vectorization information that enables conversion to increased sampling density through interpretation or processing of the subset or superset of the raster or vector samples.
[0180] For each of the 2D, 3D, 4D, or 5D provided data sets, the information is transformed by vectorized information, manual identification, computer vision analysis, automated processing, or other means of transforming the provided samples from the original data set to the 5D coordinate system. For each of the 2D, 3D, 4D, or 5D provided data sets, the information can include a plurality of samples or layers of samples and additional vectorized characteristics of the original angular sampling components for each provided data set, or a combination of contributing samples for any of the 2D, 3D, 4D, or 5D additional provided data sets, with reference to a second coordinate of a second plane of the 4D energy directing surface.
[0181] Each of the provided samples includes an intrinsic energy for each desired coordinate, which may include additional extrinsic energy attributes, where the intrinsic energy represents the value at a given 5D coordinate in the absence of other external samples, characteristics, or environmental conditions. In the electromagnetic spectrum, this corresponds to a white body that reflects all incident radiation, and is sometimes referred to as albedo, a dimensionless measure of reflectance whose range of dimensionless values apparently spans each desired sensory energy corresponding to a particular sensory energy. Within the visual system, this range is approximately 400 nm to 700 μm, and within the auditory system, this range is approximately 20 Hz to 20 kHz.
[0182] Over the past few decades, there have been enormous technological improvements that allow for the reproduction of human senses, artificially leveraging advanced pattern recognition of senses, scents, and flavors by electronic means. In other systems that may exist outside the electromagnetic spectrum, these dimensionless values can be characterized in the same way based on the sensed sharpness response. Holographic sensory energy technology is emerging, but the disclosed within this embodiment includes systems, methods, and formats for stimulating all human senses in a virtual environment, articulating a universal construction for various sensory parameters, and in so doing, providing for the appropriate processing, transmission, storage, vectorization, and transfer of data to, from, or between any sensory energy parameters or devices desired for the full population of the constructed virtual environment, and embodiments of energy propagation for holographic sensory technology will be disclosed in future applications. Furthermore, it is the intent of this disclosure to enable other analog devices, including quirky commercial products such as the classic "Smellovision," or modern versions such as the FeelReal smelling VR headset, to utilize the parameterized values provided within the vectorization of the datasets herein.
[0183] In one embodiment, the somatosensory system may be defined based on components that define sensitivity, including mechanoreceptors for texture with a pressure sensitivity range in the skin that may be normalized from 50 Hz to 300 Hz, thermoreceptors with a temperature sensitivity range in the skin that may be normalized from 0°C to 50°C (although this range may be much broader with upper and lower limits defined by excessive temperatures), or surface deformability that defines the range of viscoelastic behavior of materials measurements, both viscous and elastic, as they undergo deformations between stress and strain over time, providing a number of physical properties including variables for time, strain, elastic modulus among other dynamic properties, simplified for purposes of this disclosure to a dimensionless normalized scale having a value of 0 for immobile solids such as granite and 1 for low viscosity fluids such as water. Those skilled in the art will appreciate that the actual vectors provided include the physical properties necessary to properly define the viscoelasticity of a material and are normalized for illustrative purposes only.
[0184] Finally, state of the art advances in artificial electronic sensing, including taste and olfactory devices, indicate a viable path to further vectorize the sensory parameters disclosed for the Holodeck design parameters and enable electronic reproduction of artificial tastes and olfactory sensations through holographic waveguiding means as described herein. Artificial electronic taste and olfactory receptors have made considerable progress through emerging nanodevices, including frequency-based artificial taste receptors that use enzyme biosensors to sample the intensity of chemical stimuli through encoding and conversion into frequency-based pulses that repeatedly and precisely detect tastes as frequencies of the sampled chemical composition through pattern recognition systems that result in detection of the tastes that constitute human taste. It is believed that this technology can be extended to any type of detectable taste, and similar advances in artificial olfactory systems have demonstrated digital interfaces to stimulate the human olfactory receptor body using weak electrical pulses targeted to the nasal turbinates, along with ongoing work to further parameterize the patterns contained within the frequencies of specific olfactory responses through changes in electronic signals.
[0185] Pathways are established for the arbitrary generation of frequencies and complex electronic patterns representing olfactory, gustatory and other sensory systems, and in one embodiment, taste acuity responses can be vectorized to include normalized scales for each of the electronically controlled parameters along a scale of 0 to 1 representing the minimum and maximum taste responses to saturate the average human's 2,000 to 8,000 taste buds, which may include, but are not limited to, vectors for sour, salty, bitter (spicy), sweet, and savory (umami), and the vectors and spatial coordinates of the vectorized signals can inform the fabrication of complex olfactory realizations.
[0186] In another embodiment, a highly complex olfactory space, which may include, but is not limited to, vectors for pleasant, fruity, citrus, woody (resinous), medicinal, sweet, minty (peppermint), toasty (nutty), pungent, and rancid odors, is constructed using an average human 10 cm vector for each of the odors. 2 The olfactory acuity response can be further vectorized to include normalized scales for each of the electronically controlled parameters along a scale of 0 to 1 representing the minimum and maximum olfactory responses to saturate the olfactory epithelium, and the vectors and spatial coordinates of the vectorized signals can inform the fabrication of complex olfactory realizations.
[0187] Each of these vectors may provide normalized values representing these patterns for taste, smell, or other sensory domains that have been converted to waves, amplitudes, magnitudes, or other attributes as necessary for appropriate application of the provided vectorization values. Although olfaction and taste are two of the most degraded senses within the sensory system, with parameterized values for vectorizing complex fusions, it is further possible in embodiments to provide user-based interactive control of the sensitivity of any such sensory energy to provide customization of the individualization of each of the visual, auditory, somatosensory, taste, olfactory, vestibular, or other desired sensory system responses.
[0188] In one embodiment, each of the represented sensory albedo energy values of a sample may further include extrinsic energy attributes baked into a single sample value representing the summation of each sample provided for each of the other external samples, properties or environmental conditions. In this configuration, the composite sample value may or may not indicate potential attributes of other energies from other samples in a physically based or simulated environment. The most efficient and pure method for transmitting parameterized and reconstructed holographic datasets is based on single intrinsic sample information that provides simplified and lower bandwidth frequency information, but this method is not always possible to receive outside of a completely synthetic environment, especially in physically based imaging or acoustic systems. In any real-world environment, there is always some degree of extrinsic contribution to the resulting sample information. Certain systems such as light stages, or other systems known in the art to facilitate estimation of reflectance, shape, texture, and motion capture utilize some form of structured illumination and one or more imaging devices that provide direct or indirect analysis of surface properties such as albedo, depth information, surface normals, and bidirectional scattering distribution.
[0189] The bidirectional scattering distribution function (BSDF) is often expressed as a distribution f r (w i , w r ), which collectively act as a model for parameterizing and identifying surface properties in computer graphics and vision algorithms known in the art. This function is expressed as a function of the incident light direction w of the incoming light relative to the energy propagation path when the surface normal is perpendicular to the tangent of the target surface. i and the reflection or transmission direction of the emitted light w r describes how visible light is reflected, transmitted, or otherwise interacts with a surface given the exit path w r The reflected radiance incident along the outgoing path wi describes the fraction of irradiance incident on a surface along i , w r Each of the vectors may include a 4D function that defines the parameterized azimuth and zenith angles of each of the incoming and outgoing paths.
[0190] Energy hitting the surface λ i The first position x i , and the material properties reduce this energy to energy λ r The second position x r These functions can be further specified for the exit after internal scattering to account for visible wavelength effects such as iridescence, luminescence, subsurface scattering, non-local scattering effects, specular reflection, shadowing, masking, interreflection, etc., resulting output energy, input energy and position based on the material properties of the surface, and output energy and position across the surface of an object, volume or point.
[0191] Thus, a generalized property to describe how energy is transported between any two energy rays striking a surface, to include wavelength or frequency dependence, and spatially varying material properties or surfaces, can be expressed as a 10D function, f, for each or any of the available or provided samples in the data set. r (λ i , x i , w i , λ r , x r , w r ) to express the input energy, vectorized surface shape effects, and output results for reflection, refraction, specularity, transmission, scattering, diffusion, or other material properties from any energy domain as a function of the generalized function f r can be explained when given.
[0192] Now considering an energy directed surface, the plenoptic4D function contains the energy position and a vector of energy propagation paths f l (x l , y l , u l, v l ) is defined by the guided wave parameter u l , v l Two spatial coordinates x from the first plane, oriented through a second coordinate along a second plane including l , y l Considering multiple energy directing surfaces, the plenoptic5D function contains one or more energy locations and a vector of energy propagation paths f l (x l , y l , z l , u l , v l ) is defined by the guided wave parameter u l , v l A three-dimensional coordinate system is defined from the first coordinate to three spatial coordinates x, oriented through the second coordinate along a plane that contains l , y l , z l For 4D or 5D respectively, additional variables f for time and color l (λ l , t l ), functions and even if not explicitly noted for ease of explanation, can be considered to include any of the plenoptic functions if necessary for the application.
[0193] Along the first vector of the energy propagation path, multiple intersections involving energy convergence may occur with additional energy propagation paths. At the intersections, a 3D point or depth parameter is a position X between multiple energy propagation paths with a 4D or 5D plenoptic function. l , Y l , Z l where each x contained within the energy-directed 4D or 5D surface l , y l or x l , y l , z l For a coordinate, a single u is formed between the first coordinate and the 3D convergence point. l , v l 3D convergence point X between multiple energy propagation paths, where only the propagation path angle exists l, Y l , Z l 4D function f z (x l , y l , u l , v l ) or 5D function f z (x l , y l , z l , u l , v l ) is available for all 4Dx l , y l or 5Dx l , y l , z l coordinates, and X l , Y l , Z l For each convergence point of l , v l Collectively define the propagation path.
[0194] Convergence coordinate X l , Y l , Z l A surface is formed on the surface, which may include points, volumes, objects or other embodiments that include 3D locations of the focused energy propagation paths. The samples provided for each surface location are then analyzed to determine one or more surface features, vectors, materials, characterizations or other distinguishing properties V for characterizing or otherwise processing the resulting energy. i , as well as one or more input energy sources hitting specific points proximate to the surface location, where the reflectance function is a 11D universal object parameterization function f r (λ i , x i , w i , λ r , x r , w r , V i )
[0195] 11D universal holographic parameterization function f r (λ i , x i , wi , λ r , x r , w r , V i ) specifies the resulting values for a particular environment and vectorized properties, and the 4D function f l (x l , y l , u l , v l ) defines the energy propagation path from the energy directing device surface, and therefore the 15D universal holographic parameterization function f r (λ i , x i , w i , λ r , x r , w r (x l , y l , u l , v l ), V i ), where the transmitted direction w r Ga u l , v l The propagation path of x is defined and made equal, so that the spatial coordinate x l , y l and each transmitted direction w r Well then, w r =u l , v l There is only one f that satisfies l (x l , y l , u l , v l ) set of values. Those skilled in the art will understand the various transformations and mathematical constructs as well as the requirements for rendering associated with the disclosed universal parameterization of 4D and 5D holographic sensory energy properties.
[0196] With a complete 15D function shown to vectorize all sensory energy properties to coincide with a surface formed from convergence points in space, multiple orders of required data are radically omitted, provisioning a viable pathway to enable the transmission of truly holographic data sets.
[0197] Vectorized characteristics aim to provide precise physical characteristics of each of the sensory regions for characteristics that can be synthetically programmed, captured, or evaluated by a computer, V i For general system metadata or for specific samples in the dataset provided for each or any sensory energy domain, including: l , Y l , Z lAttributes of vectorized features can be specified in 1) System metadata can provide either sensory energy specific attributes or system wide references for surface features of each sample, including normals, depth information, environmental features, multiple angle samples for a given 3D coordinate, procedural textures, geometric shapes, point clouds, deep image data, static frames, time frames, video data, surface IDs, surface paths, coordinate maps, virtual camera coordinates, virtual lighting and visible energy information, environment maps, scene information outside the field of visual sample information, curves, vertices, time information, network data, databases, object recognition, energy devices, external data feeds, sensors for system modification and interaction, system status, voice recognition, olfactory detection, auditory detection, face recognition, somatosensory recognition, taste recognition, UI, UX, user profiles, flow and motion vectors, layers, regions, transparency, segments, animations, sequence information, procedural information, displacement maps, or any other scene data necessary to provide sufficient data for proper processing of each sample. 2) Visual energy may provide surface properties to prescribe proper rendering of visible or non-visible electromagnetic energy, iridescence, luminescence, subsurface scattering, non-local scattering effects, specular reflection, shadowing, absorbance, transmittance, masking, interreflection, albedo, transparency, physical properties, dynamic properties, reflection, refraction, diffraction, optical effects, atmospheric effects, frequency, modulation, surface shape, texture, displacement maps, physical and dynamic properties to specifically interact and respond to other sensory energies based on the provisioned energy (e.g., sound vibrations changing reflectance properties, or deformation of tactile materials causing deformation of a surface), layers, regions, transparencies, segments, curves, animations, sequence information, procedural information, material sizes, environmental conditions, room dynamics, or other material properties associated with a surface, environment, room, object, point, volume, etc.3) Auditory Energy: Vectors relating to local sound field geometry, size, amplitude, mass, material propagation parameters, material properties indicating absorbance, transmittance, acoustic reflectance, diffusion, transmittance, enhancement, masking, scattering, localization, frequency dependence or modulation, pitch, tone, viscosity, smoothness, texture, elasticity, any other parameters determining the propagation of sound waves within an object, surface, medium or other, physical and dynamic properties (e.g. temperature changing the sound of a material) specifically to correlate and respond to other sensory energies based on the provisioned energy, layers, regions, transparency, segments, curves, animations, sequence information, procedural information, material size, environmental conditions, room dynamics or other material properties related to surfaces, environments, rooms, objects, points, volumes, etc. 4) Surface deformability parameters and vectors that provide many physical properties including mechanical receptors for texture, pressure, thermoreceptors, temperature, variables for time, strain, elastic modulus among other dynamic properties and define the range of viscoelastic behavior of materials measurements, both viscous and elastic, as they undergo deformations between stress and strain over time, layers, areas, transparency, segments, curves, animations, sequence information, procedural information, material size, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other somatosensory parameters. 5) Taste energy vectors for pleasant, fruity, citrus, woody (resinous), medicinal, sweet, minty (peppermint), toasty (nutty), pungent and spoiled odors, where the vectors and spatial coordinates of the vectorized signals can inform the fabrication of complex olfactory realizations and further provide duration, magnitude, frequency, length, time, radius, modulation, layers, areas, transparency, segments, curves, animations, sequence information, procedural information, material size, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other taste parameters.6) Olfactory energy vectors for sour, salty, bitter (spicy), sweet, and savory (umami), where the vectorized signal vectors and spatial coordinates can inform the fabrication of complex olfactory realizations, further providing duration, magnitude, frequency, length, time, radius, modulation, layers, areas, transparency, segments, curves, animation, sequence information, procedural information, material size, environmental conditions, room dynamics, or other material properties related to surfaces, environments, rooms, objects, points, volumes, or other taste parameters; 7) Or other interrelated sensory dynamics based on physical, synthetic, conveyed, or computational interdependencies from any other sensory sample data set, required, designed, or requested sensory vectors, and any additional sensory characteristics where parameterization of specific features is beneficial for the reconstruction, storage, processing, or transmission of generalized holographic configuration data.
[0198] Given a received dataset containing 2D data with a single angle sample, 3D data with two or more angle samples in a single dimension, 4D data with multiple angle samples in two dimensions, or 5D data with multiple angle samples in three or more dimensions.
[0199] For all provided source materials, each source material may undergo additional processing to properly prepare it for efficient vectorization of the holographic data set. For any provided sheath material that exhibits lower spatial or angular resolution of its energy directing aspects, a conversion process may be required to accurately convert the original source into a 4D or 5D data set.
[0200] For proper preparation, in one embodiment, the 2D or 3D source material provided includes photographic capture from a standard imaging system. Within this sequence of images, there are raster reflections, refractions, transparent elements, and other similar examples of material property interactions with physically based lighting.
[0201] If content is prepared by simply identifying surface IDs for surfaces that already have rasterized material properties, the valid data may be sufficient to converge to a 4D coordinate system, but any additional rendering applied to these surfaces will show double images due to the physical properties of the photographic properties, and the parameterized, synthetically rendered reflectance properties. An ideal source data set for efficient holographic transmission would include an albedo representation of the sample source information, and vectorized material properties for each of the specific energy regions with metadata that form an object-based volumetric sampling of the albedo multiview samples, with all material properties precisely based on the specific vectorized surface properties to provide accurate surface identification and rendering, as well as localization or projection of other sensory energies.
[0202] In one embodiment, a manual, semi-automated, computer vision, or automated process is provisioned to algorithmically or manually evaluate the content in the source sample dataset, and manual or algorithmic analysis is performed to perform segmentation and other object separation methods known in the art to identify regions that contain undesired physical rasterization effects. In one embodiment, a person is photographed in front of a background, where the material characteristics of the person include reflections from the environment, and the background objects are occluded by the photographed person.After these regions are identified as undesirable, a process may be utilized to 1) isolate the object in question, 2) isolate all object elements to account for occlusions, transparency, edges, or other features, and 3) through the facilitation of machine learning, computer vision, and additional hardware and energy devices that have additionally captured information about the scene, object, and / or environment, through image analysis, time analysis, energy analysis, or through entirely manual means, the object elements are provisioned such that any surfaces that should exhibit material properties have had any such baked-in material properties removed by computer vision, algorithms, processors, or manual visual effects, where the manual process recreates intrinsic material properties in the absence of extrinsic material properties to allow for the most efficient transmission and processing of said dataset. 3) The above additional processes include manual or computer-assisted identification of depth or 3D coordinate values for each of the desired samples; and 4) the above additional processes include manual or computer-assisted identification of depth or 3D coordinate values for each of the desired samples. 5) Further within the scope of this embodiment is the identification of associated material properties, each of which represents a point, data region, surface, object, or other representation of a material, such that the data may be easily further rendered within a display driver of the energy directing device or any additional system capable of encoding or decoding the parameterized data set.
[0203] In one embodiment, the dataset from above includes 3D multi-view samples prepared with albedo visual energy samples, each having multiple layers of rgba information, a set of vectorized material properties to associate each segmented material with a surface ID, and a set of surface parameters to closely reconstruct the original source dataset before removal of the exogenous image data, and the acoustic dataset is provisioned with material properties of the visual energy system, as well as vectorized material properties associated with multiple acoustic channels, each having identified frequency, modulation, spatial location and other sound source localization characteristics, and a somatosensory energy dataset is provided for a subset of surfaces included within the visual energy dataset, further including both viscoelastic and temperature vectorized material properties, both related to other vectorized datasets.
[0204] From any provided data set, each provided sample from the visual energy data set is evaluated for a relative depth position with respect to a surface of the energy directing device, and for any of the visual energy samples, each of the samples is placed in a 3D coordinate system, and an energy propagation path length for each provided sample is calculated based on a position X among a plurality of energy propagation paths within a 4D or 5D plenoptic function. l , Y l , Z l where each x x contained within the energy-directed 4D or 5D surface is evaluated with respect to a function that correlates each 3D coordinate with respect to multiple coexisting converging energy propagation paths that intersect with the first 3D point at l , y l or x l , y l , z l For coordinates, a single u is formed between the first coordinate and the 3D convergence point. l , v l Only the propagation path angle exists. 4D function f z (x l , y l , u l , v l ) or 5D function f z(x l , y l , z l , u l , v l ) is a 4D x l , y l , or 5D x l , y l , z l coordinates, and X l , Y l , Z l For each convergence point in l , v l The propagation paths are collectively defined and, after this analysis process is performed, the 4D x l , y l , or 5D x l , y l , z l The total number of samples per spatial coordinate is now known, and the position X l , Y l , Z l The total energy propagation path length between each 3D point in and the 4D or 5D coordinate location is now known, and a weighted distribution based on the total available samples per 4D or 5D coordinate, and the minimum path length to a sampled 3D coordinate value from the multiple available 3D coordinate data provides a complete sampling of the 4D or 5D bright field from any data set.
[0205] As a further embodiment of the above, 1) after each of the samples for any of the visual, auditory, somatosensory, and any other provided energy samples are 2) placed in a 3D coordinate system based on the provided dataset, additional processing, or additional vectorized characteristics, and before performing the coordinate analysis, 3) a 15D universal holographic parameterization function f r (λ i , x i , w i , λ r , x r , w r (x l , y l , u l , v l ), Vi ) is evaluated; 4) additional known environmental scene, geometry, metadata, etc., each having independent vectorized material properties, are provided; 5) virtual lighting information is provided and additional sensory energy metadata properties are evaluated for any possible interference between properties that may alter the rendering function; 6) a 15D parameterization function is evaluated for each provided 3D coordinate and corresponding vectorized material property; 7) given any provided dataset, a rendering process is performed via online, offline, real-time, ASIC, FPGA, cloud, or other form of rendering process to result in new multiple angle-varying material properties, where 8) this rendering process includes a propagation path u l , v l and define a transmission direction w r Thus, the spatial coordinate x l , y l and for each transmission direction w r In contrast, w r =u l , v l There is only one set of f that satisfies l (x l , y l , u l , v l 9) based on the results of the rendering and the resulting available new angle-varying material properties, for each of the 4D or 5D coordinates that contain the energy propagation path length for each of the provided samples, a position X among the multiple energy propagation paths in the 4D or 5D plenoptic function is calculated. l , Y l , Z l where each x x contained within the energy-directed 4D or 5D surface is evaluated in relation to a function relating each 3D coordinate to multiple coexisting converging energy propagation paths intersecting a first 3D point in l , y l or x l , y l , z l For a coordinate, there is only one u that forms between the first coordinate and the convergent 3D point.l , v l Only the propagation path angle exists. 4D function f z (x l , y l , u l , v l ) or 5D function f z (x l , y l , z l , u l , v l ) is a 4D x l , y l , or 5D x l , y l , z l coordinates, and X for each convergence point l , Y l , Z l There exists a corresponding u l , v l The propagation paths are collectively defined and, after this analysis process is performed, the 4D x l , y l , or 5D x l , y l , z l The total number of samples per spatial coordinate is now known, and the position X l , Y l , Z l The total energy propagation path length between each 3D point in and the 4D or 5D coordinate location is now known, and a weighted distribution based on the total available samples per 4D or 5D coordinate, and the minimum path length to a sampled 3D coordinate value from the multiple available 3D coordinate data provides a complete sampling of the 4D or 5D bright field for all provided sensory energy from any data set.
[0206] Further embodiments of the above-described systems are evaluated in which the rendering further reveals a two-way energy directing surface such that sensed electromagnetic energy representing the illumination of the real-world environment, or the absorbance of certain sound frequencies within the environment, can result in dynamic or offline updates to the rendering process or other sensed interactive real-world elements, and lighting and sound or other sources are adjusted to adapt to changing environmental conditions.
[0207] Referring again to FIG. 8 and taking into consideration the above principles, in one embodiment of process 800, the received content data may further include vectorized material property data, and process 800 further includes step 830, in which a digital volumetric representation of the content data is associated with the vectorized material property data, and in step 804, determining the energy source position value is based at least on the vectorized material property data associated with the volumetric representation of the content data.
[0208] 9 and 13, in one embodiment, the vectorization process 1300 may include step 1302 in which first content data is received and step 1304 in which a surface 915 is identified in the content data. In one embodiment, identifying the surface 915 may include using segmentation data in the content data. The vectorization process 1300 may further include step 1306 in which a surface identification of the surface 915 is determined and step 1308 in which material property data of the surface 915 is determined. In one embodiment, determining the material property data may include manual determination or using a predetermined process. After steps 1306 and 1308, the vectorization process 1300 may further include step 1310 in which the surface identification is associated with the material property data of the surface 915. The vectorization process 1300 may further include step 1312 in which a vector of material property data is generated. The vectorization process 1300 may further include step 1314 in which vectorized material property data is generated based on the generated vector.
[0209] In one embodiment, process 1300 may optionally include step 1316, in which the material property data is removed from the first content data and replaced with the material property data vectorized in step 1314. In one embodiment, the material property data vectorized in step 1314 may be used in process 800, as described above, to determine 4D plenoptic coordinates of an energy directing device of the present disclosure, as described above.
[0210] Process 1300 may be performed using any processing system of the disclosure, including processing system 1200. In one embodiment, content data may be received via data input / output interface 1201 in step 1302, and steps 1304-1314 of vectorization process 1300 may be performed using vectorization engine 1204. Additionally, vectorized material property data generated in step 1314 may be used by sensory data processor 1202 and tracing engine 1206 for processing according to the steps of process 800 as described above. Steps 808 and 812 may be performed by the tracing engine to determine 4D coordinates for the holographic display. Step 810 may be performed by sensory data processor 1202. The output of the processing subsystem may be provided to compression engine 1210, from which the compressed data may be stored in memory or provided to data input / output interface 1201 for transmission to an energy-directing system connected locally or remotely to system 1210. Data may also be stored in memory 1208 until later retrieval.
[0211] Rendering a 4D Energy Field from a 3D Environment The present disclosure further contemplates embodiments of an inverse tracing process that enable a digital volumetric representation to be rendered into a format compatible with energy directing apparatus 1000 by inverse mapping, which provides an inverse path for sampling a digital volumetric representation of content data that can be presented as a 3D environment.
[0212] Rendering a 4D energy field from a 3D environment poses many challenges. For purposes of explanation, these challenges are described in the context of bright field. However, the disclosure of this application is applicable to other forms of energy, and these discussions do not limit the application of these principles to other energy forms.
[0213] Energy in a 3D environment interacts with objects in the 3D environment in a variety of different ways depending on the physical properties of the objects in the 3D environment, which may model how the energy is generated, reflected, absorbed, transmitted, or affected by the objects. FIG. 14 illustrates how any interaction between energy and objects throughout a 3D environment affects the perception of a scene from that environment. Shown is an observation point 1402 of a virtual observer in a 3D environment 1400 having a field of view 1404. An energy ray 1406 can reach the observation point directly from an energy source 1408 or by bouncing off a reflecting object 1410A, 1410B, 1410C, which affects the attributes of the energy as it reaches the observation point 1402. As described throughout this disclosure, including paragraph 203, various attributes of the energy of the 3D environment 1400 may be considered, including visual, auditory, somatosensory, gustatory, olfactory, vestibular, or other desired energy. It should be noted that FIG. 14 only partially depicts some energy paths toward an observation point. As will be understood, sources for some of the energy paths and other reflections may be removed from the diagram. In some cases, a single energy path 1406A reflects off multiple objects 14010A, 1410C on its way to the observation point 1402, some of which 1410C may be outside the field of view 1404 of the observation point 1402. On its way to the observation point 1402, the energy 1406 may also pass through partially transparent objects 1412, or refractive objects (not shown) that change how the energy is perceived as it reaches the observation point 1402. Thus, an accurate rendering of a scene from a 3D environment may take into account the multiple interactions that occur both within and outside the field of view 1404 from the observation point 1402 of a virtual viewer in the 3D environment. This may be even more complicated in a holographic system, as the holographic system may require the simultaneous projection of multiple unique energy convergences. FIG. 14 is presented for illustrative purposes and oversimplifies some possible implementations for ease of explanation.It will be understood that a 3D environment may account for many additional physical characteristics that affect the propagation of energy within the 3D environment, and FIG. 14 is not limiting to embodiments of the present disclosure.
[0214] Prior art techniques for rendering 4D energy field data from a 3D environment include double frustum rendering, tilted rendering, and multi-view rendering. Double frustum rendering is a multi-pass rendering method that cannot account for data outside the field of view of the observer point and requires computationally complex post-integration of data from multiple rendering passes. Tilt rendering also requires complex post-integration calculations, requiring thousands to tens of thousands of rendering passes, and also makes unrealistic assumptions about the physical energy projection system, such as perfect projection optics and no aperture. Tilt rendering also cannot be easily integrated into existing workflows. And finally, multiple view rendering does not allow for simultaneous rendering from and into the field of view, and exhibits some of the other problems exhibited by double frustum rendering and tilted rendering.
[0215] The embodiments herein enable systems and methods for single-pass rendering of a 3D environment into a dataset for 4D energy projection. At the very least, the embodiments of the present disclosure generate a significantly smoother, more cohesive energy field projection because the single-pass rendering enabled by the present disclosure avoids computationally challenging post-integration efforts; and because the methods and systems of the present disclosure account for both on-screen and off-screen effects. In embodiments, the systems and methods may include wavelength / energy considerations and display-specific calibration (or any other function to move seamlessly between various systems). This may include pixel processing with a displacement map in addition to calculate if and when other 4D coordinates are advantageous.
[0216] FIG. 15 is a flow chart illustrating an embodiment of a process for rendering a 4D energy field from a 3D environment. With reference to FIG. 15 and FIG. 16, the process may include a first step 1510 for providing a scene 1600 of a 3D environment described by a plurality of energy data points 1602 located throughout the scene 1600. A visual representation of the scene 1600 of a 3D environment is depicted in FIG. 16. As can be appreciated, the energy data points 1602 may be configured in different ways in different embodiments of the 3D environment. The energy data points 1602 may include a value or a group of values describing various physical properties that determine the virtual appearance of the scene 1600 and how elements within the scene 1602 may interact. Some examples of this are color, reflectance, material properties, and energy direction. Other properties that may be considered within the scene 1600 may include visual, auditory, somatosensory, gustatory, olfactory, vestibular, or other desired energies. In some embodiments, the energy data points may store multi-dimensional functions. The functions may include values corresponding to properties of the material world that describe how energy may be reflected, refracted, transmitted, or affected when it intersects with the energy data points 1602. And, in some embodiments, the energy data points 1602 may also include intensity and directional information that collectively allows the energy data points to model the propagation of energy within the scene 1600. It should also be noted that the energy data points 1602 may include energy values for other types of energy in addition to or instead of information related to electromagnetic energy in the visible spectrum. There are no limitations on the type of information that may be stored as values of the energy data points 1602.
[0217] It will be appreciated that there are various ways to configure or arrange the energy data points 1602 in the 3D environment. The density of the energy data points 1602 in the 3D environment 1602 may vary from embodiment to embodiment. Or, in some embodiments, the density of the energy data points 1602 may vary from position to position in the 3D environment 1600 or from object to object in the 3D environment 1600.
[0218] In some embodiments of the 3D environment 1600, the energy data points 1602 may correspond to locations in virtual or physical space. And the values of the energy data points 1602 may change as different objects occupy the space in which the energy data points are mapped. The values of the energy data points 1602 may also change as light propagates through an embodiment of the 3D environment. And again, while FIG. 16 represents a scene of an electromagnetic 3D environment, it should be noted that the disclosure is not limited to energies within that region. An embodiment of the 3D environment may account for other types of energy in addition to electromagnetic energy in the visual spectrum, such as mechanical energy in the form of ultrasound that may be used to create tactile surfaces. Other energies accounted for may include visual, auditory, somatosensory, gustatory, olfactory, vestibular, or other energies.
[0219] 15 and 16, in an embodiment, a process for rendering a 4D energy field from a 3D environment may include a second step 1520 that includes arranging a plurality of virtual pixels 1604 in a virtual pixel plane 1606 in a scene 1600, each pixel having known unique 4D coordinates including 2D angular coordinates and 2D spatial coordinates that describe the angular correlation between each virtual pixel 1604 and a virtual viewpoint of a plurality of virtual viewpoints 1608 arranged in a virtual viewing plane 1610 (shown in FIG. 16 by the dashed rectangular box labeled 1610) in the scene 1600, and the 2D spatial coordinates that identify the location of a virtual aperture of a plurality of virtual apertures 1612 arranged in a virtual display plane 1614 in the scene 1600. In some embodiments, the virtual viewpoint 1608 may be arranged in two or more virtual viewing planes allowing for rendering coplanar devices.
[0220] It will be understood that the term "virtual pixel" does not limit embodiments of the present disclosure to rendering a 3D environment with a bright field. A virtual pixel may comprise an energy location of a 3D environment with any form of detectable energy field, including, but not limited to, tactile and acoustic fields. Other energy fields may include visual, auditory, somatosensory, gustatory, olfactory, vestibular, or other desired energy fields. A plurality of virtual pixels 1604 may be arranged to form a two-dimensional array of virtual pixels. In some embodiments, a plurality of virtual pixels 1604 may be arranged in different ways. For example, embodiments allow for the virtual pixels to be arranged to form a curved surface or any other shape. Similarly, a plurality of virtual viewpoints 1608 may be arranged to form different shapes in different embodiments. In some embodiments, the virtual pixel plane 1606 and the virtual display plane 1614 are parallel. In some embodiments, the virtual pixel plane 1606 and the virtual viewing plane 1610 are parallel. In some embodiments, the virtual pixel plane 1606, the virtual display plane 1614, and the virtual viewing plane 1610 are parallel.
[0221] The positions of the virtual pixels 1604 and virtual apertures 1612 relative to the virtual viewpoints 1608 in the scene 1602 may correspond to the positions of the energy sources or locations relative to the intended viewing volume for that 4D energy projection system and the apertures of the physical 4D energy projection system. And in some embodiments, the number and positions of the virtual pixels 1604, virtual apertures 1612, and virtual viewpoints 1608 may vary to render data for different 4D energy projection systems depending on the number of energy sources or locations, field of view, number of apertures, waveguides, or other characteristics of the physical 4D energy projection system. In some embodiments, an algorithm may be used to calibrate the defined number and positions of the virtual pixels 1604, virtual apertures 1612, and virtual viewpoints 1608 for any type of 4D energy projection system. It will also be understood that while FIG. 16 depicts only a small number of virtual pixels 1604, virtual apertures 1612, and virtual viewpoints 1608, an embodiment may comprise tens, hundreds, thousands, or more of each of these elements. Additionally, the virtual display plane 1614 may correspond to the location of an energy projection surface or screen within a physical 4D energy projection system.
[0222] 15 and 16, in an embodiment, the process for rendering a 4D energy field from a 3D environment may include a third step 1530 that includes sampling energy data points of a plurality of energy data points 1602 in the scene 1600 from a virtual viewing plane 1610 along a plurality of rays 1616. Each ray of the plurality of rays 1616 intersects with one virtual viewpoint of the plurality of virtual viewpoints 1608 and one virtual pixel of the plurality of virtual pixels 1606 at an angle determined by the 2D angular coordinate of one virtual pixel of the plurality of virtual pixels 1604. Each ray of the plurality of rays 1616 intersects with one virtual aperture of the plurality of virtual apertures 1612, which determines the 2D spatial coordinate of the ray. Together, the 2D spatial coordinate and the 2D angular coordinate form a single 4D energy field coordinate. For example, a first ray 1618 of the multiple rays 1616 intersects a first virtual pixel 1620 of the multiple pixels 1604 and a first virtual viewpoint 1622 of the multiple virtual viewpoints 1608 at an angle determined by the 2D angular coordinates of the known unique 4D coordinates of the first virtual pixel 1620. The first ray 1618 also intersects a first aperture 1624 of the multiple virtual apertures 1612, which is identified by the 2D spatial coordinates of the first aperture.
[0223] FIG. 17 further illustrates the relationship represented by known unique 4D coordinates of virtual pixels in some embodiments of the method or system for rendering a 4D energy field from a 3D environment of the present disclosure by partially representing a scene 1700 of the 3D environment. It will be understood that many more virtual pixels, virtual apertures, and virtual viewpoints may be placed in the scene 1700. To render data for a 4D energy field, some embodiments may include a large number of virtual pixels and virtual viewpoints to provide an energy field that is perceived to be seamless. The 2D angular coordinates of one of the virtual pixels 1702, 1704, 1706 describe the angular correlation between the pixel 1702, 1704, 1706 and the virtual viewpoints L1, L2, L3. For example, the 2D angular coordinates (U1, V1) of the first virtual pixel 1702 describe the angular correlation between the pixel 1702 and the virtual viewpoint L3. The 2D angular coordinates (U2, V1) of the second virtual pixel 1704 account for the angular correlation between the pixel 1704 and the virtual viewpoint L2. And the 2D angular coordinates (U3, V1) of the third virtual pixel 1706 account for the angular correlation between the pixel 1706 and the virtual viewpoint L1. In some embodiments, the angular coordinates may correspond to the 2D angular coordinates of a 4D energy projection system that defines a unique direction of the first propagation path, as described elsewhere in this disclosure. Some embodiments, such as those depicted in Figures 16 and 17, require only one ray to intersect with any one of the virtual pixels for a particular energy type.
[0224] FIG. 17 also depicts 2D spatial coordinates (X1, Y1) that identify the location of one virtual aperture 1708 of the plurality of virtual apertures. The 2D spatial coordinates may correspond to 2D spatial coordinates of 4D bright field coordinates for a physical 4D energy directing system as disclosed elsewhere in this disclosure, such as the embodiment discussed herein with respect to FIGS. 7C and 7D. Together, the 2D angular coordinates and the 2D spatial coordinates define a unique 4D coordinate. The unique 4D coordinate may correspond to the 4D bright field coordinates of a physical 4D energy projection system. In an embodiment such as the one depicted in FIG. 17, a first virtual pixel 1702 has 4D coordinates (X1, Y1, U1, V1), a second virtual pixel 1704 has 4D coordinates (X1, Y1, U2, V1), and a third virtual pixel 1706 has 4D coordinates (X1, Y1, U3, V1). 16 and 17, multiple rays may converge to each virtual aperture 1708 of a plurality of virtual apertures. The coordinates of a first virtual pixel 1702, a second virtual pixel 1704, and a third virtual pixel 1706 share the same spatial coordinates such that the rays 1710 of those virtual pixels converge to the same virtual aperture 1708. However, because the 2D angular coordinates differ among the virtual pixels, the rays intersect with different virtual viewing positions L1, L2, and L3.
[0225] In some embodiments of the systems and methods of the present disclosure, at least one ray 1710 of the multiple rays intersects with each virtual viewpoint L1, L2, L3 of the multiple virtual viewpoints.
[0226] In some embodiments, as depicted in FIG. 18, which further illustrates the relationship described by the known unique 4D coordinates, multiple light rays 1802 may also converge to a single viewpoint L2. In some of those embodiments, only one of the light rays 1802 converging to one virtual viewpoint L2 may intersect the same virtual aperture. As depicted in FIG. 18, one of the converging light rays 1802 intersects at (X1, Y1), one at (X2, Y1), and one at (X3, Y1). FIG. 17 and FIG. 18 are for illustrative purposes and are not intended to limit the embodiments of the present disclosure. As will be understood, even if FIG. 16, FIG. 17, and FIG. 18 depict a small number of virtual pixels, virtual apertures, light rays, and virtual viewpoints, different embodiments may comprise tens, hundreds, thousands, or other numbers of these elements. It should be understood that the virtual pixels may be located anywhere in the virtual plane. And the virtual positions may be located anywhere in the virtual viewing plane. Also, each of these diagrams is a stylized representation that may vary depending on the embodiment, and the disclosure of this application may apply to any type of energy and is not limited to electromagnetic energy in the visible spectrum.
[0227] 16 , in an embodiment, the multiple rays 1616 extend beyond the virtual pixel plane 1606. In some embodiments, the multiple rays 1616 may extend indefinitely beyond the virtual pixel plane 1606. This allows for sampling of energy data points from an object in front 1626 and from an object such as a photograph 1628 behind the virtual pixel plane 1606.
[0228] In some embodiments, the energy data points 1602 along multiple rays 1616 are sampled by tracing multiple rays 1616 from the virtual viewing plane 1610, through the virtual display plane 1614, through the virtual pixel plane 1606, and beyond. This allows the methods and systems of the present disclosure to sample energy data points capturing information from an object, such as a table 1628A in front of the virtual display plane 1610, and an object, such as a photo 1628B behind the virtual display plane 1610, in a single pass. This also allows the systems and methods of the present disclosure to accurately account for multiple energy interactions, as described in more detail elsewhere in this disclosure, that occur outside and inside the field of view from the observation point in a single pass. This eliminates the need for difficult, cumbersome, and imperfect post-computation to integrate multiple rendering passes.
[0229] FIG. 19 represents a scene with a 3D environment from a virtual viewpoint 1906, further illustrating how an embodiment of the system and method of the present disclosure allows for a large number of different energy interactions to be accounted for in a single pass. An embodiment of the system and method of the present disclosure allows for sampling of data points 1902 from the virtual viewpoint 1906 along multiple rays 1904. It will be understood that FIG. 19 is provided for illustrative purposes and is not intended to limit the embodiments of the present disclosure. For clarity, FIG. 19 does not represent virtual pixels or virtual apertures. And the size of the viewpoint 1906 may vary from embodiment to embodiment. For purposes of this illustration, the virtual viewpoint 1906 has a field of view 1908. The size of the field of view may vary from embodiment to embodiment and may depend on the type of physical 4D energy projection system in some cases. And, as will be understood, the size of the field of view 1908 may be determined or varied for different purposes. Sampled energy data points 1902 along multiple rays 1904 illustrate objects such as exemplary objects 1910A, 1910B in a field of view 1908 and energy reflected by objects within the field of view 1908. In this example, energy from a person (not shown) outside the field of view 1908 is reflected from a mirror 1910B in the field of view 1908, producing a reflection 1912 at the mirror 1910B that is sampled by the rays 1904. FIG. 19 is provided for illustrative purposes and is not intended to limit embodiments of the present disclosure. And, although FIG. 19 represents a 3D light environment, embodiments of the systems and methods of the present disclosure are applicable to other sensory and energy domains.
[0230] Referring again to FIG. 15, in an embodiment, the process for rendering a 4D energy field from a 3D environment may include a fourth step 1540 that includes correlating energy data points sampled along each ray with an energy value for one of the multiple virtual pixels.
[0231] This step 1540 allows embodiments of the disclosed systems and methods to account for changing properties of objects in the path of the ray. For example, in some embodiments, the energy value of a virtual pixel (not shown in FIG. 19) is correlated with energy data points 1902 sampled along only one ray that intersects with the virtual pixel (not shown in FIG. 19). For example, ray 1916 intersects with a partially reflective object 1920 and with a partially transparent object 1918 that is located between the partially reflective object 1920 and the virtual viewpoint 1906. To render data for this scene 1900, the disclosed systems or methods account for how the partially transparent object 1918 affects the light that is reflected from the partially reflective object 1920 through the partially transparent object 1918 to the virtual viewpoint 1906. For example, a partially reflective object 1920 may have a spectral radiance function that, when illuminated by a light source (not shown), distributes energy in different directions that is attenuated by a partially transparent object 1918 before reaching the virtual viewpoint 1906. An embodiment may also allow the energy values for different data points to be weighted based at least on the transmission or reflection characteristics of other energy data points along the same ray. The systems and methods of the present disclosure allow for correlating the energy values of a virtual pixel (not shown in FIG. 19) with the energy data points along the length of the ray 1916 to account for these types of various characteristics. As will be appreciated, the various energy data points may be modeled to account for any number of different characteristics, with different embodiments that may be specifically tailored to different energy regions. Determining energy values can be very challenging for multi-pass rendering systems, as they must model a large number of possible interactions along the renderings, and the three-dimensional interface of each rendering, to attempt to integrate all the information necessary to arrive at an energy value. The systems and methods of the present disclosure allow for avoiding such cumbersome and imprecise measures.
[0232] The process for rendering a 4D energy field from a 3D environment may include a fifth step 1550, which includes rendering the energy value of one of the virtual pixels and the known unique 4D coordinates of one of the virtual pixels into a data set having a format operable to instruct an energy device to output a 4D energy field. In some embodiments, the systems and methods of the present disclosure may enable the data set to comprise any data set described or referenced elsewhere in this disclosure, including 4D, 5D, 10D, 11D, or 15D functions. It will be understood that other types of data sets may be rendered for different embodiments of the 3D environment that account for different types of data. In some embodiments, the rendering may calibrate the data set for different types of 4D energy projection systems. Different types of calibrations may be applied to render data for 4D energy directing systems with different architectures, elements, or numbers of multiple elements. For example, calibration may be required for systems with different fields of view, or with different numbers or locations of energy sources or other components. In some embodiments of the systems and methods of the present disclosure, the energy data points 1902 include values describing at least one of: energy frequency, energy intensity, energy transparency, energy refraction, and energy reflectivity.
[0233] Some embodiments of the method for rendering a four-dimensional energy field from a three-dimensional environment of the present disclosure may allow the method to calibrate for an energy device that projects a 4D energy field. As will be appreciated, the location of the physical energy source and the energy propagation path of the 4D projection system may differ from the theoretical location of the virtual pixel and the theoretical angular correlation between the virtual pixel and the virtual viewpoint. This may be due to slight variations from energy device to energy device and even from waveguide to waveguide of the energy device. For example, slight imperfections in the waveguide may cause the energy propagation paths to deviate slightly from their expected direction. And the energy location may vary slightly from the corresponding virtual location in the 3D environment, which may also cause deviations in the direction of energy propagation. A method for calibrating is disclosed in co-owned U.S. Patent Application No. US16064300, entitled "Method of Calibration for Holographic Energy Directing Systems," and is incorporated herein by reference.
[0234] Fig. 21 is a schematic diagram illustrating reverse tracing of a three-dimensional environment. The method illustrated in Fig. 21 may include providing a scene 21000 in the 3D environment described by a plurality of energy data points disposed throughout the scene. In one embodiment, providing the scene 21000 may be performed according to the embodiments described above in paragraphs
[0244] to
[0245] . 21 also illustrates disposing a plurality of virtual pixels 21012 in a virtual pixel plane 21004 in the scene 21000, each virtual pixel of the plurality of virtual pixels 21012 having a known unique 4D coordinate including a 2D angular coordinate and a 2D spatial coordinate, the 2D angular coordinate of each virtual pixel of the plurality of virtual pixels 21012 describing an angular correlation between each virtual pixel and a virtual viewpoint of a plurality of virtual viewpoints 21008 disposed in a virtual viewing plane 21006 in the scene 21000, and the 2D spatial coordinate of each virtual pixel identifying a location of one virtual aperture of a plurality of virtual apertures 21010 disposed in a virtual display plane 21019 in the scene 21000. In one embodiment, disposing the plurality of virtual pixels 21012 may be performed according to the embodiments described above in paragraphs
[0245] to
[0247] .
[0235] 21 further illustrates sampling energy data points of a plurality of energy data points in the scene 21000 from a virtual viewing plane 21006 along a plurality of rays 21020, each of which intersects with one of the plurality of virtual viewpoints 21008 and one of the plurality of virtual pixels 21012 at an angle determined by the 2D angular coordinates of one of the virtual pixels intersected by the ray, and each of which intersects with one of the virtual apertures determined by the 2D spatial coordinates of one of the virtual pixels intersected by the ray. In one embodiment, sampling the data points may be performed according to the embodiment described above in paragraphs
[0248] to
[0255] .
[0236] FIG. 21 further illustrates correlating the energy data points sampled along each ray of the plurality of rays 21020 with an energy value for one of the plurality of virtual pixels 21012 intersecting the ray. In an embodiment, correlating the sampled energy data points may be performed according to the embodiment described above in paragraphs
[0256] -
[0257] . The energy value of the one virtual pixel of each ray and the known unique 4D coordinate of the one virtual pixel of each ray may be rendered into a data set having a format operable to instruct an energy device to output a 4D energy field. In an embodiment, rendering may be performed according to the embodiment described above in paragraph
[0258] .
[0237] 21, multiple rays 21012 may be traced from the virtual view plane, through and beyond the virtual pixel plane 21004, capturing information about energy data points behind the virtual pixel plane 21004. This allows the method to render a 4D energy field from the 3D environment, accounting for energy reflectance behind the virtual pixel plane, even when the reflections come from distant objects 21013.
[0238] 21 also illustrates how an embodiment of this method may account for a partially transparent object 21015. As depicted, energy data is sampled through the partially transparent object 21015 and past a virtual pixel display 21004 which samples data reflected from the object 21013. In some embodiments, energy values may then be correlated according to the sampled information by weighting the energy data 21014 of the object 21013 and the partially transparent object 21015 to arrive at an energy data value for a virtual pixel of the array of virtual pixels.
[0239] It should be appreciated that in one embodiment, the inverse tracing and mapping process of Figures 20 and 21 may enable inverse ray tracing in a single process step to a desired 4D view-specific volumetric energy representation (e.g., an image). This may include wavelength / energy considerations and view-specific calibration (or any other function to move seamlessly between various systems). This may include pixel processing with a displacement map in addition to calculate if and when other 4D coordinates are advantageous.
[0240] In some embodiments of the systems and methods of this disclosure, a ray of the multiple rays intersects with each virtual pixel of the multiple virtual pixels.
[0241] In some embodiments of the systems and methods of the present disclosure, the virtual display plane corresponds to a waveguide system of the energy directing device, and energy is operable to be directed through the waveguide system in accordance with the dataset to form a detectable volumetric representation of at least a portion of the scene.
[0242] In some embodiments of the systems and methods of the present disclosure, the multiple virtual pixels correspond to multiple energy locations on a first side of the waveguide system.
[0243] In some embodiments of the systems and methods of the present disclosure, the dataset further includes vectorized material property data.
[0244] In some embodiments of the systems and methods of the present disclosure, at least a portion of the method is performed in real time.
[0245] In some embodiments of the systems and methods of the present disclosure, the methods are performed entirely in real time.
[0246] In some embodiments of the systems and methods of the present disclosure, at least two portions of the method are performed at different time periods.
[0247] In some embodiments of the systems and methods of the present disclosure, at least one ray of the multiple rays intersects with each virtual aperture of the multiple virtual apertures.
[0248] In some embodiments of the systems and methods of the present disclosure, the dataset describes at least one signal perceptible by a visual, audio, textural, sensational, or odor sensor.
[0249] In some embodiments of the systems and methods of the present disclosure, energy data points sampled along each ray of a plurality of rays are simultaneously correlated with an energy value.
[0250] In some embodiments of the systems and methods of the present disclosure, the data set is stored in a binary file format. In some embodiments of the systems and methods of the present disclosure, the energy data points of the plurality of energy data points are sampled along each ray of the plurality of rays from the virtual viewing plane, through a virtual aperture of the plurality of virtual apertures, and beyond a virtual pixel of the plurality of virtual pixels.
[0251] In some embodiments of the systems and methods of the present disclosure, the steps may be repeated indefinitely, hi some embodiments, this may allow for dynamic capture of a scene, where the scene changes over time.
[0252] In embodiments, the disclosed method for rendering a 4D energy field from a 3D environment further includes storing information about the virtual viewpoint and the 4D coordinates of the virtual viewpoint in a ray file, which can provide instructions for sampling data from those virtual viewpoints along any rays that intersect with the virtual viewpoint. A ray file is literally just a mapping of what samples are being rendered into the virtual viewing plane, and can include a list of 3D virtual viewing plane coordinates and associated 2D (u,v) angular coordinates. As an example, in some embodiments, the ray file can be used with CG scene information such as the scene's shape, color, reflections, etc. to effectively sample a synthetic 4D light field. It will be appreciated that the information stored in the ray file can vary depending on the type of energy device utilized to output the 4D energy field, as the number of pixels and acceptable observation point locations can differ from energy device to energy device. In some embodiments, the ray file can provide a means for instructing the system on how to sample the 3D environment for a given energy device. In other embodiments, the ray file is a mapping ideal for a particular energy field projection device design and does not reflect the relatively small differences from device to device, which then require arbitrary sampling of the scene using the ray file to undergo further device-dependent corrections. The following discussion refers to Figures 17 and 18, but it will be understood that the methods described herein apply to embodiments with many more virtual viewpoints, virtual pixels, virtual apertures, and rays. In one embodiment, the 3D spatial coordinates may identify the location of each virtual viewpoint L2 and the 2D angular coordinates of one virtual pixel 1702, 1704, 1706 that intersects with all rays 1710. In some embodiments, the ray file associates the 3D spatial coordinates of each virtual viewpoint L2 with the 2D angular coordinates of all virtual pixel pixels 1702, 1704, 1706 that intersect with all rays that intersect the virtual viewpoint. For example, L2 in Figure 18 intersects with ray 1802.The ray file may then link the angular coordinates of virtual pixels 1803, 1805, and 1807 with L2 and 3D spatial coordinates identifying the location of L2 within the scene. In an embodiment, the ray file provides instructions for each ray 1802 to sample energy data points of the plurality of energy data points 1802 along the ray from one virtual viewpoint L2 that intersects with the ray, the instructions being determined by at least the 3D spatial coordinates of one virtual viewpoint L2 that intersects with the ray and the 2D angular coordinates of one virtual pixel that intersects with the ray. For example, the ray file may provide instructions for a ray 1809 that is determined by at least the 3D spatial coordinates of L2 and the 2D angular coordinates of 1803. As will be appreciated, this method may be used for a variety of applications other than energy fields, including rendering data for 2D, stereoscopic, virtual reality, augmented reality, and multiple view display systems.
[0253] In some embodiments of the method for rendering a four-dimensional energy field from a three-dimensional environment of the present disclosure, rendering the energy values may also further include calibrating the energy data of the energy device. As will be understood, the theoretical positions of the virtual pixels and the theoretical angular correlation between the virtual pixels and the virtual viewpoint may vary from one physical implementation to another. This may be due to slight variations from one energy device to another, and even from one waveguide of the energy device to another. For example, slight imperfections in the waveguide may cause the energy propagation paths to deviate slightly from their expected directions. And the energy positions may vary slightly from the corresponding virtual positions in the 3D environment, which may also cause slight deviations in the direction of energy propagation. In some embodiments, these deviations may be accounted for by calibrating the energy data of the energy device or a particular type of energy device. It will be understood that other types of calibrations may also be appropriate.
[0254] The process depicted in FIG. 15 may be performed using any processing system of the present disclosure or suitable systems known in the art, including the processing system 2200. In the embodiment of FIG. 22, the processing system may include a rendering engine 2210 and a sensory data engine 2220. In some embodiments, the sensory data engine 2220 may perform step 1510 to provide a scene of a 3D environment. And the sensory data engine 2220 may also perform step 1520 to place a plurality of virtual pixels in a virtual pixel plane. The rendering engine 2210 of some embodiments may perform step 1530 to sample energy data points. The rendering engine 2230 may also perform step 1540 to correlate the energy data points sampled along each ray with an energy value. And the rendering engine 2210 may also perform step 1550 to render the energy value. In some embodiments, the system may also include a memory 2240 to store the dataset.
[0255] It will be understood that the method for rendering a 4D energy field from a 3D environment may be applied to render energy data for other applications. Such a method may include a first step 2210 for providing a scene of a 3D environment described by a plurality of energy data points located throughout the scene. A second step 2220 may include arranging a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having a known unique 4D coordinate including a 2D angular coordinate and a 2D spatial coordinate, the 2D angular coordinate of each virtual pixel describing an angular correlation between the virtual pixel and one of a plurality of virtual viewpoints located in a virtual viewing plane in the scene, and the 2D spatial coordinate of each virtual pixel identifying the position of one of a plurality of virtual apertures located in a virtual display surface in the scene. A third step 2230 may include sampling energy data points of a plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, where each ray intersects a virtual viewpoint and a virtual pixel at an angle determined by the 2D angular coordinate of a virtual pixel intersecting the ray, and where each ray intersects a virtual aperture determined by the 2D spatial coordinate of a virtual pixel intersecting the ray. A fourth step may include correlating the energy data points sampled along each ray with an energy value for a virtual pixel intersecting the ray, and a fifth step 2250 may include rendering the energy value of the virtual pixel for each ray and the known unique 4D coordinate of the virtual pixel for each ray into a data set having a format operable to instruct an energy device to output the energy data.
[0256] It will be appreciated that some forms of energy data may have far fewer virtual field pixels and virtual viewpoints required to render a 4D energy field. For example, in some embodiments, each virtual aperture intersects with two rays. In such embodiments, the multiple virtual viewpoints comprise two virtual viewpoints. However, such embodiments may be operable to generate a data set operable to instruct the energy device to output a stereoscopic image. It will be appreciated that the stereoscopic display directs a first image to a first eye of the viewer and a second image to a second eye. As a result, in some embodiments, each aperture may comprise only two virtual pixels and there may be only two virtual viewpoints - one for each eye. Such embodiments may enable a data set having a format operable to instruct the energy device to output a virtual reality image. And, such embodiments may enable a data set having a format operable to instruct the energy device to output an augmented reality image.
[0257] It will also be appreciated that in other embodiments, each virtual aperture may intersect with additional rays corresponding to additional viewpoints or views. Such an embodiment may enable a data set having a format operable for a multi-view display, where the number of views corresponds to the number of rays intersecting each virtual pixel. This principle can be illustrated by FIG. 17, which shows three rays 1710 intersecting with three viewpoints L1, L2, L3 and one aperture X1, Y1, each ray intersecting with one virtual pixel 1702, 1704, and 1706. Each viewpoint may correspond to one view of a multiple view system. It will be appreciated that some embodiments are multiple additional viewpoints.
[0258] Although various embodiments according to the principles disclosed herein have been described above, it should be understood that these embodiments are presented for illustrative purposes only and are not limiting. Thus, the breadth and scope of the invention(s) should not be limited by any of the exemplary embodiments described above, but should be defined only in accordance with the claims that derive from this disclosure, and their equivalents. Moreover, the above advantages and features are provided in the described embodiments, but do not limit the application of such claims to processes and structures that achieve any or all of the above advantages.
[0259] It should be understood that the principal features of this disclosure can be employed in various embodiments without departing from the scope of the disclosure. Those skilled in the art will recognize, or be able to discover using no more than routine experimentation, many equivalents to the specific procedures described herein. Such equivalents are considered to be within the scope of this disclosure and are covered by the claims.
[0260] Furthermore, the section headings herein are provided for consistency with the teachings under 37 CFR 1.77 or to otherwise provide organizational guidance. These headings are not intended to limit or characterize the invention(s) described in any claim that may originate from this disclosure. Specifically, as an example, even if a heading is entitled "Field of the Invention," such claims should not be limited by the language of this heading to describe the so-called technical field. Furthermore, the description of a technology in the "Background of the Invention" section should not be construed as an admission that the technology is prior art to any invention(s) in this disclosure. The "Summary" is in no way intended to characterize the invention(s) described in the claims at issue. Furthermore, references to the singular "invention" in this disclosure should not be used to assert that there is only a single novelty in this disclosure. Multiple inventions may be described according to the limitations of multiple claims originating from this disclosure, and such claims thus define the invention(s) and their equivalents protected thereby. In all instances, the scope of such claims will be considered on their own merits in light of this disclosure, but should not be constrained by the headings set forth herein.
[0261] The words "a" or "an" when used in conjunction with the term "comprising" in the claims and / or specification may mean "one," but it is also consistent with the meanings of "one or more," "at least one," and "one or more than one." The term "or" used in the claims is used to mean "and / or" unless expressly referring to alternatives only or the alternatives are mutually exclusive, but the present disclosure supports the definition referring to alternatives only and "and / or." Throughout this application, the term "about" is used to indicate that a value includes the inherent error variation of the device and the method is used to determine the value, or the variation that exists among the research subjects. Generally, but subject to the foregoing considerations, numerical values herein modified by approximation words such as "about" can vary from the stated value by at least + / - 1, 2, 3, 4, 5, 6, 7, 10, 12, or 15%.
[0262] As used in this specification and claim(s), the words "comprising" (and any form of comprising, such as "comprise" and "comprises"), "having" (and any form of having, such as "have" and "has"), "including" (and any form of including, such as "includes" and "include"), or "containing" (and any form of including, such as "contains" and "contain") are inclusive or open-ended, and do not exclude additional, unrecited elements or method steps.
[0263] Words relating to comparisons, measurements, and timings such as "at the time," "equivalent," "during," "complete," and the like, should be understood to mean "substantially at the time," "substantially equivalent," "substantially during," "substantially complete," and the like, where "substantially" means that such comparisons, measurements, and timings are practical to achieve the desired result implicitly or explicitly stated. Words relating to the relative positions of elements, such as "near," "proximate to," "adjacent to," and the like, are intended to mean close enough to substantially affect the interaction of the respective system elements. Other words of approximation similarly refer to conditions that, when so modified, are not necessarily considered absolute or complete, but would be considered close enough by one of ordinary skill in the art to warrant specifying the condition as existing. The extent to which the description may vary will depend on how significant the change is, and one of ordinary skill in the art would recognize the modified features as still having the desired properties and capabilities of the unmodified features.
[0264] The term "or combinations thereof" as used herein refers to all permutations and combinations of the listed items preceding the term. For example, "A, B, C, or combinations thereof" is intended to include at least one of A, B, C, AB, AC, BC, or ABC, as well as BA, CA, CB, CBA, BCA, ACB, BAC, or CAB, if the order is important in a particular context. Continuing with this example, combinations including repeats of one or more items or terms, such as BB, AAA, AB, BBC, AAABCCCC, CBBAAA, CABABB, etc., are expressly included. Those skilled in the art will understand that there is typically no limit to the number of items or terms in any combination, unless otherwise clear from the context.
[0265] All of the compositions and / or methods disclosed and claimed herein can be made and executed without undue experimentation in light of the present disclosure.The compositions and methods of the present disclosure are described in terms of preferred embodiments, but it will be apparent to those skilled in the art that various variations can be applied to the compositions and / or methods, and in the steps or sequence of steps of the methods described herein, without departing from the concept, spirit and scope of the present disclosure.All such similar substitutes and modifications that are apparent to those skilled in the art are deemed to be within the spirit, scope and concept of the disclosure as defined by the appended claims. (Other possible items) (Item 1) 1. A method for rendering a four-dimensional (4D) energy field from a three-dimensional (3D) environment, the method comprising: providing a scene of a 3D environment described by a plurality of energy data points distributed throughout the scene; arranging a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having known unique 4D coordinates including a 2D angular coordinate and a 2D spatial coordinate, the 2D angular coordinate of each virtual pixel describing an angular correlation between the virtual pixel and one virtual viewpoint of a plurality of virtual viewpoints arranged in a virtual viewing plane in the scene, and the 2D spatial coordinate of each virtual pixel identifying a location of a virtual aperture of a plurality of virtual apertures arranged in a virtual viewing plane in the scene; sampling energy data points of the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, each ray intersecting a virtual viewpoint and the one virtual pixel at an angle determined by the 2D angular coordinate of a one virtual pixel intersected by the ray, and each ray intersecting a virtual aperture determined by the 2D spatial coordinate of the one virtual pixel intersected by the ray; correlating the energy data points sampled along each ray with an energy value for the one virtual pixel intersected by the ray; and rendering the energy value of the one virtual pixel of each light ray and the known unique 4D coordinate of the one virtual pixel of each light ray into a dataset having a format operable to instruct an energy device to output a 4D energy field. (Item 2) Item 2. The method of item 1, wherein at least one ray of the plurality of rays intersects with each virtual viewpoint of the plurality of virtual viewpoints. (Item 3) Item 2. The method of item 1, wherein one ray of the plurality of rays intersects with each virtual pixel of the plurality of virtual pixels. (Item 4) 2. The method of claim 1, wherein the 4D energy field comprises a bright field. (Item 5) 2. The method of claim 1, wherein the 4D energy field comprises a haptic field. (Item 6) Item 14. The method of item 1, wherein the 4D energy field comprises a tactile field. (Item 7) Item 10. The method of item 1, wherein the energy data points include values describing at least one of the following: energy frequency, energy intensity, energy transparency, energy refraction, and energy reflectance. (Item 8) 2. The method of claim 1, wherein the 3D environment is determined by applying a depth map to points in a two-dimensional space. (Item 9) 2. The method of claim 1, wherein the virtual display plane corresponds to a waveguide system of an energy directing device, and wherein energy is operable to be directed through the waveguide system in accordance with the dataset to form a detectable 4D energy representation of at least a portion of the scene. (Item 10) 10. The method of claim 9, wherein the plurality of virtual pixels corresponds to a plurality of energy positions on a first side of the waveguide system. (Item 11) 2. The method of claim 1, wherein the dataset further comprises vectorized material property data. (Item 12) 2. The method of claim 1, wherein at least a portion of the method is performed in real time. (Item 13) 2. The method according to claim 1, wherein the method is performed entirely in real time. (Item 14) Item 14. The method of item 1, wherein at least two parts of the method are carried out for different periods of time. (Item 15) 2. The method of claim 1, wherein the dataset describes signals perceivable by visual, audio, textural, sensational, or olfactory sensors. (Item 16) Item 14. The method of item 1, wherein the energy data points sampled along each ray of the plurality of rays are simultaneously correlated with an energy value. (Item 17) 2. The method of claim 1, wherein the dataset is stored in a binary file format. (Item 18) 2. The method of claim 1, wherein each ray of the plurality of rays extends through the one virtual aperture of the plurality of virtual apertures to and beyond the one virtual pixel of the plurality of virtual pixels, and an energy data point of the plurality of energy data points is sampled from the virtual viewing plane. (Item 19) 2. The method of claim 1, wherein the steps are repeated indefinitely. (Item 20) 2. The method of claim 1, wherein the steps are repeated indefinitely to render a dynamic 4D energy field from the 3D environment. (Item 21) 2. The method of claim 1, wherein rendering the energy data further comprises calibrating the energy data for the energy device. (Item 22) 2. The method of claim 1, wherein a ray file stores 3D spatial coordinates identifying the position of each virtual viewpoint and the 2D angular coordinates of the one virtual pixel intersected by all rays. (Item 23) 23. The method of claim 22, wherein the ray file associates the 3D spatial coordinates of each virtual viewpoint with the 2D angular coordinates of all virtual pixels intersected by all rays that intersect the virtual viewpoint. (Item 24) 23. The method of claim 22, wherein the ray file provides instructions for each ray for sampling energy data points of the plurality of energy data points along the ray from the one virtual viewpoint that intersects the ray, the instructions being determined by at least the 3D spatial coordinate of the one virtual viewpoint that intersects the ray and the 2D angular coordinate of the one virtual pixel that intersects the ray. (Item 25) 1. A system for rendering a four-dimensional (4D) energy field from a dynamic three-dimensional (3D) environment, the system comprising: a processing subsystem including a sensory data engine and a rendering engine; the sensory data engine providing a scene of a 3D environment described by a plurality of energy data points located throughout the scene; the sensory data engine arranges a number of virtual pixels in a virtual pixel plane of the scene, each virtual pixel having a known unique 4D coordinate comprising a 2D angular coordinate and a 2D spatial coordinate, the 2D angle of each virtual pixel coordinate describing an angular correlation between the virtual pixel and a virtual viewpoint of a number of virtual viewpoints arranged in a virtual viewing plane in the scene by the sensory data engine, and the 2D spatial coordinate of each virtual pixel identifying the position of one virtual aperture of a number of virtual apertures arranged in a virtual viewing plane in the scene by the sensory data engine; the rendering engine samples energy data points of the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, each ray intersecting a virtual viewpoint and a virtual pixel at an angle determined by the 2D angular coordinate of a virtual pixel intersected by the ray, each ray intersecting a virtual aperture determined by the 2D spatial coordinate of the virtual pixel intersected by the ray; the rendering engine correlating the energy data points sampled along each ray with an energy value for the one of the virtual pixels; The system, wherein the rendering engine renders the energy value of the one virtual pixel of the plurality of virtual pixels and the known unique 4D coordinate of the one virtual pixel of the plurality of virtual pixels into a dataset having a format operable to instruct an energy device to output a 4D energy field. (Item 26) 26. The system of claim 25, wherein at least one ray of the plurality of rays intersects with each virtual viewpoint of the plurality of virtual viewpoints. (Item 27) 26. The system of claim 25, wherein one ray of the plurality of rays intersects with each virtual pixel of the plurality of virtual pixels. (Item 28) 26. The system of claim 25, wherein the 4D energy field comprises a bright field. (Item 29) 30. The system of claim 28, wherein the 4D energy field comprises a haptic field. (Item 30) 26. The system of claim 25, wherein the 4D energy field comprises a tactile field. (Item 31) 26. The system of claim 25, wherein the energy data points include values describing at least one of: energy frequency, energy intensity, energy transparency, energy refraction, and energy reflectance. (Item 32) 26. The system of claim 25, wherein the 3D environment is determined by applying a depth map to points in two-dimensional space. (Item 33) 26. The system of claim 25, wherein the virtual display plane corresponds to a waveguide system of an energy directing device, and wherein energy is operable to be directed through the waveguide system in accordance with the dataset to form a detectable 4D energy field representation of at least a portion of the scene. (Item 34) Item 34. The system of item 33, wherein the plurality of virtual pixels corresponds to a plurality of energy positions on a first side of the waveguide system. (Item 35) 26. The system of claim 25, wherein the dataset further comprises vectorized material property data. (Item 36) 26. The system of claim 25, wherein at least a portion of the system is performed in real time. (Item 37) 26. The system of claim 25, wherein the system runs entirely in real time. (Item 38) 26. The system of claim 25, wherein at least two parts of the system run for different periods of time. (Item 39) 26. The system of claim 25, wherein the dataset describes signals perceivable by visual, audio, textural, sensational, or odor sensors. (Item 40) Item 26. The system of item 25, wherein the energy data points sampled along each ray of the plurality of rays are simultaneously correlated with an energy value. (Item 41) 26. The system of claim 25, wherein the dataset is stored in a binary file format. (Item 42) 26. The system of claim 25, wherein an energy data point of the plurality of energy data points is sampled by the rendering engine along each ray of the plurality of rays, the ray extending from the virtual viewing plane, through the one virtual aperture of the plurality of virtual apertures, to and beyond the one virtual pixel of the plurality of virtual pixels. (Item 43) 26. The system of claim 25, wherein the system is operated indefinitely to render dynamic 4D energy fields from a dynamic 3D environment. (Item 44) 26. The system of claim 25, further comprising a memory for storing the dataset. (Item 45) Item 26. The method of item 25, wherein the rendering system calibrates the energy data for the energy device. (Item 46) 26. The system of claim 25, further comprising a memory for storing the plurality of energy data points. (Item 47) 26. The system of claim 25, wherein a ray file stores 3D spatial coordinates identifying the position of each virtual viewpoint and the 2D angular coordinates of the one virtual pixel intersected by all rays. (Item 48) 48. The system of claim 47, wherein the ray file associates the 3D spatial coordinates of each virtual viewpoint with the 2D angular coordinates of all virtual pixels intersected by all rays that intersect the virtual viewpoint. (Item 49) 49. The system of claim 48, wherein the ray file provides instructions for each ray for sampling energy data points of the plurality of energy data points along the ray from the one virtual viewpoint that intersects the ray, the instructions being determined by at least the 3D spatial coordinate of the one virtual viewpoint that intersects the ray and the 2D angular coordinate of the one virtual pixel that intersects the ray. (Item 50) 1. A method for rendering energy data from a three-dimensional (3D) environment, the method comprising: providing a scene of a 3D environment described by a plurality of energy data points distributed throughout the scene; arranging a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having known unique 4D coordinates including a 2D angular coordinate and a 2D spatial coordinate, the 2D angular coordinate of each virtual pixel describing an angular correlation between the virtual pixel and one virtual viewpoint of a plurality of virtual viewpoints arranged in a virtual viewing plane in the scene, and the 2D spatial coordinate of each virtual pixel identifying a location of a virtual aperture of a plurality of virtual apertures arranged in a virtual viewing plane in the scene; sampling energy data points of the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, each ray intersecting a virtual viewpoint and the one virtual pixel at an angle determined by the 2D angular coordinate of a one virtual pixel intersected by the ray, and each ray intersecting a virtual aperture determined by the 2D spatial coordinate of the one virtual pixel intersected by the ray; correlating the energy data points sampled along each ray with an energy value for the one virtual pixel intersected by the ray; rendering the energy value of the one virtual pixel of each light ray and the known unique 4D coordinate of the one virtual pixel of each light ray into a dataset having a format operable to instruct an energy device to output energy data. (Item 51) Item 51. The method according to item 50, wherein each virtual aperture is intersected by two rays. (Item 52) Item 52. The method of item 51, wherein the multiple virtual viewpoints comprise two virtual viewpoints. (Item 53) 53. The method of claim 52, wherein the dataset has a format operable to instruct an energy device to output a stereoscopic image. (Item 54) 53. The method of claim 52, wherein the dataset has a format operable to instruct an energy device to output a virtual reality image. (Item 55) Item 53. The method of item 52, wherein the dataset has a format operable to instruct an energy device to output an augmented reality image. (Item 56) 51. The method of claim 50, wherein each virtual aperture is intersected by several rays. (Item 57) 52. The method of claim 51, wherein the dataset has a format operable to instruct the energy device to output images from multiple views, the multiple views corresponding to a number of rays intersecting each virtual aperture.
Claims
[Claim 1] 1. A method for rendering a four-dimensional (4D) energy field from a three-dimensional (3D) environment, the method comprising: providing a scene of a 3D environment described by a plurality of energy data points distributed throughout the scene; arranging a plurality of virtual pixels in a virtual pixel plane in the scene, each virtual pixel having known unique 4D coordinates including a 2D angular coordinate and a 2D spatial coordinate, the 2D angular coordinate of each virtual pixel describing an angular correlation between the virtual pixel and one virtual viewpoint of a plurality of virtual viewpoints arranged in a virtual viewing plane in the scene, and the 2D spatial coordinate of each virtual pixel identifying a location of a virtual aperture of a plurality of virtual apertures arranged in a virtual display plane in the scene; sampling energy data points of the plurality of energy data points in the scene along a plurality of rays from the virtual viewing plane, each ray intersecting a virtual viewpoint and a virtual pixel at an angle determined by the 2D angular coordinate of a virtual pixel intersected by the ray, and each ray intersecting a virtual aperture determined by the 2D spatial coordinate of the virtual pixel intersected by the ray; correlating the energy data points sampled along each ray with an energy value for the one virtual pixel intersected by the ray; rendering the energy value of the one virtual pixel of each light ray and the known unique 4D coordinate of the one virtual pixel of each light ray into a dataset having a format operable to instruct an energy device to output a 4D energy field.
Citation Information
Patent Citations
Image processing device, image processing method, and program
JP2014086899A
Image processor and image processing method
JP2015114775A
Light field processing method
JP2016537901A
Capturing light-field volume image and video data using tiled light-field cameras
US20160307372A1
