Light field display system for performance events

The LF display system addresses limitations of traditional venues by offering immersive, interactive holographic performances with integrated sensory feedback, allowing unconstrained viewing and networked content distribution.

JP2024156802AInactive Publication Date: 2024-11-06LIGHT FIELD LAB INC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024124265
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-05-13
Filing Date
2024-07-31
Publication Date
2024-11-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional performance venues limit viewer access, and recorded performances fail to recreate the atmosphere and excitement of live events, especially when sold out, inconveniently timed, or geographically distant.

Method used

A light field (LF) display system that uses LF display modules to create a seamless viewing surface, projecting holographic content with integrated sensory stimuli, allowing viewers to interact and perceive performances as if they were live, with features like ultrasound for tactile feedback and networked content distribution.

Benefits of technology

Enables immersive, interactive holographic experiences that replicate live performances, providing unconstrained viewing and sensory engagement without the need for external devices, enhancing viewer immersion and interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024156802000001_ABST
    Figure 2024156802000001_ABST
Patent Text Reader

Abstract

To provide a light filed (LF) display system for displaying holographic performance content to viewers in a venue.SOLUTION: A LF display system in a venue includes LF display modules tiled together to form an array of LF modules. The array of LF modules create a performance volume for displaying performance content in the venue. The array of LF modules displays the performance content to viewers in viewing volumes. The LF display system can be included in a LF presentation network. Use of the LF presentation network allows holographic performance content to be recorded at one location and displayed at another location. The LF presentation network includes a network system to manage the digital rights of the holographic performance content.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042470, and PCT / US2017 / 042679, all of which are incorporated by reference in their entireties. [Background technology]

[0002] The present disclosure relates to performances in performance venues, and more particularly to light field display systems for displaying performances in performance venues.

[0003] Traditionally, performance venues (e.g., theaters, concert halls, etc.) are configured to allow audiences (e.g., fans, patrons, etc.) attending the venue to view a live performance (e.g., concert, play, comedy show, etc.) in real time from a performer (e.g., band, troupe, comedy, etc.). Unfortunately, in some cases, hosting a performance at the venue limits the ability of audience members who wish to view the performance. For example, the performance may be sold out, at an inconvenient time, or located far away from the audience. Performances may also be recorded and played back on a two-dimensional surface such as a movie screen or television, but these playbacks rarely replicate the atmosphere and excitement present in the venue for a live performance. As such, a venue that is configured for a performance such that an audience member may perceive the performance as if they were at the live performance may be beneficial. Summary of the Invention

[0004] A light field (LF) display system for displaying holographic content of a performance in a venue (such as a theater, concert hall, etc.). The LF display system includes LF display modules forming a surface (e.g., a stage, etc.) at the venue, each having a display area and tiled together to form a seamless display surface having an effective display area larger than the display area. The LF display modules display holographic content of the performance from the performance volume such that an audience at the venue can perceive the performance.

[0005] In some embodiments, the holographic content of a performance may be a copy of a performance occurring simultaneously at another venue and is created for display at the venue by a content creation system and / or accessed from a data store for display at the venue. The holographic content may be managed by a network system responsible for managing the digital rights of the holographic content. For example, viewers at the venue may pay a transaction fee to access the holographic content that displays at the venue.

[0006] In some embodiments, the LF display system includes a tracking system and / or a viewer profiling module. The tracking system and viewer profiling system can monitor and store characteristics of viewers in a venue, viewer profiles describing the viewers, and / or viewer responses to holographic content in the venue. The holographic content created for display in the venue can be based on either the monitored or stored information.

[0007] In some embodiments, a user can interact with the holographic content, and the interaction can serve as an input for the holographic content creation system. For example, in some embodiments, some or all of the LF display system includes a plurality of ultrasonic speakers. The plurality of ultrasonic speakers are configured to generate a tactile surface that matches at least a portion of the holographic content. A tracking system is configured to track the user's interaction with the holographic object (e.g., via images captured by an image sensor and / or some other camera of the LF display module). The LF display system is also configured to create and provide holographic content based on the interaction. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram of a light field display module for presenting holographic objects in accordance with one or more embodiments. [Figure 2A] 1 is a cross-section of a portion of a light field display module according to one or more embodiments. [Figure 2B] 1 is a cross-section of a portion of a light field display module according to one or more embodiments. [Figure 3A] FIG. 1 is a perspective view of a light field display module according to one or more embodiments. [Figure 3B] 1 is a cross-sectional view of a light field display module including an interleaved energy relay device according to one or more embodiments. [Figure 4A] FIG. 1 is a perspective view of a portion of a light field display system that is two-dimensionally tiled to form a single-sided seamless surface environment in accordance with one or more embodiments. [Figure 4B] FIG. 1 is a perspective view of a portion of a light field display system in a multi-surface seamless surface environment in accordance with one or more embodiments. [Figure 4C] FIG. 2 is a top view of a light field display system having a collective surface in a wing-type configuration in accordance with one or more embodiments. [Figure 4D] FIG. 2 is a side view of a light field display system having a collection surface in a tilted configuration in accordance with one or more embodiments. [Figure 4E] FIG. 2 illustrates a top view of a light field display system having a collection surface on the front wall of a room in accordance with one or more embodiments. [Figure 4F] FIG. 2 is a side view of a LF display system having a collection surface on the front wall of a room in accordance with one or more embodiments. [Figure 5A] FIG. 1 is a block diagram of a light field display system in accordance with one or more embodiments. [Figure 5B] 5 illustrates an exemplary LF film network 550 according to one or more embodiments. [Figure 6] FIG. 6 is a side view of a venue 600, a traditional theater augmented with an LF display system, in accordance with one or more embodiments. [Figure 7A] FIG. 2 is a cross-sectional view of a first venue including an LF display system for displaying performance content to an audience at a viewing location within a viewing volume, according to one or more embodiments. [Figure 7B] FIG. 2 is a cross-sectional view of a second venue including an LF display system for displaying performance content to an audience at a viewing location within a viewing volume, according to one or more embodiments. [Figure 8] FIG. 1 illustrates a venue that doubles as a home theater in a viewer's living room, in accordance with one or more embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating a method for displaying holographic performances in an LF performance network.

[0009] The figures depict various embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the present invention as described herein." DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] overview A light field (LF) display system is implemented in a venue that serves as a setting for a performance. For example, the performance may be a concert, a play, a comedy show, a dance recital, etc. The LF display system comprises an LF display assembly configured to present holographic content including one or more holographic objects that may be visible to one or more users within a viewing volume of the LF display system. The LF display assembly may form a multi-sided seamless surface across some or all of one or more surfaces (e.g., a stage) of the venue. The LF display system may be capable of presenting the holographic content to an audience at the venue. The audience typically attends a performance at the venue, but may be any human in a location at the venue that is capable of viewing the holographic content. The holographic objects may also be augmented with other sensory stimuli (e.g., haptics and / or sound). For example, an ultrasound emitter in the LF display system may emit ultrasound pressure waves that provide a tactile surface for some or all of the holographic object. The holographic content may include additional visual content (i.e., 2D or 3D visual content). Part of the system in a multi-emitter implementation is the coordination of emitters to ensure a cohesive experience is possible (i.e., a holographic object providing the correct tactile and sensory stimuli at any given time).

[0011] In some embodiments, the LF display system includes a plurality of LF display modules forming a stage for a venue. The LF display modules forming the stage may be configured to project holographic content of a performance at the venue. In this manner, a viewer at the venue may perceive a holographic performance at the stage. For example, the LF display system may display a band performing a concert, a dance company debuting a ballet, or a theater company performing a play. In some embodiments, the LF display system may create holographic content for display to a viewer in the venue. For example, the LF display system may create a virtual performer to perform a song created for the virtual performer to a viewer at the venue.

[0012] In some embodiments, the LF display system may include elements that allow the system to simultaneously emit at least one type of energy and absorb at least one type of energy for the purpose of responding to the created holographic content. For example, an LF display system may emit both holographic objects for viewing and ultrasound for tactile perception, while simultaneously absorbing image information for viewer tracking and other scene analysis, while absorbing ultrasound to detect touch responses by the user. As an example, such a system may project a holographic performer who is crowd surfing, giving the viewer the illusion that the performer is in their hand when the viewer effectively "touches" it. The components of the display system that perform energy sensing of the environment may be integrated into the display surface via bidirectional energy elements that both emit and absorb energy, or may be dedicated sensors separate from the display surface, such as ultrasonic speakers and imaging capture devices such as cameras.

[0013] The LF display system can be part of an LF performance network that allows LF data to be recorded in one location, encoded, transmitted to another location, decoded, and displayed as holographic content to audiences at venues, thereby allowing audiences at multiple venues to perceive a live performance occurring at another venue. In some embodiments, the LF display system includes a network system that manages digital rights for the holographic content.

[0014] Light Field Display System FIG. 1 is a diagram of a light field (LF) display module 110 presenting a holographic object 120, according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system presents holographic content, including at least one holographic object, using one or more LF display modules. The LF display system can present the holographic content to one or more viewers. In some embodiments, the LF display system can also augment the holographic content with other sensory content (e.g., touch, sound, smell, temperature, etc.). For example, as discussed below, the projection of focused ultrasound can generate hollow haptics that can simulate some or all of the surface of a holographic object. The LF display system includes one or more LF display modules 110 and is discussed in detail below with respect to FIGS. 2A-5B.

[0015] The LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more viewers (e.g., viewer 140). The LF display module 110 includes an energy device layer (e.g., a light-emitting electronic display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). Additionally, the LF display module 110 may include an energy relay layer to combine multiple energy sources or detectors together to form a single surface. At a high level, the energy device layer generates energy (e.g., holographic content), which is then directed to an area in space according to one or more four-dimensional (4D) light field functions using the energy waveguide layer. The LF display module 110 may also simultaneously project and / or sense one or more types of energy. For example, the LF display module 110 may be able to project holographic images as well as ultrasound tactile surfaces into a viewing volume while simultaneously detecting image data from the viewing volume. The holographic image between the display area 150 of the LF display module 110 and the viewer 140 is a real image in which the collection of focal points that form the image are converging light rays. Additionally, some images produced by the LF display may be virtual images (e.g., images displayed behind the display area 150) that are a collection of focal points created by the extension of diverging light rays. The operation of the LF display module 110 is discussed in more detail below with respect to Figures 2A-3B.

[0016] The LF display module 110 generates a holographic object in the holographic object volume 160 using one or more 4D light field functions (e.g., derived from a plenoptic function). The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Furthermore, the holographic object may be multi-colored (e.g., full color). The holographic object may be projected in front of a screen plane, behind a screen plane, or split by the screen plane. The holographic object 120 may be presented to be perceived anywhere in the holographic object volume 160. Here, the holographic object 120 is a real image 122 when light rays are converging, which is presented between the display area 150 and the viewer 140. The holographic object in the holographic object volume 160 may appear to the viewer 140 as floating in space.

[0017] The holographic object volume 160 represents a volume in which a holographic object can be perceived by the viewer 140. The holographic object volume 160 may extend in front of the surface of the display area 150 (i.e., towards the viewer 140) so that the holographic object can be presented in front of the plane of the display area 150. Additionally, the holographic object volume 160 may extend behind the surface of the display area 150 (i.e., away from the viewer 140), allowing the holographic object to be presented as if it were behind the plane of the display area 150. In other words, the holographic object volume 160 can include all light rays that can originate (e.g., projected) from the display area 150 and converge to create a holographic object. Here, the light rays can converge to a point in front of, at, or behind the display surface. In simpler terms, the holographic object volume 160 encompasses all volumes in which a holographic object can be perceived by the viewer.

[0018] The viewing volume 130 is the volume of space in which a holographic object (e.g., holographic object 120) presented by the LF display system in the holographic object volume 160 is fully visible. The holographic object may be presented in the holographic object volume 160 and viewed in the viewing volume 130 such that it is indistinguishable from the actual object. The holographic object is formed by projecting the same light rays that would be generated from the surface of the object if it physically existed.

[0019] In some cases, the holographic object volume 160 and corresponding viewing volume 130 may be relatively small, such as designed for a single viewer. In other embodiments, for example, as discussed in detail below with respect to Figures 4A-4F, 6, 7, 8, and 9, the LF display modules may be scaled and / or tiled to create larger holographic object volumes and corresponding viewing volumes that can accommodate a wide range of viewers (e.g., from one person to several thousand people). The LF display modules presented in this disclosure may be constructed such that the entire surface of the LF display includes holographic imaging optics, with no inactive or dead space and no need for bezels. In these embodiments, the LF display modules may be tiled such that the imaging area is continuous across the seams between the LF display modules, and the join lines between the tiled modules are substantially undetectable using eye vision. Notably, in some configurations, portions of the display surface may not include holographic imaging optics, not described in detail herein.

[0020] The flexible size and / or shape of the viewing volume 130 allows the viewer 140 to be unconstrained within the viewing volume 130. For example, the viewer 140 can move to different positions within the viewing volume 130 and see different views of the holographic object 120 from corresponding viewpoints. For illustration, with reference to FIG. 1 , the viewer 140 is in a first position relative to the holographic object 120 such that the holographic object 120 is seen in a head-on view of the dolphin. The viewer 140 can move to other locations relative to the holographic object 120 to see different views of the dolphin. For example, the viewer 140 can move to see the left side of the dolphin, the right side of the dolphin, etc., as if the viewer 140 were looking at an actual dolphin and change the viewer's relative position to the actual dolphin to see different views of the dolphin. In some embodiments, the holographic object 120 is visible to all viewers within the viewing volume 130 who have an unobstructed (i.e., unobstructed by an object / person) line of sight to the holographic object 120. These viewers may be unconstrained so that they can move around within the viewing volume to see different perspectives of the holographic object 120. Thus, the LF display system can present a holographic object in a manner that allows multiple unconstrained viewers to simultaneously view the holographic object in real-world space from different perspectives, as if the holographic object were physically present.

[0021] In contrast, traditional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) generally require each viewer to wear some external device (e.g., 3D glasses, near-eye display, or head-mounted display) to view the content. Additionally and / or alternatively, traditional displays may require the viewer to be constrained to a particular viewing position (e.g., to a chair with a fixed location relative to the display). For example, when viewing an object displayed by a stereoscopic display, the viewer always focuses on the display surface rather than the object, and the display always presents only two views of the object that follow the viewer as they try to move around the perceived object, causing distortions in the perception of that object. However, in light field displays, the viewer of a holographic object presented by an LF display system does not need to wear an external device or be confined to a particular position to view the holographic object. The LF display system presents the holographic object in a way that appears to the viewer nearly the same as a physical object appears to the viewer, without requiring special eyewear, glasses, or head-mounted accessories. Furthermore, the viewer can view the holographic content from any location within the viewing volume.

[0022] In particular, the potential locations of holographic objects within the holographic object volume 160 are limited by the size of the volume. To increase the size of the holographic object volume 160, the size of the display area 150 of the LF display module 110 may be increased and / or multiple LF display modules may be tiled together to form a seamless display surface. A seamless display surface has an effective display area that is larger than the display areas of the individual LF display modules. Some embodiments related to tiling of LF display modules are discussed below with respect to FIGS. 4A-4F and 6-9. As illustrated in FIG. 1, the display area 150 is rectangular, resulting in a holographic object volume 160 that is pyramidal. In other embodiments, the display area may have some other shape (e.g., hexagonal) that also affects the shape of the corresponding viewing volume.

[0023] Additionally, while the above discussion has focused on presenting the holographic object 120 within a portion of the holographic object volume 160 that is between the LF display module 110 and the viewer 140, the LF display module 110 can additionally present content to the holographic object volume 160 behind the plane of the display area 150. For example, the LF display module 110 can make the display area 150 appear to be the surface of the ocean with the holographic object 120 popping out of it. The displayed content may also be such that the viewer 140 can see underwater marine life through the displayed surface. Additionally, the LF display system can generate content that moves seamlessly around the holographic object volume 160, including behind and in front of the plane of the display area 150.

[0024] 2A is a cross-section 200 of a portion of an LF display module 210, according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a different display area shape than the display area 150. In an exemplary embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 have different components than those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be distributed among components in a different manner than described herein.

[0025] The display system described herein presents an emission of energy that replicates the energy that normally surrounds real-world objects, where the emitted energy is directed in a specific direction from every coordinate on the display surface. In other words, various coordinates on the display surface act as projection locations for the emitted energy. The directed energy from the display surface allows for the convergence of many energy rays, thereby creating a holographic object. For example, in visible light, the LF display projects a very large number of rays from the projection locations that may converge at any point within the holographic object volume, so that from the perspective of a viewer located farther away than the object being projected, those rays will appear to come from the surface of a real-world object located in this area of ​​space. In this way, the LF display is generating reflected rays that, from the viewer's perspective, appear to come from the surface of such an object. The viewer's viewpoint can be changed on any particular holographic object, and the viewer will see different views of that holographic object.

[0026] The energy device layer 220 includes one or more electronic displays (e.g., light-emitting displays such as OLEDs) and one or more other energy projection and / or energy receiving devices described herein. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include multiple pixels, each having an intensity that is individually controlled. Many types of commercial displays can be used within the LF display, such as light-emitting LED and OLED displays.

[0027] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic projection devices generate one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or some combination thereof. An array of ultrasonic pressure waves may be used (e.g., at the surface of the holographic object 250) for volumetric haptics. The audible pressure waves are used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming the holographic object 250 is a dolphin, one or more acoustic projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the dolphin so that the viewer can touch the holographic object 250, and (2) provide audio content corresponding to sounds made by the dolphin, such as snapping, high-pitched sounds, shrieking, etc. The acoustic receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audible pressure waves within a localized area of ​​the LF display module 210.

[0028] The energy device layer 220 may also include one or more image sensors. The image sensors may be sensitive to light in the visible light band, and possibly other bands (e.g., infrared). The image sensors may be, for example, a complementary metal oxide semiconductor (CMOS) array, a charge-coupled device (CCD), an array of photodetectors, some other sensor that captures light, or some combination thereof. The LF display system may use data captured by the one or more image sensors for viewer location tracking.

[0029] In some configurations, the energy relay layer 230 relays energy (e.g., electromagnetic energy, mechanical pressure waves, etc.) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270 and relays energy between the two surfaces. The first surface 265 of each energy relay element may be coupled to one or more energy devices (e.g., electronic displays or acoustic projection devices). The energy relay elements may be composed of, for example, glass, carbon, optical fiber, optical film, plastic, polymer, or some combination thereof. Additionally, in some embodiments, the energy relay elements may adjust the magnification (increase or decrease) of the energy passing between the first surface 265 and the second surface 270. If the relay provides magnification, the relay may take the form of an array of bonded tapered relays, called tapers, where one end of the taper may be substantially larger in area than the opposite end. The larger ends of the tapers can be plastered together to form a seamless energy surface 275. One advantage is that space is created at the smaller ends of each taper to accommodate the mechanical envelope of multiple energy sources, such as the bezels of multiple displays. This extra space allows energy sources to be placed side-by-side on the smaller taper side, with each energy source having an active area within the smaller taper surface that directs energy and is relayed to the larger seamless energy surface. Another advantage of using a taper relay is that the combined seamless energy surface formed by the larger ends of the tapers has no non-imaging dead space. Because there are no borders or bezels, the seamless energy surfaces can be tiled together to form a larger surface that is substantially seamless according to the visual acuity of the eye.

[0030] The second surfaces of adjacent energy relay elements together form an energy plane 275. In some embodiments, the separation between the edges of adjacent energy relay elements is less than the minimum perceptible contour defined, for example, by the visual acuity of a human eye having 20 / 40 vision, such that the energy plane 275 is virtually seamless from the perspective of a viewer 280 within a viewing volume 285.

[0031] In some embodiments, the second surfaces of adjacent energy relay elements are fused together using a process that may include one or more of pressure, heat, and chemical reaction such that no seams exist between them. Yet, in other embodiments, the array of energy relay elements is formed by molding one side of a continuous block of relay material into an array of small tapered ends, each configured to transport energy from an energy device attached to the small tapered end into a single bonded surface having a large undivided area.

[0032] In some embodiments, one or more energy relay elements exhibit energy localization, such that the efficiency of energy transport in a longitudinal direction substantially perpendicular to surfaces 265 and 270 is much higher than the efficiency of transport in a perpendicular transverse plane, and the energy density is highly localized in this transverse plane as an energy wave propagates between surfaces 265 and 270. This energy localization allows energy distributions, such as images, to be efficiently relayed between these surfaces without significant loss of resolution.

[0033] The energy waveguide layer 240 uses the waveguide elements in the energy waveguide layer 240 to direct energy from locations (e.g., coordinates) on the energy plane 275 into specific energy propagation paths from the display surface outward into the holographic viewing volume 285. The energy propagation paths are defined by at least two angular dimensions determined by the coordinate location of the energy plane relative to the waveguide. The waveguide is associated with spatial 2D coordinates. These four coordinates together form a four-dimensional (4D) energy field. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 direct light from locations on the seamless energy plane 275 along different propagation directions through the viewing volume 285. In various examples, the light is directed according to a 4D light field function to form a holographic object 250 in the holographic object volume 255.

[0034] Each waveguide element in the energy waveguide layer 240 may be, for example, a lenslet composed of one or more elements. In some configurations, the lenslet may be a positive lens. The positive lens may have a spherical, aspheric, or freeform surface profile. Additionally, in some embodiments, some or all of the waveguide elements may include one or more additional optical components. The additional optical components may be, for example, energy suppression structures such as baffles, positive lenses, negative lenses, spherical lenses, aspheric lenses, freeform lenses, liquid crystal lenses, liquid lenses, refractive elements, diffractive elements, or some combination thereof. In some embodiments, at least one of the lenslets and / or additional optical components may also dynamically adjust refractive power. For example, the lenslet may be a liquid crystal lens or a liquid lens. The dynamic adjustment of the surface profile, the lenslet and / or the at least one additional optical component may provide additional directional control of the light projected from the waveguide element.

[0035] In an illustrative example, the holographic object volume 255 of the LF display has a boundary formed by rays 256 and 257, but may be formed by other rays. The holographic object volume 255 is a continuous volume that extends both in front of the energy waveguide layer 240 (i.e., toward the viewer 280) and behind it (i.e., away from the viewer 280). In an illustrative example, rays 256 and 257 are projected from opposite edges of the LF display module 210 at the largest angle relative to the normal of the display surface 277 that can be perceived by the user, but they may be other projected rays. These rays define the field of view of the display and therefore the boundary of the holographic viewing volume 285. In some cases, these rays define a holographic viewing volume (e.g., an ideal viewing volume) in which the entire display may be observed without vignetting. As the field of view of the display increases, the convergence points of rays 256 and 257 become closer to the display. Thus, a display with a wider field of view allows viewer 280 to view the entire display at a closer viewing distance. Additionally, light rays 256 and 257 may form an ideal holographic object volume. A holographic object presented within the ideal holographic object volume may be viewed anywhere within viewing volume 285.

[0036] In some examples, the holographic object may be presented in only a portion of the viewing volume 285. In other words, the holographic object volume may be divided into any number of viewing sub-volumes (e.g., viewing sub-volume 290). Additionally, the holographic object may be projected outside the holographic object volume 255. For example, holographic object 251 is presented outside the holographic object volume 255. Because holographic object 251 is presented outside the holographic object volume 255, it is not visible from all locations within the viewing volume 285. For example, holographic object 251 may be visible from locations within the viewing sub-volume 290, but not from locations of the viewer 280.

[0037] For example, to illustrate viewing holographic content from different viewing sub-volumes, turn to FIG. 2B. FIG. 2B illustrates a cross section 200 of a portion of an LF display module, according to one or more embodiments. The cross section of FIG. 2B is the same as the cross section of FIG. 2A. However, FIG. 2B shows a different set of light rays projected from the LF display module 210. Light rays 256 and light rays 257 still form the holographic object volume 255 and the viewing volume 285. However, as shown, the light rays projected from the top of the LF display module 210 and the bottom of the LF display module 210 overlap to form various viewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D) within the viewing volume 285. A viewer in a first viewing sub-volume (e.g., 290A) may be able to perceive holographic content presented in holographic object volume 255 that is not perceivable by viewers in the other viewing sub-volumes (e.g., 290B, 290C, and 290D).

[0038] More simply, as illustrated in FIG. 2A, holographic object volume 255 is a volume in which a holographic object may be presented by the LF display system such that the holographic object may be perceived by a viewer (e.g., viewer 280) within viewing volume 285. Thus, viewing volume 285 is an example of an ideal viewing volume, and holographic object volume 255 is an example of an ideal object volume. However, in various configurations, a viewer may perceive a holographic object presented by LF display system 200 within other exemplary holographic object volumes. More generally, "eyeline guidelines" apply when viewing holographic content projected from an LF display module. Eyeline guidelines assert that a line formed by the viewer's eye position and the holographic object being viewed must intersect the LF display surface.

[0039] When viewing the holographic content presented by the LF display module 210, each eye of the viewer 280 will see a different perspective of the holographic object 250 because the holographic content is presented according to a 4D light field function. Furthermore, as the viewer 280 moves within the viewing volume 285, the viewer will see different perspectives of the holographic object 250 as other viewers within the viewing volume 285 would see. As will be appreciated by those skilled in the art, 4D light field functions are well known in the art and will not be described in further detail herein.

[0040] As described in more detail herein, in some embodiments, the LF display can project more than one type of energy. For example, the LF display can project two types of energy, such as, for example, mechanical and electromagnetic energy. In this configuration, the energy relay layer 230 can include two separate energy relays that are interleaved together at the energy plane 275, but are separated such that the energy is relayed to two different energy device layers 220. Here, one relay can be configured to transport electromagnetic energy and another relay can be configured to transport mechanical energy. In some embodiments, the mechanical energy can be projected from a location between the electromagnetic waveguide elements on the energy waveguide layer 240, which helps to form a structure that inhibits light from being transported from one electromagnetic waveguide element to another. In some embodiments, the energy waveguide layer 240 can also include waveguide elements that transport focused ultrasound along a specific propagation path according to display instructions from the controller.

[0041] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy plane 275 is an output plane formed using one or more adjacent electronic displays in the energy device layer 220. Also, in some embodiments without an energy relay layer, the separation between the edges of adjacent electronic displays is less than the smallest perceptible contour defined by the visual acuity of a human eye with 20 / 40 vision, so that the energy plane is virtually seamless from the perspective of a viewer 280 within the viewing volume 285.

[0042] LF Display Module FIG. 3A is a perspective view of an LF display module 300A according to one or more embodiments. The LF display module 300A may be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 300A may be some other LF display module. In an exemplary embodiment, the LF display module 300A includes an energy device layer 310, an energy relay layer 320, and an energy waveguide layer 330. The LF display module 300A is configured to present holographic content from a display surface 365 as described herein. For convenience, the display surface 365 is illustrated as a dashed outline on the frame 390 of the LF display module 300A, but more precisely, it is the surface directly in front of the waveguide element bounded by the inner rim of the frame 390. The display surface 365 includes multiple projection locations to which energy may be projected. Some embodiments of the LF display module 300A have different components than those described herein. For example, in some embodiments, the LF display module 300A does not include the energy relay layer 320. Similarly, functionality may be distributed among components in different ways than described herein.

[0043] Energy device layer 310 is one embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three are visible in the figure). The energy devices 340 may all be of the same type (e.g., all electronic displays) or may include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0044] The energy relay layer 320 is one embodiment of the energy relay layer 230. The energy relay layer 320 includes four energy relay devices 350 (three are visible in the figure). The energy relay devices 350 may all relay the same type of energy (e.g., light) or may relay one or more different types (e.g., light and sound). Each of the relay devices 350 includes a first surface and a second surface, where the second surfaces of the energy relay devices 350 are arranged to form a single seamless energy surface 360. In an exemplary embodiment, each of the energy relay devices 350 is tapered such that the first surface has a smaller surface area than the second surface, which allows the mechanical envelope of the energy device 340 to be housed at the smaller end of the taper. This allows the seamless energy surface to be borderless, since the entire area may project energy. This means that this seamless energy surface can be tiled by placing multiple instances of the LF display module 300A together without dead space or bezels, such that the entire combined surface is seamless. In other embodiments, the first surface and the second surface have the same surface area.

[0045] The energy waveguide layer 330 is one embodiment of the energy waveguide layer 240. The energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIGS. 2A-2B, the energy waveguide layer 330 is configured to direct energy from a seamless energy plane 360 ​​along specific propagation paths according to a 4D light field function to form a holographic object. Note that in an exemplary embodiment, the energy waveguide layer 330 is bounded by a frame 390. In other embodiments, the frame 390 is absent and / or the thickness of the frame 390 is reduced. Removal or reduction of the thickness of the frame 390 may facilitate tiling of the LF display module 300A with additional LF display modules.

[0046] Note that in the exemplary embodiment, the seamless energy surface 360 ​​and the energy waveguide layer 330 are planar. In alternative embodiments not shown, the seamless energy surface 360 ​​and the energy waveguide layer 330 may be curved in one or more dimensions.

[0047] The LF display module 300A may be configured with additional energy sources present on the surface of the seamless energy surface, allowing the projection of energy fields in addition to the light field. In one embodiment, an acoustic energy field may be projected from an electrostatic speaker (not illustrated) mounted at any number of locations on the seamless energy surface 360. Additionally, the electrostatic speaker of the LF display module 300A is positioned within the light field display module 300A such that the dual energy surface simultaneously projects a sound field and holographic content. For example, the electrostatic speaker may be formed of one or more diaphragm elements that are transparent to some wavelengths of electromagnetic energy and driven with one or more conductive elements (e.g., planes that sandwich one or more diaphragm elements). The electrostatic speaker may be mounted on the seamless energy surface 360 ​​such that the diaphragm elements cover some of the waveguide elements. The conductive electrodes of the speaker may be collocated with a structure designed to suppress light transmission between the electromagnetic waveguides and / or may be located at a location between the electromagnetic waveguide elements (e.g., frame 390). In various configurations, the speakers can project multiple sources of focused ultrasonic energy that produce audible sound and / or a tactile surface.

[0048] In some configurations, the energy devices 340 may sense energy. For example, the energy devices may be microphones, light sensors, acoustic transducers, etc. Thus, the energy relay devices may also relay energy from the seamless energy surface 360 ​​to the energy device layer 310. That is, the seamless energy surface 360 ​​of the LF display module forms a bidirectional energy surface when the energy devices and the energy relay devices 340 are configured to simultaneously emit and sense energy (e.g., emit a light field and sense sound).

[0049] More broadly, the energy device 340 of the LF display module 300A can be either an energy source or an energy sensor. The LF display module 300A can include various types of energy devices that function as energy sources and / or energy sensors to facilitate the projection of high quality holographic content to a user. Other sources and / or sensors include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, and the like. Many other sensors or sources are possible. Additionally, the LF display modules can be tiled such that the LF display modules can form an assembly that projects and senses multiple types of energy from a large aggregate seamless energy surface.

[0050] In various embodiments of the LF display module 300A, the seamless energy surface 360 ​​may have various surface portions, with each surface portion configured to project and / or emit a particular type of energy. For example, if the seamless energy surface is a dual energy surface, the seamless energy surface 360 ​​includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy may be located on the seamless energy surface 360 ​​between electromagnetic waveguide elements and / or may be co-located with structures designed to inhibit light transmission between the electromagnetic waveguide elements. In examples where the seamless energy surface is a bidirectional energy surface, the energy relay layer 320 may include two types of energy relay devices interleaved in the seamless energy surface 360. In various embodiments, the seamless energy surface 360 ​​may be configured such that the portions of the surface under any particular waveguide element 370 are all energy sources, all energy sensors, or a mix of energy sources and energy sensors.

[0051] FIG. 3B is a cross-sectional view of an LF display module 300B including interleaved energy relay devices, according to one or more embodiments. The energy relay device 350A transports energy between the first surface 345A of the energy relay connected to the energy device 340A and the seamless energy plane 360. The energy relay 350B transports energy between the first surface 345B of the energy relay connected to the energy device 340B and the seamless energy plane 360. Both relay devices are interleaved in an interleaved energy relay device 352 connected to the seamless energy plane 360. In this configuration, the plane 360 ​​includes interleaved energy locations of both energy devices 340A and 340B, which may be energy sources or energy sensors. Thus, the LF display module 300B can be configured either as a dual energy projection device for projecting two or more types of energy, or as a bidirectional energy device for simultaneously projecting one type of energy and sensing another type of energy. LF display module 300B may be LF display module 110 and / or LF display module 210. In other embodiments, LF display module 300B may be some other LF display module.

[0052] The LF display module 300B includes many components configured similarly to those of the LF display module 300A of FIG. 3A. For example, in an exemplary embodiment, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330 that include at least the same functionality as described with respect to FIG. 3A. Additionally, the LF display module 300B can present and / or receive energy from a display surface 365. In particular, the components of the LF display module 300B are alternatively connected and / or oriented differently than the components of the LF display module 300A of FIG. 3A. Some embodiments of the LF display module 300B have different components than those described herein. Similarly, functionality may be distributed among the components in a different manner than described herein. FIG. 3B illustrates a design of a single LF display module 300B that can be tiled to create a dual energy projection surface or a bidirectional energy surface with a larger area.

[0053] In one embodiment, the LF display module 300B is an LF display module of a bidirectional LF display system. The bidirectional LF display system can project energy and simultaneously sense energy from a display surface 365. The seamless energy surface 360 ​​includes both energy projection locations and energy sensing locations closely interleaved on the seamless energy surface 360. Thus, in the example of FIG. 3B, the energy relay layer 320 is configured in a different manner than the energy relay layer of FIG. 3A. For convenience, the energy relay layer of the LF display module 300B is referred to herein as an "interleaved energy relay layer."

[0054] The interleaved energy relay layer 320 includes two legs, a first energy relay device 350A and a second energy relay device 350B. Each of the legs is illustrated as a lightly shaded area in FIG. 3B. Each of the legs is made of a flexible relay material and can be formed with a length sufficient for use with energy devices of various sizes and shapes. In some areas of the interleaved energy relay layer, the two legs are tightly interleaved together as they approach a seamless energy surface 360. In the illustrative example, the interleaved energy relay device 352 is illustrated as a darkly shaded area.

[0055] The energy relay devices are configured to relay energy between different energy devices while being interleaved in a seamless energy plane 360. The energy devices are in the energy device layer 310. As illustrated, energy device 340A is connected to energy relay device 350A and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device may be an energy source or an energy sensor.

[0056] The energy waveguide layer 330 includes a waveguide element 370 for directing energy waves from the seamless energy surface 360 ​​along a projected path toward a series of convergence points. In this example, a holographic object 380 is formed at the series of convergence points. Notably, as illustrated, the convergence of energy at the holographic object 380 occurs on the viewer side (i.e., in front) of the display surface 365. However, in other examples, the convergence of energy may be anywhere within the holographic object volume that extends both in front of the display surface 365 and behind the display surface 365. The waveguide element 370 can simultaneously direct the incoming energy to an energy device (e.g., an energy sensor), as described below.

[0057] In one exemplary embodiment of the LF display module 300B, a light-emitting display is used as the energy source (e.g., energy device 340A) and an image sensor is used as the energy sensor (e.g., energy device 340B). In this manner, the LF display module 300B can simultaneously project holographic content and detect light from a volume in front of the display surface 365. In this manner, this embodiment of the LF display module 300B functions as both an LF display and an LF sensor.

[0058] In one embodiment, the LF display module 300B is configured to simultaneously project a light field from a projection location on the display surface in front of the display surface and capture a light field from in front of the display surface at the projection location. In this embodiment, the energy relay device 350A connects a first set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340A. In one example, the energy device 340A is a light-emitting display having an array of source pixels. The energy relay device 340B connects a second set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340B. In one example, the energy device 340B is an image sensor having an array of sensor pixels. The LF display module 300B can be configured such that the locations in the seamless energy surface 365 under a particular waveguide element 370 are all light-emitting display locations, all image sensor locations, or some combination of these locations. In other embodiments, the bidirectional energy surface can project and receive various other forms of energy.

[0059] In another exemplary embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, in one embodiment, the energy device 340A is a light-emitting display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. As such, both light and sound can be projected from various locations on the seamless energy surface 360. In this configuration, the energy relay device 350A connects the energy device 340A to the seamless energy surface 360 ​​and relays the electromagnetic energy. The energy relay device is configured to have properties that make the transport of electromagnetic energy efficient (e.g., change the refractive index). The energy relay device 350B connects the energy device 340B to the seamless energy surface 360 ​​and relays the mechanical energy. The energy relay device 350B is configured to have properties for efficient transport of ultrasonic energy (e.g., distribution of materials with different acoustic impedances). In some embodiments, the mechanical energy can be projected from locations between the waveguide elements 370 on the energy waveguide layer 330. The locations of projecting mechanical energy may form structures that serve to inhibit light from being transported from one electromagnetic waveguide element to another. In one example, a spatially separated array of locations of projecting ultrasonic mechanical energy may be configured to create three-dimensional haptic shapes and surfaces in air. The surfaces may coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delays and amplitude variations across the array may assist in creating haptic shapes.

[0060] In various embodiments, the LF display module 300B with interleaved energy relay devices may include multiple energy device layers, with each energy device layer including a particular type of energy device. In these examples, the energy relay layers are configured to relay the appropriate type of energy between the seamless energy plane 360 ​​and the energy device layer 310.

[0061] Tiled LF Display Module 4A is a perspective view of a portion of an LF display system 400 that is two-dimensionally tiled to form a single-sided seamless surface environment, according to one or more embodiments. The LF display system 400 includes a plurality of LF display modules that are tiled to form an array 410. More specifically, each small square in the array 410 represents a tiled LF display module 412. The LF display module 412 may be the same as the LF display module 300A or 300B. The array 410 may cover, for example, some or all of a surface (e.g., a wall) of a room. The LF array may cover other surfaces, such as, for example, a tabletop, an advertising board, a rotunda, etc.

[0062] Array 410 can project one or more holographic objects. For example, in an exemplary embodiment, array 410 projects holographic object 420 and holographic object 422. Tiling of LF display module 412 not only allows for a much larger viewing volume, but also allows objects to be projected farther away from array 410. For example, in an exemplary embodiment, the viewing volume is approximately the entire area in front of and behind array 410, rather than a localized volume in front of (and behind) LF display module 412.

[0063] In some embodiments, the LF display system 400 presents the holographic object 420 to a viewer 430 and a viewer 434. The viewer 430 and the viewer 434 receive different perspectives of the holographic object 420. For example, the viewer 430 is presented with a straight-on view of the holographic object 420, while the viewer 434 is presented with a more oblique view of the holographic object 420. As the viewer 430 and / or the viewer 434 move, they are presented with different perspectives of the holographic object 420. This allows the viewer to visually interact with the holographic object by moving relative to the holographic object. For example, as the viewer 430 walks around the holographic object 420, the viewer 430 sees different sides of the holographic object 420, as long as the holographic object 420 remains within the holographic object volume of the array 410. Thus, the viewer 430 and the viewer 434 can simultaneously see the holographic object 420 in real-world space as if the holographic object 420 were actually there. Additionally, the viewer 430 and the viewer 434 do not need to wear an external device to view the holographic object 420, since the holographic object 420 appears to the viewer in much the same way that a physical object appears. Additionally, the holographic object 422 is illustrated here behind the array, since the viewing volume of the array extends behind the surface of the array. In this way, the holographic object 422 can be presented to the viewer 430 and / or the viewer 434. The holographic object 422 can be a virtual image 424, since the image originates from behind the plane of the LF display.

[0064] In some embodiments, LF display system 400 may include a tracking system that tracks the positions of viewer 430 and viewer 434. In some embodiments, the tracked positions are the positions of the viewers. In other embodiments, the tracked positions are the positions of the viewers' eyes. Eye position tracking differs from gaze tracking, which tracks where the eyes are looking (e.g., using orientation to determine gaze location). The eyes of viewer 430 and the eyes of viewer 434 are in different locations.

[0065] In various configurations, the LF display system 400 may include one or more tracking systems. For example, in the exemplary embodiment of FIG. 4A, the LF display system includes a tracking system 440 that is external to the array 410. Here, the tracking system may be a camera system coupled to the array 410. The external tracking system is described in more detail with respect to FIG. 5A. In other exemplary embodiments, the tracking system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340) of one or more LF display modules 412 including a bidirectional energy surface included in the array 410 may be configured to capture an image of a viewer in front of the array 410. In either case, the tracking system of the LF display system 400 determines tracking information about a viewer (e.g., viewer 430 and / or viewer 434) viewing the holographic content presented by the array 410.

[0066] The tracking information represents the position of the viewer or the position of a portion of the viewer (e.g., one or both eyes of the viewer, or a limb of the viewer) in space (e.g., relative to the tracking system). The tracking system can use any number of depth determination techniques to determine the tracking information. Depth determination techniques may include, for example, structured light, time of flight, stereo imaging, some other depth determination technique, or some combination thereof. The tracking system may include various systems configured to determine the tracking information. For example, the tracking system may include one or more infrared sources (e.g., structured light sources), one or more image sensors capable of capturing images in infrared (e.g., red-blue-green-infrared cameras), and a processor that executes a tracking algorithm. The tracking system may determine the viewer's position using depth estimation techniques. In some embodiments, the LF display system 400 generates a holographic object based on the tracked positions, actions, or gestures of the viewer 430 and / or the viewer 434 as described herein. For example, the LF display system 400 may generate a holographic object in response to a viewer coming within a threshold distance and / or a particular position of the array 410.

[0067] LF display system 400 can present one or more holographic objects customized for each viewer based in part on the tracking information. For example, viewer 430 can be presented with holographic object 420, but not with holographic object 422. Similarly, viewer 434 can be presented with holographic object 422, but not with holographic object 420. For example, LF display system 400 tracks the positions of each of viewer 430 and viewer 434. LF display system 400 determines the viewpoint of the holographic object that should be seen by the viewer based on the viewer's position relative to where the holographic object should be presented. LF display system 400 selectively projects light from specific pixels that correspond to the determined viewpoint. Thus, viewer 434 and viewer 430 can potentially have completely different experiences simultaneously. In other words, LF display system 400 can present holographic content to viewing sub-volumes of a viewing volume (i.e., similar to viewing sub-volumes 290A, 290B, 290C, and 290D shown in FIG. 2B). For example, as illustrated, LF display system 400 can track the location of viewer 430 so that LF display system 400 can present space content (e.g., holographic object 420) to a viewing sub-volume surrounding viewer 430 and safari content (e.g., holographic object 422) to a viewing sub-volume surrounding viewer 434. In contrast, conventional systems would have to use individual headsets to provide a similar experience.

[0068] In some embodiments, the LF display system 400 may include one or more sensory feedback systems. The sensory feedback systems provide other sensory stimuli (e.g., touch, sound, or smell) that enhance the holographic objects 420 and 422. For example, in the exemplary embodiment of FIG. 4A, the LF display system 400 includes a sensory feedback system 442 external to the array 410. In one example, the sensory feedback system 442 may be an electrostatic speaker coupled to the array 410. The external sensory feedback system is described in more detail with respect to FIG. 5A. In other exemplary embodiments, the sensory feedback system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340A of FIG. 3B) of the LF display module 412 included in the array 410 may be configured to project ultrasonic energy to a viewer in front of the array and / or receive image information from a viewer in front of the array. In either case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by array 410 (e.g., holographic object 420 and / or holographic object 422).

[0069] The LF display system 400 may include a sensory feedback system 442 including one or more acoustic projection devices external to the array. Alternatively or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410 as described herein. The acoustic projection device may consist of an array of ultrasound sources configured to project a volumetric tactile surface. In some embodiments, the tactile surface may coincide with the holographic object (e.g., at the surface of the holographic object 420) on one or more surfaces of the holographic object when a portion of the viewer comes within a threshold distance of the one or more surfaces. Volumetric haptics may allow a user to touch and feel the surface of the holographic object. The multiple acoustic projection devices may project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Thus, the ultrasound pressure waves and / or the audible pressure waves may serve to complement the holographic object.

[0070] In various embodiments, the LF display system 400 can provide other sensory stimuli based in part on the tracked position of the viewer. For example, the holographic object 422 illustrated in FIG. 4A is a lion, and the LF display system 400 can cause the holographic object 422 to roar both visually (i.e., the holographic object 422 appears to roar) and audibly (i.e., one or more sound projection devices project pressure waves that the viewer 430 perceives as a lion's roar emanating from the holographic object 422).

[0071] Note that in the example configuration, the holographic viewing volume may be limited in a manner similar to the viewing volume 285 of the LF display system 200 of Figures 2A-2B. This can limit the amount of perceived immersion a viewer may experience with a single wall display unit. One way to address this is to use multiple LF display modules tiled along multiple sides, as described below with respect to Figures 4B-4F.

[0072] FIG. 4B is a perspective view of a portion of a LF display system 402 in a multi-sided seamless surface environment, according to one or more embodiments. The LF display system 402 is substantially similar to the LF display system 400, except that multiple LF display modules are tiled to create a multi-sided seamless surface environment. More specifically, the LF display modules are tiled to form an array that is a six-sided collective seamless surface environment. In FIG. 4B, multiple LF display modules cover all walls, ceiling, and floor of a room. In other embodiments, multiple LF display modules may cover some, but not all, of the walls, floor, ceiling, or some combination thereof. In other embodiments, multiple LF display modules are tiled to form some other collective seamless surface. For example, walls may be curved such that a cylindrical collective energy environment is formed. Additionally, as described below with respect to FIGS. 6-9, in some embodiments, the LF display modules may be tiled to form a surface (e.g., walls, etc.) of a venue.

[0073] The LF display system 402 can project one or more holographic objects. For example, in an exemplary embodiment, the LF display system 402 projects the holographic object 420 into an area surrounded by a six-sided aggregated seamless surface environment. In this example, the viewing volume of the LF display system is also included within the six-sided aggregated seamless surface environment. Note that in an exemplary configuration, the viewer 434 can be positioned between the holographic object 420 and the LF display module 414 that is projecting the energy (e.g., light and / or pressure waves) used to form the holographic object 420. Thus, the positioning of the viewer 434 may prevent the viewer 430 from perceiving the holographic object 420 formed from the energy from the LF display module 414. However, in an exemplary configuration, there is at least one other LF display module, such as, for example, the LF display module 416, that is not obstructed (e.g., by the viewer 434) and that can project energy to form the holographic object 420 and be observed by the viewer 430. Thus, while viewer occlusion in the space may obscure parts of the holographic projection, this effect is much smaller than if a holographic display panel were present on only one side of the volume. Holographic object 422 is illustrated "outside" the walls of the six-sided aggregated seamless surface environment because the holographic object volume extends behind the aggregated surfaces. Viewer 430 and / or viewer 434 may therefore perceive holographic object 422 as "outside" the enclosed six-sided environment, which the viewer may move throughout.

[0074] As discussed above with reference to Figure 4A, in some embodiments, the LF display system 402 may actively track the viewer's position and dynamically instruct different LF display modules to present holographic content based on the tracked position. Thus, the multi-surface configuration may provide a more robust environment (e.g., as compared to Figure 4A) for presenting holographic objects, where an unconstrained viewer may freely move throughout an area surrounded by the multi-surface seamless surface environment.

[0075] In particular, various LF display systems may have different configurations. Furthermore, each configuration may have a particular orientation of the surfaces that together form a seamless display surface ("aggregate surface"). That is, the LF display modules of an LF display system may be tiled to form various aggregate surfaces. For example, in FIG. 4B, LF display system 402 includes LF display modules that are tiled to form a six-sided aggregate surface that approximates the walls of a room. In some other examples, the aggregate surface may occur only on a portion of a surface (e.g., half of a wall) rather than the entire surface (e.g., the entire wall). Several examples are described herein.

[0076] In some configurations, a collective surface of an LF display system may include a collective surface configured to project energy toward a localized viewing volume. Projecting energy into a localized viewing volume may enable a higher quality viewing experience, for example, by increasing the density of projected energy within a particular viewing volume, widening the FOV of a viewer within that volume, and moving the viewing volume closer to the display surface.

[0077] For example, Figure 4C shows a top view of an LF display system 450A with a collective surface in a "wing" configuration. In this example, the LF display system 450A is located in a room having a front wall 452, a back wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown). The first side wall 456, the second side wall 458, the back wall 454, the floor, and the ceiling are all orthogonal. The LF display system 450A includes LF display modules tiled to form a collective surface 460 that covers the front wall. The front wall 452, and therefore the collecting surface 460, includes three portions: (i) a first portion 462 (i.e., a central surface) that is generally parallel to the rear wall 454; (ii) a second portion 464 (i.e., a first side surface) that connects the first portion 462 to the first side 456 and is angled to project energy toward the center of the room; and (iii) a third portion 466 (i.e., a second side surface) that connects the first portion 462 to the second side 458 and is angled to project energy toward the center of the room. The first portion is a vertical plane in the room and has a horizontal and vertical axis. The second and third portions are angled toward the center of the room along the horizontal axis.

[0078] In this example, the viewing volume 468A of the LF display system 450A is in the center of the room and is partially surrounded by three portions of the collecting surface 460. The collecting surfaces that at least partially surround the viewer (the "surrounding surfaces") increase the viewer's immersive experience.

[0079] For illustration, consider, for example, a collective surface having only a central surface. Referring to FIG. 2A, rays projected from either edge of the display surface create an ideal holographic volume and an ideal viewing volume as described above. Now, consider, for example, the case where the central surface includes two side surfaces angled toward the viewer. In this case, rays 256 and 257 will be projected at a larger angle from the normal of the central surface. Thus, the field of view of the viewing volume will be enlarged. Similarly, the holographic viewing volume will be closer to the display surface. Additionally, a holographic object projected at a certain distance from the display surface will be closer to its viewing volume because the second and third two parts are tilted closer to the viewing volume.

[0080] Simply put, a display surface with only a central surface has a planar field of view, a planar threshold separation between the (central) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer increases the field of view relative to the planar field, decreases the separation between the display surface and the viewing volume relative to the planar separation, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further increases the field of view, decreases the separation, and increases the proximity. In other words, the angled arrangement of the side surfaces increases the immersive experience of the viewer.

[0081] Additionally, as described below with respect to FIG. 6, deflection optics can be used to optimize the size and position of the viewing volume for LF display parameters (eg, dimensions and FOV).

[0082] Returning to Figure 4D, in a similar example, Figure 4D shows a top view of an LF display system 450B having a collective surface in a "slope" configuration. In this example, the LF display system 450B is located in a room having a front wall 452, a back wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. The first side wall, second side wall, back wall 454, floor 474, and ceiling 472 are all orthogonal. The LF display system 450B includes LF display modules tiled to form a collective surface 460 that covers the front wall. The front wall 452, and therefore the collecting surface 460, includes three portions: (i) a first portion 462 (i.e., a central surface) that is generally parallel to the rear wall 454; (ii) a second portion 464 (i.e., a first side surface) that connects the first portion 462 to the ceiling 472 and is angled to project energy toward the center of the room; and (iii) a third portion 464 (i.e., a second side surface) that connects the first portion 462 to the floor 474 and is angled to project energy toward the center of the room. The first portion is a vertical plane in the room, having a horizontal and vertical axis. The second and third portions are angled toward the center of the room along the vertical axis.

[0083] In this example, the viewing volume 468B of the LF display system 450B is in the center of the room and is partially surrounded by three portions of the collecting surface 460. Similar to the configuration shown in FIG. 4C, two side portions (e.g., second portion 464 and third portion 466) are angled to surround the viewer and form a surrounding surface. The surrounding surface expands the viewing FOV from any viewer's viewpoint within the holographic viewing volume 468B. Additionally, the surrounding surface allows the viewing volume 468B to be closer to the surface of the display so that projected objects appear closer. In other words, the angled arrangement of the side surfaces expands the field of view, reduces separation, and increases the proximity of the collecting surface, thereby increasing the immersive experience of the viewer. Additionally, as discussed below, polarizing optics can be used to optimize the size and position of the viewing volume 468B.

[0084] The tilted configuration of the side portions of collecting surface 460 allows holographic content to be presented closer to the viewing volume 468B than if the third portion 466 was not tilted. For example, the lower extremities (such as legs) of a form in which a character is presented, with a tilted configuration of the LF display system, may appear closer and more realistic than if a LF display system with a flat front wall were used.

[0085] Additionally, the configuration of the LF display system and the environment in which it is located may inform the shape and location of the viewing volume and viewing sub-volumes.

[0086] 4E, for example, illustrates a top view of an LF display system 450C having a collection surface 460 on a front wall 452 of a room. In this example, an LF display system 450D is located in a room having a front wall 452, a back wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown).

[0087] The LF display system 450C projects various light rays from the collecting surface 460. From each location on the display surface, light rays are projected at a range of angles centered on the viewing volume. Light rays projected from the left side of the collecting surface 460 have a horizontal angular range 481, light rays projected from the right side of the collecting surface have a horizontal angular range 482, and light rays projected from the center of the collecting surface 460 have a horizontal angular range 483. Between these points, the projected light rays can take intermediate values ​​of the angular range, as described below with respect to FIG. 6. In this way, a viewing volume 468C is created by having a gradient deflection angle for the projected light rays across the display surface. Furthermore, this configuration avoids wasting display resolution when projecting light rays onto the side walls 456 and 458.

[0088] FIG. 4F illustrates a side view of an LF display system 450D having a collecting surface 460 on a front wall 452 of a room. In this example, the LF display system 450E is located in a room having a front wall 452, a back wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. In this example, the floor is stepped such that each step is higher in a staircase as one moves from the front wall to the back wall. Here, each step of the floor contains a viewing subvolume (e.g., viewing subvolumes 470A and 470B). The stepped floor allows for non-overlapping viewing subvolumes. That is, each viewing subvolume has a line of sight from the viewing subvolume to the collecting surface 460 that does not pass through another viewing subvolume. In other words, this orientation creates a "stadium seating" effect, with the vertical offset between the steps allowing for unobstructed lines of sight, allowing each step to "see through" the viewing subvolumes of the other steps. LF display systems that include non-overlapping viewing sub-volumes can provide a higher quality viewing experience than LF display systems with overlapping viewing volumes. For example, in the configuration shown in FIG. 4F, different holographic content can be projected to an audience in viewing sub-volumes 470A and 470B.

[0089] Controlling the LF Display System FIG. 5A is a block diagram of an LF display system 500, according to one or more embodiments. The LF display system 500 comprises an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display modules 512 may include a source / sensor system 514 that includes an integrated energy source and / or energy sensor that projects and / or senses other types of energy. The controller 520 includes a data store 522, a network interface 524, and an LF processing engine 530. The controller 520 may also include a tracking module 526, and a viewer profiling module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. The LF display systems described in the context of FIGS. 1, 2, 3, and 4 are embodiments of the LF display system 500. In other embodiments, the LF display system 500 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or different entities in ways different than described herein. Applications of LF display system 500 are also discussed in more detail below with respect to Figures 6-9.

[0090] The LF display assembly 510 provides holographic content within a holographic object volume that is visible to a viewer located within the viewing volume. The LF display assembly 510 can provide the holographic content by executing display instructions received from the controller 520. The holographic content can include one or more holographic objects projected in front of a collecting surface, onto the LF display assembly 510, behind the collecting surface of the LF display assembly 510, or some combination thereof. Generation of display instructions using the controller 520 is described in more detail below.

[0091] The LF display assembly 510 provides holographic content using one or more LF display modules (e.g., any of the LF display module 110, the LF display system 200, and the LF display module 300) included in the LF display assembly 510. For convenience, the one or more LF display modules may be described herein as LF display modules 512. The LF display modules 512 may be tiled to form the LF display assembly 510. The LF display modules 512 may be structured as various seamless surface environments (e.g., single surface, multi-surface, venue wall, curved surface, etc.). That is, the tiled LF display modules form a collective surface. As described above, the LF display module 512 includes an energy device layer (e.g., energy device layer 220) and an energy waveguide layer (e.g., energy waveguide layer 240) that present the holographic content. The LF display module 512 may also include an energy relay layer (e.g., energy relay layer 230) that transfers energy between the energy device layer and the energy waveguide layer when presenting the holographic content.

[0092] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing, as described above. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and integrated energy sensing system of the LF display module 512 may be collectively described herein as a source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 such that the source / sensor system 514 shares the same seamless energy surface as the LF display module 512. In other words, the collective surface of the LF display assembly 510 includes the functionality of both the LF display module 512 and the source / sensor module 514. That is, the LF assembly 510 including the LF display module 512 with the source / sensor system 514 can simultaneously project energy and / or sense energy while projecting a light field. For example, the LF display assembly 510 may include an LF display module 512 and a source / sensor system 514 configured as a dual energy plane or a bidirectional energy plane as described above.

[0093] In some embodiments, the LF display system 500 uses the sensory feedback system 570 to augment the generated holographic content with other sensory content (e.g., conditioned touch, sound, or smell, etc.). The sensory feedback system 570 can augment the projection of the holographic content by executing display instructions received from the controller 520. In general, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., the sensory feedback system 442) external to the LF display assembly 510. Some exemplary sensory feedback devices may include conditioned sound projection and receiving devices, aroma projection devices, temperature regulation devices, force actuation devices, pressure sensors, transducers, etc. In some cases, the sensory feedback system 570 may have similar functionality as the light field display assembly 510, or vice versa. For example, both the sensory feedback system 570 and the light field display assembly 510 may be configured to generate a sound field. As another example, the sensory feedback system 570 may be configured to generate a tactile surface, while the light field display assembly 510 is not.

[0094] To illustrate, in one exemplary embodiment of the light field display system 500, the sensory feedback system 570 may include one or more acoustic projection devices. The one or more acoustic projection devices are configured to generate one or more pressure waves that complement the holographic content when executing the display instructions received from the controller 520. The generated pressure waves may be, for example, audible (for sound), ultrasonic (for touch), or some combination thereof. Similarly, the sensory feedback system 570 may include an aroma projection device. The projection device may be configured to impart an aroma to some or all of the target area when executing the display instructions received from the controller. The aroma device may be tied into an air circulation system (e.g., ducts, fans, vents, etc.) to regulate airflow within the target area. Additionally, the sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature of some or all of the target area when executing the display instructions received from the controller 520.

[0095] In some embodiments, the sensory feedback system 570 is configured to receive input from a viewer of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the viewer. The sensory feedback devices can include devices such as acoustic receiving devices (e.g., microphones), pressure sensors, joysticks, motion detectors, transducers, etc. The sensory feedback system can transmit the detected input to the controller 520 to adjust the generation of the holographic content and / or sensory feedback.

[0096] To illustrate, in one exemplary embodiment of a light field display assembly, the sensory feedback system 570 includes a microphone. The microphone is configured to record sounds (e.g., gasps, screams, laughter, etc.) made by one or more viewers. The sensory feedback system 570 provides the recorded sounds to the controller 520 as viewer input. The controller 520 can use the viewer input to generate the holographic content. Similarly, the sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure a force applied to the pressure sensor by the viewer. The sensory feedback system 570 can provide the measured force to the controller 520 as viewer input.

[0097] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to determine the position, movement, and / or characteristics of a viewer within a target area. Generally, the tracking devices are external to the LF display assembly 510. Some exemplary tracking devices include a camera assembly ("camera"), a depth sensor, structured light, a LIDAR system, a card scanning system, or other tracking device capable of tracking a viewer within a target area.

[0098] The tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. In some cases, however, when presenting holographic content, the target area is illuminated with natural light and / or ambient light from the LF display assembly 510. The energy sources project light when executing instructions received from the controller 520. The light may be, for example, a structured light pattern, a pulse of light (e.g., an IR flash), or some combination thereof. The tracking system may project light within the visible band (approximately 380 nm to 750 nm), within the infrared (IR) band (approximately 750 nm to 1700 nm), within the ultraviolet band (10 nm to 380 nm), some other portion of the electromagnetic spectrum, or some combination thereof. Sources may include, for example, light emitting diodes (LEDs), micro LEDs, laser diodes, TOF depth sensors, tunable lasers, etc.

[0099] The tracking system 580, when executing instructions received from the controller 520, can adjust one or more emission parameters. An emission parameter is a parameter that affects how light is projected from a source of the tracking system 580. Emission parameters can include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, any other parameter that affects how light is projected from the source assembly, or any combination thereof. In one embodiment, the source projects pulses of light in a time-of-flight operation.

[0100] The camera of the tracking system 580 captures an image of the light (e.g., structured light pattern) reflected from the target area. The camera captures the image as it executes the tracking instructions received from the controller 520. As previously described, the light may be projected by the source of the tracking system 580. The camera may include one or more cameras. That is, the camera may be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by the tracking system 580, or some combination thereof. In one embodiment, the tracking system 580 may include a light field camera external to the LF display assembly 510. In other embodiments, the camera is included as part of the LF display source / sensor module 514 included in the LF display assembly 510. For example, as previously described, if the energy relay element of the light field module 512 is a bidirectional energy layer that interleaves both a light-emitting display and an image sensor with the energy device layer 220, the LF display assembly 510 may be configured to simultaneously project a light field and record image information from a viewing area in front of the display. In one embodiment, the images captured from the bidirectional energy surface form a light field camera. The camera provides the captured images to the controller 520.

[0101] The cameras of tracking system 580 may adjust one or more imaging parameters when executing tracking instructions received from controller 520. Imaging parameters are parameters that affect how the cameras capture images. Imaging parameters may include, for example, frame rate, aperture, gain, exposure length, frame timing, rolling shutter or global shutter capture mode, any other parameter that affects how the cameras capture images, or any combination thereof.

[0102] The controller 520 controls the LF display assembly 510 and any other components of the LF display system 500. The controller 520 comprises a data store 522, a network interface 524, a tracking module 526, a viewer profiling module 528, and a light field processing engine 530. In other embodiments, the controller 520 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different from that described herein. For example, the tracking module 526 may be part of the LF display assembly 510 or the tracking system 580.

[0103] The data store 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by the camera, one or more viewer profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 including the resolution and orientation of the LF module 512, desired viewing volume shape, 3D models, scenes and environments, content for graphics creation including materials and textures, other information that may be used by the LF display system 500, or some combination thereof. The data store 522 is a memory such as a read only memory (ROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), or some combination thereof.

[0104] The network interface 524 allows the light field display system to communicate with other systems or environments over a network. In one example, the LF display system 500 receives holographic content from a remote light field display system via the network interface 524. In another example, the LF display system 500 uses the network interface 524 to transmit holographic content to a remote data store.

[0105] The tracking module 526 tracks viewers watching content presented by the LF display system 500. To do so, the tracking module 526 generates tracking instructions that control the operation of sources and / or cameras of the tracking system 580 and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking inputs to the tracking module 526.

[0106] The tracking module 526 may determine the location of one or more viewers (e.g., sitting in seats at a venue) within the target area. The determined location may be, for example, relative to some reference point (e.g., the display surface). In other embodiments, the determined location may be within a virtual model of the target area. The tracked location may be, for example, a tracked location of the viewers and / or a tracked location of a portion of the viewers (e.g., eye location, hand location, etc.). The tracking module 526 determines the location using one or more images captured from cameras of the tracking system 580. The cameras of the tracking system 580 may be distributed around the LF display system 500 and may capture images in stereo, allowing the tracking module 526 to passively track the viewers. In other embodiments, the tracking module 526 actively tracks the viewers. That is, the tracking system 580 illuminates a portion of the target area, images the target area, and the tracking module 526 determines the location using time-of-flight and / or structured light depth determination techniques. The tracking module 526 uses the determined positions to generate tracking information.

[0107] The tracking module 526 can also receive tracking information as input from a viewer of the LF display system 500. The tracking information can include body movements of the viewer corresponding to various input options provided by the LF display system 500. For example, the tracking module 526 can track the body movements of the viewer and assign any of the various movements as input to the LF processing engine 530. The tracking module 526 can provide the tracking information to the data store 522, the LF processing engine 530, the viewer profiling module 528, any other component of the LF display system 500, or some combination thereof.

[0108] To provide context for the tracking module 526, consider an exemplary embodiment of the LF display system 500 displaying a play in which a player scores a winning touchdown. In response to the scene, the viewer pumps their fist to indicate excitement. The tracking system 580 can record the viewer's hand movements and transmit the recording to the tracking module 526. This can be accomplished with the tracking system 580 consisting of a camera, depth sensor, or other device external to the light field display assembly 510, as described above, or simultaneously with a display surface that projects a light field image and records the image, where the image recorded from the display surface may be a light field image, or with any combination of these devices. The tracking module 526 tracks the viewer's hand movements in the recording and transmits input to the LF processing engine 530. As described below, the viewer profiling module 528 determines that information in the image indicates that the viewer's hand movements are associated with a positive response. Thus, if a sufficient number of viewers are recognized to have a positive response, the LF processing engine 530 generates appropriate holographic content celebrating the touchdown. For example, the LF processing engine 530 may project confetti onto a scene.

[0109] The LF display system 500 includes a viewer profiling module 528 configured to identify and profile viewers. The viewer profiling module 528 generates a profile of one or more viewers viewing holographic content displayed by the LF display system 500. The viewer profiling module 528 generates the viewer profile based in part on viewer input and monitored viewer behavior, actions, and reactions. The viewer profiling module 528 can access information obtained from the tracking system 580 (e.g., recorded images, videos, audio, etc.) and process the information to determine various information. In various examples, the viewer profiling module 528 can use any number of machine vision or machine hearing algorithms to determine viewer behavior, actions, and reactions. Monitored viewer behaviors can include, for example, smiles, cheers, applause, laughter, fear, screams, excitement levels, recoil, changes in other gestures or movements by the viewer, etc.

[0110] More generally, a viewer profile may include any information received and / or determined regarding a viewer viewing holographic content from the LF display system. For example, each viewer profile may record that viewer's actions or responses to content displayed by the LF display system 500. Some example information that may be included in a viewer profile is provided below.

[0111] In some embodiments, the viewer profile may describe the viewer's response to the displayed performers (e.g., actors, singers, etc.), venues (e.g., stages, concert halls, etc.), etc. For example, the viewer profile may indicate that the viewer has a generally positive response to the performers performing as a group of handsome male singers with matte blonde hair, ages 18-25.

[0112] In some embodiments, the viewer profile may indicate characteristics of a viewer viewing a performance. For example, a viewer at a venue is wearing a sweatshirt with a university logo on it. In this case, the viewer profile may indicate that the viewer is likely to prefer holographic content associated with the university that the sweatshirt is associated with. More broadly, viewer characteristics that may be indicated in a viewer profile may include, for example, age, gender, ethnicity, clothing, viewing location at the venue, etc.

[0113] In some embodiments, the viewer profile may indicate viewer preferences with respect to desired performance and / or venue characteristics. For example, the viewer profile may indicate that the viewer only wants to view holographic content appropriate for the age of each member of their household. In another example, the viewer profile may indicate holographic object volumes in which holographic content should be displayed (e.g., on walls) and holographic object volumes in which holographic content should not be displayed (e.g., above the viewer's head). The viewer profile may also indicate that the viewer wants haptic interfaces presented near them or wants to avoid haptic interfaces.

[0114] In another example, the viewer profile indicates a history of performances viewed for a particular viewer. For example, the viewer profiling module 528 determines that a viewer or group of viewers have attended a performance before. Thus, the LF display system 500 may display different holographic content than the viewer last time they attended the performance. As an example, a performance including holographic content may have three different endings, and the LF display system 500 may display the different endings based on the viewer attending. In another example, each of the three endings may be presented in a different viewing volume of the same venue.

[0115] In some embodiments, the viewer profile may also indicate characteristics and preferences of a group of viewers, rather than a particular viewer. For example, the viewer profiling module 528 may generate a viewer profile for an audience member viewing a performance at a venue. In one example, the viewer profiling module 528 creates a viewer profile for an audience member viewing a performance about a teenager who is in love with a vampire. The profile indicates that 86.3% of the audience member are female, between the ages of 20 and 35, and have had a positive response to the performance. The profile also indicates that the remaining 23.7% of the audience member are male, between the ages of 20 and 35, and have had a negative response to the performance. Any of the foregoing information and characteristics may be applied to a group of viewers.

[0116] The viewer profiling module 528 may also access profiles associated with a particular viewer or viewers from one or more third party systems to build a viewer profile. For example, a viewer purchases a ticket to a performance using a third party vendor that is linked to the viewer's social media account. Thus, the viewer's ticket is linked to the viewer's social media account. When the viewer uses the ticket to enter the venue for the performance, the viewer profiling module 528 may access information from the viewer's social media account to build (or extend) the viewer profile.

[0117] In some embodiments, the data store 522 includes a viewer profile store that stores viewer profiles generated, updated, and / or maintained by the viewer profiling module 528. The viewer profiles may be updated in the data store at any time by the viewer profiling module 528. For example, in one embodiment, the viewer profile store receives and stores information about a particular viewer in a viewer profile for that viewer when that viewer views holographic content provided by the LF display system 500. In this example, the viewer profiling module 528 includes a facial recognition algorithm that can recognize the viewer and reliably identify the viewer when the viewer views the presented holographic content. To illustrate, when a viewer enters a target area of ​​the LF display system 500, the tracking system 580 captures an image of the viewer. The viewer profiling module 528 inputs the captured image and uses a facial recognition algorithm to identify the viewer's face. The identified face is associated with a viewer profile in the profile store, such that all input information captured about that viewer may be stored in that viewer's profile. The viewer profiling module may also utilize a card identification scanner, a voice identifier, a radio frequency identification (RFID) chip scanner, a bar code scanner, etc. to positively identify the viewer.

[0118] In embodiments where the viewer profiling module 528 can reliably identify a viewer, the viewer profiling module 528 can determine each visit of each viewer to the LF display system 500. The viewer profiling module 528 can then store the date and time of each visit in a viewer profile for each viewer. Similarly, the viewer profiling module 528 can store input from the viewer received from any combination of the sensory feedback system 570, the tracking system 580, and / or the LF display assembly 510 as the input occurs. The viewer profile system 528 can additionally receive further information about the viewer from other modules or components of the controller 520, which information can be stored with the viewer profile. Other components of the controller 520 can then also access the stored viewer profile to determine future content to be provided to that viewer.

[0119] The LF processing engine 530 generates holographic content consisting of light field data as well as data for all sensory regions supported by the LF display system 500. For example, the LF processing engine 530 can generate 4D coordinates in a rasterized format ("rasterized data") that, when executed by the LF display assembly 510, causes the LF display assembly 510 to present the holographic content. The LF processing engine 530 can access the rasterized data from the data store 522. Additionally, the LF processing engine 530 can construct the rasterized data from a vectorized data set. The vectorized data is described below. The LF processing engine 530 can also generate sensory instructions necessary to provide sensory content that augments the holographic object. As described above, the sensory instructions, when executed by the LF display system 500, can generate haptic surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF data processing engine 530 can access sensory instructions from the data store 522 or construct sensory instructions to form a vectorized data set. Collectively, the 4D coordinates and sensory data represent holographic content as display instructions executable by the LF display system to generate holographic and sensory content. More generally, holographic content can take the form of CG content with idealized light field coordinates, live action content, rasterized data, vectorized data, electromagnetic energy transported by a set of relays, instructions sent to a group of energy devices, energy locations on one or more energy planes, a set of energy propagation paths projected from a display surface, a holographic object visible to a viewer or spectator, and many other similar forms.

[0120] The amount of rasterized data describing the flow of energy through the various energy sources in the LF display system 500 is incredibly large. While it is possible to display the rasterized data on the LF display system 500 when accessed from the data store 522, it is not possible to efficiently transmit (e.g., via the network interface 524), receive, and then display the rasterized data on the LF display system 500. Take, for example, rasterized data representing a short performance for holographic projection by the LF display system 500. In this example, the LF display system 500 includes a display including several gigapixels, and the rasterized data includes information for each pixel location on the display. The corresponding size of the rasterized data is enormous (e.g., many gigabytes per second of performance display time) and unmanageable for efficient transfer over a commercial network via the network interface 524. The problem of efficient transfer may be amplified in applications involving live streaming of holographic content. If an interactive experience is desired using input from the sensory feedback system 570 or tracking module 526, additional problems with simply storing rasterized data in the data store 522 arise. To enable interactive experiences, the light field content generated by the LF processing engine 530 may be modified in real-time in response to sensory or tracking input. In other words, in some cases, the LF content cannot simply be read from the data store 522.

[0121] Thus, in some configurations, data representing holographic content for display by the LF display system 500 may be transferred to the LF processing engine 530 in a vectorized data format ("vectorized data"). The vectorized data may be orders of magnitude smaller than rasterized data. Furthermore, the vectorized data provides high image quality while having a data set size that allows for efficient sharing of the data. For example, the vectorized data may be a sparse data set derived from a denser data set. Thus, the vectorized data may have an adjustable balance between image quality and data transmission size based on how the sparse vectorized data is sampled from the denser rasterized data. The adjustable sampling to generate the vectorized data allows for optimization of image quality at a particular network speed. As a result, the vectorized data allows for efficient transmission of the holographic content over the network interface 524. The vectorized data may also allow for live streaming of the holographic content over commercial networks.

[0122] In summary, the LF processing engine 530 can generate holographic content derived from rasterized data accessed from the data store 522, vectorized data accessed from the data store 522, or vectorized data received via the network interface 524. In various configurations, the vectorized data can be encoded by an encoder prior to data transmission and decoded by a decoder after receipt by the LF controller 520. The encoder and decoder pair can be part of the same proprietary system codec. In some examples, the vectorized data is encoded for additional data security and performance improvements associated with data compression. For example, the vectorized data received by the network interface can be encoded and vectorized data received from a holographic streaming application. In some examples, the vectorized data can require the decoder, the LF processing engine 530, or both, to access the information content encoded in the vectorized data. The encoder and / or decoder systems may be available to customers or may be licensed to third-party vendors. Other exemplary encoding and / or decoding schemes can be used to transmit and / or present holographic content.

[0123] The vectorized data includes information for each of the sensory areas supported by the LF display system 500 in a manner that can support an interactive experience. For example, the vectorized data for an interactive holographic experience can include any vectorized characteristics that can provide accurate physics for each of the sensory areas supported by the LF display system 500. The vectorized characteristics can include any characteristics that can be synthetically programmed, captured, computationally evaluated, etc. The LF processing engine 530 can be configured to convert the vectorized characteristics of the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content converted from the vectorized data using the LF display assembly 510. In various configurations, the vectorized properties may include one or more red / green / blue / alpha channel (RGBA) + depth images, multi-view images with or without depth information at various resolutions that may include one high-resolution central image and other views at lower resolution, albedo and reflectance, surface normals, other optical effects, surface identity, geometric object coordinates, virtual camera coordinates, display surface location, lighting coordinates, surface contact stiffness, contact ductility, contact intensity, amplitude and coordinates of sound fields, environmental conditions, somatosensory energy vectors associated with mechanoreceptors of texture or temperature, sound, and coordinates of other sensory domain properties. Many other vectorized properties are possible.

[0124] The LF display system 500 can also generate interactive viewing experiences. That is, the holographic content can respond to input stimuli, including information about the viewer's location, gestures, interactions, interactions with the holographic content, or other information derived from the viewer profiling module 528 and / or tracking module 526. For example, in one embodiment, the LF processing system 500 uses vectorized data of a real-time performance received via the network interface 524 to create an interactive viewing experience. In another example, if a holographic object needs to immediately move in a particular direction in response to a viewer's interaction, the LF processing engine 530 can update the rendering of the scene so that the holographic object moves in that requested direction. This may require the LF processing engine 530 to use the vectorized data set to render a light field in real time based on a 3D graphical scene with proper object placement and movement, collision detection, occlusion, color, shading, lighting, etc., to properly respond to the viewer's interactions. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510. The LF display system 500 may employ a variety of other encoding / decoding techniques that enable the LF display system to present holographic content in near real-time.

[0125] The rasterized data includes holographic content instructions and sensory instructions (display instructions) that represent the real-time performance. The LF display assembly 510 executes the display instructions to simultaneously project the holographic and sensory content of the real-time performance. The LF display system 500 monitors viewer interactions (e.g., voice responses, touch, etc.) with the presented real-time performance content using a tracking module 526 and a viewer profiling module 528. In response to viewer interactions, the LF processing engine can create an interactive experience by generating additional holographic and / or sensory content for display to the viewer.

[0126] To illustrate, consider an exemplary embodiment of the LF display system 500 including an LF processing engine 530 that generates a number of holographic objects representing balloons dropping from the ceiling of a venue during a performance. A viewer may move to touch the holographic objects representing the balloons. In response, a tracking system 580 tracks the movement of the viewer's hands relative to the holographic objects. The viewer's movements are recorded by the tracking system 580 and transmitted to the controller 520. The tracking module 526 continuously determines the viewer's hand movements and transmits the determined movements to the LF processing engine 530. The LF processing engine 530 determines the placement of the viewer's hands in the scene and adjusts the real-time rendering of the graphics to include any required changes (such as position, color, or occlusion) to the holographic objects. The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a tactile surface using a volumetric haptic projection system (e.g., using ultrasonic speakers). The generated haptic surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the exterior surface of the holographic object. The LF processing engine 530 uses the tracking information to dynamically direct the LF display assembly 510 to move the position of the haptic surface along with the position of the rendered holographic object such that the viewer is given both the visual and haptic perception of touching the balloon. In simpler terms, as the viewer sees his or her hand touching the holographic balloon, the viewer simultaneously feels haptic feedback indicating that the hand is touching the holographic balloon and that the balloon changes position or motion in response to the touch. In some examples, rather than presenting the interactive balloon at a performance accessed from the data store 522, the interactive balloon may be received as part of the holographic content received from a live streaming application via the network interface 524.In other words, the holographic content displayed by the LF display system 500 can be a live stream of holographic content.

[0127] The LF processing engine 530 may provide holographic content for display to audiences at the venue before, during, and / or after a performance to enhance the venue experience. The holographic content may be provided by a publisher of the performance, provided by the venue, provided by an advertiser, generated by the LF processing engine 530, etc. The holographic content may be content associated with the performance, the genre of the performance, the location of the venue, advertisements, etc. In either case, the holographic content may be stored in the data store 522 or streamed to the LF display system 500 in a vectorized format via the network interface 524. For example, a performance may be shown in a venue augmented with LF display modules on the walls. A distributor of the performance may provide holographic content to present on the wall displays before the performance begins. The LF processing engine 530 accesses the holographic content and presents the accessed content from the venue's wall displays before the performance begins. In another example, a venue with the LF display system 500 is in San Francisco. A holographic representation of the Golden Gate Bridge is stored in the venue and is presented to the venue prior to the performance if no performance-specific content has been provided. Here, since no performance-specific holographic content has been provided, the LF processing engine 530 accesses and presents the Golden Gate Bridge in the venue. In another example, an advertiser has provided holographic content for its product as an advertisement to the venue to display after the film. After the performance has ended, the LF processing engine 530 presents the advertisement as the audience leaves the venue. In another example, as described below, the LF processing engine can dynamically generate holographic content for display on the walls of the theater.

[0128] The LF processing engine 500 may also modify the holographic content to fit the venue in which it is being presented. For example, not all venues are the same size, have the same number of seats, or have the same technical configuration. Thus, the LF processing engine 530 can modify the holographic content so that it appears properly within the venue. In one embodiment, the LF processing engine 530 can access a configuration file for the venue, including the venue's layout, resolution, field of view, other technical specifications, etc. The LF processing engine 530 can render and present the holographic content based on the information contained in the configuration file.

[0129] The LF processing engine 530 can also create holographic content for display by the LF display system 500. Importantly, creating holographic content for display is distinct from accessing or receiving holographic content for display. That is, when creating content, the LF processing engine 530 generates entirely new content for display rather than accessing previously generated and / or received content. The LF processing engine 530 can create holographic content for display using information from the tracking system 580, the sensory feedback system 570, the viewer profiling module 528, the tracking module 526, or some combination thereof. In some examples, the LF processing engine 530 can access information from elements of the LF display system 500 (e.g., tracking information and / or viewer profile) and, accordingly, create holographic content based on that information and display the created holographic content using the LF display system 500. The created holographic content can be augmented with other sensory content (e.g., touch, sound, or smell) when displayed by the LF display system 500. Additionally, the LF display system 500 can store the generated holographic content so that it can be displayed in the future.

[0130] Dynamic Content Generation for LF Display Systems In some embodiments, the LF processing engine 530 incorporates artificial intelligence (AI) models to create holographic content for display by the LF display system 500. The AI ​​models may include supervised or unsupervised learning algorithms, including but not limited to regression models, neural networks, classifiers, or other AI algorithms. The AI ​​models may be used to determine viewer preferences based on viewer information recorded by the LF display system 500 (e.g., by the tracking system 580), which may include information about viewer behavior.

[0131] The AI ​​model can access information from the data store 522 to create the holographic content. For example, the AI ​​model can access viewer information from one or more viewer profiles in the data store 522 or can receive viewer information from various components of the LF display system 500. To illustrate, the AI ​​model can determine that a viewer enjoys viewing holographic content in which a performer wears a bow tie. The AI ​​model can determine a preference based on a group of viewers' positive reactions or responses to previously viewed holographic content that includes an actor wearing a bow tie. That is, the AI ​​model can create holographic content that is personalized for a set of viewers according to the learned viewer preferences. So, for example, the AI ​​model can create bow ties for actors that appear in holographic content viewed by a group of viewers using the LF display system 500. The AI ​​model can also store each learned viewer preference in a viewer profile store in the data store 522. In some examples, the AI ​​model can create holographic content for an individual viewer rather than a group of viewers.

[0132] One example of an AI model that may be used to identify viewer characteristics, identify responses, and / or generate holographic content based on the identified information is a convolutional neural network model with layers of nodes where the value of a node in a current layer is a transformation of the value at a node in a previous layer. The transformations in the model are determined through a set of weights and parameters that connect the current layer and the previous layer. For example, the AI ​​model may include five layers of nodes, namely layers A, B, C, D, and E. The transformation from layer A to layer B is given by function W1, the transformation from layer B to layer C is given by W2, the transformation from layer C to layer D is given by W3, and the transformation from layer D to layer E is given by W4. In some examples, the transformations may also be determined through a set of weights and parameters used to transform between previous layers in the model. For example, the transformation W4 from layer D to layer E may be based on the parameters used to perform the transformation W1 from layer A to layer B.

[0133] The input to the model may be an image captured by tracking system 580, encoded on convolutional layer A, and the output of the model is the decoded holographic content from output layer E. Alternatively or additionally, the output may be viewer features determined in the image. In this example, the AI ​​model identifies latent information in the image that represents viewer features in identification layer C. The AI ​​model reduces the dimensionality of convolutional layer A to the dimensionality of identification layer C to identify any features, actions, responses, etc. in the image. In some examples, the AI ​​model then increases the dimensionality of identification layer C to generate the holographic content.

[0134] The image from the tracking system 580 is encoded into a convolutional layer A. The image input into the convolutional layer A may be associated with various features and / or reaction information, etc., in the identification layer C. The associated information between these elements may be retrieved by applying a set of transformations between the corresponding layers. That is, the convolutional layer A of the AI ​​model represents the encoded image, and the identification layer C of the model represents the smiling viewer. The smiling viewer in a given image may be identified by applying the transformations W1 and W2 to the pixel values ​​of the image in the space of the convolutional layer A. The weights and parameters of the transformations may indicate the relationship between the information contained in the image and the identification of the smiling viewer. For example, the weights and parameters may be quantizations of the shape, color, size, etc., contained in the information representing the smiling viewer in the image. The weights and parameters may be based on historical data (e.g., previously tracked viewers).

[0135] Smiling viewers in the image are identified in identification layer C. Identification layer C represents smiling viewers identified based on latent information about smiling viewers in the image.

[0136] The identified smiling viewers in the image can be used to generate holographic content. To generate the holographic content, the AI ​​model starts with the identification layer C and applies transformations W2 and W3 to a given identified smiling viewer value in the identification layer C. The transformations result in a set of nodes in the output layer E. The weights and parameters of the transformations can indicate a relationship between the identified smiling viewer and a particular holographic content and / or preference. The holographic content may be output directly from the nodes in the output layer E, or the content generation system may decode the nodes in the output layer E into holographic content. For example, if the output is a set of identified features, the LF processing engine can use the features to generate the holographic content.

[0137] Additionally, the AI ​​model may include layers known as intermediate layers. An intermediate layer is a layer that does not correspond to an image and identifies features / responses, etc., or generates holographic content. For example, in the given example, layer B is an intermediate layer between convolutional layer A and identification layer C. Layer D is an intermediate layer between identification layer C and output layer E. Hidden layers are latent representations of various aspects of identification that are not observed in the data, but may govern the relationship between elements of the image when identifying features and generating holographic content. For example, a node in the hidden layer may have a strong connection (e.g., a large weight value) with input values ​​and identification values ​​that share a commonality of "laughing human smile". As another example, another node in the hidden layer may have a strong connection with input values ​​and identification values ​​that share a commonality of "scared human scream". Of course, there may be any number of linkages in a neural network. Additionally, each intermediate layer is a combination of functions, such as, for example, residual blocks, convolutional layers, pooling operations, skip connections, concatenation, etc. Any number of hidden layers B may serve to reduce the convolutional layer down to a discriminative layer, and any number of hidden layers D may serve to augment the discriminative layer up to an output layer.

[0138] In one embodiment, the AI ​​model includes a deterministic method trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve the quality of the performance using measurements from the tracking system 580 as inputs and modifications to the created holographic content as outputs.

[0139] Reinforcement learning is a machine learning system in which a machine learns "what to do" (how to map situations to actions) to maximize a numerical reward signal. The learner (e.g., LF processing engine 530) is not told what action to take (e.g., generating a given holographic content), but instead discovers, by trying actions, which action will yield the greatest reward (e.g., increasing the quality of the holographic content by making more people cheer). In some cases, an action may affect not only the immediate reward, but also the next situation and, through it, all subsequent rewards. These two features (trial-and-error exploration and delayed reward) are the two distinguishing features of reinforcement learning.

[0140] Reinforcement learning is defined by characterizing the learning problem, not by characterizing the learning method. Essentially, a reinforcement learning system captures the key aspects of the problem facing a learning agent that interacts with an environment to achieve a goal. That is, in the example of generating a song for a performer, a reinforcement learning system captures information about the audience in the venue (e.g., age, temperament, etc.). Such an agent senses the state of the environment and performs actions that affect the state to achieve one or more goals (e.g., create a pop song that the audience will cheer). In its most basic form, a reinforcement learning formulation includes three aspects for the learner: senses, actions, and goals. Continuing with the song example, the LF processing engine 530 senses the state of the environment with sensors in the tracking system 580, displays holographic content to the audience in the environment, and achieves a goal that is a measure of the audience's acceptance of the song.

[0141] One of the challenges that arise in reinforcement learning is the trade-off between exploration and exploitation. To increase rewards in a system, a reinforcement learning agent will favor actions that it has tried in the past and found to be effective at producing rewards. However, to discover actions that produce rewards, the learning agent will choose actions that it has not previously chosen. The agent "exploits" the information it already knows to obtain rewards, but "explores" information to select better actions in the future. The learning agent tries various actions, gradually favoring those that it thinks are best while still trying new actions. In probabilistic tasks, each action is typically tried many times to obtain a reliable estimate for the expected reward. For example, if the LF processing engine creates holographic content that it knows will cause the viewer to laugh after a long time has passed, the LF processing engine can modify the holographic content so that the time it takes for the viewer to laugh decreases.

[0142] Moreover, reinforcement learning considers the entire problem of a goal-directed agent interacting with an uncertain environment. A reinforcement learning agent has a clear goal, may sense aspects of the environment, and may choose actions to receive high rewards (i.e., a roaring crowd). Furthermore, the agent generally operates despite significant uncertainty about the environment it faces. When reinforcement learning involves planning, the system addresses the interplay between planning and real-time action selection, as well as the problem of how to acquire and improve upon environmental factors. For reinforcement learning to make progress, important sub-problems must be studied in isolation, and the sub-problems play distinct roles in a complete, interactive, goal-seeking agent.

[0143] A reinforcement learning problem is a framing of a machine learning problem in which interactions are processed and actions are taken to achieve a goal. The learner and decision maker are called the agent (e.g., LF processing engine 530). What the agent interacts with, including everything external to the agent, is called the environment (e.g., the audience in the field). These two interact continuously, with the agent choosing actions (e.g., creating holographic content) and the environment responding to those actions by presenting the agent with new situations. The environment also gives rise to a reward, a special numerical value that the agent tries to maximize over time. In one context, the reward serves to maximize the audience's positive response to the holographic content. A complete specification of the environment defines a task, which is one instance of a reinforcement learning problem.

[0144] To provide more context, the agent (e.g., the LF processing engine 530) and the environment interact at each of a sequence of discrete time steps, i.e., t=0, 1, 2, 3, etc. At each time step t, the agent determines the state s of the environment. t Receive some representation of state s t is in S, where S is the set of possible states. t Based on and time step t, the agent selects an action at (e.g., have the performer do the splits). The action at is A(s t ) and A(s t ) is a set of possible actions. After one time state, and partly as a result of the agent's actions, the agent receives a numerical reward r t+1 Receive the state r t+1 is in R, where R is the set of possible rewards. When the agent receives a reward, it creates a new state s t+1 Select .

[0145] At each time step, the agent implements a mapping from states to the probability of choosing each possible action. This mapping is called the agent's policy, π t where π t (s,a) is s t =s then a t = a. Reinforcement learning methods can dictate how the agent changes its policy as a result of the states and rewards that result from the agent's actions. The agent's goal is to maximize the total amount of rewards the agent receives over time.

[0146] This reinforcement learning framework is flexible and can be applied in many different ways to many different problems (e.g., generating holographic content).The framework proposes that any problem (or objective) of learning goal-directed behavior, whatever the details of the senses, memories, and controllers, can be boiled down to three signals passing between the agent and its environment: one signal representing the choice made by the agent (action), one signal representing the rationale for the choice made (state), and one signal defining the agent's goal (reward).

[0147] Of course, the AI ​​model may include any number of machine learning algorithms. Some other AI models that may be used are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In either case, generally, the LF processing engine 530 takes input from the tracking module 526 and / or the viewer profiling module 528, and the machine learning model creates the holographic content accordingly. Similarly, the AI ​​model may direct the rendering of the holographic content.

[0148] In one example, the LF processing engine 530 creates a virtual pop star. The LF processing engine 530 can create the virtual pop star using information contained in the viewer profiles stored in the data store 522. For example, information contained in the stored viewer profiles indicates that many viewers enjoy pop music performed by trendy female vocalists in their mid-twenties. In this manner, the LF processing engine 530 creates a virtual pop star that is displayed by the LF display system 500 as a trendy female vocalist in her mid-twenties. More specifically, the LF processing engine 530 accesses viewer profiles of audience members at the venue. The LF processing engine 530 parameterizes (e.g., quantifies) the information in each viewer profile. For example, the LF processing engine 530 can quantify characteristics such as the viewer's age, location, and gender. Additionally, the LF processing engine 530 can parameterize other information contained in the viewer profiles. For example, if the viewer profile indicates that the viewer attended four performances of a female vocalist, the content creation module can quantify this tendency (e.g., generate a score indicative of the viewer's interest in female vocalists). The LF processing engine 530 inputs the parameterized user profile into an AI model (e.g., a neural network) configured to generate characteristics of a virtual performer based on the input parameters, and receives the performer characteristics in response. The LF processing engine 530 then inputs the characteristics of the virtual performer into an AI model (e.g., a procedural generation algorithm) configured to generate a performer given the set of characteristics, and in response generates a virtual performer that is a trendy mid-20s female vocalist. Additionally, the LF processing engine 530 can create holographic content (e.g., songs, performances, etc.) that would follow the virtual pop star's persona. For example, the content generation module 530 can create a song for a virtual mid-20s female vocalist about a jealous ex-boyfriend.More specifically, the LF processing engine 530 may access information about the virtual performer's characteristics and audience and input that information into an AI model (e.g., a recurrent neural network "RNN"). Again, the features and information may be parameterized (e.g., using classification and regression trees) and input into the RNN, where the RNN may be trained using real-world songs with similar input parameters. In this way, the RNN generates virtual songs for the virtual star to the audience at the venue that share similar characteristics to songs created by other trendy female vocalists in their mid-twenties.

[0149] The LF processing engine 530 can create holographic content based on a performance shown at the venue. For example, a performance shown at the venue may be associated with a set of metadata that describes the characteristics of the performer. The metadata may include, for example, setting, genre, performer, performance type, theme, title, runtime, etc. The LF processing engine 530 can access any of the metadata describing the performance and, in response, generate holographic content for presentation within the venue. For example, a performance titled "The Last Merman" is a play about to be performed at a venue augmented with the LF display system 500. The LF processing engine 530 accesses the performance's metadata to create holographic content for the walls of the venue before the performance begins. Here, the metadata includes that the setting is underwater and the genre is romance. The LF processing engine 530 inputs the metadata into an AI model and, in response, receives holographic content for display on the walls of the venue. In this example, the LF processing engine 530 creates a seaside sunset for display on the walls of the venue before the performance begins to be performed.

[0150] In one example, the LF processing engine 530 creates holographic content based on viewers in a venue that includes the LF display system 500. For example, a group of viewers enter the venue to view a performance enhanced with holographic content displayed by the LF display system 500. The viewer profiling module 528 generates viewer profiles for the viewers in the venue and an aggregate viewer profile that represents all viewers in the venue. The LF processing engine 530 accesses the aggregate viewer profile and creates holographic content for display to the viewers in the venue. For example, the viewers in the venue are a group of couples who are viewing a famous performer for romantic ballads, and therefore the aggregate viewer profile includes information (e.g., by parameterizing and inputting into an AI model) indicating that holographic content appropriate for a dating couple would be enjoyed. Thus, the LF processing engine 530 generates holographic content such that the venue has a more romantic atmosphere (e.g., candles, dim lighting, Marvin Gaye music, etc.).

[0151] In a similar example, LF processing engine 530 may create holographic content based on previously existing content, where previously existing may be previously recorded songs, existing real-world performers, etc. For example, a viewer's favorite band is "The Town People" and they would like to see a concert of their favorite Town People songs. Thus, LF processing engine 530 creates a performance of the Town People singing "DMCA" and "In the Gravy." In this example, LF processing engine 530 may access existing songs and models of singers from Town People (e.g., stored in data store 522) and use that content to create a new performance for display by LF display system 500. More specifically, the content creation system may access the songs and models from data store 522. LF processing engine 530 inputs the songs and models into an AI model (e.g., a procedural generation algorithm) configured to create a performance by, for example, a rock band, and the AI ​​model responsively outputs holographic content of the performance. In some cases, because the LF processing engine 530 is creating holographic content based on copyrighted content, the viewer and / or the LF processing engine 530 may pay a fee to the copyright owner.

[0152] In one example, the LF processing engine 530 creates holographic content based on the response of the audience watching the performance. For example, an audience at a venue is watching a performance at a venue augmented by the LF display system 500. The tracking module 526 and the audience profiling module 528 monitor the response of the audience watching the performance. For example, the tracking module 526 may capture images of the audience as they watch the performance. The tracking module 526 may identify the audience and the audience profiling module 528 may use machine vision algorithms to determine the audience's response based on the information contained in the images. For example, an AI model may be used to identify whether the audience watching the performance is smiling and, accordingly, the audience profiling module 528 may indicate in the audience profile whether the audience has a positive or negative response to the performance based on the smiling. Other responses may also be determined. The tracking module may determine information about the audience including the viewer's location, the viewer's movement, the viewer's gestures, the viewer's expressions, the viewer's age, the viewer's gender, the viewer's ethnicity, or the clothing worn by the viewer. This information may be shared with the audience profiling module 528 to generate an audience profile. As an illustration, the LF processing engine 530 generates a comedy routine to be performed by a virtual comedian. As the LF display system 500 displays the performance, the tracking system 580 and the audience profiling module 528 monitor the audience's reactions. In this example, the audience is not laughing at the "dad joke" being told by the virtual comedian. In response, the LF processing engine 530 modifies the comedy performance such that the virtual comedian begins to make fun of the particular audience member. The LF processing engine 530 may then input this reaction into an AI model (e.g., a reinforcement learning model) configured to induce viewers watching the holographic content to laugh. Based on the audience's reactions and characteristics in the audience, the LF processing engine 530 modifies the approach to telling the joke.For example, a virtual comedian might make fun of the leopard print pants worn by a particular audience member, where the model could be trained using previously performed comedy performances and recorded user responses to those performances.

[0153] In a similar example, LF processing engine 530 may create holographic content based on previously existing content, where previously existing may be a previously recorded song, an existing real-world performer, etc. For example, a viewer's favorite band is "The Town People" and they would like to see a concert of their favorite Town People songs. Thus, LF processing engine 530 creates a performance of the Town People singing "DMCA" and "In the Gravy." In this example, LF processing engine 530 may access existing songs and models of singers from Town People (e.g., stored in data store 522) and use that content to create a new performance for display by LF display system 500. More specifically, the content creation system may access the songs and models from data store 522. LF processing engine 530 inputs the songs and models into an AI model (e.g., a procedural generation algorithm) configured to create a performance by, for example, a rock band, and the AI ​​model responsively outputs holographic content of the performance. In some cases, because the LF processing engine 530 is creating holographic content based on copyrighted content, the viewer and / or the LF processing engine 530 may pay a fee to the copyright owner.

[0154] In a similar example, the LF processing engine 530 can create holographic content based on previously existing or provided advertising content. That is, for example, the LF processing engine 530 can request an advertisement from the network system via the network interface 524, the network system provides the holographic content in response, and the LF processing engine 530 creates the holographic content for display, including the advertisement. Some examples of advertisements can include products, text, video, etc. Advertisements can be presented to a particular viewing volume based on the viewers within that viewing volume. Similarly, holographic content can augment a performance with advertisements (e.g., product placement). Most generally, the LF processing engine 530 can create the advertising content based on any of the characteristics and / or responses of the viewers within the venue, as previously described.

[0155] The above examples of creating content are not limiting. In the broadest sense, the LF processing engine 530 creates holographic content for display to a viewer of the LF display system 500. The holographic content can be created based on any of the information contained in the LF display system 500.

[0156] Holographic Content Delivery Network 5B illustrates an exemplary LF performance network 550, according to one or more embodiments. One or more LF display systems may be included in the LF performance network 550. The LF performance network 550 includes any number of LF display systems (e.g., 500A, 500B, and 500C), an LF generation system 554, and a networking system 556 coupled together via a network 552. In other embodiments, the LF performance network 550 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among different entities in a manner different than described herein.

[0157] In the illustrated embodiment, LF performance network 550 includes LF display systems 500A, 500B, and 500C that can receive holographic content over network 552 and display the holographic content to an audience. LF display systems 500A, 500B, and 500C are collectively referred to as LF display systems 500.

[0158] The LF generation system 554 is a system that generates holographic content for display in a performance venue that includes an LF display system. In other words, the LF generation system is configured to generate any of the LF content described herein. The holographic content may be a performance or may be holographic content that enhances a traditional performance. The LF generation system 554 may include a light field recording assembly consisting of any number of sensors and / or processors for recording energy data of an event and a processing engine configured to convert this recorded energy data into holographic content. For example, the sensors of the light field recording assembly may include cameras for recording images, microphones for recording audio, pressure sensors for recording interactions with objects, etc. In some examples, the light field recording assembly of the LF generation system 554 includes one or more recording modules (e.g., LF display modules configured to record energy data from the event, or simple 2D cameras for capturing images of the event) positioned around an area (e.g., a performance venue) to record the event from multiple viewpoints. In this case, the processing engine of the LF generation system 554 is configured to convert the energy from multiple viewpoints into holographic content. In some examples, the light field recording assembly includes two or more two-dimensional recording systems that are used by the processing engine to convert multiple perspectives of an event into three-dimensional holographic content. The light field recording assembly may also include other sensors, such as, for example, a depth sensor and / or a plenoptic camera.

[0159] The processing engine of the LF generation system 554 may also generate synthetic light field data from computer-generated imagery (CGI) animations. The synthetic light field data may be from, for example, virtual worlds or animated movies and may be used to augment the sensory data from the light field recording assembly. The processing engine of the LF generation system 554 combines the recorded sensory information, the synthetic data, and encodes the information as holographic and sensory content. The LF generation system 554 may transmit the encoded holographic content to one or more of the LF display systems 500 for display to the viewer. As previously discussed, for efficient transfer speeds, data from the LF display systems 500A, 500B, 500C, etc. may be transferred over the network 552 as vectorized data.

[0160] More broadly, the LF generation system 554 generates holographic content for display at the venue by using any recorded sensory data or synthetic data that can be projected by the LF display system when performing a performance. For example, the sensory data can include recorded audio, recorded images, recorded interactions with objects, etc. Many other types of sensory data can be used. For illustration, the recorded visual content can include 3D graphics scenes, 3D models, object placement, textures, colors, shading, and lighting, 2D performance data that can be converted to a holographic format using large datasets of AI models and similar performance transformations, multi-view camera data from a camera rig with many cameras with or without a depth channel, plenoptic camera data, CG content, or other recorded sensory data types described herein.

[0161] In various examples, the event recorded by one or more sensors across one or more energy regions may be a concert, a show, a scene, a singing performance, a band performance, an actor performance, a dancer performance, a magician performance, or another type of event found at a performance venue, which may include one or more of a performance hall, an event space, a theater, a concert hall, a studio, or a stage.

[0162] In some configurations, the LF production system 554 may use its own encoder to perform an encoding operation that reduces the sensory data recorded for the performance to a vectorized data format as described above. That is, encoding the data into vectorized data may include image processing, audio processing, or any other calculations that may result in a reduced data set that is easier to transmit over the network 552. The encoder may support formats used by professionals in the performance production industry. In other configurations, the LF production system may transmit the performance content to the network system 556 and or the LF display system without encoding the content.

[0163] Each LF display system (e.g., 500A, 500B, 500C) can receive encoded data from the network 552 via the network interface 524. In this example, each LF display system includes a decoder for decoding the encoded LF display data. More specifically, the LF processing engine 530 generates rasterized data for the LF display assembly 510 by applying a decoding algorithm provided by the decoder to the received encoded data. In some examples, the LF processing engine can further generate rasterized data for the LF display assembly using inputs from the tracking module 526, the viewer profiling module 528, and the sensory feedback system 570, as described herein. The rasterized data generated for the LF display assembly 510 reproduces the holographic content recorded by the LF generation system 554. Importantly, each LF display system 500A, 500B, and 500C generates rasterized data that is suitable for the particular configuration of the LF display assembly in terms of shape, resolution, etc. In some configurations, the encoding and decoding process is part of a proprietary encoding / decoding system pair that may be provided to viewing customers or licensed by a third party. In some cases, the encoding / decoding system pair may be implemented as a proprietary API that may provide a common programming interface to content creators.

[0164] In some configurations, various systems in the LF performance network 550 (e.g., LF display system 500, LF generation system 554, etc.) may have different hardware configurations. The hardware configurations may include arrangements of physical systems, energy sources, energy sensors, haptic interfaces, sensor capabilities, resolutions, fields of view, LF display module configurations, or any other hardware description of the systems in the LF performance network 550. Each hardware configuration may generate or utilize sensory data in a different data format. Thus, the decoder system may be configured to decode the encoded data for the LF display system on which it is presented. For example, an LF display system having a first hardware configuration (e.g., LF display system 500A) receives encoded data from an LF generation system having a second hardware configuration (e.g., LF generation system 554). The decoding system accesses information describing the first hardware configuration of the LF display system 500A. The decoding system decodes the encoded data using the accessed hardware configuration, such that the decoded data may be processed by the LF processing engine 530 of the receiving LF display system 500A. The LF processing engine 530 generates and presents the rasterized content of the first hardware configuration, even though it is recorded in the second hardware configuration. Similarly, the holographic content recorded by the LF generation system 554 can be presented by any LF display system (e.g., LF display system 500B, LF display system 500C) regardless of the hardware configuration. Various other aspects that may be included in the hardware configuration may include the resolution, the number of projected rays per degree, the field of view, the deflection angle on the display surface, and the dimensions of the display surface, etc. Additionally, the hardware configuration may also include the number of display panels of the LF display assembly, the relative orientation of the display panels, the height of the display panels, the width of the display panels, and the layout of the display panels.

[0165] Similarly, various systems within the LF performance network 550 may have different geometric orientations. The geometric orientation reflects the physical size, layout, and arrangement of the various modules and systems included in the LF display system. Thus, the decoder system may be configured to decode data encoded for the LF display system in the geometric configuration in which it is presented. For example, an LF display system having a first geometric configuration (e.g., LF display system 500A) receives encoded data from an LF generation system having a second geometric configuration (e.g., LF generation system 554). The decoding system accesses information describing the first geometric configuration of the LF display system 500A. The decoding system decodes the encoded data using the accessed geometric configuration, such that the decoded data may be processed by the LF processing engine 530 of the receiving LF display system 500A. The LF processing engine 530 generates and presents the content for the first geometric configuration, despite being recorded in the second geometric configuration. Similarly, the holographic content recorded by LF generation system 554 can be presented by any LF display system (e.g., LF display system 500B, LF display system 500C) regardless of the geometric configuration. Various other aspects that may be included in the geometric configuration may include the number of display panels (or surfaces) of the LF display assembly, the relative orientation of the display panels.

[0166] Similarly, various venues in the LF performance network 550 may have different configurations. The venue configuration reflects any of the number and / or location of holographic object volumes, the number and / or location of viewing volumes, and the number and / or location of viewing locations relative to the LF display system. Thus, a decoder system may be configured to decode data encoded for the LF display system at the venue where it is presented. For example, an LF display system (e.g., LF display system 500A) at a first venue receives encoded data from an LF generation system (e.g., LF generation system 554) recorded at another venue (or other space). The decoding system accesses information describing the venue. The decoding system decodes the encoded data using the accessed venue configuration, such that the decoded data may be processed by an LF processing engine 530 located at the venue. The LF processing engine 530 generates and presents the content for the venue, despite being recorded at another location.

[0167] Network system 556 is any system configured to manage the transmission of holographic content between systems in LF performance network 550. For example, network system 556 may receive requests for holographic content from LF display system 500A and facilitate transmission of holographic content from LF generation system 554 to LF display system 500A. Network system 556 may also store holographic content, viewer profiles, holographic content, etc. for transmission to and / or storage by other LF display systems 500 in LF performance network 550. Network system 556 may also include LF processing engine 530 that can create holographic content as described above.

[0168] The network system 556 may include a digital rights management (DRM) module for managing digital rights of the holographic content. For example, the LF generation system 554 can send the holographic content to the network system 556, and the DRM module can encrypt the holographic content using a digital encryption format. In another example, the LF generation system 554 encodes the recorded light field data into a holographic content format that can be managed by the DRM module. The network system 556 can provide the LF display systems with a key to the digital encryption so that each LF display system 500 can decrypt and subsequently display the holographic content to the viewer. Most commonly, the network system 556 and / or the LF generation system 554 encode the holographic content, and the LF display system can decode the holographic content.

[0169] The network system 556 can act as a repository of previously recorded and / or created holographic content. Each piece of holographic content, when received, can be associated with a transaction fee that causes the network system 556 to transmit the holographic content to the LF display system 500, which provides the transaction fee. For example, the LF display system 500A can request access to the holographic content over the network 552. The request includes a transaction fee for the holographic content. In response, the network system 556 transmits the holographic content to the LF display system for display to the viewer. In another example, the network system 556 can also act as a subscription service for the holographic content stored on the network system. In another example, the LF generation system 554 is recording light field data of a performance in real time and generating holographic content representing the performance. The LF display system 500 transmits a request for the holographic content to the LF generation system 554. The request includes a transaction fee for the holographic content. In response, the LF generation system 554 transmits the holographic content for simultaneous display on the LF display system 500. The network system 556 may act as an intermediary in exchanging transaction fees and / or managing the holographic content data flow across the network 552 .

[0170] Network 552 represents the communication paths between systems in LF performance network 550. In one embodiment, the network is the Internet, but may be any network, including, but not limited to, a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, cloud computing network, private network, or virtual private network, and any combination thereof. Additionally, all or part of the links may be encrypted using conventional encryption techniques such as Secure Socket Layer (SSL), Secure HTTP, and / or Virtual Private Network (VPN). In another embodiment, the entities may use custom and / or proprietary data communication technologies instead of or in addition to those described above.

[0171] Examples of venues 6-8 show some example venues where a LF display system (e.g., LF display system 500) can be used to display holographic content of a performance. The holographic content of the performance ("performance content") displayed in the venue can be any of the holographic content described herein. The performance content can be displayed in lieu of a live performance by an actual performer. Within the venue, any number of viewers are located in any number of viewing volumes. The LF display system is configured to display the performance content in a holographic object volume ("performance volume") such that viewers in the viewing volumes perceive the performance content. Generally, the LF display system in the venue includes an array of light field modules 210 around a performer volume that creates a seamless multi-sided LF display. The performance volume can therefore be an aggregate holographic object volume of the array of LF display modules 210.

[0172] FIG. 6 shows a side view of venue 600, a traditional theater augmented with an LF display system, according to one or more embodiments. In the theater, both the stage 602 and the walls 604 surrounding the proscenium are backed with an array 640 of LF display modules, such that the area above the stage is the performance volume 610 of the LF display system. Although the performance volume 610 (e.g., holographic object volume 255) is shown as a bounded square for clarity, the performance volume 610 shown is only a portion of the actual performance volume. For example, the performance volume could extend to the walls 604 or the stage 602. In FIG. 6, the LF display system is an embodiment of LF display system 500. Additionally, the LF display modules in array 640 are an embodiment of LF display assembly 510.

[0173] Here, the theater house is shown with three levels, but any number of levels may be included. Each level includes several viewing locations (e.g., viewing locations 622A, 622B) for viewers to view performance content 630 in a performance volume 610. As shown, the performance content is a real image 632, but may also be a virtual image. The viewing locations 622 in each level are included in a viewing volume (e.g., 620A, 620B, and 620C) of the LF display system. The LF display system may display the same or different performance content 630 to viewers in each viewing volume 620. For example, as described below, a viewer at a viewing location 622A located in a viewing volume 620A in the bottom level may view different performance content 630 than a viewer at a viewing location 622B in a viewing volume 620B in the middle level.

[0174] In other embodiments, the theater may be configured differently. For example, the stage may be sloped, the theater may include an orchestra or standing room area, etc. Any surface of the theater (proscenium arch, ceiling, backdrop, drapes, etc.) may be included in the array of LF display modules 640. Additionally, the theater may have additional tiers and viewing volumes, which may be arranged in any number of configurations.

[0175] As an example of context, the illustrated venue 600 is a theater in San Francisco that is billed to show live performer content of famous performer "NJ." However, NJ is performing at a venue in New York City. The New York City venue includes an LF generation system 554 for recording NJ's performer as performer content 630 and transmitting it over a network (e.g., network 552) to other venues. The San Francisco venue 600 includes an LF display system 500 configured to display the received performer content 630 to viewers at viewing locations 622 in a viewing volume 620. A transaction fee is paid (e.g., to network system 556 or LF generation system 554) to receive the performance content. In various examples, the fee can be paid by the owner of the venue 600, attendees at the venue 600, a performance manager, or any other agent. The performance content 630 allows the audience in San Francisco to perceive NJ as if he were performing live at their previous performance volume 610. This allows the San Francisco audience to see the live performance without having to travel to New York City.

[0176] In some embodiments, the venue 600 charges an admission fee to view the live performance display of NJ by the LF display system 500. Each viewing location 622 may have a different admission fee, and generally, the admission fee for the bottom viewing volume 620A is more expensive than the top viewing volume 620C. The LF display system 500 may display different performance content 630 for each viewing volume 620 based on the admission fee for the viewing volume 620. For example, fully rendering the NJ performance in San Francisco may be more costly (e.g., processing power, energy, etc.) than partially rendering the NJ performance. Thus, the LF processing engine 530 may display only a portion of the performance content 630 in the viewing volume 620C in the top tier of the theater while playing all of the performance content 630 in the viewing volume 620A in the bottom tier of the theater. For example, when displaying only a portion of the performance content 630, the LF processing engine 530 may display the performance content 630 only in a portion of the performance volume 610 rather than the entire performance volume 610, the LF processing engine 530 may remove the performance content 630 (e.g., backup dancers, props, etc.), the LF processing engine 530 may render the performance content 630 at a lower resolution for the top viewing volume 620C than the bottom viewing volume 620A, etc.

[0177] Alternatively or additionally, the LF processing engine 530 can create holographic content to augment the performance content 630 based on admission fees for the viewing volume 620. For example, the LF processing engine 530 can create (or access from the network system 556) advertisements for displaying simultaneously with the performance content 630. The advertisements can be based on information obtained by the viewer profiling system 590 or the tracking system 580. For example, the LF processing engine 530 can access a viewer profile that includes viewer characteristics and responses to the holographic content. The LF processing engine 530 accesses an advertisement from the data store 522 associated with the viewer characteristics and responses and displays the advertisement. The venue can receive payment from an advertiser to display the advertisement when displaying the performance content 630. In another example, rather than overtly displayed advertisements, the LF processing engine 530 can augment the performance content 630 with sponsored holographic content (e.g., product placement). For example, the LF processing engine 530 can replace the clothing worn by NJ during a performance in New York City with clothing from an advertiser. Thus, a viewer viewing performance content 630 at a theater in San Francisco will perceive NJ as wearing the clothes from the advertiser, rather than the clothes he wore in New York City.

[0178] 7A shows a cross-section of another venue 700A including an LF display system for displaying performance content to viewers at viewing locations within a viewing volume, according to one or more embodiments. At the venue, the LF display system can display real-time and / or recorded holographic content, or other additional holographic content. The holographic content may include content representing concerts, performances, shows, scenes, events, and the like. The holographic content may include singers, bands, actors, dancers, comedians, performers, and the like. The holographic content may also include representations of sets, stages, props, clothing, costumes, instruments, and the like. All of the venues described herein may be configured to display similar holographic content.

[0179] Here, venue 700A is designed and constructed to display performance content 730 rather than expanding an existing venue. As shown, Figure 7A is a cross-section of venue 600 shaped as a cylindrical ring, where LF display system 740 covers the exterior surface of the inner cylinder of the ring. In Figure 7A, LF display system 740 is an embodiment of LF display system 510. Additionally, LF display system 740 is an embodiment of LF display system 500.

[0180] The venue 700A includes several viewing locations arranged in tiers, where each tier is on the inner surface of an outer cylinder with a radius larger than the radius of the inner cylinder. The viewing locations 722 of each tier are within a viewing volume (e.g., 720A, 720B, 720C, 720D, 720E, and 720F), where the performance volumes (e.g., 710A, 710B, 710C, 710D, 710E, and 710F) are located between the inner wall of the outer cylinder and the outer wall of the inner cylinder. The performance volumes are positioned such that viewers at the viewing locations 722 can perceive the performance content 730 displayed in the performance volume 710. Again, the performance volume 710 (e.g., holographic object volume 255) is shown as a bounding square for clarity, but the performance volume 710 shown is only a portion of the actual performance volume. For example, the performance volume could extend into the inner cylinder of the venue 700A.

[0181] In the illustrated example, a portion of each performance volume 710 and viewing volume 720 spatially overlaps. Although shown as partial overlap, the performance volumes and viewing volumes may spatially overlap completely. The spatial overlap is an area where viewers can interact with the performance content 730, as previously described. Additionally, users interacting with the performance content 730 in the spatial overlap area may be monitored by the sensory feedback assembly 570, and the performance content 730 may change accordingly. In some cases, the venue 700 may have a higher admission price for viewing locations 722 in areas where viewers can interact with the performance content 730.

[0182] Venues designed to display performance content using a series of LF display modules have several advantages over venues expanded using a series of LF display modules. For example, viewers in different viewing volumes of venue 700A view different performance volumes (or different sub-portions of a performance volume), while viewers in viewing volume 620 of venue 600 view the same performance volume. Thus, the LF display system of venue 700 can generate a better viewing experience for each viewing volume 720. For example, viewers in viewing volume 720A of venue 700 can be provided with the same view of performance content 730 as viewers in viewing volume 720C. However, in venue 600, viewers in viewing volume 620C view performance content 730 further away and lower than viewers in viewing volume 620A. In this way, all viewers in venue 700A can have the best seats in the house.

[0183] Additionally, managing the rendering of performance content taking into account LF display parameters is easier for performance content 730 displayed in viewing volume 710 of FIGS. 7A-7B than for performance content 630 displayed in viewing volume 620 of FIG. 6. Some LF display parameters may include, for example, resolution, field of view, projection distance, and viewing sub-volume. For illustrative purposes, the viewing volumes 720 in the venue of FIGS. 7A-7B may be designed to be approximately equal in shape and approximately equally spaced from the performance volume 710. Thus, less rendering of performance content 730 into each of the viewing volumes 720 occurs because each viewing volume 720 is the same. However, in the venue of FIG. 6, each viewing volume 620 is different, requiring independent rendering of performance content 630 for each viewing volume 620.

[0184] The venue can take on other shapes such that an array of light field display modules can display performance content within the performance volume to viewers at viewing locations within the viewing volume 720. Examples of venues of various shapes include domes, canopies, tunnels, globe theaters, etc.

[0185] For example, Figure 7B illustrates a cross-section of another venue 700B including an LF display system for displaying performance content to viewers at viewing locations within a viewing volume, according to one or more embodiments. The venue 700B is designed and constructed to display performance content 730, rather than expanding an existing venue. As shown, Figure 7B is a cross-sectional view of the venue 700B. In Figure 7B, the LF display system 740 is an embodiment of the LF display system 510. Additionally, the LF display system 740 is an embodiment of the LF display system 500.

[0186] In the illustrated example, the venue 700B resembles a circular theater with viewing locations 722 circling the stage. The floor of the stage 702 is covered with an array 740 of LF display modules such that the area above the stage forms a performance volume (e.g., performance volume 710G). The LF displays 710 present performance content in the performance volume 710G such that viewers in the venue 700B may perceive the performance content. In the venue 700B, the viewing locations 722 are positioned at an angle such that the line of sight of each viewing location allows unobstructed viewing of the performance content 730 from the viewing volume (e.g., viewing volume 720G). Here, the venue contains one viewing volume 720G such that all viewers are presented with the same performance. In other configurations, there may be one or more display volumes.

[0187] More generally, an LF display system may have a display surface that is substantially horizontal or nearly horizontal. In some examples, an LF display system may include a display surface that is (i) at least a portion of a venue floor, (ii) at least a portion of a venue stage (e.g., stage 702), and / or (iii) at least a portion of a raised viewing platform within the venue. Other types of horizontal surfaces are possible. For these configurations, viewers may be elevated relative to the display surface and look down to view holographic performance content projected from the display surface, and viewers may partially or completely surround the display surface. There are many other configurations of light field display surfaces, including vertically mounted display surfaces (450A shown in FIG. 4C, 450B shown in FIG. 4D, and 450C shown in FIG. 4E) arranged approximately in front of the LF display surface and with display positions described elsewhere in this disclosure, as well as curved LF display surfaces.

[0188] Although the performance volume 710G is shown as a bounded square for clarity, the performance volume 710G may only be a portion of the actual performance volume 710G. For example, the performance volume 710G may extend further toward the top of the venue 700B. Also, there is spatial overlap of a portion of the performance volume 710G and the viewing volume 720G. Although shown as a partial overlap, the performance volume and viewing volume may spatially overlap completely. The spatial overlap is the area where viewers can interact with the performance content 730, as discussed above.

[0189] A venue can also be a much smaller location. For example, Figure 8 shows a venue 800 that doubles as a home theater in a viewer's living room 802, according to one or more embodiments. Here, the home theater includes an LF display system 810 consisting of an array of LF display modules on one wall. In Figure 8, LF display system 810 is one embodiment of LF display system 500.

[0190] LF display system 810 may be configured such that the performance volume (e.g., holographic object volume 255) and the viewing volume fully overlap in living room 802. That is, at any viewing location (e.g., viewing location 822) in living room 802, a viewer may view and interact with performance content 830. Although viewing location 822 is shown as a chair, a viewer may move around the living room to view performance content 830. That is, viewing location 822 may be a viewer's location in living room 802.

[0191] In some embodiments, the LF display system 810 creates (or modifies) the performance content 830 based on the viewer's interaction with the performance content 830. For example, the viewer may fist-bump the performer in the performance content 830. In this situation, the viewer may say "Great, NJ" and move their hand to fist-bump the performer in the performance content 830. The tracking system 580 monitors the viewer and identifies (e.g., via machine hearing, machine vision, neural networks, etc.) that the viewer wishes to fist-bump the performer in the performance content 830. The content generation system 530 uses the monitored viewer information to create performance content (e.g., using machine learning, neural networks, or some combination thereof) that represents the performer returning the fist-bump. The tracking system 580 monitors the viewer's hand position, and when the viewer's fist and the performer's fist spatially overlap, the sensory feedback assembly 570 creates the sensation that the viewer has fist-bumped the performer in the performance content.

[0192] In some cases, the viewer can interact with the network system 556 to acquire performance content 830 in the living room 802. Continuing with the example described with respect to Figure 5B, the viewer can transmit a transaction fee to the network system 556 and the LF generation system 554 can transmit holographic content of NJ performing in New York City to the venue 800.

[0193] Displaying performance content to viewers in the venue 9 is a flow diagram of a method 900 for displaying performance content to an audience at a venue (e.g., venue 600) in the context of an LF performance network (e.g., LF performance network 550). Method 900 may include additional or fewer steps, which may occur in a different order. Additionally, various steps, or combinations of steps, may be repeated any number of times during the execution of the method.

[0194] Initially, a venue including an LF display system (e.g., LF display system 500) sends a request for a live stream (or previously recorded) of performance content (e.g., performance content 630) over a network (e.g., network 552) to a network system (e.g., network system 556) at 910. The request includes a transaction fee sufficient to pay royalties to the copyright owner of the performance content.

[0195] An LF production system (e.g., LF production system 554) records LF data of a live performance and transmits corresponding performance content to a network system that transmits the performance content to an LF display system for display at approximately the same time as it is being recorded by the LF production system.

[0196] The LF display system receives 920 the holographic performance content 630 from the network system over the network.

[0197] The LF display system determines the configuration of the LF display system and / or the performance space at 930. For example, the LF display system may access a configuration file that includes several parameters describing the HW configuration of the LF display, including resolution, projected rays per degree, field of view, deflection angle on the display surface, or dimensions of the LF viewing surface. The configuration file may also include information regarding the geometric orientation of the LF display assembly, including the number of LF display panels, relative orientation, width, height, and layout of the LF display panels. Additionally, the configuration file may include configuration parameters of the performance venue, including the holographic object volume, the viewing volume, and the position of the audience relative to the display panels.

[0198] To illustrate by way of example, the LF display system 500 determines viewing volumes (e.g., viewing volumes 620A, 620B, and 620C) for displaying performance content at 930. For example, the LF display system 500 can access information within the LF display system that describes the layout, geometric configuration, and / or hardware configuration of the venue. For illustrative purposes, the layout may include the locations, spacing, and sizes of viewing locations (e.g., viewing locations 622) within the venue. Thus, the LF display system may determine that viewing locations for a first tier of the venue are in a first viewing volume, viewing locations for a second tier of the venue are in a second viewing volume, and viewing locations for a third tier of the venue are in a third viewing volume. In various other embodiments, the LF display system may determine any number and configuration of viewing sub-volumes at any location within the venue.

[0199] The LF display system 940 generates holographic content (and other sensory content) for presentation on the LF display system based on the hardware configuration of the LF display system within the performance venue and the particular layout and configuration of the performance venue. Determining the presentation content to display can include appropriate rendering of the presentation content for the venue or viewing volume. For example, the LF display system may (i) augment the performance content for display in a third tier viewing volume with advertising and remove live performance aspects from the performance content, (ii) reduce the fidelity of the performance content for display in a second tier viewing volume of the venue, and (iii) fully render the performance content for display in a third tier viewing volume of the venue.

[0200] The LF display system presents 950 the performance content in the performance volumes of the venue such that viewers at viewing locations within each viewing volume perceive the appropriate performance content, i.e. viewers in the top viewing volume perceive the performance content with advertisements, viewers in the middle viewing volume perceive the lower resolution performance content, and viewers in the bottom viewing volume perceive the fully rendered performance content.

[0201] The LF display system may determine information about the viewers in the viewing volumes at any time while the viewers are viewing a performance. For example, a tracking system may monitor the facial responses of viewers in the viewing volumes, and a viewer profiling system may access information about the characteristics of viewers in those viewing volumes.

[0202] The LF display system may create (or modify) performance content for simultaneous display based on the determined information. For example, the LF processing engine creates a light show for simultaneous display by the LF display system based on the determined information about the audience.

[0203] Additional configuration information The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will recognize that numerous modifications and variations are possible in light of the above disclosure.

[0204] Some portions of this specification describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has proven convenient at times to refer to arrangements of these operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0205] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented in a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0206] The embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, that may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0207] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information resulting from the computing processes, which information is stored in a non-transitory, tangible, computer-readable storage medium, and may include any embodiment of the computer program products or other data combinations described herein.

[0208] Finally, the language used herein has been selected primarily for ease of reading and instructional purposes, and may not have been selected to define or limit the subject matter of the present invention. Accordingly, the scope of the present disclosure is intended to be limited not by this detailed description, but by any claims issued on an application based thereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure, which is set forth in the following claims.

[0209] [Item 1] 1. A light field (LF) display system, comprising: a network interface configured to receive the encoded holographic content over a network; a decoder configured to decode the encoded holographic content into a format presentable by an LF display assembly; and a light field display assembly configured to present the decoded holographic content to an audience at the venue. [Item 2] 2. The LF display system of claim 1, wherein an encoder decodes the encoded holographic content in near real-time. [Item 3] Item 14. The LF display system of item 1, wherein the holographic content is a live stream of an event, the holographic content being presented in near real-time. [Item 4] 2. The LF display system of claim 1, wherein the encoded holographic content is in a first format and the decoded holographic content is in a second format. [Item 5] 5. The LF display system of claim 4, wherein the first format is a vectorized data format and the second format is a rasterized data format. [Item 6] Item 2. The LF display system of item 1, wherein the decoder uses a proprietary codec to decode the encoded streaming holographic content. [Item 7] Item 2. The LF display system of item 1, wherein the holographic content is encoded in a format decodable by a proprietary codec. [Item 8] a LF processing engine configured to determine a hardware configuration of the LF display system; 2. The LF display system of claim 1, wherein the decoder decodes the encoded holographic content based on the determined hardware configuration of the LF display system. [Item 9] The hardware configuration includes: Resolution and the number of projected rays per degree; and Field of view and the deflection angle of the display surface; 8. The LF display system of item 7, comprising any one of: a. a dimension of the display surface; [Item 10] and a LF processing engine configured to determine a geometric orientation of the LF display system, Item 1. The LF display system of item 1, wherein the decoder decodes the encoded holographic content based on the determined geometric orientation of the LF display system. [Item 11] The geometric orientation is a number of display panels in the LF display assembly; and a relative orientation of the display panels; and The height of the display panel; and a width of the display panel; and 11. The LF display system of item 10, comprising: [Item 12] and a LF processing engine configured to determine a configuration of the venue. 2. The LF display system of claim 1, wherein the decoder decodes the encoded holographic content based on the determined configuration of the venue. [Item 13] The configuration of the venue, one or more holographic object volumes; One or more listening volumes; and the location of the spectator relative to the LF display assembly. [Item 14] Item 1. The LF display system of item 1, further comprising a digital rights management system configured to manage digital rights for the received encoded holographic content, enabling the LF display assembly to project decrypted holographic content for which the digital rights management system provides a digital key. [Item 15] Item 1. The LF display system of item 1, wherein the holographic content is encrypted and the LF display system is configured to decrypt the holographic content. [Item 16] 2. The LF display system of claim 1, wherein the encoded holographic content is received from a holographic content repository connected to the LF display system via the network. [Item 17] 2. The LF display system of claim 1, wherein the holographic content is received in response to the LF display system transmitting a transaction fee to a holographic content repository for the holographic content. [Item 18] The holographic content is received from a holographic content generation system, the holographic content generation system comprising: Recording live performances, encoding said recording of said performance into encoded holographic content; Item 2. The LF display system of item 1, configured to transmit the encoded holographic content to the LF display system over the network. [Item 19] 2. The LF display system of claim 1, wherein the network is a public network. [Item 20] Item 2. The LF display system of item 1, wherein the network is a private network configured to transmit holographic content. [Item 21] the LF display assembly is configured to present acoustic data; 2. The LF display system of claim 1, wherein the encoded holographic content includes acoustic data, such that when the holographic content is decoded, the LF display system presents the decoded holographic content and the decoded acoustic data. [Item 22] 2. The LF display system of claim 1, wherein the presented holographic content comprises a first type of energy and a second type of energy. [Item 23] 23. The LF display system of claim 22, wherein the first type of energy is electromagnetic energy and the second type of energy is ultrasonic energy. [Item 24] Item 24. The LF display system of item 23, wherein the first type of energy and the second type of energy are presented at the same location such that the LF display assembly presents a volumetric tactile surface. [Item 25] the LF display assembly is configured to record image data; Item 14. The LF display system of item 1, wherein the LF display assembly records image data simultaneously with presenting holographic content. [Item 26] the LF display assembly is configured to record holographic content; Item 14. The LF display system of item 1, wherein the LF display assembly simultaneously presents and records holographic content. [Item 27] Item 1. The LF display system of item 1, further comprising a tracking system configured to obtain information regarding one or more viewers in the audience viewing the holographic content. [Item 28] Item 28. The LF display system of item 27, wherein the LF display assembly presents holographic content to the spectator based on information obtained by the tracking system. [Item 29] The information obtained by the tracking system is: a response by the one or more viewers in the audience to the presented holographic content; and and wherein the one or more viewer characteristics of the audience are included. [Item 30] 28. The LF display system of claim 27, wherein the information about the viewer includes any of the viewer's position, the viewer's movements, the viewer's gestures, the viewer's facial expressions, the viewer's age, the viewer's gender, and clothing worn by the viewer. [Item 31] The method further comprises: identifying viewers in the audience who are viewing the holographic content; 2. The LF display system of claim 1, configured to generate a viewer profile for each of the identified viewers. [Item 32] Item 32. The LF display system of item 31, wherein the LF display assembly presents holographic content to the audience based on identified viewers in the audience. [Item 33] Item 32. The LF display system of item 31, wherein the LF display system presents holographic content to the audience based on the identified viewer profile. [Item 34] Item 32. The LF display system of item 31, wherein the viewer profiling system is configured to identify a viewer's response to the holographic content or characteristics of a viewer viewing the holographic content and include the identified response or characteristics in a viewer profile. [Item 35] The LF Viewer Profiling System identifies characteristics of the viewer, the characteristics comprising: The position of the viewer; and A movement of the viewer; and a gesture by the viewer; and A facial expression of the viewer; and The gender of the viewer; and The age of the viewer; and and the viewer's clothing. [Item 36] 27. The LF display system of claim 26, wherein the viewer profiling system accesses social media accounts of the one or more identified viewers to generate a viewer profile. [Item 37] 2. The LF display system of item 1, wherein the venue is a theater. [Item 38] Item 2. The LF display system of item 1, wherein the venue includes a plurality of viewing locations generally surrounding a display surface of the LF display system that is substantially horizontal. [Item 39] Item 39. The LF display system of item 38, wherein the display surface is at least a portion of the venue floor. [Item 40] Item 39. The LF display system of item 38, wherein the display surface is at least a portion of a stage within the venue. [Item 41] Item 39. The LF display system of item 38, wherein the display surface is at least a portion of a raised viewing platform within the venue. [Item 42] 2. The LF display system of claim 1, wherein the venue includes a plurality of viewing locations arranged generally in front of the LF display assembly. [Item 43] 2. The LF display system of claim 1, wherein the venue includes multiple viewing locations positioned to have unobstructed line of sight to the presented holographic content. [Item 44] 2. The LF display system of claim 1, wherein the venue has a slope and one or more viewing locations are positioned along the slope. [Item 45] The venue: Item 1. The LF display system of item 1, comprising one or more viewing stages, each configured to contain a portion of the audience. [Item 46] Item 46. The LF display system of item 45, wherein the LF display system presents the same decoded holographic content to each tier of the venue. [Item 47] Item 46. The LF display system of item 45, wherein the LF display system presents different decoded holographic content on one or more stages of the venue. [Item 48] Item 46. The LF display system of item 45, wherein the steps are generally circular and positioned such that they surround the LF display assembly. [Item 49] the venue comprises a first wall; 2. The LF display system of claim 1, wherein the LF display assembly is positioned against the first wall such that the holographic content is presented to the audience from the first wall. [Item 50] the venue comprises a second wall; 50. The LF display system of claim 49, wherein the LF display assembly is positioned on the first wall and the second wall such that the holographic content is presented to the audience from the first wall and the second wall. [Item 51] The venue has a stage; 2. The LF display system of claim 1, wherein the LF display assembly is positioned at the stage such that the holographic content is presented to the audience from the stage. [Item 52] the venue comprises a floor; 2. The LF display system of claim 1, wherein the LF display assembly is positioned on the floor such that the holographic content is presented to the audience from the floor. [Item 53] Item 10. The LF display system of item 1, wherein the LF display assembly is a substantially flat surface. [Item 54] Item 1. The LF display system of item 1, wherein the LF display system is curved. [Item 55] 1. A light field (LF) display system, comprising: a network interface configured to receive a live stream comprising real-time holographic content over a network; an LF processing engine configured to generate additional holographic content; and an LF display assembly configured to present the real-time holographic content and additional holographic content to an audience at a venue. [Item 56] The live stream, Concerts and Performances and With the show, Scene and Item 56. The LF display system of item 55, including real-time holographic content representing a simultaneous broadcast of one or more of the event. [Item 57] The live stream, Concerts and Performances and With the show, Scene and Item 56. The LF display system of item 55, including additional holographic content representing one or more of the following: [Item 58] The additional holographic content is Neural networks and A procedural generation algorithm; Machine learning algorithms and 58. The LF display system of item 57, generated by one or more of the following: [Item 59] The additional holographic content is A computer-generated rendering of an object, event, or scene; Item 56. The LF display system of item 55, comprising one or more of: previously recorded live-action holographic content; and [Item 60] The live stream, With singer, With the band, With the actor, With dancers, Comedians and Item 56. The LF display system of item 55, including holographic content representing one or more of: [Item 61] Item 56. The LF display system of item 55, wherein the additional holographic content extends the live stream with real-time holographic content. [Item 62] Item 56. The LF display system of item 55, wherein the additional holographic content replaces real-time holographic content included in the live stream. [Item 63] The additional holographic content is Set and Stage and With props, Clothes and Costumes and Item 56. The LF display system of item 55, representing one or more of: [Item 64] The additional holographic content is people, objects, events, or scenes from another time or place; Computer-generated renderings of people, objects, events, or scenes; 56. The LF display system of item 55, which displays one or more of: previously recorded holographic content; [Item 65] The additional holographic content is Advertisements and Item 56. The LF display system of item 55, which represents one or more of: [Item 66] The additional holographic content is Neural networks and A procedural generation algorithm; Machine learning algorithms and 56. The LF display system of item 55, generated by one or more of the following: [Item 67] 56. The LF display system of item 55, wherein viewers in the audience can interact with the additional holographic content. [Item 68] Item 56. The LF display system of item 55, wherein the viewer can interact with the additional holographic content but cannot interact with the real-time holographic content. [Item 69] Item 56. The LF display system of item 55, wherein a viewer can interact with the real-time holographic content. [Item 70] Item 56. The LF display system of item 55, wherein the processing engine is configured to generate additional holographic content based on the viewer's interaction with the holographic content presented by the LF display assembly. [Item 71] The viewers in the audience, Physical interaction and Auditory interaction and 56. The LF display system of claim 55, wherein the LF display system interacts with the presented holographic content through an interaction that includes any of the following: [Item 72] Item 56. The LF display system of item 55, further comprising a tracking system configured to obtain information regarding one or more viewers in the audience viewing the holographic content. [Item 73] 73. The LF display system of claim 72, wherein the additional holographic content generated by the processing engine is based on the information regarding the one or more viewers in the audience obtained by the tracking system. [Item 74] The information obtained by the tracking system is: a response by the one or more viewers in the audience to the presented holographic content; and and one or more viewer characteristics of the audience. [Item 75] Item 73. The LF display system of item 72, wherein the information about the viewer includes any of the viewer's position, the viewer's movements, the viewer's gestures, the viewer's facial expressions, the viewer's age, the viewer's gender, and clothing worn by the viewer. [Item 76] the tracking system is configured to determine aggregate information regarding the one or more viewers in the audience; Item 73. The LF display system of item 72, wherein the additional holographic content generated by the processing engine is based on the aggregate information. [Item 77] The method further comprises: identifying one or more viewers in the audience viewing the holographic content; Item 56. The LF display system of item 55, configured to generate a viewer profile for the one or more identified viewers. [Item 78] Item 78. The LF display system of item 77, wherein the additional holographic content generated by the processing engine is based on the one or more viewers identified by the viewer profiling system. [Item 79] Item 78. The LF display system of item 77, wherein the additional holographic content generated by the processing engine is based on the viewer profile generated by the viewer profiling system. [Item 80] Item 78. The LF display system of item 77, wherein the viewer profiling system is configured to identify a viewer's response to the holographic content or characteristics of a viewer viewing the holographic content and include the identified response or characteristics in a viewer profile. [Item 81] The viewer profiling system comprises: The position of the viewer; and A movement of the viewer; and a gesture by the viewer; and A facial expression of the viewer; and The gender of the viewer; and The age of the viewer; and Item 78. The method of item 77, further comprising identifying features including any of: the viewer's clothing; [Item 82] the viewer profiling system is configured to generate an aggregate profile for the one or more viewers in the audience; Item 78. The LF display system of item 77, wherein the additional holographic content generated by the processing engine is based on the aggregate profile. [Item 83] a LF processing engine configured to determine a hardware configuration of the LF display system; Item 56. The LF display system of item 55, wherein the LF display assembly presents the holographic content based on the determined hardware configuration of the LF display system. [Item 84] The hardware configuration includes: Resolution and the number of projected rays per degree; and Field of view and the deflection angle of the display surface; 84. The LF display system of item 83, comprising any one of: a. a dimension of the display surface; [Item 85] and a LF processing engine configured to determine a geometric orientation of the LF display system, Item 56. The LF display system of item 55, wherein the holographic content is presented to the audience based on the determined geometric orientation of the LF display system. [Item 86] The geometric orientation is a number of display panels in the LF display assembly; and a relative orientation of the display panels; and The height of the display panel; and a width of the display panel; and Item 86. The LF display system of item 85, comprising any one of the following: a layout of the display panel. [Item 87] and a LF processing engine configured to determine a configuration of the venue. Item 56. The LF display system of item 55, wherein the LF display assembly presents holographic content based on the determined configuration of the venue. [Item 88] The configuration of the venue, one or more holographic object volumes; One or more listening volumes; Item 88. The LF display system of item 87, comprising any one of: a. the location of the spectator relative to the LF display assembly. [Item 89] the LF display assembly is configured to present acoustic data; Item 56. The LF display system of item 55, wherein the holographic content includes the acoustic data such that the LF display system presents the holographic content and the acoustic data. [Item 90] Item 56. The LF display system of item 55, wherein the presented holographic content includes a first type of energy and a second type of energy. [Item 91] Item 91. The LF display system of item 90, wherein the first type of energy is electromagnetic energy and the second type of energy is ultrasonic energy. [Item 92] Item 92. The LF display system of item 91, wherein the first type of energy and the second type of energy are presented at the same location such that the LF display assembly presents a volumetric tactile surface. [Item 93] the LF display assembly is configured to record image data; Item 56. The LF display system of item 55, wherein the LF display assembly records image data simultaneously with presenting holographic content. [Item 94] the LF display assembly is configured to record holographic content; Item 56. The LF display system of item 55, wherein the LF display assembly simultaneously presents and records holographic content. [Item 95] Item 56. The LF display system of item 55, wherein the live stream containing real-time holographic content is received in a first format and the LF processing engine decodes the holographic content into a second format. [Item 96] Item 96. The LF display system of item 95, wherein the first format is a vectorized data format and the second format is a rasterized data format. [Item 97] Item 56. The LF display system of item 55, wherein the processing engine uses a proprietary codec to decode streaming holographic content encoded by the same codec. [Item 98] 1. A light field (LF) generation system, comprising: a light field recording assembly configured to record one or more types of energy representative of an event at the venue; a processing engine configured to convert the recorded energy into holographic content representative of the performance; and and a network interface configured to transmit the holographic content over a network to one or more LF display systems. [Item 99] The event is Concerts and With the show, Performances and Scene and Item 99. The LF generating system of item 98, wherein the LF generating system is one or more of: [Item 100] The event is With singer, With the band, Magician and With the actor, With dancers, Comedians and Item 99. The LF generation system of item 98, comprising one or more of: [Item 101] The venue: A performance hall and Event space and Theater and The concert hall and The studio and Item 99. The LF generating system according to item 98, wherein the LF generating system is any one of the following stages: [Item 102] Item 99. The LF generating system of item 98, wherein the network is a public network. [Item 103] Item 99. The LF generation system of item 98, wherein the network is a private network configured to transmit holographic content. [Item 104] Item 99. The LF generation system of item 98, wherein the processing engine uses a proprietary codec to encode the recorded energy as holographic content. [Item 105] Item 105. The LF generating system of item 104, wherein the holographic content is encoded in a vectorized format. [Item 106] Item 105. The LF performance network system of item 104, wherein the holographic content is encoded in a format decodable by an LF display system utilizing the proprietary codec. [Item 107] the light field recording assembly comprising: one or more light field modules including one or more energy sensors configured to record electromagnetic energy as light field content; Item 99. The LF generating system of item 98, wherein the holographic content includes the light field content. [Item 108] the light field recording assembly comprising: one or more acoustic recording devices configured to record acoustic energy as audio content; Item 99. The LF generation system of item 98, wherein the holographic content includes the audio content. [Item 109] the light field recording assembly comprising: one or more pressure sensors configured to record mechanical energy; Item 99. The LF generating system of item 98, wherein the holographic content includes instructions for generating mechanical energy to generate one or more tactile surfaces. [Item 110] the light field recording assembly comprising: Item 99. The LF generation system of item 98, comprising one or more recording modules positioned around the venue to capture energy from the performance at multiple viewpoints. [Item 111] Item 111. The LF generation system of item 110, wherein the processing engine converts the energy from multiple viewpoints into the holographic content. [Item 112] the light field recording assembly comprising: Item 99. The LF generation system of item 98, comprising one or more two-dimensional cameras, each two-dimensional camera recording energy from the performance as one or more two-dimensional images. [Item 113] Item 113. The LF generation system of item 112, wherein the processing engine converts the one or more two-dimensional images into holographic content. [Item 114] Item 114. The LF generation system of item 113, wherein the processing engine uses a machine learning algorithm to generate the holographic content. [Item 115] the light field recording assembly comprising: Item 99. The LF generation system of item 98, comprising one or more depth sensors for determining the depth of an object. [Item 116] the light field recording assembly comprising: Item 99. The LF generation system of item 98, comprising one or more plenoptic camera systems for recording light field data. [Item 117] Item 99. The LF generation system of item 98, wherein the LF generation system transmits the holographic content to an LF display system in near real time over the network. [Item 118] Item 99. The LF generation system of item 98, wherein the LF generation system stores the holographic content in a local storage device. [Item 119] Item 99. The LF generation system of item 98, wherein the LF generation system transmits the holographic content to a network system and stores it in a network storage system. [Item 120] Item 99. The LF generation system of item 98, wherein the LF generation system transmits the holographic content to an LF display system in response to receiving a transaction fee.

Claims

[Claim 1] 1. A light field (LF) display system, comprising: a network interface configured to receive the encoded holographic content over a network; a decoder configured to decode the encoded holographic content into a format presentable by an LF display assembly; and a light field display assembly configured to present the decoded holographic content to an audience at the venue.

Citation Information

Patent Citations

  • Image pickup device, image generation system, server, and electronic equipment

    JP2013090059A

  • Image processing apparatus, image processing method, and program

    JP2014135680A

  • Image processor, imaging apparatus, image processing method and program

    JP2014160912A

  • Method and apparatus for omnidirectional parallax light field display system

    JP2018518866A

  • Depth key compositing for video and holographic projection

    US20150070467A1