A light field display system for video communication

Abstract

A video communication system uses a light field display to present a holographic image of a remote scene (e.g., a hologram of a remote participant). The system may include a local light field display assembly and a controller. The controller generates display instructions based on visual data corresponding to the remote scene received from a remote image capture system (e.g., a remote light field display system). The display instructions cause the local light field display assembly to generate a holographic image of the remote scene.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042470, and PCT / US2017 / 042679, all of which are incorporated herein by reference in their entirety. Background Art

[0003] The present disclosure relates generally to communication systems and, more particularly, to video communications using light field display systems.

[0004] As the availability of communication bandwidth increases and the cost of digital cameras decreases, video has become an increasingly popular form of communication. However, several factors limit the quality of user experience with existing video communication solutions. Typically, the camera and screen are positioned close together, but at distinctly different locations. As a result, participants don't make eye contact and often fail to notice other gestures and expressions that humans use to add context to what is being said. Direct interaction is also difficult because the display presenting the video acts as a barrier between the participants. While some existing video communication technologies allow viewing documents through screen sharing, this results in the participant's video being removed or reduced in size. Summary of the Invention

[0005] A video communication system uses a light field display to present a holographic image of a remote scene, which may include one or more remote participants. In one embodiment, the system includes a local light field display assembly and a controller. The controller generates display instructions based on visual data corresponding to the remote scene received from a remote image capture system (e.g., another light field display assembly or a remote light field display system). The display instructions cause the local light field display assembly to generate a holographic image of the remote scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a diagram of a light field display module for presenting a holographic object according to one or more embodiments.

[0007] Figure 2A is a cross-section of a portion of a light field display module according to one or more embodiments.

[0008] Figure 2B is a cross-section of a portion of a light field display module according to one or more embodiments.

[0009] Figure 3A is a perspective view of a light field display module according to one or more embodiments.

[0010] Figure 3B is a cross-sectional view of a light field display module including interwoven energy relays according to one or more embodiments.

[0011] Figure 4A is a perspective view of a portion of a lightfield display system tiled in two dimensions to form a single-sided seamless surface environment in accordance with one or more embodiments.

[0012] Figure 4B is a perspective view of a portion of a lightfield display system in a multi-faceted seamless surface environment in accordance with one or more embodiments.

[0013] Figure 4C is a top view of a lightfield display system with a converging surface in a wing-like configuration according to one or more embodiments.

[0014] Figure 4D is a side view of a lightfield display system with a converging surface in a tilted configuration according to one or more embodiments.

[0015] Figure 4E is a top view of a lightfield display system having a converging surface on the front wall of a room according to one or more embodiments.

[0016] Figure 4F is a side view of a side view of a lightfield display system having a converging surface on the front wall of a room in accordance with one or more embodiments.

[0017] Figure 5 is a block diagram of a light field display system according to one or more embodiments.

[0018] Figure 6 is an illustration of an example light field display system for video conferencing in accordance with one or more embodiments.

[0019] Figure 7 is an illustration of an alternative configuration for a video conferencing space in accordance with one or more embodiments.

[0020] Figure 8A is an illustration of a light field display system presenting holographic content including holographic video chat participants in accordance with one or more embodiments.

[0021] Figure 8BAccording to one or more embodiments Figure 8A Illustration of a light field display system presenting holographic content including images of video chat participants.

[0022] Figure 9 is an illustration of an LF display system presenting holographic content including representations of participants in a group video chat in accordance with one or more embodiments. DETAILED DESCRIPTION

[0023] Light field (LF) display systems provide video communications, such as video conferencing, video chat, or pre-recorded video messages. Video communications include holographic content representing the remote scene, such as holographic images of remote participants, holographic images of props, and holographic whiteboards. Although the terms "video conferencing" and "video chat" are used for convenience to refer to more formal and less formal communications, respectively, any functionality described with reference to one term can also be provided with the other.

[0024] In various embodiments, a holographic image of a participant in one location is presented to a participant in a different location. The holographic image provides a three-dimensional (3D) representation of the participant that can be viewed without glasses, headphones, or other viewing devices. In the case of real-time communication, a bidirectional surface that both emits and absorbs light can be used. As a result, there can be a one-to-one correspondence between the gaze directions of participants in the two locations, allowing the participants to make eye contact as if they were in the same physical space. In addition, the presence of the 3D image can help participants notice gestures and / or expressions that convey information that may be missed in traditional video communication. Therefore, the use of an LF display system can give participants the impression that they are in the same space, even if they are thousands of miles apart.

[0025] In some embodiments, additional holographic images may be provided to improve and / or facilitate the communication experience. For example, holographic props such as product prototypes may be provided to further facilitate communication, as if the participants were all in the same space. Similarly, a holographic whiteboard may be provided on which participants can draw, and content may be synchronized between two or more locations. Because the whiteboard is holographic, it does not need to be limited to a two-dimensional (2D) surface. In one embodiment, participants may draw in 3D within a spatial area (e.g., a box) designated as a virtual whiteboard. As another example, participants may be able to change their own appearance and / or the appearance of other participants. This may include partial changes, such as changing clothing, hair color, lighting, etc., as well as complete changes, such as representing a participant with an avatar whose movements are mapped to the movements of the corresponding participant.

[0026] The holographic content presented by the LF display system can also be enhanced by other sensory stimuli (e.g., tactile and / or audio). For example, an ultrasound source in the LF display system can project ultrasonic pressure waves that produce a body tactile projection. The body tactile projection provides a tactile surface corresponding to some or all of the projected holographic objects. The holographic content can also include additional visual content (i.e., 2D or 3D visual content). In embodiments with multiple energy sources (i.e., holographic objects that provide the correct tactile sensation and sensory stimulation at any given point in time), coordination of the energy sources to achieve a cohesive experience is part of the LF system. For example, the LF system can include a controller that coordinates the presentation of the holographic content and the tactile surface.

[0027] In some embodiments, an LF display system may include elements that enable the system to project at least one type of energy and simultaneously sense at least one type of energy. The sensed energy can be used to record how a viewer responds to the holographic content. For example, an LF display system can project both holographic objects for viewing and ultrasound for tactile perception, and simultaneously record imaging information for tracking viewers (e.g., video conference participants) and other scene analysis. As an example, such a system can project a holographic product prototype that a participant can manipulate by touch (e.g., by reaching out and rotating the holographic prototype to view it from different angles), where this interaction with the holographic prototype is recorded by the LF display system. The LF display system component that performs ambient energy sensing can be integrated into the display surface, or it can be a dedicated sensor separate from the display surface.

[0028] Light Field Display System Overview

[0029] Figure 1 FIG100 is a diagram of a light field (LF) display module 110 presenting a holographic object 120 according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system uses one or more LF display modules to present holographic content including at least one holographic object. The LF display system can present the holographic content to one or more viewers. In some embodiments, the LF display system can also enhance the holographic content with other sensory content (e.g., touch, audio, smell, temperature, etc.). For example, as discussed below, the projection of focused ultrasound can generate a mid-air tactile sensation that can simulate the surface of some or all of the holographic objects. The LF display system includes one or more LF display modules 110 and is discussed in detail below with respect to FIG2-9.

[0030] The LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more viewers (e.g., viewer 140). The LF display module 110 includes an energy device layer (e.g., an emissive electron display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). Additionally, the LF display module 110 may include an energy relay layer for the purpose of combining multiple energy sources or detectors to form a single surface. At a high level, the energy device layer generates energy (e.g., holographic content) and then uses the energy waveguide layer to direct the energy to a region in space according to one or more four-dimensional (4D) light field functions. The LF display module 110 can also simultaneously project and / or sense one or more types of energy. For example, the LF display module 110 may be capable of projecting a holographic image and an ultrasonic tactile surface into a viewing volume while simultaneously detecting imaging data from the viewing volume. The operation of the LF display module 110 is discussed in more detail below with respect to Figures 2-3.

[0031] The LF display module 110 uses one or more 4D light field functions (e.g., derived from a plenoptic function) to generate a holographic object within the holographic object volume 160. The holographic object can be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Furthermore, the holographic object can be multi-colored (e.g., full-color). The holographic object can be projected in front of, behind, or separated by a screen plane. The holographic object 120 can be presented so that it can be perceived anywhere within the holographic object volume 160. The holographic object within the holographic object volume 160 can appear to float in space to the viewer 140.

[0032] The holographic object volume 160 represents the volume in which the viewer 140 can perceive the holographic object. The holographic object volume 160 can extend in front of the surface of the display area 150 (i.e., toward the viewer 140) so that the holographic object can appear in front of the plane of the display area 150. Additionally, the holographic object volume 160 can extend behind the surface of the display area 150 (i.e., away from the viewer 140), allowing the holographic object to appear as if it is behind the plane of the display area 150. In other words, the holographic object volume 160 can contain all light rays that originate from the display area 150 (e.g., are projected) and can converge to create the holographic object. Herein, the light rays can converge at a point in front of, at, or behind the display surface. More simply, the holographic object volume 160 encompasses all volumes from which the viewer can perceive the holographic object.

[0033] The viewing volume 130 is a volume of space from which a holographic object (e.g., holographic object 120) presented within the holographic object volume 160 by the LF display system is fully visible. The holographic object can be presented within the holographic object volume 160 and viewed within the viewing volume 130 so that the holographic object is indistinguishable from an actual object. The holographic object is formed by projecting the same light rays that would be generated from the surface of the object if it were physically present.

[0034] In some cases, the holographic object volume 160 and corresponding viewing volume 130 can be relatively small - such that they are designed for a single viewer. In other embodiments, as discussed in detail below with respect to, for example, Figures 4 and 6-9, the LF display modules can be enlarged and / or tiled to create larger holographic object volumes and corresponding viewing volumes that can accommodate a wide range of viewers (e.g., one to thousands). The LF display modules presented in this disclosure can be constructed so that the entire surface of the LF display contains holographic imaging optics, there are no dead or dead spaces, and no bezels are required. In these embodiments, the LF display modules can be tiled so that the imaging area is continuous across the seams between the LF display modules, and the join lines between the tiled modules are nearly undetectable using the visual acuity of the eye. Notably, in some configurations, although not described in detail herein, some portions of the display surface may not contain holographic imaging optics.

[0035] The flexible size and / or shape of the viewing volume 130 allows the viewer to be unconstrained within the viewing volume 130. For example, the viewer 140 can move to different locations within the viewing volume 130 and see different views of the holographic object 120 from corresponding viewing angles. Figure 1 , viewer 140 is located in a first position relative to holographic object 120 such that holographic object 120 appears to be a frontal view of a dolphin. Viewer 140 can move to other positions relative to holographic object 120 to see different views of the dolphin. For example, viewer 140 can move so that he / she sees the dolphin's left side, the dolphin's right side, and so on, much as if viewer 140 were viewing an actual dolphin and changing his / her relative position to the actual dolphin to see different views of the dolphin. In some embodiments, holographic object 120 is visible to all viewers within viewing volume 130, all of whom have an unobstructed view of holographic object 120 (i.e., not blocked by objects / people). These viewers can be unconstrained, allowing them to move around within the viewing volume to see different perspectives of holographic object 120. Thus, the LF display system can present a holographic object such that multiple unconstrained viewers can simultaneously see different perspectives of the holographic object in real-world space, as if the holographic object were physically present.

[0036] In contrast, conventional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) typically require each viewer to wear some kind of external device (e.g., 3-D glasses, a near-eye display, or a head-mounted display) in order to see the content. Additionally and / or alternatively, conventional displays may require the viewer to be constrained to a specific viewing position (e.g., in a chair with a fixed position relative to the display). For example, when viewing an object shown by a stereoscopic display, the viewer always focuses on the display surface, not on the object, and the display will always present only two views of the object, which will follow the viewer as they attempt to move around the perceived object, resulting in a distortion of the object's perception. However, with light field displays, viewers of holographic objects presented by an LF display system do not need to wear external devices or be constrained to a specific position in order to see the holographic object. The LF display system presents the holographic object in a manner visible to the viewer that is much the same as a physical object, without the need for special goggles, glasses, or head-mounted accessories. Furthermore, the viewer can view holographic content from any position within the viewing volume.

[0037] It is worth noting that there is a limitation on the size of the potential position receptors of the holographic object within the holographic object volume 160. In order to increase the size of the holographic object volume 160, the size of the display area 150 of the LF display module 110 can be increased and / or multiple LF display modules can be tiled together in a manner to form a seamless display surface. The effective display area of the seamless display surface is larger than the display area of each LF display module. Some embodiments related to tiling LF display modules are discussed below with respect to Figures 4 and 6-9. Figure 1 As shown, the display area 150 is rectangular, resulting in a pyramidal shape for the holographic object volume 160. In other embodiments, the display area may have some other shape (eg, hexagonal), which also affects the shape of the corresponding viewing volume.

[0038] Additionally, while the above discussion focuses on presenting the holographic object 120 within a portion of the holographic object volume 160 located between the LF display module 110 and the viewer 140, the LF display module 110 can also present content in the holographic object volume 160 behind the plane of the display area 150. For example, the LF display module 110 can make the display area 150 appear to be the surface of the ocean with the holographic object 120 emerging from it. The displayed content can then enable the viewer 140 to peer through the displayed surface to see underwater marine life. Furthermore, the LF display system can generate content that seamlessly moves around the holographic object volume 160, both behind and in front of the plane of the display area 150.

[0039] Figure 2AA cross-section 200 of a portion of an LF display module 210 is shown, according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a display area shaped differently from the display area 150. In the illustrated embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 may have components different from those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be distributed among components differently from that described herein.

[0040] The display system described herein presents an energy emission that replicates the energy that typically surrounds objects in the real world. Here, the emitted energy is directed from each coordinate on the display surface toward a specific direction. In other words, each coordinate on the display surface serves as a projection location for the emitted energy. The directed energy from the display surface causes many energy rays to converge, which can thereby create a holographic object. For example, for visible light, an LF display will project a large number of light rays from the projection location that can converge at any point in the holographic object volume, so that from the perspective of a viewer positioned farther away than the projected object, the light rays appear to come from the surface of a real object positioned in this area of space. In this way, the LF display generates reflected light rays that move away from the surface of the object from the viewer's perspective. The viewer's perspective can change on any given holographic object, and the viewer will see different views of the holographic object.

[0041] As described herein, the energy device layer 220 includes one or more electronic displays (e.g., emissive displays such as OLEDs) and one or more other energy projection and / or energy receiving devices. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include a plurality of pixels, each with independently controlled intensity. Many types of commercially available displays can be used in LF displays, such as emissive LEDs and OLED displays.

[0042] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic projection devices generate one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or some combination thereof. An array of ultrasonic pressure waves may be used for somatotactile sensations (e.g., at the surface of the holographic object 250). Audible pressure waves are used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming that the holographic object 250 is a dolphin, one or more acoustic projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the dolphin so that the viewer can touch the holographic object 250; and (2) provide audio content corresponding to the sounds made by the dolphin (e.g., clicks, chirps, or squeaks). The acoustic receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audible pressure waves within a local area of the LF display module 210.

[0043] Energy device layer 220 may also include one or more imaging sensors. The imaging sensors may be sensitive to light in the visible wavelength band and, in some cases, may be sensitive to light in other wavelength bands (e.g., infrared). The imaging sensors may be, for example, complementary metal oxide semiconductor (CMOS) arrays, charge coupled devices (CCDs), photodetector arrays, some other sensor that captures light, or some combination thereof. The LF display system may use the data captured by the one or more imaging sensors to locate and track the viewer's position.

[0044] In some configurations, the energy relay layer 230 relays energy (e.g., electromagnetic energy, mechanical pressure waves, etc.) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270, and relays energy between the two surfaces. The first surface 265 of each energy relay element can be coupled to one or more energy devices (e.g., an electronic display or an acoustic projection device). The energy relay elements can be constructed from, for example, glass, carbon, optical fiber, optical film, plastic, polymer, or some combination thereof. Additionally, in some embodiments, the energy relay elements can adjust the amplification (increase or decrease) of energy passing between the first surface 265 and the second surface 270. If the relays provide amplification, the relays can take the form of an array of bonded conical repeaters, referred to as cones, in which the area of one end of the cone can be substantially larger than the area of the opposite end. The large ends of the cones can be bonded together to form a seamless energy surface 275. One advantage is that space is created on the multiple small ends of each cone to accommodate multiple energy mechanical envelopes, such as the bezels of multiple displays. This additional room allows energy sources to be placed side by side on the small cone sides, with the active area of each energy source directing energy into the small cone surface and relaying it to the large seamless energy surface. Another advantage of using cone-shaped repeaters is that there is no non-imaging dead space on the combined seamless energy surface formed by the large ends of the cones. There are no borders or frames, and therefore the seamless energy surfaces can then be tiled together to form a larger surface with virtually no seams, depending on the visual acuity of the eye.

[0045] The second surfaces of adjacent energy-relay elements converge to form energy surface 275. In some embodiments, the spacing between edges of adjacent energy-relay elements is less than the minimum perceptible outline defined by the visual acuity of a human eye having, for example, 20 / 40 vision, such that energy surface 275 is effectively seamless from the perspective of viewer 280 within viewing volume 285.

[0046] In some embodiments, the second surfaces of adjacent energy relay elements are fused together using a process that may include one or more of pressure, heat, and a chemical reaction, such that no seams exist between them. In still other embodiments, an array of energy relay elements is formed by molding one side of a continuous block of relay material into an array of small tapered ends, each energy relay element being configured to transfer energy from an energy device attached to the small tapered end to a larger, undivided, single, combined surface.

[0047] In some embodiments, one or more of the energy relay elements exhibit energy localization, wherein the efficiency of energy transfer in a longitudinal direction substantially normal to surfaces 265 and 270 is much higher than the efficiency of transfer in a perpendicular transverse plane, and wherein the energy density is highly localized in this transverse plane as the energy wave propagates between surfaces 265 and 270. This localization of energy enables efficient relaying of energy distributions (e.g., images) between these surfaces without any significant loss of resolution.

[0048] The energy waveguide layer 240 uses waveguide elements in the energy waveguide layer 240 to guide energy from locations (e.g., coordinates) on the energy surface 275 into specific energy propagation paths from the display surface outward into the holographic viewing volume 285. The energy propagation paths are defined by at least two angular dimensions determined by the energy surface coordinate positions relative to the waveguides. The waveguides are associated with spatial 2D coordinates. Together, these four coordinates form a four-dimensional (4D) energy field. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 guide light from locations on the seamless energy surface 275 along different propagation directions through the viewing volume 285. In various examples, the light is guided according to a 4D light field function to form the holographic object 250 within the holographic object volume 255.

[0049] Each waveguide element in the energy waveguide layer 240 can be, for example, a lenslet comprised of one or more elements. In some configurations, the lenslet can be a positive lens. The positive lens can have a spherical, aspherical, or free-form surface profile. Furthermore, in some embodiments, some or all of the waveguide elements can include one or more additional optical components. The additional optical components can be, for example, energy-reducing structures such as baffles, positive lenses, negative lenses, spherical lenses, aspherical lenses, free-form lenses, liquid crystal lenses, liquid lenses, refractive elements, diffractive elements, or some combination thereof. In some embodiments, at least one of the lenslet and / or additional optical component can dynamically adjust its optical power. For example, the lenslet can be a liquid crystal lens or a liquid lens. Dynamic adjustment of the surface profile of the lenslet and / or at least one additional optical component can provide additional directional control of light projected from the waveguide element.

[0050] In the illustrated example, the LF display's holographic object volume 255 has boundaries formed by rays 256 and 257, but may be formed by other rays. The holographic object volume 255 is a continuous volume that extends both in front of (i.e., toward) and behind (i.e., away from) the energy waveguide layer 240. In the illustrated example, rays 256 and 257, perceptible to the user, are projected from opposite edges of the LF display module 210 at maximum angles relative to the normal to the display surface 277, but the rays may be other projected rays. The rays define the display's field of view and, therefore, the boundaries of the holographic viewing volume 285. In some cases, the rays define a holographic viewing volume within which the entire display can be viewed without vignetting (e.g., an ideal viewing volume). As the display's field of view increases, the convergence point of rays 256 and 257 moves closer to the display. Therefore, displays with larger fields of view allow the viewer 280 to see the entire display at closer viewing distances. Alternatively, rays 256 and 257 may form an ideal holographic object volume. A holographic object presented as an ideal holographic object volume may be viewed anywhere in viewing volume 285.

[0051] In some instances, a holographic object can be presented to only a portion of the viewing volume 285. In other words, the holographic object volume can be divided into any number of viewing sub-volumes (e.g., viewing sub-volume 290). In addition, a holographic object can be projected outside the holographic object volume 255. For example, the holographic object 251 is presented outside the holographic object volume 255. Because the holographic object 251 is presented outside the holographic object volume 255, it cannot be viewed from every position in the viewing volume 285. For example, the holographic object 251 is visible from a position in the viewing sub-volume 290, but not from the position of the viewer 280.

[0052] For example, go to Figure 2B To show the holographic content viewed from different viewing sub-bodies. Figure 2B A cross-section 200 of a portion of a LF display module is shown according to one or more embodiments. Figure 2B The cross section and Figure 2A The cross section is the same. However, Figure 2BDifferent sets of light rays are shown projected from the LF display module 210. Rays 256 and 257 still form the holographic object volume 255 and the viewing volume 285. However, as shown, the rays projected from the top of the LF display module 210 and the rays projected from the bottom of the LF display module 210 overlap to form various viewing sub-volumes (e.g., viewing sub-volumes 290A, 290B, 290C, and 290D) within the viewing volume 285. A viewer in the first viewing sub-volume (e.g., 290A) may be able to perceive the holographic content presented in the holographic object volume 255, while viewers in the other viewing sub-volumes (e.g., 290B, 290C, and 290D) may not be able to perceive the holographic content presented in the holographic object volume 255.

[0053] More simply, Figure 2A As shown, holographic object volume 255 is a volume in which holographic objects can be presented by the LF display system so that the holographic objects can be perceived by a viewer (e.g., viewer 280) in viewing volume 285. In this manner, viewing volume 285 is an example of an ideal viewing volume, and holographic object volume 255 is an example of an ideal object volume. However, in various configurations, a viewer can perceive holographic objects presented by the LF display system 200 in other example holographic object volumes. More generally, when viewing holographic content projected from an LF display module, "sight line guidelines" apply. The sight line guidelines assert that the line formed by the viewer's eye position and the holographic object being viewed must intersect the LF display surface.

[0054] Because the holographic content is presented according to the 4D light field function, when viewing the holographic content presented by the LF display module 210, each eye of the viewer 280 sees a different perspective of the holographic object 250. Furthermore, as the viewer 280 moves within the viewing volume 285, he / she will also see different perspectives of the holographic object 250, just like other viewers within the viewing volume 285. As one of ordinary skill in the art will appreciate, 4D light field functions are well known in the art and will not be described in further detail herein.

[0055] As described in greater detail herein, in some embodiments, the LF display can project more than one type of energy. For example, the LF display can project two types of energy, such as mechanical energy and electromagnetic energy. In this configuration, the energy relay layer 230 can include two separate energy relays that are interwoven at the energy surface 275 but separated so that energy is relayed to two different energy device layers 220. Here, one relay can be configured to transmit electromagnetic energy, while the other relay can be configured to transmit mechanical energy. In some embodiments, mechanical energy can be projected from locations between electromagnetic waveguide elements on the energy waveguide layer 240, thereby helping to form a structure that inhibits light from being transmitted from one electromagnetic waveguide element to another. In some embodiments, the energy waveguide layer 240 can also include waveguide elements that transmit focused ultrasound along specific propagation paths based on displayed instructions from the controller.

[0056] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy surface 275 is an emitting surface formed using one or more adjacent electronic displays within the energy device layer 220. And in some embodiments, without the energy relay layer, the spacing between the edges of the adjacent electronic displays is less than the minimum perceptible outline defined by the visual acuity of a human eye with 20 / 40 vision, so that the energy surface is effectively seamless from the perspective of the viewer 280 within the viewing volume 285.

[0057] LF display module

[0058] Figure 3A is a perspective view of an LF display module 300A according to one or more embodiments. The LF display module 300A may be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 300A may be some other LF display module. In the illustrated embodiment, the LF display module 300A includes an energy device layer 310, an energy relay layer 320, and an energy waveguide layer 330. The LF display module 300A is configured to present holographic content from a display surface 365, as described herein. For convenience, the display surface 365 is shown in dashed outline on the frame 390 of the LF display module 300A, but is more accurately the surface directly in front of the waveguide element defined by the inner edge of the frame 390. The display surface 365 includes multiple projection locations from which energy can be projected. Some embodiments of the LF display module 300A have components different from those described herein. For example, in some embodiments, the LF display module 300A does not include the energy relay layer 320. Similarly, functionality may be distributed among components in ways other than described here.

[0059] Energy device layer 310 is an embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three of which are visible in the figure). Energy devices 340 can all be of the same type (e.g., all electronic displays) or can include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0060] Energy relay layer 320 is an embodiment of energy relay layer 230. Energy relay layer 320 includes four energy relay devices 350 (three of which are visible in the figure). Energy relay devices 350 can all relay the same type of energy (e.g., light) or can relay one or more different types (e.g., light and sound). Each relay device 350 includes a first surface and a second surface, with the second surface of each relay device 350 arranged to form a single, seamless energy surface 360. In the illustrated embodiment, each relay device 350 is tapered, with the first surface having a smaller surface area than the second surface. This allows the mechanical envelope of energy device 340 to be accommodated at the small end of the cone. Because energy can be projected over the entire area, this also makes the seamless energy surface borderless. This means that this seamless energy surface can be tiled by placing multiple instances of LF display module 300A together without dead space or borders, making the entire combined surface seamless. In other embodiments, the first and second surfaces have the same surface area.

[0061] Energy waveguide layer 330 is an embodiment of energy waveguide layer 240. Energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIG. 2 , energy waveguide layer 330 is configured to guide energy from seamless energy surface 360 along a specific propagation path according to a 4D light field function to form a holographic object. Note that in the illustrated embodiment, energy waveguide layer 330 is bounded by frame 390. In other embodiments, frame 390 is absent and / or its thickness is reduced. Removing or reducing the thickness of frame 390 can facilitate tiling LF display module 300A with other LF display modules.

[0062] Note that in the illustrated embodiment, the seamless energy surface 360 and the energy waveguide layer 330 are flat. In alternative embodiments not shown, the seamless energy surface 360 and the energy waveguide layer 330 can be curved in one or more dimensions.

[0063] The LF display module 300A can be configured with additional energy sources residing on the surface of the seamless energy surface, allowing for the projection of energy fields in addition to light fields. In one embodiment, an acoustic energy field can be projected from electrostatic speakers (not shown) mounted at any number of locations on the seamless energy surface 360. Furthermore, the electrostatic speakers of the LF display module 300A are positioned within the light field display module 300A, allowing the dual energy surface to simultaneously project both an acoustic field and holographic content. For example, the electrostatic speakers can be formed with one or more diaphragm elements that transmit electromagnetic energy of a certain wavelength and driven by one or more conductive elements (e.g., a plane sandwiching the one or more diaphragm elements). The electrostatic speakers can be mounted on the seamless energy surface 360 such that the diaphragm elements cover some of the waveguide elements. The conductive electrodes of the speakers can be co-located with structures designed to suppress light transmission between electromagnetic waveguides and / or positioned between electromagnetic waveguide elements (e.g., frame 390). In various configurations, the speakers can project audible sounds and / or generate a variety of sources of focused ultrasound energy for the tactile surface.

[0064] In some configurations, the energy device 340 can sense energy. For example, the energy device can be a microphone, a light sensor, an acoustic transducer, etc. Thus, the energy relay device can also relay energy from the seamless energy surface 360 to the energy device layer 310. In other words, when the energy device and the energy relay device 340 are configured to simultaneously emit and sense energy (e.g., emit a light field and sense sound), the seamless energy surface 360 of the LF display module forms a bidirectional energy surface.

[0065] More broadly, the energy device 340 of the LF display module 340 can be an energy source or an energy sensor. The LF display module 300A can include various types of energy devices that act as energy sources and / or energy sensors to facilitate the projection of high-quality holographic content to the user. Other sources and / or sensors can include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, and the like. Many other sensors or sources are also possible. Furthermore, the LF display modules can be tiled so that the LF display modules can form an assembly that projects and senses multiple types of energy from a large, seamless aggregate energy surface.

[0066] In various embodiments of the LF display module 300A, the seamless energy surface 360 can have various surface portions, each configured to project and / or transmit a specific type of energy. For example, when the seamless energy surface is a dual-energy surface, the seamless energy surface 360 includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy can be positioned on the seamless energy surface 360 between electromagnetic waveguide elements and / or co-located with structures designed to inhibit light transmission between electromagnetic waveguide elements. In instances where the seamless energy surface is a dual-energy surface, the energy relay layer 320 can include two types of energy relay devices interwoven at the seamless energy surface 360. In various embodiments, the seamless energy surface 360 can be configured such that the portion of the surface underlying any particular waveguide element 370 is all energy sources, all energy sensors, or a mixture of energy sources and energy sensors.

[0067] Figure 3B FIG2 is a cross-sectional view of an LF display module 300B including interwoven energy relay devices, according to one or more embodiments. Energy relay device 350A transfers energy between an energy relay first surface 345A connected to energy device 340A and a seamless energy surface 360. Energy relay device 350B transfers energy between an energy relay first surface 345B connected to energy device 340B and a seamless energy surface 360. The two relay devices are interwoven at an interwoven energy relay device 352 connected to seamless energy surface 360. In this configuration, surface 360 contains interwoven energy locations for both energy devices 340A and 340B, which can be energy sources or energy sensors. Thus, LF display module 300B can be configured as a dual energy projection device for projecting more than one type of energy, or as a bidirectional energy device for simultaneously projecting one type of energy and sensing another type of energy. LF display module 300B can be LF display module 110 and / or LF display module 210. In other embodiments, LF display module 300B can be some other LF display module.

[0068] LF display module 300B includes Figure 3A For example, in the embodiment shown, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330, including at least the same components as those of the LF display module 300A. Figure 3A Additionally, the LF display module 300B can present and / or receive energy from the display surface 365. Figure 3AThe components of the LF display module 300B may be connected and / or oriented differently than the components of the LF display module 300A in FIG. Some embodiments of the LF display module 300B have components that differ from those described herein. Similarly, functionality may be allocated between components in a manner different from that described herein. Figure 3B A design of a single LF display module 300B is shown that can be tiled to create a dual-energy projection surface or a bidirectional energy surface with a larger area.

[0069] In one embodiment, the LF display module 300B is a LF display module of a bidirectional LF display system. The bidirectional LF display system can simultaneously project energy from the display surface 365 and sense energy. The seamless energy surface 360 contains both energy projection locations and energy sensing locations that are closely interwoven on the seamless energy surface 360. Figure 3B In the example of Figure 3A For convenience, the energy relay layer of the LF display module 300B will be referred to herein as an "interwoven energy relay layer."

[0070] The interwoven energy relay layer 320 includes two legs: a first energy relay device 350A and a second energy relay device 350B. Figure 3B In the example shown, each of the legs is illustrated as a light shaded area. Each of the legs can be made of a flexible relay material and formed to a sufficient length for use with energy devices of various sizes and shapes. In some areas of the interwoven energy relay layer, two legs are tightly interwoven as they approach seamless energy surface 360. In the illustrated example, interwoven energy relay device 352 is illustrated as a dark shaded area.

[0071] When interwoven at seamless energy surface 360, the energy relay devices are configured to relay energy to / from different energy devices. The energy devices are located at energy device layer 310. As shown, energy device 340A is connected to energy relay device 350A, and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device can be an energy source or an energy sensor.

[0072] The energy waveguide layer 330 includes waveguide elements 370 to guide energy waves from the seamless energy surface 360 along a projected path toward a series of convergence points. In this example, a holographic object 380 is formed at the series of convergence points. Notably, as shown, the convergence of energy at the holographic object 380 occurs on the viewer side (i.e., the front side) of the display surface 365. However, in other examples, the convergence of energy can be anywhere in the holographic object volume, extending both in front of the display surface 365 and behind the display surface 365. The waveguide elements 370 can also guide the incoming energy to an energy device (e.g., an energy sensor), as described below.

[0073] In one example embodiment of the LF display module 300B, an emissive display serves as an energy source (e.g., energy device 340A), and an imaging sensor serves as an energy sensor (e.g., energy device 340B). In this manner, the LF display module 300B can simultaneously project holographic content and detect light from a volume in front of the display surface 365. In this manner, this embodiment of the LF display module 300B serves as both an LF display and an LF sensor.

[0074] In one embodiment, the LF display module 300B is configured to simultaneously project a light field from a projection location on the display surface to the front of the display surface and capture a light field from the front of the display surface at the projection location. In this embodiment, an energy relay device 350A connects a first set of locations on the seamless energy surface 360 that are positioned below the waveguide element 370 to an energy device 340A. In one example, the energy device 340A is an emissive display having an array of source pixels. The energy relay device 340B connects a second set of locations on the seamless energy surface 360 that are positioned below the waveguide element 370 to the energy device 340B. In one example, the energy device 340B is an imaging sensor having an array of sensor pixels. The LF display module 300B can be configured so that the locations on the seamless energy surface 365 that are positioned below a particular waveguide element 370 are all emissive display locations, all imaging sensor locations, or some combination thereof. In other embodiments, the bidirectional energy surface can project and receive various other forms of energy.

[0075] In another example embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, in one embodiment, the energy device 340A is an emissive display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. In this manner, both light and sound can be projected from various locations on the seamless energy surface 360. In this configuration, an energy relay 350A connects the energy device 340A to the seamless energy surface 360 and relays electromagnetic energy. The energy relay is configured to have properties that enable it to efficiently transmit electromagnetic energy (e.g., a varying refractive index). The energy relay 350B connects the energy device 340B to the seamless energy surface 360 and relays mechanical energy. The energy relay 350B is configured to have properties that facilitate efficient transmission of ultrasonic energy (e.g., a distribution of materials with different acoustic impedances). In some embodiments, mechanical energy can be projected from locations between the waveguide elements 370 on the energy waveguide layer 330. The locations where the mechanical energy is projected can form a structure that inhibits the transmission of light from one electromagnetic waveguide element to another. In one example, an array of spatially separated locations where ultrasonic mechanical energy is projected can be configured to form a three-dimensional tactile shape and surface in mid-air. The surface can coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delay and amplitude variations across the array can help form the tactile shape.

[0076] In various embodiments, the LF display module 300B with interwoven energy relay devices may include multiple energy device layers, where each energy device layer includes a specific type of energy device. In these examples, the energy relay layer is configured to relay the appropriate type of energy between the seamless energy surface 360 and the energy device layer 310.

[0077] Tiled LF display modules

[0078] Figure 4A FIG4 is a perspective view of a portion of an LF display system 400 tiled in two dimensions to form a single-sided seamless surface environment, according to one or more embodiments. LF display system 400 includes a plurality of LF display modules tiled to form an array 410. More specifically, each of the small squares in array 410 represents a tiled LF display module 412. LF display module 412 can be the same as LF display module 300A or 300B. Array 410 can cover some or all of a surface (e.g., a wall) of a room, for example. LF arrays can also cover other surfaces, such as tabletops, cubicle dividers, and the like.

[0079] Array 410 can project one or more holographic objects. For example, in the illustrated embodiment, array 410 projects holographic object 420 and holographic object 422. Tiling of LF display modules 412 allows for a larger viewing volume and allows objects to be projected at a greater distance from array 410. For example, in the illustrated embodiment, the viewing volume is approximately the entire area in front of and behind array 410, rather than a localized volume in front of (and behind) LF display modules 412.

[0080] In some embodiments, the LF display system 400 presents the holographic object 420 to a viewer 430 and a viewer 434. The viewer 430 and the viewer 434 receive different perspectives of the holographic object 420. For example, the viewer 430 is presented with a direct view of the holographic object 420, while the viewer 434 is presented with a more oblique view of the holographic object 420. As the viewer 430 and / or the viewer 434 move, they are presented with different perspectives of the holographic object 420. This allows the viewers to visually interact with the holographic object by moving relative to the holographic object. For example, as the viewer 430 walks around the holographic object 420, the viewer 430 sees different sides of the holographic object 420, as long as the holographic object 420 remains within the holographic object volume of the array 410. Thus, the viewer 430 and the viewer 434 can simultaneously see the holographic object 420 in real-world space, as if the holographic object were actually present. Additionally, viewers 430 and 434 do not need to wear external devices to view holographic object 420, as holographic object 420 is visible to the viewers in much the same way a physical object would be visible. Furthermore, here, holographic object 422 is displayed behind the array, as the viewing volume of the array extends behind the surface of the array. In this way, holographic object 422 can be presented to viewers 430 and / or 434.

[0081] In some embodiments, LF display system 400 may include a tracking system that tracks the location of viewer 430 and viewer 434. In some embodiments, the tracked location is the location of the viewer. In other embodiments, the tracked location is the location of the viewer's eyes. Eye location tracking is different from gaze tracking, which tracks where the eyes are looking (e.g., using orientation to determine gaze location). The eyes of viewer 430 and the eyes of viewer 434 are located in different locations.

[0082] In various configurations, the LF display system 400 may include one or more tracking systems. For example, in the illustrated Figure 4A In the embodiment of FIG, the LF display system includes a tracking system 440 external to the array 410. Here, the tracking system can be a camera system coupled to the array 410. Figure 54. In other example embodiments, the tracking system may be incorporated into the array 410 as described herein. For example, an energy device (e.g., energy device 340) of one or more LF display modules 412 containing a bidirectional energy surface included in the array 410 may be configured to capture an image of a viewer in front of the array 410. In any case, the one or more tracking systems of the LF display system 400 determine tracking information about a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by the array 410.

[0083] The tracking information describes the position of the viewer or a portion of the viewer (e.g., one or both eyes of the viewer, or a limb of the viewer) in space (e.g., relative to the tracking system). The tracking system can use any number of depth determination techniques to determine the tracking information. Depth determination techniques can include, for example, structured light, time of flight, stereoscopic imaging, some other depth determination technique, or some combination thereof. The tracking system can include various systems configured to determine the tracking information. For example, the tracking system can include one or more infrared sources (e.g., structured light sources), one or more imaging sensors (e.g., red-blue-green-infrared cameras) that can capture infrared images, and a processor that executes a tracking algorithm. The tracking system can use depth estimation techniques to determine the position of the viewer. In some embodiments, as described herein, the LF display system 400 generates a holographic object based on the tracked position, motion, or gesture of the viewer 430 and / or viewer 434. For example, the LF display system 400 can generate a holographic object in response to the viewer entering within a threshold distance and / or a specific position of the array 410.

[0084] The LF display system 400 can present one or more holographic objects customized for each viewer based in part on the tracking information. For example, the holographic object 420 can be presented to the viewer 430 instead of the holographic object 422. Similarly, the holographic object 422 can be presented to the viewer 434 instead of the holographic object 420. For example, the LF display system 400 tracks the position of each of the viewers 430 and 434. The LF display system 400 determines the viewing angle of the holographic object that should be visible to the viewer based on the position of the viewer relative to where the holographic object is to be presented. The LF display system 400 selectively projects light from specific pixels corresponding to the determined viewing angle. Therefore, the viewer 434 and the viewer 430 can have potentially completely different experiences at the same time. In other words, the LF display system 400 can present holographic content to a viewing sub-body of the viewing body (i.e., similar to Figure 2B4. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 401. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402. FIGURE 4B shows a schematic diagram of a LF display system 400 and a display device 402.

[0085] In some embodiments, the LF display system 400 may include one or more sensory feedback systems. The sensory feedback system provides other sensory stimulation (e.g., tactile, audio, or scent) that enhances the holographic objects 420 and 422. For example, in the illustrated Figure 4A In an embodiment of the present invention, the LF display system 400 includes a sensory feedback system 442 external to the array 410. In one example, the sensory feedback system 442 can be an electrostatic speaker coupled to the array 410. Figure 5 The external sensory feedback system is described in more detail. In other example embodiments, the sensory feedback system can be incorporated into the array 410 as described herein. For example, the energy device (e.g., Figure 3B The energy device 340A in the array 410 can be configured to project ultrasound energy to a viewer in front of the array and / or receive imaging information from a viewer in front of the array. In any case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content (e.g., holographic object 420 and / or holographic object 422) presented by the array 410.

[0086] The LF display system 400 may include a sensory feedback system 442 comprising one or more acoustic projection devices external to the array. Alternatively or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410, as described herein. The acoustic projection device may consist of an array of ultrasound sources configured to project a body tactile surface. In some embodiments, for one or more surfaces of a holographic object, if a portion of the viewer is within a threshold distance of the one or more surfaces, the tactile surface may coincide with the holographic object (e.g., at the surface of the holographic object 420). Body tactile sensations may allow the user to touch and feel the surface of the holographic object. Multiple acoustic projection devices may also project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Thus, the ultrasonic pressure waves and / or audible pressure waves may serve to supplement the holographic object.

[0087] In various embodiments, the LF display system 400 may provide other sensory stimuli based in part on the tracked position of the viewer. For example, Figure 4A The holographic object 422 shown in the figure is a lion, and the LF display system 400 can make the holographic object 422 roar both visually (i.e., the holographic object 422 appears to be roaring) and auditorily (i.e., one or more acoustic projection devices project pressure waves), so that the viewer 430 perceives it as the roar of a lion emitted by the holographic object 422.

[0088] Note that in the illustrated configuration, the holographic viewing volume can be limited in a manner similar to the viewing volume 285 of the LF display system 200 in FIG2 . This can limit the perceived immersion that a viewer would experience with a single wall display unit. One approach to addressing this issue is to use multiple LF display modules tiled along multiple sides, as described below with respect to Figures 4B-4F described.

[0089] Figure 4B is a perspective view of a portion of an LF display system 402 in a multi-sided seamless surface environment according to one or more embodiments. The LF display system 402 is substantially similar to the LF display system 400, except that a plurality of LF display modules are tiled to create the multi-sided seamless surface environment. More specifically, the LF display modules are tiled to form an array as a six-sided aggregate seamless surface environment. Figure 4B In one embodiment, the multiple LF display modules cover all walls, ceiling, and floor of the room. In other embodiments, the multiple LF display modules may cover some, but not all, of the walls, floor, ceiling, or some combination thereof. In other embodiments, the multiple LF display modules are tiled to form some other aggregate seamless surface. For example, the walls may be curved so as to form a cylindrical aggregate energy environment. Additionally, as described below with respect to Figure 6-9 As described, in some embodiments, the LF display modules can be tiled to form a surface (eg, a wall, etc.) in a conference room or office.

[0090] LF display system 402 can project one or more holographic objects. For example, in the illustrated embodiment, LF display system 402 projects holographic object 420 into an area surrounded by a six-sided polymeric seamless surface environment. In this example, the viewing volume of the LF display system is also contained within the six-sided polymeric seamless surface environment. Note that in the illustrated configuration, viewer 434 can be positioned between holographic object 420 and LF display module 414, which projects the energy (e.g., light and / or pressure waves) used to form holographic object 420. Therefore, the positioning of viewer 434 can prevent viewer 430 from perceiving holographic object 420 formed by the energy from LF display module 414. However, in the illustrated configuration, there is at least one other LF display module, such as LF display module 416, which is unobstructed (e.g., by viewer 434) and can project energy to form holographic object 420 and be observed by viewer 430. In this way, being occluded by the viewer in space may cause some portions of the holographic projection to disappear, but this effect is much smaller than if only one side of the volume were filled with a holographic display panel. Holographic object 422 is displayed as "outside" of the walls of the six-sided, aggregated, seamless surface environment because the holographic object volume extends behind the aggregated surface. Thus, viewer 430 and / or viewer 434 can perceive holographic object 422 as "outside" of the enclosed, six-sided environment in which it can move.

[0091] As mentioned above Figure 4A As described, in some embodiments, the LF display system 402 actively tracks the viewer's position and can dynamically instruct different LF display modules to present holographic content based on the tracked position. Thus, the multi-faceted configuration can provide a more robust environment (e.g., relative to Figure 4A ) to provide holographic objects where an unconstrained viewer can move freely throughout the area surrounded by a multi-faceted seamless surface environment.

[0092] It is worth noting that various LF display systems can have different configurations. Further, each configuration can have a specific orientation of surfaces that aggregate to form a seamless display surface ("aggregate surface"). In other words, the LF display modules of the LF display system can be tiled to form various aggregate surfaces. For example, in Figure 4B In the example embodiment, LF display system 402 includes LF display modules that are tiled to form a six-sided converging surface that approximates a wall of a room. In some other examples, the converging surface may appear on only a portion of the surface (e.g., half of the wall) rather than the entire surface (e.g., the entire wall). Some examples are described herein.

[0093] In some configurations, the converging surface of the LF display system can include a converging surface configured to project energy toward a local viewing volume. Projecting energy toward a local viewing volume allows for a higher quality viewing experience by, for example, increasing the density of projected energy in a particular viewing volume, increasing the viewer's FOV in that viewing volume, and bringing the viewing volume closer to the display surface.

[0094] For example, Figure 4C A top view of an LF display system 450A having a converging surface in a "wing-like" configuration is shown. In this example, the LF display system 450A is positioned in a room having a front wall 452, a rear wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown). The first side wall 456, the second side wall 458, the rear wall 454, the floor, and the ceiling are all orthogonal. The LF display system 450A includes LF display modules tiled to form a converging surface 460 covering the front wall. The front wall 452, and therefore the converging surface 460, comprises three sections: (i) a first section 462 that is substantially parallel to the rear wall 454 (i.e., the center surface), (ii) a second section 464 that connects the first section 462 to the first side wall 456 and is positioned at an angle to project energy toward the center of the room (i.e., the first side surface), and (iii) a third section 466 that connects the first section 462 to the second side wall 458 and is positioned at an angle to project energy toward the center of the room (i.e., the second side surface). The first section is a vertical plane in the room and has a horizontal axis and a vertical axis. The second and third sections are angled along the horizontal axis toward the center of the room.

[0095] In this example, viewing volume 468A of LF display system 450A is located in the center of the room and is partially surrounded by three portions of converging surface 460. The converging surface that at least partially surrounds the viewer ("surrounding surface") increases the viewer's immersive experience.

[0096] For illustration, consider, for example, a polymeric surface having only a central surface. Figure 2A As described above, rays projected from either end of the display surface create an ideal holographic volume and an ideal viewing volume. Now consider, for example, if the central surface includes two side faces angled toward the viewer. In this case, rays 256 and 257 would be projected at a greater angle from the normal to the central surface. Consequently, the viewing volume's field of view would increase. Similarly, the holographic viewing volume would be closer to the display surface. Additionally, because the two second and third portions are tilted closer to the viewing volume, the holographic object projected at a fixed distance from the display surface is closer to the viewing volume.

[0097] For simplicity, a display surface with only a central surface has a planar field of view, a planar threshold separation between the (central) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer increases the field of view relative to the planar field of view, reduces the separation between the display surface and the viewing volume relative to the planar separation, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further increases the field of view, reduces the separation, and increases the proximity. In other words, the angled placement of the side surfaces increases the immersive experience of the viewer. Additionally, the deflection optics can be used to optimize the size and positioning of the viewing volume (e.g., dimensions and FOV) of the viewing volume for LF display parameters.

[0098] Return to Figure 4D , in a similar instance, Figure 4D A side view of an LF display system 450B having a converging surface in a "tilted" configuration is shown. In this example, the LF display system 450B is positioned in a room having a front wall 452, a rear wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. The first side wall, the second side wall, the rear wall 454, the floor 474, and the ceiling 472 are all orthogonal. The LF display system 450B includes LF display modules tiled to form a converging surface 460 covering the front wall. The front wall 452, and therefore the converging surface 460, comprises three sections: (i) a first section 462 that is substantially parallel to the rear wall 454 (i.e., the center surface), (ii) a second section 464 that connects the first section 462 to the ceiling 472 and is angled to project energy toward the center of the room (i.e., the first side surface), and (iii) a third section 464 that connects the first section 462 to the floor 474 and is angled to project energy toward the center of the room (i.e., the second side surface). The first section is a vertical plane in the room and has a horizontal axis and a vertical axis. The second and third sections are angled along the vertical axis toward the center of the room.

[0099] In this example, the viewing volume 468B of the LF display system 450B is located in the center of the room and is partially surrounded by three portions of the converging surface 460. Figure 4CSimilar to the configuration shown, the two side portions (e.g., second portion 464 and third portion 466) are angled to surround the viewer and form a surrounding surface. From the perspective of any viewer in the holographic viewing volume 468B, the surrounding surface increases the viewing FOV. Additionally, the surrounding surface allows the viewing volume 468B to be closer to the surface of the display, making the projected objects appear closer. In other words, the angled placement of the side surfaces increases the field of view, reduces the separation, and increases the proximity of the converging surfaces, thereby increasing the viewer's immersive experience. Further, as will be discussed below, deflection optics can be used to optimize the size and positioning of the viewing volume 468B.

[0100] The tilted configuration of the side portions of the converging surface 460 enables the holographic content to be presented closer to the viewing volume 468B than if the third portion 466 were not tilted. For example, the lower limbs (e.g., legs) of a character presented from an LF display system in a tilted configuration may appear closer and more realistic than if an LF display system with a flat front wall were used.

[0101] Additionally, the configuration of the LF display system and the environment in which it is located can inform the shape and position of the viewing body and viewing sub-bodies.

[0102] Figure 4E For example, a top view of an LF display system 450C is shown having a converging surface 460 on a front wall 452 of a room. In this example, an LF display system 450D is positioned in a room having a front wall 452, a back wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown).

[0103] LF display system 450C projects various rays from converging surface 460. Rays are projected from each location on the display surface into a range of angles centered on the viewing volume. Rays projected from the left side of converging surface 460 have a horizontal angular range 481, rays projected from the right side of the converging surface have a horizontal angular range 482, and rays projected from the center of converging surface 460 have a horizontal angular range 483. Between these points, projected rays can have angular ranges intermediate to the desired range. Having a gradient deflection angle in the projected rays passing through the display surface in this manner creates viewing volume 468C. Further, this configuration avoids wasting display resolution when projecting rays into sidewalls 456 and 458.

[0104] Figure 4FA side view of an LF display system 450D having a converging surface 460 on a front wall 452 of a room is shown. In this example, an LF display system 450E is positioned in a room having a front wall 452, a back wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. In this example, the floor is layered such that each layer rises in a stepwise manner moving from the front wall to the back wall. Here, each layer of the floor contains viewing sub-volumes (e.g., viewing sub-volumes 470A and 470B). The layered floor allows for non-overlapping viewing sub-volumes. In other words, each viewing sub-volume has a line of sight from the viewing sub-volume to the converging surface 460 that does not pass through another viewing sub-volume. In other words, this orientation creates a "stadium seating" effect, where the vertical offset between the layers allows each layer to "see" the viewing sub-volumes of the other layers. An LF display system containing non-overlapping viewing sub-volumes can provide a more enhanced viewing experience than an LF display system with viewing sub-volumes that do overlap. For example, in Figure 4F In the configuration shown, different holographic content may be projected to viewers in viewing sub-volumes 470A and 470B.

[0105] LF display system control

[0106] Figure 5 is a block diagram of an LF display system 500 according to one or more embodiments. The LF display system 500 includes an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display module 512 may include a source / sensor system 514 that includes an integrated energy source and / or energy sensor that projects and / or senses other types of energy. The controller 520 includes a data memory 522, a network interface 524, and a LF processing engine 530. The controller 520 may also include a tracking module 526 and a viewer profiling module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. Figure 1 Embodiments of the LF display system 500 are discussed in the context of the LF display system described in the foregoing. In other embodiments, the LF display system 500 includes additional or fewer modules than those described herein. Similarly, functionality may be allocated between modules and / or different entities in a manner different from that described herein. Applications of the LF display system 500 will also be discussed below with respect to FIG. Figure 6-9 Discuss in detail.

[0107] The LF display assembly 510 provides holographic content in a holographic object volume that can be visible to a viewer positioned within the viewing volume. The LF display assembly 510 can provide the holographic content by executing display instructions received from the controller 520. The holographic content can include one or more holographic objects projected in front of a converging surface of the LF display assembly 510, behind a converging surface of the LF display assembly 510, or some combination thereof. Generating display instructions using the controller 520 is described in more detail below.

[0108] The LF display assembly 510 uses one or more LF display modules (e.g., any of the LF display module 110, the LF display system 200, and the LF display module 300) included in the LF display assembly 510 to provide holographic content (e.g., images of participants, avatars of participants, holographic objects, and / or other sensory content). For convenience, the one or more LF display modules may be described herein as LF display modules 512. The LF display modules 512 may be tiled to form the LF display assembly 510. The LF display modules 512 may be configured to form various seamless surface environments (e.g., single-sided, multi-sided, cinema walls, curved surfaces, etc.). In other words, the tiled LF display modules form a cohesive surface. As previously described, the LF display module 512 includes an energy device layer (e.g., the energy device layer 220) and an energy waveguide layer (e.g., the energy waveguide layer 240) that presents the holographic content. The LF display module 512 may also include an energy relay layer (eg, the energy relay layer 230 ) that transfers energy between the energy device layer and the energy waveguide layer when presenting holographic content.

[0109] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing as previously described. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and the integrated energy sensing system of the LF display module 512 may be collectively described herein as the source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 such that the source / sensor system 514 and the LF display module 512 share the same seamless energy surface. In other words, the collective surface of the LF display assembly 510 includes the functionality of both the LF display module 512 and the source / sensor module 514. In other words, the LF assembly 510, including the LF display module 512 with the source / sensor system 514, can simultaneously project and / or sense energy while projecting a light field. For example, the LF display assembly 510 may include a LF display module 512 and a source / sensor system 514 configured as a dual energy surface or a bidirectional energy surface as previously described.

[0110] In some embodiments, the LF display system 500 uses a sensory feedback system 570 to enhance the generated holographic content with other sensory content (e.g., coordinated touch, audio, or scent, etc.). The sensory feedback system 570 can enhance the projection of the holographic content by executing display instructions received from the controller 520. Typically, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., the sensory feedback system 442) external to the LF display assembly 510. Some example sensory feedback devices may include a coordinated acoustic projection and reception device, an aroma projection device, a temperature adjustment device, a force actuation device, a pressure sensor, a transducer, etc. In some cases, the sensory feedback system 570 may have similar functionality to the lightfield display assembly 510, and vice versa. For example, both the sensory feedback system 570 and the lightfield display assembly 510 may be configured to generate an acoustic field. As another example, the sensory feedback system 570 may be configured to generate a tactile surface without the lightfield display assembly 510.

[0111] For illustration, in an example embodiment of the light field display system 500, the sensory feedback system 570 may include one or more acoustic projection devices. The one or more acoustic projection devices are configured to generate one or more pressure waves that complement the holographic content when executing display instructions received from the controller 520. The generated pressure waves may be, for example, audible (for sound), ultrasonic (for touch), or some combination thereof. Similarly, the sensory feedback system 570 may include an aroma projection device. The aroma projection device may be configured to provide a scent to some or all of the target areas in the target area when executing display instructions received from the controller. The aroma device may be connected to an air circulation system (e.g., a duct, fan, or vent) to coordinate air flow within the target area. Further, the sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature of some or all of the target areas in the target area when executing display instructions received from the controller 520.

[0112] In some embodiments, the sensory feedback system 570 is configured to receive input from a viewer of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the viewer. The sensory feedback devices may include devices such as acoustic receiving devices (e.g., microphones), pressure sensors, joysticks, motion detectors, transducers, etc. The sensory feedback system may transmit the detected input to the controller 520 to coordinate the generation of holographic content and / or sensory feedback.

[0113] To illustrate, in an example embodiment of lightfield display assembly 510, sensory feedback system 570 includes a microphone. The microphone is configured to record audio generated by one or more viewers (e.g., participants in a video conference). Sensory feedback system 570 provides the recorded audio as viewer input to controller 520. Controller 520 can use the viewer input to generate holographic content. For example, if a participant mentions a particular product, a holographic representation of the product can be generated to provide context for the participant's comment. Similarly, sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure the force applied to the pressure sensor by the viewer. Sensory feedback system 570 can provide the measured force as viewer input to controller 520.

[0114] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to determine the position, movement, and / or characteristics of a viewer in a target area. Typically, the tracking device is external to the LF display assembly 510. Some example tracking devices include a camera assembly ("camera"), a depth sensor, a structured light, a LIDAR system, a card scanning system, or any other tracking device that can track a viewer in a target area.

[0115] Tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. However, in some cases, when presenting holographic content, the target area is illuminated by natural and / or ambient light from LF display assembly 510. The energy sources project light in response to instructions received from controller 520. The light may be, for example, a structured light pattern, a light pulse (e.g., an IR flash), or some combination thereof. The tracking system may project light in the visible band (approximately 380 nm to 750 nm), the infrared (IR) band (approximately 750 nm to 1700 nm), the ultraviolet band (10 nm to 380 nm), some other portion of the electromagnetic spectrum, or some combination thereof. The sources may include, for example, light-emitting diodes (LEDs), micro-LEDs, laser diodes, time-of-flight depth sensors, tunable lasers, and the like.

[0116] When executing instructions received from controller 520, tracking system 580 can adjust one or more emission parameters. Emission parameters are parameters that affect how light is projected from the source of tracking system 580. Emission parameters can include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, some other parameter that affects how light is projected from the source assembly, or some combination thereof. In one embodiment, the source projects light pulses in time-of-flight operation.

[0117] The camera of tracking system 580 captures images of light (e.g., a structured light pattern) reflected from the target area. The camera captures images when executing tracking instructions received from controller 520. As previously described, light can be projected by a source of tracking system 580. The camera can include one or more cameras. In other words, the camera can be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by tracking system 580, or some combination thereof. In one embodiment, tracking system 580 may include a light field camera external to LF display assembly 510. In other embodiments, the camera is included as part of LF display source / sensor module 514 included in LF display assembly 510. For example, as previously described, if the energy relay element of light field module 512 is a bidirectional energy layer that interweaves both an emissive display and an imaging sensor at energy device layer 220, LF display assembly 510 can be configured to simultaneously project a light field and record imaging information from a viewing area in front of the display. In one embodiment, the image captured from the bidirectional energy surface forms a light field camera. The camera provides the captured image to the controller 520 .

[0118] When executing tracking instructions received from controller 520, the camera of tracking system 580 can adjust one or more imaging parameters. Imaging parameters are parameters that affect how the camera captures an image. Imaging parameters can include, for example, frame rate, aperture, gain, exposure length, frame timing, rolling shutter or global shutter capture mode, some other parameter that affects how the camera captures an image, or some combination thereof.

[0119] Controller 520 controls LF display assembly 510 and any other components of LF display system 500. Controller 520 includes data storage 522, network interface 524, tracking module 526, viewer profiling module 528, and light field processing engine 530. In other embodiments, controller 520 includes additional or fewer modules than those described herein. Similarly, functionality can be allocated between modules and / or different entities in a manner different from that described herein. For example, tracking module 526 can be part of LF display assembly 510 or tracking system 580.

[0120] The data store 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by the camera, one or more viewer profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 (including the resolution and orientation of the LF module 512), the desired viewing volume geometry, content for graphic creation including 3D models, scenes and environments, materials and textures, and other information that may be used by the LF display system 500, or some combination thereof. The data store 522 is a memory such as read-only memory (ROM), dynamic random access memory (DRAM), static random access memory (SRAM), or some combination thereof.

[0121] The network interface 524 allows the light field display system to communicate with other systems or environments over a network. In one example, the LF display system 500 receives holographic content from a remote light field display system via the network interface 524. In another example, the LF display system 500 uses the network interface 524 to transmit holographic content to a remote data store.

[0122] The tracking module 526 tracks the viewer viewing the content presented by the LF display system 500. To this end, the tracking module 526 generates tracking instructions that control the operation of the tracking system 580 or the source and / or camera, and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking input to the tracking module 526.

[0123] Tracking module 526 can determine the position of one or more viewers within the target area (e.g., sitting in a specific chair in a conference room, walking around the conference room, etc.). The determined position can be relative to a reference point (e.g., a display surface, a conference table), for example. In other embodiments, the determined position can be within a virtual model of the target area. The tracked position can be, for example, the tracked position of the viewer and / or the tracked position of a portion of the viewer (e.g., eye position, hand position, etc.). Tracking module 526 uses one or more captured images from cameras of tracking system 580 to determine the position. The cameras of tracking system 580 can be distributed around LF display system 500 and can capture stereo images, allowing tracking module 526 to passively track the viewer. In other embodiments, tracking module 526 actively tracks the viewer. In other words, tracking system 580 illuminates a portion of the target area to image the target area, and tracking module 526 uses time-of-flight and / or structured light depth determination techniques to determine the position. Tracking module 526 uses the determined position to generate tracking information.

[0124] The tracking module 526 may also receive as input tracking information from a viewer of the LF display system 500. The tracking information may include body movements corresponding to various input options provided to the viewer by the LF display system 500. For example, the tracking module 526 may track the body movements of the viewer and assign any of the various movements as input to the LF processing engine 530. The tracking module 526 may provide the tracking information to the data store 522, the LF processing engine 530, the viewer profiling module 528, any other component of the LF display system 500, or some combination thereof.

[0125] To provide context for the tracking module 526, consider an example embodiment of an LF display system 500 that provides video conferencing for a meeting. In response to a participant voting on a proposal, one or more participants raise their hands. The tracking system 580 can record the movement of the participant's hand and transmit the recording to the tracking module 526. The tracking module 526 tracks the movement of the participant's hand in the recording and sends the input to the LF processing engine 530. As described below, the viewer profile analysis module 528 determines whether the information in the image indicates that the movement of the participant's hand is associated with a vote in favor of the proposal. The LF processing engine 530 can generate appropriate holographic content indicating the result of the vote. For example, the LF processing engine 530 can project the number of votes in the scene.

[0126] The LF display system 500 includes a viewer profiling module 528 configured to identify and profile viewers. The viewer profiling module 528 generates a profile of a viewer (or multiple viewers) viewing holographic content displayed by the LF display system 500. The viewer profiling module 528 generates a viewer profile based in part on viewer input and monitored viewer behavior, actions, and reactions. The viewer profiling module 528 can access information obtained from the tracking system 580 (e.g., recorded images, video, sound, etc.) and process the information to determine various information. In various embodiments, the viewer profiling module 528 can use any number of machine vision or machine hearing algorithms to determine viewer behavior, actions, and reactions. The monitored viewer behavior can include, for example, a viewer's smile, hand raising, cheering, clapping, laughing, and / or other gesture changes or movements.

[0127] More generally, a viewer profile may include any information received and / or determined about a viewer viewing holographic content from the LF display system. For example, each viewer profile may record the viewer's actions or responses to content displayed by the LF display system 500. Some example information that may be included in a viewer profile is provided below.

[0128] In some embodiments, the viewer profile may indicate the role of the viewer associated with the LF display system 500 in relation to an organization or group. For example, in a video conferencing system operated by an enterprise, the viewer profile may include, for example, the viewer's job title, responsibilities, authorization to view confidential information, etc. The viewer profile may additionally or alternatively indicate more general viewer characteristics, such as age, gender, race, clothing, location, etc.

[0129] In some embodiments, the viewer profile may indicate the viewer's preferences regarding the presentation of holographic content. For example, the viewer profile may indicate that the holographic object volume displays holographic content (e.g., to the viewer's right) and / or indicate that the holographic object volume does not display holographic content (e.g., to the viewer's left). The viewer profile may also indicate that the viewer prefers a haptic interface to be presented near them, or prefers to avoid the haptic interface.

[0130] In some embodiments, a viewer profile can also describe the characteristics and preferences of a group of viewers rather than a specific viewer. For example, the viewer profiling module 528 can generate a viewer profile for a specific combination of viewers in a video conference. In one example, the viewer profiling module 528 creates a profile for a pair of viewers that indicates that the pair prefers a specific configuration of holographic viewing areas, tactile interfaces, background images, and / or avatars when participating in a video conference session using the LF display system 500. Any of the previously described information and characteristics can be applied to a group of viewers.

[0131] The viewer profiling module 528 may also access profiles associated with a particular viewer (or multiple particular viewers) from one or more third-party systems to build a viewer profile. For example, a viewer may connect one or more social media accounts to a viewer profile maintained by the viewer profiling module 528. The viewer profiling module 528 may access information from one or more of the social media accounts to build (or augment) the viewer profile.

[0132] In some embodiments, the data store 522 includes a viewer profile store that stores viewer profiles generated, updated, and / or maintained by the viewer profiling module 528. The viewer profiles in the data store can be updated at any time by the viewer profiling module 528. For example, in one embodiment, when a particular viewer views holographic content provided by the LF display system 500, the viewer profile store receives and stores information about the particular viewer in its viewer profile. In this example, the viewer profiling module 528 includes a facial recognition algorithm that can recognize the viewer and positively identify the viewer when viewing the presented holographic content. To illustrate, when a viewer enters a target area of the LF display system 500, the tracking system 580 obtains an image of the viewer. The viewer profiling module 528 inputs the captured image and uses the facial recognition algorithm to identify the viewer's face. The identified face is associated with the viewer profile in the profile store, and thus, all input information obtained about the viewer can be stored in the viewer's profile. The viewer profiling module 528 may also utilize a card identification scanner, voice identifier, radio frequency identification (RFID) chip scanner, bar code scanner, etc. to positively identify the viewer.

[0133] In embodiments where the viewer profiling module 528 can positively identify a viewer, the viewer profiling module 528 can determine each visit to the LF display system 500 by each viewer. The viewer profiling module 528 can then store the time and date of each visit in a viewer profile for each viewer. Similarly, the viewer profiling module 528 can store input received from a viewer from any combination of the sensory feedback system 570, the tracking system 580, and / or the LF display assembly 510 each time it occurs. The viewer profiling system 528 can also receive additional information about the viewer from other modules or components of the controller 520, which can then be stored with the viewer profile. Other components of the controller 520 can then access the stored viewer profile to determine the content to be subsequently provided to the viewer.

[0134] The LF processing engine 530 generates holographic content that includes light field data and data for all sensory domains supported by the LF display system 500. For example, the LF processing engine 530 can generate 4D coordinates in a rasterized format ("rasterized data") that, when executed by the LF display assembly 510, enable the LF display assembly 510 to render holographic content. The LF processing engine 530 can access the rasterized data from the data store 522. Additionally, the LF processing engine 530 can construct the rasterized data from a vectorized dataset. The vectorized data is described below. The LF processing engine 530 can also generate sensory instructions necessary to provide sensory content that enhances the holographic object. As described above, when executed by the LF display system 500, the sensory instructions can generate tactile surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF processing engine 530 can access the sensory instructions from the data store 522 or construct the sensory instructions to form a vectorized dataset. The 4D coordinates and sensory data collectively represent the holographic data as display instructions that can be executed by the LF display system to generate holograms and sensory content. More generally, holographic content can take the form of computer graphics (CG) content having idealized light field coordinates, live action content, rasterized data, vectorized data, electromagnetic energy transmitted by a set of repeaters, instructions sent to a set of energy devices, energy locations on one or more energy surfaces, a set of energy propagation paths projected from a display surface, holographic objects visible to a viewer or viewers, and many other similar forms.

[0135] The amount of rasterized data describing the energy flow through the various energy sources in the LF display system 500 is very large. While it is possible to display the rasterized data on the LF display system 500 when accessed from the data store 522, it is not feasible to efficiently transmit, receive (e.g., via the network interface 524), and then display the rasterized data on the LF display system 500. Consider, for example, the rasterized data for a short film holographically projected by the LF display system 500. In this example, the LF display system 500 includes a display containing several gigapixels, and the rasterized data contains information about the location of each pixel on the display. The corresponding size of the rasterized data is enormous (e.g., several gigabytes per second of movie display time) and is unmanageable for efficient transmission over a commercial network via the network interface 524. For applications involving real-time streaming of holographic content, the efficient transmission problem can be magnified. Additional problems arise with storing the rasterized data solely on the data store 522 when interactive experiences require input from the sensory feedback system 570 or the tracking module 526. To enable an interactive experience, the light field content generated by the LF processing engine 530 can be modified in real time in response to sensory or tracking input. In other words, in some cases, the LF content cannot simply be read from the data store 522.

[0136] Thus, in some configurations, data representing holographic content displayed by the LF display system 500 can be transmitted to the LF processing engine 530 in a vectorized data format ("vectorized data"). Vectorized data can be several orders of magnitude smaller than rasterized data. Further, vectorized data provides high image quality while having a data set size that enables efficient sharing of data. For example, the vectorized data can be a sparse data set derived from a denser data set. Thus, based on how the sparse vectorized data is sampled from the dense rasterized data, the vectorized data can have an adjustable balance between image quality and data transmission size. The adjustable sampling used to generate the vectorized data can optimize image quality for a given network speed. Thus, the vectorized data enables efficient transmission of holographic content over the network interface 524. The vectorized data also enables real-time streaming of holographic content over commercial networks.

[0137] In summary, the LF processing engine 530 can generate holographic content derived from: rasterized data accessed from the data store 522, vectorized data accessed from the data store 522, or vectorized data received through the network interface 524. In various configurations, the vectorized data can be encoded prior to data transmission and decoded after being received by the LF controller 520. In some instances, the vectorized data is encoded for increased data security and performance improvements related to data compression. For example, the vectorized data received by the network interface can be encoded vectorized data received from a holographic streaming application. In some instances, the vectorized data may require a decoder, the LF processing engine 530, or both to access the information content encoded in the vectorized data. The encoder and / or decoder systems may be available for customer use or licensed to third-party vendors.

[0138] The vectorized data contains all information for each of the sensory domains supported by the LF display system 500 in a manner that can support an interactive experience. For example, the vectorized data for an interactive holographic experience can include any vectorized properties that can provide accurate physics for each of the sensory domains supported by the LF display system 500. The vectorized properties can include any properties that can be synthetically programmed, captured, computationally evaluated, etc. The LF processing engine 530 can be configured to convert the vectorized properties in the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content converted from the vectorized data using the LF display assembly 510. In various configurations, vectorized properties may include the following: one or more red / green / blue / alpha channels (RGBA) + depth images; multi-view images with or without depth information of varying resolutions, which may include a high-resolution center image and other lower-resolution views; material properties such as albedo and reflectivity; surface normals; other optical effects; surface identification; geometric object coordinates; virtual camera coordinates; display plane position; lighting coordinates; tactile stiffness of the surface; tactile extensibility; tactile intensity; amplitude and coordinates of the sound field, environmental conditions, somatosensory energy vectors associated with mechanoreceptors for texture or temperature, audio, and any other sensory domain properties. Many other vectorized properties are also possible.

[0139] The LF display system 500 can also generate interactive viewing experiences. In other words, holographic content can respond to input stimuli containing information about the viewer's position, gestures, interactions, interactions with the holographic content, or other information derived from the viewer profiling module 528 and / or tracking module 526. For example, in one embodiment, the LF processing system 500 uses vectorized data of real-time performance received via the network interface 524 to create an interactive viewing experience. In another example, if a holographic object needs to move in a certain direction immediately in response to a viewer interaction, the LF processing engine 530 can update the rendering of the scene so that the holographic object moves in the desired direction. This may require the LF processing engine 530 to use the vectorized data set to render a light field in real time based on the 3D graphics scene with appropriate object placement and movement, collision detection, occlusion, color, shadows, lighting, etc., so as to correctly respond to viewer interactions. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510.

[0140] The rasterized data includes holographic content instructions and sensory instructions (display instructions) representing the real-time performance. The LF display assembly 510 simultaneously projects the holographic and sensory content of the real-time performance by executing the display instructions. The LF display system 500 uses a tracking module 526 and a viewer profiling module 528 to monitor viewer interactions with the presented real-time performance (e.g., vocal responses, touch, etc.). In response to viewer interactions, the LF processing engine can create an interactive experience by generating additional holographic and / or sensory content for display to the viewer.

[0141] To illustrate, consider an example embodiment of an LF display system 500 that includes an LF processing engine 530, which generates a holographic object representing a product prototype. A viewer can move to touch the holographic object representing the product prototype. Accordingly, a tracking system 580 tracks the movement of the viewer's hand relative to the holographic object. The viewer's movements are recorded by the tracking system 580 and sent to the controller 520. The tracking module 526 continuously determines the movement of the viewer's hand and sends the determined movement to the LF processing engine 530. The LF processing engine 530 determines the placement of the viewer's hand in the scene and adjusts the real-time rendering of the graphics to incorporate any desired changes (such as positioning, color, or occlusion) in the holographic object. The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a tactile surface using a body haptic projection system (e.g., using an ultrasonic speaker). The generated tactile surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the exterior surfaces of the holographic object. The LF processing engine 530 uses the tracking information to dynamically instruct the LF display assembly 510 to move the position of the tactile surface along with the position of the rendered holographic object, giving the viewer both the visual and tactile perception of touching the product prototype. More simply, as the viewer watches their hand touch the holographic prototype, the viewer simultaneously feels tactile feedback indicating that their hand has touched the holographic prototype, and the prototype changes position or movement in response to the touch. In some instances, rather than presenting an interactive prototype accessed from the data store 522, the interactive prototype can be received as part of the holographic content received from the real-time streaming application via the network interface 524 (e.g., the holographic prototype can be a holographic representation of a physical prototype at a different physical location imaged by a different LF display system 500).

[0142] In embodiments where the LF display system is used to provide video conferencing, the holographic content may include holographic images of one or more participants in the video conference. The holographic content may also include other holographic objects, such as holographic handouts, holographic whiteboards, holographic movies or videos, holographic simulations, holographic product prototypes, holographic models, holographic experiences, holographic games, holographic objects, holographic assistants, any other holographic objects, or any combination thereof. In some embodiments, the holographic content may be received from a third-party system separate from the LF display system 500.

[0143] In a video conferencing configuration, participants are located at two or more physical locations. At least one of the physical locations has an LF capture system. The LF capture system, located external to the LF display, can be a plenoptic light field camera or a multi-view camera system with multiple lenses and sensors. Alternatively, the LF capture system can be integrated into the LF display assembly 510 as a bidirectional energy surface that both projects LF and absorbs and relays incident light to an imaging sensor.

[0144] The dataset generated when capturing a full LF can be unmanageable for most processors and for efficient network transmission. To address this issue, the data recorded from the light field capture system can be compressed. In some embodiments, the data is simplified to a vectorized format, including, for example, any of the following: N red / green / blue / alpha channels (RGBA) + depth image; N-view images with or without depth of different resolutions, wherein the N-view images may include a high-resolution center image and other lower-resolution views; or any simplified dataset. In this way, the amount of data required to represent the full LF can be significantly reduced (e.g., by multiple orders of magnitude), creating a viable path to achieve the transmission of a truly holographic dataset. Such techniques can provide high image quality while balancing the dataset size for efficient sharing of data, thereby providing advantages, including, for example, reduced storage requirements, enabling data to be streamed over a network in real time, etc.

[0145] In some embodiments, the data is encoded using a proprietary encoding block. This encoding process is part of a proprietary encoding / decoding pair. The receiving system or systems include a matching decoding block. These encoding and decoding blocks can be licensed from third-party vendors. The encoding block can compress the vectorized format while balancing image quality and transmission speed, and automatically adjust to network speed, for example, to provide the highest possible image quality for the available bandwidth. The encoding process can use a selectable or variable compression ratio and can include real-time or offline image processing, other computations, and / or data reduction to a sparser data set.

[0146] The encoded data is sent from the LF capture system to one or more LF display systems 500 (e.g., located at a physical location or locations of another participant or participants) via a network interface (e.g., network interface 524, if the LF capture system is integrated with the LF display system 500). The encoded data is received and decoded (e.g., by a proprietary decoder) by the LF display system 500. The decoder processes the encoded signal and, in combination with the display driver and display hardware configuration, allows the LF processing engine 530 to agnostically project the available information as a fully rasterized 4D light field, taking into account the resolution of the LF display assembly 510 and the available tactile interface and other sensory projection capabilities of the LF display system 500.

[0147] As previously described, the encoded data sent over the network is in a vectorized format. The LF processing engine 530 takes this data and converts it into a rasterized format that drives the LF display assembly 510. The rasterized format can be many orders of magnitude larger than the vectorized data set. In some embodiments, both holographic video conferencing information and CG content are sent.

[0148] In some embodiments, the encoded data also includes matching tactile surface instructions for some or all of the holographic objects. In one embodiment, the vectorized format generated from the lightfield display assembly 510 includes vectorized properties that provide accurate physics for multiple sensory domains, whose characteristics can be synthetically programmed, captured, or computationally evaluated, including, for example, any of the following: N red / green / blue / alpha channels (RGBA) + depth image; N-view images with or without depth at different resolutions, which may include a high-resolution center image and other lower-resolution views; material properties such as albedo and reflectivity; surface normals; other optical effects; surface identification; geometric object coordinates; virtual camera coordinates; display plane position; lighting coordinates; tactile stiffness of the surface; tactile extensibility; tactile intensity; amplitude and coordinates of the sound field, environmental conditions, somatosensory energy vectors associated with mechanoreceptors for texture or temperature, game audio, and any other sensory domain properties. Vectorization of data can eliminate orders of magnitude of required data, creating a feasible path to achieve the transmission of data sets containing properties in multiple sensory domains.

[0149] The LF processing engine 500 can also modify the holographic content to suit the space in which it is being presented. For example, not every conference room has the same size, the same number of seats, or the same technical configuration. Thus, the LF processing engine 530 can modify the holographic content so that it will be displayed appropriately in the conference room. In one embodiment, the LF processing engine 530 can access a configuration file for the conference room, including the conference room's layout, resolution, field of view, and other technical specifications. The LF processing engine 530 can render and present the holographic content based on the information contained in the configuration file.

[0150] The LF processing engine 530 can also create holographic content for display by the LF display system 500. Importantly, creating holographic content for display is distinct from accessing or receiving holographic content for display. In other words, when creating content, the LF processing engine 530 generates entirely new content for display rather than accessing previously generated and / or received content. The LF processing engine 530 can use information from the tracking system 580, the sensory feedback system 570, the viewer profiling module 528, the tracking module 526, or some combination thereof to create holographic content for display. In some instances, the LF processing engine 530 can access information from elements of the LF display system 500 (e.g., tracking information and / or viewer profiles), create holographic content based on that information, and, in response, display the created holographic content using the LF display system 500. When displayed by the LF display system 500, the created holographic content can be enhanced with other sensory content (e.g., touch, audio, or scent).

[0151] Dynamic content generation for LF display systems

[0152] In some embodiments, the LF processing engine 530 incorporates an artificial intelligence (AI) model to create holographic content for display by the LF display system 500. The AI model can include a supervised or unsupervised learning algorithm, including but not limited to a regression model, a neural network, a classifier, or any other AI algorithm. The AI model can be used to determine viewer preferences based on viewer information recorded by the LF display system 500 (e.g., via the tracking system 580), which can include information about the viewer's behavior.

[0153] The AI model can access information from the data store 522 to create holographic content. For example, the AI model can access viewer information from one or more viewer profiles in the data store 522, or it can receive viewer information from various components of the LF display system 500. To illustrate, the AI model may determine that one viewer prefers to view financial data in a chart, while another prefers to view financial data in a table. The AI model can determine this preference based on the viewer's reaction or response to previously viewed holographic content containing financial data (e.g., did the viewer request an explanation? Did the user request to view the data in a different format? etc.). The LF display system 500 can then present the same data to different users using different types of holographic representations. In other words, the AI model can create holographic content personalized for a group of viewers based on the learned preferences of those viewers. Thus, for example, the AI model can create a holographic chart of financial data for one user and a holographic table of the same data for another user. The AI model can also store the learned preferences of each viewer in the viewer profile store in the data store 522. In some instances, the AI model can create holographic content tailored for a group of viewers rather than a single viewer.

[0154] An example of an AI model that can be used to identify characteristics of a viewer, identify reactions, and / or generate holographic content based on the identified information is a convolutional neural network model with a node layer, where the value at the node of the current layer is a transformation of the value at the node of the previous layer. The transformation in the model is determined by a set of weights and parameters connecting the current layer and the previous layer. For example, the AI model may include five node layers: layers A, B, C, D, and E. The transformation from layer A to layer B is given by function W1, the transformation from layer B to layer C is given by W2, the transformation from layer C to layer D is given by W3, and the transformation from layer D to layer E is given by W4. In some instances, the transformation can also be determined by a set of weights and parameters used to transform between previous layers in the model. For example, the transformation W4 from layer D to layer E can be based on the parameters used to complete the transformation W1 from layer A to B.

[0155] The input to the model can be an image captured by the tracking system 580 and encoded onto the convolution layer A, and the output of the model is holographic content decoded from the output layer E. Alternatively or in addition, the output can be the characteristics of the viewer in the determined image. In this example, the AI model identifies potential information in the image that represents the viewer's characteristics in the recognition layer C. The AI model reduces the dimensionality of the convolution layer A to the dimensionality of the recognition layer C to identify any characteristics, actions, responses, etc. in the image. In some examples, the AI model then increases the dimensionality of the recognition layer C to generate holographic content.

[0156] The image from the tracking system 580 is encoded into convolutional layer A. The image input in convolutional layer A can be correlated with various characteristics and / or reaction information in identification layer C, among other things. The correlation information between these elements can be retrieved by applying a set of transformations between the corresponding layers. In other words, convolutional layer A of the AI model represents the encoded image, and identification layer C of the model represents a smiling viewer. A smiling viewer in a given image can be identified by applying transformations W1 and W2 to the pixel values of the image in the space of convolutional layer A. The weights and parameters used for the transformations can indicate the relationship between the information contained in the image and the identification of the smiling viewer. For example, the weights and parameters can be quantifications of the shape, color, size, etc. contained in the information representing the smiling viewer in the image. The weights and parameters can be based on historical data (e.g., previously tracked viewers).

[0157] The smiling viewer in the image is identified in the identification layer C. The identification layer C represents the identified smiling viewer based on the underlying information about the smiling viewer in the image.

[0158] The smiling viewer in the identified image can be used to generate holographic content. To generate holographic content, the AI model starts with the identification layer C and applies transformations W2 and W3 to the values of the identified smiling viewer given in the identification layer C. The transformation produces a set of nodes in the output layer E. The weights and parameters used for the transformation can indicate the relationship between the identified smiling viewer and specific holographic content and / or preferences. In some cases, the holographic content is output directly from the nodes of the output layer E, while in other cases, the content generation system decodes the nodes of the output layer E into holographic content. For example, if the output is a set of identified features, the LF processing engine can use the features to generate holographic content.

[0159] Additionally, AI models may include layers known as intermediate layers. Intermediate layers are layers that do not correspond to images, do not identify features / reactions, etc., or generate holographic content. For example, in the example given, layer B is an intermediate layer between convolutional layer A and identification layer C. Layer D is an intermediate layer between identification layer C and output layer E. Hidden layers are latent representations of different aspects of identification that are not observed in the data but can control the relationship between image elements when identifying features and generating holographic content. For example, a node in a hidden layer may have a strong connection (e.g., a large weight value) to input values and identification values that share the commonality of "a smiling person smiles." As another example, another node in a hidden layer may have a strong connection to input values and identification values that share the commonality of "a scared person screams." Of course, any number of links can exist in a neural network. Furthermore, each intermediate layer is a combination of functions, such as residual blocks, convolutional layers, pooling operations, skip connections, concatenation, etc. Any number of intermediate layers B can be used to reduce a convolutional layer to an identification layer, and any number of intermediate layers D can be used to add an identification layer to the output layer.

[0160] In one embodiment, the AI model comprises a deterministic method that has been trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained using measurements from the tracking system 580 as input and changes to the created holographic content as output to improve the quality of performance.

[0161] Reinforcement learning is a machine learning system in which the machine learns "what to do" - how to map situations to actions - so as to maximize a digital reward signal. The learner (e.g., the LF processing engine 530) is not told which actions to take (e.g., generate a prescribed holographic content), but rather discovers which actions produce the highest reward by trying the actions (e.g., improving the quality of the holographic content by getting more people to cheer). In some cases, an action can affect not only the immediate reward, but also the next situation, and therefore all subsequent rewards. These two characteristics, trial and error search and delayed rewards, are two distinguishing features of reinforcement learning.

[0162] Reinforcement learning is not defined by characterizing the learning method, but by characterizing the learning problem. Fundamentally, the reinforcement learning system captures those important aspects of the problem faced by a learning agent interacting with its environment to achieve a goal. In other words, in the example of generating a song for a performer, the reinforcement learning system captures information about the viewers in the venue (e.g., age, temperament, etc.). This agent senses the state of the environment and takes actions that affect the state to achieve one or more goals (e.g., creating a pop song that the viewers will cheer for). In its most basic form, the formulation of reinforcement learning contains three aspects of the learner: sensations, actions, and goals. Continuing with the song example, the LF processing engine 530 uses the sensors of the tracking system 580 to sense the state of the environment, display holographic content to viewers in the environment, and achieve a goal that is a measure of the viewer's reception of the song.

[0163] One of the challenges that arise in reinforcement learning is the trade-off between exploration and exploitation. To increase the reward in the system, a reinforcement learning agent prefers actions that have been tried in the past and found to be effective in producing reward. However, to discover actions that produce reward, the learning agent chooses actions that it has not previously chosen. The agent "exploits" information that it already knows to obtain rewards, but it also "explores" information to make better action choices in the future. The learning agent tries various actions and gradually prefers those that seem to work best, while still trying new actions. On a random task, each action is typically tried many times to get a reliable estimate of its expected reward. For example, if the LF processing engine creates holographic content that the LF processing engine knows will cause a viewer to laugh after a long period of time, the LF processing engine can change the holographic content so that the time until the viewer laughs is reduced.

[0164] Further, reinforcement learning considers the entire problem of goal-directed agents interacting with uncertain environments. Reinforcement learning agents have clear goals, can sense aspects of their environment, and can choose actions that receive high rewards (i.e., roaring at the crowd). Moreover, the agent typically operates despite significant uncertainty in the environment it faces. When reinforcement learning involves planning, the system addresses the interplay between planning and real-time action selection, as well as how to acquire and improve elements of the environment. For reinforcement learning to make progress, it is necessary to isolate and study important subproblems that play a clear role in the complete interactive goal-seeking agent.

[0165] The reinforcement learning problem is a framework for machine learning problems in which interactions are processed and actions are performed to achieve a goal. The learner and the decision maker are called agents (e.g., the LF processing engine 530). The things that interact with it (including everything outside the agent) are called the environment (e.g., viewers in the venue, etc.). The two interact continuously, the agent chooses actions (e.g., creating holographic content), and the environment responds to these actions and presents new situations to the agent. The environment also brings rewards, that is, special numerical values that the agent tries to maximize over time. In one context, the reward plays the role of maximizing the viewer's positive response to the holographic content. The complete environment specification defines a task as an instance of a reinforcement learning problem.

[0166] To provide more context, the agent (e.g., LF processing engine 530) interacts with the environment at each discrete time step in a series of discrete time steps (i.e., t=0, 1, 2, 3, etc.). At each time step t, the agent receives some environment state s t A representation of (e.g., measurements from tracking system 580). State s t In S, where S is the set of possible states. Based on state s t At time step t, the agent chooses an action at (e.g., make the performer split). t ), where A(s t ) is the set of possible actions. One time state later (partially as a result of its actions), the agent receives a numerical reward r t+1 . Status r t+1 Within R, where R is the set of possible rewards. Once the agent receives the reward, it chooses a new state s t+1 .

[0167] At each time step, the agent implements a mapping from states to the probability of choosing each possible action. This mapping is called the agent’s policy and is denoted as π t , where if s t =s, then π t (s, a) is a t = the probability of a. Reinforcement learning methods can determine how an agent changes its policy due to the states and rewards generated by its actions. The agent's goal is to maximize the total number of rewards received over time.

[0168] This reinforcement learning framework is very flexible and can be applied to many different problems in many different ways (for example, generating holographic content). The framework proposes that any problem (or purpose) of learning goal-directed behavior, regardless of the details of the sensory, memory, and control devices, can be reduced to three signals passed back and forth between the agent and its environment: a signal representing the choice made by the agent (the action), a signal representing the basis for the choice (the state), and a signal defining the agent's goal (the reward).

[0169] Of course, the AI model can include any number of machine learning algorithms. Some other AI models that can be employed are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, and the like. Regardless, generally, the LF processing engine 530 takes input from the tracking module 526 and / or the viewer profile analysis module 528, and in response, the machine learning model creates holographic content. Similarly, the AI model can guide the presentation of holographic content.

[0170] In some embodiments, the LF processing engine 530 generates display instructions for presenting a holographic assistant to a participant in a video conference. Such an assistant can provide information related to the video conference to the participant, assist in the presentation, and the like. The LF processing engine 530 can retrieve holographic assistants stored as holographic objects in a data store. Each holographic assistant can have various parameters that determine how the holographic assistant is presented. For example, parameters that a holographic assistant can have include, but are not limited to, the type of assistant (e.g., human avatar, alien, robot, humanoid, etc.), the size of the assistant, the gender of the assistant (when available), the voice of the assistant, the personality of the assistant, or any combination thereof.

[0171] In another embodiment, the LF processing engine 530 further accesses a viewer profile that may include the participant's preferences for the assistant. For example, the viewer profile may include a preference (e.g., provided by the participant or inferred by the viewer profiling module 528) that the participant prefers the assistant to be a male human with brown hair and a deep voice, a female alien with black hair and a high-pitched voice, etc. Furthermore, the LF processing engine 530 may incorporate an AI model, as described above, to generate instructions for presenting the assistant, thereby presenting a holographic assistant that interacts with the participant. The AI model may be used to continuously generate instructions for the sensory feedback system 570 to provide audio feedback in response to voice input from the participant, i.e., to simulate a real-time conversation between the participant and the holographic assistant. Furthermore, the LF processing engine 530 may use tracking information from the tracking system 580 and / or the tracking module 526 to generate display instructions for the LF display assembly 510 to adjust the presentation of the holographic assistant's eyes to move or track with the participant's gaze or body movements.

[0172] The foregoing examples of creating content are not limiting. In the broadest sense, the LF processing engine 530 creates holographic content for display to a viewer of the LF display system 500. The holographic content may be created based on any of the information contained in the LF display system 500.

[0173] Light field display video conferencing system

[0174] Figure 6 is an illustration of an LF display system 600 for video conferencing according to one or more embodiments. The LF display system 600 is an embodiment of the LF display system 500. The LF display system 600 is positioned in a meeting space 610. The meeting space 610 is typically a room containing an LF display formed by an LF display module 620 of the LF display assembly 510. Figure 6 In the illustrated configuration, the LF display module 620 is positioned to cover one wall of a video conferencing room. However, the conference space 610 can be any physical space where the LF display module 620 can be temporarily or permanently installed. In one embodiment, the LF display system 600 has an imaging system integrated into the LF display assembly 510. Thus, the LF display assembly 510 can function as a bidirectional display surface that both projects a light field and relays light from the display surface to at least one imaging sensor. The LF display system can additionally or alternatively include a camera external to the display, with the image analyzed for use by the light field display assembly 510 using tracking software in the controller 520.

[0175] exist Figure 6 6, meeting space 610 also includes a table 630 and chairs 640. Table 640 is positioned against a wall having an LF display module 620, and chairs 640 are positioned so that a participant is seated at table 630. LF display module 620 is generating a holographic image of table 632 and chairs 642 located in a different physical space 612, where another video conference participant is located. The holographic image can be generated based on visual data collected by a remote light field display assembly (e.g., one or more remote LF display modules) located in the remote meeting space 612. For example, in one embodiment, each meeting space includes an LF display system 600 having an imaging system integrated into LF display assembly 510. Thus, each display assembly 510 acts as a bidirectional surface that can be configured to give the impression that two meeting spaces are physically adjacent to each other, with LF display assembly 510 acting as a window between them.

[0176] The visual data used to generate the holographic image can be a digital representation of light detected by one or more imaging sensors (e.g., imaging sensors of one or more LF display modules 620). For example, in the case of chair 642, the visual data can be light reflected from the chair and detected by LF display module 620 of LF display system 600 in remote meeting space 612. Alternatively or in addition, remote light field display assembly 510 can use other sensors such as independent cameras to collect visual data. In some embodiments, the visual data can be vectorized and / or compressed, as previously described with respect to Figure 5 described.

[0177] Holographic table 632 is adjacent to physical table 630. Thus, to a participant in meeting space 610 (e.g., sitting in chair 640), the images of physical chair 640 and chair 642 in another physical location 612 surround a single table (composed of images of physical table 630 and table 632 in another physical location). Similarly, in another physical location 612, holographic table 632 is presented along with holographic chair 642 as a physical instance adjacent to table 630. Although Figure 6 Two identical tables are shown to improve the illusion of being two halves of a single table, but tables of different sizes and shapes may be used.

[0178] When a participant sits in chairs 640, 642 (or otherwise positions themselves in the corresponding viewing space), a hologram of the participant can be provided in another physical location. For example, a hologram of a first user sitting in chair 640 can be provided in second meeting space 612, and a hologram of a second participant sitting in chair 642 can be provided in first meeting space 610. Although each physical location is described as having a single chair and participant, it should be understood that one or both locations can contain multiple participants. It should also be understood that in some embodiments, there can be more than two physical locations.

[0179] In one embodiment, remote meeting space 612 does not include an LF display system. Instead, it captures images using an LF camera and sends the images to LF display system 600 in meeting space 610. In other embodiments, both remote meeting space 612 and local meeting space 610 include LF display systems 600 and light field capture devices. In another embodiment, holographic data is captured in either space using one or more conventional 2D cameras and a depth sensor. In yet another embodiment, 2D image data is captured using one or more 2D cameras in either the remote or local meeting space, and holographic data is generated from the 2D image data using 2D-to-3D conversion techniques known in the art. In one embodiment, LF display module 620 includes a bidirectional surface that projects a light field and simultaneously absorbs incident light, relaying it to one or more image sensors, which can be used to record the light field from the area near the display surface. In a configuration where a light field display system 600 with such a bidirectional surface is located at two locations involved in a video conference, a one-to-one correspondence can exist between the gaze directions of participants in the physical locations connected by the video conferencing solution, as LF display module 620 acts as a co-located display and camera. Thus, the participants can make eye contact as if they were located in the same physical space. In other words, the LF display module 620 acts as a window between the two locations, leaving the impression that the participants are sitting around a table.

[0180] The LF display module 620 may also include an audible sound field projection system. This can be emitted from an electrostatic speaker mounted on the display surface, having an optically transparent membrane, located between the waveguide elements, and co-located with or helping to form a structure designed to suppress light transmission between the waveguides, allowing both a sound field and a light field to be projected simultaneously from the display surface. Sound detected by a microphone in one physical location can be emitted by a corresponding LF display module 620 in another physical location. Therefore, when a participant in one physical location speaks or otherwise makes a sound, they can appear to be in another physical location, as if the sound is coming from a hologram of the participant. Alternatively, a separate microphone and speaker can be used to establish an audio connection between the physical locations.

[0181] In some embodiments, the tactile projection system is integrated into the display surface formed by the LF display module 620. Thus, the display surface can be a bidirectional energy surface that simultaneously projects both light fields and focused ultrasound energy to create a tactile surface. This can allow participants in a video conference to experience the sensation of physical contact with a remote participant, such as shaking hands. It can also be used to generate holographic objects that participants can touch and manipulate (e.g., holographic props 650, which are described in more detail below).

[0182] In some embodiments, the display surface includes speakers (e.g., electrostatic speakers) for generating an audio field. Thus, the system can control the direction from which participants hear sounds. For example, the controller 520 of the LF display system 600 can use tracking software to determine which remote participant is speaking and cause the speakers to emit an audio field that substantially co-locates the apparent source of the generated audio and a holographic representation of the speaking participant. This can further enhance the impression that the participants are all located in the same physical space. Additionally or alternatively, the LF system 600 can include one or more speakers external to the display.

[0183] exist Figure 6 In the illustrated embodiment, meeting space 610 also includes a mobile LF system 660. Mobile LF system 660 can have a display surface area that is smaller than the display surface area of one or more other LF displays, such as the surface formed by LF modules 620. Although mobile LF system 660 is shown as a robot on wheels, other forms of mobility can also be used. For example, mobile LF system 660 can be mounted on tracks, mounted on an articulated robotic arm, or carried by a human participant. Regardless of the precise form employed, mobile LF system 660 can provide functionality that further enhances the impression that participants from different physical locations are present in the same space.

[0184] In various embodiments, the mobile LF system 660 may include an image capture system, an image display system, or both. The image capture system of the mobile LF system 660 may include one or more 2D cameras, depth sensors, and / or LF cameras. Thus, the mobile LF system 660 may be configured to capture 2D image data and / or holographic data of an area near the mobile LF system 660 (e.g., in front of the mobile LF system). The mobile LF system 660 may also convert the 2D image data into holographic data (e.g., using depth sensor data and / or 2D to 3D techniques known in the art) in a configuration that captures 2D image data. The mobile LF system 660 may also be configured to capture other types of data. For example, the mobile LF system 660 may include a microphone configured to capture audio data.

[0185] The image display system of the mobile LF system 660 may include a 2D display, a LF display system, or both. In some embodiments, the mobile LF system 660 includes a bidirectional surface that can be configured to simultaneously capture LF image data and project holographic content.

[0186] In one embodiment, mobile LF system 660 is located in one space (e.g., local meeting space 610) but is controlled by a participant located in a remote location (e.g., remote meeting space 612). Thus, the controlling participant can navigate mobile LF system 660 around meeting space 610 to view perspectives that might otherwise be unavailable (e.g., the back of chair 640). Furthermore, mobile LF display 660 can create holographic images of some (e.g., head and shoulders) or all of the controlling participant (or an avatar of the controlling participant). Thus, the controlling participant can explore meeting space 610, and other participants physically located within meeting space 610 can interact with the controlling participant's holographic image as if they were actually there. This can further enhance the impression that the participants are located within a common physical space.

[0187] The LF display system 600 can also present supplemental holographic content in the meeting space 610 . Figure 6 The illustrated embodiment includes two examples of supplemental holographic content: a holographic prop 650 (in this case, a holographic image of a car) and a holographic whiteboard 670. Holographic prop 650 and holographic whiteboard 670 can be displayed in both the local meeting space and the remote meeting space, or in only one of these spaces. In other embodiments, different and / or additional supplemental holographic content can be presented by the LF display system.

[0188] The holographic prop 650 is a holographic object such as a 3D CAD model. The holographic prop 650 can be an image of a physical object (e.g., a product prototype located in the second meeting space 612) or a CG virtual object (e.g., generated from a file on a computer such as the controller 520). This can further enable participants in different physical locations to interact as if they were in the same space. For example, a participant in the second meeting space 612 can select a 3D CAD model and the LF display system 600 can generate the holographic prop 650 in the holographic viewing volume in the first meeting space 610. The location of the holographic viewing volume can be selected based on the preferences of one or more participants located in the meeting space 610. Alternatively, the holographic prop 650 can appear in a default location.

[0189] In some embodiments, as previously described, ultrasound waves are used to provide a tactile surface for the holographic prop 650. The tracking system 580 of the LF display system 600 can track the movements of participants in the meeting space 610 to enable manipulation of the holographic prop. For example, a participant may be able to reach out and "grab" the holographic prop 650 to rotate and / or move it. The tracking system 580 can also recognize certain gestures as commands related to the holographic prop 650. For example, if a participant places both hands on (or near) the holographic prop 650 and moves their hands apart or together, the tracking system 580 can interpret these gestures as commands to increase and decrease the size of the holographic prop 650, respectively.

[0190] If both meeting spaces 610 and 612 include an LF display system 600, a shared instance of holographic prop 650 can be created in each meeting space. Thus, interactions with holographic prop 650 by participants in one space can be reflected in the presentation of holographic prop 650 in the other space. For example, if holographic prop 650 was generated from a 3D CAD model, participants in one space can rotate the model to make specific features visible while describing the features as part of the presentation. Holographic prop 650 in the other space can then automatically rotate in a manner that allows all participants to see the described features in the same way. In some embodiments, participants can use a stylus (or other such tool) to annotate holographic prop 650. Tracking system 580 can track the stylus's movements and add the annotations to a file that generates a holographic handout and / or display a holographic version of the annotations in conjunction with holographic prop 650. Returning to the previous example, the presenter can draw a circle around the described feature, provide a written annotation that provides additional information about the feature, add arrows indicating how the feature moves during operation, and so on.

[0191] The holographic whiteboard 670 provides participants with an interface for drawing on it similar to a physical whiteboard. In some embodiments, the LF display system 600 provides a holographic image (e.g., a holographic image of a physical whiteboard) indicating the extent of the holographic whiteboard 670 in each meeting space 610, 612. For example, in Figure 6 In the configuration shown, each meeting space 610, 612 can include a holographic whiteboard 670 adjacent to a physical table 630, 632. The holographic whiteboards 670 in each location are synchronized, meaning any additions or modifications made to one whiteboard appear on the others. Thus, participants in different physical locations can collaborate on the holographic whiteboards 670 as if they were in the same physical space. Ultrasound can be used to create a tactile surface for some or all of the holographic whiteboards 670 to give participants the sensation of drawing on a physical whiteboard (or some other surface, such as paper, a blackboard, a canvas, etc.).

[0192] In one embodiment, contributions from different participants are stored as different layers. Participants can select which layers to display on the holographic whiteboard 670 (e.g., using a physical remote control and / or holographic controls). Layer selection can be synchronized across instances of the holographic whiteboard 670. Alternatively, participants at each location can independently select which layer to display. In other embodiments, layers can be added, used, and displayed in different ways. For example, participants can be provided with (e.g., physical or holographic) controls to add and delete layers and select which layer is currently active. New content can be added to the currently active layer. Controls can also enable participants to split layers and move content between layers. Thus, participants can control which content on the holographic whiteboard 670 is in which layers.

[0193] In some embodiments, the holographic whiteboard 670 is 3D. Because the whiteboard is holographic, it need not be limited to a 2D surface. Participants can draw lines that move forward / backward, up / down, and / or left / right. This allows participants to directly draw 3D structures and may make it easier to express complex concepts and relationships than when limited to a 2D representation. In one embodiment, the 3D whiteboard is divided into slices along the front / back axis. Participants can select which slices are displayed to view different cross-sections of the depicted 3D structure.

[0194] In some embodiments, the holographic whiteboard 670 can be used to display 3D holographic objects, such as CAD models, building schematics, park plans, and the like. The 3D objects can be generated from data files or by imaging physical objects (e.g., product prototypes, a set of blueprints, etc.) located within one of the meeting spaces (e.g., remote meeting space 612). In one example, each page of a document can be presented as a different layer on the holographic whiteboard 670. Participants can select which layers to view to read a specific page. Participants can also independently move pages around within the 3D volume of the holographic whiteboard. Thus, participants can view any combination of pages (e.g., side by side). Similarly, participants can arrange corresponding pages of two different versions of a document for easy comparison. As another example, the LF processing engine 530 can take a set of blueprints for a building and use them to create an approximate 3D model of the building. Participants can then interact with the holographic whiteboard 670 to, for example, view different cross-sections of the building, remove certain elements (e.g., wiring, doors, specific walls, etc.) from the displayed model, modify the model (which can then be used to generate updated blueprints), add annotations, and so on.

[0195] LF display system 600 can modify the images of participants and / or other holographic objects. In one embodiment, to improve the impression that the holographic objects are located within meeting space 610, LF display system 600 determines the lighting parameters within meeting space 610 (e.g., overall brightness and spectral distribution) and adjusts the holographic image to better match the lighting parameters. For example, if meeting space 610 has bright light on the left side, the holographic image can be adjusted so that the left side is brighter and the right side is in shadow. This can increase the impression that the image is a physical object located within meeting space 610.

[0196] The LF display system 600 can also adjust the audio parameters used for the generated audible sound to account for the acoustics within the meeting space 610. For example, if a corner typically has muffled sound, the LF display system can apply equalization to the high-frequency band of acoustic energy toward that corner to provide clearer sound. The LF display system can also provide noise cancellation. For example, one or more LF display modules 620 at the back of the meeting space 610 can emit sound waves that partially or completely cancel out sound waves emitted from the LF display modules 620 at the front of the meeting space 610.

[0197] Light field display video chat system

[0198] Figure 7 An alternative configuration of a conference space 710 including a LF display system 700 is shown according to one or more embodiments. The LF display system 700 is an embodiment of the LF display system 500. Figure 7 In FIG, , rather than generating holographic images of the table and chairs positioned behind the display surface, the LF display system 700 generates holographic images of the table 730 and chairs 742 in front of the display surface. Thus, the physical table 730 and chairs 742 in the meeting space 710 are not as physically present as they would be in the conference room. Figure 6 To do this, the LF display system transforms the holographic image by projecting it so that it appears in front of the LF display module 720, rather than behind it (as in FIG. Figure 6 Similarly, in another physical location (where table 732 and chair 742 are located), holographic images of table 730 and chair 740 can be formed by transforming the images by projecting them so that they also appear in front of the display surface in the other physical location.

[0199] Transforming the image in this way can improve the impression that all holographic images (including participants) are located in the same physical space. Rather than acting as a window that participants can see but not pass through, the LF display module 720 projects the images of remote participants into the meeting space 710. Thus, participants can walk up to each other. In addition, an ultrasonic tactile surface can be used to simulate participants in remote locations touching each other. Although Figure 7 As not shown, the conference space 710 may also include other holographic objects, such as holographic handouts, whiteboards, etc., as described above with reference to Figure 6 described.

[0200] Figure 8A FIG8 is an illustration of an LF display system 800 presenting holographic content including holographic video chat participants, according to one or more embodiments. LF display system 800 is an embodiment of LF display system 500. LF display system 800 includes an LF display module 820 that forms an LF display assembly that forms a single-sided seamless surface environment. LF display system 800 provides video chat functionality in which participants view holographic images of some or all of the other participants, generated by LF display module 820. LF display system 800 may also include any combination of other components of LF display system 500, such as sensory feedback assembly 570, tracking system 580, viewer profiling module 528, and controller 520. In other embodiments, LF display system 900 includes an additional camera for capturing image data, separate from the tracking system.

[0201] exist Figure 8A In the illustration, first video chat participant 830 is positioned in the viewing area of LF display system 800. LF display system 800 captures image data of first participant 830 via any combination of a tracking system (e.g., tracking system 580), LF display module 820, one or more cameras included in LF display system 800, and any additional tracking devices. In some instances, LF display system 800 receives image data of first participant 830 from multiple perspectives, such as from one or more LF display modules 820 or from one or more cameras separate from or as part of the tracking system. In some embodiments, LF display system 800 has cameras surrounding first participant 830 that capture image data covering all 360° perspectives. Alternatively, LF display system 800 can be "unidirectional," meaning that a holographic representation of second participant 840 is presented to first participant 830, but LF display system 800 does not collect image data of first participant 830 and / or does not stream such image data to a display device being used by second participant 840.

[0202] The LF display system 800 generates holographic content including a holographic representation of the second video chat participant 840. The holographic representation of the second video chat participant 840 is a holographic image. In one embodiment, the LF display module 820 is a bidirectional surface that also collects image data of the first participant 830 and sends it to the LF display system of the second participant 840 for presentation as a holographic image. In other words, the LF display module 820 can act as a window through which the participants can view each other. Figure 6 As with the video conferencing system described above, there can be a one-to-one correspondence between the gaze directions of the participants, enabling the participants to make eye contact with each other as if they were located in the same physical space. Alternatively, the LF display system 800 can be "one-way," wherein only one of the participants is presented with a holographic representation of the other participant.

[0203] Figure 8B According to one or more embodiments Figure 8A 800 presenting holographic content including a holographic image of a second video chat participant 850. Figure 7 In the illustrated video conferencing configuration, the LF display system 800 projects a holographic image of the second participant 850 into the same physical space as the first participant 730. Figure 8A , the first participant 830 experiences the impression that the second participant 850 is in the same physical location as them. The same or similar techniques as previously described for adjusting lighting and / or sound parameters to improve the overall experience can be applied. In addition, as previously described, the ultrasound projection system can be used to create somatotactile tactile surfaces for some or all of the participants' holographic representations. These tactile surfaces can simulate the participants touching each other, further leaving the impression that they are located in the same physical space. The ultrasound projection system can work in conjunction with the tracking system 580 to update the tactile surfaces. For example, using this approach, remote participants in a video chat can experience the feeling of shaking hands with each other, where the tactile tactile surface experienced by each participant is updated based on the movement of the other participant's hand.

[0204] The LF display system 800 can also enable video chat participants to change their appearance and / or voice. In one embodiment, the LF display system 800 provides (e.g., physical or holographic) controls that enable participants to adjust parameters of their holographic representation presented to other participants. For example, a participant can blur their face, select a custom background, add balloons or other objects floating around them, make it appear as if they are sitting behind a table, and so on. Similarly, the LF display system 800 can enable participants to apply filters to change their voice, such as increasing or decreasing the pitch, making it sound robotic, selectively reducing the intensity of certain frequencies, and so on.

[0205] In other embodiments, the LF display system 800 provides (e.g., physical or holographic) controls that enable participants to apply filters to change various aspects of their own or other participants' appearance. For example, filters can change a participant's clothing, change a participant's hair color, replace a participant's image with an avatar mapped to the participant's movements, and / or apply any other visual filters to how a participant appears. In one embodiment, the filters are modular and participants can install (e.g., by downloading from an online store, etc.) the filters they wish to use. In some cases, third parties may provide additional filters. Such third-party filters may be provided free of charge and / or for a price (e.g., through a marketplace system).

[0206] Figure 9 is an illustration of an LF display system 900 presenting holographic content including holographic representations of participants in a group video chat in accordance with one or more embodiments. The LF display system 900 is an embodiment of the LF display system 500. A first video chat participant 930 is in the same physical location as the LF display system 900. The LF display system includes an LF display module 920 that generates a holographic image representation of an active participant 950 in the voice chat. The LF display module 920 also generates a holographic image representation of one or more additional participants 960. The representations may be images of the participants and / or avatars of the participants. In Figure 9 In the example embodiment, two additional participants 960 are shown, but the voice chat can include any number of additional participants (including zero participants). Furthermore, while the representations of additional participants 960 are shown as reduced in size, in other embodiments, other indicators can be used for active participants 950, such as displaying the representations of active participants 950 in a particular location (e.g., centrally), displaying a particular visual indicator in conjunction with the representations of active participants 950 (e.g., a glow above the representation of the participant's head, etc.), or any other suitable indicator. In some embodiments, not all participants are represented by holographic representations. For example, participants who do not have access to the LF camera (or choose not to use it) can be represented by static avatars.

[0207] The LF display system 900 may use any suitable method to determine which participant is the active participant 950. In one embodiment, the LF display system identifies the participant currently speaking as the active participant 950 based on the collected acoustic energy. In another embodiment, the first participant 930 may change the active participant 950 using controls provided by the LF display system 900 (e.g., by pointing to one of the holographic representations of the additional participants 960). Although Figure 9 The holographic representations are shown standing, but need not be. For example, representations of participants can be presented as sitting around a holographic (or physical) table, with active participants 950 placed at the head of the table.

[0208] In various embodiments, the LF display system 500 provides a holographic voicemail feature. This operates substantially as described above with respect to video chat, except that the LF display system 500 creates a message for later presentation by collecting audio and visual data corresponding to an individual and saving it (e.g., in data storage 530) rather than presenting it in substantially real time at another location. For example, an individual can use physical and / or holographic controls to start, pause, and end a recording, and then select one or more individuals to receive the recorded message. Later, the recipient can use physical and / or holographic controls of the LF display system 500 (which can be a different or the same system as that used to record the message) to trigger playback of the message. During playback, the LF display system 500 presents a holographic image of the individual who recorded the message, along with the corresponding audio content. In some embodiments, the same or similar techniques as described above for modifying an individual's appearance and / or voice can be used.

[0209] In one embodiment, the LF display system 500 provides controls to enable the recipient to control the playback of the message, such as controls to start, pause, fast forward, rewind, and / or end playback. As the LF display system 500 creates holographic content from the visual data, the recipient can move around to view the visual content of the message from different perspectives. Alternatively, the controls can include controls for rotating, panning, zooming in and out, etc., so that the recipient can view different perspectives without moving. Thus, the recipient can play the message multiple times to view it from different perspectives.

[0210] In some embodiments, a similar playback function can be provided for recordings of video conferences and / or video chat sessions. The LF display system can provide an option for recording a video communication session. Participants (and anyone else with access to the recording) can then replay the recorded session to view the recorded scene from a different perspective.

[0211] Additional configuration information

[0212] The foregoing description of the embodiments of the present disclosure is presented for illustrative purposes; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations of the LF display system are possible in light of the above disclosure.

[0213] Some parts of this description describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are typically used by those skilled in the art of data processing to effectively convey the essence of their work to other persons skilled in the art. Although these operations are described functionally, computationally, or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it is sometimes convenient to refer to the arrangement of these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0214] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In some embodiments, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0215] Embodiments of the present disclosure may also relate to devices for performing the operations described herein. This device may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions that can be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that uses multiple processors to increase computing power.

[0216] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information generated by the computing processes, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0217] Finally, the language used in the specification is primarily selected for readability and instructional purposes and may not be selected to describe or limit the subject matter of the present invention. Accordingly, it is intended that the scope of the present disclosure be limited not by this detailed description, but rather by any claims based upon the application appended hereto. Accordingly, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure as set forth in the following claims.

Claims

1. A light field display system for video communication, the light field display system comprising: a controller configured to generate display instructions based on visual data corresponding to a remote scene, the visual data being received from a remote image capture system, and a local light field display assembly configured to generate a holographic image of the remote scene based on display instructions from the controller; wherein the local light field display assembly comprises a bidirectional surface that projects the light field and simultaneously absorbs incident light, relaying it to one or more image sensors for recording the light field from an area near the display surface; wherein the remote image capture system is a remote light field display system, the remote light field display system comprising a bidirectional surface that simultaneously captures light field image data corresponding to the remote scene and projects holographic content; There is a one-to-one correspondence between gaze directions in a local physical space containing the local light field display assembly and a remote physical space containing the remote image capture system. 2 . The light field display system of claim 1 , wherein the local light field display assembly comprises a plurality of light field display modules. 3 . The light field display system of claim 2 , wherein the plurality of light field display modules are tiled to form a seamless display surface. 4 . The light field display system of claim 1 , wherein the local light field display assembly is configured to generate visual data corresponding to a local scene and transmit the visual data corresponding to the local scene to a remote light field display system.

5. The light field display system of claim 1, wherein the display instructions further cause the local light field display assembly to generate a holographic prop.

6. The light field display system of claim 5, wherein the holographic prop is a holographic image generated from a 3D model of an object.

7. The light field display system of claim 5 , further comprising a tracking system configured to generate tracking data representing motion of a local participant positioned within a physical space containing the local light field display assembly, wherein the holographic prop is manipulated based on the tracking data.

8. The light field display system of claim 1, wherein the display instructions further cause the local light field display assembly to generate a holographic whiteboard, wherein content on the holographic whiteboard is synchronized with a corresponding holographic whiteboard generated by a remote light field display assembly.

9. The light field display system of claim 8, wherein the holographic whiteboard is a three-dimensional holographic whiteboard.

10. The light field display system of claim 1, wherein the display instructions further cause the local light field display assembly to emit ultrasound to generate a tactile surface.

11. The light field display system of claim 10, wherein the tactile surface is juxtaposed with at least a portion of a holographic image.

12. The light field display system of claim 11, wherein the holographic image is a holographic image of a remote participant.

13. The light field display system of claim 11, wherein the holographic image is a holographic prop.

14. The lightfield display system of claim 1, further comprising a mobile lightfield system positioned within a physical space including the local lightfield display assembly.

15. The light field display system of claim 14, wherein a remote participant controls movement of the mobile light field system within the physical space containing the local light field display assembly.

16. The light field display system of claim 15, wherein the mobile light field system is configured to capture image data corresponding to an area near the mobile light field system and provide the captured image data to a remote light field display assembly for use in generating holographic content in a remote physical space.

17. The light field display system of claim 1, further comprising a mobile light field system comprising a bidirectional surface configured to simultaneously capture light field image data and project holographic content.

18. The light field display system of claim 17, wherein a remote participant controls movement of the mobile light field system within a local physical space containing the local light field display assembly.

19. The light field display system of claim 18, wherein image data captured by the mobile light field system is projected to the remote participant as holographic content.

20. The light field display system of claim 19, wherein a hologram of at least a portion of the remote participant is projected into the local physical space simultaneously with projecting the holographic content to the remote participant.

21. The light field display system of claim 1, wherein the display instructions causing the local light field display assembly to generate the holographic image of the remote scene include instructions causing the local light field display assembly to generate holographic images of one or more remote participants.

22. The light field display system of claim 21, wherein the holographic image of a first remote participant of the one or more additional remote participants is reduced in size relative to the holographic image of a second remote participant of the one or more additional remote participants.

23. The light field display system of claim 22, wherein in response to determining that the second remote participant is currently speaking, a size of the holographic image of the first remote participant is reduced.

24. The light field display system of claim 1, wherein the holographic image of the remote scene comprises an avatar of a remote participant, wherein movement of the avatar is synchronized to movement of the remote participant.

25. The light field display system of claim 1, wherein the holographic image of the remote scene comprises a holographic image of a remote participant, the holographic image of the remote participant adjusted to match one or more lighting parameters of a physical location at which the local light field display assembly is positioned.

26. The light field display system of claim 1, wherein the visual data is vectorized, and the light field display system further comprises a processing engine configured to convert the vectorized visual data into rasterized visual data, the display instructions being generated from the rasterized visual data. 27 . The light field display system of claim 1 , wherein the light field display system further comprises a local decoding block matched with a remote encoding block for compressing the visual data, the local decoding block being configured to decompress the visual data.

28. The light field display system according to claim 27, wherein the light field display system further comprises: a local light field capture system configured to generate visual data corresponding to a local participant; a local encoding block configured to compress the visual data corresponding to the local participant; as well as A network interface is configured to transmit compressed visual data corresponding to the local participant to a remote light field display system.

29. The light field display system of claim 1, wherein the visual data corresponds to a pre-recorded holographic video, the visual data causing the local light field display assembly to generate the holographic video in response to user input provided by a local participant.

30. The light field display system of claim 29, wherein the light field display system further comprises a control configured to enable the local participant to change a viewing angle of the holographic video.

31. The light field display system of claim 1 , wherein the controller receives audio data, and the light field display system further comprises one or more electrostatic speakers mounted on a display surface of the light field display assembly and configured to generate an audio field based on the audio data.

32. The light field display system of claim 31 , wherein the one or more electrostatic speakers comprise a film transparent to visible light.

33. The light field display system of claim 31 , wherein the audio data comprises audio content and direction information, and the electrostatic speaker is configured to reproduce the audio content such that a local participant perceives the audio content as coming from a direction determined according to the direction information.

34. The light field display system of claim 31 , wherein the one or more electrostatic speakers are configured to generate ultrasonic energy that forms the tactile surface.

35. The light field display system of claim 1, wherein the holographic image of the remote scene is modified.

36. The light field display system of claim 35, wherein the holographic image of the remote scene is modified by at least one of applying a blur effect to the holographic image, changing a remote participant's clothing, changing the remote participant's hair color, replacing the remote participant's image with an avatar mapped to the remote participant's movements, applying a custom background, or augmenting the remote scene with one or more holographic objects.

37. The light field display system of claim 35, wherein the holographic image of the remote scene is modified by a filter provided by a third party.

Images