Video communication including holographic content

Abstract

To solve the problem in which existing video communication solutions limit the quality of user experience.SOLUTION: A video communication system uses a light field (LF) display to present a holographic image of a remote scene (e.g., a hologram of a remote participant). The system may include a local light field display assembly and a controller. The controller generates display instructions based on visual data corresponding to a remote scene received from a remote image capture system (e.g., a remote light field display system). The display instructions cause the local light field display assembly to generate a holographic image of the remote scene.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042470, and PCT / US2017 / 042679, all of which are incorporated herein by reference in their entireties. [Background technology]

[0002] The present disclosure relates generally to communication systems, and more particularly to video communication using light field display systems.

[0003] As the availability of communication bandwidth increases and the cost of digital cameras decreases, video is becoming an increasingly popular method of communication. However, several factors limit the quality of the user experience with existing video communication solutions. Typically, the camera and screen are located close to each other but in significantly different positions. As a result, participants often do not make eye contact and are unaware of other gestures and facial expressions that humans use to add context to spoken words. Direct interaction is also difficult because the display on which the video is presented creates a barrier between participants. While some existing video communication technologies allow documents to be viewed via screen sharing features, this results in participants' videos being removed or reduced in size. Summary of the Invention

[0004] A video communication system uses a light field display to present a holographic image of a remote scene that may include one or more remote participants. In one embodiment, the system includes a local light field display assembly and a controller. The controller generates display instructions based on visual data corresponding to the remote scene received from a remote image capture system, such as another light field display assembly or a remote light field display system. The display instructions cause the local light field display assembly to generate a holographic image of the remote scene. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a diagram of a light field display module for presenting holographic objects, according to one or more embodiments. [Figure 2A] 1 is a cross-section of a portion of a light field display module according to one or more embodiments. [Figure 2B] 1 is a cross-section of a portion of a light field display module according to one or more embodiments. [Figure 3A] FIG. 1 is a perspective view of a light field display module according to one or more embodiments. [Figure 3B] 1 is a cross-sectional view of a light field display module including interleaved energy relay devices according to one or more embodiments. [Figure 4A] FIG. 1 is a perspective view of a portion of a light field display system that is two-dimensionally tiled to form a single-sided seamless surface environment, according to one or more embodiments. [Figure 4B] FIG. 1 is a perspective view of a portion of a light field display system in a multi-sided seamless surface environment, according to one or more embodiments. [Figure 4C] FIG. 1 is a top view of a light field display system having a collecting surface in a wing configuration, according to one or more embodiments. [Figure 4D] 1 is a side view of a light field display system having a collecting surface in a tilted configuration according to one or more embodiments. FIG. [Figure 4E] 1 is a top view of a light field display system having a collecting surface on the front wall of a room in accordance with one or more embodiments. FIG. [Figure 4F] FIG. 1 is a side view of a light field display system having a collection surface on the front wall of a room in accordance with one or more embodiments. [Figure 5] FIG. 1 is a block diagram of a light field display system according to one or more embodiments. [Figure 6] 1 is an illustration of an exemplary light field display system for video conferencing, in accordance with one or more embodiments. [Figure 7] 1 is an illustration of an alternative configuration of a video conference space, according to one or more embodiments. [Figure 8A] 1 is an illustration of a light field display system presenting holographic content including holographic video chat participants, according to one or more embodiments. [Figure 8B] 8B is an illustration of the light field display system of FIG. 8A presenting holographic content including images of video chat participants, according to one or more embodiments. [Figure 9] 1 is an illustration of an LF display system presenting holographic content including representations of participants in a group video chat, according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] Light field (LF) display systems provide video communications such as video conferencing, video chat, or pre-recorded video messages. The video communications include holographic content representing remote scenes, such as holographic images of remote participants, holographic images of props, and holographic whiteboards. The terms "video conferencing" and "video chat" are used for convenience to refer to more and less formal communications, respectively, although any features described in connection with one may be provided for the other.

[0007] In various embodiments, a holographic image of a participant at one location is presented to a participant at a different location. The holographic image provides a three-dimensional (3D) representation of the participant that can be viewed without the need for glasses, headsets, or other viewing equipment. When communication is live, a two-way surface that both emits and absorbs light may be used. Thus, there may be a one-to-one correspondence between the gaze directions of participants at both locations, allowing participants to make eye contact as if they were in the same physical space. Furthermore, the presence of a 3D image may help participants notice gestures and / or facial expressions that convey information that may be missed in traditional video communication. Thus, the use of an LF display system may provide participants with the impression that they are located in the same space, even if they are thousands of miles apart.

[0008] In some embodiments, additional holographic imagery may be provided to improve and / or facilitate the communication experience. For example, holographic props, such as product prototypes, may be provided to further facilitate communication as if participants were located in the same space. Similarly, a holographic whiteboard may be provided on which participants can draw, and content may be synchronized between two or more locations. Because the whiteboard is holographic, it need not be constrained to a two-dimensional (2D) surface. In one embodiment, participants may draw in 3D within a region of space (e.g., a box) designated as a virtual whiteboard. As another example, participants may change their own appearance and / or the appearance of other participants. This may include partial changes, such as clothing, hair color, lighting, etc., as well as complete changes, such as representing participants with avatars whose movements are mapped to the movements of the corresponding participants.

[0009] The holographic content presented by the LF display system may also be augmented with other sensory stimuli (e.g., haptics and / or sound). For example, an ultrasound source within the LF display system may project ultrasound pressure waves that generate a volumetric tactile projection. The volumetric tactile projection provides a tactile surface that corresponds to some or all of the projected holographic object. The holographic content may also include additional visual content (i.e., 2D or 3D visual content). Coordination of energy sources to enable a cohesive experience is part of the LF system in implementations with multiple energy sources (i.e., holographic objects that provide the correct tactile and sensory stimuli at any given time). For example, the LF system may include a controller for coordinating the presentation of the holographic content and tactile surfaces.

[0010] In some embodiments, an LF display system may include elements that enable the system to project at least one type of energy and simultaneously sense at least one type of energy. The sensed energy may be used to record how viewers respond to the holographic content. For example, an LF display system may project both holographic objects for viewing and ultrasound for tactile perception, while simultaneously recording image information for viewer (e.g., video conference participants) tracking and other scene analysis. As an example, such a system may project a holographic product prototype that participants can manipulate via touch (e.g., by reaching out and rotating the holographic prototype to view it from different angles), and this interaction with the holographic prototype is recorded by the LF display system. The components of an LF display system that perform environmental energy sensing may be integrated into the display surface or may be dedicated sensors separate from the display surface.

[0011] Light Field Display System Overview FIG. 1 is a diagram 100 of a light field (LF) display module 110 presenting a holographic object 120, according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system presents holographic content, including at least one holographic object, using one or more LF display modules. The LF display system can present the holographic content to one or more viewers. In some embodiments, the LF display system can also augment the holographic content with other sensory content (e.g., touch, sound, smell, temperature, etc.). For example, as described below, the projection of focused ultrasound can generate mid-air haptics that can simulate some or all of the surface of a holographic object. The LF display system includes one or more LF display modules 110 and is described in detail below with respect to FIGS. 2-9.

[0012] LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more viewers (e.g., viewer 140). LF display module 110 includes an energy device layer (e.g., a light-emitting electronic display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). Additionally, LF display module 110 may include an energy relay layer to combine multiple energy sources or detectors together to form a single surface. At a high level, the energy device layer generates energy (e.g., holographic content), which is then directed to a region in space according to one or more four-dimensional (4D) light field functions using the energy waveguide layer. LF display module 110 can also simultaneously project and / or sense one or more types of energy. For example, LF display module 110 may project a holographic image as well as an ultrasound tactile surface into a viewing volume while simultaneously detecting image data from the viewing volume. The operation of the LF display module 110 is described in more detail below with respect to FIGS.

[0013] The LF display module 110 generates a holographic object within the holographic object volume 160 using one or more 4D light field functions (e.g., derived from a plenoptic function). The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Additionally, the holographic object may be multicolor (e.g., full color). The holographic object may be projected in front of a screen plane, behind a screen plane, or separated by a screen plane. The holographic object 120 may be presented to be perceived anywhere within the holographic object volume 160. The holographic object within the holographic object volume 160 may appear to the viewer 140 as floating in space.

[0014] Holographic object volume 160 represents the volume in which a holographic object can be perceived by viewer 140. Holographic object volume 160 can extend in front of the surface of display area 150 (i.e., toward viewer 140) so that the holographic object can be presented in front of the plane of display area 150. Additionally, holographic object volume 160 can extend behind the plane of display area 150 (i.e., away from viewer 140), allowing the holographic object to be presented as if it were behind the plane of display area 150. In other words, holographic object volume 160 can include all light rays that originate (e.g., projected) from display area 150 and can converge to create a holographic object. Here, the light rays can converge to a point in front of, at, or behind the display surface. More simply, holographic object volume 160 encompasses all volumes in which a holographic object can be perceived by a viewer.

[0015] Viewing volume 130 is the volume of space in which a holographic object (e.g., holographic object 120) presented by the LF display system in holographic object volume 160 is fully visible. The holographic object is presented in holographic object volume 160 and can be viewed in viewing volume 130 so as to be indistinguishable from the actual object. The holographic object is formed by projecting light rays identical to those that would be generated from the surface of the object if it were physically present.

[0016] In some cases, the holographic object volume 160 and corresponding viewing volume 130 may be relatively small, designed for a single viewer. In other embodiments, for example, as described in detail below with respect to FIGS. 4 and 6-9, LF display modules may be scaled and / or tiled to create larger holographic object volumes and corresponding viewing volumes that can accommodate a wide range of viewers (e.g., from one to several thousand). The LF display modules presented in this disclosure may be constructed such that the entire surface of the LF display includes holographic imaging optics, with no inactive or dead space and no need for bezels. In these embodiments, the LF display modules may be tiled such that the imaging area is continuous across seams between LF display modules, and the join lines between tiled modules are substantially undetectable using the human eye's visual acuity. Notably, in some configurations, portions of the display surface, not described in detail herein, may not include holographic imaging optics.

[0017] The flexible size and / or shape of viewing volume 130 allows the viewer 140 to be unconstrained within viewing volume 130. For example, viewer 140 can move to different positions within viewing volume 130 and see different views of holographic object 120 from corresponding perspectives. For example, referring to FIG. 1 , viewer 140 is in a first position relative to holographic object 120 such that holographic object 120 appears as a head-on view of the dolphin. Viewer 140 can move to other positions relative to holographic object 120 to see different views of the dolphin. For example, viewer 140 can move to see the left side of the dolphin, the right side of the dolphin, etc., as if viewer 140 were looking at a real dolphin and change the viewer's position relative to the real dolphin to see different views of the dolphin. In some embodiments, holographic object 120 is visible to all viewers within viewing volume 130 who have an unobstructed line of sight (i.e., not blocked by an object / person) to holographic object 120. These viewers may be unconstrained so that they can move around within the viewing volume to see different perspectives of the holographic object 120. Thus, the LF display system can present a holographic object in a way that allows multiple unconstrained viewers to simultaneously view the holographic object in real-world space from different perspectives, as if the holographic object were physically present.

[0018] In contrast, traditional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) generally require each viewer to wear some kind of external device (e.g., 3D glasses, a near-eye display, or a head-mounted display) to view the content. Additionally and / or alternatively, traditional displays may require the viewer to be constrained to a specific viewing position (e.g., in a chair with a fixed position relative to the display). For example, when viewing an object displayed by a stereoscopic display, the viewer always focuses on the display surface rather than the object, and the display always presents only two views of the object that follow the viewer as they attempt to move around the perceived object, causing distortions in the perception of the object. However, in light field displays, viewers of holographic objects presented by LF display systems do not need to wear external devices or be confined to a specific position to view the holographic object. LF display systems present holographic objects in a way that appears to the viewer substantially the same as physical objects appear to the viewer, without requiring special eyewear, glasses, or head-mounted accessories. Furthermore, viewers can view holographic content from any position within the viewing volume.

[0019] In particular, the potential positions of the holographic object within holographic object volume 160 are limited by the size of the volume. To increase the size of holographic object volume 160, the size of display area 150 of LF display module 110 may be increased, and / or multiple LF display modules may be tiled together to form a seamless display surface. A seamless display surface has an effective display area that is larger than the display areas of the individual LF display modules. Some embodiments related to tiling LF display modules are described below with respect to FIGS. 4 and 6-9. As illustrated in FIG. 1, display area 150 is rectangular, resulting in holographic object volume 160 that is pyramidal. In other embodiments, the display area may have some other shape (e.g., hexagonal) that also affects the shape of the corresponding viewing volume.

[0020] Additionally, while the above discussion has focused on presenting holographic object 120 within a portion of holographic object volume 160 that is between LF display module 110 and viewer 140, LF display module 110 can additionally present content to holographic object volume 160 behind the plane of display area 150. For example, LF display module 110 can make display area 150 appear to be the surface of the ocean with holographic object 120 protruding from it. The displayed content may then be such that viewer 140 can see underwater marine life through the displayed surface. Furthermore, LF display systems can generate content that moves seamlessly around holographic object volume 160, including behind and in front of the plane of display area 150.

[0021] 2A is a cross-section 200 of a portion of an LF display module 210 according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a different display area shape than the display area 150. In an exemplary embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 have different components than those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be distributed among components in a manner different from that described herein.

[0022] The display systems described herein exhibit energy emissions that replicate the energy that typically surrounds real-world objects. Here, the emitted energy is directed in a specific direction from every coordinate on the display surface. In other words, various coordinates on the display surface serve as projection locations for the emitted energy. The directed energy from the display surface allows for the convergence of many energy rays, thereby creating a holographic object. For example, in visible light, an LF display projects a very large number of light rays from projection locations that can converge at any point within the holographic object volume, so that from the perspective of a viewer located farther away than the object being projected, these light rays will appear to originate from the surface of a real-world object located in this region of space. In this way, from the viewer's perspective, the LF display generates reflected light rays that appear to originate from the surface of such an object. The viewer's viewpoint can be changed on any particular holographic object, and the viewer will see different views of that holographic object.

[0023] The energy device layer 220 includes one or more electronic displays (e.g., light-emitting displays such as OLEDs) and one or more other energy-projecting and / or energy-receiving devices described herein. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include multiple pixels, each with an individually controlled intensity. Many types of commercial displays can be used within an LF display, such as light-emitting LED and OLED displays.

[0024] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic projection devices generate one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or some combination thereof. An array of ultrasonic pressure waves may be used for volumetric haptics (e.g., at the surface of the holographic object 250). The audio pressure waves may be used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming the holographic object 250 is a dolphin, one or more audio projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the dolphin so that the viewer can touch the holographic object 250, and (2) provide audio content corresponding to sounds made by the dolphin, such as snapping, squeals, and shrill calls. The audio receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audio pressure waves within a localized area of the LF display module 210.

[0025] The energy device layer 220 can also include one or more image sensors. The image sensors may be sensitive to light in the visible light band, and in some cases may be sensitive to light in other bands (e.g., infrared). The image sensors may be, for example, a complementary metal-oxide semiconductor (CMOS) array, a charge-coupled device (CCD), an array of photodetectors, some other sensor that captures light, or some combination thereof. The LF display system can use data captured by one or more image sensors for viewer location tracking.

[0026] In some configurations, the energy relay layer 230 relays energy (e.g., electromagnetic energy, mechanical pressure waves, etc.) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270 and relays energy between the two surfaces. The first surface 265 of each energy relay element may be coupled to one or more energy devices (e.g., electronic displays or acoustic projection devices). The energy relay elements may be composed of, for example, glass, carbon, optical fiber, optical film, plastic, polymer, or some combination thereof. Additionally, in some embodiments, the energy relay elements can adjust the magnification (increase or decrease) of the energy passing between the first surface 265 and the second surface 270. When the relay provides magnification, the relay may take the form of an array of bonded tapered relays, called tapers, where one end of the taper may be substantially larger in area than the opposite end. The larger ends of the tapers can be glued together to form a seamless energy surface 275. One advantage is that space is created at the multiple smaller ends of each taper to accommodate the mechanical envelopes of multiple energy sources, such as multiple display bezels. This extra space allows energy sources to be placed side-by-side on the smaller tapered sides, with each energy source having an active area within the smaller tapered surface that directs energy and is relayed to the larger seamless energy surface. Another advantage of using tapered relays is that the combined seamless energy surface formed by the larger ends of the tapers has no non-imaging dead space. Because there are no borders or bezels, the seamless energy surfaces can be tiled together to form a larger surface that is virtually seamless to the eye.

[0027] The second surfaces of adjacent energy relay elements together form energy plane 275. In some embodiments, the separation between the edges of adjacent energy relay elements is less than the smallest perceptible contour defined, for example, by the visual acuity of a human eye with 20 / 40 vision, such that energy plane 275 is virtually seamless from the perspective of viewer 280 within viewing volume 285.

[0028] In some embodiments, the second surfaces of adjacent energy relay elements are fused together using a process that may include one or more of pressure, heat, and chemical reaction so that there are no seams between them. Additionally, in other embodiments, the array of energy relay elements is formed by molding one side of a continuous block of relay material into an array of small tapered ends, each configured to transport energy from an energy device attached to the small tapered end into a single bonded surface having a large undivided area.

[0029] In some embodiments, one or more energy relay elements exhibit energy localization, such that the efficiency of energy transport in a longitudinal direction substantially perpendicular to surfaces 265 and 270 is much higher than the efficiency of transport in a perpendicular cross-section, and the energy density is highly localized in this cross-section as an energy wave propagates between surfaces 265 and 270. This energy localization allows energy distributions, such as images, to be efficiently relayed between these surfaces without significant loss of resolution.

[0030] The energy waveguide layer 240 uses waveguide elements within the energy waveguide layer 240 to direct energy from locations (e.g., coordinates) on the energy plane 275 into specific energy propagation paths from the display surface outward into the holographic viewing volume 285. The energy propagation paths are defined by at least two angular dimensions determined by the coordinate position of the energy plane relative to the waveguide. The waveguide is associated with spatial 2D coordinates. These four coordinates together form a four-dimensional (4D) energy field. As an example, for electromagnetic energy, the waveguide elements within the energy waveguide layer 240 direct light from locations on the seamless energy plane 275 along different propagation directions through the viewing volume 285. In various examples, the light is directed according to a 4D light field function to form the holographic object 250 within the holographic object volume 255.

[0031] Each waveguide element in the energy waveguide layer 240 may be, for example, a lenslet composed of one or more elements. In some configurations, the lenslet may be a positive lens. The positive lens may have a spherical, aspherical, or freeform surface profile. Additionally, in some embodiments, some or all of the waveguide elements may include one or more additional optical components. The additional optical component may be, for example, an energy suppression structure such as a baffle, a positive lens, a negative lens, a spherical lens, an aspherical lens, a freeform lens, a liquid crystal lens, a liquid lens, a refractive element, a diffractive element, or some combination thereof. In some embodiments, at least one of the lenslet and / or additional optical components may also dynamically adjust the refractive power. For example, the lenslet may be a liquid crystal lens or a liquid lens. The dynamic adjustment of the surface profile, the lenslet, and / or the at least one additional optical component may provide additional directional control of the light projected from the waveguide element.

[0032] In the illustrative example, holographic object volume 255 of the LF display has a boundary formed by light rays 256 and 257, although it may be formed by other light rays. Holographic object volume 255 is a continuous volume that extends both in front of (i.e., toward viewer 280) and behind (i.e., away from viewer 280) energy waveguide layer 240. In the illustrative example, light rays 256 and 257 are projected from opposite edges of LF display module 210 at the largest angle relative to the normal to display surface 277 that can be perceived by a user, although these may be other projected light rays. These light rays define the field of view of the display and therefore define the boundary of holographic viewing volume 285. In some cases, these light rays define a holographic viewing volume (e.g., an ideal viewing volume) in which the entire display can be observed without vignetting. As the field of view of the display increases, the convergence point of light rays 256 and 257 becomes closer to the display. Thus, a display with a wider field of view allows viewer 280 to view the entire display at a closer viewing distance. Additionally, light rays 256 and 257 may form an ideal holographic object volume. A holographic object presented within the ideal holographic object volume may be seen anywhere within viewing volume 285.

[0033] In some examples, a holographic object may be presented in only a portion of viewing volume 285. In other words, the holographic object volume may be divided into any number of viewing sub-volumes (e.g., viewing sub-volume 290). Additionally, a holographic object may be projected outside holographic object volume 255. For example, holographic object 251 is presented outside holographic object volume 255. Because holographic object 251 is presented outside holographic object volume 255, it may not be visible from all positions within viewing volume 285. For example, holographic object 251 may be visible from positions within viewing sub-volume 290, but not from the position of viewer 280.

[0034] For example, to illustrate viewing holographic content from different viewing sub-volumes, turn to FIG. 2B . FIG. 2B illustrates a cross-section 200 of a portion of an LF display module, according to one or more embodiments. The cross-section of FIG. 2B is the same as the cross-section of FIG. 2A . However, FIG. 2B illustrates a different set of light rays projected from LF display module 210. Light rays 256 and 257 still form holographic object volume 255 and viewing volume 285. However, as shown, light rays projected from the top of LF display module 210 and the bottom of LF display module 210 overlap to form various viewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D) within viewing volume 285. A viewer in a first viewing sub-volume (e.g., 290A) may be able to perceive holographic content presented in holographic object volume 255 that is not perceptible by viewers in the other viewing sub-volumes (e.g., 290B, 290C, and 290D).

[0035] More simply, as illustrated in FIG. 2A , holographic object volume 255 is a volume in which a holographic object can be presented by an LF display system such that the holographic object can be perceived by a viewer (e.g., viewer 280) within viewing volume 285. Thus, viewing volume 285 is an example of an ideal viewing volume, and holographic object volume 255 is an example of an ideal object volume. However, in various configurations, a viewer can perceive a holographic object presented by LF display system 200 within other exemplary holographic object volumes. More generally, "eyeline guidelines" apply when viewing holographic content projected from an LF display module. Eyeline guidelines assert that the line formed by the viewer's eye position and the holographic object being viewed must intersect the LF display surface.

[0036] When viewing the holographic content presented by LF display module 210, each eye of viewer 280 will see a different perspective of holographic object 250 because the holographic content is presented according to a 4D light field function. Furthermore, as viewer 280 moves within viewing volume 285, the viewer will see different perspectives of holographic object 250 as would other viewers within viewing volume 285. As will be appreciated by those skilled in the art, 4D light field functions are known in the art and will not be described in further detail herein.

[0037] As described in more detail herein, in some embodiments, the LF display can project two or more types of energy. For example, the LF display can project two types of energy, such as mechanical and electromagnetic energy. In this configuration, the energy relay layer 230 can include two separate energy relays interleaved together at the energy plane 275 but separated so that energy is relayed to two different energy device layers 220. Here, one relay can be configured to transport electromagnetic energy and the other relay can be configured to transport mechanical energy. In some embodiments, the mechanical energy can be projected from a location between electromagnetic waveguide elements on the energy waveguide layer 240, which helps to form a structure that inhibits light from being transported from one electromagnetic waveguide element to the other. In some embodiments, the energy waveguide layer 240 can also include waveguide elements that transport focused ultrasound along specific propagation paths according to display instructions from the controller.

[0038] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy plane 275 is an emission plane formed using one or more adjacent electronic displays in the energy device layer 220. And, in some embodiments without an energy relay layer, the separation between the edges of adjacent electronic displays is less than the smallest perceptible contour defined by the visual acuity of a human eye with 20 / 40 vision, so that the energy plane is virtually seamless from the perspective of a viewer 280 within the viewing volume 285.

[0039] LF Display Module FIG. 3A is a perspective view of an LF display module 300A according to one or more embodiments. LF display module 300A may be LF display module 110 and / or LF display module 210. In other embodiments, LF display module 300A may be some other LF display module. In an exemplary embodiment, LF display module 300A includes an energy device layer 310, an energy relay layer 320, and an energy waveguide layer 330. LF display module 300A is configured to present holographic content from display surface 365 as described herein. For convenience, display surface 365 is illustrated as a dashed outline on frame 390 of LF display module 300A, but more precisely, it is the surface directly in front of the waveguide element bounded by the inner rim of frame 390. Display surface 365 includes multiple projection positions onto which energy may be projected. Some embodiments of LF display module 300A have different components than those described herein. For example, in some embodiments, the LF display module 300A does not include the energy relay layer 320. Similarly, functionality may be distributed among components in ways different from those described herein.

[0040] Energy device layer 310 is one embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three are visible in the figure). The energy devices 340 may all be of the same type (e.g., all electronic displays) or may include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0041] Energy relay layer 320 is one embodiment of energy relay layer 230. Energy relay layer 320 includes four energy relay devices 350 (three are visible in the figure). The energy relay devices 350 may all relay the same type of energy (e.g., light) or may relay one or more different types (e.g., light and sound). Each of the relay devices 350 includes a first surface and a second surface, and the second surfaces of the energy relay devices 350 are arranged to form a single, seamless energy surface 360. In an exemplary embodiment, each of the energy relay devices 350 is tapered such that the first surface has a smaller surface area than the second surface, allowing the mechanical envelope of the energy device 340 to be housed at the smaller end of the taper. This allows the seamless energy surface to be boundaryless, as the entire area may project energy. This means that this seamless energy surface can be tiled by placing multiple instances of the LF display module 300A together without dead space or bezels so that the entire combined surface is seamless. In other embodiments, the first surface and the second surface have the same surface area.

[0042] Energy waveguide layer 330 is one embodiment of energy waveguide layer 240. Energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIG. 2 , energy waveguide layer 330 is configured to direct energy from seamless energy plane 360 along specific propagation paths according to a 4D light field function to form a holographic object. Note that in the exemplary embodiment, energy waveguide layer 330 is bounded by frame 390. In other embodiments, frame 390 is absent and / or has a reduced thickness. Removing or reducing the thickness of frame 390 may facilitate tethering LF display module 300A with additional LF display modules.

[0043] Note that in the exemplary embodiment, the seamless energy surface 360 and the energy waveguide layer 330 are planar. In alternative embodiments not shown, the seamless energy surface 360 and the energy waveguide layer 330 may be curved in one or more dimensions.

[0044] The LF display module 300A may be configured with an additional energy source present on the surface of the seamless energy surface, allowing for the projection of an energy field in addition to the light field. In one embodiment, an acoustic energy field may be projected from an electrostatic speaker (not shown) mounted at any number of locations on the seamless energy surface 360. Additionally, the electrostatic speaker of the LF display module 300A may be positioned within the light field display module 300A so that the dual energy surface simultaneously projects a sound field and holographic content. For example, the electrostatic speaker may be formed of one or more diaphragm elements that are transparent to some wavelengths of electromagnetic energy and driven with one or more conductive elements (e.g., planes sandwiching one or more diaphragm elements). The electrostatic speaker may be mounted on the seamless energy surface 360 so that the diaphragm elements cover some of the waveguide elements. The speaker's conductive electrodes may be collocated with a structure designed to suppress light transmission between the electromagnetic waveguides and / or may be located between the electromagnetic waveguide elements (e.g., in the frame 390). In various configurations, the speakers can project multiple sources of focused ultrasonic energy that produce audible sound and / or a tactile surface.

[0045] In some configurations, the energy devices 340 may sense energy. For example, the energy devices may be microphones, light sensors, acoustic transducers, etc. Thus, the energy relay devices may also relay energy from the seamless energy surface 360 to the energy device layer 310. That is, the seamless energy surface 360 of the LF display module forms a bidirectional energy surface when the energy devices and the energy relay devices 340 are configured to simultaneously emit and sense energy (e.g., emit a light field and sense sound).

[0046] More broadly, the energy device 340 of the LF display module 340 can be either an energy source or an energy sensor. The LF display module 300A can include various types of energy devices that function as energy sources and / or energy sensors to facilitate the projection of high-quality holographic content to a user. Other sources and / or sensors include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, and the like. Many other sensors or sources are possible. Additionally, LF display modules can be tiled so that they can form assemblies that project and sense multiple types of energy from a large, collective, seamless energy surface.

[0047] In various embodiments of the LF display module 300A, the seamless energy surface 360 may have various surface portions, each configured to project and / or emit a particular type of energy. For example, if the seamless energy surface is a dual energy surface, the seamless energy surface 360 includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy may be located on the seamless energy surface 360 between electromagnetic waveguide elements and / or may be co-located with structures designed to inhibit optical transmission between the electromagnetic waveguide elements. In examples where the seamless energy surface is a bidirectional energy surface, the energy relay layer 320 may include two types of energy relay devices interleaved in the seamless energy surface 360. In various embodiments, the seamless energy surface 360 may be configured so that the surface portions under any particular waveguide element 370 are all energy sources, all energy sensors, or a mix of energy sources and energy sensors.

[0048] 3B is a cross-sectional view of an LF display module 300B including interleaved energy relay devices, according to one or more embodiments. Energy relay device 350A transfers energy between an energy relay first surface 345A connected to energy device 340A and a seamless energy surface 360. Energy relay 350B transfers energy between an energy relay first surface 345B connected to energy device 340B and the seamless energy surface 360. Both relay devices are interleaved in an interleaved energy relay device 352 connected to the seamless energy surface 360. In this configuration, surface 360 includes interleaved energy locations of both energy devices 340A and 340B, which may be energy sources or energy sensors. Thus, LF display module 300B can be configured either as a dual energy projection device for projecting multiple types of energy or as a bidirectional energy device for simultaneously projecting one type of energy and sensing another type of energy. LF display module 300B may be LF display module 110 and / or LF display module 210. In other embodiments, LF display module 300B may be some other LF display module.

[0049] The LF display module 300B includes many components configured similarly to those of the LF display module 300A of FIG. 3A. For example, in an exemplary embodiment, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330, which include at least the same functionality as described with respect to FIG. 3A. Additionally, the LF display module 300B can present and / or receive energy from a display surface 365. Notably, the components of the LF display module 300B are connected and / or oriented differently than the components of the LF display module 300A of FIG. 3A. Some embodiments of the LF display module 300B have different components than those described herein. Similarly, functionality may be distributed among the components in a manner different from that described herein. FIG. 3B illustrates the design of a single LF display module 300B that can be tiled to create a dual-energy projection surface or a bidirectional energy surface with a larger area.

[0050] In one embodiment, LF display module 300B is an LF display module of a bidirectional LF display system. A bidirectional LF display system can simultaneously project energy and sense energy from display surface 365. Seamless energy surface 360 includes both energy projection locations and energy sensing locations closely interleaved on seamless energy surface 360. Thus, in the example of FIG. 3B, energy relay layer 320 is configured differently than the energy relay layer of FIG. 3A. For convenience, the energy relay layer of LF display module 300B is referred to herein as an “interleaved energy relay layer.”

[0051] The interleaved energy relay layer 320 includes two legs, a first energy relay device 350A and a second energy relay device 350B. Each leg is illustrated as a lightly shaded region in FIG. 3B. Each leg is made of a flexible relay material and can be formed with a length sufficient for use with energy devices of various sizes and shapes. In some regions of the interleaved energy relay layer, the two legs are tightly interleaved together as they approach a seamless energy plane 360. In the illustrative example, the interleaved energy relay device 352 is illustrated as a darkly shaded region.

[0052] The energy relay devices are configured to relay energy between different energy devices while being interleaved in a seamless energy plane 360. The energy devices are in the energy device layer 310. As illustrated, energy device 340A is connected to energy relay device 350A, and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device may be an energy source or an energy sensor.

[0053] Energy waveguide layer 330 includes waveguide elements 370 for directing energy waves from seamless energy surface 360 along a projected path toward a series of convergence points. In this example, holographic object 380 is formed at the series of convergence points. Notably, as illustrated, the energy convergence at holographic object 380 occurs on the viewer-side (i.e., front) of display surface 365. However, in other examples, the energy convergence may be anywhere within the holographic object volume, extending both in front of and behind display surface 365. Waveguide elements 370 can simultaneously direct incoming energy to energy devices (e.g., energy sensors), as described below.

[0054] In one exemplary embodiment of LF display module 300B, a light-emitting display is used as the energy source (e.g., energy device 340A) and an image sensor is used as the energy sensor (e.g., energy device 340B). In this manner, LF display module 300B can simultaneously project holographic content and detect light from the volume in front of display surface 365. In this manner, this embodiment of LF display module 300B functions as both an LF display and an LF sensor.

[0055] In one embodiment, the LF display module 300B is configured to project a light field from a projection location on the display surface in front of the display surface and simultaneously capture a light field from in front of the display surface at the projection location. In this embodiment, the energy relay device 350A connects a first set of locations in the seamless energy surface 360 positioned below the waveguide element 370 to the energy device 340A. In one example, the energy device 340A is an emissive display having an array of source pixels. The energy relay device 340B connects a second set of locations in the seamless energy surface 360 positioned below the waveguide element 370 to the energy device 340B. In one example, the energy device 340B is an image sensor having an array of sensor pixels. The LF display module 300B can be configured such that the locations in the seamless energy surface 365 below a particular waveguide element 370 are all emissive display locations, all image sensor locations, or some combination of these locations. In other embodiments, the bidirectional energy surface can project and receive various other forms of energy.

[0056] In another exemplary embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, in one embodiment, the energy device 340A is a light-emitting display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. Thus, both light and sound can be projected from various positions on the seamless energy surface 360. In this configuration, the energy relay device 350A connects the energy device 340A to the seamless energy surface 360 and relays the electromagnetic energy. The energy relay device is configured with properties that allow for efficient transport of electromagnetic energy (e.g., changing the refractive index). The energy relay device 350B connects the energy device 340B to the seamless energy surface 360 and relays the mechanical energy. The energy relay device 350B is configured with properties for efficient transport of ultrasonic energy (e.g., distribution of materials with different acoustic impedances). In some embodiments, mechanical energy can be projected from positions between the waveguide elements 370 on the energy waveguide layer 330. The locations that project mechanical energy can form structures that act to inhibit light from being transported from one electromagnetic waveguide element to another. In one example, a spatially separated array of locations that project ultrasonic mechanical energy can be configured to create three-dimensional haptic shapes and surfaces in space. The surfaces can coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delay and amplitude variation across the array can assist in creating the haptic shapes.

[0057] In various embodiments, the LF display module 300B with interleaved energy relay devices may include multiple energy device layers, with each energy device layer including a particular type of energy device. In these examples, the energy relay layers are configured to relay the appropriate type of energy between the seamless energy plane 360 and the energy device layer 310.

[0058] Tiled LF Display Module 4A is a perspective view of a portion of an LF display system 400 tiled in two dimensions to form a single-sided seamless surface environment, according to one or more embodiments. LF display system 400 includes multiple LF display modules tiled to form an array 410. More specifically, each small square in array 410 represents a tiled LF display module 412. LF display module 412 may be the same as LF display module 300A or 300B. Array 410 can, for example, cover some or all of a room's surface (e.g., a wall). LF arrays can also cover other surfaces, such as tabletops, cubicle dividers, etc.

[0059] Array 410 can project one or more holographic objects. For example, in an exemplary embodiment, array 410 projects holographic object 420 and holographic object 422. Tiling of LF display module 412 not only allows for a much larger viewing volume, but also allows objects to be projected farther from array 410. For example, in an exemplary embodiment, the viewing volume is substantially the entire area in front of and behind array 410, rather than a localized volume in front of (and behind) LF display module 412.

[0060] In some embodiments, LF display system 400 presents holographic object 420 to viewer 430 and viewer 434. Viewer 430 and viewer 434 receive different perspectives of holographic object 420. For example, viewer 430 is presented with a straight-on view of holographic object 420, while viewer 434 is presented with a more oblique view of holographic object 420. As viewer 430 and / or viewer 434 move, they are presented with different perspectives of holographic object 420. This allows the viewer to visually interact with the holographic object by moving relative to the holographic object. For example, as viewer 430 walks around holographic object 420, viewer 430 sees different sides of holographic object 420, as long as holographic object 420 remains within the holographic object volume of array 410. Thus, viewer 430 and viewer 434 can simultaneously view holographic object 420 in real-world space as if holographic object 420 were actually there. Additionally, because holographic object 420 appears to the viewer in much the same way that a physical object appears, viewer 430 and viewer 434 do not need to wear an external device to view holographic object 420. Additionally, here, holographic object 422 is illustrated behind the array because the array's viewing volume extends behind the surface of the array. In this manner, holographic object 422 can be presented to viewer 430 and / or viewer 434.

[0061] In some embodiments, LF display system 400 may include a tracking system that tracks the positions of viewer 430 and viewer 434. In some embodiments, the tracked positions are the viewer's positions. In other embodiments, the tracked positions are the viewer's eye positions. Eye position tracking differs from gaze tracking, which tracks where the eyes are looking (e.g., uses orientation to determine gaze position). The eyes of viewer 430 and viewer 434 are in different positions.

[0062] In various configurations, LF display system 400 may include one or more tracking systems. For example, in the exemplary embodiment of FIG. 4A , LF display system includes tracking system 440 that is external to array 410. Here, the tracking system may be a camera system coupled to array 410. External tracking systems are described in more detail with respect to FIG. 5 . In other exemplary embodiments, a tracking system may be incorporated into array 410, as described herein. For example, an energy device (e.g., energy device 340) of one or more LF display modules 412 including a bidirectional energy surface included in array 410 may be configured to capture images of a viewer in front of array 410. In either case, the tracking system of LF display system 400 determines tracking information about a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by array 410.

[0063] The tracking information represents the position of the viewer or the position of a portion of the viewer (e.g., one or both of the viewer's eyes, or the viewer's limbs) in space (e.g., relative to the tracking system). The tracking system can use any number of depth determination techniques to determine the tracking information. Depth determination techniques may include, for example, structured light, time-of-flight, stereo imaging, some other depth determination technique, or some combination thereof. The tracking system may include various systems configured to determine the tracking information. For example, the tracking system may include one or more infrared sources (e.g., structured light sources), one or more image sensors capable of capturing images in infrared (e.g., red-blue-green-infrared cameras), and a processor that executes a tracking algorithm. The tracking system can determine the viewer's position using depth estimation techniques. In some embodiments, the LF display system 400 generates a holographic object based on the tracked position, movement, or gesture of the viewer 430 and / or the viewer 434, as described herein. For example, the LF display system 400 may generate a holographic object in response to a viewer coming within a threshold distance and / or a specific position of the array 410.

[0064] Based in part on the tracking information, LF display system 400 can present one or more holographic objects customized for each viewer. For example, viewer 430 may be presented with holographic object 420, but not with holographic object 422. Similarly, viewer 434 may be presented with holographic object 422, but not with holographic object 420. For example, LF display system 400 tracks the positions of each of viewer 430 and viewer 434. LF display system 400 determines the viewpoint of the holographic object that should be seen by the viewer based on the viewer's position relative to where the holographic object should be presented. LF display system 400 selectively projects light from specific pixels that correspond to the determined viewpoint. Thus, viewer 434 and viewer 430 can potentially have completely different experiences simultaneously. In other words, LF display system 400 can present holographic content in viewing sub-volumes of a viewing volume (i.e., similar to viewing sub-volumes 290A, 290B, 290C, and 290D shown in FIG. 2B ). For example, as illustrated, LF display system 400 can track the position of viewer 430, so that LF display system 400 can present space content (e.g., holographic object 420) in a viewing sub-volume surrounding viewer 430 and safari content (e.g., holographic object 422) in a viewing sub-volume surrounding viewer 434. In contrast, conventional systems would have to use individual headsets to provide a similar experience.

[0065] In some embodiments, LF display system 400 may include one or more sensory feedback systems. The sensory feedback system provides other sensory stimuli (e.g., touch, sound, or smell) that augment holographic objects 420 and 422. For example, in the exemplary embodiment of FIG. 4A , LF display system 400 includes a sensory feedback system 442 external to array 410. In one example, sensory feedback system 442 may be an electrostatic speaker coupled to array 410. External sensory feedback systems are described in more detail with respect to FIG. 5 . In other exemplary embodiments, a sensory feedback system may be incorporated into array 410, as described herein. For example, an energy device (e.g., energy device 340A of FIG. 3B ) of an LF display module 412 included in array 410 may be configured to project ultrasonic energy to a viewer in front of the array and / or receive image information from a viewer in front of the array. In either case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing the holographic content presented by array 410 (e.g., holographic object 420 and / or holographic object 422).

[0066] The LF display system 400 may include a sensory feedback system 442 that includes one or more acoustic projection devices external to the array. Alternatively, or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410, as described herein. The acoustic projection device may consist of an array of ultrasound sources configured to project a volumetric tactile surface. In some embodiments, the tactile surface may coincide with the holographic object on one or more surfaces of the holographic object (e.g., on the surface of the holographic object 420) when a portion of the viewer comes within a threshold distance of the one or more surfaces. The volumetric tactile sensation may allow the user to touch and feel the surface of the holographic object. Multiple acoustic projection devices may project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Thus, ultrasound pressure waves and / or audible pressure waves can serve to complement the holographic object.

[0067] In various embodiments, the LF display system 400 can provide other sensory stimuli based in part on the viewer's tracked position. For example, the holographic object 422 illustrated in FIG. 4A is a lion, and the LF display system 400 can cause the holographic object 422 to roar both visually (i.e., the holographic object 422 appears to roar) and audibly (i.e., one or more sound projection devices project pressure waves that the viewer 430 perceives as a lion's roar emanating from the holographic object 422).

[0068] Note that in the exemplary configuration, the holographic viewing volume may be limited in a manner similar to the viewing volume 285 of the LF display system 200 of Figure 2. This can limit the amount of perceived immersion a viewer will experience with a single wall display unit. One way to address this is to use multiple LF display modules that are tiled along multiple sides, as described below with respect to Figures 4B-4F.

[0069] FIG. 4B is a perspective view of a portion of an LF display system 402 in a multi-sided seamless surface environment, according to one or more embodiments. LF display system 402 is substantially similar to LF display system 400, except that multiple LF display modules are tiled to create the multi-sided seamless surface environment. More specifically, the LF display modules are tiled to form an array that is a six-sided collective seamless surface environment. In FIG. 4B, multiple LF display modules cover all walls, ceiling, and floor of a room. In other embodiments, multiple LF display modules may cover some, but not all, of the walls, floor, ceiling, or some combination thereof. In other embodiments, multiple LF display modules are tiled to form some other collective seamless surface. For example, a wall may be curved to form a cylindrical collective energy environment. Additionally, as described below with respect to FIGS. 6-9, in some embodiments, LF display modules may be tiled to form a surface (e.g., a wall) within a conference room or office.

[0070] LF display system 402 can project one or more holographic objects. For example, in an exemplary embodiment, LF display system 402 projects holographic object 420 into an area surrounded by a six-sided aggregated seamless surface environment. In this example, the viewing volume of the LF display system is also contained within the six-sided aggregated seamless surface environment. Note that in the exemplary configuration, viewer 434 may be positioned between holographic object 420 and LF display module 414, which is projecting the energy (e.g., light and / or pressure waves) used to form holographic object 420. Thus, the positioning of viewer 434 may prevent viewer 430 from perceiving holographic object 420 formed from energy from LF display module 414. However, in the exemplary configuration, there is at least one other LF display module, such as LF display module 416, that is unobstructed (e.g., by viewer 434) and that projects energy to form holographic object 420 and can be observed by viewer 430. In this way, viewer occlusion in the space may obscure portions of the holographic projection, but this effect is much smaller than if a holographic display panel were present on only one side of the volume. Holographic object 422 is illustrated "outside" the walls of the six-sided aggregated seamless surface environment because the holographic object volume extends behind the aggregated surfaces. Viewer 430 and / or viewer 434 can therefore perceive holographic object 422 as "outside" the enclosed six-sided environment, which the viewer can move throughout.

[0071] 4A, in some embodiments, the LF display system 402 may actively track the viewer's position and dynamically direct different LF display modules to present holographic content based on the tracked position. Thus, a multi-surface configuration may provide a more robust environment (e.g., compared to FIG. 4A) for presenting holographic objects, where an unconstrained viewer can freely move throughout an area surrounded by the multi-surface seamless surface environment.

[0072] In particular, various LF display systems may have different configurations. Furthermore, each configuration may have a particular orientation of the surfaces that together form a seamless display surface ("aggregate surface"). That is, the LF display modules of an LF display system may be tiled to form various aggregate surfaces. For example, in FIG. 4B, LF display system 402 includes LF display modules tiled to form a six-sided aggregate surface that approximates the walls of a room. In some other examples, the aggregate surface may occur only on a portion of a surface (e.g., half of a wall) rather than the entire surface (e.g., the entire wall). Several examples are described herein.

[0073] In some configurations, the collecting surface of an LF display system may include a collecting surface configured to project energy toward a localized viewing volume. Projecting energy toward a localized viewing volume may enable a higher quality viewing experience, for example, by increasing the density of projected energy within a particular viewing volume, widening the FOV of a viewer within that volume, and moving the viewing volume closer to the display surface.

[0074] For example, Figure 4C shows a top view of an LF display system 450A having a collective surface in a "wing-shaped" configuration. In this example, the LF display system 450A is located in a room having a front wall 452, a rear wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown). The first side wall 456, the second side wall 458, the rear wall 454, the floor, and the ceiling are all orthogonal. The LF display system 450A includes LF display modules tiled to form a collective surface 460 that covers the front wall. The front wall 452, and therefore the collecting surface 460, includes three portions: (i) a first portion 462 (i.e., a central surface) that is generally parallel to the rear wall 454; (ii) a second portion 464 (i.e., a first side surface) that connects the first portion 462 to the first side wall 456 and is angled to project energy toward the center of the room; and (iii) a third portion 466 (i.e., a second side surface) that connects the first portion 462 to the second side wall 458 and is angled to project energy toward the center of the room. The first portion is a vertical plane within the room, having horizontal and vertical axes. The second and third portions are angled toward the center of the room along the horizontal axis.

[0075] In this example, the viewing volume 468A of the LF display system 450A is in the center of the room and is partially surrounded by three portions of the collecting surface 460. The collecting surfaces that at least partially surround the viewer (the "surrounding surfaces") increase the viewer's immersive experience.

[0076] For illustrative purposes, consider a collective surface having only a central surface. Referring to FIG. 2A, rays projected from either edge of the display surface create an ideal holographic volume and an ideal viewing volume, as described above. Now, consider, for example, a case in which the central surface includes two side surfaces angled toward the viewer. In this case, rays 256 and 257 would be projected at a larger angle from the normal to the central surface. Thus, the field of view of the viewing volume would be enlarged. Similarly, the holographic viewing volume would be closer to the display surface. Additionally, a holographic object projected at a fixed distance from the display surface would be closer to the viewing volume because the second and third portions are tilted closer to the viewing volume.

[0077] Simply put, a display surface with only a central surface has a planar field of view, a planar threshold separation between the (central) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer expands the field of view relative to the planar field of view, reduces the separation between the display surface and the viewing volume relative to the planar separation, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further expands the field of view, reduces the separation, and increases the proximity. In other words, the angled placement of the side surfaces enhances the viewer's immersive experience. Additionally, deflection optics can be used to optimize the size and position of the viewing volume for LF display parameters (e.g., dimensions and FOV).

[0078] Returning to FIG. 4D , in a similar example, FIG. 4D shows a side view of an LF display system 450B having a collective surface in a “tilted” configuration. In this example, the LF display system 450B is located in a room having a front wall 452, a rear wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. The first side wall, the second side wall, the rear wall 454, the floor 474, and the ceiling 472 are all orthogonal. The LF display system 450B includes LF display modules tiled to form a collective surface 460 that covers the front wall. The front wall 452, and therefore the collecting surface 460, includes three portions: (i) a first portion 462 (i.e., a central surface) that is generally parallel to the rear wall 454; (ii) a second portion 464 (i.e., a first side surface) that connects the first portion 462 to the ceiling 472 and is angled to project energy toward the center of the room; and (iii) a third portion 464 (i.e., a second side surface) that connects the first portion 462 to the floor 474 and is angled to project energy toward the center of the room. The first portion is a vertical plane within the room, having horizontal and vertical axes. The second and third portions are angled toward the center of the room along the vertical axis.

[0079] In this example, the viewing volume 468B of the LF display system 450B is located in the center of a room and is partially surrounded by three portions of the collecting surface 460. Similar to the configuration shown in FIG. 4C, two side portions (e.g., second portion 464 and third portion 466) are angled to surround the viewer and form a surrounding surface. The surrounding surface expands the viewing FOV from any viewer's viewpoint within the holographic viewing volume 468B. Additionally, the surrounding surface allows the viewing volume 468B to be closer to the display surface so that projected objects appear closer. In other words, the angled arrangement of the side surfaces expands the field of view, reduces separation, and increases the proximity of the collecting surface, thereby enhancing the viewer's immersive experience. Furthermore, as described below, deflection optics can be used to optimize the size and position of the viewing volume 468B.

[0080] The tilted configuration of the side portions of collecting surface 460 allows holographic content to be presented closer to viewing volume 468B than if third portion 466 were not tilted. For example, the lower extremities (such as legs) of a character presented on a tilted LF display system may appear closer and more realistic than if a LF display system with a flat front wall were used.

[0081] Additionally, the configuration of the LF display system and the environment in which it is located may inform the shape and location of the viewing volume and viewing sub-volumes.

[0082] 4E, for example, illustrates a top view of an LF display system 450C having a collection surface 460 on a room's front wall 452. In this example, an LF display system 450D is located in a room having a front wall 452, a back wall 454, a first side wall 456, a second side wall 458, a ceiling (not shown), and a floor (not shown).

[0083] LF display system 450C projects various light rays from collecting surface 460. From each location on the display surface, light rays are projected over a range of angles centered on the viewing volume. Light rays projected from the left side of collecting surface 460 have horizontal angular range 481, light rays projected from the right side of collecting surface 460 have horizontal angular range 482, and light rays projected from the center of collecting surface 460 have horizontal angular range 483. Light rays projected between these points can have intermediate values in the angular range. In this way, viewing volume 468C is created by having a gradient deflection angle for the projected light rays across the display surface. Furthermore, this configuration avoids wasting display resolution when projecting light rays onto side walls 456 and 458.

[0084] FIG. 4F illustrates a side view of an LF display system 450D having a collecting surface 460 on the front wall 452 of a room. In this example, the LF display system 450E is located within a room having a front wall 452, a rear wall 454, a first side wall (not shown), a second side wall (not shown), a ceiling 472, and a floor 474. In this example, the floor is stepped, with each step increasing in height as one moves from the front wall to the rear wall. Here, each step of the floor includes a viewing subvolume (e.g., display subvolumes 470A and 470B). The stepped floor allows for non-overlapping viewing subvolumes. That is, each viewing subvolume has a line of sight from the viewing subvolume to the collecting surface 460 that does not pass through another viewing subvolume. In other words, this orientation creates a "stadium seating" effect, with the vertical offset between the steps allowing each step to "see through" the viewing subvolumes of the other steps. LF display systems that include non-overlapping viewing sub-volumes can provide a higher quality viewing experience than LF display systems with overlapping viewing volumes. For example, in the configuration shown in Figure 4F, different holographic content can be projected to viewers in viewing sub-volumes 470A and 470B.

[0085] Controlling the LF display system FIG. 5 is a block diagram of an LF display system 500 according to one or more embodiments. The LF display system 500 comprises an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display modules 512 may include a source / sensor system 514 that includes an integrated energy source and / or energy sensors that project and / or sense other types of energy. The controller 520 includes a data store 522, a network interface 524, and an LF processing engine 530. The controller 520 may also include a tracking module 526 and a viewer profiling module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. The LF display systems described in the context of FIGS. 1-4 are embodiments of the LF display system 500. In other embodiments, the LF display system 500 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or different entities in ways different from those described herein. Applications of LF display system 500 are also described in more detail below with respect to Figures 6-9.

[0086] The LF display assembly 510 provides holographic content within a holographic object volume that is visible to a viewer located within the viewing volume. The LF display assembly 510 can provide the holographic content by executing display instructions received from the controller 520. The holographic content may include one or more holographic objects projected in front of a collecting surface, onto the LF display assembly 510, behind the collecting surface of the LF display assembly 510, or some combination thereof. Generation of display instructions using the controller 520 is described in more detail below.

[0087] The LF display assembly 510 provides holographic content (e.g., participant images, participant avatars, holographic objects, and / or other sensory content) using one or more LF display modules included in the LF display assembly 510 (e.g., any of the LF display module 110, the LF display system 200, and the LF display module 300). For convenience, the one or more LF display modules may be referred to herein as LF display modules 512. The LF display modules 512 can be tiled to form the LF display assembly 510. The LF display modules 512 can be structured as a variety of seamless surface environments (e.g., single surface, multi-surface, cinema wall, curved surface, etc.). That is, the tiled LF display modules form a collective surface. As described above, the LF display module 512 includes an energy device layer (e.g., the energy device layer 220) and an energy waveguide layer (e.g., the energy waveguide layer 240) that present the holographic content. The LF display module 512 may also include an energy relay layer (e.g., energy relay layer 230) that transfers energy between the energy device layer and the energy waveguide layer when presenting holographic content.

[0088] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing, as described above. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and integrated energy sensing system of the LF display module 512 may be collectively referred to herein as a source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 such that the source / sensor system 514 shares the same seamless energy surface as the LF display module 512. In other words, the collective surface of the LF display assembly 510 includes the functionality of both the LF display module 512 and the source / sensor module 514. That is, an LF assembly 510 including an LF display module 512 with a source / sensor system 514 can project energy and / or sense energy while simultaneously projecting a light field. For example, the LF display assembly 510 may include an LF display module 512 and a source / sensor system 514 configured as a dual energy plane or a bidirectional energy plane as described above.

[0089] In some embodiments, the LF display system 500 uses a sensory feedback system 570 to augment the generated holographic content with other sensory content (e.g., cooperative touch, sound, or smell). The sensory feedback system 570 can augment the projection of the holographic content by executing display instructions received from the controller 520. Generally, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., the sensory feedback system 442) external to the LF display assembly 510. Some exemplary sensory feedback devices may include cooperative sound projection and reception devices, aroma projection devices, temperature adjustment devices, force actuation devices, pressure sensors, transducers, etc. In some cases, the sensory feedback system 570 may have similar functionality as the light field display assembly 510, or vice versa. For example, both the sensory feedback system 570 and the light field display assembly 510 may be configured to generate sound fields. As another example, the sensory feedback system 570 may be configured to generate a tactile surface, while the light field display assembly 510 is not.

[0090] To illustrate, in one exemplary embodiment of light field display system 500, sensory feedback system 570 may include one or more acoustic projection devices. The one or more acoustic projection devices are configured to generate one or more pressure waves that complement the holographic content when executing display instructions received from controller 520. The generated pressure waves may be, for example, audible (in the case of sound), ultrasonic (in the case of touch), or some combination thereof. Similarly, sensory feedback system 570 may include an aroma projection device. The aroma projection device may be configured to provide a scent to some or all of the target area when executing display instructions received from the controller. The aroma device may be coupled within an air circulation system (e.g., a duct, fan, or vent) to regulate airflow within the target area. Additionally, sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature of some or all of the target area when executing display instructions received from controller 520.

[0091] In some embodiments, the sensory feedback system 570 is configured to receive input from a viewer of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the viewer. The sensory feedback devices can include devices such as acoustic receiving devices (e.g., microphones), pressure sensors, joysticks, motion detectors, transducers, etc. The sensory feedback system can transmit the detected input to the controller 520 to adjust the generation of the holographic content and / or sensory feedback.

[0092] By way of illustration, in one exemplary embodiment of the light field display assembly 510, the sensory feedback system 570 includes a microphone. The microphone is configured to record audio generated by one or more viewers (e.g., video conference participants). The sensory feedback system 570 provides the recorded audio to the controller 520 as viewer input. The controller 520 can use the viewer input to generate holographic content. For example, if a participant mentions a particular product, a holographic representation of the product can be generated to provide context for the participant's comment. Similarly, the sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure a force applied to the pressure sensor by the viewer. The sensory feedback system 570 can provide the measured force to the controller 520 as viewer input.

[0093] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to determine the position, movement, and / or characteristics of viewers within a target area. Generally, the tracking devices are external to the LF display assembly 510. Some exemplary tracking devices include a camera assembly (“camera”), a depth sensor, structured light, a LIDAR system, a card scanning system, or other tracking device capable of tracking viewers within a target area.

[0094] The tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. However, in some cases, when presenting holographic content, the target area is illuminated with natural light and / or ambient light from the LF display assembly 510. The energy source projects light when executing instructions received from the controller 520. The light may be, for example, a structured light pattern, a pulse of light (e.g., an IR flash), or some combination thereof. The tracking system may project light within the visible band (approximately 380 nm to 750 nm), the infrared (IR) band (approximately 750 nm to 1700 nm), the ultraviolet band (10 nm to 380 nm), some other portion of the electromagnetic spectrum, or some combination thereof. Sources may include, for example, light-emitting diodes (LEDs), micro-LEDs, laser diodes, time-of-flight depth sensors, tunable lasers, etc.

[0095] Tracking system 580, when executing instructions received from controller 520, can adjust one or more emission parameters. Emission parameters are parameters that affect how light is projected from a source of tracking system 580. Emission parameters can include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, any other parameter that affects how light is projected from the source assembly, or any combination thereof. In one embodiment, the source projects pulses of light in a time-of-flight operation.

[0096] The camera of tracking system 580 captures images of light (e.g., a structured light pattern) reflected from the target area. The camera captures the images as it executes tracking instructions received from controller 520. As previously described, the light may be projected by a source of tracking system 580. The camera may include one or more cameras. That is, the camera may be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by tracking system 580, or some combination thereof. In one embodiment, tracking system 580 may include a light field camera external to LF display assembly 510. In other embodiments, the camera is included as part of LF display source / sensor module 514 included in LF display assembly 510. For example, as described above, if the energy relay element of the light field module 512 is a bidirectional energy layer that interleaves both a light-emitting display and an image sensor with the energy device layer 220, the LF display assembly 510 can be configured to simultaneously project a light field and record image information from a viewing area in front of the display. In one embodiment, images captured from the bidirectional energy plane form a light field camera. The camera provides the captured images to the controller 520.

[0097] The cameras of tracking system 580 may adjust one or more imaging parameters when executing tracking instructions received from controller 520. Imaging parameters are parameters that affect how the cameras capture images. Imaging parameters may include, for example, frame rate, aperture, gain, exposure length, frame timing, rolling shutter or global shutter capture mode, any other parameter that affects how the cameras capture images, or any combination thereof.

[0098] The controller 520 controls the LF display assembly 510 and any other components of the LF display system 500. The controller 520 comprises a data store 522, a network interface 524, a tracking module 526, a viewer profiling module 528, and a light field processing engine 530. In other embodiments, the controller 520 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or different entities in a manner different from that described herein. For example, the tracking module 526 may be part of the LF display assembly 510 or the tracking system 580.

[0099] The data store 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by the camera, one or more viewer profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 including the resolution and orientation of the LF module 512, desired viewing volume shape, 3D models, content for graphics creation including scenes and environments, materials, and textures, other information that may be used by the LF display system 500, or some combination thereof. The data store 522 is a memory such as a read-only memory (ROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), or some combination thereof.

[0100] The network interface 524 allows the light field display system to communicate with other systems or environments over a network. In one example, the LF display system 500 receives holographic content from a remote light field display system via the network interface 524. In another example, the LF display system 500 uses the network interface 524 to transmit holographic content to a remote data store.

[0101] The tracking module 526 tracks viewers viewing content presented by the LF display system 500. To do so, the tracking module 526 generates tracking instructions that control the operation of the sources and / or cameras of the tracking system 580 and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking input to the tracking module 526.

[0102] The tracking module 526 may determine the positions of one or more viewers within the target area (e.g., sitting in a particular chair in a conference room, walking around the conference room, etc.). The determined positions may be, for example, relative to some reference point (e.g., a display surface, a conference table, etc.). In other embodiments, the determined positions may be within a virtual model of the target area. The tracked positions may be, for example, the tracked positions of the viewers and / or the tracked positions of parts of the viewers (e.g., eye positions, hand positions, etc.). The tracking module 526 determines the positions using one or more images captured from cameras of the tracking system 580. The cameras of the tracking system 580 may be distributed around the LF display system 500 and may capture images in stereo, allowing the tracking module 526 to passively track the viewers. In other embodiments, the tracking module 526 actively tracks the viewers. That is, the tracking system 580 illuminates a portion of the target area and images the target area, and the tracking module 526 determines the positions using time-of-flight and / or structured light depth determination techniques. The tracking module 526 uses the determined location to generate tracking information.

[0103] The tracking module 526 may also receive tracking information as input from a viewer of the LF display system 500. The tracking information may include body movements of the viewer corresponding to various input options provided by the LF display system 500. For example, the tracking module 526 may track the viewer's body movements and assign any of the various movements as input to the LF processing engine 530. The tracking module 526 may provide the tracking information to the data store 522, the LF processing engine 530, the viewer profiling module 528, any other component of the LF display system 500, or some combination thereof.

[0104] To provide context for the tracking module 526, consider one exemplary embodiment of the LF display system 500, which provides video conferencing for a meeting. In response to a participant requesting a vote on a proposal, one or more participants raise their hands. The tracking system 580 may record the participants' hand movements and send the recording to the tracking module 526. The tracking module 526 tracks the participants' hand movements in the recording and sends input to the LF processing engine 530. As described below, the viewer profiling module 528 determines whether information in the image indicates that the viewer's hand movements are associated with a vote in favor of the proposal. The LF processing engine 530 may generate appropriate holographic content indicating the results of the vote. For example, the LF processing engine 530 may project the number of votes.

[0105] The LF display system 500 includes a viewer profiling module 528 configured to identify and profile viewers. The viewer profiling module 528 generates a profile of a viewer (or multiple viewers) viewing holographic content displayed by the LF display system 500. The viewer profiling module 528 generates the viewer profile based in part on viewer input and monitored viewer behavior, actions, and reactions. The viewer profiling module 528 can access information obtained from the tracking system 580 (e.g., recorded images, video, audio, etc.) and process the information to determine various information. In various examples, the viewer profiling module 528 can use any number of machine vision or machine hearing algorithms to determine viewer behavior, actions, and reactions. Monitored viewer behavior may include, for example, smiling, raising a hand, cheering, clapping, laughter, and / or other changes in gestures or movements by the viewer.

[0106] More generally, a viewer profile may include any information received and / or determined about a viewer viewing holographic content from the LF display system. For example, each viewer profile may record that viewer's actions or responses to content displayed by the LF display system 500. Some example information that may be included in a viewer profile is provided below.

[0107] In some embodiments, the viewer profile may indicate the viewer's role with respect to an organization or group associated with the LF display system 500. For example, in a video conferencing system operated by an enterprise, the viewer profile may include, for example, the viewer's job title, responsibilities, permission to view confidential information, etc. The viewer profile may additionally or alternatively indicate more general viewer characteristics, such as, for example, age, gender, ethnicity, clothing, location, etc.

[0108] In some embodiments, the viewer profile may indicate the viewer's preferences regarding the presentation of holographic content. For example, the viewer profile may indicate holographic object volumes in which holographic content should be displayed (e.g., to the viewer's right) and holographic object volumes in which holographic content should not be displayed (e.g., to the viewer's left). The viewer profile may also indicate that the viewer prefers to have a haptic interface presented near them or to avoid a haptic interface.

[0109] In some embodiments, the viewer profile may also indicate characteristics and preferences of a group of viewers, rather than a particular viewer. For example, the viewer profiling module 528 may generate a viewer profile for a particular combination of viewers in a video conference. In one example, the viewer profiling module 528 generates a profile for a pair of viewers indicating that the pair prefers a particular configuration, such as a holographic viewing zone, haptic interface, background image, and / or avatar, when engaging in a video conference session using the LF display system 500. Any of the foregoing information and characteristics may be applied to a group of viewers.

[0110] The viewer profiling module 528 may access profiles associated with a particular viewer (or multiple viewers) from one or more third-party systems to construct a viewer profile. For example, a viewer may link one or more social media accounts to a viewer profile maintained by the viewer profiling module 528. The viewer profiling module 528 may access information from one or more of the social media accounts to construct a viewer profile.

[0111] In some embodiments, data store 522 includes a viewer profile store that stores viewer profiles generated, updated, and / or maintained by viewer profiling module 528. The viewer profiles may be updated in the data store by viewer profiling module 528 at any time. For example, in one embodiment, the viewer profile store receives and stores information about a particular viewer in a viewer profile for that viewer when that viewer views holographic content provided by LF display system 500. In this example, viewer profiling module 528 includes a facial recognition algorithm that can recognize the viewer and reliably identify the viewer when the viewer views the presented holographic content. To illustrate, when a viewer enters a target area of LF display system 500, tracking system 580 captures an image of the viewer. Viewer profiling module 528 inputs the captured image and identifies the viewer's face using a facial recognition algorithm. The identified face is associated with a viewer profile in the profile store, so that all input information acquired about that viewer can be stored in that viewer's profile. The viewer profiling module 528 may also utilize a card identification scanner, a voice identifier, a radio frequency identification (RFID) chip scanner, a bar code scanner, or the like to positively identify the viewer.

[0112] In embodiments in which the viewer profiling module 528 can reliably identify a viewer, the viewer profiling module 528 can determine each visit of each viewer to the LF display system 500. The viewer profiling module 528 can then store the date and time of each visit in a viewer profile for each viewer. Similarly, the viewer profiling module 528 can store input from the viewer received from any combination of the sensory feedback system 570, the tracking system 580, and / or the LF display assembly 510 as the input occurs. The viewer profile system 528 can additionally receive further information about the viewer from other modules or components of the controller 520, which information can be stored along with the viewer profile. Other components of the controller 520 can then also access the stored viewer profile to determine subsequent content to be provided to that viewer.

[0113] The LF processing engine 530 generates holographic content consisting of light field data as well as data for all sensory domains supported by the LF display system 500. For example, the LF processing engine 530 can generate 4D coordinates in a rasterized format (“rasterized data”) that, when executed by the LF display assembly 510, cause the LF display assembly 510 to present the holographic content. The LF processing engine 530 can access the rasterized data from the data store 522. Additionally, the LF processing engine 530 can construct the rasterized data from vectorized data sets. Vectorized data is described below. The LF processing engine 530 can also generate the sensory instructions necessary to provide sensory content that augments the holographic object. As described above, the sensory instructions, when executed by the LF display system 500, can generate tactile surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF processing engine 530 can access the sensory instructions from the data store 522 or construct the sensory instructions to form the vectorized data sets. Collectively, the 4D coordinates and sensory data represent the holographic data as display instructions executable by the LF display system to generate holographic and sensory content. More generally, holographic content can take the form of computer graphics (CG) content with idealized light field coordinates, live-action content, rasterized data, vectorized data, electromagnetic energy transported by a set of relays, instructions sent to a group of energy devices, energy locations on one or more energy planes, a set of energy propagation paths projected from a display surface, a holographic object visible to a viewer or audience, and many other similar forms.

[0114] The amount of rasterized data describing the flow of energy through the various energy sources within LF display system 500 is extremely large. While the rasterized data can be displayed on LF display system 500 when accessed from data store 522, the rasterized data cannot be efficiently transmitted, received (e.g., via network interface 524), and subsequently displayed on LF display system 500. Take, for example, rasterized data representing a short film for holographic projection by LF display system 500. In this example, LF display system 500 includes a display containing several gigapixels, and the rasterized data includes information about each pixel location on the display. The corresponding size of the rasterized data is enormous (e.g., many gigabytes per second of film display time) and unmanageable for efficient transfer over a commercial network via network interface 524. The problem of efficient transfer may be amplified in applications involving live streaming of holographic content. An additional problem with simply storing rasterized data on the data store 522 arises when an interactive experience is desired, using input from the sensory feedback system 570 or the tracking module 526. To enable an interactive experience, the light field content generated by the LF processing engine 530 may be modified in real time in response to sensory or tracking input. In other words, in some cases, the LF content cannot simply be read from the data store 522.

[0115] Thus, in some configurations, data representing holographic content for display by the LF display system 500 may be transferred to the LF processing engine 530 in a vectorized data format (“vectorized data”). Vectorized data may be orders of magnitude smaller than rasterized data. Furthermore, vectorized data has a dataset size that allows for efficient sharing of data while providing high image quality. For example, vectorized data may be a sparse dataset derived from a denser dataset. Thus, vectorized data may have an adjustable balance between image quality and data transmission size based on how the sparse vectorized data is sampled from the dense rasterized data. Adjustable sampling to generate the vectorized data allows for optimization of image quality for a given network speed. Consequently, vectorized data enables efficient transmission of holographic content over the network interface 524. Vectorized data also allows holographic content to be live streamed over commercial networks.

[0116] In summary, the LF processing engine 530 may generate holographic content derived from rasterized data accessed from the data store 522, vectorized data accessed from the data store 522, or vectorized data received via the network interface 524. In various configurations, the vectorized data may be encoded prior to data transmission and decoded after receipt by the LF controller 520. In some examples, the vectorized data is encoded for added data security and performance improvements associated with data compression. For example, the vectorized data received by the network interface may be encoded vectorized data received from a holographic streaming application. In some examples, the vectorized data may require a decoder, the LF processing engine 530, or both, to access the information content encoded in the vectorized data. The encoder and / or decoder systems may be available to customers or licensed to third-party vendors.

[0117] The vectorized data includes all information for each of the sensory domains supported by the LF display system 500 in a manner that can support an interactive experience. For example, the vectorized data for an interactive holographic experience can include any vectorized characteristics that can provide accurate physics for each of the sensory domains supported by the LF display system 500. The vectorized characteristics can include any characteristics that can be synthetically programmed, captured, and computationally evaluated. The LF processing engine 530 can be configured to convert the vectorized characteristics in the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content translated from the vectorized data using the LF display assembly 510. In various configurations, vectorized properties may include one or more red / green / blue / alpha channel (RGBA) plus depth images, multi-view images with or without depth information at various resolutions that may include one high-resolution central image and other views at lower resolutions, material properties such as albedo and reflectance, surface normals, other optical effects, surface identity, geometric object coordinates, virtual camera coordinates, viewing plane position, lighting coordinates, haptic stiffness of surfaces, haptic ductility, haptic intensity, sound field amplitude and coordinates, environmental conditions, somatosensory energy vectors related to mechanoreceptors for texture or temperature, sound, and any other sensory domain properties. Many other vectorized properties are possible.

[0118] The LF display system 500 can also generate interactive viewing experiences. That is, the holographic content may respond to input stimuli, including information about the viewer's position, gestures, interactions, interactions with the holographic content, or other information derived from the viewer profiling module 528 and / or tracking module 526. For example, in one embodiment, the LF processing system 500 uses real-time performance vectorized data received via the network interface 524 to create the interactive viewing experience. In another example, if a holographic object needs to immediately move in a particular direction in response to a viewer's interaction, the LF processing engine 530 can update the rendering of the scene so that the holographic object moves in the requested direction. This may require the LF processing engine 530 to use the vectorized data set to render a light field, on a real-time basis, into a 3D graphical scene with appropriate object placement and movement, collision detection, occlusion, color, shading, lighting, etc., in response to the viewer's interaction. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510 .

[0119] The rasterized data includes holographic content instructions and sensory instructions (display instructions) that represent the real-time performance. The LF display assembly 510 executes the display instructions to simultaneously project the holographic and sensory content of the real-time performance. The LF display system 500 monitors viewer interactions (e.g., voice responses, touch, etc.) with the presented real-time performance using a tracking module 526 and a viewer profiling module 528. In response to viewer interactions, the LF processing engine can create an interactive experience by generating additional holographic and / or sensory content for display to the viewer.

[0120] To illustrate, consider one exemplary embodiment of an LF display system 500 that includes an LF processing engine 530 that generates a holographic object representing a product prototype. A viewer may move to touch the holographic object representing the product prototype. In response, a tracking system 580 tracks the viewer's hand movements relative to the holographic object. The viewer's movements are recorded by the tracking system 580 and transmitted to the controller 520. The tracking module 526 continuously determines the viewer's hand movements and transmits the determined movements to the LF processing engine 530. The LF processing engine 530 determines the viewer's hand placement within the scene and adjusts the real-time rendering of the graphics to include any desired changes to the holographic object (such as position, color, or occlusion). The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a haptic surface using a volumetric haptic projection system (e.g., using an ultrasonic speaker). The generated haptic surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the holographic object's exterior surface. The LF processing engine 530 uses the tracking information to dynamically direct the LF display assembly 510 to move the position of the haptic surface along with the position of the rendered holographic object so that the viewer is given both the visual and tactile perception of touching the prototype. In simpler terms, as the viewer sees their hand touching the holographic prototype, the viewer simultaneously feels haptic feedback indicating that their hand is touching the holographic prototype and that the prototype changes position or movement in response to the touch.In some examples, rather than presenting an interactive prototype accessed from data store 522, the interactive prototype may be received as part of holographic content received from a live streaming application via network interface 524 (e.g., the holographic prototype may be a holographic representation of a physical prototype at different physical locations being imaged by different LF display systems 500).

[0121] In embodiments where the LF display system is used to provide video conferencing, the holographic content may include holographic images of one or more participants in the video conference. The holographic content may also include other holographic objects, such as, for example, holographic handouts, holographic whiteboards, holographic movies or videos, holographic simulations, holographic product prototypes, holographic models, holographic experiences, holographic games, holographic items, holographic assistants, any other holographic objects, or any combination thereof. In some embodiments, the holographic content may be received from a third-party system separate from the LF display system 500.

[0122] In a videoconferencing configuration, participants are located at two or more physical locations. At least one of the physical locations has an LF capture system. The LF capture system can be a plenoptic light field camera external to the LF display or a multi-view camera system with multiple lenses and sensors. Alternatively, the LF capture system can be integrated into the LF display assembly 510 as a bidirectional energy surface that projects LF and absorbs incident light, relaying it to an image sensor.

[0123] The datasets generated when capturing a complete LF can be unmanageable for most processors and efficient network transmission. To address this, data recorded from a light field capture system can be compressed. In some embodiments, the data is reduced to a vectorized format containing, for example, N images of red / green / blue / alpha channels (RGBA) plus depth, N multi-view images with or without depth at various resolutions, which may include one high-resolution central image and other views at lower resolutions, or any other reduced dataset. In this way, the amount of data required to represent a complete LF can be significantly reduced (e.g., by orders of magnitude), creating a viable path to enabling the transmission of truly holographic datasets. Such techniques may provide high image quality while balancing dataset size for efficient sharing of data, offering benefits such as reduced storage requirements, allowing data to be live-streamed over a network, etc.

[0124] In some embodiments, the data is encoded by a proprietary encoding block. This encoding process is part of a proprietary encoding / decoding pair. One or more receiving systems include a matching decoding block. These encoding and decoding blocks may be licensed by a third-party vendor. The encoding block compresses the vectorized format while balancing image quality with transmission speed, and may automatically adjust to network speeds, for example, to provide the best possible image quality for the available bandwidth. The encoding process may use selectable or variable compression ratios, which may include real-time or offline image processing, other computations, and / or data reduction to a sparser data set.

[0125] The encoded data is transmitted from the LF capture system via a network interface (e.g., network interface 524 if the LF capture system is integrated with the LF display system 500) using a steaming engine to one or more LF display systems 500 (e.g., located at one or more physical locations of one or more other participants). The encoded data is received and decoded (e.g., by a proprietary decoder) by the LF display system 500. The decoder processes the encoded signal and, in combination with the display driver and display hardware configuration, enables the LF processing engine 530 to agnostically project the available information as a fully rasterized 4D light field, taking into account the resolution of the LF display assembly 510 and the available haptic interface and other sensory projection capabilities of the LF display system 500.

[0126] As previously mentioned, the encoded data transmitted over the network is in a vectorized format. The LF processing engine 530 takes this data and converts it into a rasterized format that drives the LF display assembly 510. The rasterized format can be orders of magnitude larger than the vectorized data set. In some embodiments, both holographic videoconferencing information and CG content are transmitted.

[0127] In some embodiments, the encoded data also includes matching haptic surface instructions for some or all of the holographic object. In one embodiment, the vectorized format generated from the light field display assembly 510 includes vectorized properties that provide accurate physics for multiple sensory domains, where properties can be synthetically programmed, captured, or computationally accessed, including, for example, N images of red / green / blue / alpha channels (RGBA) plus depth, N multi-view images with or without depth at various resolutions, which may include one high-resolution central image and other views at lower resolutions, material properties such as albedo and reflectivity, surface normals, other optical effects, surface identification, geometric object coordinates, virtual camera coordinates, display plane position, lighting coordinates, haptic stiffness of surfaces, haptic ductility, haptic intensity, sound field amplitude and coordinates, environmental conditions, somatosensory energy vectors related to mechanoreceptors for texture or temperature, game sounds, and any other sensory domain properties. Vectorizing the data can eliminate multiple orders of required data and create a viable path to enabling the transmission of datasets containing properties within multiple sensory domains.

[0128] The LF processing engine 530 can also modify the holographic content to fit the space in which it is being presented. For example, not all conference rooms are the same size, have the same number of seats, or have the same technical configuration. Thus, the LF processing engine 530 can modify the holographic content so that it appears appropriately within the conference room. In one embodiment, the LF processing engine 530 can access a conference room configuration file that includes the conference room's layout, resolution, field of view, other technical specifications, etc. The LF processing engine 530 can render and present the holographic content based on the information contained in the configuration file.

[0129] The LF processing engine 530 can also create holographic content for display by the LF display system 500. Importantly, creating holographic content for display is distinct from accessing or receiving holographic content for display. That is, when creating content, the LF processing engine 530 generates entirely new content for display rather than accessing previously generated and / or received content. The LF processing engine 530 can create holographic content for display using information from the tracking system 580, the sensory feedback system 570, the viewer profiling module 528, the tracking module 526, or some combination thereof. In some examples, the LF processing engine 530 can access information from elements of the LF display system 500 (e.g., tracking information and / or viewer profiles) and, accordingly, create holographic content based on that information and display the created holographic content using the LF display system 500. The created holographic content can be augmented with other sensory content (e.g., touch, sound, or smell) when displayed by the LF display system 500.

[0130] Dynamic Content Generation for LF Display Systems In some embodiments, the LF processing engine 530 incorporates artificial intelligence (AI) models to create holographic content for display by the LF display system 500. The AI models may include supervised or unsupervised learning algorithms, including, but not limited to, regression models, neural networks, classifiers, or other AI algorithms. The AI models may be used to determine viewer preferences based on viewer information recorded by the LF display system 500 (e.g., by the tracking system 580), which may include information about viewer behavior.

[0131] The AI model can access information from data store 522 to create the holographic content. For example, the AI model can access viewer information from one or more viewer profiles in data store 522 or can receive viewer information from various components of LF display system 500. To illustrate, the AI model may determine that one viewer prefers to view financial data in charts, while another viewer prefers to view financial data in tables. The AI model can determine preferences based on the viewer's reaction or response to previously viewed holographic content, such as financial data (e.g., did the viewer ask for an explanation? Did the user request to see the data in a different format?), etc. The LF display system 500 can then present the same data to different users using different types of holographic representations. That is, the AI model can create personalized holographic content for sets of viewers according to the learned viewer preferences. So, for example, the AI model can create a holographic chart of financial data for one user and a holographic table of the same data for another user. The AI model can also store each learned viewer's preferences in a viewer profile store in data store 522. In some instances, AI models can create holographic content tailored for groups of viewers, rather than a single viewer.

[0132] One example of an AI model that can be used to identify viewer characteristics, identify responses, and / or generate holographic content based on the identified information is a convolutional neural network model having layers of nodes, where the value of a node in a current layer is the transformation of the value of a node in a previous layer. Transformations in the model are determined by sets of weights and parameters connecting the current layer and the previous layer. For example, the AI model may include five layers of nodes: Layers A, B, C, D, and E. The transformation from Layer A to Layer B is given by function W1, the transformation from Layer B to Layer C is given by W2, the transformation from Layer C to Layer D is given by W3, and the transformation from Layer D to Layer E is given by W4. In some examples, transformations may also be determined via sets of weights and parameters used to transform between previous layers in the model. For example, the transformation W4 from Layer D to Layer E may be based on the parameters used to perform the transformation W1 from Layer A to Layer B.

[0133] The input to the model may be an image captured by tracking system 580, encoded onto convolutional layer A, and the output of the model is holographic content decoded from output layer E. Alternatively or additionally, the output may be viewer features determined within the image. In this example, the AI model identifies latent information within the image that represents viewer features in identification layer C. The AI model reduces the dimensionality of convolutional layer A to the dimensionality of identification layer C to identify any features, actions, responses, etc. within the image. In some examples, the AI model then increases the dimensionality of identification layer C to generate holographic content.

[0134] Images from tracking system 580 are encoded into convolutional layer A. The images input into convolutional layer A may be associated with various features and / or reaction information, etc., in identification layer C. The associated information between these elements may be retrieved by applying a set of transformations between corresponding layers. That is, convolutional layer A of the AI model represents the encoded image, and identification layer C of the model represents the smiling viewer. The smiling viewer in a given image may be identified by applying transformations W1 and W2 to the image's pixel values in the space of convolutional layer A. The weights and parameters of the transformations may indicate the relationship between the information contained in the image and the identification of the smiling viewer. For example, the weights and parameters may be quantizations of the shape, color, size, etc., contained in the information representing the smiling viewer in the image. The weights and parameters may be based on historical data (e.g., previously tracked viewers).

[0135] Smiling viewers in the image are identified in identification layer C. Identification layer C represents smiling viewers identified based on latent information about smiling viewers in the image.

[0136] The identified smiling viewer in the image can be used to generate holographic content. To generate the holographic content, the AI model starts with identification layer C and applies transformations W2 and W3 to a given identified smiling viewer value in identification layer C. The transformations result in a set of nodes in output layer E. The weights and parameters of the transformations can indicate a relationship between the identified smiling viewer and specific holographic content and / or preferences. The holographic content may be output directly from the nodes in output layer E, or the content generation system may decode the nodes in output layer E into holographic content. For example, if the output is a set of identified features, the LF processing engine can use the features to generate the holographic content.

[0137] Additionally, an AI model may include layers known as hidden layers. Hidden layers are layers that do not correspond to images and identify features / responses, etc., or generate holographic content. For example, in the given example, Layer B is an intermediate layer between Convolutional Layer A and Identification Layer C. Layer D is an intermediate layer between Identification Layer C and Output Layer E. Hidden layers are latent representations of various aspects of identification that are not observed in the data but may govern the relationships between elements of the image when identifying features and generating holographic content. For example, a node in a hidden layer may have a strong connection (e.g., a large weight value) between input values and identification values that share the commonality of "a smiling person's smile." As another example, another node in a hidden layer may have a strong connection between input values and identification values that share the commonality of "a scared person's scream." Of course, there can be any number of linkages in a neural network. Additionally, each hidden layer is a combination of functions, such as residual blocks, convolutional layers, pooling operations, skip connections, and concatenation. Any number of hidden layers B may serve to reduce the convolutional layers down to discriminative layers, and any number of hidden layers D may serve to augment the discriminative layers up to the output layer.

[0138] In one embodiment, the AI model comprises a deterministic method trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve performance quality using measurements from the tracking system 580 as inputs and modifications to the created holographic content as outputs.

[0139] Reinforcement learning is a machine learning system in which a machine learns "what to do" (how to map situations to actions) to maximize a numerical reward signal. The learner (e.g., LF processing engine 530) is not told which action to take (e.g., generating a given holographic content), but instead, by trying actions, discovers which action will yield the greatest reward (e.g., increasing the quality of the holographic content by getting more people to cheer). In some cases, actions can affect not only the immediate reward, but also the next situation and, through it, all subsequent rewards. These two properties (trial-and-error exploration and delayed reward) are two distinguishing features of reinforcement learning.

[0140] Reinforcement learning is defined not by characterizing the learning method, but by characterizing the learning problem. Essentially, a reinforcement learning system captures key aspects of the problem facing a learning agent interacting with an environment to achieve a goal. That is, in the example of generating a song for a performer, the reinforcement learning system captures information about the audience in the venue (e.g., age, temperament, etc.). Such an agent senses the state of the environment and performs actions that affect the state to achieve one or more goals (e.g., creating a pop song that the audience will cheer). In its most basic form, a reinforcement learning formulation includes three aspects for the learner: senses, actions, and goals. Continuing with the song example, the LF processing engine 530 senses the state of the environment with sensors in the tracking system 580, displays holographic content to the audience in the environment, and achieves a goal that is a measure of the audience's reception of the song.

[0141] One challenge that arises in reinforcement learning is the trade-off between exploration and exploitation. To increase rewards in a system, a reinforcement learning agent favors actions that it has tried in the past and found to be effective at generating rewards. However, to discover actions that generate rewards, the learning agent chooses actions that it has not previously chosen. The agent "exploits" information it already knows to obtain rewards, but "exploits" information to select better actions in the future. The learning agent tries various actions, gradually favoring those that appear to be best while still trying new actions. In probabilistic tasks, each action is typically tried multiple times to obtain a reliable estimate of the expected reward. For example, if an LF processing engine creates holographic content that it knows will cause viewers to laugh after a long time has passed, the LF processing engine can modify the holographic content to decrease the time it takes for viewers to laugh.

[0142] Furthermore, reinforcement learning considers the entire problem of a goal-directed agent interacting with an uncertain environment. Reinforcement learning agents have clear goals, can sense aspects of the environment, and can choose actions to receive high rewards (i.e., a cheering crowd). Furthermore, agents typically operate despite significant uncertainty about the environment they face. When reinforcement learning involves planning, the system addresses the interplay between planning and real-time action selection, as well as the problem of how to acquire and improve upon environmental factors. For reinforcement learning to make progress, important subproblems must be studied in isolation, and the subproblems play distinct roles in a complete, interactive, goal-seeking agent.

[0143] A reinforcement learning problem is a framing of a machine learning problem in which interactions are processed and actions are taken to achieve a goal. The learner and decision maker are called the agent (e.g., the LF processing engine 530). What the agent interacts with, including everything external to the agent, is called the environment (e.g., the audience in the venue). These two interact continuously, with the agent choosing actions (e.g., creating holographic content) and the environment responding to those actions by presenting the agent with new situations. The environment also generates a reward, a specific numerical value that the agent attempts to maximize over time. In one context, the reward functions to maximize the audience's positive response to the holographic content. A complete specification of the environment defines a task, which is one instance of a reinforcement learning problem.

[0144] To provide more context, the agent (e.g., LF processing engine 530) and the environment interact at each of a sequence of discrete time steps, i.e., t=0, 1, 2, 3, etc. At each time step t, the agent determines the state s of the environment. t Receive some representation of state s (e.g., measurements from tracking system 580). t is in S, where S is the set of possible states. t Based on time step t and time step t, the agent selects an action at (e.g., make the performer do a wide stride). The action at is A(s t ) and A(s t ) is a set of possible actions. After one time state, and partly as a result of the agent's actions, the agent receives a numerical reward r t+1 Receive the state r t+1 is in R, where R is the set of possible rewards. When the agent receives a reward, it moves to a new state s t+1 Select .

[0145] At each time step, the agent implements a mapping from the state to the probability of choosing each possible action. This mapping is called the agent's policy, and π t where π t (s,a) is s t =s then a t = a. Reinforcement learning methods can dictate how an agent changes its policy as a result of the states and rewards that result from its actions. The agent's goal is to maximize the total amount of rewards it receives over time.

[0146] This reinforcement learning framework is flexible and can be applied in many different ways to many different problems (e.g., generating holographic content).The framework proposes that any problem (or objective) of learning goal-directed behavior, whatever the details of the senses, memories, and controllers, can be boiled down to three signals passing between the agent and its environment: one signal representing the choice made by the agent (action), one signal representing the rationale for the choice (state), and one signal defining the agent's goal (reward).

[0147] Of course, the AI model may include any number of machine learning algorithms. Some other AI models that may be used are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In either case, generally, the LF processing engine 530 takes input from the tracking module 526 and / or the viewer profiling module 528, and the machine learning model creates the holographic content accordingly. Similarly, the AI model may direct the rendering of the holographic content.

[0148] In some embodiments, the LF processing engine 530 generates display instructions for presenting a holographic assistant to participants in a video conference. Such an assistant may provide participants with information related to the video conference, assist with presentations, and the like. The LF processing engine 530 can retrieve holographic assistants stored as holographic objects in a data store. Each holographic assistant may have various parameters that direct the presentation of the holographic assistant. For example, a holographic assistant may have parameters including, but not limited to, the type of assistant (e.g., human avatar, alien, robot, humanoid, etc.), the size of the assistant, the gender of the assistant if available, the voice of the assistant, the personality of the assistant, or any combination thereof.

[0149] In additional embodiments, the LF processing engine 530 further accesses a viewer profile that may include the participant's preferences for an assistant. For example, the viewer profile may include a preference (e.g., provided by the participant or inferred by the viewer profiling module 528) that the participant prefers to have the assistant as a male human with brown hair and a deep voice, a female alien with black hair and a high-pitched voice, etc. Additionally, the LF processing engine 530 may incorporate an AI model, as described above, to generate instructions for the presentation of the assistant to present the holographic assistant to engage with the participant. Using the AI model, the sensory feedback system 570 can constantly generate instructions to provide audio feedback in response to audio input from the participant, i.e., to simulate a real-time conversation between the participant and the holographic assistant. Furthermore, the LF processing engine 530 may use tracking information from the tracking system 580 and / or the tracking module 526 to generate display instructions for the LF display assembly 510 to adjust the presentation of the holographic assistant's eyes to move or track with the participant's gaze or body movements.

[0150] The foregoing examples of creating content are not limiting. In the broadest sense, the LF processing engine 530 creates holographic content for display to a viewer of the LF display system 500. The holographic content may be created based on any of the information contained in the LF display system 500.

[0151] Lightfield Display Video Conferencing System FIG. 6 is an illustration of an LF display system 600 for videoconferencing, according to one or more embodiments. LF display system 600 is an embodiment of LF display system 500. LF display system 600 is located within a conference space 610. Conference space 610 is typically a room that includes an LF display formed by LF display modules 620 of LF display assembly 510. In the configuration shown in FIG. 6, LF display module 620 is configured to cover one wall of a videoconferencing room. However, conference space 610 can be any physical space in which LF display module 620 can be temporarily or permanently installed. In one embodiment, LF display system 600 has an imaging system integrated into LF display assembly 510. LF display assembly 510 can thus function as a two-way display surface that both projects a light field and relays light from the display surface to at least one image sensor. The LF display system may additionally or alternatively have a camera external to the display, and images are analyzed using tracking software in the controller 520 for the light field display assembly 510.

[0152] In FIG. 6 , conference space 610 also includes table 630 and chair 640. Table 640 is positioned against a wall with LF display module 620, and chair 640 is positioned so that a participant is seated facing table 630. LF display module 620 generates a holographic image of table 632 and chair 642 located in a different physical space 612 where another video conference participant is located. The holographic image may be generated based on visual data collected by a remote light field display assembly (e.g., one or more remote LF display modules) located in remote conference space 612. For example, in one embodiment, each conference space includes LF display system 600 having an imaging system integrated into LF display assembly 510. Each display assembly 510 thus functions as a bidirectional surface, which may be configured to create the impression that two conference spaces are physically adjacent to each other, with LF display assembly 510 acting as a window between them.

[0153] The visual data used to generate the holographic image may be a digital representation of light detected by one or more image sensors (e.g., image sensors of one or more LF display modules 620). For example, in the case of chair 642, the visual data may be light reflected from the chair and detected by LF display module 620 of LF display system 600 in remote conference space 612. Alternatively or additionally, remote light field display assembly 510 may collect visual data using other sensors, such as standalone cameras. In some embodiments, the visual data may be vectorized and / or compressed, as described above with respect to FIG. 5.

[0154] Holographic table 632 is adjacent to physical table 630. Thus, participants in conference space 610 (e.g., sitting in chair 640) see the physical chair 640 and an image of chair 642 at the other physical location 612 around a single table (composed of physical table 630 and an image of table 632 at the other location). Similarly, at the other physical location 612, holographic table 632 is presented adjacent to the physical instance of table 630, along with holographic chair 642. While FIG. 6 shows that two identical tables are used to enhance the illusion that they are two halves of a single table, tables of different sizes and shapes may also be used.

[0155] As participants sit in chairs 640, 642 (or otherwise position themselves within their respective viewing volumes), holograms of the participants may be provided at the other physical location. For example, a hologram of a first user sitting in chair 640 may be provided in second conference space 612, and a hologram of a second participant sitting in chair 642 may be provided in first conference space 610. While each physical location is described as having a single chair and participant, it should be understood that either or both locations may include multiple participants. It should also be understood that in some embodiments, there may be more than two physical locations.

[0156] In one embodiment, the remote conference space 612 does not include an LF display system, but is captured with an LF camera and images are sent to the LF display system 600 in the conference space 610. In another embodiment, both the remote conference space 612 and the local conference space 610 include the LF display system 600 and a light field capture device. In another embodiment, holographic data is captured within either space using one or more conventional 2D cameras along with a depth sensor. In yet another embodiment, 2D image data is captured using one or more 2D cameras within either the remote or local conference space, and holographic data is generated from the 2D image data using 2D-to-3D conversion techniques known in the art. In one embodiment, the LF display module 620 includes a bidirectional surface that projects a light field and simultaneously absorbs incident light, relaying the incident light to one or more image sensors, which can be used to record the light field from an area near the display surface. In a configuration where there is a light field display system 600 with such a bidirectional surface located at both locations involved in the video conference, there can be a one-to-one correspondence between the gaze directions of participants at the physical locations connected by the video conference solution because the LF display module 620 acts as a co-located display and camera. Thus, participants can make eye contact as if they were located in the same physical space. In other words, the LF display module 620 acts as a window between the two locations, creating the impression that the participants are sitting around a single table.

[0157] The LF display module 620 may also include an audible sound field projection system. The audible sound field may emanate from electrostatic speakers mounted on the surface of the display, positioned between the waveguide elements and co-located with structures designed to block light transmission between the waveguides or with optically transparent membranes that help to create such structures. Sound detected by a microphone at one physical location may be emitted by a corresponding LF display module 620 at another physical location. Thus, when a participant at one physical location speaks or otherwise makes a sound, at another physical location, the sound may appear to come from the participant's hologram. Alternatively, separate microphones and speakers may be used to establish an audio connection between the physical locations.

[0158] In some embodiments, the haptic projection system is integrated into the display surface formed by the LF display module 620. Thus, the display surface may be a dual-energy surface that projects both light field and focused ultrasound energy to create a haptic surface. This may enable video conference participants to experience the sensation of physical contact with remote participants, such as a handshake. The haptic projection system may also be used to generate holographic objects (e.g., holographic props 650, described in more detail below) that participants can touch and manipulate.

[0159] In some embodiments, the display surface includes a speaker (e.g., an electrostatic speaker) for generating the sound field. Thus, the system can control the direction from which participants hear sound. For example, the controller 520 of the LF display system 600 can use tracking software to determine which remote participant is speaking and cause the speaker to emit a sound field that substantially co-locates the apparent source of the generated sound with the holographic representation of the speaking participant. This can further enhance the impression that the participants are all located in the same physical space. Additionally or alternatively, the LF system 600 can include one or more speakers external to the display.

[0160] In the embodiment shown in FIG. 6 , the conference space 610 also includes a mobile LF system 660. The mobile LF system 660 may have a display surface area that is smaller than the display surface area of one or more other LF displays, such as the surface formed by the LF module 620. While the mobile LF system 660 is shown as a robot on wheels, other forms of locomotion may be used. For example, the mobile LF system 660 may be mounted on a track, mounted on an articulated robotic arm, or carried by a human participant. Regardless of the exact form employed, the mobile LF system 660 may provide the ability to further enhance the impact of participants from different physical locations being present in the same space.

[0161] In various embodiments, the mobile LF system 660 may include an image capture system, an image display system, or both. The image capture system of the mobile LF system 660 may include one or more 2D cameras, depth sensors, and / or LF cameras. Thus, the mobile LF system 660 may be configured to capture 2D image data and / or holographic data of an area near the mobile LF system 660 (e.g., in front of the mobile LF system). The mobile LF system 660, in a configuration that captures 2D image data, may also convert the 2D image data into holographic data (e.g., using depth sensor data and / or 2D-to-3D techniques known in the art). The mobile LF system 660 may also be configured to capture other types of data. For example, the mobile LF system 660 may include a microphone configured to capture audio data.

[0162] The image display system of the mobile LF system 660 may include a 2D display, an LF display system, or both. In some embodiments, the mobile LF system 660 includes a two-way surface that may be configured to capture LF image data and simultaneously project holographic content.

[0163] In one embodiment, the mobile LF system 660 is located within one space (e.g., the local conference space 610) but is controlled by a participant located at a remote location (e.g., the remote conference space 612). Thus, the controlling participant can navigate the mobile LF system 660 around the conference space 610 to see fields of view (e.g., the back of the chair 640) that may not otherwise be available. Additionally, the mobile LF display 660 may create a holographic image of part (e.g., the head and shoulders) or all of the controlling participant (or the controlling participant's avatar). As a result, the controlling participant can explore the conference space 610, and other participants physically located within the conference space 610 can interact with the holographic image of the controlling participant as if they were physically present. This can further enhance the impression that the participants are located within a common physical space.

[0164] LF display system 600 may also present supplemental holographic content within conference space 610. The embodiment shown in FIG. 6 includes two examples of holographic content: holographic prop 650 (in this case, a holographic image of a car) and holographic whiteboard 670. Holographic prop 650 and holographic whiteboard 670 may be displayed in both the local and remote conference spaces, or in only one of these spaces. In other embodiments, different and / or additional supplemental holographic content may be presented by the LF display system.

[0165] The holographic prop 650 is a holographic object such as a 3D CAD model. The holographic prop 650 can be an image of a physical object (e.g., a product prototype located in the second conference space 612) or a CG virtual object (e.g., generated from a file on a computer such as the controller 520). This may further enable participants in different physical locations to interact as if they were in the same space. For example, a participant in the second conference space 612 may select a 3D CAD model, and the LF display system 600 generates the holographic prop 650 in a holographic viewing volume in the first conference space 610. The location of the holographic viewing volume may be selected based on the preferences of one or more participants located in the conference space 610. Alternatively, the holographic prop 650 may appear in a default location.

[0166] In some embodiments, ultrasound is used to provide the tactile surface of the holographic prop 650, as described above. The tracking system 580 of the LF display system 600 may track the movements of participants within the conference space 610 to enable manipulation of the holographic prop. For example, a participant may be able to reach out and "grab" the holographic prop 650, rotating and / or moving the holographic prop 650. The tracking system 580 may also recognize certain gestures as commands related to the holographic prop 650. For example, if a participant places both hands on (or near) the holographic prop 650 and moves them apart or together, the tracking system 580 may interpret these gestures as commands to increase and decrease the size of the holographic prop 650, respectively.

[0167] If both conference spaces 610, 612 include the LF display system 600, a shared instance of the holographic prop 650 may be formed in each conference space. Thus, interactions with the holographic prop 650 by participants in one space may be reflected in the presentation of the holographic prop 650 in the other space. For example, if the holographic prop 650 is generated from a 3D CAD model, a participant in one space may rotate the model to reveal a particular feature while describing that feature as part of the presentation. The holographic prop 650 in the other space may similarly be automatically rotated to allow all participants to view the feature being described. In some embodiments, participants may annotate the holographic prop 650 using a stylus (or other such tool). The tracking system 580 may track the movement of the stylus and add the annotations to the file from which the holographic handout was generated and / or display a holographic version of the annotations in conjunction with the holographic prop 650. Returning to the previous example, the presenter could draw a circle around the feature being described, provide written annotations that provide additional information about the feature, and add arrows that show how the feature moves in operation.

[0168] The holographic whiteboard 670 provides an interface for participants to draw on, similar to a physical whiteboard. In some embodiments, the LF display system 600 provides a holographic image showing the extent of the holographic whiteboard 670 (e.g., a holographic image of a physical whiteboard) within each conference space 610, 612. For example, in the configuration shown in FIG. 6, each conference space 610, 612 may include a holographic whiteboard 670 next to a physical table 630, 632. The holographic whiteboard 670 at each location is synchronized, meaning any additions or modifications made in one location appear in the other. Thus, participants at different physical locations can collaborate on the holographic whiteboard 670 as if they were present in the same physical space. Ultrasound can be used to create a tactile surface for part or all of the holographic whiteboard 670, giving participants the sensation of drawing on a physical whiteboard (or some other surface, such as paper, a chalkboard, or a canvas).

[0169] In one embodiment, different participants' contributions are stored as different layers. Participants may select (e.g., using a physical remote control and / or holographic control device) which layer is displayed on the holographic whiteboard 670. The layer selection may be synchronized across instances of the holographic whiteboard 670. Alternatively, participants at each location may independently select which layer to display. In other embodiments, layers may be added, used, and displayed in different ways. For example, participants may be provided with controls (e.g., physical or holographic) to add and remove layers and to select which layer is currently active. New content may be added to the currently active layer. The controls may also allow participants to split layers and move content between layers. Thus, participants may control which content on the holographic whiteboard 670 is on which layer.

[0170] In some embodiments, the holographic whiteboard 670 is 3D. Because the whiteboard is holographic, it need not be limited to a 2D surface. Participants can draw lines that move forward / backward, as well as up / down and / or left / right. This may enable participants to directly draw 3D structures and express complex concepts and relationships more easily than would be possible if limited to a 2D representation. In one embodiment, the 3D whiteboard is divided into slices along the front-to-back axis. Participants may select which slices are displayed to see different cross-sections of the depicted 3D structure.

[0171] In some embodiments, the holographic whiteboard 670 can be used to display 3D holographic objects such as CAD models, building schematics, park plans, etc. The 3D objects can be generated from data files or by imaging physical objects (e.g., product prototypes, sets of blueprints, etc.) located within one of the conference spaces (e.g., remote conference space 612). In one example, each page of a document can be presented as a different layer on the holographic whiteboard 670. Participants can select which layer to view in order to read a particular page. Participants can also independently move pages around within the 3D volume of the holographic whiteboard. Thus, participants can view any combination of pages (e.g., side by side). Similarly, participants can position corresponding pages of two different versions of a document and easily compare them. As another example, the LF processing engine 530 can obtain a set of blueprints for a building and use them to create an approximate 3D model of the building. Participants can then interact with the holographic whiteboard 670 to, for example, view different cross sections of the building, remove particular elements from the displayed model (e.g., wiring, doors, particular walls, etc.), make modifications to the model (which may then be used to generate updated blueprints), add annotations, etc.

[0172] The LF display system 600 can modify images of participants and / or other holographic objects. In one embodiment, to enhance the impression that the holographic object is located within the conference space 610, the LF display system 600 determines lighting parameters (e.g., overall brightness and spectral distribution) within the conference space 610 and adjusts the holographic image to better match the lighting parameters. For example, if the conference space 610 has a bright light on the left side, the holographic image can be adjusted so that the left side is brighter while the right side is in shadow. This can enhance the impression that the image is a physical object located within the conference space 610.

[0173] The LF display system 600 may also adjust sound parameters for the generated audible sound to account for the acoustics within the conference space 610. For example, if one corner typically has a muffled sound, the LF display system can apply an equalization boost to the high frequency band of the acoustic energy directed toward that corner to provide a clearer sound. The LF display system may also provide noise cancellation. For example, one or more LF display modules 620 at the back of the conference space 610 may emit sound waves that partially or completely cancel sound waves emitted from LF display modules 620 at the front of the conference space 610.

[0174] Light Field Display Video Chat System FIG. 7 illustrates an alternative configuration of a conference space 710 including an LF display system 700, according to one or more embodiments. The LF display system 700 is one embodiment of the LF display system 500. In FIG. 7, rather than generating holographic images of a table and chairs located behind the display surface, the LF display system 700 generates holographic images of a table 730 and a chair 742 in front of the display surface. Thus, the physical table 730 and chair 740 in the conference space 710 are not positioned opposite the LF display module 720, as they were in FIG. 6. To do so, the LF display system translates the holographic images by projecting them so that they appear in front of the LF display module 720, rather than behind it (as they were in FIG. 6). Similarly, at the other physical location (where the table 732 and chair 742 are located), holographic images of the table 730 and chair 740 can be formed by translating those images by projecting them so that they also appear in front of the display surface at the other physical location.

[0175] This translation of the images enhances the impression that all of the holographic images (including the participants) are located in the same physical space. Rather than functioning as a window that participants can see through but cannot pass through, the LF display module 720 projects the images of the remote participants into the conference space 710. Thus, participants walk around and approach each other throughout. Additionally, ultrasound tactile surfaces can be used to simulate remote participants touching each other. Although not shown in FIG. 7, the conference space 710 may also include additional holographic objects, such as holographic handouts, whiteboards, etc., as described above with reference to FIG. 6.

[0176] FIG. 8A is an illustration of an LF display system 800 presenting holographic content including holographic video chat participants, according to one or more embodiments. LF display system 800 is one embodiment of LF display system 500. LF display system 800 has an LF display module 820 of an LF display assembly that forms a one-sided seamless surface environment. LF display system 800 provides video chat functionality in which participants view holographic images of some or all of the other participants generated by LF display module 820. LF display system 800 may also include any combination of the other components of LF display system 500, such as sensory feedback assembly 570, tracking system 580, viewer profiling module 528, and controller 520. In other embodiments, LF display system 900 includes an additional camera separate from the tracking system for capturing image data.

[0177] 8A , a first video chat participant 830 is located within the viewing zone of an LF display system 800. The LF display system 800 captures image data of the first participant 830 via any combination of a tracking system (e.g., tracking system 580), an LF display module 820, one or more cameras included in the LF display system 800, and any additional tracking devices. In some cases, the LF display system 800 receives image data of the first participant 830 from multiple viewpoints, such as from one or more LF display modules 820 or from one or more cameras separate from or as part of the tracking system. In some embodiments, the LF display system 800 has cameras around the first participant 830 that capture image data encompassing a full 360° viewpoint. Alternatively, the LF display system 800 may be "one-way," meaning that a first participant 830 is presented with a holographic representation of a second participant 840, but the LF display system 800 does not collect image data of the first participant 830 and / or provide such image data to a display device being used by the second participant 840.

[0178] The LF display system 800 generates holographic content, including a holographic representation of the second video chat participant 840. The holographic representation of the second video chat participant 840 is a holographic image. In one embodiment, the LF display module 820 is a two-way surface that also collects image data of the first participant 830 and transmits the image data to the LF display system of the second participant 840 for presentation as a holographic image. In other words, the LF display module 820 can function as a window through which participants can see each other. Similar to the video conferencing system described with reference to FIG. 6 , there can be a one-to-one correspondence between the gaze directions between participants, allowing them to make eye contact with each other as if they were located in the same physical space. Alternatively, the LF display system 800 can be “one-way,” in which only one of the participants is presented with a holographic representation of the other.

[0179] FIG. 8B is an illustration of the LF display system 800 of FIG. 8A presenting holographic content, including a holographic image of a second video chat participant 850, in accordance with one or more embodiments. Similar to the video conferencing configuration illustrated in FIG. 7, the LF display system 800 projects the holographic image of the second participant 850 within the same physical space as the first participant 730. Thus, in contrast to the example shown in FIG. 8A, the first participant 830 experiences the impression that the second participant 850 is in the same physical location as them. To improve the overall experience, the same or similar techniques as described above for adjusting lighting and / or sound parameters may be applied. Additionally, as described above, an ultrasound projection system may be used to create volumetric haptic surfaces for some or all of the participants' holographic representations. These haptic surfaces may simulate participants touching each other, further creating the impression that the participants are located within the same physical space. The ultrasound projection system may operate in conjunction with the tracking system 580 to update the haptic surfaces. For example, using this approach, remote participants in a video chat can experience the sensation of shaking hands with each other, with each participant's experience updated based on the other participant's hand movements thanks to the haptic surface.

[0180] The LF display system 800 may also allow video chat participants to make changes to their appearance and / or voice. In one embodiment, the LF display system 800 provides controls (e.g., physical or holographic) that allow participants to adjust parameters of the holographic representation of themselves that is presented to other participants. For example, participants can blur their faces, select a custom background, add balloons or other objects that float around them, make them appear to be sitting behind a desk, etc. Similarly, the LF display system 800 may allow participants to apply filters to modify their voices, such as to raise or lower the pitch, make the voice sound robotic, selectively reduce the intensity of certain frequencies, etc.

[0181] In other embodiments, the LF display system 800 provides controls (e.g., physical or holographic) that allow participants to apply filters to change aspects of their own or other participants' appearance. For example, a filter may change a participant's clothing, change a participant's hair color, replace a participant's image with an avatar that is mapped to the participant's movements, and / or apply any other visual filter to how a participant looks. In one embodiment, the filters are modular, and participants may install filters they want to use (e.g., by downloading them from a web store, etc.). In some cases, additional filters may be provided by third parties. Such third-party filters may be free and / or offered for a fee (e.g., through a marketplace).

[0182] FIG. 9 is an illustration of an LF display system 900 presenting holographic content including holographic representations of participants in a group video chat, according to one or more embodiments. The LF display system 900 is one embodiment of the LF display system 500. A first video chat participant 930 is in the same physical location as the LF display system 900. The LF display system includes an LF display module 920 that is generating a holographic image representation of a participant 950 who is active in a voice chat. The LF display module 920 is also generating holographic image representations of one or more additional participants 960. The representations may be images of the participants and / or avatars of the participants. Although two additional participants 960 are shown in FIG. 9, the voice chat may include any number of additional participants (including zero). Additionally, although the representations of the additional participants 960 are shown reduced in size, in other embodiments, other indicators may be used for the active participants 950, such as displaying the representation of the active participant 950 in a particular location (e.g., centered), displaying a particular visual indication along with the representation of the active participant 950 (e.g., a glow over the representation of the participant's head), or any other suitable indicator. In some embodiments, not all participants are represented by holographic representations. For example, participants who do not have access to (or choose not to use) an LF camera may instead be represented by a static avatar.

[0183] The LF display system 900 may use any suitable method to determine which participants are the active participants 950. In one embodiment, the LF display system identifies the participant currently speaking as the active participant 950 based on collected acoustic energy. In another embodiment, the first participant 930 can change the active participant 950 using a control device provided by the LF display system 900 (e.g., by pointing at one of the holographic representations of the additional participants 960). While FIG. 9 shows the holographic representations standing, this need not be the case. For example, representations of the participants may be presented as sitting around a holographic (or physical) table, with the active participant 950 positioned at the head of the table.

[0184] In various embodiments, the LF display system 500 provides a holographic voicemail feature. This feature operates substantially as described above with respect to video chat, except that the LF display system 500 creates a message for later presentation by collecting audio and visual data corresponding to an individual and storing it (e.g., in data store 530) rather than presenting it elsewhere in substantially real time. For example, an individual can use physical and / or holographic controls to start, pause, and stop recording and then select one or more individuals to receive the recorded message. Later, the recipient can trigger playback of the message using physical and / or holographic controls on the LF display system 500 (which may be different from or the same system as used to record the message). During playback, the LF display system 500 presents a holographic image of the individual who recorded the message along with the corresponding audio content. In some embodiments, the same or similar techniques as described above may be used to modify the individual's appearance and / or voice.

[0185] In one embodiment, the LF display system 500 provides controls that allow the recipient to control the playback of the message, for example, controls for starting, pausing, fast-forwarding, rewinding, and / or ending playback. As the LF display system 500 creates holographic content from the visual data, the recipient can move around and view the message's visual content from different perspectives. Alternatively, the controls may include controls for rotation, translation, zooming in and out, etc., so that the recipient can see different perspectives without moving. Thus, the recipient can play the message multiple times to view the message from different perspectives.

[0186] In some embodiments, a similar playback feature may be provided for recordings of video conference and / or video chat sessions. The LF display system may offer the option for a video communication session to be recorded. Participants (and anyone else given access to the recording) can then play back the recorded session to view the recorded scene from different perspectives.

[0187] Additional configuration information The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations of the LF display system are possible in light of the above disclosure.

[0188] Some portions of this specification describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it is understood that they may be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Further, it has proven convenient at times to refer to arrangements of these operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0189] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some embodiments, the software modules are implemented in a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0190] Embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory, tangible, computer-readable storage medium that may be coupled to a computer system bus, or any type of medium suitable for storing electronic instructions. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0191] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information resulting from the computing processes, where the information is stored on a non-transitory, tangible, computer-readable storage medium, and may include any embodiment of the computer program product or other data combination described herein.

[0192] Finally, the language used herein has been chosen primarily for ease of reading and instructional purposes, and may not be chosen to define or limit the subject matter of the present disclosure. Accordingly, the scope of the present disclosure is intended to be limited not by this detailed description, but by any claims that issue on an application based thereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not limit, the scope of the present disclosure, which is set forth in the following claims.

Claims

[Claim 1] 1. A light field display system for video communication, comprising: a local light field display assembly; a controller configured to generate display instructions based on visual data corresponding to the remote scene; Equipped with the visual data is received from a remote image capture system; the display instructions cause the local light field display assembly to generate a holographic image of the remote scene; the local light field display assembly is configured to generate visual data corresponding to a local scene and to transmit the visual data corresponding to the local scene to a remote light field display system; the local light field display assembly includes an interactive surface that captures image data corresponding to the local scene and simultaneously projects holographic content corresponding to the remote scene generated from the display instructions; The light field display system comprises: a data store that stores a viewer profile indicating viewer preferences regarding the presentation of holographic content; Furthermore, the holographic content corresponding to the remote scene is projected in response to the viewer profile of a viewer in the local scene. Light field display system.

Images