Light field display for mobile device

Abstract

To solve the problem in which 2D visual display offers limited opportunities to provide a user with an immersive operational experience.SOLUTION: A light field display system is implemented in a mobile device to present to a user, holographic content which includes at least one holographic object, and provides the user with an immersive operational experience. The system generates and presents holographic content for the user. In one embodiment, the system receives a command from the user. In some embodiments, the presented holographic content may comprise a holographic user interface that is used by the system to receive the commands from the user of the mobile device. Subsequently, the system recognizes the received commands, determines one or more computational commands for execution by the system, and executes the determined computational command.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042414, and PCT / US2017 / 042679, all of which are incorporated by reference in their entireties. [Technical field]

[0002] The present disclosure relates to mobile devices such as smartphones and tablets, and in particular to light field displays for mobile devices.

[0003] Mobile devices generally rely on two-dimensional (2D) visual displays for content presentation and user interface. In addition to displaying two-dimensional content such as text, images, and videos, the 2D visual displays of mobile devices are designed to provide an interactive interface to display various features that the mobile device presents to the user and allow the user to select and configure the desired features. Interactivity on mobile devices is provided using physical elements such as buttons, keyboards, and touch interfaces on the 2D visual displays. However, the size of the 2D visual displays is limited to the footprint that the displays occupy on the surface of the mobile device. As a result, the display content is also limited to the size of the 2D visual displays, limiting the interactivity and viewing experience provided to mobile device users. In particular, 2D visual displays offer limited opportunities to provide users with an immersive interaction experience.

[0004] The described embodiments relate to techniques including a light field display system in a mobile device that provides an immersive interaction experience to a user while using the mobile device. Summary of the Invention

[0005] A light field (LF) display system of a mobile device generates holographic content for viewing by a user of the mobile device. The generated holographic content may be a holographic user interface that is displayed to the user of the mobile device. The light field display system may receive commands from the user. The LF display system determines computational commands based on the received user commands and appropriately executes the determined computational commands. In some embodiments, the holographic content is augmented with a tactile surface over the generated holographic content. The tactile surface may provide a haptic sensation to the user.

[0006] In some embodiments, a mobile device comprises an LF display system including a controller configured to generate holographic content, an LF display assembly, and a command interface, The LF display assembly comprises one or more LF display modules configured to present holographic content in a holographic object volume to one or more users of the mobile device positioned within a viewing volume of the one or more LF display modules.

[0007] In some embodiments, the LF display system includes a tracking system and / or a viewer profiling module. The tracking system and viewer profiling module can monitor and store characteristics of a user of a mobile device, a user profile describing the user, and / or the user's responses to holographic content presented during operation of the mobile device. Holographic content can be created for display based on either the monitored or stored information.

[0008] In some embodiments, a user of the mobile device can interact with the holographic content, and the interaction can serve as an input for the holographic content creation system. In some embodiments, the LF display system of the mobile device can generate sensory stimuli in addition to the holographic content. For example, in some embodiments, some or all of the LF display system includes one or more ultrasonic speakers. The ultrasonic speakers can be configured to generate a tactile surface that matches at least a portion of the holographic content. A tracking system is configured to track the user's interaction with the holographic object (e.g., via images captured by an image sensor and / or some other camera of the LF display module). The LF display system is then configured to provide the holographic content based on the interaction. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram of a light field display module for presenting holographic objects in accordance with one or more embodiments.

[0010] [Figure 2A] 1 is a cross-section of a portion of a light field display module according to one or more embodiments.

[0011] [Figure 2B] 1 is a cross-section of a portion of a light field display module according to one or more embodiments.

[0012] [Figure 3A] FIG. 1 is a perspective view of a light field display module according to one or more embodiments.

[0013] [Figure 3B] 1 is a cross-sectional view of a light field display module including an interleaved energy relay device according to one or more embodiments.

[0014] [Figure 4A] FIG. 1 is a perspective view of a portion of a light field display system that is two-dimensionally tiled to form a single-sided seamless surface environment in accordance with one or more embodiments.

[0015] [Figure 4B] FIG. 1 is a perspective view of a portion of a light field display system in a multi-surface seamless surface environment in accordance with one or more embodiments.

[0016] [Figure 4C] FIG. 2 is a top view of a light field display system having a collective surface in a wing-type configuration in accordance with one or more embodiments.

[0017] [Figure 4D] FIG. 2 is a side view of a light field display system having a collection surface in a tilted configuration in accordance with one or more embodiments.

[0018] [Figure 4E] FIG. 2 is a top view of a light field display system having a collection surface on the front panel of the device in accordance with one or more embodiments.

[0019] [Diagram 5] FIG. 1 is a block diagram of a light field display system in accordance with one or more embodiments.

[0020] [Figure 6] FIG. 1 is a block diagram of a network environment for a mobile device light field display system in accordance with one or more embodiments.

[0021] [Figure 7] 1 is an example display of an embodiment using a light field display on a mobile device.

[0022] [Figure 8] 1 is a flowchart illustrating a method for enabling interaction with a mobile device using a light field display system, in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] A light field (LF) display system is implemented in a mobile device to present holographic content, including at least one holographic object, to a mobile device user. The terms "user," "mobile device user," "device user," and "viewer" are used interchangeably herein. The terms "mobile device" and "device" are also used interchangeably herein and refer to handheld electronic computing devices designed for portability and compactness. Examples of mobile devices include, but are not limited to, mobile phones, smartphones, PDAs, tablets, e-book readers, portable music players, portable computers, and the like.

[0024] To provide an immersive operational experience to a user using the mobile device, the LF display system of the mobile device is integrated with the functionality of the device. The functionality of the mobile device is realized by executing computer program modules, such as a web browser and a software client application program. This allows the device user to browse the Internet, consume media content, and execute various software applications. The LF display system comprises an LF display assembly configured to present holographic content including one or more holographic objects that would be visible to a viewer within a viewing volume of the LF display system. The holographic content presented by the LF display system is integrated with the functionality of the device. This enhances the immersive experience of the device user while executing the software application on the device. The holographic content presented by the LF display system may also be augmented by other sensory stimuli (e.g., tactile, acoustic, temperature, pressure, force, or olfactory). For example, an ultrasound emitter of the LF display system may project ultrasound pressure waves that create a volumetric tactile projection. The volumetric tactile projection provides a tactile surface that corresponds to some or all of the projected holographic object. The holographic content may include additional visual content (i.e., 2D or 3D visual content). Coordination of energy sources to enable a cohesive experience is part of the LF system for implementations using multiple energy sources (i.e., holographic objects that provide the correct haptic and sensory stimuli at any given time). For example, the LF system may include a controller for coordinating the presentation of holographic content and haptic surfaces.

[0025] In some embodiments, an LF display system may include elements that allow it to project at least one type of energy and simultaneously sense at least one type of energy to respond to the user and create an interactive experience. The sensed energy may be used to record how the viewer responds to the holographic content. For example, an LF display system may project both holographic objects for viewing and ultrasound for tactile perception, while simultaneously recording image information for user tracking and other scene analysis, while sensing ultrasound to detect touch responses by the user. As an example, such a system may project holographic objects that, when virtually "touched" by a viewer, modify their "behavior" according to the touch stimulus. The components of the display system that perform the energy sensing of the environment may be integrated into the display surface or may be dedicated sensors separate from the display surface, such as ultrasound speakers and imaging capture devices such as cameras.

[0026] The LF display system may be constructed to provide a variety of experiences on a mobile device based on the generated holographic objects. For example, in one embodiment, an LF display system on a mobile device may be implemented to present a user with a holographic view of the terrain via a map display application. The LF display assembly may form a single-surface or multi-surface seamless surface environment. The holographic content may extend or enhance a physical object such as the mobile device screen itself. For example, in one embodiment, an LF display system on a device may be implemented to present a 3D holographic progress indicator gif, such as an enveloping sphere, to indicate that an application is loading on the device. Additionally, the user may freely view the holographic content without the need for eyewear devices such as 3D glasses or other headgear.

[0027] In some embodiments, the LF display system can receive input via a tracking system and / or a sensory feedback system. Based on the input, the LF display system can adjust the holographic content as well as provide feedback to associated components.

[0028] The LF display system may also incorporate a system for tracking the movement of a user within the viewing volume of the LF display system. The tracked movement of the user may be used to enhance the experience of a user using a mobile device. For example, the LF display system may use the tracking information to facilitate the user's interaction with the holographic content (e.g., pressing a holographic button). The LF display system may use the tracking information to monitor the position of a finger relative to a holographic object. For example, the holographic object may be a button that the viewer can "press." The LF display system may project ultrasonic energy to generate a tactile surface that corresponds to the button and occupies substantially the same space as the button. The LF display system may use the tracking information to dynamically move the button as well as dynamically move the position of the tactile surface as the button is "pressed" by the viewer. The LF display system may use the tracking information to render a holographic character that looks at the user and / or makes eye contact with the user or otherwise interacts with the user. The LF display system can use the tracking information to render holographic objects that "touch" the user by being projected in close proximity to the user, and ultrasonic speakers create a tactile surface where the holographic objects generate haptic sensations for the user.

[0029] Additionally, the LF display system may incorporate a user profiling system to identify each user in order to provide personalized content to each device user. The user profiling system may further record information about the use of the device (e.g., type of content viewed, locations navigated) that can be retrieved upon subsequent use in order to personalize the holographic content. Light Field Display System Overview

[0030] FIG. 1 is a diagram 100 of a light field (LF) display module 110 presenting a holographic object 120, according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system presents holographic content, including at least one holographic object, using one or more LF display modules. The LF display system can present the holographic content to one or more viewers. In some embodiments, the LF display system can also augment the holographic content with other sensory content (e.g., touch, sound, smell / olfaction, pressure, force, or temperature, etc.). For example, as discussed below, the projection of focused ultrasound can generate hollow haptics that may simulate some or all of the surface of a holographic object. The LF display system includes one or more LF display modules 110 and is discussed in detail below with respect to FIGS. 2-8.

[0031] The LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more viewers (e.g., viewer 140). The LF display module 110 includes an energy device layer (e.g., a light-emitting electronic display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). Additionally, the LF display module 110 may include an energy relay layer to combine multiple energy sources or detectors together to form a single surface. At a high level, the energy device generates energy (e.g., holographic content), which is then directed to an area in space according to one or more four-dimensional (4D) light field functions using an energy waveguide. The LF display module 110 may also simultaneously project and / or sense one or more types of energy. For example, the LF display module 110 may be capable of projecting holographic images as well as an ultrasound tactile surface into a viewing volume while simultaneously detecting image data from the viewing volume. The operation of the LF display module 110 is discussed in detail below with respect to FIGS. 2-3.

[0032] The LF display module 110 generates a holographic object within the holographic object volume 160 using one or more 4D light field functions (e.g., derived from a plenoptic function). The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Additionally, the holographic object may be multi-colored (e.g., full color). The holographic object may be projected in front of a screen plane, behind a screen plane, or segmented by a screen plane. The holographic object 120 may be presented to be perceived anywhere within the holographic object volume 160. The holographic object within the holographic object volume 160 may appear to the viewer 140 as floating in space.

[0033] The holographic object volume 160 represents a volume in which a holographic object can be perceived by the viewer 140. The holographic object volume 160 may extend in front of the surface of the display surface 150 (i.e., toward the viewer 140) so that the holographic object can be presented in front of the plane of the display area 150. Additionally, the holographic object volume 160 may extend behind the plane of the display area 150 (i.e., away from the viewer 140), allowing the holographic object to be presented as if it were behind the plane of the display area 150. In other words, the holographic object volume 160 can include all light rays that originate (e.g., projected) from the display area 150 and converge to create a holographic object. Here, the light rays may converge to a point in front of, at, or behind the display surface. In simpler terms, the holographic object volume 160 encompasses all volumes in which a holographic object can be perceived by the viewer.

[0034] The viewing volume 130 is the volume of space in which a holographic object (e.g., holographic object 120) presented by the LF display system in the holographic object volume 160 is fully visible. The holographic object may be presented in the holographic object volume 160 and viewed in the viewing volume 130 such that it is indistinguishable from the actual object. The holographic object is formed by projecting the same light rays that would be generated from the surface of the object if it physically existed.

[0035] In some cases, the holographic object volume 160 and corresponding viewing volume 130 may be relatively small, such as designed for a single viewer. In other embodiments, for example, as described in detail below with respect to Figures 4A and 4B, the LF display modules may be scaled and / or tiled to create larger holographic object volumes and corresponding viewing volumes that can accommodate a wide range of viewers (e.g., from one person to several thousand people). The LF display modules presented in this disclosure are constructed such that the entire surface of the module includes holographic imaging optics, with no inactive or dead space and no need for bezels. In these embodiments, the LF display modules may be tiled such that the imaging area is continuous across the seams between the LF display modules, and the join lines between the tiled modules are substantially undetectable using eye vision. Notably, in some configurations, portions of the display surface may not include holographic imaging optics, not described in detail herein.

[0036] The flexible size and / or shape of the viewing volume 130 allows the viewer 140 to be unconstrained within the viewing volume 130. For example, the viewer 140 can move to different positions within the viewing volume 130 and see different views of the holographic object 120 from corresponding viewpoints. For illustration, with reference to FIG. 1 , the viewer 140 is in a first position relative to the holographic object 120 such that the holographic object 120 is seen in a head-on view of the dolphin. The viewer 140 can move to other locations relative to the holographic object 120 to see different views of the dolphin. For example, the viewer 140 can move to see the left side of the dolphin, the right side of the dolphin, etc., as if the viewer 140 were looking at an actual dolphin and change the viewer's relative position with respect to the actual dolphin to see different sides of the dolphin. In some embodiments, the holographic object 120 is visible to all viewers within the viewing volume 130 who have an unobstructed (i.e., unobstructed by an object / person) line of sight to the holographic object 120. These viewers may be untethered so that they can move around within the viewing volume to see different perspectives of the holographic object 120. Thus, the LF display system can present a holographic object such that multiple untethered viewers can simultaneously view different perspectives of the holographic object in real-world space as if the holographic object were physically present.

[0037] In contrast, traditional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) generally require each viewer to wear some external device (e.g., 3D glasses, near-eye display, or head-mounted display) to view the content. Additionally and / or alternatively, traditional displays may require the viewer to be constrained to a particular viewing position (e.g., in a chair with a fixed location relative to the display). For example, when viewing an object displayed by a stereoscopic display, the viewer always focuses on the display surface rather than the object, and the display always presents only two views of the object that follow the viewer as they try to move around the perceived object, causing distortions in the perception of that object. However, in light field displays, a viewer of a holographic object presented by an LF display system does not need to wear an external device to view the holographic object. The LF display system displays the holographic object such that the holographic object appears to the viewer in much the same way that a physical object appears to the viewer, without the need for special eyewear, glasses, or head-mounted accessories. Furthermore, the viewer can view the holographic content from any location within the viewing volume.

[0038] In particular, the potential locations of holographic objects within the holographic object volume 160 are limited by the size of the volume. To increase the size of the holographic object volume 160, the size of the display area 150 of the LF display module 110 may be increased and / or multiple LF display modules may be tiled together to form a seamless display surface with an effective display area larger than the display area of ​​an individual LF display module. Some embodiments related to tiling of LF display modules are discussed below with respect to Figures 4A and 4B. As illustrated in Figure 1, the display area 150 is rectangular, resulting in a holographic object volume 160 that is pyramidal. In other embodiments, the display area may have some other shape (e.g., hexagonal) that also affects the shape of the corresponding viewing volume.

[0039] Additionally, while the above discussion has focused on presenting the holographic object 120 within the portion of the holographic object volume 160 that is between the LF display module 110 and the viewer 140, the LF display module 110 can also present content that appears behind the plane of the display area 150. For example, the LF display module 110 can make a portion of the displayed holographic object appear to make the display area 150 appear to be the surface of the ocean with the holographic object 120 popping out of it. The displayed content can be such that the viewer 140 can see underwater marine life through the displayed surface. Additionally, the LF display system can generate content that moves seamlessly around the holographic object volume 160, including behind and in front of the plane of the display surface 150.

[0040] 2A is a cross-section 200 of a portion of an LF display module 210, according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a different display area shape than the display area 150. In an exemplary embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 have different components than those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be distributed among components in a different manner than described herein.

[0041] The display system described herein presents an emission of energy that replicates the energy that typically surrounds real-world objects, where the emitted energy is directed in a specific direction from every coordinate on the display surface. In other words, various coordinates on the display surface act as projection locations for the emitted energy. The directed energy from the display surface allows for the convergence of many energy rays, thereby creating a holographic object. For example, in visible light, an LF display projects a very large number of light rays to converge at any point within the holographic object volume, so that from the perspective of a viewer located farther away than the object being projected, the light rays will appear to come from the surface of a real-world object located in this region of space. In this way, the LF display is generating reflected light rays that, from the viewer's perspective, appear to come from the surface of such an object. The viewer's viewpoint may be changed with respect to any holographic object, so that the viewer sees a different view of the holographic object.

[0042] The energy device layer 220 includes one or more electronic displays (e.g., light-emitting displays such as OLEDs) and one or more other energy projection and / or energy receiving devices described herein. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include multiple pixels, each having an intensity that is individually controlled. Many types of commercial displays can be used within the LF display, such as light-emitting LED and OLED displays.

[0043] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic projection devices generate one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or some combination thereof. An array of ultrasonic pressure waves may be used (e.g., at the surface of the holographic object 250) for volumetric haptics. The audible pressure waves are used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming the holographic object 250 is a dolphin, one or more acoustic projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the dolphin so that the viewer can touch the holographic object 250, and (2) provide audio content corresponding to the sounds the dolphin makes, such as clicking, chirping, shrill calls, etc. An acoustic receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audible pressure waves within a localized area of ​​the LF display module 210.

[0044] The energy device layer 220 may also include one or more image sensors. The image sensors are sensitive to light in at least the visible light band, and may possibly be sensitive to light in other bands (e.g., infrared). The image sensors may be, for example, a complementary metal oxide semiconductor (CMOS) array, a charge-coupled device (CCD), an array of photodetectors, some other sensor that captures light, or some combination thereof. The LF display system may use data captured by the one or more image sensors for viewer location tracking.

[0045] The energy relay layer 230 relays energy (e.g., electromagnetic energy, mechanical pressure waves, etc.) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270 and relays energy between the two surfaces. The first surface 265 of each energy relay element may be coupled to one or more energy devices (e.g., electronic displays or acoustic projection devices). The energy relay elements may be composed of, for example, glass, carbon, optical fiber, optical films, plastics, polymers, or some combination thereof. Additionally, in some embodiments, the energy relay elements may adjust the magnification (increase or decrease) of the energy passing between the first surface 265 and the second surface 270. If the relay provides magnification, the relay may take the form of an array of bonded tapered relays, called tapers, where one end of the taper may be substantially larger in area than the opposite end. The larger ends of the tapers can be tiled together to form a seamless energy surface 275. One advantage is that space is created at the smaller ends of each taper to accommodate the mechanical envelope of multiple energy sources, such as the bezels of multiple displays. This extra space allows energy sources to be placed side-by-side on the smaller taper side, with each energy source having an active area within the smaller taper surface that directs energy and is relayed to the larger seamless energy surface. Another advantage of using a taper relay is that the combined seamless energy surface formed by the larger ends of the tapers has no non-imaging dead space. Because there are no borders or bezels, the seamless energy surfaces can be tiled together to form a larger surface that is substantially seamless according to the visual acuity of the eye.

[0046] The second surfaces of adjacent energy relay elements together form an energy plane 275. In some embodiments, the separation between the edges of adjacent energy relay elements is less than the minimum perceptible contour defined, for example, by the visual acuity of a human eye having better than 20 / 40 vision, such that the energy plane 275 is virtually seamless from the perspective of a viewer 280 within the viewing volume 285.

[0047] In some embodiments, the second surfaces of adjacent energy relay elements are fused together using a process that may include one or more of pressure, heat, and chemical reaction such that there are no seams between them. Yet, in other embodiments, the array of energy relay elements is formed by molding one side of a continuous block of relay material into an array of small tapered ends, each configured to transport energy from an energy device attached to the small tapered end into a single bonded surface having a large undivided area.

[0048] In some embodiments, one or more energy relay elements exhibit energy localization, such that the efficiency of energy transport in a longitudinal direction substantially perpendicular to surfaces 265 and 270 is much higher than the efficiency of transport in a perpendicular cross-plane, and the energy density is highly localized in this cross-plane as an energy wave propagates between surfaces 265 and 270. This energy localization allows energy distributions, such as images, to be efficiently relayed between these surfaces without significant loss of resolution.

[0049] The energy waveguide layer 240 uses waveguide elements in the energy waveguide layer 240 to direct energy from locations (e.g., coordinates) on the energy plane 275 into specific propagation paths from the display surface outward into the holographic viewing volume 285. The energy propagation paths are defined by at least two angular dimensions determined by the coordinate location of the energy plane relative to the waveguide. The waveguide is associated with spatial 2D coordinates. These four coordinates together form a four-dimensional (4D) energy field. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 direct light from locations on the seamless energy plane 275 along different propagation directions through the viewing volume 285. In various examples, the light is directed according to a 4D light field function to form a holographic object 250 in the holographic object volume 255.

[0050] Each waveguide element in the energy waveguide layer 240 may be, for example, a lenslet composed of one or more elements. In some configurations, the lenslet may be a positive lens. The positive lens may have a spherical, aspheric, or freeform surface profile. Additionally, in some embodiments, some or all of the waveguide elements may include one or more additional optical components. The additional optical components may be, for example, energy suppression structures such as baffles, positive lenses, negative lenses, spherical lenses, aspheric lenses, freeform lenses, liquid crystal lenses, liquid lenses, refractive elements, diffractive elements, or some combination thereof. In some embodiments, at least one of the lenslets and / or additional optical components may also dynamically adjust refractive power. For example, the lenslet may be a liquid crystal lens or a liquid lens. The dynamic adjustment of the surface profile, the lenslet and / or the at least one additional optical component may provide additional directional control of the light projected from the waveguide element.

[0051] In an illustrative example, the holographic object volume 255 of the LF display has a boundary formed by light rays 256 and 257, but may be formed by other light rays. The holographic object volume 255 is a continuous volume that extends both in front of the energy waveguide layer 240 (i.e., toward the viewer 280) and behind it (i.e., away from the viewer 280). In an illustrative example, light rays 256 and 257 are projected from opposite edges of the LF display module 210 at the largest angle relative to the normal of the display surface 277 that can be perceived by the user, but they may be other projected light rays. These light rays define the field of view of the display and therefore the boundary of the holographic viewing volume 285. In some cases, these light rays define a holographic viewing volume (e.g., an ideal viewing volume) where the entire display may be observed without vignetting. As the field of view of the display increases, the convergence point of light rays 256 and 257 becomes closer to the display. Thus, a display with a wider field of view allows viewer 280 to view the entire display at a closer viewing distance. Additionally, light rays 256 and 257 may form an ideal holographic object volume. A holographic object presented within the ideal holographic object volume may be viewed anywhere within viewing volume 285.

[0052] In some examples, the holographic object may be presented in only a portion of the viewing volume 285. In other words, the holographic object volume may be divided into any number of viewing sub-volumes (e.g., viewing sub-volume 290). Additionally, the holographic object may be projected outside the holographic object volume 255. For example, the holographic object 251 is presented outside the holographic object volume 255. Because the holographic object 251 is presented outside the holographic object volume 255, it is not visible from all locations within the viewing volume 285. For example, the holographic object 251 may be visible from locations within the viewing sub-volume 290, but not from locations of the viewer 280.

[0053] For example, to illustrate viewing holographic content from different viewing sub-volumes, turn to FIG. 2B. FIG. 2B illustrates a cross section 200 of a portion of an LF display module, according to one or more embodiments. The cross section of FIG. 2B is the same as the cross section of FIG. 2A. However, FIG. 2B shows a different set of light rays projected from the LF display module 210. Light rays 256 and light rays 257 still form the holographic object volume 255 and the viewing volume 285. However, as shown, the light rays projected from the top of the LF display module 210 and the bottom of the LF display module 210 overlap to form various viewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D) within the viewing volume 285. A viewer in a first viewing sub-volume (e.g., 290A) may be able to perceive holographic content presented in holographic object volume 255 that is not perceivable by viewers in the other viewing sub-volumes (e.g., 290B, 290C, and 290D).

[0054] More simply, as illustrated in FIG. 2A, holographic object volume 255 is a volume in which a holographic object may be presented by the LF display system such that the holographic object may be perceived by a viewer (e.g., viewer 280) in viewing volume 285. As such, viewing volume 285 is an example of an ideal viewing volume, and holographic object volume 255 is an example of an ideal object volume. However, in various configurations, a viewer may perceive a holographic object presented by LF display system 200 in other exemplary holographic object volumes as a viewer in other exemplary viewing volumes perceives holographic content. More generally, "eyeline guidelines" are applied when viewing holographic content projected from an LF display module. Eyeline guidelines assert that the line formed by the viewer's eye position and the holographic object being viewed necessarily intersects the LF display surface.

[0055] When viewing the holographic content presented by the LF display module 210, each eye of the viewer 280 will see a different perspective of the holographic object 250 because the holographic content is presented according to a 4D light field function. Furthermore, as the viewer 280 moves within the viewing volume 285, the viewer will see different perspectives of the holographic object 250 as other viewers within the viewing volume 285 would see. As will be appreciated by those skilled in the art, 4D light field functions are well known in the art and will not be described in further detail herein.

[0056] As described in more detail herein, in some embodiments, the LF display can project more than one type of energy. For example, the LF display can project two types of energy, such as acoustic energy and electromagnetic energy. In this configuration, the energy relay layer 230 includes two separate energy relays that are interleaved together at the energy plane 275, but are separated such that the energy is relayed to two different energy device layers 220. Here, one relay can be configured to transport electromagnetic energy. The other relay can be configured to transport mechanical energy. In some embodiments, the mechanical energy can be projected from a location between the electromagnetic waveguide elements on the energy waveguide layer 240, which helps to form a structure that inhibits light from being transported from one electromagnetic waveguide element to the other. In some embodiments, the energy waveguide layer 240 can also include waveguide elements that transport focused ultrasound along specific propagation paths according to display instructions from the controller.

[0057] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy plane 275 is an output plane formed using one or more adjacent electronic displays in the energy device layer 220. Also, in some embodiments, the separation between the edges of adjacent electronic displays is less than the smallest perceptible contour defined by the visual acuity of a human eye with 20 / 40 vision, such that the energy plane is virtually seamless from the perspective of a viewer 280 within the viewing volume 285. LF Display Module

[0058] FIG. 3A is a perspective view of an LF display module 300A according to one or more embodiments. The LF display module 300A may be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 300A may be some other LF display module. In an exemplary embodiment, the LF display module 300A includes an energy device layer 310, an energy relay layer 320, and an energy waveguide layer 330. The LF display module 300A is configured to present holographic content from a display surface 365 as described herein. For convenience, the display surface 365 is illustrated as a dashed outline on the frame 390 of the LF display module 300A, but more precisely, it is the surface directly in front of the waveguide element bounded by the inner rim of the frame 390. The display surface 365 includes multiple projection positions to which energy may be projected. Some embodiments of the LF display module 300A have different components than those described herein. For example, in some embodiments, the LF display module 300A does not include the energy relay layer 320. Similarly, functionality may be distributed among components in different ways than described herein.

[0059] Energy device layer 310 is one embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three are visible in the figure). The energy devices 340 may all be of the same type (e.g., all electronic displays) or may include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0060] The energy relay layer 320 is one embodiment of the energy relay layer 230. The energy relay layer 320 includes four energy relay elements 350 (three are visible in the figure). The energy relay devices 350 may all relay the same type of energy (e.g., light) or may relay one or more different types (e.g., light and sound). Each of the relay elements 350 includes a first surface and a second surface, where the second surfaces of the energy relay elements 350 are arranged to form a single seamless energy surface 360. In an exemplary embodiment, each of the energy relay elements 350 is tapered such that the first surface has a smaller surface area than the second surface, which allows the mechanical envelope of the energy device 340 to be housed at the smaller end of the taper. This allows the seamless energy surface to be borderless, since the entire area may project energy. This means that this seamless energy surface can be tiled by placing multiple instances of 300 together with no dead space or bezels such that the entire combined surface is seamless. In other embodiments, the first surface and the second surface have the same surface area.

[0061] The energy waveguide layer 330 is one embodiment of the energy waveguide layer 240. The energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIG. 2, the energy waveguide layer 330 is configured to direct energy from a seamless energy plane 360 ​​along specific propagation paths according to a 4D light field function to form a holographic object. Note that in an exemplary embodiment, the energy waveguide layer 330 is bounded by a frame 390. In other embodiments, the frame 390 is not present and / or the thickness of the frame 390 is reduced. Removal or reduction in thickness of the frame 390 may facilitate tiling of the LF display module 300A with additional LF display modules 300.

[0062] Note that in the exemplary embodiment, the seamless energy surface 360 ​​and the energy waveguide layer 330 are planar. In alternative embodiments not shown, the seamless energy surface 360 ​​and the energy waveguide layer 330 may be curved in one or more dimensions.

[0063] The LF display module 300A may be configured with additional energy sources present on the surface of the seamless energy surface, allowing the projection of energy fields in addition to the light field. In one embodiment, an acoustic energy field may be projected from an electrostatic speaker (not illustrated) mounted at any number of locations on the seamless energy surface 360. Additionally, the electrostatic speaker of the LF display module 300A is positioned within the light field display module 300A such that the dual energy surface simultaneously projects a sound field and holographic content. For example, the electrostatic speaker may be formed of one or more diaphragm elements that are transparent to some wavelengths of electromagnetic energy and driven with conductive elements. The electrostatic speaker may be mounted on the seamless energy surface 360 ​​such that the diaphragm elements cover some of the waveguide elements. The conductive electrodes of the speaker may be collocated with a structure designed to inhibit light transmission between the electromagnetic waveguides and / or may be located at a location (e.g., frame 390) between the electromagnetic waveguide elements. In various configurations, the speakers can project multiple sources of focused ultrasonic energy that produce audible sound and / or a tactile surface.

[0064] In some configurations, the energy devices 340 may sense energy. For example, the energy devices may be microphones, light sensors, acoustic transducers, etc. Thus, the energy relay devices may also relay energy from the seamless energy surface 360 ​​to the energy device layer 310. That is, the seamless energy surface 360 ​​of the LF display module forms a bidirectional energy surface when the energy devices and the energy relay devices 340 are configured to simultaneously emit and sense energy (e.g., emit a light field and sense sound).

[0065] More broadly, the energy device 340 of the LF display module 300A can be either an energy source or an energy sensor. The LF display module 300A can include various types of energy devices that function as energy sources and / or energy sensors to facilitate the projection of high quality holographic content to a user. Other sources and / or sensors include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, and the like. Many other sensors or sources are possible. Additionally, the LF display modules can be tiled such that the LF display modules can form an assembly that projects and senses multiple types of energy from a large aggregate seamless energy surface.

[0066] In various embodiments of the LF display module 300A, the seamless energy surface 360 ​​may have various surface portions, with each surface portion configured to project and / or emit a particular type of energy. For example, if the seamless energy surface is a dual energy surface, the seamless energy surface 360 ​​includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy may be located on the seamless energy surface 360 ​​between the waveguide elements and / or may be co-located with structures designed to inhibit optical transmission between the waveguide elements. In examples where the seamless energy surface is a bidirectional energy surface, the energy relay layer 320 may include two types of energy relay devices interleaved in the seamless energy surface 360. In various embodiments, the seamless energy surface 360 ​​may be configured such that the portions of the surface under a particular waveguide element 370 are all energy sources, all energy sensors, or a mix of energy sources and energy sensors.

[0067] FIG. 3B is a cross-sectional view of an LF display module 300B including interleaved energy relay devices, according to one or more embodiments. The energy relay device 350A transports energy between a first surface 345A of an energy relay connected to an energy device 340A and a seamless energy surface 360. The energy relay 350B transports energy between a first surface 345B of an energy relay connected to an energy device 340B and a seamless energy surface 360. Both relay devices are interleaved in an interleaved energy relay device 352 connected to the seamless energy surface 360. In this configuration, the surface 360 ​​includes interleaved energy locations of both energy devices 340A and 340B, which may be energy sources or energy sensors. Thus, the LF display module 300B can be configured either as a dual energy projection device for projecting two or more types of energy, or as a bidirectional energy device for simultaneously projecting one type of energy and sensing another type of energy. LF display module 300B may be LF display module 110 and / or LF display module 210. In other embodiments, LF display module 302 may be some other LF display module.

[0068] The LF display module 300B includes many components configured similarly to those of the LF display module 300A of FIG. 3A. For example, in an exemplary embodiment, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330 that include at least the same functionality as described with respect to FIG. 3A. Additionally, the LF display module 300B can present and / or receive energy from a display surface 365. In particular, the components of the LF display module 300B are alternatively connected and / or oriented differently than the components of the LF display module 300A of FIG. 3A. Some embodiments of the LF display module 300B have different components than those described herein. Similarly, functionality may be distributed among the components in a different manner than described herein. FIG. 3B illustrates a design of a single LF display module 302 that can be tiled to create a dual energy projection surface or a bidirectional energy surface with a larger area.

[0069] In one embodiment, the LF display module 300B is an LF display module of a bidirectional LF display system. The bidirectional LF display system can project energy and simultaneously sense energy from a display surface 365. The seamless energy surface 360 ​​includes both energy projection locations and energy sensing locations closely interleaved on the seamless energy surface 360. Thus, in the example of FIG. 3B, the energy relay layer 320 is configured in a different manner than the energy relay layer of FIG. 3A. For convenience, the energy relay layer of the LF display module 300B is referred to herein as an "interleaved energy relay layer."

[0070] The interleaved energy relay layer 320 includes two legs, a first energy relay device 350A and a second energy relay device 350B. Each of the legs is illustrated as a lightly shaded area. Each of the legs is made of a flexible relay material and can be formed with a length sufficient for use with energy devices of various sizes and shapes. In some areas of the interleaved energy relay layer, the two legs are tightly interleaved together as they approach a seamless energy surface 360. In the illustrative example, the interleaved energy relay device 352 is illustrated as a darkly shaded area.

[0071] The energy relay devices are configured to relay energy between different energy devices while being interleaved in a seamless energy plane 360. The energy devices are in the energy device layer 310. As illustrated, energy device 340A is connected to energy relay device 350A and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device may be an energy source or an energy sensor.

[0072] The energy waveguide layer 330 includes a waveguide element 370 for directing energy waves from the seamless energy surface 360 ​​along a projected path toward a series of convergence points. In this example, a holographic object 380 is formed at the series of convergence points. Notably, as illustrated, the convergence of energy at the holographic object 380 occurs on the viewer side of the display surface 365. However, in other examples, the convergence of energy may be anywhere within the holographic object volume that extends both in front of the display surface 365 and behind the display surface 365. The waveguide element 370 can simultaneously direct the incoming energy to an energy device (e.g., an energy sensor), as described below.

[0073] In one exemplary embodiment of the LF display module 300B, a light-emitting display is used as the energy source and an image sensor is used as the energy sensor. In this manner, the LF display module 300B can simultaneously project holographic content and detect light from the volume in front of the display surface 365. In this manner, this embodiment of the LF display module 300B functions as both an LF display and an LF sensor.

[0074] In one embodiment, the LF display module 300B is configured to project a light field from a projection location of the display surface in front of the display surface 365 while simultaneously capturing a light field from in front of the display surface 365. In this embodiment, the energy relay device 350A connects a first set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340A. In one example, the energy device 340A is an emissive display having an array of source pixels. The energy relay device 340B connects a second set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340B. In one example, the energy device 340B is an image sensor having an array of sensor pixels. The LF display module 302 may be configured such that the locations in the seamless energy surface 365 under a particular waveguide element 370 are all emissive display locations, all image sensor locations, or some combination of locations. In other embodiments, the bidirectional energy surface may project and receive various other forms of energy.

[0075] In another exemplary embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, the energy device 340A is a light-emitting display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. As such, both light and sound can be projected from various positions on the seamless energy surface 360. In this configuration, the energy relay device 350A connects the energy device 340A to the seamless energy surface 360 ​​and relays the electromagnetic energy. The energy relay device is configured to have properties that make the transport of electromagnetic energy efficient (e.g., change the refractive index). The energy relay device 350B connects the energy device 340B to the seamless energy surface 360 ​​and relays the mechanical energy. The energy relay device 350B is configured to have properties for efficient transport of ultrasonic energy (e.g., distribution of materials with different acoustic impedances). In some embodiments, the mechanical energy can be projected from positions between the waveguide elements 370 on the energy waveguide layer 330. The locations that project the mechanical energy may form structures that serve to inhibit light from being transported from one electromagnetic waveguide element to another. In one example, a spatially separated array of locations that project ultrasonic mechanical energy may be configured to create three-dimensional haptic shapes and surfaces in air. The surfaces may coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delays and amplitude variations across the array may assist in creating haptic shapes.

[0076] In various embodiments, the interactive LF display module 302 may include multiple energy device layers, with each energy device layer including a particular type of energy device. In these examples, the energy relay layer is configured to relay the appropriate type of energy between the seamless energy plane 360 ​​and the energy device layer 330. Tiled LF Display Module

[0077] 4A is a perspective view of a portion of an LF display system 400 that is two-dimensionally tiled to form a single-sided seamless surface environment, in accordance with one or more embodiments. The LF display system 400 includes a plurality of LF display modules that are tiled to form an array 410. More specifically, each small square in the array 410 represents a tiled LF display module 412. The array 410 can, for example, cover some or all of a surface.

[0078] Array 410 can project one or more holographic objects. For example, in an exemplary embodiment, array 410 projects holographic object 420 and holographic object 430. Tiling of the LF display modules not only allows for a much larger viewing volume, but also allows objects to be projected farther away from array 410. For example, in an exemplary embodiment, the viewing volume is approximately the entire area in front of and behind array 410, rather than a localized volume in front of (and behind) LF display module 412.

[0079] In some embodiments, the LF display system 400 presents the holographic object 420 to a viewer 430 and a viewer 434. The viewer 430 and the viewer 434 receive different perspectives of the holographic object 420. For example, the viewer 430 is presented with a straight-on view of the holographic object 420, while the viewer 434 is presented with a more oblique view of the holographic object 430. As the viewer 430 and / or the viewer 434 move, the viewer is presented with different perspectives of the holographic object 420. This allows the viewer to visually interact with the holographic object by moving relative to the holographic object. For example, as the viewer 430 walks around the holographic object 420, the viewer 430 sees different sides of the holographic object 420, as long as the holographic object 420 remains within the holographic object volume of the array 410. The viewer 430 and the viewer 434 can simultaneously view the holographic object 420 in real-world space as if it were actually there. Additionally, because holographic object 420 appears to the viewer in much the same way that a physical object appears, viewers 440 and 450 of holographic object 420 do not need to wear external devices to view holographic object 420. Additionally, holographic object 422 is illustrated here behind the array because the viewing volume of the array extends behind the surface of the array. In this manner, holographic object 422 may be presented to viewer 430 and / or viewer 434.

[0080] In some embodiments, LF display system 400 may include a tracking system that tracks the positions of viewer 430 and viewer 434. In some embodiments, the tracked positions are the positions of the viewers. In other embodiments, the tracked positions are the positions of the viewers' eyes. Eye position tracking differs from gaze tracking, which tracks where the eyes are looking (e.g., using orientation to determine gaze location). The eyes of viewer 430 and the eyes of viewer 434 are in different positions.

[0081] In various configurations, the LF display system 400 may include one or more tracking systems. For example, in the exemplary embodiment of FIG. 4A, the LF display system includes a tracking system 440 that is external to the array 410. Here, the tracking system may be a camera system coupled to the array 410. The external tracking system is described in more detail with respect to FIG. 5A. In other exemplary embodiments, the tracking system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340) of an LF display module 412 included in the array 410 may be configured to capture an image of a viewer in front of the array 440. In either case, the tracking system(s) of the LF display system 400 determine tracking information regarding a viewer (e.g., viewer 430 and / or viewer 434) viewing the holographic content presented by the array 410.

[0082] The tracking information represents the position of the viewer or the position of a portion of the viewer (e.g., one or both eyes of the viewer, or a limb of the viewer) in space (e.g., relative to the tracking system). The tracking system can use any number of depth determination techniques to determine the tracking information. Depth determination techniques can include, for example, structured light, time of flight, stereo imaging, some other depth determination technique, or some combination thereof. The tracking system can include a variety of systems configured to determine the tracking information. For example, the tracking system can include one or more infrared sources (e.g., structured light sources), one or more image sensors capable of capturing images in infrared (e.g., red-blue-green-infrared cameras), and a processor that executes a tracking algorithm. The tracking system can determine the viewer's position using depth estimation techniques. In some embodiments, the LF display system 400 generates a holographic object based on the tracked positions of the viewer 430 and / or the viewer 434. For example, the LF display system 400 can generate a holographic object in response to a viewer coming within a threshold distance and / or a specific position of the array 410.

[0083] LF display system 400 can present one or more holographic objects customized for each viewer based in part on the tracking information. For example, viewer 430 can be presented with holographic object 420, but not with holographic object 422; and viewer 434 can be presented with holographic object 422, but not with holographic object 420. For example, LF display system 400 tracks the positions of each of viewer 430 and viewer 434. LF display system 400 determines the viewpoint of the holographic object that should be seen by the viewer based on the viewer's relative position to where the holographic object should be presented. LF display system 400 selectively projects light from specific pixels that correspond to the determined viewpoint. Thus, viewer 434 and viewer 430 can potentially have completely different experiences simultaneously. In other words, LF display system 400 can present holographic content in viewing sub-volumes of a viewing volume (i.e., similar to viewing sub-volumes 290A, 290B, 290C, and 290D shown in FIG. 2B). For example, as shown, LF display system 400 can track the location of viewer 430 so that LF display system 400 can present space content (e.g., holographic object 420) in a viewing sub-volume surrounding viewer 430 and safari content (e.g., holographic object 422) in a viewing sub-volume surrounding viewer 434. In contrast, conventional systems would have to use individual headsets to provide a similar experience.

[0084] In some embodiments, the LF display system 400 may include one or more sensory feedback systems. The sensory feedback systems provide other sensory stimuli (e.g., touch, sound, pressure, force, or smell / olfaction) that enhance the holographic objects 420 and 422. For example, in the exemplary embodiment of FIG. 4A, the LF display system 400 includes a sensory feedback system 442 external to the array 410. In one example, the sensory feedback system 442 may be an electrostatic speaker coupled to the array 410. The external sensory feedback system is described in more detail with respect to FIG. 5A. In other exemplary embodiments, the sensory feedback system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340A of FIG. 3B) of the LF display module 412 included in the array 410 may be configured to project ultrasonic energy to a viewer in front of the array and / or receive image information from a viewer in front of the array. In either case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by array 410 (e.g., holographic object 420 and / or holographic object 422).

[0085] The LF display system 400 may include a sensory feedback system including one or more acoustic projection devices external to the array. Alternatively or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410 as described herein. The acoustic projection device may consist of an array of ultrasound sources configured to project a volumetric tactile surface. In some embodiments, the volumetric surface may be projected to coincide with the holographic object. In some embodiments, the volumetric tactile surface may be projected within a threshold distance of the holographic object. In some embodiments, the tactile surface may coincide with the holographic object (e.g., at the surface of the holographic object 420) at one or more surfaces of the holographic object if a portion of the viewer falls within a threshold distance of the one or more surfaces. Volumetric haptics allows the user to touch and feel the surface of the holographic object. The multiple acoustic projection devices may project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Thus, the ultrasound pressure waves and / or the audible pressure waves may serve to complement the holographic object.

[0086] In various embodiments, the LF display system 400 can provide other sensory stimuli based in part on the tracked position of the viewer. For example, the holographic object 422 illustrated in FIG. 4A is a lion, and the LF display system 400 can cause the holographic object 422 to roar both visually (i.e., the holographic object 430 appears to roar) and audibly (i.e., one or more sound projection devices project pressure waves that the viewer 430 perceives as a lion's roar emanating from the holographic object 422).

[0087] Note that in an example configuration, the holographic viewing volume may be limited in a manner similar to the viewing volume 285 of the LF display system 200 of Figure 2. This can limit the perceived amount of immersion a viewer may experience with a single device. One way to address this is to use multiple LF display modules tiled along multiple sides, as described below with respect to Figure 4B.

[0088] 4B is a perspective view of a portion of a LF display system 402 in a multi-sided seamless surface environment, according to one or more embodiments. The LF display system 402 is substantially similar to the LF display system 400, except that multiple LF display modules are tiled to create a multi-sided seamless surface environment. More specifically, the LF display modules are tiled to form an array that is a six-sided collective seamless surface environment. In other embodiments, multiple LF display modules may cover a portion, but not all, of a surface, or some combination thereof. In other embodiments, multiple LF display modules are tiled to form some other collective seamless surface. For example, the surface may be curved such that a cylindrical collective energy environment is formed.

[0089] The LF display system 402 can project one or more holographic objects. For example, in an exemplary embodiment, the LF display system 402 projects the holographic object 420 into an area surrounded by a six-sided aggregated seamless surface environment. In this example, the viewing volume of the LF display system is also included within the six-sided aggregated seamless surface environment. Note that in an exemplary configuration, the viewer 432 can be positioned between the holographic object 420 and the LF display module 414 that is projecting the energy (e.g., light and / or pressure waves) used to form the holographic object 420. Thus, the positioning of the viewer 434 may prevent the viewer 430 from perceiving the holographic object 420 formed from the energy from the LF display module 414. However, in the exemplary configuration, there is at least one other LF display module, such as, for example, the LF display module 416, that is not obstructed (e.g., by the viewer 434) and can project energy to form the holographic object 420. In this way, it is possible that some of the holographic projection may be obscured by viewers in the space, but this effect is much smaller than if there were a holographic display panel on only one side of the volume. Holographic object 422 is illustrated "outside" the enclosure of the six-sided aggregated seamless surface environment because the holographic object volume extends behind the aggregated surfaces. Viewer 430 and / or viewer 434 may therefore perceive holographic object 422 as "outside" the six-sided environment.

[0090] As discussed above with reference to Figure 4A, in some embodiments, the LF display system 402 may actively track the viewer's position and dynamically instruct different LF display modules to present holographic content based on the tracked position. Thus, the multi-surface configuration may provide a more robust environment (e.g., compared to Figure 4A) for presenting holographic objects, where an untethered viewer may freely move throughout an area surrounded by the multi-surface seamless surface environment.

[0091] In particular, various LF display systems may have different configurations. Furthermore, each configuration may have a particular orientation of the surfaces that together form a seamless display surface ("aggregate surface"). That is, the LF display modules of an LF display system may be tiled to form various aggregate surfaces. For example, in FIG. 4B, LF display system 402 includes LF display modules that are tiled to form a six-sided aggregate surface. In some other examples, the aggregate surface may occur only on a portion of the surface, rather than the entire surface. Some examples are described herein.

[0092] In some configurations, a collective surface of an LF display system may include a collective surface configured to project energy toward a localized viewing volume, which may enable a higher quality viewing experience, for example, by increasing the density of projected energy within a particular viewing volume, widening the FOV of a viewer within that volume, and moving the viewing volume closer to the display surface.

[0093] For example, Figure 4C shows a top view of an LF display system 450A having a collective surface in a "wing" configuration. The LF display system 450A includes LF display modules tiled to form a collective surface 460. The LF display system 450A includes a collective surface 460 that includes three portions: (i) a first central portion 462, (ii) a second portion 464 (i.e., a first side surface) connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front of the central portion, and (iii) a third portion 466 (i.e., a second side surface) connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front of the central portion. The mobile device may be designed in a configuration similar to that shown in Figure 4C, such that the first central portion 462 is a primary display screen designed to face the viewer, and the second portion 464 and third portion 466 are angled toward the viewer to partially surround the viewer. When the mobile device is held such that the first central portion is in a vertical plane having horizontal and vertical axes, the second and third portions are angled toward the front of the central portion along the horizontal axis.

[0094] In this example, the viewing volume 468A of the LF display system 450A lies in front of the device and is partially surrounded by three portions of the collecting surface 460. The collecting surfaces that at least partially surround the viewer (the "surrounding surfaces") increase the viewer's immersive experience.

[0095] For illustration, consider, for example, a collective surface having only a central surface. Referring to FIG. 2A, rays projected from either edge of the display surface will create an ideal holographic volume and an ideal viewing volume as described above. Now consider, for example, the case where the central surface includes two side surfaces angled toward the viewer. In this case, rays 256 and 257 will be projected at a larger angle from the normal of the central surface. Thus, the field of view of the viewing volume will be enlarged. Similarly, the holographic viewing volume will be closer to the display surface. Additionally, a holographic object projected at a certain distance from the display surface will be closer to its viewing volume because the second and third two parts are tilted closer to the viewing volume.

[0096] Simply put, a display surface with only a central surface has a planar field of view, a planar threshold separation between the (central) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer increases the field of view relative to the planar field, decreases the separation between the display surface and the viewing volume relative to the planar separation, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further increases the field of view, decreases the separation, and increases the proximity. In other words, the angled arrangement of the side surfaces increases the immersive experience of the viewer.

[0097] Additionally, as described below with respect to FIG. 6, deflection optics can be used to optimize the size and position of the viewing volume for LF viewing parameters (eg, dimensions and FOV).

[0098] Returning to FIG. 4D, in a similar example, FIG. 4D shows a top view of an LF display system 450B having a collective surface in a "slope" configuration. The LF display system 450B includes LF display modules tiled to form a collective surface 460. The collective surface 460 includes three portions: (i) a first central portion 462; (ii) a second portion 464 (i.e., a first side surface) connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front of the central portion; and (iii) a third portion 464 (i.e., a second side surface) connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front of the central portion. A mobile device may be designed in a similar configuration to that shown in FIG. 4D, such that the first central portion 462 is a primary display screen designed to face the viewer, and the second portion 464 and third portion 466 are angled toward the viewer to partially surround the viewer. When the mobile device is held such that the first portion is in a vertical plane having a horizontal axis and a vertical axis, the second and third portions are angled toward the front of the central portion along the vertical axis.

[0099] In this example, the viewing volume 468B of the LF display system 450B is in front of the central portion and is partially surrounded by three portions of the collecting surface 460. Similar to the configuration shown in FIG. 4C, the two side portions (e.g., the second portion 464 and the third portion 466) are angled to surround the viewer and form a surrounding surface. The surrounding surface expands the viewing FOV from any viewer's viewpoint within the holographic viewing volume 468B. Additionally, the surrounding surface allows the viewing volume 468B to be closer to the surface of the display so that projected objects appear closer. In other words, the angled arrangement of the side surfaces expands the field of view, reduces separation, and increases the proximity of the collecting surface, thereby increasing the immersive experience of the viewer. Additionally, as discussed below, deflection optics can be used to optimize the size and position of the viewing volume 468B.

[0100] The tilted configuration of the side portions of the collecting surface 460 allows the holographic content to be presented closer to the viewing volume 468B than if the third portion 466 was not tilted. For example, the lower extremities (such as legs) of a character in a presentation form in a tilted LF display system may appear closer and more realistic than if a LF display system with a flat front display was used.

[0101] Additionally, the configuration of the LF display system and the environment in which it is located may allow the shape and location of the viewing volume and viewing sub-volumes to be known.

[0102] 4E, for example, illustrates a top view of an LF display system 450C having a collecting surface 460. The LF display system 450C projects various light rays from the collecting surface 460. The light rays projected from the left side of the collecting surface 460 have a horizontal angular range 481, the light rays projected from the right side of the collecting surface have a horizontal angular range 482, and the light rays projected from the center of the collecting surface 460 have a horizontal angular range 483. Between these points, the projected light rays can take intermediate values ​​of the angular range. In this way, a viewing volume 468C is created by having a gradient deflection angle for the projected light rays across the display surface. Controlling the LF Display System

[0103] FIG. 5 is a block diagram of an LF display system 500, according to one or more embodiments. The LF display system 500 comprises an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display modules 512 may include a source / sensor system 514 that includes integrated energy source(s) and / or energy sensor(s) that project and / or sense other types of energy. The controller 520 includes a data store 522, a network interface 524, an LF processing engine 530, a command library 532, and a security module 534. The controller 520 may also include a tracking module 526, and a viewer profiling module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. The LF display systems described in the context of FIGS. 1, 2, 3, and 4 are embodiments of the LF display system 500. In other embodiments, LF display system 500 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different than described herein. Applications of LF display system 500 are also discussed in more detail below with respect to Figures 6 and 7.

[0104] The LF display assembly 510 provides holographic content within the holographic object volume that may be visible to a viewer located within the viewing volume. The LF display assembly 510 may provide the holographic content by executing display instructions received from the controller 520. The holographic content may include one or more holographic objects projected in front of a collecting surface, onto the LF display assembly 510, behind the collecting surface of the LF display assembly 510, or some combination thereof. Generation of display instructions using the controller 520 is described in more detail below.

[0105] The LF display assembly 510 provides holographic content using one or more LF display modules (e.g., any of the LF display module 110, the LF display system 200, and the LF display module 300) included in the LF display assembly 510. For convenience, the one or more LF display modules may be described herein as LF display modules 512. The LF display modules 512 may be tiled to form the LF display assembly 510. The LF display modules 512 may be structured as a variety of seamless surface environments (e.g., single-sided, multi-sided, curved, etc.). That is, the tiled LF display modules form a collective surface. As described above, the LF display module 512 includes an energy device layer (e.g., energy device layer 220) and an energy waveguide layer (e.g., energy waveguide layer 240) that present the holographic content. The LF display module 512 may also include an energy relay layer (e.g., energy relay layer 230) that transfers energy between the energy device layer and the energy waveguide layer when presenting the holographic content.

[0106] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing, as described above. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and integrated energy sensing system of the LF display module 512 may be collectively described herein as a source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 such that the source / sensor system 514 shares the same seamless energy surface as the LF display module 512. In other words, the collective surface of the LF display assembly 510 includes the functionality of both the LF display module 512 and the source / sensor module 514. That is, the LF assembly 510 including the LF display module 512 with the source / sensor system 514 can project energy and / or sense energy while simultaneously projecting a light field. For example, the LF display assembly 510 may include an LF display module 512 and a source / sensor system 514 configured as a dual energy plane or a bidirectional energy plane as described above.

[0107] In some embodiments, the LF display system 500 uses the sensory feedback system 570 to augment the generated holographic content with other sensory content (e.g., collaborative touch, sound, pressure, force, or smell). The sensory feedback system 570 can augment the projection of the holographic content by executing display instructions received from the controller 520. In general, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., the sensory feedback system 442) external to the LF display assembly 510. Some exemplary sensory feedback devices may include collaborative sound projection and receiving devices, aroma projection devices, temperature adjustment devices, force actuation devices, pressure sensors, accelerometers, gyroscopes, transducers, and the like. In some cases, the sensory feedback system 570 may have similar functionality as the light field display assembly 510, or vice versa. For example, both the sensory feedback system 570 and the light field display assembly 510 may be configured to generate a sound field. As another example, the sensory feedback system 570 may be configured to generate a tactile surface, while the light field display assembly 510 is not.

[0108] To illustrate, in one exemplary embodiment of the light field display system 500, the sensory feedback system 570 may include an acoustic projection device. The acoustic projection device is configured to generate one or more pressure waves that complement the holographic content when executing the display instructions received from the controller 520. The generated pressure waves may be, for example, audible (for sound), ultrasonic (for touch), or some combination thereof. Similarly, the sensory feedback system 570 may include an aroma projection device. The aroma projection device may be configured to impart an aroma to some or all of the target area when executing the display instructions received from the controller. Additionally, the sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature of some or all of the target area when executing the display instructions received from the controller 520.

[0109] In some embodiments, the sensory feedback system 570 may include a force actuator. The force actuator can be used to move some physical objects in the target area. In one example, the LF display system 500 can be integrated with a multi-sensory gaming application, and the force actuator can move parts related to the game.

[0110] In some embodiments, the sensory feedback system 570 is configured to receive input from a viewer of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the viewer. The sensory feedback devices can include devices such as acoustic receiving devices (e.g., microphones), pressure sensors, motion detectors, ultrasonic transducers, other transducers, force sensors, temperature sensors, touch sensors, proximity sensors, and the like. The sensory feedback system can also be comprised of buttons, dials, knobs, keypads, fingerprint sensors, joysticks, other input hardware, or any combination thereof. The sensory feedback system 570 can include sensory feedback devices configured to provide sensory feedback when a holographic character is presented. The sensory feedback system can transmit the detected input to the controller 520 to adjust the generation of the holographic content and / or sensory feedback. In some embodiments, the sensory feedback system 570 identifies the input received from the viewer as a received command from the viewer to the mobile device.

[0111] To illustrate, in one exemplary embodiment of a light field display assembly, the sensory feedback system 570 includes a microphone. The microphone is configured to record audio generated by a user of the mobile device. The sensory feedback system 570 provides the recorded audio as user input to the controller 520. The controller 520 can use the user input to generate the holographic content. Similarly, the sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure a force applied to the pressure sensor by the user. The sensory feedback system 570 can provide the measured force to the controller 520 as user input.

[0112] In some embodiments, the sensory feedback assembly 570 is also configured to redirect the user's movements. In one embodiment, the sensory feedback assembly 570 includes an acoustic projection device that can generate haptic pressure waves to discourage movement, for example, if the tracking system 580 detects the presence of a user's hand blocking the mobile device's camera while the camera sensor is actively recording image data. In another embodiment, the sensory feedback assembly 570 can use an acoustic emitting device to audibly cue the user not to take further action (e.g., "Camera is recording - do not block the camera").

[0113] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to obtain information about a user of the device, including the user's position within a target area, movements, gestures, facial expressions, gaze, and / or characteristics such as gender and age. Typically, the tracking devices are external to the LF display assembly 510. Some exemplary tracking devices include a camera assembly ("camera"), a depth sensor, structured light, a LIDAR system, a card scanning system, or other tracking device capable of tracking a user within a target area.

[0114] The tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. In some cases, however, when presenting holographic content, the target area is illuminated with natural light and / or ambient light from the LF display assembly 510. The energy sources project light when executing instructions received from the controller 520. The light may be, for example, a structured light pattern, a pulse of light (e.g., an IR flash), or some combination thereof. The tracking system may project light within the visible band (approximately 380 nm to 750 nm), within the infrared (IR) band (approximately 750 nm to 1700 nm), within the ultraviolet band (10 nm to 380 nm), some other portion of the electromagnetic spectrum, or some combination thereof. Sources may include, for example, light emitting diodes (LEDs), micro LEDs, laser diodes, TOF depth sensors, tunable lasers, etc.

[0115] Tracking system 580, when executing instructions received from controller 520, can adjust one or more emission parameters. An emission parameter is a parameter that affects how light is projected from a source of tracking system 580. Emission parameters can include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, any other parameter that affects how light is projected from the source assembly, or any combination thereof. In one embodiment, the source projects pulses of light during time-of-flight operation.

[0116] The camera of the tracking system 580 captures an image of the light (e.g., structured light pattern) reflected from the target area. The camera captures the image as it executes the tracking instructions received from the controller 520. As previously described, the light may be projected by the source of the tracking system 580. The camera may include one or more cameras. That is, the camera may be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by the tracking system 580, or some combination thereof. In one embodiment, the tracking system 580 may include a light field camera external to the LF display assembly 510. In other embodiments, the camera is included as part of an LF display module included in the LF display assembly 510. For example, as previously described, if the energy relay element of the light field module 512 is a bidirectional energy layer that interleaves both a light-emitting display and an image sensor with the energy device layer 220, the LF display assembly 510 may be configured to project a light field and simultaneously record image information from a viewing area in front of the display. In one embodiment, the images captured from the bidirectional energy surface form a light field camera. The camera provides the captured images to the controller 520.

[0117] The cameras of tracking system 580 may adjust one or more imaging parameters when executing tracking instructions received from controller 520. Imaging parameters are parameters that affect how the camera assembly captures images. Imaging parameters may include, for example, frame rate, aperture, gain, exposure length, frame timing, any other parameter that affects how the camera assembly captures images, or any combination thereof.

[0118] In some embodiments, the tracking system 580 is configured to receive input from a viewer of the LF display system 500. The tracking system 580 can track the body movements of the viewer and transmit information regarding the specific tracked movements to the controller 520. In some embodiments, the tracking system 580 is used in coordination with a holographic user interface that is presented to the user.

[0119] The controller 520 controls the LF display assembly 510 and any other components of the LF display system 500. The controller 520 comprises a data store 522, a network interface 524, a tracking module 526, a viewer profiling module 528, a light field processing engine 530, and a command library 532. In other embodiments, the controller 520 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different from that described herein. For example, the tracking module 526 may be part of the LF display assembly 510 or the tracking system 580.

[0120] The data store 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by the camera, one or more user profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 including the resolution and orientation of the LF module 512, desired viewing volume geometry, 3D models, scenes and environments, content for graphics creation including materials and textures, security related information including authentication elements that may be used by the security module 534, such as passwords, smart cards, biometric data, other information that may be used by the LF display system 500, or any combination thereof. The data store 522 is a memory such as a read only memory (ROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), or any combination thereof.

[0121] The network interface 524 allows the light field display system to communicate with other systems or environments over a network. In one example, the LF display system 500 receives holographic content from a remote server, such as an online holographic content store, via the network interface 524. In another example, the LF display system 500 uses the network interface 524 to transmit holographic content to a remote data store.

[0122] The tracking module 526 tracks viewers watching content presented by the LF display system 500. To do so, the tracking module 526 generates tracking instructions that control the operation of the source(s) and / or camera(s) of the tracking system 580 and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking input to the tracking module 526.

[0123] The tracking module 526 can determine the location of one or more viewers within the target area of ​​the mobile device. The determined location can be relative to some reference point (e.g., the display surface). In other embodiments, the determined location can be within a virtual model of the target area. The tracked location can be, for example, a tracked location of the viewer and / or a tracked location of a portion of the viewer (e.g., eye location, hand location, etc.). The tracking module 526 determines the location using one or more images captured from the cameras of the tracking system 580. The cameras of the tracking system 580 can be distributed around the LF display system 500 and can capture images in stereo, allowing the tracking module 526 to passively track the user. In other embodiments, the tracking module 526 actively tracks the user. That is, the tracking system 580 illuminates a portion of the target area, images the target area, and the tracking module 526 determines the location using time-of-flight and / or structured light depth determination techniques. The tracking module 526 uses the determined location to generate tracking information.

[0124] The tracking module 526 can also receive tracking information as input from a viewer of the LF display system 500, for example, while using a holographic user interface on a mobile device (described further below). The tracking information can include body movements of a user corresponding to various input options provided by the LF display system 500. For example, the tracking module 526 can track the user's body movements and assign any of the various movements as input to the LF processing engine 530. The tracking module 526 can provide the tracking information to the data store 522, the LF processing engine 530, the viewer profiling module 528, the command store 532, any other component of the LF display system 500, or some combination thereof. The LF processing engine 530 can create holographic content based in part on the provided tracking information.

[0125] To provide context for the tracking module 526, consider an exemplary embodiment of the LF display system 500 used in a gaming application in a mobile device. When a user responds to a win by raising and lowering their fist in the air to express their excitement, the tracking system 580 can record the user's hand movements and transmit the recording to the tracking module 526. The tracking module 526 tracks the user's hand actions in the recording and transmits input to the LF processing engine 530, described below. As described below, the viewer profiling module 528 determines that information in the image indicates that the user's hand movements are associated with a positive reaction. Thus, the LF processing engine 530 generates appropriate holographic content to celebrate the win. For example, the LF processing engine 530 can project confetti into a volume of a holographic object associated with the gaming application.

[0126] The LF display system 500 includes a viewer profiling module 528 configured to identify and profile device users. The viewer profiling module 528 generates a profile of a user (or users) viewing holographic content displayed by the LF display system 500. The viewer profiling module 528 generates a user profile based in part on user input and monitored user behavior, actions, and reactions. The viewer profiling module 528 can access information obtained from the tracking system 580 (e.g., recorded images, videos, audio, etc.) and process the information to determine various information. In various examples, the viewer profiling module 528 can use machine vision or machine hearing algorithms to determine user behavior, actions, and reactions. Monitored user behaviors can include, for example, smiles, cheers, clapping, laughter, fear, screams, excitement levels, repulsion, changes in other gestures or movements by the user, etc.

[0127] More generally, a user profile may include any information received and / or determined regarding a user viewing holographic content from the LF display system. For example, each user profile may record that user's actions or reactions to content displayed by the LF display system 500. Some exemplary information that may be included in a user profile is provided below.

[0128] The user profile may be based on the user's behavior with respect to content displayed by the LF display system 500. In some embodiments, software applications resident on the device may be functionally integrated with the LF display system 500, and the user profile is based on the user's behavior that is monitored while running these software applications on the device.

[0129] Thus, for example, monitored behavior may include the number of times a user uses an application over a period such as a day or week, how often a user uses a particular setting of an application (e.g., the "Level 8" setting used in a particular puzzle game application on a device), how a user responds to a particular type of holographic content (e.g., a clock alarm application - a preferred setting for the user - a holographic image of a flashing button accompanied by a high frequency whistle or a holographic image of just a flashing button), other behavior associated with use of a particular device, or combinations thereof.

[0130] In another example, a user may indicate a preference for a tactile interface presented in conjunction with the holographic content, e.g., a preferred key click sensitivity in a holographic keyboard touch interface that may be generated by the LF display system 100 and optionally provided to a mobile device user.

[0131] Other user characteristics that may be stored as part of the user profile information may include, for example, the user's name, the user's age, ethnicity, gender, viewing location, occupational information, education, income, amount spent on purchases, hobbies, location, viewing history, categories of online items viewed, purchasing history, device and application setting preferences, geographic location, other demographic information, or any combination thereof.

[0132] The viewer profiling module 528 may also access profiles associated with a particular user(s) from third party or online systems to build and / or update a user profile. The viewer profiling module 528 may be configured to use information from the user(s)' social media accounts to update the user profile. For example, a user's interactions with third party vendors linked to the user's social media or other online accounts may provide further user profile information that may be accessed by the viewer profiling module 528. Thus, the user profile information may store information such as favorite musicians, favorite movie characters, least favorite music genres, etc., and leverage this information when accessing content from an online holographic content store.

[0133] In some embodiments, the data store 522 includes a user profile store that stores user profiles generated, updated, and / or maintained by the viewer profiling module 528. The user profiles may be updated in the data store at any time by the viewer profiling module 528. For example, in one embodiment, the user profile store receives and stores information about a particular user in a user profile for that user when that user views holographic content provided by the LF display system 500. In this example, the viewer profiling module 528 includes a facial recognition algorithm that can recognize the user and reliably identify them when they view the presented holographic content. To illustrate, when a user enters a target area of ​​the LF display system 500, the tracking system 580 captures an image of the user. The viewer profiling module 528 inputs the captured image and uses a facial recognition algorithm to identify the user's face. The identified face is associated with the user profile in the profile store, such that all input information captured about that user may be stored in that user's profile. The user profiling module may also utilize a card identification scanner, a voice identifier, a radio frequency identification (RFID) chip scanner, a bar code scanner, etc. to positively identify the user.

[0134] Because the viewer profiling module 528 can reliably identify users, the viewer profiling module 528 can determine information about each user of the LF display system 500 and about each user. This is particularly pertinent when a mobile device may be shared by multiple users, such as, for example, a mobile tablet device shared by multiple students in a classroom setting or shared among family members. The viewer profiling module 528 can then store the date and time of each visit in a user profile for each user. Similarly, the viewer profiling module 528 can store input from the user received from any combination of the sensory feedback system 570, the tracking system 580, and / or the LF display assembly 510, whenever the input occurs. The user profile system 528 can additionally receive further information about the user from other modules or components of the controller 520, which information can be stored with the user profile. Other components of the controller 520 can then also access the stored user profile to determine future content to be provided to that user. The controller 520 may be configured to receive or generate holographic content based at least in part on accessed data from stored user profiles. For example, a user interface generated for the teenage girl in the family may receive verbal commands using an avatar of her favorite holographic comic book superheroine, while a user interface generated for the grandfather may be an enlarged, flashing holographic interface with easy-to-see rendering of large buttons matched with haptic cues to make "pushing" easy.

[0135] The LF processing engine 530 generates 4D coordinates in a rasterized format ("rasterized data") that, when executed by the LF display assembly 510, cause the LF display assembly 510 to present holographic content. The LF processing engine 530 can access the rasterized data from the data store 522. Additionally, the LF processing engine 530 can construct the rasterized data from vectorized data sets. The vectorized data is described below. The LF processing engine 530 can also generate sensory instructions necessary to provide sensory content that augments the holographic object. As described above, the sensory instructions, when executed by the LF display system 500, can generate tactile surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF processing engine 530 can access the sensory instructions from the data store 522 or construct the sensory instructions to form the vectorized data sets. Collectively, the 4D coordinates and sensory data represent display instructions executable by the LF display system to generate holographic and sensory content.

[0136] The amount of rasterized data describing the flow of energy through the various energy sources in the LF display system 500 is incredibly large. While it is possible to display the rasterized data on the LF display system 500 when accessed from the data store 522, it is not possible to efficiently transmit (e.g., via the network interface 524), receive, and then display the rasterized data on the LF display system 500. Take, for example, rasterized data representing a short video for holographic projection by the LF display system 500. In this example, the LF display system 500 includes a display including several gigapixels, and the rasterized data includes information for each pixel location on the display. The corresponding size of the rasterized data is enormous (e.g., many gigabytes per second of video display time) and unmanageable for efficient transfer over a commercial network via the network interface 524. The problem of efficient transfer may be amplified in applications involving live streaming of holographic content. If an interactive experience is desired using input from the sensory feedback system 570 or tracking module 526, additional problems with simply storing rasterized data in the data store 522 arise. To enable interactive experiences, the light field content generated by the LF processing engine 530 may be modified in real-time in response to sensory or tracking input. In other words, in some cases, the LF content cannot simply be read from the data store 522.

[0137] Thus, in some configurations, data representing holographic content for display by the LF display system 500 may be transferred to the LF processing engine 530 in a vectorized data format ("vectorized data"). The vectorized data may be orders of magnitude smaller than rasterized data. Furthermore, the vectorized data provides high image quality while having a data set size that allows for efficient sharing of the data. For example, the vectorized data may be a sparse data set derived from a denser data set. Thus, the vectorized data may have an adjustable balance between image quality and data transmission size based on how the sparse vectorized data is sampled from the denser rasterized data. The adjustable sampling to generate the vectorized data allows for optimization of image quality at a particular network speed. As a result, the vectorized data allows for efficient transmission of the holographic content over the network interface 524. The vectorized data may also allow for live streaming of the holographic content over commercial networks.

[0138] In summary, the LF processing engine 530 can generate holographic content derived from rasterized data accessed from the data store 522, vectorized data accessed from the data store 522, or vectorized data received via the network interface 524. In various configurations, the vectorized data can be encoded prior to data transmission and decoded after receipt by the LF controller 520. In some examples, the vectorized data is encoded for additional data security and performance improvements associated with data compression. For example, the vectorized data received by the network interface can be encoded, vectorized data received from a holographic streaming application. In some examples, the vectorized data can require a decoder, the LF processing engine 530, or both, to access the information content encoded within the vectorized data. The encoder and / or decoder systems may be available to customers or licensed to third-party vendors.

[0139] The vectorized data includes information for each of the sensory areas supported by the LF display system 500 in a manner that supports an interactive experience. For example, the vectorized data for an interactive holographic experience includes any vectorized characteristics that can provide accurate physics for each of the sensory areas supported by the LF display system 500. The vectorized characteristics can include any characteristics that can be synthetically programmed, captured, computationally evaluated, etc. The LF processing engine 530 can be configured to convert the vectorized characteristics of the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content converted from the vectorized data from the LF display assembly 510. In various configurations, the vectorized properties may include one or more red / green / blue / alpha channel (RGBA) + depth images, multi-view images with or without depth information at various resolutions that may include one high-resolution central image and other views at lower resolution, albedo and reflectance, surface normals, other optical effects, surface identity, geometric object coordinates, virtual camera coordinates, display surface location, lighting coordinates, surface contact stiffness, contact ductility, contact intensity, amplitude and coordinates of sound fields, environmental conditions, somatosensory energy vectors associated with mechanoreceptors of texture or temperature, sound, and coordinates of other sensory domain properties. Many other vectorized properties are possible.

[0140] The LF display system 500 can also generate interactive viewing experiences. That is, the holographic content can respond to input stimuli, including user positions, gestures, interactions, information about interactions with the holographic content, or other information derived from the viewer profiling module 528 and / or tracking module 526. For example, in one embodiment, the LF processing system 500 uses vectorized data of real-time performance received as part of a live streaming application on the device via the network interface 524 to create an interactive viewing experience. In another example, if a holographic object needs to move in a particular direction immediately in response to a user's interaction, the LF processing engine 530 can update the rendering of the scene so that the holographic object moves in that requested direction. This may require the LF processing engine 530 to use the vectorized data set to render a light field in real time based on a 3D graphical scene with proper object placement and movement, collision detection, occlusion, color, shading, lighting, etc., to respond correctly to the user's interactions. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510.

[0141] The rasterized data includes holographic content instructions and sensory instructions (display instructions) that represent the real-time performance. The LF display assembly 510 simultaneously projects the holographic and sensory content of the real-time performance by executing the display instructions. The LF display system 500 monitors user interactions (e.g., voice responses, touch, etc.) with the presented real-time performance content using a tracking module 526 and a viewer profiling module 528. In response to the user interactions, the LF processing engine creates an interactive experience by generating additional holographic and / or sensory content for display to the user.

[0142] To illustrate, consider an exemplary embodiment of an LF display system 500 that includes an LF processing engine 530 that generates a number of holographic objects representing balloons floating in a viewing volume of a user of a mobile device during execution of an application on the mobile device. The user may move to touch the holographic objects representing the balloons. In response, a tracking system 580 tracks the movement of the user's hand relative to the holographic objects. The user's movements are recorded by the tracking system 580 and transmitted to the controller 520. The tracking module 526 continuously determines the movement of the user's hand and transmits the determined movements to the LF processing engine 530. The LF processing engine 530 determines the placement of the user's hand in the scene and adjusts the real-time rendering of the graphics to include any required changes (such as position, color, or occlusion) to the holographic objects. The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a tactile surface using a volumetric haptic projection system (e.g., using an ultrasonic speaker). The generated haptic surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the exterior surface of the holographic object. The LF processing engine 530 uses the tracking information to dynamically direct the LF display assembly 510 to move the position of the haptic surface along with the position of the rendered holographic object such that the user is given both the visual and haptic perception of touching a balloon. In simpler terms, when a user sees his or her hand touching a holographic balloon, the user simultaneously feels haptic feedback indicating that the hand is touching the holographic balloon and that the balloon changes position or motion in response to the touch.In some examples, rather than presenting an interactive balloon in association with content accessed from data store 522 and executing an application, the interactive balloon may be received as part of holographic content received from a live streaming application via network interface 524.

[0143] The LF processing engine 530 can provide holographic content for simultaneous display to a viewer of the media content on a portable mobile device. For example, a video being viewed on a mobile device enhanced with the LF display system 500 includes holographic content (a "holographic content track") that is presented to the user during the presentation of the video. The holographic content track can be received by the device and stored in the data store 522. The holographic content track includes holographic content that enhances the viewing experience of a user watching the video on the device.

[0144] The LF processing engine 530 can create, modify, and / or update holographic content based on a user profile. Updates to the user profile can result in updates to the displayed holographic content. In some embodiments, the LF processing engine 530 can create, modify, and / or update holographic content based on the user profile in conjunction with an artificial intelligence model.

[0145] The holographic content in the holographic content track may be associated with any number of temporal, auditory, visual, or other cues for displaying the holographic content. For example, the holographic content track may include holographic content that is displayed at a particular time during the presentation of a video. As an example, the holographic content track may include a holographic dolphin pod that is displayed during the presentation of a video 35 minutes and 42 seconds after the start of the video "Bottlenosed." In another example, the holographic content track includes holographic content that is presented when the sensory feedback system 570 records a particular audio cue. As an example, the holographic content track includes a laughing holographic avatar for presentation during the execution of a gaming application when audio recorded by the sensory feedback assembly 570 indicates that the user / player is laughing. In another example, the holographic content track includes holographic content that is presented when the tracking system 580 records a particular visual cue. As an example, the holographic content track includes a holographic flashing sign with the words "Stop?" that is displayed when the tracking system 580 records information indicating that the user is moving away from the device. The determination of auditory and visual cues is described in more detail below.

[0146] The holographic content track may also include spatial rendering information. That is, the holographic content track may indicate spatial locations for presenting the holographic content of the displayed video track. For example, the holographic content track may indicate that a particular holographic content is presented in some holographic viewing volumes and not in others. To illustrate, the LF processing engine 530 may present a holographic screen that displays an email application in a holographic viewing volume that is directly in front of the user's face. Similarly, the holographic content track may indicate that the holographic content is presented in some viewing volumes and not in others. For example, the LF processing engine may display a holographic screen in a viewing volume that is directly in front of the user's face but not in other viewing volumes, such that the screen is private to the user.

[0147] The LF processing engine 530 may provide holographic content for display to a user in conjunction with an application running on the device. Holographic content may be received at a device of the LF display system 500 over a network from an online holographic content store that makes holographic content available for downloading to a mobile device in exchange for a transaction fee. For example, a user may access an online holographic content store and obtain a particular superhero character in exchange for a transaction fee for use as a holographic avatar in conjunction with an application running on the device. Holographic content may be received at the LF display system 500 over a network from an online holographic application store that makes holographic applications available for downloading and running on a mobile device in exchange for a transaction fee. For example, a user may access a holographic application store to purchase an application for generating and customizing a holographic avatar, and the executable application may download a dedicated holographic user interface for generating and customizing a holographic avatar. The holographic content may be provided as part of advertising content associated with a particular application running on the device. For example, when a user clicks on a gaming application on a mobile device, the LF display system 500 can access holographic content of products as advertisements provided by advertisers and display them during the course of the gaming application. The holographic content can be stored in the data store 522 or streamed to the LF display system 500 in a vectorized format via the network interface 524.

[0148] The LF processing engine 500 can also modify the holographic content to suit the environmental characteristics of a location. For example, based on whether the user is in a darkened space, a bright room, a sunny outdoor location, or other location, the LF processing engine can modify the holographic content on the mobile device to display in a brightness appropriate for the lighting conditions of the location. The LF processing engine can use the data stored in the data store 522, along with information from the camera system of the tracking module 580, to customize the display of the holographic content. Additionally, the modification of the holographic content can also utilize user viewing preference settings determined by the viewer profiling module 528.

[0149] The LF processing engine 530 can also modify the holographic content in response to information received from the sensory feedback system 570. For example, an accelerometer in the sensory feedback system 570 can measure the linear acceleration of the mobile device. The LF processing engine 530 can modify the displayed holographic content to remove visual jitter in the holographic display caused by acceleration. Similarly, a gyroscope in the sensory feedback system 570 can detect changes in the orientation of the mobile device and reorient the displayed holographic content based on the detected changes. In some embodiments, the tracking module 526 can track the rotation of the user's head and reorient the displayed holographic UI in response.

[0150] The LF processing engine 530 can also create holographic content for display by the LF display system 500. Importantly, creating holographic content for display is distinct from accessing or receiving holographic content for display. That is, when creating content, the LF processing engine 530 generates entirely new content for display rather than accessing previously generated and / or received content. The LF processing engine 530 can obtain or receive information from the data store 522, the tracking module 526, the viewer profiling module 528, the sensory feedback system 570, the tracking system 580, or some combination thereof to create holographic content for display. In some examples, the LF processing engine 530 can access information from elements of the LF display system 500 (e.g., tracking information and / or user profiles), create holographic content based on that information, and display the created holographic content using the LF display system 500. The created holographic content can be augmented with other sensory content (e.g., touch, sound, pressure, force, or smell / olfaction) when displayed by the LF display system 500. Additionally, the LF display system 500 can store the created holographic content in a data store 522 for future use.

[0151] In some embodiments, the LF processing engine 530 can update, modify, remove, pause, or a combination thereof, the generated holographic content in response to changes in the operational state of the mobile device. In some embodiments, the LF processing engine 530 can update, modify, remove, pause, or a combination thereof, the generated holographic display in response to information received by the tracking system 580 and interpreted by the tracking module 526. For example, the tracking module 580 can provide user gaze information captured from the tracking system 580 to the LF display module to re-render the holographic character so that the holographic character makes eye contact with the user interacting with the device. In some embodiments, operational selections made by the user can affect the display of the holographic content. For example, if the LF display system determines that the user has made an operational selection to have an alarm application generate an alert at 6:00 a.m., the LF display system can present a holographic depiction of a flashing red button and an alert sound only at 6:00 a.m., or as configured by the user.

[0152] In some embodiments, the LF display system 500 includes a command library 532. The command library 532 can function as a command interface that recognizes and interprets user commands. The recognized commands can change the operating state of the mobile device. The user commands can be interpreted based on tracking information received from the sensory feedback system 570 (e.g., voice), tracking information received from the tracking module 526 (e.g., tracked gestures or facial expressions), tracking information received from the viewer profiling module 528 (e.g., stored individual preferences), or some combination thereof. The command library 532 can apply heuristics to interpret the user commands and direct the LF processing engine 530 to generate appropriate holographic content, haptics, and any other multi-sensory information. The command library 532 stores a mapping of received user commands to computational commands that can be executed by the LF display system 500. The command library 532 also stores heuristics associated with the stored mappings. The command library 532 can receive user commands via a holographic user interface displayed by the LF display system 500. Examples of user commands that may be received by the LF display system 500 via a mobile device may include various touch contacts of holographic objects by a user (e.g., touching a particular displayed holographic "button" with a finger) and various touch movements relative to the surface of a holographic object (e.g., turning a displayed holographic "knob"). In some embodiments, a touch contact or touch movement may be determined to have occurred when the tracking module 526 determines that the user is within a threshold distance of a displayed holographic object that is projected as part of a holographic user interface.In some embodiments, user commands may include sensory feedback provided by the user, such as verbal commands spoken by the user while within the target area (e.g., a voice command in the form of "tell me where the next gas station is" captured using an acoustic receiver on the mobile device), gestures and body movements captured by the tracking system 580 (e.g., a user raising their hand within the target area to indicate that the timer should be paused), the sensing of a physical switch, knob, button, or dial connected to the display as part of the sensory feedback system 570, or a combination thereof.

[0153] Computational commands executed by the LF processing engine in response to received user commands may include displaying a particular holographic object, updating the display of holographic content, modifying the display of a particular holographic object, displaying a sequence of holographic objects, pausing or playing the content being presented by the LF display system 500, removing a displayed object, or combinations thereof. Other computational commands may include, for example, activating other sensors or controls, such as joystick game controls. Thus, in some embodiments, a detected user action, such as turning a displayed holographic "dial," is reported by tracking system 580 to command library 532. In response, command library 532 may apply a heuristic from among its stored heuristics and determine that a computational command needs to be executed, including, for example, sending instructions to a system to properly set a timer application on a displayed holographic clock.

[0154] In some embodiments, the command library 540 receives information from the data store 522 and the tracking module 580 to resolve ambiguous mappings based on user commands that may be detected with a lower degree of confidence. However, information from the tracking system 580 and user profile information from the data store 522 may provide sufficient heuristic information to disambiguate the mapping and determine the specific computational command to be executed. This determined computational command is provided to the LF processing engine 530 for execution.

[0155] In some embodiments, the LF display system 500 includes a security module 534. The security module 534 enforces secure access to at least some of the functionality provided by the LF display system. To do so, the security module 534 can generate a user interface element by the LF display system 500 for receiving authentication credentials, such as a password or biometric data, from a user of the mobile device. The security module can authenticate the credentials based on data stored in the data store 522. After authenticating the user, the security module 534 can enable secure access by the user to some or all of the functionality of the LF display system 500. In some embodiments, the security module 534 can enable the display of certain holographic content accessed from a data store or online source, or the execution of certain applications on the mobile device. For example, the LF display system may present certain holographic content to a viewer only upon authenticating user credentials, and not in the absence of authenticated user credentials. The security module 534 may provide information regarding the authentication of the mobile device user to the data store 522, the viewer profiling module 528, the LF processing engine 530, the command library 532, any other components of the LF display system 500, or some combination thereof. Dynamic Content Generation for LF Display Systems

[0156] In some embodiments, the LF processing engine 530 incorporates an artificial intelligence (AI) model to create holographic content for display by the LF display system 500. The AI ​​model may include supervised or unsupervised learning algorithms, including but not limited to regression models, neural networks, classifiers, or other AI algorithms. The AI ​​model may be used to determine user preferences based on user information recorded by the LF display system 500 (e.g., by the tracking system 580), which may include information about user behavior. The AI ​​model may also store each user's learned preferences in a user profile store in the data store 522. In some examples, when multiple users operate a mobile device, the AI ​​model may create holographic content for a particular individual user based on that user's learned preferences. The AI ​​model may access information from the data store 522 to create and / or modify the holographic content. In some embodiments, the AI ​​model may use the results of the learning algorithm to establish user-specific heuristics for command mappings stored in the command library 532.

[0157] One example of an AI model that may be used to identify user characteristics, identify responses, and / or generate holographic content based on identified information is a convolutional neural network model with layers of nodes where the value of a node in the current layer is the transformation of the value at the node in the previous layer. The transformation in the model is determined by a set of weights and parameters that connect the current layer and the previous layer. In some examples, the transformation may also be determined via a set of weights and parameters used to transform between previous layers in the model.

[0158] The input to the model may be an image captured by the tracking system 580, encoded on the first convolutional layer, and the output of the model is the holographic content decoded from the output layer of the neural network. Alternatively or additionally, the output may be determined features of the user in the image. In this example, the AI ​​model may identify latent information in the image that represents the features of the user in one of the intermediate layers of the neural network. The association information between the elements identified in each layer may be obtained by applying a set of transformations between the corresponding intermediate layers. The weights and parameters of the transformations may indicate the relationship between the information contained in the starting layer and the information obtained from the final output layer. For example, the weights and parameters may be quantization of the shape, color, size, etc., contained in the information representing the smiling user in the image. The weights and parameters may be based on historical data (e.g., previously tracked users). In some embodiments, the information from the tracking system may be used in conjunction with the AI ​​model to generate a volumetric haptic surface that matches or is within a threshold distance of the presented holographic content.

[0159] In one embodiment, the AI ​​model includes a deterministic method trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve the quality of its performance using measurements from the tracking system 580 as inputs and modifications to the created holographic content as outputs.

[0160] Reinforcement learning is a machine learning system in which a machine learns "what to do" (how to map situations to actions) to maximize a numerical reward signal. The learner (e.g., LF processing engine 530) is not told what action to take (e.g., generating a given holographic content), but instead discovers, by trying actions, which action will yield the greatest reward (e.g., increasing the quality of the holographic content by making the user smile more). In some cases, actions can affect not only the immediate reward, but also the next situation and, through it, all subsequent rewards. These two features (trial-and-error exploration and delayed reward) are the two distinguishing features of reinforcement learning.

[0161] The AI ​​model can include any number of machine learning algorithms. Some other AI models that may be used are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In general, the LF processing engine 530 takes input from the tracking module 526 and / or the viewer profiling module 528, and the machine learning model creates the holographic content accordingly. Similarly, the AI ​​model may direct the rendering of the holographic content. Network environment

[0162] 6 depicts a block diagram of a network environment 600 according to one or more embodiments. The network environment 600 includes one or more mobile devices 610, a network 630, and one or more remote servers 640.

[0163] In one embodiment, the mobile device 610 is a portable computing device that executes computer program modules, e.g., a web browser or client applications, that enable a user to browse the Internet, consume media content, and execute a variety of software applications. The mobile device 610 may be, for example, a personal computer, a tablet computer, a smartphone, a dedicated e-reader, or any other type of portable, network-enabled device. The mobile device 610 includes a light field display system 612 and a user interface 614. In other embodiments, the mobile device 610 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different than described herein.

[0164] The mobile device 610 includes, among other components, an LF display system 612, a user interface 614, a mobile operating system (OS) 616, a power system 618, and a global positioning system (GPS) module 620. The LF display system 612 is one embodiment of the LF display system 500 depicted in Figure 5. A user 650 can interact with the mobile device 610 via the user interface 614.

[0165] The LF display system 612 can access hardware components of the mobile device 610. For example, the LF display system 612 can access a power circuit in the mobile device to power itself by drawing power from a power source such as a battery or a charging unit on the mobile device. The tracking system 580 can be accessible to one or more cameras, or a depth sensor, or a GPS module 620 of the mobile device 610. The sensory feedback system 570 can access buttons, switches, microphones, or keyboards of the user interface 614. The controller 520 can access real-time clock (RTC) circuits in the mobile device as needed for time-based functions. The viewer profiling module 528 can determine user identification by directly accessing a fingerprint sensor or keyboard. The user profiling module can download or upload user profile information via a network interface 614 connected to a network 630. The LF display assembly can project holographic or 2D content depending on user settings. In general, the LF display system 612 is integrated with other hardware components of the mobile device 610.

[0166] The mobile device 610 includes a mobile OS 616 that provides a mobile operating experience to a user 650 of the mobile device 610. The mobile OS 616 manages the provision of cellular functionality, LF display system operation functionality, and user interface functionality suitable for a mobile environment including displays and I / O devices. The LF display system 612 may be a holographic display such as that provided by the LF display system 500. The mobile OS 616 manages application programs running on the mobile device 610 that may be controlled by a user through a user interface of the mobile OS 616. The user interface may be a graphical user interface (GUI) projected from the LF display system 612. The mobile OS 616 has various abstraction layers, each layer interfacing with the layers below. The mobile OS 616 also includes a kernel for managing mobile device resources such as CPU, memory, and I / O. The kernel provides functionality such as low-level scheduling of processes, inter-process communication, process creation and deletion, and interrupt handling. The other abstraction layers of the mobile OS interact with the kernel through system calls or application programming interface (API) layers. A user 650 of the mobile device 610 can interact with the mobile OS 616 via the user interface 614 .

[0167] A functional perspective of the Mobile OS is to view the Mobile OS 616 as an event-driven system, residing in a waiting loop for events delivered by the hardware of the mobile device 610, enabling appropriate responses to delivered events that are in the event queue. Examples of such events may be, for example, a touch on a particular location of a 2D touch screen, an incoming call over a cellular network, a message over a network interface, etc. Additionally, the event-driven Mobile OS 616 may respond to events reported by the LF display system 612, such as a touch of a hand, finger, or other body part against a projected holographic interface, a gesture reported by the tracking module 526 based on tracking data received from the tracking system 580, a pressure-related touch event on a matching tactile surface co-located with a projected holographic button, etc. Responses by the Mobile OS to these and other events delivered by the hardware components of the mobile device include actions performed by the Mobile OS at all abstraction layers of the Mobile OS functionality.

[0168] The mobile OS 616 or applications running on the mobile OS 616 manage the LF display system 612 to allow the LF display system 612 to access the hardware components of the mobile device as needed. Applications running on the mobile OS 616 can manage the presentation of holographic content as requested by a user. Similarly, the mobile OS 616 can present a holographic user interface and accept operational selections from the user based on tracking information provided by the tracking module 526, which can analyze data from the tracking system 580. The mobile OS 616 can run applications that access social media accounts and interface with the viewer profiling module 528 to present personalized holographic content. In general, the mobile OS 616 can interface with the controller 520 of the LF display system 612 to manage the controller, display holographic content, and provide access to the hardware components of the mobile device 610. In some embodiments, the mobile OS 616 can receive pluggable modules that function as drivers for managing the controller 520 of the LF display system. The mobile OS can interface with these modules or applications running on the mobile OS 616 and access API calls to the LF display system's controller 520 to display holographic content, provide or retrieve user profiling information, exchange tracking information, etc.

[0169] The mobile device 610 includes a power system 618 for providing power to various components of the mobile device 610. The power system 618 may include a power management system, one or more power sources (e.g., batteries, alternating current (AC), a recharging system, power fault detection circuitry, power converters or inverters, power status indicators, and other components associated with the generation, management, and distribution of power for the mobile device. As previously mentioned, the LF display system 612 can provide power to itself by accessing the power system 618 on the mobile device.

[0170] The mobile device 510 includes a GPS module 620. The GPS module 620 determines the location of the mobile device and provides this information for use in various applications. This location information can be provided to the LF display system 500 for use in applications that can leverage location information, such as map-based applications. In some embodiments, the LF display system 500 can display holographic content in response to data from the GPS module 620 and based on the location data received from the GPS module 620. For example, the LF display system 500 can be associated with a running navigation application to display holographic content indicating arrival at a destination when the GPS module provides location coordinates that are mapped to the destination by the navigation application running on the mobile device.

[0171] Network 630 may include any combination of local area and / or wide area networks and may use both wired and / or wireless communication systems. For example, local area networks may include WiFi and Bluetooth networks. In one embodiment, network 630 uses standard communication technologies and / or protocols. For example, network 630 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of networking protocols used to communicate over network 630 include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged over network 630 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 630 may be encrypted using any suitable technology or technologies.

[0172] One or more remote servers 640 can communicate with the mobile device 610 via the network 630. In one embodiment, the remote server 640 can be an application provider that provides applications for execution as well as content for display on the device 610. The remote server can be an online holographic application store that makes available holographic applications for downloading and execution on the mobile device in exchange for a transaction fee. The remote server 640 can be an online holographic content store that makes available holographic content for downloading to the mobile device in exchange for a transaction fee. The remote server 640 can be a third party application and / or content provider. New applications, new content, and updates to existing applications and content can be available from the remote server 640 for execution and / or display on the device 610 via either push-enabled or pull-enabled updates over the network 630. These applications and content can include holographic content tracks that are stored in the data store 522 for projection and display by the light field display system 612. Content available to the mobile display device 610 may include advertisements, product offers, and product coupons displayed by the light field display system 612 in association with the execution of software applications on the mobile device 610. For example, holographic content depicting popular animated characters may be downloaded from a third party content provider over the network 630 and stored in the light field display system 612 for use by the user 650 as a "holographic avatar" that represents the user 650 in one or more applications.

[0173] In one or more embodiments, to ensure that unauthorized network access to the device 610 is prevented, the system may use security settings to authenticate network access to the device 610 by entities to establish a secure connection to the device 610 and / or a secure connection for downloading content to the device 610. In some embodiments, the device 610 may capture authentication factors such as passwords and smart cards or use biometric authentication methods to enforce access control when accessing the mobile device 610 via either the network 630 or the user interface 614. Holographic user interface and "touch" interpretation in LF display systems

[0174] The following is a description of several embodiments of an LF display system for generating a holographic user interface (UI) for a user to interact with a mobile device. The description refers to elements shown in the LF display system 500 of FIG.

[0175] In some embodiments, the generated holographic UI is a user interface that includes one or more holographic UI objects that are presented to a user. In some embodiments, the presented holographic UI objects may also be part of a virtual image display that, for example, depicts a virtual screen floating above the surface of the mobile device on which the holographic objects appear to reside. In some embodiments, the presented holographic objects and / or virtual images may extend the physical UI on the device. The generated holographic UI objects may include, but are not limited to, 3D holographic objects such as "buttons," "knobs," "dials," "keypads," and the like. The generated holographic objects may also include 2D holographic objects, such as a holographic "screen" that displays a holographic menu that includes a list of multiple options. The user may be prompted (e.g., through an audio cue or a visually displayed prompt) to select one or more options by pointing (e.g., with a finger) or touching a particular option in the holographic menu.

[0176] In some embodiments, the presented holographic UI object may be augmented with haptic stimuli to generate a haptic surface co-located with the surface of the presented holographic object. Thus, for example, the sensory feedback system 570 may project ultrasonic energy, e.g., pressure waves, to generate a haptic surface on the holographic "button" that is collocated with the frontally displayed surface. The tracking system 580 may track the user's movement (e.g., the user's finger movement) toward the holographic "button." The movement information may be transmitted by the tracking system 580 to the controller 520. The tracking module 526 of the controller 520 may determine that the finger is "touching" or "pressing" the holographic "button" when the tracked finger is within a certain threshold distance of the presented haptic surface. Further, in response to this determination by the tracking module 526, the command library 532 may cause the LF processing engine 530 to render the holographic "button" and a collocated and presented haptic surface corresponding to the movement of the holographic "button" as if it had been "pressed" by the user.

[0177] In some embodiments, the holographic UI may be modified or updated in response to information received from the sensory feedback system 570. For example, an accelerometer in the sensory feedback system 570 may measure the linear acceleration of the mobile device and incrementally update the displayed holographic UI in sync with the measured acceleration. Similarly, a gyroscope in the sensory feedback system 570 may detect changes in the orientation of the mobile device and reorient the UI based on the detected changes. In some embodiments, the tracking module 526 may track the rotation of the user's head and reorient the displayed holographic UI in response.

[0178] In another embodiment, the holographic UI may include presenting a holographic character that greets the user. In some embodiments, the holographic character may be presented along with the presentation of the holographic UI object. In other embodiments, the holographic character may be presented to the user first, and then a holographic "menu" with the holographic UI object may be presented. In some embodiments, the holographic character may be modified in response to commands received from the user. In yet other embodiments, all interactions may include commands verbally issued by the user and received by the acoustic receiver of the sensory feedback system 570. In some embodiments, the presented holographic character may be augmented with acoustic stimuli to issue audio responses to the user via the mobile device's speaker.

[0179] In some embodiments, the holographic character may be generated and personalized by an AI model. The holographic character may have various characteristics that can be modified to suit preferences, such as appearance (e.g., that of a celebrity), clothing (or not), personality, voice (e.g., accent or celebrity voice), temperament, language (e.g., amount of swear words used), or other characteristics. The holographic character may have the appearance of a CG character (e.g., a superhero), a celebrity such as an actor or athlete, a random selection of one of many available avatars, or another appearance. The appearance of the holographic character may be accompanied by a recognizable acoustic voice (e.g., the voice of a celebrity matching the appearance) and can be easily modified by the user. The holographic character may also have access to a user profile or social media account to make it more appealing to the user. For example, the holographic character may provide playful feedback based on characteristics observed in photos and discussions displayed on the user's social media account, or provide jokes based on calendar reminders of the day of a scheduled event. For the user's entertainment, the holographic character may be in a bad mood or a good mood depending on the user's selection. Selections of the holographic character's characteristics to be viewed or experienced on the mobile device may be downloadable over the network 630 via a remote server 640 of a third party content provider, such as an online store, possibly in exchange for a transaction fee.

[0180] In some embodiments, the tracking module 526 determines the presence of a user in the target view of the LF display system based on information received from the tracking system 580, analyzes the received information, and transmits the analysis results to the command library 532. The command library 532 can use heuristics stored in the command library 532 based on the received analysis to determine that a user interface should be displayed to the user. Additionally, using the user profile information stored by the user profiling system 528, the command library 532 can apply other heuristics in the command library 532 to determine from the stored mappings that a customized user interface should be generated that includes the display of a particular holographic character that greets the user of the device. The command library 532 can issue a computational command to the LF processing engine 530 to display the determined holographic character that greets the user for display in the holographic object volume. Executing this computational command can include the LF processing engine 530 retrieving image data related to the rendering and display of the holographic character from the data store 522. The command library 532 can also trigger an acoustic transducer in the sensory feedback system 570 to transmit an audible greeting in synchronization with the displayed holographic character. The LF processing engine 530 can use the tracking information obtained from the tracking system 580 to render the holographic character to look at the user and / or make eye contact with the user or otherwise interact with the user. Verbal responses from the user can be received by an acoustic receiving device (e.g., a microphone or microphone array) located in the sensory feedback system 570, and this received response data is presented to the controller 520 for further processing.The command library 532 can use the data received from the sensory feedback system 570 and apply heuristics to determine that a navigation assistance application interface should be displayed to the user, for example, in response to a verbal utterance by the user.

[0181] In another embodiment, a user may configure the LF display system to always start with the presentation of a particular customized 3D holographic user interface augmented with the display of a particular desired application, such as an email application and / or a social media application. In some embodiments, a mobile device user may configure application settings for various applications associated with the LF display system to have a particular desired holographic user interface presented for the user's use while running the particular application.

[0182] In other embodiments, the holographic user interface includes additional or fewer functions than those described herein. Similarly, functionality may be distributed among modules and / or different entities of LF display system 500 in ways different from those described herein.

[0183] The tracking system 580 may track the user's movements (e.g., by using a camera, a set of cameras, a depth sensor, or some other sensor in the tracking system 580). The user's movements may be transmitted by the tracking system 580 to the controller 520 for touch interpretation. In some embodiments, touch interpretation is performed as follows: The tracking system 580 first tracks the user's movements as the user approaches the presented holographic object (e.g., with one or more fingers). The tracking system 580 transmits the tracked movement information to the controller 520. The tracking module 526 determines that the tracked fingers are within a threshold distance from the presented holographic object and, in response, determines that the user is "touching" the holographic object. In other embodiments, touch interpretation may include tracking other types of movements with respect to the presented holographic object by the tracking system 580. Tracked movements by the user may include, but are not limited to, determining that the user is performing one or more of the following with respect to one or more presented holographic objects: repositioning, moving, pushing, pressing, dialing, rotating, typing, poking, etc.

[0184] In some embodiments, the sensory feedback system 570 can sense the "touch" of a tactile surface that augments the displayed holographic object. This determination may be based on receiving input from a pressure sensor of the sensory feedback system 570 that detects changes caused by contact with the ultrasonic energy generating tactile surface. This information may be transmitted to the controller 520 for further analysis.

[0185] FIG. 7 is an illustration of an LF display system 700 according to one embodiment implemented as part of a mobile device according to one or more embodiments. The LF display system 700 is one embodiment of the LF display system 500 depicted in FIG. 5. The mobile device 710 includes an LF display assembly 720 of the LF display system 700. The LF display assembly 720 may be comprised of one or more LF display modules (e.g., the LF display module 210 of FIG. 2B or the LF display module 412 of FIG. 4A). In one or more embodiments, the mobile device 710 is accompanied by one or more control panels 730, which may include physical features (such as buttons) and / or a 2D digital display (including a 2D touch screen) for accessing the device by a particular user. Additionally, the mobile device 710 also includes an LF display assembly 720 (e.g., the LF display assembly 510 of FIG. 5) for generating holographic content for a user of the device (e.g., viewer A 772 or viewer B 774). LF display system 700 may also include any combination of the other components of LF display system 500 shown in FIG. 5, such as sensory feedback system 570, tracking system 580, user profiling system 590, and controller 520.

[0186] The LF display system can present holographic content, including at least one holographic object, to a user using a LF display assembly 720, which is composed of one or more LF display modules. The holographic object can be three-dimensional (3D), two-dimensional (2D), or a combination thereof. Additionally, the holographic object can be multi-colored (e.g., full color). As shown in FIG. 7, a holographic object volume 740 extends both in front of and behind the LF display surface of the LF display assembly 720 (e.g., similar to the holographic object volume 160 shown in FIG. 1). A holographic viewing volume 750 represents a viewing space in which a user can view the holographic objects presented by the LF display assembly 720. The holographic objects 762 and 764 can be positioned anywhere within the holographic object volume 740. Viewer A 772 and viewer B 774 can view the displayed holographic objects 762 and 764. Holographic objects 762 and 764 may also have tactile surfaces projected from LF display system 700 to match one or more surfaces of the holographic objects. The tracking system of LF display system 700 may track a user touching the surface of the displayed holographic object 762 or 764, which may cause the sensory feedback system to trigger a response. Thus, for example, the movement of finger 770 towards holographic object 762, as well as the "touch" of finger 770 on the tactile surface of holographic object 762 may be established by the tracking system of LF display system 700. The projected tactile surfaces matching holographic objects 762 and 764 may be appropriately adjusted by the sensory feedback system in response to the movement of finger 770 (e.g., changing the position of a tactile button simultaneously with a change in the hologram content indicating the button has been pressed). Customized holographic displays on mobile devices

[0187] In some embodiments, the LF display system can use the tracking system in combination with user information obtained by the user profiling system 528 of FIG. 5 to generate customized interactive experiences. For example, in an application with a holographic UI, the generated holographic "buttons" can be different sizes for different users. For example, they can be small for children and larger in size with bright colors to make them easier to see for older users.

[0188] Further, in some embodiments, the LF display system may be customized to display specific content for specific users. Thus, for example, the LF display system may be customized to display a holographic "button" to one user and a holographic "dial" to another user. Thus, with reference to FIG. 7, when tracking system 580 tracks the presence of viewer A 772, the configured settings may result in a customized display of holographic object 762 (only) (i.e., the holographic "button") for viewer A 772. If viewer B 774 is not present, holographic object 764 is not presented. Similarly, upon sensing the presence of viewer B 774, the configured settings may result in a customized display of holographic object 764 (i.e., the holographic "dial") for viewer 774.

[0189] In other embodiments, the LF display module can display a customized holographic character to interact with the device user (e.g., see, speak, hear, touch, or some combination thereof). One or more characteristics of the holographic character can be customized by the user. The customizable characteristics can be any of the visual, vocal, and personality characteristics of the holographic character. In some embodiments, some or all of the characteristics associated with the holographic character can be obtained from one or more online systems over a network in exchange for a transaction fee (e.g., adding a "Bart Simpson" face to the holographic character). Additionally, a tracking system can track the user's gaze. The tracking system can provide gaze information to the LF display system 500 to re-render the holographic character to make eye contact with the user interacting with the device.

[0190] In some additional embodiments, if multiple users use a mobile device, the user profiling system can build a user profile for each of the device users to store user settings or other user characteristics. A user profile can be built by the viewer profiling module 528 for each user, and these usage profiles can be stored in the data store 522.

[0191] The LF display system can utilize the user profiling system 590 to personalize holographic content to the user during each subsequent use of the device. For example, the LF display system 500 addresses the user by name (e.g., visually or audibly). This may include using tracking information to determine user identity or user authentication such as face or voice recognition, entering a password, or other methods. In some embodiments, the LF display system 500 generates holographic content corresponding to the user's response to previously generated holographic content. For example, in some embodiments, the AI ​​machine learning module of the LF processing engine 530 can learn that a user begins each day by checking their email, followed by solving a Sudoku puzzle. The LF display system 500 can generate a holographic 3D "home" screen display with an email application open, along with a display of a Sudoku puzzle that is retrieved and rendered based on data stored in the data store 522.

[0192] In some embodiments, customizing the light field display for a user may include a privacy mode in which the tracking system 580 tracks the user's location and generates customized holographic content that is only visible to the user. FIG. 2B illustrates multiple viewing sub-volumes 290 that may be formed by the LF display system 200. As described in the relevant sections, a user in the viewing sub-volume 290A may perceive holographic objects presented in the holographic object volume 255 that are not perceptible to users in the viewing sub-volumes 290B, 290C, and 290D, thus ensuring privacy of viewing. Thus, under these circumstances, a user reading an email on a holographic screen display may be assured that the display is only visible to the user and not to anyone nearby. In other embodiments, all users within the viewing volume of the LF display assembly may be able to see all holographic objects that are within the holographic viewing volume (although all users may see different perspectives depending on the user's location). A mobile device may provide a privacy setting to the LF display system to render holographic content in a privacy mode for one or more tracked users. Exemplary User Applications Using the LF Display System on a Mobile Device

[0193] Mobile devices are used in combination with a variety of software applications. Some of these applications can be further enhanced using the device's LF display system. Some of these examples are briefly described below.

[0194] In one embodiment, the LF display system 500 can be used with a puzzle application that includes blocks. The LF display system 500 can generate a holographic display of an "empty structure" with markings indicating the blocks that need to "fill" the holographic "empty structure." At the same time, the system can display a holographic set of "blocks" of various shapes to the viewing player. The image rendering data for generating the structure and the set of "blocks" can be stored in the data store 530 upon a previous download of the puzzle application over the network 630. When the player touches a block and moves the player's finger to the correct 3D location within the structure that will house the block, the tracking system 580 tracks this movement and the LF processing engine 530 generates a holographic rendering of the "structure" that moves the block to match the position of the player's finger, and the corresponding 3D location is "filled" as the block is "moved" to that location.

[0195] Other holographic user applications may include using the sensory feedback system 570 in concert with the tracking system 580 to generate holographic renderings that "move" when "touched." Examples include presenting holographic renderings to facilitate furniture assembly, instructional do-it-yourself applications, and educational applications, such as medical students learning human anatomy using holographic human body renderings that change in response to the user's specified movements.

[0196] In one embodiment, the LF display system 500 may generate advertisements for display, and the ultrasonic emitters of the sensory feedback system 570 may be used to generate tactile surface samples that provide tactile information to the user about the product (e.g., a carpet manufacturer may display an advertisement showing rug samples of various textures that a user can experience).

[0197] Other example applications include generating holographic renderings of a "cyberpet" that respond to sensory stimuli such as touch or voice commands, providing the "cyberpet" owner with an immersive experience of the "cyberpet." The tracking system 580 and sensory feedback system 570 together can provide the user with a responsive experience with the holographic "cyberpet."

[0198] 8 is a flow chart illustrating a process for enabling user interaction with a mobile device using a light field display system, according to one or more embodiments. In one embodiment, the process of FIG. 8 is performed by light field display system 500 (as shown in FIG. 5). Other entities may perform some or all of the steps of this process in other embodiments. Similarly, embodiments may include different and / or additional steps, or perform steps in a different order.

[0199] The light field display system 500 receives commands from a user (810). The user commands can be received via physical controls, such as physical buttons that can be pressed and a physical touch-enabled screen, all of which can be located on the mobile device and can be part of the sensory feedback system 570. In another example, the user commands can be received as voice commands at an acoustic receiver, such as an acoustic microphone, that is part of the sensory feedback system 570. The user commands can also be received through the user's interaction with a holographic object, such as a displayed holographic UI. The user commands can also be received through a tracking system 580 that is tracking the user via instructions from the tracking module 526. In one embodiment, the user commands can be captured as gestures, body movements, or facial expressions by the user, recorded by the tracking system 580, and analyzed by the tracking module 526. In another embodiment, the user commands can be received when the tracking system 580 detects that the user has touched a tactile holographic surface, such as a holographic "button" on a displayed holographic UI. The user commands can also be received via the mobile device's communication network. Received user commands are relayed via the controller to the command library 532 for interpretation and analysis.

[0200] The light field display system 500 processes the user command to determine a computation command (820). A command library 532 in the light field display system 500 may associate the received user command with user profile information, possibly stored by the user profiling system 528, and may work with a tracking module, if necessary, to interpret the user's gestures, body movements, or facial expressions and apply stored heuristics to determine a computation command based on the received user command. One example of a determined computation command is a command to generate more holographic content for a user of the mobile device to view. Another example may be to modify the generated holographic content. The determined computation command may activate a sensor in the LF display system, such as, for example, an imaging sensor to track the user's gaze. The determined computation command may also activate a control in the mobile device. One example of a determined computation command may be to generate more holographic content for a user of the mobile device to view. Another example may be to modify the generated holographic content. The generated holographic content may be modified depending on the value of a parameter received at the LF display system. Thus, for example, a holographic "sphere" may be projected to indicate download progress as an application is being downloaded over a network to a mobile device. The value of a parameter indicating the percentage of downloaded content as the download progresses may be used by the LF display system to render the holographic sphere filling with a color indicating the progress of the application download. The determined computational command may activate a sensor in the LF display system, such as, for example, a trigger sensor for tracking the user's gaze. The determined computational command may also activate a control in the mobile device.For example, a calculation command can be used to increase the volume of a mobile device's speaker or set a particular audio tone as an alarm alert.

[0201] The light field display system generates holographic content for display based on the computational commands (830). The LF processing engine 530 of the light field display system 500 receives the determined computational commands from the command library 532 and generates holographic content for display based on the computational commands (830).

[0202] The light field display system 500 displays the generated holographic content (840). The controller 520 sends display instructions for projecting the holographic content from the LF processing engine 530 to the LF display assembly 510. The LF display assembly 510 displays the holographic content to a user. Additional configuration information

[0203] The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will recognize that numerous modifications and variations are possible in light of the above disclosure.

[0204] Some portions of this specification describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has proven convenient at times to refer to arrangements of these operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0205] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented in a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0206] The embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, that may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0207] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information resulting from the computing processes, which information is stored in a non-transitory, tangible, computer-readable storage medium, and may include any embodiment of the computer program products or other data combinations described herein.

[0208] Finally, the language used herein has been selected primarily for ease of reading and instructional purposes, and may not have been selected to define or limit the subject matter of the present invention. Accordingly, the scope of the present disclosure is intended to be limited not by this detailed description, but by any claims issued on an application based thereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure, which is set forth in the following claims. [Other possible items] [Item 1] 1. A mobile device comprising: A controller configured to generate holographic content; an LF display assembly including one or more LF display modules configured to present the generated holographic content to a user of the mobile device; and a command interface configured to recognize one or more commands from the user. [Item 2] Item 10. The mobile device of item 1, wherein the controller is configured to update the generated holographic content in response to changes in an operational state of the mobile device. [Item 3] Item 10. The mobile device of item 1, wherein the controller is further configured to modify holographic content based on one or more recognized commands from the user. [Item 4] 2. The mobile device of claim 1, wherein the one or more recognized commands cause a change in an operational state of the mobile device. [Item 5] The LF display system comprises: a tracking system including one or more tracking devices configured to obtain information regarding the user of the mobile device; Item 10. The mobile device of item 1, wherein the controller is configured to generate the holographic content for the user based in part on information obtained by the tracking system. [Item 6] Item 6. The mobile device of item 5, wherein the generated holographic content includes a holographic user interface (UI) with one or more holographic features for controlling operation of the mobile device, and the command interface is further configured to recognize the one or more commands received from the user via the generated holographic UI. [Item 7] Item 6. The mobile device of item 5, wherein the command interface is further configured to receive commands from the user via the holographic UI based on a determination that the user is within a threshold distance of one or more holographic features in a presented holographic UI. [Item 8] 6. The mobile device of item 5, wherein the information acquired by the tracking system includes any combination of the user's location, the user's movement, the user's gestures, the user's facial expressions, the user's gaze, the user's gender, and the user's age. [Item 9] Item 6. The mobile device of item 5, wherein the controller is configured to generate the holographic content using the information obtained by the tracking system and an artificial intelligence model. [Item 10] Item 6. The mobile device of item 5, wherein the tracking system includes any combination of sensors including a light field camera, one or more 2D imaging sensors, and a depth sensor. [Item 11] Item 11. The mobile device of item 10, wherein the controller is further configured to generate the holographic content by capturing data from the one or more sensors in the tracking system and generating the holographic content based on the captured data. [Item 12] Item 12. The mobile device of item 11, wherein the controller is further configured to modify the generated holographic content based on data captured from the one or more sensors in the tracking system. [Item 13] The LF display system comprises: Item 10. The mobile device of item 1, further comprising a sensory feedback system including at least one sensory feedback device configured to receive sensory feedback when the holographic content is presented. [Item 14] Item 14. The mobile device of item 13, wherein the command interface is further configured to recognize the received sensory feedback as one or more commands received from the user. [Item 15] Item 14. The mobile device of item 13, wherein the sensory feedback system includes an ultrasonic transducer, a pressure sensor, a force sensor, a temperature sensor, a touch sensor, a proximity sensor, or any combination thereof. [Item 16] Item 14. The mobile device of item 13, wherein the sensory feedback system further comprises one or more of a button, a dial, a knob, a keypad, a fingerprint sensor, a joystick, other input hardware, or any combination thereof. [Item 17] Item 14. The mobile device of item 13, wherein the sensory feedback system further comprises a microphone and voice recognition software. [Item 18] The LF display system comprises: Item 10. The mobile device of item 1, further comprising a sensory feedback system including at least one sensory feedback device configured to provide sensory feedback when the holographic content is presented. [Item 19] Item 19. The mobile device of item 18, wherein the controller is further configured to augment the generated holographic content with sensory content including tactile stimuli, acoustic stimuli, temperature stimuli, olfactory stimuli, pressure stimuli, force stimuli, or any combination thereof. [Item 20] Item 19. The mobile device of item 18, wherein the sensory feedback system further comprises an ultrasonic energy projection device configured to generate a volumetric tactile surface within a threshold distance of or coinciding with a surface of the presented holographic object. [Item 21] 21. The mobile device of claim 20, wherein the ultrasonic energy projection device is further configured to adjust one or more of the resistance of the generated volumetric tactile surface to a user touch, the texture of the generated volumetric tactile surface, or the tactile intensity based on the value of a parameter received at the controller. [Item 22] Item 6. The mobile device of item 5, further comprising an ultrasonic energy projection device configured to generate a volumetric tactile surface generated based in part on the information acquired by the tracking system. [Item 23] 23. The mobile device of claim 22, wherein the ultrasonic energy projection device is further configured to update the generated volumetric tactile surface in response to information acquired by the tracking system. [Item 24] 23. The mobile device of claim 22, wherein the ultrasonic energy projection device is further configured to generate the volumetric tactile surface using the information acquired by the tracking system and an artificial intelligence model. [Item 25] The controller: obtaining at least a portion of the holographic content in a first format from an online system via a network interface; Item 1 , the mobile device of item 1 , further configured to convert the vectorized format of the holographic content into a second format for display on an LF display assembly. [Item 26] Item 26. The mobile device of item 25, wherein the first format includes a vectorized format and the second format includes a rasterized format. [Item 27] Item 10. The mobile device of item 1, wherein the LF display system is configured to modify the presentation of the holographic content based on environmental characteristics at a location where the holographic content is presented to the user. [Item 28] Item 1 , the mobile device being configured to modify the presentation of the holographic content based on values ​​of parameters received at the LF display system. [Item 29] Item 1. The LF display system of item 1, wherein the controller is further configured to generate the holographic content based in part on a user profile maintained for the user at the LF display system. [Item 30] The LF display system comprises: a tracking system including one or more tracking devices configured to obtain information regarding the user of the mobile device; Accessing the information obtained by the tracking system; configured to process the information to identify the user; Based on the information acquired by the tracking device, generating a user profile for said user; and modifying the user profile of the user; Item 10. The LF display system of item 1, wherein the controller is further configured to generate holographic content for the user based in part on the user profile. [Item 31] The controller uses the user profile and an artificial intelligence model to: generating said holographic content; Item 31. The LF display system of item 30, further configured to perform one or more of: a. [Item 32] The user profiling module includes: further configured to update the user profile using information from the user's social media account; Item 31. The LF display system of item 30, wherein the controller is configured to generate the holographic content based in part on the updated user profile. [Item 33] Item 13. The mobile device of item 1, wherein the holographic content is presented in conjunction with an application running on the mobile device. [Item 34] Item 34. The mobile device of item 33, wherein the presented holographic content is based on location coordinates obtained from a Global Positioning System (GPS). [Item 35] Item 34. The mobile device of item 33, wherein the presented holographic content is based on content received from an online source. [Item 36] Item 34. The mobile device of item 33, wherein the presented holographic content is based on content received from an online source in response to an operational selection made by the user. [Item 37] Item 14. The mobile device of item 1, wherein the presented holographic content includes a holographic character. [Item 38] Item 38. The mobile device of item 37, wherein characteristics associated with the presented holographic character are configured, at least in part, by the user. [Item 39] Item 38. The mobile device of item 37, wherein the characteristics associated with the presented holographic character include visual, audio, and personality characteristics of the presented holographic character. [Item 40] Item 38. The mobile device of item 37, wherein characteristics associated with the presented holographic character are retrieved from an online system via a network. [Item 41] 40. The mobile device of claim 39, wherein the features are retrieved from the online system via the network in exchange for a transaction fee. [Item 42] Item 9. The mobile device of item 8, wherein the controller is configured to generate a holographic character for the user based on the information acquired by the tracking system. [Item 43] Item 43. The mobile device of item 42, wherein the LF display assembly is configured to update the eyes of the generated holographic character to maintain eye contact with the user's gaze. [Item 44] The LF display system comprises: receiving security credentials from a user; authenticating the received security credentials; enabling secure access to one or more features of the mobile device; Item 1. The mobile device of item 1, further comprising a security module having an authentication module configured to enable generation of specific holographic content in response to the authentication. [Item 45] 2. The mobile device of claim 1, wherein the operational functions of the controller, the LF display assembly, and the command interface are managed in association with other operational functions of the mobile device by a mobile operating system (mobile OS). [Item 46] Item 46. The mobile device of item 45, wherein the mobile OS receives information related to input received from one or more of the controller, the LF display assembly, and the command interface. [Item 47] Item 47. The mobile device of item 46, wherein the mobile OS performs actions at one or more layers of functional abstraction in response to the received information. [Item 48] Item 46. The mobile device of item 45, wherein the mobile OS manages the operational functions of the controller, the LF display assembly, and the command interface through one or more of pluggable modules or applications that run on the mobile OS. [Item 49] Item 49. The mobile device of item 48, wherein the mobile OS interfaces with the controller to display holographic content. [Item 50] Item 46. The mobile device of item 45, wherein the mobile OS facilitates the LF display system to access hardware components of the mobile device.

Claims

[Claim 1] 1. A mobile device comprising: A controller configured to generate holographic content; an LF display assembly including one or more LF display modules configured to present the generated holographic content to a user of the mobile device; and a command interface configured to recognize one or more commands from the user.

Images