Light Field Displays for Mobile Devices

Abstract

A light field display system is implemented in a mobile device to present holographic content including at least one holographic object to a user, thereby providing an immersive operating experience for the user. The system generates and presents holographic content for the user. In one embodiment, the system receives a command from the user. In some embodiments, the presented holographic content may include a holographic user interface used by the system to receive the command from the user of the mobile device. Subsequently, the system recognizes the received command, determines one or more computing commands executed by the system, and executes the determined computing commands.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042414 and PCT / US2017 / 042679, the entire contents of which are incorporated herein by reference in their entirety. Background Art

[0003] The present disclosure relates to mobile devices such as smartphones and tablet computers, and in particular, to light field displays in mobile devices.

[0004] Mobile devices typically rely on two-dimensional (2D) visual displays - for displaying content and for user interfaces. In addition to displaying two-dimensional content including text, images, and videos, the 2D visual display on the mobile device is designed to provide an interactive interface for displaying the range of functions provided to the user by the mobile device, and for allowing the user to select and configure the desired functions. Interactivity in the mobile device is provided using physical elements such as buttons, keyboards, or touch interfaces on the 2D visual display. However, the 2D visual display is limited in size to the coverage area occupied by the display on the surface of the mobile device. Therefore, the displayed content is also limited to the size of the 2D visual display, which limits interactivity and the viewing experience that can be provided to the mobile device user. Specifically, the 2D visual display has limited opportunities to provide an immersive operating experience to the user.

[0005] The described embodiments relate to techniques including a light field display system in a mobile device that provides an immersive operating experience to a mobile device user when using the device. Summary of the invention

[0006] A light field (LF) display system in a mobile device generates holographic content for viewing by a user of the mobile device. The generated holographic content may be a holographic user interface displayed to the user of the mobile device. The light field display system may receive commands from the user. The LF display system determines a computing command based on the received user command, and appropriately executes the determined computing command. In some embodiments, the holographic content enhances a tactile surface on the generated holographic content. The tactile surface may provide a sense of touch to the user.

[0007] In some embodiments, a mobile device includes a LF display system including a controller configured to generate holographic content, a LF display assembly, and a command interface. The LF display assembly includes one or more LF display modules configured to present the holographic content as a holographic object volume to one or more users of the mobile device located in a viewing volume of the one or more LF display modules.

[0008] In some embodiments, the LF display system includes a tracking system and / or a viewer profile module. The tracking system and viewer profile module may monitor and store characteristics of a user of the mobile device, a user profile describing the user, and / or the user's response to holographic content presented during operation of the mobile device. Holographic content may be created for display based on any of the monitored or stored information.

[0009] In some embodiments, a user of a mobile device may interact with the holographic content, and the interaction may serve as an input to the holographic content creation system. In some embodiments, the LF display system of the mobile device may generate sensory stimuli in addition to the holographic content. For example, in some embodiments, some or all of the LF display systems include one or more ultrasonic speakers. Multiple ultrasonic speakers may be configured to generate a tactile surface consistent with at least a portion of the holographic content. The tracking system is configured to track the user's interaction with the holographic object (e.g., images captured via an imaging sensor of the LF display module and / or some other camera). And the LF display system is configured to provide holographic content based on the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of a light field display module for presenting a holographic object according to one or more embodiments.

[0011] Figure 2A is a cross-section of a portion of a light field display module according to one or more embodiments.

[0012] Figure 2B is a cross-section of a portion of a light field display module according to one or more embodiments.

[0013] Figure 3A is a perspective view of a light field display module according to one or more embodiments.

[0014] Figure 3B is a cross-sectional view of a light field display module including interwoven energy relay devices according to one or more embodiments.

[0015] Figure 4A A perspective view of a portion of a light field display system tiled in two dimensions to form a single-sided seamless surface environment in accordance with one or more embodiments.

[0016] Figure 4B is a perspective view of a portion of a light field display system in a multi-faceted seamless surface environment according to one or more embodiments.

[0017] Figure 4C is a top view of a light field display system with a converging surface in a wing-like configuration according to one or more embodiments.

[0018] Figure 4D is a side view of a light field display system having a converging surface in an inclined configuration according to one or more embodiments.

[0019] Figure 4E is a top view of a light field display system having a polymeric surface on a front panel of the device according to one or more embodiments.

[0020] Figure 5 is a block diagram of a light field display system according to one or more embodiments.

[0021] Figure 6 is a block diagram of a network environment for a light field display system in a mobile device according to one or more embodiments.

[0022] Figure 7 A display that is an example of an embodiment that uses a light field display for a mobile device.

[0023] Figure 8 A flow chart illustrating a method for implementing interaction with a mobile device using a light field display system according to one or more embodiments. DETAILED DESCRIPTION

[0024] Overview

[0025] A light field (LF) display system is implemented in a mobile device to present holographic content including at least one holographic object to a mobile device user. The terms "user," "mobile device user," "device user," and "viewer" are used interchangeably herein. The terms "mobile device" and "device" are also used interchangeably herein to refer to a handheld electronic computing device designed for portability and compactness. Examples of mobile devices include, but are not limited to, mobile phones, smart phones, PDAs, tablet computers, e-readers, portable music layers, portable computers, etc.

[0026] The LF display system in the mobile device is integrated with the device functionality to provide an immersive operating experience to the user when using the device. The mobile device functionality is implemented by executing a computer program module (e.g., a web browser or a software client application) that allows the device user to browse the Internet, consume media content, and execute a large variety of software applications. The LF display system includes an LF display assembly configured to present holographic content including one or more holographic objects that will be visible to a viewer in a viewing volume of the LF display system. The holographic content presented by the LF display system is integrated with the device functionality; this increases the immersive experience of the device user when executing software applications on the device. The holographic content presented by the LF display system can also be enhanced with other sensory stimuli (e.g., touch, sound, temperature, pressure, force, or smell). For example, an ultrasonic transmitter in the LF display system can project ultrasonic pressure waves that create a body tactile projection. The body tactile projection provides a tactile surface corresponding to some or all of the projected holographic objects. The holographic content may include additional visual content (i.e., 2D or 3D visual content). In implementations with multiple energy sources (i.e., holographic objects that provide the correct haptic sensation and sensory stimulation at any given point in time), energy source coordination to achieve a cohesive experience is part of the LF system. For example, the LF system may include a controller that coordinates the presentation of holographic content and haptic surfaces.

[0027] In some embodiments, the LF display system may include elements that enable the system to project at least one type of energy and simultaneously sense at least one type of energy for the purpose of responding to the user and generating an interactive experience. The sensed energy can be used to record how the viewer responds to the holographic content. For example, the LF display system may project both holographic objects for viewing and ultrasound for tactile perception, and simultaneously record imaging information for tracking the user and other scene analysis, while also sensing ultrasound to detect the user's touch response. As an example, such a system may project a holographic object that modifies its "behavior" according to the touch stimulus when virtually "touched" by a viewer. The display system component that performs energy sensing of the environment may be integrated into the display surface, or it may be a dedicated sensor separate from the display surface, such as an ultrasound speaker and an imaging capture device such as a camera.

[0028] The LF display system can be constructed to provide various experiences of the mobile device based on the generated holographic objects. For example, in one embodiment, the LF display system on the mobile device can be implemented to present a holographic view of the terrain to the user through a map viewing application. The LF display assembly can form a single-sided or multi-sided seamless surface environment. The holographic content can strengthen or enhance physical objects, such as the mobile device screen itself. For example, in one embodiment, the LF display system on the device can be implemented to present a 3D holographic progress indicator gif, such as an envelope sphere, to indicate that the application is loading on the device. In addition, the user can freely view the holographic content without the need for a goggle device (e.g., 3D glasses or any other head protection).

[0029] In some embodiments, the LF display system may receive input via a tracking system and / or a sensory feedback system. Based on the input, the LF display system may adjust the holographic content and provide feedback to relevant components.

[0030] The LF display system may also incorporate a system for tracking the movement of a user within the viewing volume of the LF display system. The tracked movement of the user may be used to enhance the user's experience of using the mobile device. For example, the LF display system may use tracking information to facilitate user interaction with holographic content (e.g., pressing a holographic button). The LF display system may use tracking information to monitor the position of a finger relative to a holographic object. For example, the holographic object may be a button that a viewer may "press". The LF display system may project ultrasonic energy to generate a tactile surface that corresponds to the button and occupies substantially the same space as the button. The LF display system may use tracking information to dynamically move the position of the tactile surface, and dynamically move the button as the button is "pressed" by the viewer. The LF display system may use tracking information to render a holographic character that looks at the user and / or makes eye contact with it, or otherwise interacts. The LF display system may use tracking information to render a holographic object that "touches" the user by projecting it near the user, wherein the ultrasonic speaker creates a tactile surface through which the holographic object generates a touch sensation for the user.

[0031] In addition, the LF display system may incorporate a user profile system for identifying each user in order to provide personalized content to each device user. The user profile system may further record information about device usage (e.g., type of content viewed, locations traveled), which may be retrieved for subsequent use for personalized holographic content.

[0032] Light Field Display System Overview

[0033] Figure 1FIG100 is a diagram of a light field (LF) display module 110 presenting a holographic object 120 according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system uses one or more LF display modules to present holographic content including at least one holographic object. The LF display system can present the holographic content to one or more viewers. In some embodiments, the LF display system can also enhance the holographic content with other sensory content (e.g., touch, audio, smell / olfaction, pressure, force, or temperature). For example, as discussed below, the projection of focused ultrasound can generate an air tactile sensation that can simulate the surface of some or all of the holographic object. The LF display system includes one or more LF display modules 110, and is discussed in detail below with respect to FIGS. 2-8.

[0034] The LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more viewers (e.g., viewer 140). The LF display module 110 includes an energy device layer (e.g., an emissive electronic display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). In addition, the LF display module 110 may include an energy relay layer for combining multiple energy sources or detectors together to form a single surface. At a high level, the energy device layer generates energy (e.g., holographic content), which is then guided to a region in space using an energy waveguide according to one or more four-dimensional (4D) light field functions. The LF display module 110 may also simultaneously project and / or sense one or more types of energy. For example, the LF display module 110 may be able to project a holographic image as well as an ultrasonic tactile surface in a viewing volume while detecting imaging data from the viewing volume. The operation of the LF display module 110 is discussed in detail below with respect to FIGS. 2-3.

[0035] The LF display module 110 uses one or more 4D light field functions (e.g., derived from a plenoptic function) to generate a holographic object within a holographic object volume 160. The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. In addition, the holographic object may be multi-color (e.g., full color). The holographic object may be projected in front of a screen plane, behind a screen plane, or separated by a screen plane. The holographic object 120 may be presented such that the holographic object may be perceived anywhere within the holographic object volume 160. The holographic object within the holographic object volume 160 may appear to be floating in space to the viewer 140.

[0036] The holographic object volume 160 represents a volume in which the viewer 140 can perceive a holographic object. The holographic object volume 160 can extend in front of the display surface 150 (i.e., toward the viewer 140) so that the holographic object can appear in front of the plane of the display area 150. In addition, the holographic object volume 160 can extend behind the surface of the display area 150 (i.e., away from the viewer 140), allowing the holographic object to be presented as if it is behind the plane of the display area 150. In other words, the holographic object volume 160 can include all light rays originating from the display area 150 (e.g., being projected) and can converge to create the holographic object. Herein, the light rays can converge at a point in front of the display surface, at the display surface, or behind the display surface. More simply, the holographic object volume 160 encompasses all volumes from which the viewer can perceive the holographic object.

[0037] The viewing volume 130 is a volume of space from which a holographic object (e.g., holographic object 120) rendered within the holographic object volume 160 by the LF display system can be fully seen. A holographic object can be rendered within the holographic object volume 160 and viewed within the viewing volume 130 such that the holographic object is indistinguishable from an actual object. The holographic object is formed by projecting the same light rays that are generated from the surface of the object when physically rendered.

[0038] In some cases, the holographic object volume 160 and the corresponding viewing volume 130 may be relatively small - so that they are designed for a single viewer. In other embodiments, as described below with respect to, for example Figure 4A and 4B As discussed in detail, LF display modules may be enlarged and / or tiled to create larger holographic object volumes and corresponding viewing volumes that may accommodate a wide range of viewers (e.g., one to thousands). The LF display modules presented in the present disclosure may be constructed such that the entire surface of the module contains holographic imaging optics, there are no dead or dead spaces, and no bezels are required. In these embodiments, the LF display modules may be tiled such that the imaging area is continuous across the seams of the tiles between LF display modules, and the join lines between tiled modules are barely detectable using the visual acuity of the eye. It is noteworthy that in some configurations, although not described in detail herein, a portion of the display surface may not contain holographic imaging optics.

[0039] The flexible size and / or shape of the viewing volume 130 allows the viewer to be unconstrained within the viewing volume 130. For example, the viewer 140 can move to different locations within the viewing volume 130 and see different views of the holographic object 120 from corresponding viewing angles. Figure 1, the viewer 140 is located at a first position relative to the holographic object 120, so that the holographic object 120 appears to be a front view of the dolphin. The viewer 140 can move to other positions relative to the holographic object 120 to see different views of the dolphin. For example, the viewer 140 can move so that he / she sees the left side of the dolphin, the right side of the dolphin, etc., just as if the viewer 140 is watching an actual dolphin and changes his / her relative position to the actual dolphin to see different sides of the dolphin. In some embodiments, the holographic object 120 is visible to all viewers within the viewing volume 130, and all viewers have an unobstructed view of the holographic object 120 (i.e., not blocked by objects / people). These viewers can be unconstrained so that they can move around within the viewing volume to see different perspectives of the holographic object 120. Therefore, the LF display system can present the holographic object so that multiple unconstrained viewers can see different perspectives of the holographic object simultaneously in the real world space, as if the holographic object was physically present.

[0040] In contrast, conventional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) typically require each viewer to wear some kind of external device (e.g., 3-D glasses, near-eye displays, or head-mounted displays) in order to see the content. Additionally and / or alternatively, conventional displays may require the viewer to be constrained to a specific viewing position (e.g., on a chair with a fixed position relative to the display). For example, when viewing an object depicted by a stereoscopic display, the viewer is always focused on the display surface, rather than on the object, and the display will always present only two views of the object, which will follow the viewer attempting to move around the perceived object, resulting in a perceptual distortion of the object. However, with a light field display, a viewer of a holographic object presented by an LF display system does not need to wear an external device in order to see the holographic object. The LF display system presents the holographic object in a manner visible to the viewer, in much the same way that the viewer can see a physical object, without the need for special goggles, glasses, or head-mounted accessories. Furthermore, the viewer can view the holographic content from any position within the viewing volume.

[0041] It is worth noting that there are size limits on potential position receptors of holographic objects within the holographic object volume 160. To increase the size of the holographic object volume 160, the size of the display area 150 of the LF display module 110 may be increased and / or multiple LF display modules may be tiled together in a manner to form a seamless display surface having an effective display area that is larger than the display area of ​​the individual LF display modules. Figure 4A and 4B Some embodiments related to tiling LF display modules are discussed. Figure 1As shown in , the display area 150 is rectangular, resulting in the holographic object volume 160 being pyramidal. In other embodiments, the display area may have some other shape (eg, hexagonal), which also affects the shape of the corresponding viewing volume.

[0042] Additionally, while the above discussion focuses on presenting the holographic object 120 within a portion of the holographic object volume 160 between the LF display module 110 and the viewer 140, the LF display module 110 may additionally present content that appears behind the plane of the display area 150. For example, the LF display module 110 may make a portion of the displayed holographic object appear so that the display area 150 appears to be the surface of the ocean from which the holographic object 120 is jumping out. The displayed content may enable the viewer 140 to look through the displayed surface to see underwater marine life. Furthermore, the LF display system may generate content that moves seamlessly around the holographic object volume 160, including behind and in front of the plane of the display surface 150.

[0043] Figure 2A 2 is a cross-section 200 of a portion of an LF display module 210 according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a display area shape different from the display area 150. In the illustrated embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 have components different from those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be allocated between components in a manner different from that described herein.

[0044] The display system described herein presents an energy emission that replicates the energy that typically surrounds an object in the real world. Here, the emitted energy is directed from each coordinate on the display surface toward a specific direction. In other words, each coordinate on the display surface acts as a projection location for the emitted energy. The directed energy from the display surface causes many energy rays to converge, which can thereby create a holographic object. For example, for visible light, the LF display will project a very large number of rays that converge at any point in the holographic object volume, so from the perspective of a viewer positioned farther than the projected object, the rays appear to come from the surface of a real-world object positioned in an area of ​​this space. In this way, the LF display generates reflected light that will leave the surface of this object from the viewer's perspective. The viewer's perspective can change on any given holographic object, and the viewer will see different views of the holographic object.

[0045] As described herein, the energy device layer 220 includes one or more electronic displays (e.g., emissive displays such as OLEDs) and one or more other energy projection and / or energy receiving devices. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include a plurality of pixels, each with an independently controlled intensity. Many types of commercial displays can be used in LF displays, such as emissive LED and OLED displays.

[0046] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic projection device generates one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or a combination thereof. An array of ultrasonic pressure waves may be used for somatic tactile sensations (e.g., at the surface of the holographic object 250). Audible pressure waves are used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming that the holographic object 250 is a dolphin, one or more acoustic projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the dolphin so that the viewer can touch the holographic object 250; and (2) provide audio content corresponding to the sounds made by the dolphin (e.g., clicks, chirps, or squeaks). An acoustic receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audible pressure waves within a local area of ​​the LF display module 210.

[0047] The energy device layer 220 may also include one or more imaging sensors. The imaging sensors are sensitive to light in at least the visible light band, and in some cases, may be sensitive to light in other bands (e.g., infrared). The imaging sensors may be, for example, a complementary metal oxide semiconductor (CMOS) array, a charge coupled device (CCD), a photodetector array, some other sensor that captures light, or some combination thereof. The LF display system may use the data captured by the one or more imaging sensors for position location tracking of the viewer.

[0048] The energy relay layer 230 relays energy (e.g., electromagnetic, mechanical pressure waves, etc.) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270, and it relays energy between the two surfaces. The first surface 265 of each energy relay element can be coupled to one or more energy devices (e.g., an electronic display or an acoustic projection device). The energy relay element can be composed of, for example, glass, carbon, optical fiber, optical film, plastic, polymer, or a combination thereof. In addition, in some embodiments, the energy relay element can adjust the amplification (increase or decrease) of the energy transferred between the first surface 265 and the second surface 270. If the repeater provides amplification, the repeater can take the form of an array of bonded conical repeaters called cones, wherein the area of ​​one end of the cone can be substantially larger than the area of ​​the opposite end. The large ends of the cones can be bonded together to form a seamless energy surface 275. One advantage is that space is created on the multiple small ends of each cone to accommodate the mechanical envelope of multiple energy sources, such as the bezels of multiple displays. This additional room allows the energy sources to be placed side by side on the small cone sides, where the active area of ​​each energy source directs energy into the small cone surface and relays to the large seamless energy surface. Another advantage of using a cone repeater is that there is no non-imaging dead space on the combined seamless energy surface formed by the large ends of the cones. There are no borders or borders, and therefore the seamless energy surfaces can be tiled together later to form a larger surface with almost no seams, depending on the visual acuity of the eye.

[0049] The second surfaces of adjacent energy-relay elements converge to form energy surface 275. In some embodiments, the spacing between edges of adjacent energy-relay elements is less than the minimum perceptible outline defined by the visual acuity of a human eye having greater than 20 / 40 vision, such that energy surface 275 is effectively seamless from the perspective of viewer 280 within viewing volume 285.

[0050] In some embodiments, the second surfaces of adjacent energy relay elements are fused together by a processing step that may include one or more of pressure, heat, and chemical reaction, in such a way that there are no seams between them. And in still other embodiments, an array of energy relay elements is formed by molding one side of a continuous block of relay material into an array of small tapered ends, each energy relay element being configured to transfer energy from an energy device attached to the small tapered end to a larger area of ​​a single combined surface that is not subdivided.

[0051] In some embodiments, one or more of the energy relay elements exhibit energy localization, wherein the efficiency of energy transfer in a longitudinal direction substantially perpendicular to surfaces 265 and 270 is much higher than the efficiency of transfer in a perpendicular transverse plane, and wherein the energy density is highly localized in this transverse plane as the energy wave propagates between surfaces 265 and 270. This localization of energy enables efficient relaying of energy distributions (e.g., images) between these surfaces without any significant loss of resolution.

[0052] The energy waveguide layer 240 guides energy from a location (e.g., a coordinate) on the energy surface 275 to a specific propagation path from the display surface outward into the holographic viewing volume 285 using waveguide elements in the energy waveguide layer 240. The energy propagation path is defined by at least two angular dimensions determined by the energy surface coordinate location relative to the waveguide. The waveguide is associated with a spatial 2D coordinate. These four coordinates together form a four-dimensional (4D) energy field. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 will guide light from a location on the seamless energy surface 275 through the viewing volume 285 along different propagation directions. In various examples, the light is guided according to a 4D light field function to form a holographic object 250 within the holographic object volume 255.

[0053] Each waveguide element in the energy waveguide layer 240 may be, for example, a lenslet composed of one or more elements. In some configurations, the lenslet may be a positive lens. The positive lens may have a spherical, non-spherical, or free-form surface profile. Additionally, in some embodiments, some or all of the waveguide elements may include one or more additional optical components. The additional optical components may be, for example, energy suppression structures such as baffles, positive lenses, negative lenses, spherical lenses, non-spherical lenses, free-form lenses, liquid crystal lenses, liquid lenses, refractive elements, diffractive elements, or some combination thereof. In some embodiments, at least one of the lenslets and / or additional optical components is capable of dynamically adjusting its optical power. For example, the lenslet may be a liquid crystal lens or a liquid lens. Dynamic adjustment of the surface profile of the lenslet and / or at least one additional optical component may provide additional directional control of the light projected from the waveguide element.

[0054] In the example shown, the holographic object volume 255 of the LF display has boundaries formed by rays 256 and 257, but may be formed by other rays. The holographic object volume 255 is a continuous volume that extends both in front of the energy waveguide layer 240 (i.e., toward the viewer 280) and behind it (i.e., away from the viewer 280). In the example shown, the rays 256 and 257 that are perceptible to the user are projected from opposite edges of the LF display module 210 at the maximum angle relative to the normal of the display surface 277, but the rays may be other projected rays. The rays define the field of view of the display and therefore define the boundaries of the holographic viewing volume 285. In some cases, the rays define a holographic viewing volume in which the entire display can be viewed without vignetting (e.g., an ideal viewing volume). As the field of view of the display increases, the convergence point of the rays 256 and 257 will be closer to the display. Therefore, a display with a larger field of view allows the viewer 280 to see the entire display at a closer viewing distance. In addition, rays 256 and 257 may form an ideal holographic object volume. A holographic object presented in an ideal holographic object volume may be seen anywhere in viewing volume 285.

[0055] In some examples, the holographic object may be presented to only a portion of the viewing volume 285. In other words, the holographic object volume may be divided into any number of viewing sub-volumes (e.g., viewing sub-volumes 290). In addition, the holographic object may be projected outside the holographic object volume 255. For example, the holographic object 251 is presented outside the holographic object volume 255. Because the holographic object 251 is presented outside the holographic object volume 255, it cannot be viewed from every position in the viewing volume 285. For example, the holographic object 251 may be visible from a position in the viewing sub-volume 290, but not from the position of the viewer 280.

[0056] For example, go to Figure 2B To illustrate viewing the holographic content from different viewing sub-volumes. Figure 2B A cross section 200 of a portion of a LF display module is shown in accordance with one or more embodiments. Figure 2B The cross section and Figure 2A The cross-sections are the same. However, Figure 2BA set of different light rays projected from the LF display module 210 is shown. Rays 256 and ray 257 still form the holographic object volume 255 and the viewing volume 285. However, as depicted, the rays projected from the top of the LF display module 210 and the rays projected from the bottom of the LF display module 210 overlap to form various viewing sub-volumes (e.g., viewing sub-volumes 290A, 290B, 290C, and 290D) within the viewing volume 285. Viewers in the first viewing sub-volume (e.g., 290A) may be able to perceive the holographic content presented in the holographic object volume 255, while viewers in other viewing sub-volumes (e.g., 290B, 290C, and 290D) may not be able to perceive it.

[0057] More simply, Figure 2A As shown, holographic object volume 255 is a volume in which holographic objects can be presented by the LF display system, so that the holographic objects can be perceived by a viewer (e.g., viewer 280) in viewing volume 285. In this way, viewing volume 285 is an instance of an ideal viewing volume, and holographic object volume 255 is an instance of an ideal object volume. However, in various configurations, viewers can perceive holographic objects presented by the LF display system 200 in other instance holographic object volumes, so that viewers in other instance viewing volumes can perceive holographic content. More generally, when viewing holographic content projected from an LF display module, "sight guides" will be applied. Sight guides assert that the line formed by the viewer's eye position and the holographic object being viewed must intersect the LF display surface.

[0058] Because the holographic content is presented according to the 4D light field function, each eye of the viewer 280 sees a different perspective of the holographic object 250 when viewing the holographic content presented by the LF display module 210. In addition, as the viewer 280 moves within the viewing volume 285, he / she will also see different perspectives of the holographic object 250, as will other viewers within the viewing volume 285. As one of ordinary skill in the art will appreciate, 4D light field functions are well known in the art and will not be described in further detail herein.

[0059] As described in more detail herein, in some embodiments, the LF display may project more than one type of energy. For example, the LF display may project two types of energy, such as mechanical energy and electromagnetic energy. In this configuration, the energy relay layer 230 includes two separate energy repeaters that are interwoven at the energy surface 275, but separated so that the energy is relayed to two different energy device layers 220. Here, one repeater may be configured to transmit electromagnetic energy, while the other repeater may be configured to transmit mechanical energy. In some embodiments, mechanical energy may be projected from a location between electromagnetic waveguide elements on the energy waveguide layer 240, thereby helping to form a structure that inhibits light from being transmitted from one electromagnetic waveguide element to another. In some embodiments, the energy waveguide layer 240 may also include a waveguide element that transmits focused ultrasound along a specific propagation path according to display instructions from a controller.

[0060] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy surface 275 is an emitting surface formed using one or more adjacent electronic displays within the energy device layer 220. And in some embodiments, the spacing between the edges of the adjacent electronic displays is less than the minimum perceptible outline defined by the visual acuity of a human eye with 20 / 40 vision, so that the energy surface is effectively seamless from the perspective of the viewer 280 within the viewing volume 285.

[0061] LF Display Module

[0062] Figure 3A 3 is a perspective view of an LF display module 300A according to one or more embodiments. The LF display module 300A may be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 300A may be some other LF display module. In the illustrated embodiment, the LF display module 300A includes an energy device layer 310 and an energy relay layer 320 and an energy waveguide layer 330. The LF display module 300A is configured to present holographic content from a display surface 365, as described herein. For convenience, the display surface 365 is shown in dashed outline on the frame 390 of the LF display module 300A, but is more accurately the surface directly in front of the waveguide element defined by the inner edge of the frame 390. The display surface 365 includes a plurality of projection locations from which energy can be projected. Some embodiments of the LF display module 300A have components different from those described herein. For example, in some embodiments, the LF display module 300A does not include an energy relay layer 320. Similarly, functionality may be distributed among components in ways other than those described here.

[0063] Energy device layer 310 is an embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three are visible in the figure). Energy devices 340 can all be of the same type (e.g., all electronic displays) or can include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0064] The energy relay layer 320 is an embodiment of the energy relay layer 230. The energy relay layer 320 includes four energy relay elements 350 (three of which are visible in the figure). The energy relay devices 350 may all relay the same type of energy (e.g., light) or may relay one or more different types (e.g., light and sound). Each of the relay elements 350 includes a first surface and a second surface, and the second surface of the energy relay element 350 is arranged to form a single seamless energy surface 360. In the illustrated embodiment, each of the energy relay elements 350 is conical so that the first surface has a smaller surface area than the second surface, which allows the mechanical envelope of the energy device 340 to be accommodated on the small end of the cone. Since the entire area can project energy, this also makes the seamless energy surface borderless. This means that this seamless energy surface can be tiled by placing multiple instances of 300 together without dead space or borders, so that the entire combined surface is seamless. In other embodiments, the surface area of ​​the first surface and the second surface is the same.

[0065] The energy waveguide layer 330 is an embodiment of the energy waveguide layer 240. The energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIG. 2 , the energy waveguide layer 330 is configured to guide energy from the seamless energy surface 360 ​​along a specific propagation path according to a 4D light field function to form a holographic object. Note that in the illustrated embodiment, the energy waveguide layer 330 is defined by a frame 390. In other embodiments, the frame 390 is absent and / or the thickness of the frame 390 is reduced. Removal or reduction of the thickness of the frame 390 may facilitate tiling the LF display module 300A with another LF display module 300.

[0066] Note that in the illustrated embodiment, the seamless energy surface 360 ​​and the energy waveguide layer 330 are flat. In alternative embodiments not depicted, the seamless energy surface 360 ​​and the energy waveguide layer 330 may be curved in one or more dimensions.

[0067] The LF display module 300A may be configured with additional energy sources residing on the surface of the seamless energy surface and allowing projection of energy fields in addition to the light field. In one embodiment, the acoustic energy field may be projected from an electrostatic speaker (not shown) mounted at any number of locations on the seamless energy surface 360. In addition, the electrostatic speaker of the LF display module 300A is positioned within the light field display module 300A so that the dual energy surface simultaneously projects the acoustic field and the holographic content. For example, the electrostatic speaker may be formed with one or more diaphragm elements that transmit electromagnetic energy of some wavelengths and are driven by a conductive element. The electrostatic speaker may be mounted on the seamless energy surface 360 ​​so that the diaphragm element covers some of the waveguide elements. The conductive electrodes of the speaker may be positioned at the same location as a structure designed to suppress light transmission between electromagnetic waveguides, and / or positioned at a location between electromagnetic waveguide elements (e.g., frame 390). In various configurations, the speaker may project audible sounds and / or many sources of focused ultrasonic energy that produce a tactile surface.

[0068] In some configurations, the energy device 340 can sense energy. For example, the energy device can be a microphone, a light sensor, an acoustic transducer, etc. Therefore, the energy relay device can also relay energy from the seamless energy surface 360 ​​to the energy device layer 310. That is, when the energy device and the energy relay device 340 are configured to simultaneously transmit and sense energy (e.g., transmit a light field and sense sound), the seamless energy surface 360 ​​of the LF display module forms a bidirectional energy surface.

[0069] More generally, the energy device 340 of the LF display module 340 may be an energy source or an energy sensor. The LF display module 300A may include various types of energy devices that act as energy sources and / or energy sensors to facilitate projection of high-quality holographic content to the user. Other sources and / or sensors may include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, etc. A plurality of other sensors or sources are possible. Furthermore, the LF display modules may be tiled so that the LF display modules may form an assembly that projects and senses multiple types of energy from a large aggregate seamless energy surface.

[0070] In various embodiments of the LF display module 300A, the seamless energy surface 360 ​​may have various surface portions, each of which is configured to project and / or emit a specific type of energy. For example, when the seamless energy surface is a dual energy surface, the seamless energy surface 360 ​​includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy may be located on the seamless energy surface 360 ​​between waveguide elements and / or in the same location as a structure designed to inhibit light transmission between waveguide elements. In the example where the seamless energy surface is a dual energy surface, the energy relay layer 320 may include two types of energy relay devices interwoven at the seamless energy surface 360. In various embodiments, the seamless energy surface 360 ​​may be configured so that the portion of the surface below a particular waveguide element 370 is all energy sources, all energy sensors, or a mixture of energy sources and energy sensors.

[0071] Figure 3B 300B is a cross-sectional view of an LF display module 300B including an interlaced energy relay device according to one or more embodiments. The energy relay device 350A transmits energy between the energy relay first surface 345A connected to the energy device 340A and the seamless energy surface 360. The energy relay 350B transmits energy between the energy relay first surface 345B connected to the energy device 340B and the seamless energy surface 360. The two relay devices are interlaced at the interlaced energy relay device 352 connected to the seamless energy surface 360. In this configuration, the surface 360 ​​contains interlaced energy locations of both energy devices 340A and 340B, both of which can be energy sources or energy sensors. Therefore, the LF display module 300B can be configured as a dual energy projection device for projecting more than one type of energy, or as a bidirectional energy device for simultaneously projecting one type of energy and sensing another type of energy. The LF display module 300B can be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 302 can be some other LF display module.

[0072] LF display module 300B includes Figure 3A For example, in the illustrated embodiment, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330, which include at least the same components as those of the LF display module 300A. Figure 3A In addition, the LF display module 300B can present and / or receive energy from the display surface 365. It is worth noting that Figure 3AComponents of the LF display module 300B may be connected and / or oriented alternatively compared to the components of the LF display module 300A in FIG. Some embodiments of the LF display module 300B have components that differ from those described herein. Similarly, functionality may be allocated between components in a manner that differs from that described herein. Figure 3B A design of a single LF display module 302 is shown that can be tiled to create a dual energy projection surface or a bi-directional energy surface with a larger area.

[0073] In one embodiment, the LF display module 300B is a LF display module of a bidirectional LF display system. The bidirectional LF display system can simultaneously project energy from the display surface 365 and sense energy. The seamless energy surface 360 ​​contains both energy projection locations and energy sensing locations that are closely interwoven on the seamless energy surface 360. Figure 3B In the example of Figure 3A For convenience, the energy relay layer of the LF display module 300B will be referred to herein as an "interwoven energy relay layer".

[0074] The interwoven energy relay layer 320 includes two branches: a first energy relay device 350A and a second energy relay device 350B. Each of the branches is shown as a light shaded area. Each of the branches can be made of a flexible relay material and formed with sufficient length to be used for energy devices of various sizes and shapes. In some areas of the interwoven energy relay layer, the two branches are tightly interwoven when approaching the seamless energy surface 360. In the example shown, the interwoven energy relay device 352 is shown as a dark shaded area.

[0075] When interwoven at seamless energy surface 360, the energy relay device is configured to relay energy to / from different energy devices. The energy devices are located at energy device layer 310. As shown, energy device 340A is connected to energy relay device 350A, and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device can be an energy source or an energy sensor.

[0076] The energy waveguide layer 330 includes a waveguide element 370 to guide the energy waves from the seamless energy surface 360 ​​along the projected path toward a series of convergence points. In this example, a holographic object 380 is formed at the series of convergence points. Notably, as shown, the energy convergence at the holographic object 380 occurs on the viewer side of the display surface 365. However, in other examples, the convergence of energy can be anywhere in the holographic object volume, extending both in front of the display surface 365 and behind the display surface 365. The waveguide element 370 can simultaneously guide the incoming energy to an energy device (e.g., an energy sensor), as described below.

[0077] In one example embodiment of the LF display module 300B, an emissive display is used as an energy source, and an imaging sensor is used as an energy sensor. In this way, the LF display module 300B can simultaneously project holographic content and detect light from a volume in front of the display surface 365. In this way, this embodiment of the LF display module 300B acts as both a LF display and a LF sensor.

[0078] In an embodiment, the LF display module 300B is configured to simultaneously project a light field in front of the display surface 365 and capture a light field from in front of the display surface 365. In this embodiment, the energy relay device 350A connects a first set of positions at the seamless energy surface 360 ​​that are positioned below the waveguide element 370 to the energy device 340A. In an example, the energy device 340A is an emissive display with an array of source pixels. The energy relay device 340B connects a second set of positions at the seamless energy surface 360 ​​that are positioned below the waveguide element 370 to the energy device 340B. In an example, the energy device 340B is an imaging sensor with an array of sensor pixels. The LF display module 302 can be configured so that the positions at the seamless energy surface 365 that are below a particular waveguide element 370 are all emissive display positions, all imaging sensor positions, or some combination of positions. In other embodiments, the bidirectional energy surface can project and receive various other forms of energy.

[0079] In another example embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, the energy device 340A is an emissive display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. Therefore, both light and sound can be projected from various locations at the seamless energy surface 360. In this configuration, the energy relay device 350A connects the energy device 340A to the seamless energy surface 360 ​​and relays electromagnetic energy. The energy relay device is configured to have characteristics (e.g., a varying refractive index) that enable the energy relay device to effectively transmit electromagnetic energy. The energy relay device 350B connects the energy device 340B to the seamless energy surface 360 ​​and relays mechanical energy. The energy relay device 350B is configured to have characteristics for efficient transmission of ultrasonic energy (e.g., a distribution of materials with different acoustic impedances). In some embodiments, mechanical energy can be projected from locations between waveguide elements 370 on the energy waveguide layer 330. The locations where the mechanical energy is projected can form a structure for inhibiting the transmission of light from one electromagnetic waveguide element to another electromagnetic waveguide element. In one example, an array of spatially separated locations where ultrasonic mechanical energy is projected can be configured to produce three-dimensional tactile shapes and surfaces in midair. The surface can coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delays and amplitude variations across the array can help produce tactile shapes.

[0080] In various embodiments, the bidirectional LF display module 302 may include multiple energy device layers, where each energy device layer includes a specific type of energy device. In these examples, the energy relay layer is configured to relay the appropriate type of energy between the seamless energy surface 360 ​​and the energy device layer 330.

[0081] Tiled LF display modules

[0082] Figure 4A 4 is a perspective view of a portion of an LF display system 400 tiled in two dimensions to form a single-sided seamless surface environment according to one or more embodiments. The LF display system 400 includes a plurality of LF display modules tiled to form an array 410. More specifically, each of the small squares in the array 410 represents a tiled LF display module 412. The array 410 may cover some or all of a surface, for example.

[0083] The array 410 may project one or more holographic objects. For example, in the embodiment shown, the array 410 projects holographic object 420 and holographic object 430. The tiling of the LF display modules allows for a larger viewing volume, and allows objects to be projected at a greater distance from the array 410. For example, in the embodiment shown, the viewing volume is approximately the entire area in front of and behind the array 410, rather than a partial volume in front of (and behind) the LF display module 412.

[0084] In some embodiments, the LF display system 400 presents the holographic object 420 to the viewer 430 and the viewer 434. The viewer 430 and the viewer 434 receive different perspectives of the holographic object 420. For example, the viewer 430 is presented with a direct view of the holographic object 420, while the viewer 434 is presented with a more oblique view of the holographic object 430. As the viewer 430 and / or the viewer 434 move, different perspectives of the holographic object 420 are presented to them. This allows the viewer to visually interact with the holographic object by moving relative to the holographic object. For example, as the viewer 430 walks around the holographic object 420, the viewer 430 sees different sides of the holographic object 420 as long as the holographic object 420 remains in the holographic object volume of the array 410. Therefore, the viewer 430 and the viewer 434 can see the holographic object 420 in the real world space at the same time, as if the holographic object really existed. In addition, the viewer 440 and the viewer 450 of the holographic object 420 do not need to wear an external device in order to see the holographic object 420, because the holographic object 420 is visible to the viewer in much the same way as a physical object would be visible. In addition, here, the holographic object 422 is shown behind the array because the viewing volume of the array extends behind the surface of the array. In this way, the holographic object 422 can be presented to the viewer 430 and / or the viewer 434.

[0085] In some embodiments, LF display system 400 may include a tracking system that tracks the position of viewer 430 and viewer 434. In some embodiments, the tracked position is the position of the viewer. In other embodiments, the tracked position is the position of the viewer's eyes. Eye position tracking is different from gaze tracking, which tracks where the eyes are looking (e.g., using orientation to determine gaze position). The eyes of viewer 430 and the eyes of viewer 434 are located in different positions.

[0086] In various configurations, the LF display system 400 may include one or more tracking systems. Figure 4A In the embodiment of FIG. 4 , the LF display system includes a tracking system 440 external to the array 410. Here, the tracking system may be a camera system coupled to the array 410. Figure 4AThe external tracking system is described in more detail. In other example embodiments, the tracking system may be incorporated into the array 410 as described herein. For example, an energy device (e.g., energy device 340) of the LF display module 412 included in the array 410 may be configured to capture an image of a viewer in front of the array 440. In any case, one or more tracking systems of the LF display system 400 determine tracking information about a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by the array 410.

[0087] The tracking information describes the position of the viewer or a portion of the viewer (e.g., one or both eyes of the viewer, or a limb of the viewer) in space (e.g., relative to the tracking system). The tracking system may use any number of depth determination techniques to determine the tracking information. The depth determination techniques may include, for example, structured light, time of flight, stereoscopic imaging, some other depth determination technique, or some combination thereof. The tracking system may include various systems configured to determine the tracking information. For example, the tracking system may include one or more infrared sources (e.g., structured light sources), one or more imaging sensors that can capture infrared images (e.g., red-blue-green-infrared cameras), and a processor that executes a tracking algorithm. The tracking system may use depth estimation techniques to determine the position of the viewer. In some embodiments, the LF display system 400 generates a holographic object based on the tracked position of the viewer 430 and / or the viewer 434. For example, the LF display system 400 may generate a holographic object in response to the viewer entering a threshold distance and / or a specific position of the array 410.

[0088] The LF display system 400 may present one or more holographic objects customized for each viewer based in part on the tracking information. For example, the holographic object 420 may be presented to the viewer 430 instead of the holographic object 422. And the holographic object 422 may be presented to the viewer 434 instead of the holographic object 420. For example, the LF display system 400 tracks the position of each of the viewer 430 and the viewer 434. The LF display system 400 determines the viewing angle of the holographic object that should be visible to the viewer based on the viewer's position relative to where the holographic object is to be presented. The LF display system 400 selectively projects light from specific pixels corresponding to the determined viewing angle. Therefore, the viewer 434 and the viewer 430 may have potentially completely different experiences at the same time. In other words, the LF display system 400 may present the holographic object to the viewing sub-volume of the viewing volume (i.e., similar to Figure 2B4. In the embodiment of the present invention, the LF display system 400 can present holographic content to the viewing sub-volumes 290A, 290B, 290C, and 290D shown in FIG. For example, as shown, because the LF display system 400 can track the position of the viewer 430, the LF display system 400 can present spatial content (e.g., holographic object 420) to the viewing sub-volumes surrounding the viewer 430, and present safari content (e.g., holographic object 422) to the viewing sub-volumes surrounding the viewer 434. In contrast, conventional systems would have to use separate headphones to provide a similar experience.

[0089] In some embodiments, the LF display system 400 may include one or more sensory feedback systems. The sensory feedback system provides other sensory stimulation (e.g., touch, audio, pressure, force, or smell / smell) that enhances the holographic objects 420 and 422. For example, in Figure 4A In the illustrated embodiment of , the LF display system 400 includes a sensory feedback system 442 external to the array 410. In one example, the sensory feedback system 442 may be an electrostatic speaker coupled to the array 410. Figure 4A The external sensory feedback system is described in more detail. In other example embodiments, the sensory feedback system may be incorporated into the array 410 as described herein. For example, the energy device (e.g., Figure 3B Energy device 340A in array 410 may be configured to project ultrasound energy to a viewer in front of the array and / or receive imaging information from a viewer in front of the array. In any case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content (e.g., holographic object 420 and / or holographic object 422) presented by array 410.

[0090] The LF display system 400 may include a sensory feedback system including one or more acoustic projection devices outside the array. Alternatively or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410, as described herein. The acoustic projection device may consist of an array of ultrasound sources configured to project a body tactile surface. In some embodiments, the body surface may be projected to coincide with the holographic object. In some embodiments, the body tactile surface may be projected within a threshold distance of the holographic object. In some embodiments, for one or more surfaces of the holographic object, if a portion of the viewer is within a threshold distance of one or more surfaces, the tactile surface may coincide with the holographic object (e.g., at the surface of the holographic object 420). The body tactile sensation allows the user to touch and feel the surface of the holographic object. Multiple acoustic projection devices may also project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Therefore, ultrasonic pressure waves and / or audible pressure waves may play a role in supplementing the holographic object.

[0091] In various embodiments, the LF display system 400 may provide other sensory stimulation based in part on the tracked position of the viewer. For example, Figure 4A The holographic object 422 shown in the figure is a lion, and the LF display system 400 can make the holographic object 422 roar both visually (i.e., the holographic object 430 seems to be roaring) and audibly (i.e., one or more acoustic projection devices project pressure waves), so that the viewer 430 perceives it as the roar of a lion emitted by the holographic object 422.

[0092] It should be noted that in the configuration shown, the holographic viewing volume may be limited in a manner similar to the viewing volume 285 of the LF display system 200 in FIG. 2 . This may limit the perceived immersion that a viewer would experience with a single device. One way to address this issue is to use multiple LF display modules tiled along multiple sides, as described below with respect to FIG. Figure 4B described.

[0093] Figure 4B 4 is a perspective view of a portion of a LF display system 402 in a multi-faceted seamless surface environment according to one or more embodiments. The LF display system 402 is substantially similar to the LF display system 400, except that a plurality of LF display modules are tiled to create the multi-faceted seamless surface environment. More specifically, the LF display modules are tiled to form an array as a six-sided aggregate seamless surface environment. In other embodiments, the plurality of LF display modules may cover some but not all of the surfaces, or some combination thereof. In other embodiments, the plurality of LF display modules are tiled to form some other aggregate seamless surface. For example, the surface may be curved such that a cylindrical aggregate energy environment is formed.

[0094] The LF display system 402 may project one or more holographic objects. For example, in the illustrated embodiment, the LF display system 402 projects a holographic object 420 into an area surrounded by a six-sided polymeric seamless surface environment. In this example, the viewing volume of the LF display system is also contained within the six-sided polymeric seamless surface environment. It should be noted that in the illustrated configuration, the viewer 432 may be positioned between the holographic object 420 and the LF display module 414, which projects energy (e.g., light and / or pressure waves) used to form the holographic object 420. Therefore, the positioning of the viewer 434 may prevent the viewer 430 from perceiving the holographic object 420 formed by the energy from the LF display module 414. However, in the illustrated configuration, there is at least one other LF display module, such as the LF display module 416, which is not obstructed (e.g., by the viewer 434) and may project energy to form the holographic object 420. In this way, being occluded in space by a viewer may cause some portions of the holographic projection to disappear, but this effect is much less than if only one side of the volume were filled with a holographic display panel. The holographic object 422 is shown as "outside" of the shell of the six-sided polymeric seamless surface environment because the holographic object volume extends behind the polymeric surface. Thus, the viewer 430 and / or the viewer 434 may perceive the holographic object 422 as being "outside" of the six-sided environment.

[0095] As mentioned above Figure 4A As described, in some embodiments, the LF display system 402 actively tracks the position of the viewer and can dynamically instruct different LF display modules to present holographic content based on the tracked position. Thus, the multi-faceted configuration can provide a more robust environment (e.g., relative to Figure 4A ) to provide a holographic object where an unconstrained viewer can move freely throughout the area surrounded by a multi-faceted seamless surface environment.

[0096] It is worth noting that various LF display systems may have different configurations. Furthermore, each configuration may have a specific orientation of surfaces that aggregate to form a seamless display surface ("aggregate surface"). That is, the LF display modules of a LF display system may be tiled to form various aggregate surfaces. For example, in Figure 4B In , LF display system 402 includes LF display modules that are tiled to form a six-sided aggregate surface. In some other examples, the aggregate surface may appear on only a portion of the surface, rather than the entire surface. Some examples are described herein.

[0097] In some configurations, the converging surface of the LF display system may include a converging surface configured to project energy toward a local viewing volume. Projecting energy to a local viewing volume allows for a higher quality viewing experience by, for example, increasing the density of projected energy in a particular viewing volume, increasing the FOV of a viewer in the viewing volume, and bringing the viewing volume closer to the display surface.

[0098] For example, Figure 4C A top view of an LF display system 450A having a converging surface in a "wing-like" configuration is shown. The LF display system 450A includes LF display modules tiled to form a converging surface 460. The LF display system 450A has a converging surface 460 that includes three portions: (i) a first central portion 462, (ii) a second portion 464 connected to the first portion 462 and positioned at an angle to the central portion to project energy in front of the central portion (i.e., a first side surface), and (iii) a third portion 466 connected to the first portion 462 and positioned at an angle to the central portion to project energy in front of the central portion (i.e., a second side surface). The mobile device may be designed to have a configuration similar to Figure 4C The configuration shown is such that the first central portion 462 is the main display screen designed to face the viewer, and the second portion 464 and the third portion 466 are angled toward the viewer to partially surround the viewer. When the mobile device is held such that the first central portion is located in a vertical plane having a horizontal axis and a vertical axis, the second and third portions are angled along the horizontal axis toward the front of the central portion.

[0099] In this example, a viewing volume 468A of LF display system 450A is in front of a device that is partially surrounded by three portions of converging surface 460. A converging surface that at least partially surrounds a viewer ("surrounding surface") increases the viewer's immersive experience.

[0100] For illustration, consider, for example, a polymeric surface having only a central surface. Figure 2A , as described above, rays projected from either end of the display surface create an ideal holographic volume and an ideal viewing volume. Now consider, for example, if the central surface includes two side surfaces that are angled toward the viewer. In this case, rays 256 and 257 will be projected at a larger angle from the normal to the central surface. Therefore, the field of view of the viewing volume will increase. Similarly, the holographic viewing volume will be closer to the display surface. In addition, since the two second and third parts are tilted closer to the viewing volume, the holographic object projected at a fixed distance from the display surface is closer to the viewing volume.

[0101] For simplicity, a display surface with only a center surface has a planar field of view, a planar threshold spacing between the (center) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer increases the field of view relative to the planar field of view, reduces the spacing between the display surface and the viewing volume relative to the planar spacing, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further increases the field of view, reduces the spacing, and increases the proximity. In other words, the angled placement of the side surfaces increases the immersive experience of the viewer.

[0102] In addition, as mentioned below Figure 6 As described, deflection optics can be used to optimize the size and position of the viewing volume with respect to LF display parameters (eg, size and FOV).

[0103] Return to Figure 4D , in similar instances, Figure 4D A side view of an LF display system 450B having a converging surface in a "tilted" configuration is shown. The LF display system 450B includes LF display modules that are tiled to form a converging surface 460. The converging surface 460 includes three portions: (i) a first central portion 462 (i.e., the central surface) (ii) a second portion 464 that is connected to the first portion 462 and is positioned at an angle to the central portion to project energy in front of the central portion (i.e., the first side surface), and (iii) a third portion 464 that is connected to the first portion 462 and is positioned at an angle to the central portion to project energy in front of the central portion (i.e., the second side surface). The mobile device can be designed to have a configuration similar to Figure 4D The configuration shown is such that the first central portion 462 is the main display screen designed to face the viewer, and the second portion 464 and the third portion 466 are angled toward the viewer to partially surround the viewer. When the mobile device is held such that the first central portion is in a vertical plane having a horizontal axis and a vertical axis, the second and third portions are angled along the vertical axis toward the front of the central portion.

[0104] In this example, the viewing volume 468B of the LF display system 450B is in front of the central portion and is partially surrounded by three portions of the converging surface 460. Figure 4CSimilar to the configuration shown, the two side portions (e.g., second portion 464 and third portion 466) are angled to surround the viewer and form a surrounding surface. The surrounding surface increases the viewing FOV from the perspective of any viewer in the holographic viewing volume 468B. In addition, the surrounding surface allows the viewing volume 468B to be closer to the surface of the display, making the projected objects appear closer. In other words, the angled placement of the side surfaces increases the field of view, reduces the separation, and increases the proximity of the converging surfaces, thereby increasing the immersive experience of the viewer. In addition, as will be discussed below, deflection optics can be used to optimize the size and position of the viewing volume 468B.

[0105] The tilted configuration of the side portions of the converging surface 460 enables the holographic content to be presented closer to the viewing volume 468B than if the third portion 466 were not tilted. For example, the lower extremities (e.g., legs) of a character presented from a LF display system in a tilted configuration may appear closer and more realistic than if a LF display system with a flat front display were used.

[0106] Additionally, the configuration of the LF display system and the environment in which it is located may inform the shape and position of the viewing volume and viewing sub-volumes.

[0107] For example, Figure 4E A top view of a LF display system 450C with a converging surface 460 is shown. The LF display system 450C projects various rays from the converging surface 460. Rays projected from the left side of the converging surface 460 have a horizontal angle range 481, rays projected from the right side of the converging surface have a horizontal angle range 482, and rays projected from the center of the converging surface 460 have a horizontal angle range 483. Between these points, the projected rays may take on intermediate values ​​of the angle range. Having a gradient deflection angle in the projected rays across the display surface in this manner creates a viewing volume 468C.

[0108] LF display system control

[0109] Figure 5is a block diagram of an LF display system 500 according to one or more embodiments. The LF display system 500 includes an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display module 512 may include a source / sensor system 514, which includes one or more integrated energy sources and / or one or more energy sensors that project and / or sense other types of energy. The controller 520 includes a data storage device 522, a network interface 524, an LF processing engine 530, a command library 532, and a security module 534. The controller 520 may also include a tracking module 526 and a user profile module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. In Figure 1 The LF display system described in the context of , 2, 3, and 4 is an embodiment of the LF display system 500. In other embodiments, the LF display system 500 includes more or fewer modules than those described herein. Similarly, functionality may be allocated between modules and / or different entities in a manner different from that described herein. Applications of the LF display system 500 are also described below with respect to Figure 6-7 Discuss in detail.

[0110] The LF display assembly 510 provides holographic content in a holographic object volume that may be visible to a viewer positioned within the viewing volume. The LF display assembly 510 may provide holographic content by executing display instructions received from the controller 520. The holographic content may include one or more holographic objects projected in front of a gathering surface of the LF display assembly 510, behind a gathering surface of the LF display assembly 510, or a combination thereof. Generating display instructions with the controller 520 is described in more detail below.

[0111] The LF display assembly 510 uses one or more LF display modules (e.g., any one of the LF display module 110, the LF display system 200, and the LF display module 300) included in the LF display assembly 510 to provide holographic content. For convenience, one or more LF display modules may be described herein as LF display modules 512. The LF display modules 512 may be tiled to form the LF display assembly 510. The LF display modules 512 may be structured into various seamless surface environments (e.g., single-sided, multi-sided, curved surfaces, etc.). That is, the tiled LF display modules form a polymeric surface. As previously described, the LF display module 512 includes an energy device layer (e.g., the energy device layer 220) and an energy waveguide layer (e.g., the energy waveguide layer 240) that presents holographic content. The LF display module 512 may also include an energy relay layer (e.g., the energy relay layer 230) that transmits energy between the energy device layer and the energy waveguide layer when presenting holographic content.

[0112] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing as previously described. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and the integrated energy sensing system of the LF display module 512 may be collectively described herein as a source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 so that the source / sensor system 514 shares the same seamless energy surface with the LF display module 512. In other words, the aggregate surface of the LF display assembly 510 includes the functions of both the LF display module 512 and the source / sensor module 514. That is, the LF assembly 510 including the LF display module 512 with the source / sensor system 514 can project energy and / or sense energy while simultaneously projecting a light field. For example, the LF display assembly 510 may include a LF display module 512 and a source / sensor system 514 configured as a dual energy surface or a bidirectional energy surface as previously described.

[0113] In some embodiments, the LF display system 500 uses a sensory feedback system 570 to enhance the generated holographic content with other sensory content (e.g., coordinated touch, audio, pressure, force, or smell). The sensory feedback system 570 can enhance the projection of the holographic content by executing display instructions received from the controller 520. Typically, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., sensory feedback system 442) outside the LF display assembly 510. Some example sensory feedback devices may include coordinated acoustic projection and reception devices, aroma projection devices, temperature adjustment devices, force actuation devices, pressure sensors, accelerometers, gyroscopes, transducers, etc. In some cases, the sensory feedback system 570 may have similar functions to the light field display assembly 510, and vice versa. For example, both the sensory feedback system 570 and the light field display assembly 510 can be configured to generate an acoustic field. As another example, the sensory feedback system 570 can be configured to generate a tactile surface without a light field display 510 assembly.

[0114] For illustration, in an example embodiment of the light field display system 500, the sensory feedback system 570 may include an acoustic projection device. The acoustic projection device is configured to generate one or more pressure waves that supplement the holographic content when executing the display instructions received from the controller 520. The generated pressure waves may be, for example, audible (for sound), ultrasonic (for touch), or a combination thereof. Similarly, the sensory feedback system 570 may include an aroma projection device. The aroma projection device may be configured to provide a scent to some or all of the target area when executing the display instructions received from the controller. In addition, the sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature in some or all of the target area when executing the display instructions received from the controller 520.

[0115] In some embodiments, the sensory feedback system 570 may include a force-actuated device. The force-actuated device may be used to move some physical objects in the target area. In one example, the LF display system 500 may be integrated with a multi-sensory gaming application, where the force-actuated device may move pieces associated with the game.

[0116] In some embodiments, the sensory feedback system 570 is configured to receive input from a viewer of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the viewer. The sensory feedback device may include, for example, an acoustic receiving device (e.g., a microphone), a pressure sensor, a motion detector, an ultrasonic transducer, other transducers, a force sensor, a temperature sensor, a touch sensor, a proximity sensor, and the like. The sensor feedback system may also include buttons, dials, knobs, keyboards, fingerprint sensors, joysticks, other input hardware, or any combination thereof. The sensory feedback system 570 may include a sensory feedback device configured to provide sensory feedback when presenting holographic content. The sensory feedback system may transmit the detected input to the controller 520 to coordinate the generation of holographic content and / or sensory feedback. In some embodiments, the sensory feedback system 570 recognizes the input received from the viewer as a received command from the viewer to the mobile device.

[0117] To illustrate, in an example embodiment of a light field display assembly, the sensory feedback system 570 includes a microphone. The microphone is configured to record audio generated by a mobile device user. The sensory feedback system 570 provides the recorded audio as user input to the controller 520. The controller 520 can use the user input to generate holographic content. Similarly, the sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure the force applied to the pressure sensor by the user. The sensory feedback system 570 provides the measured force as user input to the controller 520.

[0118] In some embodiments, the sensory feedback assembly 570 is also configured to redirect user movement. In one embodiment, the sensory feedback assembly 570 includes an acoustic projection device that can generate tactile pressure waves to discourage movement, such as when the tracking system 580 tracks the presence of a user's hand blocking the mobile device camera while the camera sensor is actively recording image data. In another embodiment, the sensory feedback assembly 570 can use an acoustic emission device to audibly signal further action to the user (e.g., "The camera is recording - don't block the camera").

[0119] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to obtain information about users of the devices, including the location, movement, gestures, expressions, line of sight, and / or characteristics (e.g., gender and age) of users in a target area. Typically, the tracking devices are external to the LF display assembly 510. Some example tracking devices include a camera assembly ("camera"), a depth sensor, structured light, a LIDAR system, a card scanning system, or any other tracking device that can track a user in a target area.

[0120] The tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. However, in some cases, when presenting holographic content, the target area is illuminated by natural light and / or ambient light from the LF display assembly 510. The energy source projects light when executing instructions received from the controller 520. The light may be, for example, a structured light pattern, a light pulse (e.g., an IR flash), or some combination thereof. The tracking system may project light in the visible band (approximately 380nm to 750nm), the infrared (IR) band (approximately 750nm to 1700nm), the ultraviolet band (10nm to 380nm), some other part of the electromagnetic spectrum, or some combination thereof. The source may include, for example, a light emitting diode (LED), a micro-LED, a laser diode, a TOF depth sensor, a tunable laser, etc.

[0121] Tracking system 580 may adjust one or more emission parameters when executing instructions received from controller 520. Emission parameters are parameters that affect how light is projected from the source of tracking system 580. Emission parameters may include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, some other parameter that affects how light is projected from the source assembly, or some combination thereof. In one embodiment, the source projects light pulses in a time-of-flight operation.

[0122] The camera of the tracking system 580 captures an image of light (e.g., a structured light pattern) reflected from the target area. The camera captures the image when executing the tracking instructions received from the controller 520. As previously described, the light may be projected by the source of the tracking system 580. The camera may include one or more cameras. That is, the camera may be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by the tracking system 580, or some combination thereof. In an embodiment, the tracking system 580 may contain a light field camera external to the LF display assembly 510. In other embodiments, the camera is included as part of the LF display module included in the LF display assembly 510. For example, as previously described, if the energy relay element of the light field module 512 is a bidirectional energy layer that interweaves both the emission display and the imaging sensor at the energy device layer 220, the LF display assembly 510 may be configured to simultaneously project a light field and record imaging information from the viewing area in front of the display. In one embodiment, the image captured from the bidirectional energy surface forms a light field camera. The camera provides the captured image to the controller 520.

[0123] When executing tracking instructions received from controller 520, the camera of tracking system 580 may adjust one or more imaging parameters. Imaging parameters are parameters that affect how the camera assembly captures an image. Imaging parameters may include, for example, frame rate, aperture, gain, exposure length, frame timing, some other parameter that affects how the camera assembly captures an image, or some combination thereof.

[0124] In some embodiments, the tracking system 580 is also configured to receive input from a viewer of the LF display system 500. The tracking system 580 can track the body movements of the viewer and send information about the specific tracked movements to the controller 520. In some embodiments, the tracking system 580 is used with a holographic user interface presented to the user.

[0125] Controller 520 controls LF display assembly 510 and any other components of LF display system 500. Controller 520 includes data storage 522, network interface 524, tracking module 526, user profile module 528, light field processing engine 530, and command library 532. In other embodiments, controller 520 includes more or fewer modules than described herein. Similarly, functionality may be allocated between modules and / or different entities in a manner different from that described herein. For example, tracking module 526 may be part of LF display assembly 510 or tracking system 580.

[0126] The data storage device 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by a camera, one or more user profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 (including the resolution and orientation of the LF module 512), the desired viewing volume geometry, content for graphic creation including 3D models, scenes and environments, materials and textures, security related information including authentication factors (e.g., passwords), smart cards, biometric data that can be used by the security module 534, other information that can be used by the LF display system 500, or some combination thereof. The data storage device 522 is a memory, such as a read-only memory (ROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), or some combination thereof.

[0127] The network interface 524 allows the light field display system to communicate with other systems or environments via a network. In one example, the LF display system 500 receives holographic content from a remote server (e.g., an online holographic content storage device) via the network interface 524. In another example, the LF display system 500 transmits the holographic content to a remote data storage device using the network interface 524.

[0128] The tracking module 526 tracks viewers viewing content presented by the LF display system 500. To this end, the tracking module 526 generates tracking instructions that control the operation of one or more sources and / or one or more cameras of the tracking system 580 and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking input to the tracking module 526.

[0129] The tracking module 526 may determine the position of one or more viewers within the target area of ​​the mobile device. The determined position may be relative to some reference point (e.g., a display surface). In other embodiments, the determined position may be within a virtual model of the target area. The tracked position may be, for example, the tracked position of the viewer and / or the tracked position of a portion of the viewer (e.g., eye position, hand position, etc.). The tracking module 526 uses one or more captured images from the camera of the tracking system 580 to determine the position. The cameras of the tracking system 580 may be distributed around the LF display system 500 and may capture stereo images, thereby allowing the tracking module 526 to passively track the user. In other embodiments, the tracking module 526 actively tracks the user. That is, the tracking system 580 illuminates a portion of the target area, images the target area, and the tracking module 526 uses time of flight and / or structured light depth determination technology to determine the position. The tracking module 526 uses the determined position to generate tracking information.

[0130] The tracking module 526 may also receive tracking information as input from a viewer of the LF display system 500, such as when using a holographic user interface on a mobile device (described further below). The tracking information may include body movements corresponding to various input options provided to the user by the LF display system 500. For example, the tracking module 526 may track user body movements and distribute any of the various movements as input to the LF processing engine 530. The tracking module 526 may provide the tracking information to the data storage device 522, the LF processing engine 530, the user profile module 528, the command storage device 532, any other component of the LF display system 500, or some combination thereof. The LF processing engine 530 may generate holographic content based in part on the provided tracking information.

[0131] To provide context for the tracking module 526, consider an example embodiment of the LF display system 500 used in a gaming application within a mobile device. When a user responds to a win by pumping their fist in the air to show their excitement, the tracking system 580 may record the movement of the user's hand and transmit the record to the tracking module 526. The tracking module 526 tracks the movement of the user's hand in the record and sends the input to the LF processing engine 530, as described below. As described below, the user profile module 528 determines that the information in the image indicates that the movement of the user's hand is associated with a positive response. Therefore, the LF processing engine 530 generates appropriate holographic content to celebrate the win. For example, the LF processing engine 530 may project confetti in a holographic object volume associated with the gaming application.

[0132] The LF display system 500 includes a user profile module 528 configured to identify a device user and form a profile for the device user. The user profile module 528 generates a profile of a user (or multiple users) viewing the holographic content displayed by the LF display system 500. The user profile module 528 generates a user profile based in part on user input and monitored user behavior, actions, and reactions. The user profile module 528 can access information obtained from the tracking system 580 (e.g., recorded images, videos, sounds, etc.) and process the information to determine various information. In various examples, the user profile module 528 can use machine vision or machine learning algorithms to determine user behavior, actions, and reactions. The monitored user behavior may include, for example, smiling, cheering, clapping, laughing, frightening, screaming, excitement level, backing off, other changes in gestures, or the movement of the user, etc.

[0133] More generally, a user profile may include any information received and / or determined about a user viewing holographic content from a LF display system. For example, each user profile may record the user's actions or responses to content displayed by the LF display system 500. Some example information that may be included in a user profile is provided below.

[0134] The user profile may be based on the user's behavior relative to content displayed by the LF display system 500. In some embodiments, software applications resident in the device may be functionally integrated with the LF display system 500, and the user profile is based on user behavior monitored while these software applications are executed in the device.

[0135] Thus, for example, the monitored behavior may include such things as how many times a user uses an application over a period of time such as a day or week, how often a user uses particular settings on an application (e.g., how often the "level 8" setting is used with a particular jigsaw puzzle application on a device), how a user responds to certain types of holographic content (e.g., in a clock alarm application - what is the user's preferred setting - a holographic image of a flashing button accompanied by a high-frequency whistle or just a flashing button), some other behavior related to use of a particular device, or some combination thereof.

[0136] In another example, a user may express a preference for a tactile interface presented in conjunction with holographic content, such as a preferred key click sensitivity in a holographic keyboard touch interface, which may be generated by the LF display system 100 and provided as an option to the mobile device user.

[0137] Other user characteristics stored as part of the user profile information may include, for example, the user's name, the user's age, race, gender, viewing location, work information, education, income, money spent on purchases, hobbies, location, viewing history, categories of online items viewed, purchase history, device and application setting preferences, location where resides, any other demographic information, or some combination thereof.

[0138] The user profile module 528 may also access profiles associated with a particular user (or users) from a third-party system or online system to establish and / or update a user profile. The user profile module 528 may be configured to update the user profile using information from the social media account of the user (or users). For example, user interactions with third-party providers linked to the user's social media or other online accounts may provide additional user profile information that may be accessed by the user profile module 528. Thus, the user profile information may store information such as favorite musicians, favorite movie characters, disliked music genres, etc., and take advantage of this information when accessing content from an online holographic content store.

[0139] In some embodiments, the data storage device 522 includes a user profile storage device that stores user profiles generated, updated and / or maintained by the user profile module 528. The user profile can be updated in the data storage device at any time by the user profile module 528. For example, in an embodiment, when a specific user views the holographic content provided by the LF display system 500, the user profile storage device receives and stores information about the specific user in its user profile. In this example, the user profile module 528 includes a facial recognition algorithm that can identify the user and positively identify it when it views the presented holographic content. For illustration, when the user enters the target area of ​​the LF display system 500, the tracking system 580 obtains an image of the user. The user profile module 528 inputs the captured image and uses the facial recognition algorithm to identify the user's face. The identified face is associated with the user profile in the profile storage device, and therefore, all the input information obtained about the user can be stored in its profile. The user profile module may also utilize a card recognition scanner, voice identifier, radio frequency identification (RFID) chip scanner, bar code scanner, etc. to positively identify the user.

[0140] Because the user profile module 528 can positively identify the user, the user profile module 528 can determine each user of the LF display system 500, and information about each user. This is particularly relevant when the mobile device can be shared by multiple users, such as a mobile tablet device shared by multiple students in a classroom environment or at home. The user profile module 528 can then store the time and date of each access in the user profile of each user. Similarly, the user profile module 528 can store input from the user received from any combination of the sensory feedback system 570, the tracking system 580 and / or the LF display assembly 510 whenever it occurs. The user profile system 528 can also receive other information about the user from other modules or components of the controller 520, which can then be stored with the user profile. Other components of the controller 520 can also subsequently access the stored user profile to determine the subsequent content provided to the user. The controller 520 can be configured to receive or generate holographic content based at least in part on the accessed data from the stored user profile. For example, the user interface generated for the teenager in the family might use an avatar of her favorite holographic comic book superhero to receive verbal commands, while the user interface generated for the grandfather might be an enlarged, flickering holographic interface with large, easy-to-see button renderings with coincident tactile stimuli for easy "pressing."

[0141] The LF processing engine 530 generates 4D coordinates in a rasterized format ("rasterized data") that, when executed by the LF display assembly 510, cause the LF display assembly 510 to present holographic content. The LF processing engine 530 may access the rasterized data from the data storage device 522. In addition, the LF processing engine 530 may construct the rasterized data from the vectorized data set. The vectorized data is described below. The LF processing engine 530 may also generate sensory instructions required to provide sensory content that enhances the holographic object. As described above, when executed by the LF display system 500, the sensory instructions may generate tactile surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF processing engine 530 may access the sensory instructions from the data storage device 522, or construct the sensory instructions from the vectorized data set. In general, the 4D coordinates and sensory data represent display instructions that can be executed by the LF display system to generate holographic and sensory content.

[0142] The amount of rasterized data describing the energy flow through the various energy sources in the LF display system 500 is very large. Although it is possible to display the rasterized data on the LF display system 500 when accessed from the data storage device 522, the rasterized data cannot be effectively transmitted, received (e.g., via the network interface 524) and then displayed on the LF display system 500. For example, take the rasterized data representing a short video of a holographic projection performed by the LF display system 500. In this example, the LF display system 500 includes a display containing several billion pixels, and the rasterized data contains information for each pixel position on the display. The corresponding size of the rasterized data is huge (e.g., several gigabytes of video display time per second) and is unmanageable for efficient transmission over a commercial network via the network interface 524. For applications that include real-time streaming of holographic content, the efficient transmission problem can be magnified. When an interactive experience is required using input from the sensory feedback system 570 or the tracking module 526, additional problems arise that store the rasterized data only on the data storage device 522. To enable an interactive experience, the light field content generated by the LF processing engine 530 may be modified in real time in response to sensory or tracking input. In other words, in some cases, the LF content cannot simply be read from the data storage device 522.

[0143] Thus, in some configurations, data representing holographic content displayed by the LF display system 500 may be transmitted to the LF processing engine 530 in a vectorized data format ("vectorized data"). Vectorized data may be several orders of magnitude smaller than rasterized data. In addition, vectorized data provides high image quality while having a data group size that enables efficient sharing of data. For example, vectorized data may be a sparse data group derived from a denser data group. Thus, based on how the sparse vectorized data is sampled from the dense rasterized data, the vectorized data may have an adjustable balance between image quality and data transmission size. The adjustable sampling used to generate the vectorized data enables optimization of image quality at a given network speed. Thus, the vectorized data enables efficient transmission of holographic content via the network interface 524. The vectorized data also enables real-time streaming of holographic content over commercial networks.

[0144] In summary, the LF processing engine 530 can generate holographic content derived from rasterized data accessed from the data storage device 522, vectorized data accessed from the data storage device 522, or vectorized data received via the network interface 524. In various configurations, the vectorized data can be encoded prior to data transmission and decoded after being received by the LF controller 520. In some instances, the vectorized data is encoded for additional data security and performance improvements related to data compression. For example, the vectorized data received via the network interface may be encoded vectorized data received from a holographic streaming media application. In some instances, the vectorized data may require a decoder, the LF processing engine 530, or both to access the information content encoded in the vectorized data. The encoder and / or decoder systems may be available for consumer use or licensed to third-party vendors.

[0145] The vectorized data contains all the information in each sensory domain supported by the LF display system 500 in a manner that supports an interactive experience. For example, the vectorized data for an interactive holographic experience includes any vectorized characteristics that can provide accurate physical effects for each sensory domain supported by the LF display system 500. The vectorized characteristics can include any characteristics that can be synthetically programmed, captured, computationally evaluated, etc. The LF processing engine 530 can be configured to convert the vectorized characteristics in the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content converted from the vectorized data from the LF display assembly 510. In various configurations, vectorized characteristics may include: one or more red / green / blue / alpha channels (RGBA) + depth images; multiple view images with or without depth information of different resolutions, which may include a high-resolution center image and other views of lower resolution; material properties such as albedo and reflectivity; surface normals; other optical effects; surface identification; geometric object coordinates; virtual camera coordinates; display plane position; lighting coordinates; tactile stiffness of the surface; tactile extensibility; tactile intensity; amplitude and coordinates of the sound field; environmental conditions; somatosensory energy vectors associated with mechanical receptors for texture or temperature, audio; and any other sensory domain characteristics. Many other vectorized characteristics are also possible.

[0146] The LF display system 500 may also generate an interactive viewing experience. That is, the holographic content may respond to input stimuli that contain information about the user's position, gestures, interactions, interactions with the holographic content, or other information derived from the user profile module 528 and / or the tracking module 526. For example, in an embodiment, the LF processing system 500 generates an interactive viewing experience using vectorized data of real-time performance received on the device via the network interface 524 as part of a real-time streaming application. In another example, if a holographic object needs to move in a certain direction immediately in response to a user interaction, the LF processing engine 530 may update the rendering of the scene so that the holographic object moves in the desired direction. This may require the LF processing engine 530 to use a vectorized data set to render a light field in real time based on a 3D graphics scene with appropriate object placement and movement, collision detection, occlusion, color, shadows, lighting, etc., so as to correctly respond to user interactions. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510.

[0147] The rasterized data includes holographic content instructions and sensory instructions (display instructions) representing the real-time performance. The LF display assembly 510 simultaneously projects the holographic and sensory content of the real-time performance by executing the display instructions. The LF display system 500 monitors the user's interaction with the presented real-time performance (e.g., sound response, touch, etc.) using the tracking module 526 and the user profile module 528. In response to the user interaction, the LF processing engine generates an interactive experience by generating additional holographic and / or sensory content for display to the user.

[0148] To illustrate, consider an example embodiment of an LF display system 500 that includes an LF processing engine 530 that generates a plurality of holographic objects representing balloons floating in the viewing volume of a user of the mobile device during execution of an application in a mobile device. The user can move to touch the holographic objects representing the balloons. Accordingly, a tracking system 580 tracks the movement of the user's hand relative to the holographic objects. The user's movements are recorded by the tracking system 580 and sent to the controller 520. The tracking module 526 continuously determines the movement of the user's hand and sends the determined movement to the LF processing engine 530. The LF processing engine 530 determines the placement of the user's hand in the scene, adjusts the real-time rendering of the graphics to include any desired changes (e.g., position, color, or occlusion) in the holographic objects. The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a tactile surface using a body tactile projection system (e.g., using an ultrasonic speaker). The generated tactile surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the external surface of the holographic object. The LF processing engine 530 uses the tracking information to dynamically instruct the LF display assembly 510 to move the position of the tactile surface together with the position of the rendered holographic object so that the user is given both the visual and tactile perception of touching the balloon. More simply, when the user watches his hand touching the holographic balloon, the user simultaneously feels tactile feedback indicating that his hand touched the holographic balloon, and the balloon changes position or movement in response to the touch. In some examples, rather than presenting an interactive balloon associated with executing an application with content accessed from the data storage device 522, the interactive balloon can be received as part of the holographic content received from the real-time streaming application via the network interface 524.

[0149] The LF processing engine 530 may provide holographic content for simultaneous display to a viewer of media content on a portable mobile device. For example, a video being viewed on a mobile device augmented with the LF display system 500 may include holographic content (a "holographic content track") to be presented to the user during the video. The holographic content track may be received by the device and stored in the data storage device 522. The holographic content track includes holographic content that enhances the viewing experience of the user viewing the video on the device.

[0150] The LF processing engine 530 may create, modify and / or update holographic content based on the user profile. The update of the user profile may cause the update of the displayed holographic content. In some embodiments, the LF processing engine 530 may create, modify and / or update holographic content based on the user profile in combination with the artificial intelligence model.

[0151] The holographic content in the holographic content track may be associated with any number of temporal, auditory, visual, etc. cues to display the holographic content. For example, a holographic content track may include holographic content to be displayed at a specific time during a video. As an illustration, a holographic content track may include a group of holographic dolphins displayed during the video "Bottlenosed" at 35 minutes and 42 seconds after the start of the video. In another example, the holographic content track includes holographic content to be presented when the sensory feedback system 570 records a specific audio cue. As an illustration, the holographic content track includes a holographic avatar smile that is presented when the audio recorded by the sensory feedback system 570 indicates that the user / player is laughing during a gaming application. In another example, the holographic content track includes holographic content to be displayed when the tracking system 580 records a specific visual cue. As an illustration, the holographic content track includes a holographic flashing logo, and the text "Do you want to stop?" is displayed on the display when the tracking system 580 records information indicating that the user is moving away from the device. Determining auditory and visual cues is described in more detail below.

[0152] The holographic content track may also include spatial rendering information. That is, the holographic content track may indicate the spatial locations for presenting holographic content in the displayed video track. For example, the holographic content segment may indicate that certain holographic content will be presented in some holographic viewing volumes but not in other holographic viewing volumes. To illustrate, the LF processing engine 530 may present a holographic screen displaying an email application in a holographic viewing volume directly in front of the user's face. Similarly, the holographic content track may indicate that the holographic content is presented to some viewing volumes but not to other viewing volumes. For example, the LF processing engine may present the holographic screen to the viewing volume directly in front of the user's face but not in any other viewing volume, making the screen private only to the user.

[0153] The LF processing engine 530 may provide holographic content for display to a user in conjunction with an application executed on the device. The holographic content may be received at the LF display system 500 device over a network from an online holographic content store, which makes available holographic content for downloading to a mobile device in exchange for a transaction fee. For example, a user may visit an online holographic content store and obtain a specific superhero character in exchange for a transaction fee to be used as a holographic avatar in conjunction with an application executed on the device. The holographic content may be received at the LF display system 500 over a network from an online holographic application store, which makes available holographic applications for downloading to and executing on a mobile device in exchange for a transaction fee. For example, a user may visit a holographic application store to purchase an application for generating and customizing a holographic avatar, and the executable application may download a specific holographic user interface for generating and customizing a holographic avatar. The holographic content may be provided as part of advertising content associated with a specific application executed on the device. For example, when a user clicks on a game application on a mobile device, the LF display system 500 can access the holographic content of the product as an advertisement provided by the advertiser and display the content during the game application. The holographic content can be stored in the data storage device 522 or streamed to the LF display system 500 in a vectorized format through the network interface 524.

[0154] The LF processing engine 500 may also modify the holographic content to adapt to the environmental characteristics of the location. For example, based on whether the user is in a dark room space, a brightly lit room, a sunny outdoor location, etc., the LF processing engine may modify the holographic content on the mobile device to display a brightness appropriate for the lighting of the location. The LF processing engine may use data stored in the data storage device 522 and information from the camera system in the tracking module 580 to customize the display of the holographic content. In addition, the modification of the holographic content may also employ the user's display preferences determined by the user profile module 528.

[0155] The LF processing engine 530 may also modify the holographic content in response to information received from the sensory feedback system 570. For example, an accelerometer in the sensory feedback system 570 may measure the linear acceleration of the mobile device. The LF processing engine 530 may modify the displayed holographic content to filter out any visual jitter in the holographic display caused by the acceleration. Similarly, a gyroscope in the sensory feedback system 570 may detect changes in the orientation of the mobile device and may reorient the displayed holographic content based on the detected changes. In some embodiments, the tracking module 526 may track the user's head rotation and may reorient the displayed holographic UI in response thereto.

[0156] The LF processing engine 530 may also create holographic content for display by the LF display system 500. Importantly, here, creating holographic content for display is different from accessing or receiving holographic content for display. That is, when creating content, the LF processing engine 530 generates brand new content for display, rather than accessing previously generated and / or received content. The LF processing engine 530 may obtain or receive information from the data storage device 522, the tracking module 526, the user profile module 528, the sensory feedback system 570, the tracking system 580, or a combination thereof to create holographic content for display. In some examples, the LF processing engine 530 may access information from elements of the LF display system 500 (e.g., tracking information and / or user profile), create holographic content based on the information, and display the created holographic content using the LF display system 500. When displayed by the LF display system 500, the created holographic content may be enhanced with other sensory content (e.g., touch, audio, pressure, force, or smell / smell). Furthermore, the LF display system 500 may store the created holographic content in the data storage device 522 for future use.

[0157] In some embodiments, the LF processing engine 530 may cause the generated holographic content to be updated, modified, deleted, paused, or a combination thereof in response to changing the operating state of the mobile device. In some embodiments, the LF processing engine 530 may cause the generated holographic display to be updated, modified, deleted, paused, or a combination thereof in response to information received from the tracking system 580 and interpreted by the tracking module 526. For example, the tracking module 580 may provide the user's line of sight information captured from the tracking system 580 to the LF display module for re-rendering the holographic character so that the holographic character makes eye contact with the user who is interacting with the device. In some embodiments, the operation selection made by the user may affect the display of the holographic content. For example, when the LF display system determines that the user has made an operation selection to cause the alarm clock application to generate a reminder at 6:00am, the LF display system may only present a holographic depiction of a flashing red button and a reminder sound at 6:00am or at a time configured by the user.

[0158] In some embodiments, the LF display system 500 includes a command library 532. The command library 532 may act as a command interface for identifying and interpreting user commands. The identified commands may cause the operating state of the mobile device to change. The user commands may be interpreted based on tracking information received from the sensory feedback system 570 (e.g., audio), the tracking module 526 (e.g., tracked gestures or expressions), the user profile module 528 (e.g., stored individual preferences), or a combination thereof. The command library 532 may apply heuristics to interpret user commands and guide the LF processing engine 530 to generate appropriate holographic content, tactile, and any other multi-sensory information. The command library 532 stores the mapping of received user commands to computational commands that can be executed by the LF display system 500. The command library 532 also stores heuristics associated with the stored mappings. The command library 532 may receive user commands through a holographic user interface displayed by the LF display system 500. Examples of possible user commands received by the LF display system 500 via the mobile device may include different touch contacts by the user to the holographic object (e.g., a finger touch of a particular displayed holographic "button") and different touch movements with respect to the surface of the holographic object (e.g., turning a displayed holographic "knob"). In some embodiments, a touch contact or touch movement may be determined to have occurred when it is determined by the tracking module 526 that the user is within a threshold distance of a displayed holographic object projected as part of a holographic user interface. In some embodiments, the user command may include sensory feedback provided by the user, such as a verbal command spoken by the user within a target area (e.g., a verbal command in the form of "show me where the next gas station is located" captured using an acoustic receiver on the mobile device), gestures and body movements captured by the tracking system 580 (e.g., a hand raised by the user at the target area to indicate that a timer should be paused), sensing of a physical switch, knob, button, or dial connected to the display as part of the sensory feedback system 570, and the like, or a combination thereof.

[0159] The computational commands corresponding to the received user commands to be executed by the LF processing engine may include commands to display a particular holographic object, update the display of holographic content, modify the display of a particular holographic object, display a sequence of holographic objects, pause or play content being presented by the LF display system 500, delete a displayed object, or a combination thereof. Other computational commands may involve actuating other sensors and controls, such as a joystick gaming control. Thus, in some embodiments, a detected user action (e.g., turning a displayed holographic "dial") is reported by the tracking system 580 to the command library 532. In response, the command library 532 may apply heuristics from the stored heuristics and determine that a computational command involving sending instructions to the system to, for example, appropriately set a timer application in a displayed holographic clock needs to be executed.

[0160] In some embodiments, command library 540 receives information from data storage device 522 and tracking module 580 to resolve ambiguous mappings based on user commands that can be detected with less confidence. However, information from tracking system 580 and user profile information from data storage device 522 may provide sufficient heuristic information to unambiguate the mapping and determine the specific computing command to be executed. This determined computing command is then provided to LF processing engine 530 for execution.

[0161] In some embodiments, the LF display system 500 includes a security module 534. The security module 534 enforces secure access to at least some of the functions provided by the LF display system. To this end, the security module 534 may generate a user interface element through the LF display system 500 for receiving verification credential information (e.g., a password or biometric data) from a user of the mobile device. The security module may verify the credential information based on data stored in the data storage device 522. After verifying the user, the security module 534 may enable the user to securely access some or all of the functions of the LF display system 500. In some embodiments, the security module 534 may enable the display of specific holographic content accessed from the data storage device or from an online source, or the execution of certain applications on the mobile device. For example, the LF display system may present specific holographic content to the viewer only after verifying the user's credentials, and not present it to the viewer without verifying the user's credentials. The security module 534 may provide information about authenticating the user of the mobile device to the data store 522, the user profile module 528, the LF processing engine 530, the command library 532, any other component of the LF display system 500, or some combination thereof.

[0162] Dynamic content generation for LF display systems

[0163] In some embodiments, the LF processing engine 530 incorporates an artificial intelligence (AI) model to create holographic content for display by the LF display system 500. The AI ​​model may include a supervised or unsupervised learning algorithm, including but not limited to a regression model, a neural network, a classifier, or any other AI algorithm. The AI ​​model can be used to determine user preferences based on user information recorded by the LF display system 500 (e.g., through the tracking system 580), which may include information about the user's behavior. The AI ​​model can also store the learned preferences of each user in a user profile storage device of the data storage device 522. In some examples, when multiple users can operate the mobile device, the AI ​​model can create holographic content for a specific individual user based on the learned user's preferences. The AI ​​model can access information from the data storage device 522 to create and / or modify holographic content. In some embodiments, the AI ​​model can use the results of the learning algorithm to establish user-specific heuristics for command mappings stored in the command library 532.

[0164] One example of an AI model that can be used to identify characteristics of a user, identify reactions, and / or generate holographic content based on the identified information is a convolutional neural network model with a node layer, where the value at the node of the current layer is a transformation of the value at the node of the previous layer. The transformation in the model is determined by a set of weights and parameters connecting the current layer and the previous layer. In some instances, the transformation can also be determined by a set of weights and parameters used to transform between previous layers in the model.

[0165] The input to the model may be an image captured by the tracking system 580 and encoded onto the first convolutional layer, and the output of the model may be holographic content decoded from the output layer of the neural network. Alternatively or in addition, the output may be a determined characteristic of the user in the image. In this example, the AI ​​model may identify potential information in the image that represents the characteristics of the user in one of the intermediate layers of the neural network. The relevant information between the elements identified in the layer may be retrieved by applying a set of transformations between the corresponding intermediate layers. The weights and parameters used for the transformation may indicate the relationship between the information contained in the starting layer and the information obtained from the final output layer. For example, the weights and parameters may be quantifications of the shape, color, size, etc. contained in the information representing a smiling user in the image. The weights and parameters may be based on historical data (e.g., previously tracked users). In some embodiments, information from the tracking system may be used in conjunction with the AI ​​model to generate a somatotactile surface that is consistent with the presented holographic content or within a threshold distance of the presented holographic content.

[0166] In one embodiment, the AI ​​model comprises a deterministic method that has been trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve the quality of performance using measurements from the tracking system 580 as input and changes in the created holographic content as output.

[0167] Reinforcement learning is a machine learning system in which the machine learns "what to do" - how to map situations to actions - so as to maximize a digital reward signal. The learner (e.g., LF processing engine 530) is not told which actions to take (e.g., generate a prescribed holographic content), but rather discovers which actions produce the highest reward by trying the actions (e.g., improving the quality of the holographic content by making more users smile). In some cases, an action can affect not only the immediate reward, but also the next situation, and therefore all subsequent rewards. These two features - trial and error search and delayed rewards - are two notable features of reinforcement learning.

[0168] The AI ​​model may include any number of machine learning algorithms. Some other AI models that may be employed are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In general, the LF processing engine 530 takes input from the tracking module 526 and / or the user profile module 528, and the machine learning model generates holographic content in response. Similarly, the AI ​​model may guide the rendering of the holographic content.

[0169] Network environment

[0170] Figure 6 A block diagram of a network environment 600 is depicted in accordance with one or more embodiments. The network environment 600 includes one or more mobile devices 610, a network 630, and one or more remote servers 640.

[0171] In one embodiment, mobile device 610 is a portable computing device that executes a computer program module (e.g., a web browser or client application) that allows a user to browse the Internet, consume media content, and execute a variety of software applications. Mobile device 610 can be, for example, a personal computer, a tablet computer, a smart phone, a dedicated electronic reader, or any other type of network-enabled device. Mobile device 610 includes a light field display system 612 and a user interface 614. In other embodiments, mobile device 610 includes more or fewer modules than those described herein. Similarly, functionality can be allocated between modules and / or different entities in a manner different from that described herein.

[0172] The mobile device 610 includes a LF display system 612, a user interface 614, a mobile operating system (OS) 616, a power system 618, and a global positioning system (GPS) module 620, among other components. The LF display system 612 is Figure 5 An embodiment of a LF display system 500 is depicted in FIG. A user 650 may interact with a mobile device 610 through a user interface 614 .

[0173] The LF display system 612 can access the hardware components of the mobile device 610. For example, the LF display system 612 can be powered by accessing the power circuit within the mobile device to derive power from a power source (e.g., from a battery or from a charging unit on the mobile device). The tracking system 580 may be able to access one or more cameras or depth sensors of the mobile device 610 or the GPS module 620. The sensory feedback system 570 can access buttons, switches, microphones, or keyboards of the user interface 614. The controller 520 can access the real-time clock (RTC) circuit within the mobile device as needed to implement time-based functions. The user profile module 528 can determine user identification through direct access to the fingerprint sensor or keyboard. The user profile module can download or upload user profile information via the network interface 614 connected to the network 630. Depending on the user settings, the LF display assembly can project holographic content or 2D content. Typically, the LF display system 612 is integrated into other hardware components of the mobile device 610.

[0174] The mobile device 610 includes a mobile OS 616 that provides a mobile operating experience to a user 650 of the mobile device 610. The mobile OS 616 manages the provision of mobile phone capabilities, the operating functions of the LF display system, the user interface capabilities suitable for the mobile environment including the display and I / O devices, etc. The LF display system 612 may be a holographic display such as provided by the LF display system 500. The mobile OS 616 manages applications executed on the mobile device 610 that can be controlled by the user through the user interface of the mobile OS 616. The user interface may be a graphical user interface (GUI) projected from the LF display system 612. The mobile OS 616 has various abstraction layers, each of which interfaces with a lower layer and includes a kernel for managing mobile device resources such as CPU, memory, and I / O. The kernel provides features such as low-level scheduling of processes, inter-process communication, process creation and deletion, interrupt handling, etc. Other abstraction layers of the mobile OS interact with the kernel through system calls or application programming interface (API) layers. A user 650 of the mobile device 610 interacts with the mobile OS 616 through a user interface 614 .

[0175] The functional view of the mobile OS is to view the mobile OS 616 as an event-driven system that resides in a wait loop for events delivered by the hardware of the mobile device 610 and implements appropriate responses to delivered events residing in the event queue. Examples of such events may be, for example, a touch to a specific location on a 2D touch screen, an incoming call on a cellular network, a message on a network interface, etc. In addition, the event-driven mobile OS 616 may respond to events reported by the LF display system 612, including touches of a hand, finger, or other body part on the projected holographic interface, gestures reported by the tracking module 526 based on tracking data received from the tracking system 580, pressure-related touch events on coincident tactile surfaces at the same location on the projected holographic button. The mobile operating system's response to these and other events delivered by the hardware components of the mobile device involves actions performed by the mobile OS at all abstraction levels of the mobile OS functionality.

[0176] The mobile OS 616 or an application running on the mobile OS 616 can manage the LF display system 612, thereby allowing the LF display system 612 to access the hardware components of the mobile device as needed. The application running on the mobile OS 616 can manage the presentation of holographic content after being requested by the user. Similarly, the mobile OS 616 can display a holographic user interface and accept operation selections from the user based on the tracking information provided by the tracking module 526, wherein the tracking module 526 can analyze data from the tracking system 580. The mobile OS 616 can run an application that accesses a social media account and interfaces with the user profile module 528 to present personalized holographic content. Typically, the mobile OS 616 can interface with the controller 520 of the LF display system 612 to manage the controller, display holographic content, and provide access to the hardware components of the mobile device 610. In some embodiments, the mobile OS 616 can receive a pluggable module to act as a driver for the controller 520 for managing the LF display system. The mobile OS may interface with these modules or applications running on the mobile OS 616 and access API calls to the controller 520 of the LF display system in order to display holographic content, provide or retrieve user profile information, exchange tracking information, and the like.

[0177] The mobile device 610 includes a power system 618 for powering the various components of the mobile device 610. The power system 618 may include a power management system, one or more power sources (e.g., batteries, alternating current (AC), a recharging system, a power fault detection circuit, a power converter or inverter, a power status display, and other components related to the generation, management, and distribution of power in the mobile device. As previously mentioned, the LF display system 612 can power itself by accessing the power system 618 on the mobile device.

[0178] The mobile device 510 includes a GPS module 620. The GPS module 620 determines the location of the mobile device and provides this information for use in various applications. The location information can be provided to the LF display system 500 for use in applications that can utilize location information, such as map-based applications, etc. In some embodiments, the LF display system 500 can display holographic content in response to data from the GPS module 620 and based on the location data received from the GPS module 620. For example, once the GPS module provides location coordinates mapped to a destination location by a navigation application executed on the mobile device, the LF display system 500 associated with the execution of the navigation application can display holographic content indicating arrival at a certain location.

[0179] The network 630 may include any combination of local area networks and / or wide area networks using both wired and / or wireless communication systems. For example, the local area network may include WiFi and Bluetooth networks. In one embodiment, the network 630 uses standard communication technologies and / or protocols. For example, the network 630 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of networking protocols for communicating via the network 630 include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). The data exchanged on the network 630 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or part of the communication links of the network 630 may be encrypted using any suitable one or more technologies.

[0180] One or more remote servers 640 may communicate with the mobile device 610 via the network 630. In one embodiment, the remote server 640 may be an application provider that provides applications for execution and content for display on the device 610. The remote server may be an online holographic application store that makes available holographic applications for downloading to and executing on the mobile device in exchange for a transaction fee. The remote server 640 may be an online holographic content store that makes available holographic content for downloading to the mobile device in exchange for a transaction fee. The remote server 640 may be a third-party application and / or content provider. New applications, new content, and updates to existing applications and content may be available for execution and / or display from the remote server 640 at the device 610 through any push-enabled or pull-enabled updates of the network 630. These applications and content may contain holographic content tracks to be stored in the data storage device 522 for projection and display by the light field display system 612. The content available to the mobile display device 610 may include advertisements, product offers, and product coupons associated with executing software applications in the mobile device 610 to be displayed by the light field display system 612. For example, holographic content depicting popular animated characters may be downloaded from a third-party content provider via the network 630 and stored in the light field display system 612 for use by the user 650 as a "holographic avatar" to represent the user 650 in one or more applications.

[0181] In one or more embodiments, to ensure prevention of unauthorized network access to device 610, the system may use security settings to authenticate network access to device 610 by an entity for establishing a secure connection to device 610, and / or downloading content to device 610. In some embodiments, device 610 may incorporate authentication factors (e.g., passwords and smart cards) or use biometric methods to enforce access controls when accessing mobile device 610 through network 630 or user interface 614.

[0182] Holographic user interface and interpretation of "touch" in LF display system

[0183] The following is a description of some embodiments of a LF display system for generating a holographic user interface (UI) for a user to interact with a mobile device. Figure 5 Elements of the LF display system 500 are shown in FIG.

[0184] In some embodiments, the generated holographic UI is a user interface that includes one or more holographic UI objects presented to the user. In some embodiments, the presented holographic UI objects may also be part of a virtual image display that depicts, for example, a virtual screen floating above the surface of a mobile device on which the holographic objects appear to reside. In some embodiments, the presented holographic objects and / or virtual images may enhance the physical UI on the device. The generated holographic UI objects may include, but are not limited to, 3D holographic objects, such as "buttons," "knobs," "dials," "keyboards," and the like. The generated holographic objects may also include 2D holographic objects, such as a holographic "screen" that displays a holographic menu with a list of many options. The user may be prompted (e.g., by an audio prompt, or by a visually displayed prompt) to select one or more options by pointing to or touching a specific option in the holographic menu (e.g., with their finger).

[0185] In some embodiments, the presented holographic UI object may be enhanced with tactile stimulation to produce a tactile surface that is in the same position as the surface of the presented holographic object. Thus, for example, the sensory feedback system 570 may project ultrasonic energy (e.g., pressure waves) to produce a tactile surface that is in the same position as the front display surface on the holographic "button". The tracking system 580 may track the user's movement toward the holographic "button" (e.g., the movement of the user's finger). The movement information may be sent by the tracking system 580 to the controller 520. The tracking module 526 in the controller 520 may determine that the finger is "touching" or "pressing" the holographic "button" when the tracked finger is within a certain threshold distance of the presented tactile surface. In addition, in response to this determination by the tracking module 526, the command library 532 may instruct the LF processing engine 530 to present the holographic "button" and juxtapose the presented tactile surface at different locations, corresponding to the movement of the holographic "button" as if it were "pressed" by the user.

[0186] In some embodiments, the holographic UI can be modified or updated in response to information received from the sensory feedback system 570. For example, an accelerometer in the sensory feedback system 570 can measure the linear acceleration of the mobile device, and can incrementally update the displayed holographic UI in synchronization with the measured acceleration. Similarly, a gyroscope in the sensory feedback system 570 can detect changes in the orientation of the mobile device, and can reorient the UI based on the detected changes. In some embodiments, the tracking module 526 can track the user's head rotation and can reorient the displayed holographic UI in response.

[0187] In another embodiment, the holographic UI may involve presenting a holographic character that greets the user. In some embodiments, the holographic character may be presented together with the presentation of a holographic UI object. In other embodiments, the holographic character may be presented to the user first, and then a holographic "menu" with holographic UI objects may be presented. In some embodiments, the holographic character may be modified in response to commands received from the user. In still other embodiments, all interactions may involve spoken commands from the user received by an acoustic receiver in the sensory feedback system 570. In some embodiments, the presented holographic character may be reinforced with acoustic stimulation to issue a verbal response to the user through a speaker in the mobile device.

[0188] In some embodiments, a holographic character may be generated and personalized by an AI model. A holographic character may have a variety of characteristics that may change for taste, including any of the following: appearance (e.g., the appearance of a celebrity), clothing (or not), personality, voice (e.g., the accent or voice of a celebrity), temper, diction (e.g., the number of swear words used), or other characteristics. A holographic character may have the appearance of a CG character (e.g., a superhero), a famous person such as an actor or athlete, a random selection of one of many available avatars, or another appearance. The appearance of a holographic character may be accompanied by a recognizable acoustic voice (e.g., a celebrity voice that matches the appearance) and may be easily changed by the user. A holographic character may also access a user profile or social media account to make it more attractive to the user. For example, a holographic character may provide interesting feedback based on characteristics observed in photos or discussions that appear on a user's social media account, or provide jokes based on calendar reminders for the day of a scheduled event. For the user's entertainment, a holographic character may be placed in a bad mood or a happy mood depending on the user's choice. A selection of holographic character features for viewing and experience on a mobile device may be downloaded via a remote server 640 at a third party content provider, such as an online store, via a network 630, in some cases in exchange for a transaction fee.

[0189] In some embodiments, the tracking module 526 determines the presence of the user within the target view of the LF display system based on the information received from the tracking system 580, analyzes the received information and sends the analysis results to the command library 532. The command library 532 may use the heuristics stored in the command library 532 based on the received analysis and determine that a user interface needs to be displayed to the user. In addition, using the user profile information stored by the user profile system 528, the command library 532 may apply other heuristics in the command library 532 and determine from the stored mapping that a customized user interface involving displaying a specific holographic character that greets the user of the device needs to be generated. The command library 532 may issue a computational command to the LF processing engine 530 to display the determined holographic character displayed in the holographic object volume that greets the user. Executing this computational command may involve the LF processing engine 530 obtaining image data related to rendering and displaying the holographic character from the data storage device 522. The command library 532 may also trigger an acoustic transducer in the sensory feedback system 570 to send an audible greeting synchronized with the displayed holographic character. The LF processing engine 530 can use the tracking information obtained from the tracking system 580 to render the holographic character to view and / or make eye contact, or otherwise interact with the user. The verbal response from the user can be received by an acoustic receiving device (e.g., a microphone or microphone array) located within the sensory feedback system 570, which presents this received response data to the controller 520 for further processing. The command library 532 can use the data received from the sensory feedback system 570, apply heuristics and determine in response to the user's verbal utterances, for example, that a navigation assistance application interface needs to be displayed to the user.

[0190] In another embodiment, the user may configure the LF display system to always start with presentation of a particular customized 3D holographic user interface that is also enhanced with presentation of a particular desired application, such as an email application and / or a social media application. In some embodiments, the mobile device user may configure application settings for various applications associated with the LF display system to have a desired particular holographic user interface presented for use by the user when executing a particular application.

[0191] In other embodiments, the holographic user interface includes more or less functionality than described herein. Similarly, the functionality may be distributed between modules and / or different entities in the LF display system 500 in a manner different than that described herein.

[0192] The tracking system 580 may track the user's movements (e.g., by using a camera, a set of cameras, a depth sensor, or some other sensor within the tracking system 580). The user's movements may be sent by the tracking system 580 to the controller 520 for touch interpretation. In some embodiments, touch interpretation is performed as follows: the tracking system 580 first tracks the user's movements as the user approaches a presented holographic object (e.g., with one or more fingers). The tracking system 580 sends the tracked movement information to the controller 520. The tracking module 526 determines that the tracked finger is within a threshold distance of the presented holographic object, and in response determines that the user is "touching" the holographic object. In other embodiments, touch interpretation may involve tracking other categories of movement with respect to the presented holographic object by the tracking system 580. The tracked movement of the user may include, but is not limited to, determining that the user is performing one or more of the following with respect to one or more presented holographic objects: changing position, moving, pressing, pressing, dialing, rotating, typing, poking, etc.

[0193] In some embodiments, the sensory feedback system 570 may sense a "touch" of a tactile surface that enhances a displayed holographic object. This determination may be based on receiving input from a pressure sensor in the sensory feedback system 570 that detects modifications caused by the touch in the ultrasonic energy that generates the tactile surface. This information may be sent to the controller 520 for further analysis.

[0194] Figure 7 FIG. 7 is a diagram of an embodiment of a LF display system 700 implemented as part of a mobile device according to one or more embodiments. Figure 5 . The mobile device 710 contains an LF display assembly 720 of the LF display system 700. The LF display assembly 720 may include one or more LF display modules (e.g., Figure 2B LF display module 210 or Figure 4A 4. In one or more embodiments, the mobile device 710 includes one or more control panels 730 and / or 2D digital displays (including 2D touch screens) that may contain physical features (e.g., buttons, etc.) to access the device by a particular user. In addition, the mobile device 710 also includes a LF display assembly 720 (e.g., Figure 5 LF display assembly 510 in the LF display assembly 510) for generating holographic content for a user of the device (e.g., viewer A 772 or viewer B 774). The LF display system 700 may also include Figure 5 Any combination of other components of the LF display system 500 shown in FIG. 5 , such as the sensory feedback system 570 , the tracking system 580 , the user profile system 590 , and the controller 520 .

[0195] The LF display system may present holographic content including at least one holographic object to a user using an LF display assembly 720 including one or more LF display modules. The holographic object may be three-dimensional (3D), two-dimensional (2D), or a combination thereof. In addition, the holographic object may be multi-color (e.g., full color). Figure 7 As shown in FIG. 7 , the holographic object volume 740 extends in front of and behind the LF display surface of the LF display assembly 720 (e.g., similar to Figure 1 762 and 764 can be placed anywhere within the holographic object volume 740. Viewer A 772 and viewer B 774 can view the displayed holographic objects 762 and 764. The holographic objects 762 and 764 can also have a tactile surface projected from the LF display system 700 so that the tactile surface coincides with one or more surfaces of the holographic objects. The tracking system of the LF display system 700 can track a user touching the surface of the displayed holographic object 762 or 764, which can cause the sensory feedback system to trigger a response. Thus, movement of the finger 770 toward, for example, the holographic object 762 and the "touch" of the finger 770 on the tactile surface of the holographic object 762 can be established by the tracking system in the LF display system 700. The projected tactile surface coincident with holographic objects 762 and 764 may be appropriately adjusted by the sensory feedback system in response to movement of finger 770 (eg, changing the holographic content depicting the pressed button while changing the position of the tactile button).

[0196] Customizable holographic displays in mobile devices

[0197] In some embodiments, the LF display system may be combined with Figure 5 The user information obtained by the user profile system 528 in the user profile system 528 is used to track the system to produce a customized interactive experience. As an example, in an application with a holographic UI, the size of the generated holographic "buttons" can be different based on the user - for example, they may be small for children, while the size may be larger and brighter in color for easier viewing by elderly users.

[0198] Furthermore, in some embodiments, the LF display system may be customized to display specific content for a specific user. Thus, for example, the LF display system may be customized to display a holographic "button" for one user and a holographic "dial" for another user. Figure 7, when the tracking system 580 tracks the presence of viewer A 772, the configured settings may result in a customized display of the holographic object 762 (i.e., a holographic "button") (only) for viewer A 772. If viewer B 774 is not present, then the display of the holographic object 764 is not presented. Similarly, when the presence of viewer B 774 is sensed, the configured settings may result in a customized display of the holographic object 764 (i.e., a holographic "dial") for viewer 774.

[0199] In other embodiments, the LF display module may display a customized holographic character to interact with the device user (e.g., see, speak, hear, touch, or some combination thereof). One or more characteristics of the holographic character may be customized by the user. The customizable characteristics may be any of the visual, audio, and personality characteristics of the holographic character. In some embodiments, some or all characteristics associated with the holographic character may be retrieved from one or more online systems over a network in exchange for a transaction fee (e.g., adding a "Bart Simpson" face to the holographic character). Additionally, the tracking system may track the user's gaze. The tracking system may provide gaze information to the LF display system 500 to re-render the holographic character to make eye contact with the user who is interacting with the device.

[0200] In some other embodiments, when multiple users use the mobile device, the user profile system can build a user profile for each device user to store user preferences or other user characteristics. The user profile can be built by the user profile module 528 for each of the users, and these usage profiles are stored in the data storage device 522.

[0201] The LF display system may utilize the user profile system 590 to personalize holographic content for the user during each subsequent use of the device. For example, the LF display system 500 may address the user by name (e.g., visually or by audio). This may involve using tracking information to determine the identity of the user or by user authentication, such as facial or voice recognition, entering a password, or some other method. In some embodiments, the LF display system 500 generates holographic content corresponding to the user's response to previously generated holographic content. For example, in some embodiments, the AI ​​machine learning module in the LF processing engine 530 may learn how the user starts each day by checking their email and then solving a Sudoku puzzle. The LF display system 500 may generate a holographic 3D "home" screen display with an email application open and a Sudoku puzzle displayed, which is obtained and presented based on data stored in the data storage device 522.

[0202] In some embodiments, customizing the light field display for a user may involve a privacy mode, where the tracking system 580 tracks the user's position and generates customized holographic content to be visible only to the user. Figure 2BA plurality of viewing sub-volumes 290 that may be formed by the LF display system 200 are shown. As described in part therein, a user in viewing sub-volume 290A may perceive holographic objects presented in holographic object volume 255 that the user may not perceive when viewing sub-volumes 290B, 290C, and 290D, thereby ensuring viewing privacy. Thus, in these cases, a user reading an email on a holographic screen display may ensure that the display is visible only to the user and not to nearby persons. In other embodiments, all users in a viewing volume of the LF display assembly may see all holographic objects positioned in the holographic viewing volume (but all may see different perspectives depending on the user's position). The mobile device may provide a privacy setting for the LF display system to present holographic content for one or more tracked users in a privacy mode.

[0203] Exemplary User Application Using LF Display System in Mobile Devices

[0204] Mobile devices are used in conjunction with various software applications. Some of these applications can be further enhanced using the LF display system in the device. Some of these examples are briefly presented below.

[0205] In one embodiment, the LF display system 500 may be used with a puzzle application involving blocks. The LF display system 500 may generate a holographic display of an "empty structure" with markings indicating blocks that need to be "filled" with the holographic "empty structure". At the same time, the system may display a holographic set of "blocks" of various shapes to the viewing player. When the puzzle application was previously downloaded via the network 630, the image rendering data used to generate the structure and the set of "blocks" may be stored in the data storage device 530. When the player touches a block and moves his or her finger to the correct 3D position in the structure that accommodates the block, the tracking system 580 tracks this movement, and the LF processing engine 530 generates a holographic rendering of the "structure" with blocks that move with the position of the player's finger, and then causes the corresponding 3D position to be "filled" with blocks as the blocks are "moved" into place.

[0206] Other holographic user applications may involve using the sensory feedback system 570 in coordination with the tracking system 580 to generate holographic renderings that "move" when "touched." Examples include presenting holographic renderings that aid in furniture assembly, do-it-yourself applications, and educational applications where, for example, medical students are taught to dissect the human body using holographic human body renderings that change in response to the user's identified activities.

[0207] In one embodiment, related to generating advertisements for display by LF display system 500, ultrasonic transmitters in sensory feedback system 570 may be used to generate tactile surface samples that provide tactile information about a product to a user (e.g., a carpet manufacturer may display an advertisement showing carpet samples with different textures that a user may experience).

[0208] Other application examples involve generating a holographic representation of a "cyber pet" that responds to sensory stimuli such as touch and verbal commands and provides an immersive experience of the "cyber pet" to the "cyber pet" owner. The tracking system 580 and the sensory feedback system 570 together can provide the user with a responsive experience of the holographic "cyber pet".

[0209] Figure 8 1 is a flow chart illustrating a process for implementing user interaction with a mobile device using a light field display system according to one or more embodiments. In one embodiment, Figure 8 The process is performed by the light field display system 500 (eg Figure 5 ). Other entities may perform some or all of the steps of this process in other embodiments. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0210] The light field display system 500 receives 810 commands from a user. The user command may be received through a physical control, such as a physical button that can be pressed and a screen that can be physically touched, both of which may be located on the mobile device and are part of the sensory feedback system 570. In another example, the user command may be received as a voice command at an acoustic receiver, such as an acoustic microphone, that is part of the sensory feedback system 570. The user command may also be received through the user's holographic object interaction, such as with a displayed holographic UI. The user command may also be received through a tracking system 580 that tracks the user via instructions from a tracking module 526. In an embodiment, the user command may be captured as a gesture, body movement, or expression of the user recorded by the tracking system 580 and analyzed by the tracking module 526. In another embodiment, the user command may be received when the tracking system 580 detects that the user has touched a tactile holographic surface, such as a holographic "button" on a displayed holographic UI. The user command may also be received at the mobile device over a communication network. The received user command is relayed to the command library 532 via the controller for interpretation and analysis.

[0211] The light field display system 500 processes 820 the user command to determine a computing command. The command library 532 in the light field display system 500 interprets the received user command, possibly in association with the user profile information stored by the user profile system 528, and if necessary, in conjunction with the tracking module to interpret user gestures, body movements, or expressions, can apply the stored heuristics, and determine the computing command based on the received user command. An example of a determined computing command is a command to generate more holographic content for display by a user of a mobile device. Another example may be to modify the generated holographic content. The determined computing command may activate a sensor in the LF display system, such as an imaging sensor for tracking the user's line of sight. The determined computing command may also activate a control within the mobile device. An example of a determined computing command is to generate more holographic content for display by a user of the mobile device. Another example may be to modify the generated holographic content. The generated holographic content may be modified in response to a parameter value received at the LF display system. Thus, for example, when an application is downloaded to a mobile device over a network, a holographic "sphere" may be projected to indicate the progress of the download. The value of the parameter indicating the percentage of the downloaded content at a point in time when the download is in progress can be used by the LF display system to depict a holographic ball filled with color that shows the progress of the application as it downloads. The determined computing command can activate a sensor in the LF display system, such as a trigger sensor for tracking the user's gaze. The determined computing command can also activate a control within the mobile device. For example, the computing command can increase the volume of a speaker in the mobile device, or set a specific audio tone as an alarm reminder.

[0212] The light field display system generates 830 holographic content for display based on the calculation command. The LF processing engine 530 in the light field display system 500 receives the determined calculation command from the command library 532 and generates 830 holographic content for display based on the calculation command.

[0213] The light field display system 500 displays 840 the generated holographic content. The controller 520 sends display instructions to project the holographic content from the LF processing engine 530 to the LF display assembly 510. The LF display assembly 510 displays the holographic content to the user.

[0214] Additional configuration information

[0215] The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Those skilled in the relevant art may appreciate that many modifications and variations are possible in light of the above disclosure.

[0216] Some parts of this description describe embodiments of the present disclosure according to algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are usually used by technical personnel in the field of data processing to effectively convey the essence of their work to other technical personnel in the field. When described functionally, computationally or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it has also been demonstrated that it is sometimes convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0217] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0218] Embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. This computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to obtain increased computing power.

[0219] Embodiments of the present disclosure may also relate to products produced by the computing process described herein. Such products may include information resulting from the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0220] The language used in the specification is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, it is intended that the scope of the present disclosure is not limited by this specific embodiment, but rather by any claims based on the application attached thereto. Therefore, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure set forth in the following claims.

Claims

1. A mobile device, comprising: A light field (LF) display system comprising: a controller configured to generate holographic content; an LF display assembly comprising one or more LF display modules configured to present the generated holographic content to a user of the mobile device; and a command interface configured to recognize one or more commands from the user; The controller is further configured to: Retrieving at least a portion of the holographic content in a vectorized format from an online system via a network interface; and A vectorized format of the holographic content is converted into a rasterized format for presentation by the LF display assembly. 2 . The mobile device of claim 1 , wherein the controller is configured to update the generated holographic content in response to a change in an operating state of the mobile device. 3 . The mobile device of claim 1 , wherein the controller is further configured to modify holographic content based on one or more identified commands from the user. The mobile device of claim 1 , wherein the one or more identified commands cause a change in an operational state of the mobile device.

5. The mobile device according to claim 1, wherein the LF display system further comprises: a tracking system comprising one or more tracking devices configured to obtain information about the user of the mobile device; as well as Wherein the controller is configured to generate the holographic content for the user based in part on the information obtained by the tracking system.

6. The mobile device of claim 5, wherein the generated holographic content comprises a holographic user interface (UI) having one or more holographic features to control the operation of the mobile device, and wherein the command interface is further configured to recognize the one or more commands received from the user through the generated holographic UI.

7. The mobile device of claim 5, wherein the command interface is further configured to receive a command from the user through the holographic UI based on determining that the user is within a threshold distance of the one or more holographic features in the presented holographic UI.

8. The mobile device of claim 5, wherein the information obtained by the tracking system includes any combination of: the user's location, the user's movement, the user's gestures, the user's expression, the user's line of sight, the user's gender, and the user's age.

9. The mobile device of claim 5, wherein the controller is configured to use the information obtained by the tracking system and an artificial intelligence model to generate the holographic content.

10. The mobile device of claim 5, wherein the tracking system comprises any combination of sensors, the sensors comprising: A light field camera, one or more 2D imaging sensors, and a depth sensor.

11. The mobile device of claim 10, wherein the controller is further configured to generate the holographic content by capturing data from the one or more sensors within the tracking system and generating the holographic content based on the captured data.

12. The mobile device of claim 11, wherein the controller is further configured to modify the generated holographic content based on data captured from the one or more sensors within the tracking system.

13. The mobile device according to claim 1, the LF display system further comprising: A sensory feedback system comprises at least one sensory feedback device configured to receive sensory feedback when presenting the holographic content.

14. The mobile device of claim 13, wherein the command interface is further configured to recognize the received sensory feedback as one or more commands received from the user.

15. The mobile device of claim 13, wherein the sensory feedback system comprises an ultrasonic transducer, a pressure sensor, a force sensor, a temperature sensor, a touch sensor, a proximity sensor, or any combination thereof.

16. The mobile device of claim 13, the sensory feedback system further comprising one or more of buttons, dials, knobs, keyboards, fingerprint sensors, joysticks, other input hardware, or any combination thereof.

17. The mobile device of claim 13, the sensory feedback system further comprising a microphone and voice recognition software.

18. The mobile device according to claim 1, the LF display system further comprising: A sensory feedback system comprises at least one sensory feedback device configured to provide sensory feedback when presenting the holographic content.

19. The mobile device of claim 18, wherein the controller is further configured to enhance the generated holographic content with sensory content, the sensory content comprising tactile stimulation, sound stimulation, temperature stimulation, olfactory stimulation, pressure stimulation, force stimulation, or any combination thereof.

20. The mobile device of claim 18, the sensory feedback system further comprising an ultrasonic energy projection device configured to generate a volumetric tactile surface within a threshold distance of a surface of the rendered holographic object or coincide with a surface of the rendered holographic object.

21. The mobile device of claim 20, the ultrasonic energy projection device further configured to adjust one or more of: resistance of the generated body tactile surface to user touch, texture of the generated body tactile surface, or tactile intensity based on the value of the parameter received at the controller.

22. The mobile device of claim 5, further comprising an ultrasonic energy projection device configured to generate a body tactile surface, the body tactile surface being generated based in part on the information obtained by the tracking system.

23. The mobile device of claim 22, wherein the ultrasound energy projection device is further configured to update the generated body tactile surface in response to information obtained by the tracking system.

24. The mobile device of claim 22, wherein the ultrasound energy projection device is further configured to use the information obtained by the tracking system and an artificial intelligence model to generate the body tactile surface.

25. The mobile device of claim 1, wherein the LF display system is configured to modify presentation of the holographic content based on environmental characteristics at a location where the holographic content is presented to the user.

26. The mobile device of claim 1, wherein the LF display system is configured to modify the presentation of the holographic content based on a value of a parameter received at the LF display system.

27. The LF display system of claim 1, wherein the controller is further configured to generate the holographic content based in part on a user profile maintained for the user in the LF display system.

28. The LF display system of claim 1, further comprising: a tracking system comprising one or more tracking devices configured to obtain information about the user of the mobile device; as well as A user profile module configured to: accessing said information obtained by said tracking system; processing the information to identify the user; and Based on the information obtained by the tracking device, the method is further configured to perform one or more of the following: generating a user profile for the user; and modifying the user profile for the user; and Wherein the controller is further configured to generate the holographic content for the user based in part on the user profile.

29. The LF display system of claim 28, wherein the controller is further configured to use the user profile and the artificial intelligence model to perform one or more of the following: generating the holographic content; and The holographic content is modified.

30. The LF display system of claim 28, wherein the user profile module is further configured to: updating the user profile using information from the user's social media account; and Wherein the controller is configured to generate the holographic content based in part on the updated user profile.

31. The mobile device of claim 1, wherein the holographic content is presented with an application executing on the mobile device.

32. The mobile device of claim 31, wherein the rendered holographic content is based on location coordinates obtained from a global positioning system (GPS).

33. The mobile device of claim 31, wherein the rendered holographic content is based on content received from an online source.

34. The mobile device of claim 31, wherein the presented holographic content is based on content received from an online source in response to an operation selection made by the user.

35. The mobile device of claim 1, wherein the presented holographic content includes a holographic character.

36. The mobile device of claim 35, wherein characteristics associated with the presented holographic character are at least partially configured by the user.

37. The mobile device of claim 35, wherein the characteristics associated with the presented holographic character include visual, auditory, and personality characteristics of the presented holographic character.

38. The mobile device of claim 35, wherein characteristics associated with the rendered holographic character are retrieved from an online system via a network.

39. The mobile device of claim 37, wherein the characteristics are retrieved from the online system over the network in exchange for a transaction fee.

40. The mobile device of claim 8, wherein the controller is configured to generate a holographic character for the user based on the information obtained by the tracking system.

41. The mobile device of claim 40, wherein the LF display assembly is configured to update eyes of the generated holographic character to maintain eye contact with the line of sight of the user.

42. The mobile device of claim 1, wherein the LF display system further comprises: A security module, comprising a verification module, wherein the security module is configured to: receiving security credentials from the user; as well as verifying the received security credentials; enabling secure access to one or more functions of the mobile device; and The generation of specific holographic content is achieved in response to the verification.

43. The mobile device of claim 1, wherein operating functions of the controller, the LF display assembly, and the command interface are managed by a mobile operating system (mobile OS) in association with other operating functions of the mobile device.

44. The mobile device of claim 43, wherein the mobile OS receives information associated with input received from one or more of: the controller, the LF display assembly, and the command interface.

45. The mobile device of claim 44, wherein the mobile OS performs actions at one or more functional abstraction layers in response to the received information.

46. ​​The mobile device of claim 43, wherein the mobile OS manages the operational functions of the controller, the LF display assembly, and the command interface through one or more of: a pluggable module or an application executed in the mobile OS.

47. The mobile device of claim 46, wherein the mobile OS interfaces with the controller to display holographic content.

48. The mobile device of claim 43, wherein the mobile OS facilitates access by the LF display system to hardware components of the mobile device.

Images