Consumer Devices

Abstract

A light field display system is implemented in a consumer device to present holographic content to a user of the consumer device, providing an immersive experience for the user. The system generates and presents the holographic content to the user. The system receives commands from the user. In some embodiments, the presented holographic content may include a holographic user interface used by the system to receive commands from the user. The system then recognizes the received commands, determines one or more computational commands to execute by the system, and executes the determined computational commands.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to International Application Nos. PCT / US2017 / 042275, PCT / US2017 / 042276, PCT / US2017 / 042418, PCT / US2017 / 042452, PCT / US2017 / 042462, PCT / US2017 / 042466, PCT / US2017 / 042467, PCT / US2017 / 042468, PCT / US2017 / 042469, PCT / US2017 / 042470, and PCT / US2017 / 042679, all of which are incorporated by reference in their entireties. [Background technology]

[0002] The present disclosure relates to consumer devices such as ovens, refrigerators, stoves, washers, dryers, home assistants (e.g., Amazon's Alexa™) and dishwashers, and in particular to light field displays in consumer devices.

[0003] Conventional consumer devices (also referred to herein as appliances) generally rely on two-dimensional (2D) visual displays for their user interfaces. The 2D display screens of appliances are designed to display various functions that the appliances offer to users and provide an interactive display to allow the users to select and configure the desired functions. However, such 2D visual displays are limited by the mounting area they occupy on the surface of the appliance. As a result, the display content is also limited to the 2D visual display, limiting the viewing experience that can be provided to users of the appliance. In particular, 2D visual displays have limited opportunities to provide users with an immersive operating experience.

[0004] The described embodiments relate to technology involving light field display systems in consumer devices that provide a user with an immersive interaction experience while using the device. Abstract

[0005] A light field (LF) display system of a consumer device generates holographic content for viewing by a user of the consumer device. The generated holographic content may be a holographic user interface that is displayed to the user of the consumer device. The light field display system may receive commands from the user. The LF display system determines computational commands based on the received user commands and appropriately executes the determined computational commands. In some embodiments, the holographic content is augmented with a tactile surface on the generated holographic content. The tactile surface may provide a haptic sensation to the user. The light field system of the consumer device may display holographic images of objects present within the consumer device, receive information about the objects, and modify the displayed holographic content based on the received information.

[0006] In some embodiments, a consumer device comprises an LF display system including a controller configured to generate holographic content, an LF display assembly, and a command interface. The LF display assembly comprises one or more LF display modules configured to present holographic content within a holographic object volume to one or more users of the consumer device positioned within a viewing volume of the one or more LF display modules.

[0007] In some embodiments, the LF display system includes a tracking system and / or a viewer profiling module. The tracking system and viewer profiling module can monitor and store characteristics of a user of a consumer device, a user profile describing the user, and / or the user's response to holographic content presented during operation of the consumer device. Holographic content can be created for display based on either the monitored or stored information.

[0008] In some embodiments, a user of a consumer device may interact with the holographic content, and the interaction may serve as an input for a holographic content creation system. For example, in some embodiments, some or all of the LF display system includes one or more ultrasonic speakers. The ultrasonic speakers may be configured to generate a tactile surface that matches at least a portion of the holographic content. A tracking system is configured to track the user's interaction with the holographic object (e.g., via images captured by an image sensor and / or some other camera of the LF display module). The LF display system is then configured to provide for creation of holographic content based on the interaction. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram of a light field display module for presenting holographic objects in accordance with one or more embodiments.

[0010] [Figure 2A] 1 is a cross-section of a portion of a light field display module according to one or more embodiments.

[0011] [Figure 2B]1 is a cross-section of a portion of a light field display module according to one or more embodiments.

[0012] [Figure 3A] FIG. 1 is a perspective view of a light field display module according to one or more embodiments.

[0013] [Figure 3B] 1 is a cross-sectional view of a light field display module including an interleaved energy relay device according to one or more embodiments.

[0014] [Figure 4A] FIG. 1 is a perspective view of a portion of a light field display system that is two-dimensionally tiled to form a single-sided seamless surface environment in accordance with one or more embodiments.

[0015] [Figure 4B] FIG. 1 is a perspective view of a portion of a light field display system in a multi-surface seamless surface environment in accordance with one or more embodiments.

[0016] [Figure 4C] FIG. 2 is a top view of a light field display system having a collective surface in a wing-type configuration in accordance with one or more embodiments.

[0017] [Figure 4D] FIG. 2 is a side view of a light field display system having a collection surface in a tilted configuration in accordance with one or more embodiments.

[0018] [Figure 4E] FIG. 2 is a top view of a light field display system having a collection surface on the front panel of the device in accordance with one or more embodiments.

[0019] [Diagram 5]FIG. 1 is a block diagram of a light field display system in accordance with one or more embodiments.

[0020] [Figure 6] FIG. 1 is a block diagram of a networked environment for a light field display system in a networked consumer device system in accordance with one or more embodiments.

[0021] [Figure 7] 1 is an example display of an embodiment using a light field display in a consumer device.

[0022] [Figure 8] 1 is a flowchart illustrating a method for enabling interaction with a consumer device using a light field display system in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] A light field (LF) display system is implemented in a consumer device to present holographic content, including at least one holographic object, to a user of the appliance. The terms "user", "appliance user", and "viewer" are used interchangeably herein. The terms "appliance", "consumer appliance", "consumer appliance device", "consumer device", and "device" are also used interchangeably herein and refer to electrical / machine tools that perform household chores such as cleaning, cooking, food preservation, and home assistants (e.g., Amazon Alexa™ or Google Home™). Examples include consumer devices such as refrigerators, stoves, washing machines, dryers, dishwashers, roti makers, pressure cookers, etc.

[0024] The LF display system of the appliance is integrated with the appliance's functionality to provide a user of the appliance with an immersive operation experience while using the appliance. The appliance's functionality is enabled using mechanical and electronic control systems, which may be enabled by executing computer program modules including various software client applications that allow a user of the appliance to operate the appliance. The LF display system comprises an LF display module configured to present holographic content including one or more holographic objects that would be visible to a viewer within a viewing volume of the appliance. The holographic content presented by the LF display system is integrated with the appliance's functionality. This enhances the immersive experience of the user of the appliance while operating the appliance. The holographic objects may also be augmented with other sensory stimuli (e.g., touch, sound, temperature, or smell). For example, an ultrasound emitter of the LF display system may project ultrasound pressure waves that create a volumetric tactile projection. The volumetric tactile projection provides a tactile surface that corresponds to some or all of the projected holographic object. The holographic content may include additional visual content (i.e., 2D or 3D visual content). Coordination of energy sources to enable a cohesive experience is part of the LF system for implementations using multiple energy sources (i.e., holographic objects that provide the correct haptic and sensory stimuli at any given time). For example, the LF system may include a controller for coordinating the presentation of holographic content and one or more haptic surfaces.

[0025] In some embodiments, an LF display system may include elements that enable the system to project at least one type of energy and simultaneously sense at least one type of energy to respond to the user and create an interactive experience. The sensed energy may be used to record how the user responds to the holographic content. For example, an LF display system may project both holographic objects for viewing and ultrasound for tactile perception, while simultaneously recording image information for user tracking and other scene analysis, while sensing ultrasound to detect touch responses by the user. As an example, such a system may project holographic objects that, when virtually "touched" by a user, modify their "behavior" according to the touch stimulus. The components of the display system that perform the energy sensing of the environment may be integrated into the display surface or may be dedicated sensors separate from the display surface, such as ultrasonic speakers and imaging capture devices such as cameras.

[0026] LF display systems can be built to provide a variety of experiences on consumer devices based on the generated holographic objects. For example, in one embodiment, an LF display system on a refrigerator can be implemented to present a user with a holographic view of the contents of the interior refrigerator shelves via a viewing application. The LF display assembly can form a single-sided or multi-sided seamless surface environment. The holographic content can extend or enhance a physical object, such as the appliance's display screen itself. For example, in one embodiment, an LF display system on a device can be implemented to present a holographic sign on the exterior surface of an oven, e.g., "Cake baking: Time until done: 12 minutes." Additionally, users can freely view the holographic content without the need for eyewear devices such as 3D glasses or other headgear.

[0027] In some embodiments, the LF display system may receive input via a tracking system and / or a sensory feedback assembly. Based on the input, the LF display system may adjust the holographic content as well as provide feedback to associated components.

[0028] The LF display system may also incorporate a system for tracking the movements of a user at least within the viewing volume of the LF display system. The tracked movements of the user may be used to enhance the user's experience when using an appliance. For example, the LF display system may use the tracking information to facilitate the user's interaction with holographic content (e.g., pressing a holographic button). The LF display system may use the tracking information to monitor the position of a finger relative to a holographic object. For example, the holographic object may be a button that may be "pressed" by a user. The LF display system may project ultrasonic energy to generate a tactile surface that corresponds to the button and occupies substantially the same space as the button. The LF display system may use the tracking information to dynamically move the button as well as dynamically move the position of the tactile surface as the button is "pressed" by the user. The LF display system may use the tracking information to render a holographic object that looks at the user and / or makes eye contact with the user or otherwise interacts with the user. The LF display system can use the tracking information to render holographic objects that are "touchable" to the user, and the ultrasonic speakers create a tactile surface that allows the holographic objects to interact with the user through touch.

[0029] Additionally, the LF display system may incorporate a user profiling system for identifying each user of the appliance in order to provide personalized content to each user of the appliance, and the user profiling system may further record other information regarding the user's particular use of the appliance that may be used for subsequent use to personalize the holographic content. Light Field Display System Overview

[0030] FIG. 1 is a diagram 100 of a light field (LF) display module 110 presenting a holographic object 120, according to one or more embodiments. The LF display module 110 is part of a light field (LF) display system. The LF display system presents holographic content, including at least one holographic object, using one or more LF display modules. The LF display system can present the holographic content to one or more users. In some embodiments, the LF display system can also augment the holographic content with other sensory content (e.g., touch, sound, smell, temperature, etc.). For example, as discussed below, the projection of ultrasound can generate hollow haptics that can simulate some or all of the surface of a holographic object. The LF display system includes one or more LF display modules 110 and is discussed in detail below with respect to FIGS. 2-8.

[0031] The LF display module 110 is a holographic display that presents a holographic object (e.g., holographic object 120) to one or more users (e.g., viewers 140). The LF display module 110 includes an energy device layer (e.g., a light-emitting electronic display or an acoustic projection device) and an energy waveguide layer (e.g., an optical lens array). Additionally, the LF display module 110 may include an energy relay layer to combine multiple energy sources or detectors together to form a single surface. At a high level, the energy device layer generates energy (e.g., holographic content), which is then directed to an area in space according to one or more four-dimensional (4D) light field functions using energy waveguides. The LF display module 110 may also simultaneously project and / or sense one or more types of energy. For example, the LF display module 110 may be able to project holographic images as well as ultrasound tactile surfaces into a viewing volume while simultaneously detecting image data from the viewing volume. The operation of the LF display module 110 is discussed in more detail below with respect to FIGS.

[0032] The LF display module 110 generates a holographic object within the holographic object volume 160 using one or more 4D light field functions (e.g., derived from a plenoptic function). The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Additionally, the holographic object may be multi-colored (e.g., full color). The holographic object 120 may be positioned anywhere within the holographic object volume 160. The holographic object may be projected in front of a screen plane, behind a screen plane, or segmented by a screen plane. The holographic object 120 may be presented to be perceived anywhere within the holographic object volume 160. The holographic object within the holographic object volume 160 may appear to the viewer 140 as floating in space.

[0033] The holographic object volume 160 represents a volume in which a holographic object can be perceived by the viewer 140. The holographic object volume 160 can extend in front of the display surface 150 (i.e., toward the viewer 140) so that the holographic object can be presented in front of the plane of the display area 150. Additionally, the holographic object volume 160 can extend behind the plane of the display area 150 (i.e., away from the viewer 140), allowing the holographic object to be presented as if it were behind the plane of the display area 150. In other words, the holographic object volume 160 can include all light rays that originate (e.g., projected) from the display area 150 and converge to create a holographic object. Here, the light rays can converge to a point in front of, at, or behind the display surface. In simpler terms, the holographic object volume 160 encompasses all volumes in which a holographic object can be perceived by the viewer.

[0034] The viewing volume 130 is a volume of space in which a holographic object (e.g., holographic object 120) presented in the holographic object volume 160 by the LF display system is fully visible. The holographic object may be presented in the holographic object volume 160 and viewed in the viewing volume 130 in a manner that is indistinguishable from the real object. The holographic object is formed by projecting the same light rays that would be generated from the surface of the object if it were physically present. Unlike stereoscopic displays, the observation position of the converging energy propagation paths in the holographic object volume 160 does not change as the viewer 140 moves around the viewing volume 130 (e.g., as discussed below with respect to FIG. 2). Instead, two people positioned in the holographic viewing volume 130 will see an appropriate view of the object as if they were actually there. Thus, the viewer 140 does not have to be constrained in the viewing volume. In contrast, conventional systems require the viewer to be constrained to a particular field of view (e.g., a chair in a fixed position relative to the display).

[0035] In some cases, the holographic object volume 160 and corresponding viewing volume 130 may be relatively small, such as designed for a single viewer. In other embodiments, for example, as described in more detail below with respect to Figures 4A and 4B, the LF display modules may be scaled and / or tiled to create larger holographic object volumes and corresponding viewing volumes that can accommodate a wide range of viewers (e.g., from one person to several thousand people). The LF display modules presented in this disclosure are constructed such that the entire surface of the module contains holographic imaging optics with no inactive or dead space and no need for bezels. In these embodiments, the LF display modules may be tiled such that the imaging area is continuous across the tile seams between the LF display modules, and the join lines between the tiled modules are substantially undetectable using eye vision.

[0036] The flexible size and / or shape of the viewing volume 130 allows the viewer 140 to be unconstrained within the viewing volume 130. For example, the viewer 140 can move to different positions within the viewing volume 130 and see different views of the holographic object 120 from corresponding viewpoints. For illustration, with reference to FIG. 1, the viewer 140 is in a first position relative to the holographic object 120 such that the holographic object 120 is seen in a head-on view of a holographic character, which may be a simulated avatar with which the user interacts. The viewer 140 may move to other locations relative to the holographic object 120 to see different views of the character. For example, the viewer 140 may move as if the viewer 140 were looking at a real character and change the viewer's relative position to the real person to see a different side of the person, see the left side of the character, the right side of the character, etc. In some embodiments, the holographic object 120 is visible to all viewers within the viewing volume 130 who have an unobstructed (i.e., unobstructed by an object / person) line of sight to the holographic object 120. These viewers may be unconstrained so that they can move around within the viewing volume to see different perspectives of the holographic object 120. Thus, the LF display system can present a holographic object such that multiple unconstrained viewers can simultaneously view the holographic object in real-world space from different perspectives, as if the holographic object were physically present.

[0037] In contrast, traditional displays (e.g., stereoscopic, virtual reality, augmented reality, or mixed reality) generally require each viewer to wear some external device (e.g., 3D glasses, near-eye display, or head-mounted display) to view the content. Additionally and / or alternatively, traditional displays may require the viewer to be constrained to a particular viewing position (e.g., in a chair with a fixed location relative to the display). For example, when viewing an object displayed by a stereoscopic display, the viewer always focuses on the display surface rather than the object, and the display always presents only two views of the object that follow the viewer as they try to move around the perceived object, causing distortions in the perception of the object. However, in light field displays, the viewer of a holographic object presented by an LF display system does not need to wear an external device to view the holographic object. The LF display system presents the holographic object in a way that appears to the viewer nearly the same as a physical object appears to the viewer, without requiring special eyewear, glasses, or head-mounted accessories. Furthermore, the viewer can view the holographic content from any location within the viewing volume.

[0038] In particular, the potential locations of holographic objects within the holographic object volume 160 are limited by the size of the volume. To increase the size of the holographic object volume 160, the size of the display area 150 of the LF display module 110 may be increased and / or multiple LF display modules may be tiled together to form a seamless viewing surface with an effective display area larger than the display area of ​​an individual LF display module. Some embodiments related to tiling of LF display modules are described below with respect to Figures 4A and 4B. As illustrated in Figure 1, the display area 150 is rectangular, resulting in a holographic object volume 160 that is pyramidal. In other embodiments, the display area may have some other shape (e.g., hexagonal) that also affects the shape of the corresponding viewing volume.

[0039] Additionally, while the above discussion has focused on presenting holographic object 120 within a portion of holographic object volume 160 that is between LF display module 110 and viewer 140, LF display module 110 can additionally present content that is visible behind the plane of display area 150. For example, LF display module 110 can make display area 150 appear to be a room with holographic object 120 in it. The displayed content may also be such that viewer 140 can see other objects in the room through the displayed surface. Furthermore, LF display systems can generate content that moves seamlessly around holographic object volume 160, including behind and in front of the plane of display surface 150.

[0040] 2A is a cross-section 200 of a portion of an LF display module 210, according to one or more embodiments. The LF display module 210 may be the LF display module 110. In other embodiments, the LF display module 210 may be another LF display module having a different display area shape than the display area 150. In an exemplary embodiment, the LF display module 210 includes an energy device layer 220, an energy relay layer 230, and an energy waveguide layer 240. Some embodiments of the LF display module 210 have different components than those described herein. For example, in some embodiments, the LF display module 210 does not include the energy relay layer 230. Similarly, functionality may be distributed among components in a different manner than described herein.

[0041] The display system described herein presents an emission of energy that replicates the energy that normally surrounds real-world objects, where the emitted energy is directed in a specific direction from every coordinate on the display surface. The directed energy from the display surface allows for the convergence of many energy rays, thereby creating a virtual object. For example, in visible light, the LF display projects an extremely large number of light rays that converge at any point within the holographic object volume, so that from the perspective of a viewer located farther away than the object being projected, those light rays will appear to come from the surface of a real-world object located in this region of space. Thus, from the viewer's perspective, the LF display is generating reflected light rays that appear to come from the surface of such an object. The viewer's viewpoint may change on any particular virtual object, and the viewer will see different views of that virtual object.

[0042] The energy device layer 220 includes one or more electronic displays (e.g., light-emitting displays such as OLEDs) and one or more other energy projection and / or energy receiving devices described herein. The one or more electronic displays are configured to display content according to display instructions (e.g., from a controller of the LF display system). The one or more electronic displays include multiple pixels, each having an intensity that is individually controlled. Many types of commercial displays can be used within the LF display, such as light-emitting LED and OLED displays.

[0043] The energy device layer 220 may also include one or more acoustic projection devices and / or one or more acoustic receiving devices. The acoustic emission devices generate one or more pressure waves that complement the holographic object 250. The generated pressure waves may be, for example, audible, ultrasonic, or some combination thereof. The ultrasonic pressure waves are used for volumetric haptics (e.g., at the surface of the holographic object 250). The audible pressure waves are used to provide audio content (e.g., immersive audio) that may complement the holographic object 250. For example, assuming the holographic object 250 is a character, one or more acoustic projection devices may be used to (1) generate a tactile surface juxtaposed with the surface of the presented character so that the viewer may touch the holographic object 250, and / or (2) provide audio content, possibly corresponding to an audio greeting from the character. The acoustic receiving device (e.g., a microphone or microphone array) may be configured to monitor ultrasonic and / or audible pressure waves within a localized area of ​​the LF display module 210.

[0044] The energy device layer 220 may also include one or more image sensors. The image sensors are sensitive to light in at least the visible light band, and may possibly be sensitive to light in other bands (e.g., infrared). The image sensors may be, for example, a complementary metal oxide semiconductor (CMOS) array, a charge-coupled device (CCD), an array of photodetectors, some other sensor that captures light, or some combination thereof. The LF display system may use data captured by the one or more image sensors for viewer location tracking.

[0045] The energy relay layer 230 relays energy (e.g., electromagnetic, mechanical pressure waves) between the energy device layer 220 and the energy waveguide layer 240. The energy relay layer 230 includes one or more energy relay elements 260. Each energy relay element includes a first surface 265 and a second surface 270 and relays energy between the two surfaces. The first surface 265 of each energy relay element may be coupled to one or more energy devices (e.g., electronic displays or acoustic projection devices). The energy relay elements may be composed of, for example, glass, carbon, optical fiber, optical films, plastics, polymers, or some combination thereof. Additionally, in some embodiments, the energy relay elements may adjust the magnification (increase or decrease) of the energy passing between the first surface 265 and the second surface 270. If the relay provides magnification, the relay may take the form of an array of bonded tapered relays, called tapers, where one end of the taper may be substantially larger in area than the opposite end. The larger ends of the tapers can be plastered together to form a seamless energy surface 275. One advantage is that space is created at the smaller ends of each taper to accommodate the mechanical envelope of multiple energy sources, such as the bezels of multiple displays. This extra space allows energy sources to be placed side-by-side on the smaller taper side, with each energy source having an active area within the smaller taper surface that directs energy and is relayed to the larger seamless energy surface. Another advantage of using a taper relay is that the combined seamless energy surface formed by the larger ends of the tapers has no non-imaging dead space. Because there are no borders or bezels, the seamless energy surfaces can be tiled together to form a larger surface that is substantially seamless according to the visual acuity of the eye.

[0046] The second surfaces of adjacent energy relay elements together form an energy plane 275. In some embodiments, the separation between the edges of adjacent energy relay elements is less than the minimum perceptible contour defined by the visual acuity of a human eye having better than 20 / 40 vision, such that the energy plane 275 is virtually seamless from the perspective of a viewer 280 within the viewing volume 285.

[0047] In some embodiments, one or more energy relay elements exhibit energy localization, such that the efficiency of energy transport in a longitudinal direction substantially perpendicular to surfaces 265 and 270 is much higher than the efficiency of transport in a perpendicular transverse plane, and the energy density is highly localized in this transverse plane as an energy wave propagates between surfaces 265 and 270. This energy localization allows energy distributions, such as images, to be efficiently relayed between these surfaces without significant loss of resolution.

[0048] The energy waveguide layer 240 uses waveguide elements in the energy waveguide layer 240 to direct energy from locations (e.g., coordinates) on the energy plane 275 into specific propagation paths from the display surface outward into the holographic viewing volume 285. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 direct light from locations on the seamless energy plane 275 along different propagation directions through the viewing volume 285. In various examples, the light is directed according to a 4D light field function to form the holographic object 250 within the holographic object volume 255.

[0049] The energy waveguide layer 240 uses waveguide elements in the energy waveguide layer 240 to direct energy from locations (e.g., coordinates) on the energy plane 275 into specific propagation paths from the display surface outward into the holographic viewing volume 285. As an example, for electromagnetic energy, the waveguide elements in the energy waveguide layer 240 direct light from locations on the seamless energy plane 275 along different propagation directions through the viewing volume 285. In various examples, the light is directed according to a 4D light field function to form the holographic object 250 within the holographic object volume 255.

[0050] Each waveguide element in the energy waveguide layer 240 may be, for example, a lenslet composed of one or more elements. In some configurations, the lenslet may be a positive lens. The positive lens may have a spherical, aspheric, or freeform surface profile. Additionally, in some embodiments, some or all of the waveguide elements may include one or more additional optical components. The additional optical components may be, for example, energy suppression structures such as baffles, positive lenses, negative lenses, spherical lenses, aspheric lenses, freeform lenses, liquid crystal lenses, liquid lenses, refractive elements, diffractive elements, or some combination thereof. In some embodiments, at least one of the lenslets and / or additional optical components may also dynamically adjust refractive power. For example, the lenslet may be a liquid crystal lens or a liquid lens. The dynamic adjustment of the surface profile, the lenslet and / or the at least one additional optical component may provide additional directional control of the light projected from the waveguide element.

[0051] In an illustrative example, the holographic object volume 255 of the LF display has a boundary formed by light rays 256 and 257, but may be formed by other light rays. The holographic object volume 255 is a continuous volume that extends both in front of the energy waveguide layer 240 (i.e., toward the viewer 280) and behind it (i.e., away from the viewer 280). In an illustrative example, light rays 256 and 257 are projected from opposite edges of the LF display module 210 at the largest angle relative to the normal of the display surface 277 that can be perceived by the user, but they may be other projected light rays. These light rays define the field of view of the display and therefore the boundary of the holographic viewing volume 285. In some cases, these light rays define a holographic viewing volume (e.g., an ideal viewing volume) where the entire display may be observed without vignetting. As the field of view of the display increases, the convergence point of light rays 256 and 257 becomes closer to the display. Thus, a display with a wider field of view allows viewer 280 to view the entire display at a closer viewing distance. Additionally, light rays 256 and 257 may form an ideal holographic object volume. A holographic object presented within the ideal holographic object volume may be viewed anywhere within viewing volume 285.

[0052] In some examples, the holographic object may be presented in only a portion of the viewing volume 285. In other words, the holographic object volume may be divided into any number of viewing sub-volumes (e.g., viewing sub-volume 290). Additionally, the holographic object may be projected outside the holographic object volume 255. For example, the holographic object 251 is presented outside the holographic object volume 255. Because the holographic object 251 is presented outside the holographic object volume 255, it is not visible from all locations within the viewing volume 285. For example, the holographic object 251 may be visible from locations within the viewing sub-volume 290, but not from locations of the viewer 280.

[0053] For example, to illustrate viewing holographic content from different viewing sub-volumes, turn to FIG. 2B. FIG. 2B illustrates a cross section 200 of a portion of an LF display module, according to one or more embodiments. The cross section of FIG. 2B is the same as the cross section of FIG. 2A. However, FIG. 2B shows a different set of light rays projected from the LF display module 210. Light rays 256 and light rays 257 still form the holographic object volume 255 and the viewing volume 285. However, as shown, the light rays projected from the top of the LF display module 210 and the bottom of the LF display module 210 overlap to form various viewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D) within the viewing volume 285. A viewer in a first viewing sub-volume (e.g., 290A) may be able to perceive holographic content presented in the holographic object volume 255 that is not perceivable by viewers in the other viewing sub-volumes (e.g., 290B, 290C, and 290D).

[0054] More simply, as illustrated in FIG. 2A, holographic object volume 255 is a volume in which a holographic object may be presented by the LF display system such that the holographic object may be perceived by a viewer (e.g., viewer 280) in viewing volume 285. Thus, viewing volume 285 is an example of an ideal viewing volume, and holographic object volume 255 is an example of an ideal object volume. However, in various configurations, a viewer may perceive a holographic object presented by LF display system 200 in another example holographic object volume, just as a viewer in another example viewing volume may perceive holographic content. More generally, "eyeline guidelines" apply when viewing holographic content projected from an LF display module. Eyeline guidelines assert that a line formed by the viewer's eye position and the holographic object being viewed must intersect the LF display surface.

[0055] When viewing the holographic content presented by the LF display module 210, each eye of the viewer 280 will see a different perspective of the holographic object 250 because the holographic content is presented according to a 4D light field function. Furthermore, as the viewer 280 moves within the viewing volume 285, the viewer will see different perspectives of the holographic object 250 as other viewers within the viewing volume 285 would see. As will be appreciated by those skilled in the art, 4D light field functions are well known in the art and will not be described in further detail herein.

[0056] As described in more detail herein, in some embodiments, the LF display can project more than one type of energy. For example, the LF display can project two types of energy, such as, for example, mechanical and electromagnetic energy. In this configuration, the energy relay layer 230 includes two separate energy relays that are interleaved together at the energy plane 275, but are separated such that the energy is relayed to two different energy device layers 220. Here, one relay can be configured to transport electromagnetic energy and another relay can be configured to transport mechanical energy. In some embodiments, the mechanical energy can be projected from a location between the electromagnetic waveguide elements on the energy waveguide layer 240, which helps to form a structure that inhibits light from being transported from one electromagnetic waveguide element to another. In some embodiments, the energy waveguide layer 240 can also include waveguide elements that transport focused ultrasound along a specific propagation path according to display instructions from the controller.

[0057] Note that in an alternative embodiment (not shown), the LF display module 210 does not include the energy relay layer 230. In this case, the energy plane 275 is an output plane formed using one or more adjacent electronic displays in the energy device layer 220. Also, in some embodiments, the separation between the edges of adjacent electronic displays is less than the smallest perceptible contour defined by the visual acuity of a human eye with 20 / 40 vision, such that the energy plane is virtually seamless from the perspective of a viewer 280 within the viewing volume 285. LF Display Module

[0058] FIG. 3A is a perspective view of an LF display module 300A according to one or more embodiments. The LF display module 300A may be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 300A may be some other LF display module. In an exemplary embodiment, the LF display module 300A includes an energy device layer 310, an energy relay layer 320, and an energy waveguide layer 330. The LF display module 300A is configured to present holographic content from a display surface 365 as described herein. For convenience, the display surface 365 is illustrated as a dashed outline on the frame 390 of the LF display module 300A, but more precisely, it is the surface directly in front of the waveguide element bounded by the inner rim of the frame 390. Some embodiments of the LF display module 300A have different components than those described herein. For example, in some embodiments, the LF display module 300A does not include the energy relay layer 320. Similarly, functionality may be distributed between components in ways different from those described herein.

[0059] Energy device layer 310 is one embodiment of energy device layer 220. Energy device layer 310 includes four energy devices 340 (three are visible in the figure). The energy devices 340 may all be of the same type (e.g., all electronic displays) or may include one or more different types (e.g., including an electronic display and at least one acoustic energy device).

[0060] The energy relay layer 320 is one embodiment of the energy relay layer 230. The energy relay layer 320 includes four energy relay elements 350 (three are visible in the figure). The energy relay devices 350 may all relay the same type of energy (e.g., light) or may relay one or more different types (e.g., light and sound). Each of the relay elements 350 includes a first surface and a second surface, where the second surfaces of the energy relay elements 350 are arranged to form a single seamless energy surface 360. In an exemplary embodiment, each of the energy relay elements 350 is tapered such that the first surface has a smaller surface area than the second surface, which allows the mechanical envelope of the energy device 340 to be housed at the smaller end of the taper. This allows the seamless energy surface to be borderless, since the entire area may project energy. This means that this seamless energy surface can be tiled by placing multiple instances of 300 together without dead space or bezels, such that the entire combined surface is seamless. In other embodiments, the first surface and the second surface have the same surface area.

[0061] The energy waveguide layer 330 is one embodiment of the energy waveguide layer 240. The energy waveguide layer 330 includes a plurality of waveguide elements 370. As discussed above with respect to FIG. 2, the energy waveguide layer 330 is configured to direct energy from a seamless energy plane 360 ​​along specific propagation paths according to a 4D plenoptic function to form a holographic object. Note that in an exemplary embodiment, the energy waveguide layer 330 is bounded by a frame 390. In other embodiments, the frame 390 is not present and / or the thickness of the frame 390 is reduced. Removal or reduction of the thickness of the frame 390 may facilitate tiling of the LF display module 300A with additional LF display modules 300.

[0062] Note that in the exemplary embodiment, the seamless energy surface 360 ​​and the energy waveguide layer 330 are planar. In alternative embodiments not shown, the seamless energy surface 360 ​​and the energy waveguide layer 330 may be curved in one or more dimensions.

[0063] The LF display module 300A may be configured with additional energy sources present on the surface of the seamless energy surface, allowing the projection of energy fields in addition to the light field. In one embodiment, an acoustic energy field may be projected from an electrostatic speaker (not illustrated) mounted at any number of locations on the seamless energy surface 360. Additionally, the electrostatic speaker of the LF display module 300A is positioned within the light field display module 300A such that the dual energy surface simultaneously projects a sound field and holographic content. For example, the electrostatic speaker may be formed of one or more diaphragm elements that are transparent to some wavelengths of electromagnetic energy and driven with conductive elements. The electrostatic speaker may be mounted on the seamless energy surface 360 ​​such that the diaphragm elements cover some of the waveguide elements. The conductive electrodes of the speaker may be collocated with a structure designed to suppress light transmission between the electromagnetic waveguides and / or may be located at a location between the electromagnetic waveguide elements (e.g., frame 390). In various configurations, the speakers can project multiple sources of focused ultrasonic energy that produce audible sound and / or a tactile surface.

[0064] In some configurations, the energy devices 340 may sense energy. For example, the energy devices may be microphones, light sensors, acoustic transducers, etc. Thus, the energy relay devices may also relay energy from the seamless energy surface 360 ​​to the energy device layer 310. That is, the seamless energy surface 360 ​​of the LF display module forms a bidirectional energy surface when the energy devices and the energy relay devices 340 are configured to simultaneously emit and sense energy (e.g., emit a light field and sense sound).

[0065] More broadly, the energy device 340 of the LF display module 300A can be either an energy source or an energy sensor. The LF display module 300A can include various types of energy devices that function as energy sources and / or energy sensors to facilitate the projection of high quality holographic content to a user. Other sources and / or sensors include thermal sensors or sources, infrared sensors or sources, image sensors or sources, mechanical energy transducers that generate acoustic energy, feedback sources, and the like. Many other sensors or sources are possible. Additionally, the LF display modules can be tiled such that the LF display modules can form an assembly that projects and senses multiple types of energy from a large aggregate seamless energy surface.

[0066] In various embodiments of the LF display module 300A, the seamless energy surface 360 ​​may have various surface portions, with each surface portion configured to project and / or emit a particular type of energy. For example, if the seamless energy surface is a dual energy surface, the seamless energy surface 360 ​​includes one or more surface portions that project electromagnetic energy and one or more other surface portions that project ultrasonic energy. The surface portions that project ultrasonic energy may be located on the seamless energy surface 360 ​​between electromagnetic waveguide elements and / or may be co-located with structures designed to inhibit light transmission between the electromagnetic waveguide elements. In examples where the seamless energy surface is a bidirectional energy surface, the energy relay layer 320 may include two types of energy relay devices interleaved in the seamless energy surface 360. In various embodiments, the seamless energy surface 360 ​​may be configured such that the portions of the surface under a particular waveguide element 370 are all energy sources, all energy sensors, or a mix of energy sources and energy sensors.

[0067] 3B is a cross-sectional view of an LF display module 300B including an interleaved energy relay device, according to one or more embodiments. The LF display module 300B can be configured either as a dual energy projection device for projecting one or more types of energy, or as a bi-directional energy device for simultaneously projecting one type of energy and sensing another type of energy. The LF display module 300B can be the LF display module 110 and / or the LF display module 210. In other embodiments, the LF display module 302 can be some other LF display module.

[0068] The LF display module 300B includes many components configured similarly to those of the LF display module 300A of FIG. 3A. For example, in an exemplary embodiment, the LF display module 300B includes an energy device layer 310, an energy relay layer 320, a seamless energy surface 360, and an energy waveguide layer 330 that include at least the same functionality as described with respect to FIG. 3A. Additionally, the LF display module 300B presents and / or receives energy from a display surface 365. In particular, the components of the LF display module 300B are alternatively connected and / or oriented differently than the components of the LF display module 300A of FIG. 3A. Some embodiments of the LF display module 300B have different components than those described herein. Similarly, functionality may be distributed among the components in a different manner than described herein. FIG. 3B illustrates a design of a single LF display module 302 that can be tiled to create a dual energy projection surface or a bidirectional energy surface with a larger area.

[0069] In one embodiment, the LF display module 300B is an LF display module of a bidirectional LF display system. The bidirectional LF display system can project energy and simultaneously sense energy from a display surface 365. The seamless energy surface 360 ​​includes both energy projection locations and energy sensing locations closely interleaved on the seamless energy surface 360. Thus, in the example of FIG. 3B, the energy relay layer 320 is configured in a different manner than the energy relay layer of FIG. 3A. For convenience, the energy relay layer of the LF display module 300B is referred to herein as an "interleaved energy relay layer."

[0070] The interleaved energy relay layer 320 includes two legs of a first energy relay device 350A and a second energy relay device 350B. Each of the legs is illustrated as a lightly shaded area. Each of the legs is made of a flexible relay material and can be formed with a length sufficient for use with energy devices of various sizes and shapes. In some areas of the interleaved energy relay layer, the two legs are tightly interleaved together as they approach a seamless energy surface 360. In the illustrative example, the interleaved energy relay device 352 is illustrated as a darkly shaded area.

[0071] The energy relay devices are configured to relay energy between different energy devices while being interleaved in a seamless energy plane 360. The energy devices are in the energy device layer 310. As illustrated, energy device 340A is connected to energy relay device 350A and energy device 340B is connected to energy relay device 350B. In various embodiments, each energy device may be an energy source or an energy sensor.

[0072] The energy waveguide layer 330 includes a waveguide element 370 for directing energy waves from the seamless energy surface 360 ​​along a projected path toward a series of convergence points. In this example, a holographic object 380 is formed at the series of convergence points. Notably, as illustrated, the convergence of energy at the holographic object 380 occurs on the viewer side of the display surface 365. However, in other examples, the convergence of energy may be anywhere within the holographic object volume that extends both in front of the display surface 365 and behind the display surface 365. The waveguide element 370 can simultaneously direct the incoming energy to an energy device (e.g., an energy sensor), as described below.

[0073] In one exemplary embodiment of the LF display module 300B, a light-emitting display is used as the energy source and an image sensor is used as the energy sensor. In this manner, the LF display module 300B can simultaneously project holographic content and detect light from the volume in front of the display surface 365.

[0074] In one embodiment, the LF display module 300B is configured to project a light field from a projection location of the display surface in front of the display surface 365 while simultaneously capturing a light field from in front of the display surface 365. In this embodiment, the energy relay device 350A connects a first set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340A. In one example, the energy device 340A is a light emitting display having an array of source pixels. The energy relay device 340B connects a second set of locations in the seamless energy surface 360 ​​positioned under the waveguide element 370 to the energy device 340B. In one example, the energy device 340B is an image sensor having an array of sensor pixels. The LF display module 302 can be configured such that the locations in the seamless energy surface 365 under a particular waveguide element 370 are all light emitting display locations, all image sensor locations, or some combination of locations. In other embodiments, the bidirectional energy surface can project and receive various other forms of energy.

[0075] In another exemplary embodiment of the LF display module 300B, the LF display module is configured to project two different types of energy. For example, the energy device 340A is a light-emitting display configured to emit electromagnetic energy, and the energy device 340B is an ultrasonic transducer configured to emit mechanical energy. Thus, both light and sound can be projected from various locations on the seamless energy surface 360. In this configuration, the energy relay device 350A connects the energy device 340A to the seamless energy surface 360 ​​and relays the electromagnetic energy. The energy relay device is configured to have properties that make the transport of electromagnetic energy efficient (e.g., change the refractive index). The energy relay device 350B connects the energy device 340B to the seamless energy surface 360 ​​and relays the mechanical energy. The energy relay device 350B is configured to have properties for efficient transport of ultrasonic energy (e.g., distribution of materials with different acoustic impedances). In some embodiments, the mechanical energy can be projected from locations between the waveguide elements 370 on the energy waveguide layer 330. The locations of projecting mechanical energy may form structures that serve to inhibit light from being transported from one electromagnetic waveguide element to another. In one example, a spatially separated array of locations of projecting ultrasonic mechanical energy may be configured to create three-dimensional haptic shapes and surfaces in air. The surfaces may coincide with a projected holographic object (e.g., holographic object 380). In some examples, phase delays and amplitude variations across the array may assist in creating haptic shapes.

[0076] In various embodiments, the interactive LF display module 302 may include multiple energy device layers, with each energy device layer including a particular type of energy device. In these examples, the energy relay layer is configured to relay the appropriate type of energy between the seamless energy plane 360 ​​and the energy device layer 330. Tiled LF Display Module

[0077] 4A is a perspective view of a portion of an LF display system 400 that is two-dimensionally tiled to form a single-sided seamless surface environment, according to one or more embodiments. The LF display system 400 includes a plurality of LF display modules that are tiled to form an array 410. More specifically, each small square in the array 410 represents a tiled LF display module 412. The array 410 may, for example, cover some or all of a surface.

[0078] Array 410 can project one or more holographic objects. For example, in an exemplary embodiment, array 410 projects holographic object 420 and holographic object 430. Tiling of the LF display modules not only allows for a much larger viewing volume, but also allows objects to be projected farther away from array 410. For example, in an exemplary embodiment, the viewing volume is approximately the entire area in front of and behind array 410, rather than a localized volume in front of (and behind) LF display module 412.

[0079] In some embodiments, the LF display system 400 presents the holographic object 420 to a viewer 430 and a viewer 434. The viewer 430 and the viewer 434 receive different perspectives of the holographic object 420. For example, the viewer 430 is presented with a straight-on view of the holographic object 420, while the viewer 434 is presented with a more oblique view of the holographic object 430. As the viewer 430 and / or the viewer 434 move, they are presented with different perspectives of the holographic object 420. This allows the viewer to visually interact with the holographic object by moving relative to the holographic object. For example, as the viewer 430 walks around the holographic object 420, the viewer 430 sees different sides of the holographic object 420, as long as the holographic object 420 remains within the holographic object volume of the array 410. Thus, viewers 430 and 434 can simultaneously view holographic object 420 in real-world space as if holographic object 420 were actually there. Additionally, viewers 440 and 450 of holographic object 420 do not need to wear external devices to view holographic object 420, since holographic object 420 appears to the viewers in much the same way that a physical object appears. Additionally, holographic object 422 is illustrated here behind the array, since the viewing volume of the array extends behind the surface of the array. In this way, holographic object 422 can be presented to viewers 430 and / or 434 as if it were farther away from the viewers than the surface of array 410.

[0080] In some embodiments, LF display system 400 may include a tracking system that tracks the positions of viewer 430 and viewer 435. In some embodiments, the tracked positions are generally the positions of the viewers. In other embodiments, the tracked positions are the positions of the viewers' eyes. Eye position tracking differs from gaze tracking, which tracks where the eyes are looking (e.g., using orientation to determine gaze location). The eyes of viewer 440 and the eyes of viewer 450 are in different locations.

[0081] In various configurations, the LF display system 400 may include one or more tracking systems. For example, in the exemplary embodiment of FIG. 4A, the LF display system includes a tracking system 440 that is external to the array 410. Here, the tracking system may be a camera system coupled to the array 410. The external tracking system is described in more detail in association with FIG. 5A. In other exemplary embodiments, the tracking system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340) of an LF display module 412 included in the array 410 may be configured to capture an image of a viewer in front of the array 440. In either case, the tracking system of the LF display system 400 determines tracking information regarding a viewer (e.g., viewer 430 and / or viewer 434) viewing the holographic content presented by the array 410.

[0082] The tracking information represents the position of the viewer or the position of a portion of the viewer (e.g., one or both eyes of the viewer, or a limb of the viewer) in space (e.g., relative to the tracking system). The tracking system can use any number of depth determination techniques to determine the tracking information. The depth determination techniques can include, for example, structured light, time of flight, stereo imaging, some other depth determination technique, or some combination thereof. The tracking system can include one or more infrared sources (e.g., structured light sources), one or more image sensors capable of capturing images in infrared (e.g., red-blue-green-infrared cameras), and a tracking module. The tracking system can determine the viewer's position using depth estimation techniques. In some embodiments, the LF display system 400 generates a holographic object based on the tracked positions of the viewer 440 and / or the viewer 450 as described herein. For example, the LF display system 400 can generate a holographic object in response to a viewer coming within a threshold distance and / or a specific position of the array 410.

[0083] LF display system 400 can present one or more holographic objects customized for each viewer based in part on the tracking information. For example, viewer 430 can be presented with holographic object 420, but not with holographic object 422. Viewer 434 can be presented with holographic object 422, but not with holographic object 420. LF display system 400 tracks the position of each of viewer 440 and viewer 450. LF display system 400 determines the viewpoint of the holographic object that should be seen by the viewer based on the position relative to where the holographic object is presented. LF display system 400 selectively projects light from specific pixels that correspond to the determined viewpoint. Thus, viewer 434 and viewer 430 can potentially have completely different experiences simultaneously. In other words, LF display system 400 can present holographic content to viewing sub-volumes of the viewing volume. For example, as shown in the figure, the viewing volume is represented as all the space before and after the array. In this example, LF display system 400 can track the positions of viewer 430 and viewer 434 so that LF display system 400 can present two different holographic interfaces to the two viewers for operating an appliance. Thus, LF display system 400 can project a "rotatable dial" (e.g., holographic object 420) into a viewing sub-volume surrounding viewer 430 and project a "set of pressable buttons" content (e.g., holographic object 422) into a viewing sub-volume surrounding viewer 434. In contrast, conventional systems would have to use individual headsets to provide such a customized experience.

[0084] In some embodiments, the LF display system 400 may include one or more sensory feedback systems. The sensory feedback systems provide other sensory stimuli (e.g., touch, sound, temperature, and / or smell) that enhance the holographic objects 420 and 422. For example, in the exemplary embodiment of FIG. 4A, the LF display system 400 includes a sensory feedback system 442 external to the array 410. In one example, the sensory feedback system 442 may be an electrostatic speaker coupled to the array 410. The external sensory feedback system is described in more detail in connection with FIG. 5A. In other exemplary embodiments, the sensory feedback system may be incorporated into the array 410, as described herein. For example, an energy device (e.g., energy device 340A of FIG. 3B) of the LF display module 412 included in the array 410 may be configured to project ultrasonic energy to a viewer in front of the array and / or receive image information from a viewer in front of the array. In either case, the sensory feedback system presents sensory content to and / or receives sensory content from a viewer (e.g., viewer 430 and / or viewer 434) viewing holographic content presented by array 410 (e.g., holographic object 420 and / or holographic object 422).

[0085] The LF display system 400 may include a sensory feedback system including one or more acoustic projection devices external to the array. Alternatively or additionally, the LF display system 400 may include one or more acoustic projection devices integrated into the array 410 as described herein. The acoustic projection devices may project ultrasonic pressure waves that generate volumetric haptics to one or more surfaces of the holographic object (e.g., at the surface of the holographic object 420) when a portion of the viewer comes within a threshold distance of the one or more surfaces. The volumetric haptics allow the user to touch and feel the surface of the holographic object. The multiple acoustic projection devices may project audible pressure waves that provide audio content (e.g., immersive audio) to the viewer. Thus, the ultrasonic pressure waves and / or audible pressure waves may serve to complement the presented holographic object by bringing the volumetric haptic surface within a threshold distance of the surface of the holographic object or matching the surface of the presented holographic object. In some embodiments, the resistance of the generated volumetric haptic surface to contact by the user may be based on the value of a parameter determined in the LF display system 400. In some embodiments, a volumetric haptic surface may be created, modified, updated, or deleted based, at least in part, on information obtained from tracking system 440 of LF display system 400. In some embodiments, the LF display system used information obtained from tracking system 440 in combination with artificial intelligence models to generate the volumetric haptic surface.

[0086] In various embodiments, the LF display system 400 can provide other sensory stimuli based in part on the tracked position of the viewer. For example, the holographic object 422 illustrated in FIG. 4A is a rotatable dial, and the LF display system 400 can rotate the holographic object 422 both visually (i.e., the holographic object 422 appears to rotate) and audibly (i.e., one or more audio projection devices emit sounds that the viewer 440 perceives as clicking rotation sounds originating from the holographic object 430).

[0087] Note that in the example configuration, the holographic viewing volume may be limited in a manner similar to viewing volume 285 of configuration 200. This can limit the amount of perceived immersion a viewer will experience with a single display unit. One way to address this is to use multiple LF display modules tiled along multiple sides of the appliance, as described below with respect to FIG. 4B.

[0088] 4B is a perspective view of a portion of a LF display system 402 in a multi-sided seamless surface environment, according to one or more embodiments. The LF display system 402 is substantially similar to the LF display system 400, except that multiple LF display modules are tiled to create a multi-sided seamless surface environment. More specifically, the LF display modules are tiled to form an array that is a six-sided collective seamless surface environment. In other embodiments, multiple LF display modules may cover some, but not all, of a surface or some combination thereof. In other embodiments, multiple LF display modules are tiled to form some other collective seamless surface. For example, the surface may be curved such that a cylindrical collective energy environment is formed.

[0089] The LF display system 402 can project one or more holographic objects. For example, in an exemplary embodiment, the LF display system 402 projects the holographic object 420 into an area surrounded by the six-sided aggregated seamless surface environment. Thus, the viewing volume of the LF display system is also contained within the six-sided aggregated seamless surface environment. Note that in an exemplary configuration, the viewer 432 can be positioned between the holographic object 420 and the LF display module 414 that is projecting the energy (e.g., light and / or pressure waves) used to form the holographic object 420. Thus, the positioning of the viewer 434 may prevent the viewer 430 from perceiving the holographic object 420 formed from the energy from the LF display module 414. However, in the exemplary configuration, there is at least one other LF display module, such as, for example, the LF display module 416, that is not obstructed (e.g., by the viewer 434) and can project energy to form the holographic object 420. Thus, while viewer occlusion in the space may obscure parts of the holographic projection, this effect is much smaller than if a holographic display panel were present on only one side of the volume. Holographic object 422 is illustrated "outside" the enclosure of the six-sided aggregated seamless surface environment because the holographic object volume extends behind the aggregated surfaces. Viewer 430 and / or viewer 434 may thus perceive holographic object 422 as "outside" the six-sided environment that the viewer may move throughout.

[0090] As discussed above with reference to Figure 4A, in some embodiments, the LF display system 402 may actively track the viewer's position and dynamically instruct different LF display modules to present holographic content based on the tracked position. Thus, the multi-surface configuration may provide a more robust environment (e.g., compared to Figure 4A) for presenting holographic objects, where an unconstrained viewer may freely move throughout an area surrounded by the multi-surface seamless surface environment.

[0091] In particular, various LF display systems may have different configurations. Furthermore, each configuration may have a particular orientation of the surfaces that together form a seamless display surface ("aggregate surface"). That is, the LF display modules of an LF display system may be tiled to form various aggregate surfaces. For example, in FIG. 4B, LF display system 402 includes LF display modules that are tiled to form a six-sided aggregate surface. In some other examples, the aggregate surface may occur only on a portion of the surface, rather than the entire surface. Some examples are described herein.

[0092] In some configurations, a collective surface of an LF display system may include a collective surface configured to project energy toward a localized viewing volume, which may enable a higher quality viewing experience, for example, by increasing the density of projected energy within a particular viewing volume, widening the FOV of a viewer within that volume, and moving the viewing volume closer to the display surface.

[0093] For example, FIG. 4C shows a top view of an LF display system 450A having a collective surface in a "wing-shaped" configuration. The LF display system 450A includes LF display modules tiled to form a collective surface 460. The LF display system 450A has a collective surface 460 that includes three portions: (i) a first central portion 462, (ii) a second portion 464 connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front (i.e., the first side) of the central portion, and (iii) a third portion 466 connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front (i.e., the second side) of the central portion. When a mobile device is held such that the first central portion is in a vertical plane having a horizontal plane and a vertical axis, the second and third portions are angled toward the front of the central portion along the horizontal axis.

[0094] In this example, the viewing volume 468A of the LF display system 450A is in front of the device and is partially surrounded by three portions of the collecting surface 460. The collecting surfaces that at least partially surround the viewer (the "surrounding surfaces") increase the viewer's immersive experience.

[0095] For illustration, consider, for example, a collective surface having only a central surface. Referring to FIG. 2A, rays projected from either edge of the display surface will create an ideal holographic volume and an ideal viewing volume as described above. Now consider, for example, the case where the central surface includes two side surfaces angled toward the viewer. In this case, rays 256 and 257 will be projected at a larger angle from the normal of the central surface. Thus, the field of view of the viewing volume will be enlarged. Similarly, the holographic viewing volume will be closer to the display surface. Additionally, a holographic object projected at a certain distance from the display surface will be closer to its viewing volume because the second and third two parts are tilted closer to the viewing volume.

[0096] Simply put, a display surface with only a central surface has a planar field of view, a planar threshold separation between the (central) display surface and the viewing volume, and a planar proximity between the holographic object and the viewing volume. Adding one or more side surfaces angled toward the viewer increases the field of view relative to the planar field, decreases the separation between the display surface and the viewing volume relative to the planar separation, and increases the proximity between the display surface and the holographic object relative to the planar proximity. Angling the side surfaces further toward the viewer further increases the field of view, decreases the separation, and increases the proximity. In other words, the angled arrangement of the side surfaces increases the immersive experience of the viewer.

[0097] Additionally, as described below with respect to FIG. 6, deflection optics can be used to optimize the size and position of the viewing volume for LF viewing parameters (eg, dimensions and FOV).

[0098] Returning to FIG. 4D, in a similar example, FIG. 4D shows a top view of an LF display system 450B having a collective surface in a "slope" configuration. The LF display system 450B includes LF display modules tiled to form a collective surface 460. The collective surface 460 includes three portions: (i) a first portion 462 (i.e., a central surface); (ii) a second portion 464 connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front (i.e., a first side) of the central portion; and (iii) a third portion 464 connected to the first portion 462 and positioned at an angle from the central portion to project energy toward the front (i.e., a second side) of the central portion. When the mobile device is held such that the first portion is a vertical surface having a horizontal plane and a vertical axis, the second and third portions are angled toward the front of the central portion along the vertical axis.

[0099] In this example, the viewing volume 468B of the LF display system 450B is in front of the central portion and is partially surrounded by three portions of the collecting surface 460. Similar to the configuration shown in FIG. 4C, the two side portions (e.g., the second portion 464 and the third portion 466) are angled to surround the viewer and form a surrounding surface. The surrounding surface expands the viewing FOV from any viewer's viewpoint within the holographic viewing volume 468B. Additionally, the surrounding surface allows the viewing volume 468B to be closer to the surface of the display so that projected objects appear closer. In other words, the angled arrangement of the side surfaces expands the field of view, reduces separation, and increases the proximity of the collecting surface, thereby increasing the immersive experience of the viewer. Additionally, as discussed below, deflection optics can be used to optimize the size and position of the viewing volume 468B.

[0100] The tilted configuration of the side portions of the collecting surface 460 allows the holographic content to be presented closer to the viewing volume 468B than if the third portion 466 were not tilted. For example, the lower extremities (such as legs) of a character presented from a tilted configuration LF display system may appear closer and more realistic than if a LF display system with a flat front display were used. Additionally, the configuration of the LF display system and the environment in which it is located can inform the shape and location of the viewing volume and viewing sub-volumes.

[0101] 4E, for example, illustrates a top view of an LF display system 450C having a collecting surface 460. The LF display system 450C projects various light rays from the collecting surface 460. The light rays projected from the left side of the collecting surface 460 have a horizontal angular range 481, the light rays projected from the right side of the collecting surface have a horizontal angular range 482, and the light rays projected from the center of the collecting surface 460 have a horizontal angular range 483. Between these points, the projected light rays can take intermediate values ​​of the angular range. In this way, a viewing volume 468C is created by having a gradient deflection angle for the projected light rays across the display surface. Controlling the LF Display System

[0102] FIG. 5 is a block diagram of an LF display system 500, according to one or more embodiments. The LF display system 500 comprises an LF display assembly 510 and a controller 520. The LF display assembly 510 includes one or more LF display modules 512 that project a light field. The LF display modules 512 may include a source / sensor system 514 that includes an integrated energy source and / or energy sensor that projects and / or senses other types of energy. The controller 520 includes a data store 522, a network interface 524, an LF processing engine 530, a command library 532, and a security module 534. The controller 520 may also include a tracking module 526, and a user profiling module 528. In some embodiments, the LF display system 500 also includes a sensory feedback system 570 and a tracking system 580. The LF display systems described in the context of FIGS. 1, 2, 3, and 4 are embodiments of the LF display system 500. In other embodiments, the LF display system 500 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or different entities in ways different than described herein. Applications of LF display system 500 are also discussed in more detail below with respect to Figures 6-8.

[0103] The LF display assembly 510 provides holographic content within a holographic object volume that is visible to a viewer located within the viewing volume. The LF display assembly 510 can provide the holographic content by executing display instructions received from the controller 520. The holographic content can include one or more holographic objects projected in front of a collecting surface, onto the LF display assembly 510, behind the collecting surface of the LF display assembly 510, or some combination thereof. Generation of display instructions using the controller 520 is described in more detail below.

[0104] The LF display assembly 510 provides holographic content using one or more LF display modules (e.g., any of the LF display module 110, the LF display system 200, and the LF display module 300) included in the LF display assembly 510. For convenience, the one or more LF display modules may be described herein as LF display modules 512. The LF display modules 512 may be tiled to form the LF display assembly 510. The LF display modules 512 may be structured as various seamless surface environments (e.g., single-sided, multi-sided, curved, etc.). That is, the tiled LF display modules form a collective surface. As previously described, the LF display module 512 may include an energy device layer (e.g., energy device layer 220) and an energy waveguide layer (e.g., energy waveguide layer 240) that present the holographic content. The LF display module 512 may also include an energy relay layer (e.g., energy relay layer 230) that transfers energy between the energy device layer and the energy waveguide layer when presenting the holographic content.

[0105] The LF display module 512 may also include other integrated systems configured for energy projection and / or energy sensing, as described above. For example, the light field display module 512 may include any number of energy devices (e.g., energy device 340) configured to project and / or sense energy. For convenience, the integrated energy projection system and integrated energy sensing system of the LF display module 512 may be collectively described herein as a source / sensor system 514. The source / sensor system 514 is integrated within the LF display module 512 such that the source / sensor system 514 shares the same seamless energy surface as the LF display module 512. In other words, the collective surface of the LF display assembly 510 includes the functionality of both the LF display module 512 and the source / sensor module 514. That is, the LF assembly 510 including the LF display module 512 with the source / sensor system 514 can simultaneously project energy and / or sense energy while projecting a light field. For example, the LF display assembly 510 may include an LF display module 512 and a source / sensor system 514 configured as a dual energy plane or a bidirectional energy plane as described above.

[0106] In some embodiments, the LF display system 500 uses the sensory feedback assembly 570 to augment the generated holographic content with other sensory content (e.g., coordinated touch, sound, or smell, etc.). The sensory feedback system 570 can augment the projection of the holographic content by executing display instructions received from the controller 520. In general, the sensory feedback system 570 includes any number of sensory feedback devices (e.g., the sensory feedback system 442) external to the LF display assembly 510. Some exemplary sensory feedback devices may include conditioned sound projection and receiving devices, aroma projection devices, temperature regulation devices, force actuation devices, pressure sensors, transducers, etc. In some cases, the sensory feedback system 570 may have similar functionality as the light field display assembly 510, or vice versa. For example, both the sensory feedback system 570 and the light field display assembly 510 may be configured to generate a sound field. As another example, the sensory feedback system 570 may be configured to generate a tactile surface, while the light field display assembly 510 is not.

[0107] To illustrate, in one exemplary embodiment of the light field display system 500, the sensory feedback system 570 may include an acoustic projection device. The acoustic projection device is configured to generate one or more pressure waves that complement the holographic content when executing the display instructions received from the controller 520. The generated pressure waves may be, for example, audible (for sound), ultrasonic (for touch), or some combination thereof. Similarly, the sensory feedback system 570 may include an aroma projection device. The projection device may be configured to impart a scent to some or all of the target area when executing the display instructions received from the controller. Additionally, the sensory feedback system 570 may include a temperature adjustment device. The temperature adjustment device is configured to increase or decrease the temperature of some or all of the target area when executing the display instructions received from the controller 520. In some embodiments, the sensory feedback system 570 may include an actuation force device. The actuation force device may be used to move some physical objects in the target area.

[0108] In some embodiments, the sensory feedback assembly 570 is configured to receive input from a user of the LF display system 500. In this case, the sensory feedback system 570 includes various sensory feedback devices for receiving input from the user. The sensor feedback devices may include devices such as acoustic receiving devices (e.g., microphones), pressure sensors, motion detectors, etc. The sensory feedback assembly 570 may include sensory feedback devices configured to provide sensory feedback when holographic content is presented. The sensory feedback system may transmit the detected input to the controller 520 to adjust the generation of the holographic content and / or sensory feedback.

[0109] To illustrate, in one exemplary embodiment of a light field display assembly, the sensory feedback system 570 includes a microphone. The microphone is configured to record audio produced by a user of one or more appliances. The sensory feedback system 570 provides the recorded audio to the controller 520 as user input. The controller 520 can use the user input to generate the holographic content. Similarly, the sensory feedback system 570 can include a pressure sensor. The pressure sensor is configured to measure a force applied to the pressure sensor by the user. The sensory feedback system 570 can provide the measured force to the controller 520 as user input.

[0110] In some embodiments, the sensory feedback assembly 570 is also configured to redirect the user's movements. In one embodiment, the sensory feedback assembly 570 includes an acoustic projection device that can generate haptic pressure waves to discourage a movement, such as opening the oven door before a timer expires. In another embodiment, the sensory feedback assembly 570 can use an acoustic emitting device to audibly signal the user from further action (e.g., "do not open the oven door").

[0111] In some embodiments, the LF display system 500 includes a tracking system 580. The tracking system 580 includes any number of tracking devices configured to obtain information about the user of the device, including location, movement, gestures, facial expressions, gaze, and / or other characteristics of people in a target area, such as age and gender. Generally, the tracking devices are external to the LF display assembly 510. Some exemplary tracking devices include a camera assembly ("camera"), a depth sensor, structured light, a LIDAR system, a card scanning system, or other tracking device capable of tracking viewers in a target area.

[0112] The tracking system 580 may include one or more energy sources that illuminate some or all of the target area with light. In some cases, however, when presenting holographic content, the target area is illuminated with natural light and / or ambient light from the LF display assembly 510. The energy sources project light when executing instructions received from the controller 520. The light may be, for example, a structured light pattern, a pulse of light (e.g., an IR flash), or some combination thereof. The tracking system may project light within the visible band (approximately 380 nm to 750 nm), within the infrared (IR) band (approximately 750 nm to 1700 nm), within the ultraviolet band (10 nm to 380 nm), some other portion of the electromagnetic spectrum, or some combination thereof. Sources may include, for example, light emitting diodes (LEDs), micro LEDs, laser diodes, TOF depth sensors, tunable lasers, etc.

[0113] Tracking system 580, when executing instructions received from controller 520, may adjust one or more emission parameters. An emission parameter is a parameter that affects how light is projected from a source of tracking system 580. Emission parameters may include, for example, brightness, pulse rate (including continuous illumination), wavelength, pulse length, any other parameter that affects how light is projected from the source assembly, or any combination thereof. In one embodiment, the source projects pulses of light in a time-of-flight operation.

[0114] The camera of the tracking system 580 captures an image of the light (e.g., structured light pattern) reflected from the target area. The camera captures the image as it executes the tracking instructions received from the controller 520. As previously described, the light may be projected by the source of the tracking system 580. The camera may include one or more cameras. That is, the camera may be, for example, an array of photodiodes (1D or 2D), a CCD sensor, a CMOS sensor, some other device that detects some or all of the light projected by the tracking system 580, or some combination thereof. In one embodiment, the tracking system 580 may include a light field camera external to the LF display assembly 510. In other embodiments, the camera is included as part of an LF display module included in the LF display assembly 510. For example, as previously described, if the energy relay element of the light field module 512 is a bidirectional energy layer that interleaves both a light-emitting display and an image sensor with the energy device layer 220, the LF display assembly 510 may be configured to project a light field and simultaneously record image information from a viewing area in front of the display. In one embodiment, images captured from the bidirectional energy surface form a light field camera. The camera provides the captured images to the controller 520. In some embodiments, the camera assembly may illuminate certain portions of the appliance to generate visual information for display to a viewer. For example, an internal sensor such as a camera may be present within the refrigerator to image the contents of the refrigerator and provide information to the controller 520 for information regarding generating a holographic display of the refrigerator contents.

[0115] The cameras of tracking system 580 may adjust one or more imaging parameters when executing tracking instructions received from controller 520. Imaging parameters are parameters that affect how the camera assembly captures images. Imaging parameters may include, for example, frame rate, aperture, gain, exposure length, frame timing, any other parameter that affects how the camera assembly captures images, or any combination thereof.

[0116] In some embodiments, the tracking system 580 is also configured to receive input from a viewer of the LF display system 500. The tracking system 580 can track the viewer's physical movements and transmit information regarding the specific tracked movements to the controller 520. In some embodiments, the tracking system 580 is used in coordination with a holographic user interface that is presented to the user.

[0117] The controller 520 controls the LF display assembly 510 and any other components of the LF display system 500. The controller 520 includes a data store 522, a network interface 524, a tracking module 526, a user profiling module 528, a light field processing engine 530, and a command library 532. In other embodiments, the controller 520 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different from that described herein. For example, the tracking module 526 may be part of the LF display assembly 510 or the tracking system 580.

[0118] The data store 522 is a memory that stores information for the LF display system 500. The stored information may include display instructions, tracking instructions, emission parameters, imaging parameters, a virtual model of the target area, tracking information, images captured by the camera, one or more user profiles, calibration data for the light field display assembly 510, configuration data for the LF display system 510 including the resolution and orientation of the LF module 512, desired viewing volume shape, 3D models, scenes and environments, content for graphics creation including materials and textures, security related information including authentication elements such as passwords, smart cards, biometric data, etc. that may be used by the security module 534, other information that may be used by the LF display system 500, or any combination thereof. The data store 522 is a memory such as a read only memory (ROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), or any combination thereof.

[0119] The network interface 524 allows the light field display system to communicate with other systems or environments over a network. In one example, the LF display system 500 receives holographic content from a remote server, such as a third party provider, via the network interface 524. In another example, the LF display system 500 uses the network interface 524 to transmit holographic content to a remote data store.

[0120] The tracking module 526 tracks viewers viewing content presented by the LF display system 500. To do so, the tracking module 526 generates tracking instructions that control the operation of sources and / or cameras of the tracking system 580 and provides the tracking instructions to the tracking system 580. The tracking system 580 executes the tracking instructions and provides tracking inputs to the tracking module 526 via the controller 520.

[0121] The tracking module 526 may determine a position of one or more viewers of the target area of ​​the mobile device. The determined position may be relative to some reference point (e.g., the display surface). In other embodiments, the determined position may be within a virtual model of the target area. The tracked position may be, for example, a tracked position of the viewer and / or a tracked position of a portion of the viewer (e.g., eye location, hand location, etc.). The tracking module 526 determines the position using one or more images captured from the cameras of the tracking system 580. The cameras of the tracking system 580 may be distributed around the LF display system 500 and may capture images in stereo, allowing the tracking module 526 to passively track the viewer. In other embodiments, the tracking module 526 actively tracks the viewer. That is, the tracking system 580 illuminates a portion of the target area, images the target area, and the tracking module 526 determines the position using time-of-flight and / or structured light depth determination techniques. The tracking module 526 uses the determined position to generate tracking information.

[0122] The tracking module 526 may also receive tracking information as input from a viewer of the LF display system 500, such as while operating an appliance using a holographic user interface (described further below). The tracking information may include body movements of the viewer corresponding to various input options provided by the LF display system 500. For example, the tracking module 526 may track the viewer's body movements and assign any of the various movements as input to the LF processing engine 530. The tracking module 526 may provide the tracking information to the data store 522, the LF processing engine 530, the user profiling module 528, the command store 532, any other component of the LF display system 500, or some combination thereof.

[0123] To provide context for the tracking module 526, consider an exemplary embodiment of the LF display system 500 used in a microwave oven. When a user reacts to a boiled egg exploding in the microwave by running to the appliance and screaming, the tracking system 580 may record the user's movements and transmit the recording to the tracking module 526. The tracking module 526 tracks the user's hand motions in the recording and transmits input to the LF processing engine 530, described below. As described below, the user profiling module 528 determines that information in the image indicates that the user's movements are associated with a negative response. Thus, the LF processing engine 530 generates appropriate holographic content to interact with the user. For example, the LF processing engine 530 may project within the holographic object volume a sad holographic avatar stating, "It'll be better next time!"

[0124] The LF display system 500 includes a user profiling module 528 configured to identify and profile users. The user profiling module 528 generates a profile of one or more users who view holographic content displayed by the LF display system 500. The user profiling module 528 generates the user profile based in part on user input and monitored user behavior, actions, and reactions. The user profiling module 528 can access information obtained from the tracking system 580 (e.g., recorded images, videos, audio, etc.) and process the information to determine various information. In various examples, the user profiling module 528 can use machine vision and / or machine learning algorithms to determine user behavior, actions, and reactions. Monitored user behavior can include, for example, smiling, cheering, clapping, laughing, fear, screams, excitement levels, recoil, changes in other gestures or movements by the user, etc.

[0125] More generally, a user profile may include any information received and / or determined regarding a user viewing holographic content from the LF display system. For example, each user profile may record that user's actions or responses to content displayed by the LF display system 500. Some example information that may be included in a user profile is provided below.

[0126] The user profile may be based on the user's behavior with respect to content displayed by the LF display system 500. In some embodiments, software applications resident in a device may be functionally integrated with the LF display system 500, and the user profile is based on the user's behavior monitored while running those software applications on the device.

[0127] Thus, for example, monitored behaviors may include the number of times a user uses an appliance over a period such as a day or week, how often a user uses a particular setting on an appliance (e.g., how often a washing machine is used on a heavy bedding wash setting), how a user responds to a particular type of holographic content (e.g., a holographic image of a roasted turkey with a sign indicating that the bake time setting has exceeded 20 minutes, or a flashing holographic image of a burnt turkey, which is what elicits the most immediate response from a user), other behaviors associated with the use of a particular appliance, or combinations thereof. In some embodiments, an appliance may also have a demographic of users (e.g., daily users, weekly users, occasional users, etc., of a washing machine at a laundromat). In these embodiments, the tracking system may be able to reliably identify a user. Once the tracking system has access to the user profiling module, the LF display system 500 may determine the status of the user (e.g., as a daily user, weekly user, occasional user) and provide holographic content specific to the user's status. The user profiling module may additionally receive further information about the user from other modules or components of the controller 520, which may be stored with the user profile. Other components of the controller 520 may then also access the stored user profile to determine future content to be provided to that user.

[0128] In another example, a user may indicate a preference for a tactile interface presented in conjunction with the holographic content, such as a preferred key click sensitivity in a holographic keyboard touch interface that may be generated by the LF display system 100 and offered as an option to an appliance user.

[0129] Other user characteristics stored as part of the user profile information may include, for example, the user's name, the user's age, ethnicity, viewing location, device and application setting preferences, place of residence, other demographic information, or any combination thereof.

[0130] The user profiling module 528 may access profiles associated with a particular user or users from one or more third party systems to build and / or update a user profile. The audience profiling module 528 may be configured to update a user profile using information from one or more users' social media accounts. For example, a user's interactions with third party vendors linked to the user's social media or other online accounts may provide further user profile information that may be accessed by the user profiling module 528. Thus, user profile information may store information such as a favorite laundry detergent product on an LF display system associated with a washing machine or a favorite music genre for playback on a home audio system.

[0131] In some embodiments, the data store 522 includes a user profile store that stores user profiles generated, updated, and / or maintained by the user profiling module 528. The user profile can be updated in the data store at any time by the user profiling module 528. For example, in one embodiment, the user profile store receives and stores information about a particular user in a user profile for that user when that user views holographic content provided by the LF display system 500. In this example, the user profiling module 528 includes a facial recognition algorithm that can recognize the user and reliably identify the user when the user views the presented holographic content. To illustrate, when a user enters a target area of ​​the LF display system 500, the tracking system 580 captures an image of the user. The user profiling module 528 inputs the captured image and identifies the user's face using a facial recognition algorithm. The identified face is associated with the user profile in the profile store, such that all input information captured about that user can be stored in that user's profile. The user profiling module may also utilize a card identification scanner, a voice identifier, a radio frequency identification (RFID) chip scanner, a barcode scanner, etc. to positively identify the user.

[0132] Because the user profiling module 528 can reliably identify users, the user profiling module 528 can determine each visit of each user to the LF display system 500. The user profiling module 528 can then store the date and time of each visit in a user profile for each user. Similarly, the user profiling module 528 can store input from the user received from any combination of the sensory feedback system 570, the tracking system 580, and / or the LF display assembly 510, whenever the input occurs. The user profile system 528 can additionally receive further information about the user from other modules or components of the controller 520, which information can be stored with the user profile. Other components of the controller 520 can then also access the stored user profile to determine subsequent content to be provided to that user. The controller 520 can be configured to generate holographic content based at least in part on data accessed from the stored user profile.

[0133] The LF processing engine 530 generates 4D coordinates in a rasterized format ("rasterized data") that, when executed by the LF display assembly 510, cause the LF display assembly 510 to present holographic content. The LF processing engine 530 can access the rasterized data from the data store 522. Additionally, the LF processing engine 530 can construct the rasterized data from a vectorized data set. The vectorized data is described below. The LF processing engine 530 can also generate sensory instructions necessary to provide sensory content that augments the holographic object. As described above, the sensory instructions, when executed by the LF display system 500, can generate haptic surfaces, sound fields, and other forms of sensory energy supported by the LF display system 500. The LF processing engine 530 can access the sensory instructions from the data store 522 or construct the sensory instructions to form the vectorized data set. Collectively, the 4D coordinates and sensory data represent display instructions executable by the LF display system to generate holographic and sensory content.

[0134] The amount of rasterized data describing the flow of energy through the various energy sources in the LF display system 500 is incredibly large. While it is possible to display the rasterized data on the LF display system 500 when accessed from the data store 522, it is not possible to efficiently transmit (e.g., via the network interface 524), receive, and then display the rasterized data on the LF display system 500. Take, for example, rasterized data representing a short video for holographic projection by the LF display system 500 (e.g., a tutorial showing a new user how to use an appliance). In this example, the LF display system 500 includes a display containing several gigapixels, and the rasterized data includes information for each pixel location on the display. The corresponding size of the rasterized data is enormous (e.g., many gigabytes per second of video display time) and unmanageable for efficient transfer over a commercial network via the network interface 524. Additional problems with simply storing the rasterized data in the data store 522 arise when an interactive experience is desired using input from the sensory feedback system 570 or the tracking module 526. To enable interactive experiences, the light field content generated by the LF processing engine 530 may be modified in real time in response to sensory or tracking inputs. In other words, in some cases, the LF content cannot simply be read from the data store 522.

[0135] Thus, in some configurations, data representing holographic content for display by the LF display system 500 may be transferred to the LF processing engine 530 in a vectorized data format ("vectorized data"). The vectorized data may be orders of magnitude smaller than rasterized data. Furthermore, the vectorized data provides high image quality while having a data set size that allows for efficient sharing of the data. For example, the vectorized data may be a sparse data set derived from a denser data set. Thus, the vectorized data may have an adjustable balance between image quality and data transmission size based on how the sparse vectorized data is sampled from the denser rasterized data. The adjustable sampling to generate the vectorized data allows for optimization of image quality at a particular network speed. As a result, the vectorized data allows for efficient transmission of the holographic content over the network interface 524. The vectorized data may also allow for live streaming of the holographic content over commercial networks.

[0136] In summary, the LF processing engine 530 may generate holographic content derived from rasterized data accessed from the data store 522, vectorized data accessed from the data store 522, or vectorized data received via the network interface 524. In some configurations, the LF processing engine 530 may generate holographic content using a combination of rasterized data accessed from the data store, vectorized data accessed from the data store 522, and vectorized data received via the network interface. In various configurations, the vectorized data may be encoded prior to data transmission and decoded after receipt by the LF controller 520. In some examples, the vectorized data is encoded for additional data security and performance improvements associated with data compression. For example, the vectorized data received by the network interface may be encoded and vectorized data received from a holographic streaming application. In some examples, the vectorized data may require a decoder, the LF processing engine 530, or both, to access information content encoded within the vectorized data. The encoder and / or decoder systems may be available to customers or may be licensed to third-party vendors.

[0137] The vectorized data includes information for each of the sensory areas supported by the LF display system 500 in a manner that supports an interactive experience. For example, the vectorized data for an interactive holographic experience includes any vectorized characteristics that can provide accurate physics for each of the sensory areas supported by the LF display system 500. The vectorized characteristics can include any characteristics that can be synthetically programmed, captured, computationally evaluated, etc. The LF processing engine 530 can be configured to convert the vectorized characteristics of the vectorized data into rasterized data. The LF processing engine 530 can then project the holographic content converted from the vectorized data from the LF display assembly 510. In various configurations, the vectorized properties may include one or more red / green / blue / alpha channel (RGBA) + depth images, multi-view images with or without depth information at various resolutions that may include one high-resolution central image and other views at lower resolution, albedo and reflectance, surface normals, other optical effects, surface identity, geometric object coordinates, virtual camera coordinates, display surface location, lighting coordinates, surface contact stiffness, contact ductility, contact intensity, amplitude and coordinates of sound fields, environmental conditions, somatosensory energy vectors associated with mechanoreceptors of texture or temperature, sound, and coordinates of other sensory domain properties. Many other vectorized properties are possible.

[0138] The LF display system 500 can also generate interactive viewing experiences. That is, the holographic content may respond to input stimuli, including information about the user's location, gestures, interactions, interactions with the holographic content, or other information derived from the user profiling module 528 and / or tracking module 526. For example, if a displayed holographic object needs to immediately move in a particular direction in response to a user's interaction, the LF processing engine 530 may update the rendering of the scene so that the holographic object moves in that requested direction. This may require the LF processing engine 530 to use the vectorized data set to render a light field in real time based on a 3D graphical scene with proper object placement and movement, collision detection, occlusion, color, shading, lighting, etc., to respond correctly to the user's interactions. The LF processing engine 530 converts the vectorized data into rasterized data for presentation by the LF display assembly 510.

[0139] The rasterized data includes holographic content instructions and sensory instructions (display instructions) that represent the real-time performance. The LF display assembly 510 simultaneously projects the holographic and sensory content of the real-time performance by executing the display instructions. The LF display system 500 monitors user interactions (e.g., voice responses, touch, etc.) with the presented real-time performance content using a tracking module 526 and a user profiling module 528. In response to the user interactions, the LF processing engine creates an interactive experience by generating additional holographic and / or sensory content for display to the user.

[0140] To illustrate, consider an exemplary embodiment of an LF display system 500 including an LF processing engine 530 that generates a holographic object representing a loaf of bread in a user's viewing volume during execution of an application in the device. The user may move to touch the holographic object representing the loaf of bread. In response, a tracking system 580 tracks the movement of the user's hand relative to the holographic object. The user's movements are recorded by the tracking system 580 and transmitted to the controller 520. The tracking module 526 continuously determines the movement of the user's hand and transmits the determined movements to the LF processing engine 530. The LF processing engine 530 determines the placement of the user's hand in the scene and adjusts the real-time rendering of the graphics to include any changes (such as position, color, or occlusion) required for the holographic object. The LF processing engine 530 instructs the LF display assembly 510 (and / or the sensory feedback system 570) to generate a tactile surface using a volumetric haptic projection system (e.g., using an ultrasonic speaker). The generated haptic surface corresponds to at least a portion of the holographic object and occupies substantially the same space as some or all of the exterior surface of the holographic object. The LF processing engine 530 uses the tracking information to dynamically direct the LF display assembly 510 to move the location of the haptic surface along with the location of the rendered holographic object such that the user is given both a visual and haptic perception of touching a loaf of bread. In simpler terms, as the user sees his or her hand touching a holographic loaf of bread, the viewer simultaneously feels haptic feedback indicating that the hand is touching the holographic loaf of bread and that the loaf changes position or movement in response to the contact.

[0141] The LF processing engine 530 may provide holographic content for simultaneous display to a user of the media content using the appliance. For example, a cooking video being viewed on an oven augmented with the LF display system 500 may include holographic content (a "holographic content track") that is presented to the user during the video. The holographic content track may be received by the appliance and stored in the data store 522. The holographic content track includes holographic content that enhances the viewing experience of a user watching the video on the appliance.

[0142] The holographic content in the holographic content track may be associated with any number of temporal, auditory, visual, etc. cues to display the holographic content. For example, the holographic content track may include holographic content that is displayed at a particular time during a video. As an example, the holographic content track may include a cooked apple pie that is displayed during a video 12 minutes and 42 seconds into the start of a "How to Bake an Apple Pie from Scratch" video. In another example, the holographic content track includes holographic content that is presented when the sensory feedback system 570 records a particular audio cue. As an example, the holographic content track may include a holographic avatar showing dough kneading instructions to present during a bread baking video when audio recorded by the sensory feedback assembly 570 indicates that the user / player has asked for more help. In another example, the holographic content track includes holographic content that is displayed when the tracking system 580 records a particular visual cue. As an example, the holographic content track may include a holographic flashing sign with the words "Stop?" when the tracking system 580 records information indicating that the user is moving away from an appliance while a video tutorial is being displayed. The determination of auditory and visual cues is described in more detail below.

[0143] The holographic content track may also include spatial rendering information. That is, the holographic content track may indicate spatial locations for presenting the holographic content in the displayed video track. For example, the holographic content tract may indicate that a particular holographic content is presented in some holographic viewing volumes and not in others. To illustrate, the LF processing engine 530 may present a holographic screen displaying the contents of a refrigerator shelf in a holographic viewing volume directly in front of the user's face. Similarly, the holographic content track may indicate that the holographic content is to be presented in some viewing volumes and not in others. For example, the LF processing engine may present a holographic screen in a viewing volume directly in front of the user's face but not in other viewing volumes, so that the displayed content is private to the user.

[0144] The LF processing engine 530 may provide holographic content for display to a user in conjunction with a particular application running on the appliance. The holographic content may be received by the appliance over a network, provided as part of advertising content associated with a particular application running on the device (e.g., laundry detergent specifically designed for silk and wool when a delicate wash cycle is initiated on the washing machine), or generated by the LF processing engine 530, etc. The holographic content may be stored in the data store 522 or streamed to the LF display system 500 in a vectorized format via the network interface 524. For example, when a user of the washing machine initiates a wash cycle on the washing machine, especially on a "delicate" cycle, the LF display system 500 may access holographic content advertising a laundry detergent specifically designed for silk and wool and display it during the course of the wash cycle.

[0145] The LF processing engine 500 may also modify the holographic content to suit the environmental characteristics of a location. For example, based on whether the consumer device is located in a dark room space, a bright room, a sunny location, etc., the LF processing engine may modify the holographic content to be displayed with a brightness appropriate for the lighting of the location. The LF processing engine may use the data stored in the data store 522, along with information from the camera system in the tracking module 580, to customize the display of the holographic content. Additionally, the modification of the holographic content may also take into account user display preferences, as determined by the user profiling module 528.

[0146] The LF processing engine 530 may also create holographic content for display by the LF display system 500. Importantly, here, creating holographic content for display is distinct from accessing or receiving holographic content for display. That is, when creating content, the LF processing engine 530 generates entirely new content for display, rather than accessing previously generated and / or received content. The LF processing engine 530 may obtain or receive information from the data store 522, the tracking module 526, the user profiling module 528, the sensory feedback system 570, the tracking system 580, or some combination thereof, to create holographic content for display. In some examples, the LF processing engine 530 may access information (e.g., tracking information and / or user profile) from elements of the LF display system 500, and accordingly create content based on that information, and display the created content using the LF display system 500. The created content may be holographic content. The created holographic content may be augmented with other sensory content (e.g., touch, sound, or smell) when displayed by the LF display system 500. Additionally, the LF display system 500 may store the created holographic content in a data store 522 for future use.

[0147] In some embodiments, the LF processing engine 530 may cause the generated holographic display to be updated, modified, removed, paused, or some combination thereof in response to changes in the operating conditions of the consumer device. In some embodiments, the LF processing engine 530 may cause the generated holographic display to be updated, modified, removed, paused, or some combination thereof in response to information received by the tracking system 580 and interpreted by the tracking module 526. For example, the tracking module 580 may provide captured gaze information from the tracking system 580 to the LF display module for re-rendering the holographic content to make eye contact with a user interacting with the device.

[0148] In some embodiments, the LF display system 500 includes a command library 532. The command library 532 may function as a command interface that recognizes and interprets user commands. The user commands may be interpreted based on tracking information received from the sensory feedback system 570 (e.g., voice), the tracking module 526 (e.g., tracked gestures or facial expressions), the user profiling module 528 (e.g., stored personal preferences), or some combination thereof. The command library 532 may apply heuristics to interpret the user commands and direct the LF processing engine 530 to generate appropriate visual, haptic, and any other multi-sensory information. The command library 532 stores mappings of received user commands to computational commands that can be executed by the LF display system 500. The command library 532 also stores heuristics associated with the stored mappings. The command library 532 may receive user commands via a holographic user interface displayed by the LF display system 500. Examples of possible user commands received via the displayed holographic user interface may include various specific touch contacts of a holographic object by a user (e.g., a finger touch on a particular displayed holographic "button") and various tactile movements relative to the surface of a holographic object (e.g., turning a displayed holographic "knob"). In some embodiments, a touch contact or tactile movement may be determined to have occurred when the tracking module 526 determines that the user is within a threshold distance of a displayed holographic object in the displayed holographic user interface.In some embodiments, user commands may include sensory feedback provided by the user, such as verbal commands spoken by the user while in the target area (e.g., a voice command in the form of "Show me Grandma's apple pie video" captured using an acoustic receiver), gestures and body movements captured by tracking system 580 (e.g., hands held by the user in the target area to indicate that the timer should be paused), the sensing of a physical switch, knob, button, or dial connected to the display as part of sensory feedback system 570, or a combination thereof.

[0149] Computational commands executed by the LF processing engine in response to received user commands may include commands to display a particular holographic object, update the display of holographic content, change the display of a particular holographic object, display a sequence of holographic objects, pause or play the content being presented by the LF display system 500, remove a displayed object, or combinations thereof. Other computational commands may include actuating other sensors and controls, such as consumer device controls, such as a baking temperature setting button, to change the operational state of the consumer device. Thus, in some embodiments, a detected user action, such as turning a displayed holographic "dial," is reported by tracking system 580 to command library 532. In response, command library 532 may apply heuristics from stored heuristics and may determine that a computational command needs to be executed, including, for example, sending instructions to a system to properly set a timer application on an appliance.

[0150] In some embodiments, the command library 532 receives information from the data store 522 and the tracking module 580 to resolve ambiguous mappings based on user commands that may be detected with a lower degree of confidence. However, information from the tracking system 580 and user profile information from the data store 522 may provide sufficient heuristic information to disambiguate the mappings and determine the specific computational command to be executed. This determined computational command is then provided to the LF processing engine 530 for execution.

[0151] In some embodiments, the LF display system 500 includes a security module 534. The security module 534 performs secure access to at least some of the functionality provided by the LF display system. To that end, the security module 534 may generate user interface elements by the LF display system 500 to receive authentication credentials, such as a password or biometric data, from a user of the consumer device. The security module authenticates the credentials based on data stored in the data store 522. After authenticating the user, the security module 534 may enable secure access by the user to some or all of the functionality of the LF display system 500. In some embodiments, the security module 534 may activate an internal sensor in the consumer device, such as an internal camera positioned to record images of elements, such as internal mechanical elements and internal circuitry. In some embodiments, the security module 534 may provide images from the internal camera directly to the authenticated user for viewing. In some embodiments, the security module 534 may receive information from the internal sensor and provide the information to the LF processing engine 530 to generate holographic content based on the received information. In some embodiments, the LF processing engine 530 may process information from the internal sensors and use the processed data to map to data stored in the data store 522, generate / retrieve holographic content, and display them to the user. In some embodiments, the LF processing engine 530 may use AI models to map information from the internal sensors to information stored in the data store 522, and then generate / retrieve holographic content. For example, a qualified oven repair technician may provide authentication credentials to gain secure access to the internal circuitry associated with the oven's broiler element. Upon authentication, the security module 534 may activate an internal camera to receive images related to the internal circuitry and provide the images to the LF processing engine 530.The LF processing engine 530 may process the received image data and map the processed image data to data stored in the data store 522 associated with a depiction of the broken heating element. The LF processing engine may then generate or retrieve holographic content depicting the broken heating element based on the mapped data and display it to a qualified oven repair technician. The security module 534 may provide information related to authenticating a user of a consumer device to the data store 522, the user profiling module 528, the LF processing engine 530, the command library 532, any other component of the LF display system 500, or some combination thereof. Dynamic Content Generation for LF Display Systems

[0152] In some embodiments, the LF processing engine 530 incorporates an artificial intelligence (AI) model to create holographic content for display by the LF display system 500. The AI ​​model may include supervised or unsupervised learning algorithms, including but not limited to regression models, neural networks, classifiers, or other AI algorithms. The AI ​​model may be used to determine user preferences based on user information recorded by the LF display system 500 (e.g., by the tracking system 580), which may include information about user behavior. The AI ​​model may also store each learned user's preferences in a user profile store in the data store 522. In some examples, when multiple users operate an appliance, the AI ​​model may create holographic content for a particular individual user based on the particular user's learned preferences. The AI ​​model may access information from the data store 522 to create the holographic content. In some embodiments, the AI ​​model may use the results of the learning algorithm to establish user-specific heuristics for command mappings stored in the command library 532.

[0153] One example of an AI model that may be used to identify user characteristics, identify responses, and / or generate holographic content based on identified information is a convolutional neural network model having layers of nodes where the value of a node in the current layer is the transformation of the value at the node in the previous layer. The transformation in the model is determined via a set of weights and parameters that connect the current layer and the previous layer. In some examples, the transformation may also be determined via a set of weights and parameters used to transform between previous layers in the model.

[0154] The input to the model may be an image captured by the tracking system 580 encoded in the first convolutional layer, and the output of the model is the holographic content decoded from the output layer of the neural network. Alternatively or additionally, the output may be determined features of the user in the image. In this example, the AI ​​model may identify latent information in the image that represents the features of the user in one of the intermediate layers of the neural network. The association information between the elements identified in the layers can be retrieved by applying a series of transformations between the corresponding intermediate layers. The weights and parameters of the transformations may indicate the relationship between the information contained in the starting layer and the information obtained from the final output layer. For example, the weights and parameters may be quantization of the shape, color, size, etc. contained in the information representing the smiling user in the image. The weights and parameters may be based on historical data (e.g., previously tracked users).

[0155] In one embodiment, the AI ​​model includes a deterministic method trained with reinforcement learning (thereby creating a reinforcement learning model). The model is trained to improve the quality of the performance using measurements from the tracking system 580 as inputs and modifications to the created holographic content as outputs.

[0156] Reinforcement learning is a machine learning system in which a machine learns "what to do" (how to map situations to actions) to maximize a numerical reward signal. The learner (e.g., LF processing engine 530) is not told what action to take (e.g., generate a given holographic content), but instead discovers, by trying actions, which action will yield the greatest reward (e.g., increasing the quality of the holographic content by making the user smile more). In some cases, actions can affect not only the immediate reward, but also the next situation and, through it, all subsequent rewards. These two features (trial-and-error exploration and delayed reward) are the two distinguishing features of reinforcement learning.

[0157] The AI ​​model may include any number of machine learning algorithms. Some other AI models that may be used are linear and / or logistic regression, classification and regression trees, k-means clustering, vector quantization, etc. In general, the LF processing engine 530 takes input from the tracking module 526 and / or the user profiling module 528, and the machine learning model creates the holographic content accordingly. Similarly, the AI ​​model may direct the rendering of the holographic content. Network environment

[0158] 6 illustrates a block diagram of a system environment 600 according to one or more embodiments. The system environment 600 includes one or more networked consumer devices 610, a network 620, and one or more third-party content providers 630.

[0159] In one embodiment, the consumer device 610 is a portable device that provides one or more consumer-oriented functions enabled by an electronic or mechanical control system or a combination thereof. A networked consumer device may also execute computer program modules, e.g., a web browser or client application, that allow a user to browse the Internet, consume media content, and execute various software applications in the course of using the appliance. The consumer device 610 may be, for example, a washing machine, an oven, a refrigerator, or any other network-enabled consumer device. The consumer device 610 includes a light field display system 612, and a user interface 614, in addition to the consumer-related functions specific to the appliance. In other embodiments, the consumer device 610 includes additional or fewer modules than those described herein. Similarly, functionality may be distributed among modules and / or among different entities in a manner different from that described herein.

[0160] The LF display system 616 shown in Figure 6 is an embodiment of the LF display system 500 shown in Figure 5. A user 640 can interact with the consumer device 610 via a user interface 614.

[0161] Network 620 may include any combination of local area and / or wide area networks using both wired and / or wireless communication systems. For example, local area networks may include WiFi and Bluetooth networks. In one embodiment, network 620 uses standard communication technologies and / or protocols. For example, network 620 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of networking protocols used to communicate over network 620 include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged over network 620 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 620 may be encrypted using any suitable technology or technologies.

[0162] One or more third party content providers 630 may communicate with the consumer device 610 via the network 620. In one embodiment, the third party content provider 630 may be an application provider that provides applications for execution as well as content for display on the consumer device 610. New applications, new content, and updates to existing applications and content may be available to run and / or display on the consumer device 610 from the third party content provider 630 via either push-enabled or pull-enabled updates over the network 620. These applications and content may include holographic content tracks stored in the data store 522 for projection and display by the light field display system 612. Content made available to the consumer device 610 may include advertisements, product offers, and product coupons displayed by the light field display system 612 in connection with use of the features of the consumer device 610. For example, holographic content depicting a popular animated character may be downloaded from a third party content provider and stored in the light field display system 612 for use by the user 640 as a "holographic avatar" to represent the user in one or more applications.

[0163] In one or more embodiments, to ensure that unauthorized network access to the consumer device 610 is prevented, the system may use security settings to authenticate network access to the consumer device 610 by entities, establish a secure connection to the consumer device 610, and / or download content to the consumer device 610. In some embodiments, the consumer device 610 may incorporate authentication elements such as passwords and smart cards, or use biometric methods to perform access control when accessing the consumer device 610 via either the network 620 or the user interface 614. Secure access to the user interface 614 may be achieved for any of the possible configurations of the user interface 614, including any of the following: a holographic user interface, a traditional physical user interface or an audio interface that may be part of the sensory feedback system 570, an interface that relies on gestures captured by the tracking system 580 and processed by the tracking module 526, or commands received via the network interface 524. Holographic user interface and "touch" interpretation in LF display systems

[0164] The following is a description of several embodiments of an LF display system for generating a holographic user interface (UI) for a user to interact with a consumer device. The description refers to elements shown in LF display system 500 of FIG.

[0165] In some embodiments, the generated holographic UI is a user interface that includes one or more holographic UI objects that are presented to a user. In some embodiments, the presented holographic UI objects may also be part of a virtual image display that, for example, depicts a virtual display screen image in which the holographic objects appear to exist. In some embodiments, the presented holographic objects and / or virtual images may augment the physical UI on the device. The generated holographic UI objects include, but are not limited to, 3D holographic objects such as "buttons," "knobs," "dials," "keypads," and the like. The generated holographic objects may also include 2D holographic objects, such as a holographic "screen" that displays a holographic menu that includes a list of numerous options. The user may be prompted (e.g., verbally or via a visually displayed prompt) to select one or more options by pointing or touching (e.g., with a finger) a particular option in the holographic menu.

[0166] In some embodiments, the presented holographic UI object may be augmented with haptic stimuli to generate a haptic surface co-located with the surface of the presented holographic object. Thus, for example, the sensory feedback system 570 may project ultrasonic energy, e.g., pressure waves, onto a holographic "button" to generate a haptic surface co-located with the surface displayed in front of it. The tracking system 580 may track the user's movement (e.g., the user's finger movement) toward the holographic button. The movement information may be transmitted by the tracking system 580 to the tracking module 520. The tracking module 526 of the controller 520 may determine that the finger is "touching" or "pressing" the holographic "button" when the tracked finger is within a certain threshold distance of the presented haptic surface. Furthermore, in response to this determination, the LF display assembly 510 may further render the holographic "button" as well as the presented haptic surface at a different location, e.g., move the location of the haptic surface to a new location by moving the holographic "button" as the user "presses."

[0167] In another embodiment, the holographic UI may include presenting a holographic character that greets the user. In some embodiments, various characteristics of the presented holographic character, such as visual, voice, and personality traits, may be configured at least in part by the user. In some embodiments, various characteristics of the presented holographic character may be based on information received from an online system over a network, perhaps in exchange for a transaction fee. In some embodiments, the holographic character may be presented along with the presentation of a holographic UI object. In other embodiments, the holographic character may be presented to the user first, followed by a holographic "menu" display with the holographic UI object. In still other embodiments, all interactions may include verbal voice commands from the user, which are received by an acoustic receiver of the LF display system. In some embodiments, the presented holographic character may be augmented with acoustic stimuli and respond verbally to the user. In some embodiments, the holographic character may be generated or modified using artificial intelligence models. In some embodiments, the holographic character may be generated based on accessed user profile data. Additionally, as the user profile information is updated, the generated holographic character may also be modified based on the changes to the user profile information. In some embodiments, the generated holographic character may be modified based on recognized commands from a user.

[0168] In some embodiments, the tracking module 526 may track information in the holographic object volume and / or the viewing volume in front of the LF display system based on information received from the tracking system 580 and transmit the tracked information to the command library 532. The command library 532 may analyze the received information and determine that a user interface should be displayed to the user using stored heuristics based on the analysis. Additionally, using the user profile information stored by the user profiling module 528, the command library 532 may apply other stored heuristics and determine from the stored mappings that a customized user interface should be generated that includes the display of a particular holographic character greeting the user of the appliance. The command library 532 may issue a computational command to the LF processing engine 530 to display the determined holographic character greeting the user to display in the holographic object volume. Executing this computational command may include the LF processing engine 530 retrieving image data from the data store 522 related to the rendering and display of the holographic character. The command library 532 may also trigger an acoustic transducer in the sensory feedback assembly 570 to transmit an audible greeting that is synchronized with the displayed holographic character. The LF processing engine 530 may render a holographic character that looks at the user and / or makes eye contact with or otherwise interacts with the user using tracking information obtained from the tracking system 580. Verbal responses from the user may be received by an acoustic receiving device (e.g., a microphone or microphone array) disposed in the sensory feedback assembly 570, and this received response data is presented to the controller 520 for further processing.The command library 532 may use the data received from the sensory feedback assembly 570 and apply heuristics and recognition software, such as speech recognition software, to determine, for example, that a timer setting interface should be displayed to the user in response to a verbal utterance by the user.

[0169] In another embodiment, a user may configure the LF display system to always initiate the presentation of a particular customized 3D holographic user interface that is augmented with a display of a particular desired appliance feature, such as a hot water setting option with a microwave. In some embodiments, a device user may configure settings of various appliance features in conjunction with the LF display system to have a particular desired holographic user interface presented for use by the user with the various features. For example, a dryer LF display system may be configured by a user such that a particular holographic UI is presented when the user selects to use the dryer on a "heavy" load, providing more selectable options than when the user selects to use the dryer on a "normal" setting.

[0170] In other embodiments, the holographic user interface includes additional or fewer functions than those described herein. Similarly, functionality may be distributed among modules and / or different entities within LF display system 500 in ways different from those described herein.

[0171] The tracking system 580 may track the user's movements using a camera within the tracking system 580. The user's movements may be transmitted by the tracking system 580 to the controller 520 for touch interpretation. In some embodiments, touch interpretation is performed as follows: The tracking system 580 first tracks the user's movements as the user approaches (e.g., with one or more fingers) the presented holographic object. The tracking system 580 transmits the tracked movement information to the controller 520. The tracking module 526 determines that the tracked fingers are within a threshold distance from the presented holographic object and, in response, determines that the user is "touching" the holographic object. In other embodiments, touch interpretation may include tracking other types of movements with respect to the presented holographic object by the tracking system 580. The tracked movements by the user may include, but are not limited to, determining that the user is performing one or more of the following with respect to one or more presented holographic objects: repositioning, moving, pressing, pressing, dialing, rotating, entering, poking, moving a switch, etc.

[0172] In some embodiments, the sensory feedback system 570 may sense the "touch" of a volumetric tactile surface that augments the displayed holographic object. This determination may be based on receiving input from a pressure sensor of the sensory feedback system 570 that detects a threshold change caused by the presence of a finger or other body part in the ultrasonic energy required to generate the volumetric tactile surface. This information may be transmitted to the controller 520 for further analysis. A sense of proximity may also be achieved by monitoring the driving force applied to the ultrasonic transducers required for the projection of the volumetric tactile surface and inferring the presence of a physical object by detecting a change in the threshold of this driving force. Customized Holographic Displays in Consumer Devices

[0173] In some embodiments, the LF display system may use such a tracking system in combination with user profile information, such as obtained by the user profiling system 528 of FIG. 5, to generate customized interactive experiences. Thus, in a holographic UI generation application, the generated holographic "buttons" may be different sizes based on the user. For example, the buttons may be smaller for children and larger in size with bright colors to make them easier to see for older users. Additionally, in some embodiments, the LF display system may be customized to display specific content for specific users. Thus, for example, the LF display system may be customized to display a holographic "button" for one user and a holographic "dial" for another user. In other embodiments, the LF display module may display a holographic character that is customized to interact with (e.g., see, speak, hear, touch, or some combination thereof) the device user. Additionally, the tracking system may track the user's gaze. The tracking system may provide gaze information to the LF display module to re-render the holographic content to make eye contact with the user interacting with the device.

[0174] In some additional embodiments, the user profiling system may build a user profile for each user of the appliance to store the user's preferences or other user characteristics. The user profile may be built using the tracking system or another component of the LF display system, such as the controller (i.e., controller 520), and stored in a data store (i.e., data store 530). A user profile may be created for each user using all data collected by the user profiling system. Additionally, the tracking system or other components, such as the sensory feedback assembly, may provide data to the user profile to be stored in the data store.

[0175] The LF display system may utilize a user profiling system to personalize holographic content for the user during each subsequent use of the appliance. For example, the LF display system may address the user by name (e.g., visually or audio). In other examples, the LF display system generates holographic content corresponding to a user's response to previously generated holographic content. For example, if Tuesday night is regular roast chicken night and a data store has this information stored, the LF display system in the oven may generate a holographic character on Tuesday evening posing the question, "Should I turn the oven to 350 degrees?"

[0176] FIG. 7 is a diagram of an embodiment of a LF display system 700 implemented as part of a consumer device 710, according to one or more embodiments. The LF display system 700 is one embodiment of the LF display system 500 described in FIG. 5. The LF display system 700 is comprised of a consumer device 710 (e.g., an appliance) with an LF display module 720. In one or more embodiments, the device 710 is accompanied by one or more control panels 730, which may include physical features (such as buttons) and / or a 2D digital display (including a 2D touch screen) for a user to access the appliance. In addition, the device 710 also includes an LF display assembly (e.g., the LF display assembly 510 of FIG. 5) for generating holographic content to a user of the device. The LF display system 700 may also include any combination of the other components of the LF display system 500 shown in FIG. 5, such as the controller 520, the user profiling module 528, the sensory feedback assembly 570, and the tracking system 580.

[0177] The LF display system may present holographic content, including at least one holographic object, to a user using one or more LF display modules. The holographic object may be three-dimensional (3D), two-dimensional (2D), or some combination thereof. Additionally, the holographic object may be multi-colored (e.g., full color). In FIG. 7, the holographic object volume 740 extends both in front of and behind the display module 720 (e.g., similar to that shown in FIG. 1). However, for ease of illustration, the portion of the holographic object volume 740 that is behind the consumer device 710 is omitted from FIG. 7. The viewing volume 750 represents a display space in which a user may view the holographic object presented by the LF display module 720. The holographic objects 762 and 764 may be located anywhere within the holographic object volume 740, which encompasses all light rays that originate from the LF display module 720 and converge to create the holographic object, with a set of convergence points located in front of the display area of ​​the device. Viewer A 772 and viewer B 774 may view the displayed holographic objects 762 and 764. The displayed holographic objects 762 and 764 may respond to user movements. For example, a holographic dial or button may move in response to a user's hand movements that may be sensed by a tracking system, rotate with user gestures sensed by the tracking system, rotate according to voice commands received by a sensory feedback system (e.g., 570), and move with other user input.

[0178] The holographic objects 762 and 764 may also have tactile surfaces projected near or coincident with their surfaces. These tactile surfaces may respond to the movements of the user. For example, contact with the surfaces may be sensed by a tracking system and trigger an appropriate tactile response in the sensory feedback system (e.g., an appropriate corresponding tactile response corresponding to the position of the user's finger or hand). Thus, for example, the movement of finger 770 towards holographic object 762, as well as the "touch" of finger 770 on the tactile surface of holographic object 762 may be established by the tracking system of the LF display system.

[0179] As described above with respect to customized holographic displays in consumer devices, when tracking system 580 tracks the presence of viewer A 772, the configured settings may provide a customized display of holographic object 762 (i.e., the holographic "button") (only) to viewer 772. Thus, if viewer B 774 is not present, the display of holographic object 764 may not be presented. Similarly, upon sensing the presence of viewer 774, the configured settings may provide a customized display of holographic object 764 (i.e., the holographic "dial") (only) to viewer 774.

[0180] Other exemplary embodiments of LF display systems are described below. The description of the embodiments is added for illustrative purposes and is not intended to be an exhaustive description of LF display systems in all possible appliances. Exemplary embodiment: LF display system in home assistant

[0181] The following is a description of several embodiments of an LF display system as part of a home assistant device (such as AMAZON's ALEXA or GOOGLE's HOME). The description refers to elements shown in LF display system 500 of FIG. Such a home assistant system may also function as a home automation system that can control home lighting, indoor climate, home entertainment systems, appliances, cameras, door locks, or other devices.

[0182] In some embodiments, the sensory feedback system 570 may receive sensory data detected near the device (e.g., audio data such as voice commands from the user captured by an acoustic microphone in the sensory feedback system 570), or the tracking system 580 may receive visual information about the user's movements captured by a camera. This received information is transmitted to the controller 520 in the home assistant light field display system 500. The controller 520 may forward this information to the command library 532 for further processing and interpretation. The command library 532, in conjunction with the tracking module, may interpret the received information together with the user profile information stored by the user profiling system 528, and may apply stored heuristics, if necessary, to interpret the user's gestures and send computational commands to the LF processing engine to present the above-mentioned holographic user interface to the user.

[0183] The displayed holographic user interface may be configured as desired by the user to facilitate interaction with the home assistant. The holographic user interface presented to the user is integrated with the home assistant to allow for the display and execution of all functionality provided by the home assistant. Examples of functionality provided by the LF display system 500 integrated with the home assistant include, but are not limited to, a holographic user interface including a holographic and tactile screen of menu selections displayed to the user that responds to touch commands from the user, and a holographic display of home security, lighting, or appliance settings that facilitates the user reconfiguring these settings via the displayed holographic interface.

[0184] A holographic user interface to a consumer device (e.g., a home assistant) may feature a holographic character, as described above. The holographic character may have various characteristics that may be modified to suit preferences, such as appearance (e.g., a celebrity appearance), clothing (or none), personality, voice (e.g., accent or celebrity voice), temperament, language (e.g., amount of offensive language used), or other characteristics. The holographic character may have a CG character (e.g., a superhero), a celebrity such as an actor or athlete, an optional appearance of one of many available avatars, or another appearance. The appearance of the holographic character may be accompanied by a recognizable home assistant voice (e.g., a celebrity voice matching the appearance) and may be easily modified by the user. The holographic character may be generated and personalized by an AI model. The holographic character may access a user profile or social media account to make it more appealing to the user. For example, the holographic character may provide playful feedback based on characteristics observed in photos and discussions displayed on the user's social media account or provide small talk based on calendar reminders for days of scheduled events. For the user's enjoyment, the holographic character can be in a bad mood or a good mood depending on the user's selection. Many options regarding the holographic character's traits to display and experience on a consumer device such as a home assistant or home automation device may be downloadable over the network 620 via a third party content provider 630 such as an online store, possibly in exchange for a transaction fee.

[0185] In other embodiments, the home assistant holographic character may be provided as a friendly virtual pet to a recognized user, or may act as a sentry animal (a guard dog or noisy parrot) that acts protectively when an unrecognized person is observed in a part of the home. Exemplary embodiment: In-oven LF display system

[0186] The following is a description of several embodiments of an LF display system as part of a consumer appliance, such as a domestic oven, which refers to elements shown in LF display system 500 of FIG.

[0187] In some embodiments, within the light field display system 500 within an appliance, a tracking module 526 may receive images captured in front of the LF display system (e.g., within the holographic object volume and / or within the viewing volume, etc.) from a tracking system 580. The tracking module 526 may determine the presence of a user. A command library 532 within the controller 520 may use the tracking information received from the tracking module 526 along with user profile information stored by a user profiling system 528, apply heuristics, and send computational commands to the LF display assembly 510 to present the holographic user interface described above to the user.

[0188] As previously mentioned, in some embodiments, the LF display system 500 may also augment the presented holographic content with other sensory content (e.g., touch, temperature). Thus, for example, an LF display system 500 associated with an oven may present a holographic image of a loaf of bread baking in the oven. The holographic "loaf of bread" may be augmented with haptic stimuli to generate a haptic surface that is co-located with the top surface of the presented holographic "loaf of bread." Thus, the sensory feedback system 570 may project acoustic energy, e.g., pressure waves, to generate a co-located haptic "loaf of bread" surface. Additionally, the quality of the co-located haptic "loaf of bread" surface may be periodically altered such that the haptics presented to the user reflect the "doneness" state of the actual freshly baked loaf of bread in the oven.

[0189] Thus, in some embodiments, the LF display system may obtain parameter values, such as time, temperature, etc., from the appliance control system and use the parameter values ​​along with data in data store 522 regarding the toast time of a loaf of bread to adjust the intensity of the presented tactile surface, such that the intensity of the presented surface may represent the "doneness" of the bread in the appliance. For example, the LF display system may use the obtained information to control ultrasonic emitters in sensory feedback assembly 570 to generate appropriately varying pressure waves such that the intensity of the rendered tactile surface at a given time reflects the "doneness" of an actual loaf of bread in the oven at a given time, in response to a "touch" from the user.

[0190] In some embodiments, tracking system 580 may track the user's movements (e.g., the user's finger movements) toward the presented holographic loaf of bread. The movement information may be transmitted by tracking system 580 to controller 520. Controller 520 may determine that the tracked finger is "touching" or "pressing" the holographic "loaf of bread" when the tracked finger is within a certain threshold distance of the presented tactile surface of the loaf of bread. Furthermore, in response to this determination, LF display assembly 510 may further render the holographic "loaf of bread" as well as the presented tactile surface at a different location, e.g., move the location of the tactile surface to a new location by moving the holographic "loaf of bread" as the user "presses". Again, the amount of movement of the "surface" to the new location may reflect the "doneness" of the loaf of bread.

[0191] Some embodiments may use input from the sensory feedback assembly 570 to determine how to display the holographic object. The display of the holographic object may be modified in response to receiving a value of a parameter at the LF display system 500. Examples of parameters may include time, temperature, pressure, force, audio decibel levels, and the like, or combinations thereof. For example, a roast turkey may be depicted as a holographic object that slowly changes color as the baking time progresses in an oven based on a timer or temperature sensor input. In another embodiment, if the LF display system 500 includes a heating unit as part of the sensory feedback assembly 570, the temperature around the holographic representation of the roast turkey may increase as the time or temperature increases. Data may be obtained from additional sensors (such as a temperature sensor) located in the oven to generate the holographic object, and may also be obtained from data stored in a data store (such as recipe information).

[0192] In some embodiments, operational choices made by a user may affect the display of holographic content. For example, if the LF display system determines that the user has made an operational choice to use the oven on a low temperature 10 hour cycle, then only at the end of the 10 hour cycle may the LF display system present a holographic depiction of the oven's contents. Exemplary embodiment: LF display system in refrigerator

[0193] Some embodiments of the LF display system are found in appliances such as a home refrigerator. In some embodiments, the tracking module 526 may track information in the holographic object volume and / or the viewing volume in front of the LF display system based on information received from the tracking system 580 (as shown in FIG. 5 ) and send this tracked information to the information to command library 532. The command library 532 uses the tracking information received from the tracking module 526 along with user profile information stored by the user profiling system 528 to apply heuristics in the command library 532 and send computational commands to the LF processing engine 530. The LF processing engine 530 executes the computational commands and causes the LF display assembly to project a holographic display of the refrigerator's contents. In some embodiments, displaying the refrigerator's contents includes an internal camera sensor located within the refrigerator to provide image data of the refrigerator's contents to the LF display system 500. In other embodiments, the LF display system 500 receives user commands, either manually entered by a user into a physical control panel or based on recognition of user gestures by the tracking system 580, to generate the holographic display. For example, tracking module 526 may detect a user gesture heuristically mapped by command library 532 and send a computational command to the LF display system to generate a holographic display depicting only a portion of the refrigerator's contents (e.g., the amount of ice currently available in the freezer's built-in ice maker, or, as another example, the contents of the refrigerator door shelf, etc.). Continuing with the example, generating the desired particular holographic display may include acquiring the desired image data using a camera sensor in sensory feedback system 570 and rendering a projected holographic object from the display surface based on the acquired image.

[0194] As previously described in some embodiments, the LF display system may include elements that enable the system to simultaneously emit at least one type of energy and simultaneously use image tracking information as well as image information from within the appliance to respond to and create an interactive experience for the user. Thus, for example, in one embodiment, the LF display system 500 may display the contents of a refrigerator. Upon sensing that a user has "moved" a particular depicted holographic object, such as a gallon of milk, the system may change the presentation to depict another holographic object, such as a bag of salad vegetables behind the gallon of milk in the refrigerator. Appliance service and maintenance support

[0195] Additional embodiments using the LF display system include facilitating service and maintenance functions within a consumer device (e.g., an appliance). The LF display system may have an access control policy to allow only certain types of holographic objects to be displayed to a user of the consumer device under normal operation of the consumer device. However, a qualified consumer device service technician may obtain secure access within the consumer device system to activate a display of holographic objects depicting internal mechanical elements within the consumer device during service maintenance and / or repair of the consumer device (e.g., an appliance such as a dishwasher). The display may be generated using various sensor data obtained from sensors located within the consumer device. Thus, for example, a qualified refrigerator technician may be able to securely activate a holographic display of the refrigerator's ice maker while performing service or maintenance work directed to the refrigerator's ice maker function, including internal mechanical parts not easily visible to a refrigerator user, such as the condenser fan and motor assembly, water tubes, ice grid assembly, etc., and circuitry associated with the refrigerator's ice maker.

[0196] 8 is a flow chart illustrating a process for enabling user interaction with a consumer device using a light field display system, according to one or more embodiments. In one embodiment, the process of FIG. 8 is performed by light field display system 500 (as shown in FIG. 5). Other entities may perform some or all of the steps of the process in other embodiments. Similarly, embodiments may include different and / or additional steps, or perform steps in a different order.

[0197] The light field display system 500 receives 810 commands from a user. The user commands may be received via physical controls, such as physical buttons that can be pressed, physical dials that can be rotated, and physically touchable screens, all of which are located on a consumer device and are part of the sensory feedback system 570. In another example, the user commands may be received as voice commands at an acoustic receiver, such as an acoustic microphone, that is part of the sensory feedback system 570. The received user commands are relayed via the controller to the command library 532 for further interpretation and analysis. The user commands may be received through the user's interaction with a holographic object, such as a displayed holographic UI. The user commands may also be received via the tracking system 580, which is tracking the user via instructions from the tracking module 526. The user commands may be received when the tracking system 580 detects that the user touches a tactile holographic surface, such as a holographic "button" on the displayed holographic UI. The received user commands are relayed via the controller to the command library 532 for further interpretation and analysis. Similarly, the user commands may be in the form of gestures or movements made by the user. User movements or gestures may be identified by the tracking system 580 and relayed to the command library 532 for analysis and interpretation. User commands may also be received over a communications network at the consumer device and transmitted to the command library 532 for analysis and interpretation.

[0198] The light field display system 500 processes 820 the user command to determine a computational command. The command library 532 of the light field display system 500 interprets the received user command and, in conjunction with the tracking module, may apply stored heuristics to interpret user gestures, if necessary, possibly in conjunction with user profile information stored by the user profiling system 528, to determine a computational command based on the received user command. One example of a determined computational command is a command to generate more holographic content for display by a user of the consumer device. Another example may change the generated holographic content. The determined computational command may activate a sensor in the LF display system, such as, for example, an image sensor to track the user's gaze. The determined computational command may also activate a control in the consumer device. For example, the computational command may lower the temperature setting of a refrigerator.

[0199] The light field display system generates 830 holographic content for display based on the computational command. The LF processing engine 530 of the light field display system 500 receives the computational command determined from the command library 532 and generates 830 holographic content for display based on the computational command. Thus, for example, the computational command may instruct the LF processing engine 530 to generate a holographic display of a loaf of bread baking in an oven. In response, the LF processing engine 530 may first determine characteristics of the holographic display, such as the color and tactile characteristics of the depicted loaf of bread, depicting the "doneness" of the baked loaf of bread, and retrieve stored user profile characteristics from the data store 522 before generating the required holographic content.

[0200] The light field display system 500 displays 840 the generated holographic content. The controller 520 sends display instructions to project the holographic content from the LF processing engine 530 to the LF display module 512. The LF display module 512 displays the holographic content to a user. Additional configuration information

[0201] The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will recognize that numerous modifications and variations are possible in light of the above disclosure.

[0202] Some portions of this specification describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they are implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has proven convenient at times to refer to arrangements of these operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0203] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented in a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0204] The embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, that may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0205] The embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information resulting from the computing processes, which information is stored in a non-transitory tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein. Finally, the language used herein has been selected primarily for readability and instructional purposes, and may not have been selected to define or limit the subject matter of the present disclosure. Accordingly, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based on this specification. Accordingly, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the disclosure, which is set forth in the following claims. (Item 1) 1. A consumer device comprising: A controller configured to generate holographic content; an LF display assembly including one or more LF display modules configured to present the generated holographic content to a user of the consumer device; and a command interface configured to recognize one or more commands from the user; 1. A consumer device comprising a light field (LF) display system comprising: (Item 2) Item 10. The consumer device of item 1, wherein the controller is configured to update the generated holographic content in response to changes in an operational state of the consumer device. (Item 3) 2. The consumer device of claim 1, wherein the generated holographic content comprises a holographic user interface (UI) having one or more holographic features for controlling operation of the consumer device, the command interface being further configured to recognize the one or more commands received from the user via the generated holographic UI. (Item 4) 4. The consumer device of claim 3, wherein the command interface is further configured to receive a command from the user via the holographic UI based on a determination that the user is within a threshold distance of the one or more holographic features in the presented holographic UI. (Item 5) 2. The consumer device of claim 1, wherein the one or more recognized commands cause a change in an operational state of the consumer device. (Item 6) The LF display system comprises: Item 13. The consumer device of item 1, further comprising a sensory feedback system comprising at least one sensory feedback device configured to provide sensory feedback when the holographic content is presented. (Item 7) 7. The consumer device of claim 6, wherein the command interface is further configured to recognize the provided sensory feedback as one or more commands from the user. (Item 8) 7. The consumer device of claim 6, wherein the sensory feedback system comprises tactile feedback, audio feedback, aroma feedback, temperature feedback, pressure feedback, force feedback, or any combination thereof. (Item 9) 7. The consumer device of claim 6, wherein the sensory feedback system further comprises one or more of buttons, dials, knobs, keypads, other input hardware, or any combination thereof. (Item 10) 7. The consumer device of claim 6, wherein the sensory feedback system further comprises a microphone and voice recognition software. (Item 11) The LF display system comprises: a tracking system, the tracking system including one or more tracking devices configured to obtain information about the user of the consumer device; Item 10. The consumer device of item 1, wherein the controller is configured to generate the holographic content for the user based in part on the information obtained by the tracking system. (Item 12) Item 12. The consumer device of item 11, wherein the information acquired by the tracking system includes any combination of the user's location, the user's movement, the user's gestures, the user's facial expressions, the user's gaze, the user's gender, and the user's age. (Item 13) Item 12. The consumer device of item 11, wherein the controller is further configured to update the generated holographic content based on the information obtained by the tracking system. (Item 14) Item 12. The consumer device of item 11, wherein the controller is configured to generate the holographic content using the information acquired by the tracking system and an artificial intelligence model. (Item 15) Item 12. The consumer device of item 11, wherein the tracking system includes any combination of sensors including a light field camera, one or more 2D image sensors, and a depth sensor. (Item 16) Item 16. The consumer device of item 15, wherein generating the holographic content includes capturing data from the one or more sensors in the tracking system and generating the holographic content based on the captured data. (Item 17) Item 10. The consumer device of item 1, wherein the controller is further configured to modify holographic content based on one or more recognized commands from the user. (Item 18) The controller: Retrieving at least a portion of the holographic content in a vectorized format from an online system via a network interface; Item 14. The consumer device of item 1, further configured to convert the vectorized format of the holographic content into a rasterized format for presentation on the LF display assembly. (Item 19) 2. The consumer device of claim 1, wherein the LF display system is configured to modify the presentation of the holographic content based on environmental characteristics at a location where the holographic content is presented to the user. (Item 20) 2. The consumer device of claim 1, wherein the LF display system is configured to modify the presentation of the holographic content based on a value of a parameter received at the LF display system. (Item 21) accessing the information obtained by the tracking system; and processing said information to identify said user; generating a user profile for said user; and a user profiling module configured to: Item 12. The LF display system of item 11, wherein the controller is further configured to generate the holographic content for the user based in part on the user profile. (Item 22) 22. The LF display system of claim 21, wherein the controller is further configured to generate the holographic content using the user profile and an artificial intelligence model. (Item 23) The user profiling module comprises: further configured to update the user profile using information from the user's social media account; Item 22. The LF display system of item 21, wherein the controller is configured to generate the holographic content based in part on the updated user profile. (Item 24) Item 1, the consumer device further comprising an ultrasonic energy projection device configured to generate a volumetric tactile surface within a threshold distance of or matching the surface of the presented holographic object. (Item 25) 25. The consumer device of claim 24, further comprising adjusting a resistance of the generated volumetric tactile surface to a user touch based on a value of a parameter received at the controller. (Item 26) Item 12. The consumer device of item 11, further comprising an ultrasonic energy projection device configured to generate a volumetric tactile surface generated based in part on the information acquired by the tracking system. (Item 27) 27. The consumer device of claim 26, wherein the ultrasonic energy projection device is further configured to update the generated volumetric tactile surface in response to information obtained by the tracking system. (Item 28) 27. The consumer device of claim 26, wherein the ultrasonic energy projection device is further configured to generate the volumetric tactile surface using the information obtained by the tracking system and artificial intelligence model. (Item 29) Item 13. The consumer device of item 1, wherein the controller is further configured to augment the generated holographic content with sensory content including tactile stimuli, acoustic stimuli, thermal stimuli, olfactory stimuli, or any combination thereof. (Item 30) 2. The consumer device of claim 1, wherein the holographic content is presented in conjunction with an application running on the consumer device. (Item 31) Item 31. The consumer device of item 30, wherein the presented holographic content is based on content received from an online source. (Item 32) Item 32. The consumer device of item 31, wherein the presented holographic content is based on content received from an online source in response to operational selections made by the user. (Item 33) 2. The consumer device of claim 1, wherein the presented holographic content includes a holographic character. (Item 34) Item 34. The consumer device of item 33, wherein characteristics associated with the presented holographic character are configured, at least in part, by the user. (Item 35) 35. The consumer device of claim 34, wherein the characteristics include visual, voice, and personality characteristics of the presented holographic character. (Item 36) Item 35. The consumer device of item 34, wherein the characteristics associated with the presented holographic character are retrieved from an online system via a network. (Item 37) Item 37. The consumer device of item 36, wherein the features are retrieved from the online system via the network in exchange for a transaction fee. (Item 38) 35. The consumer device of item 34, wherein the holographic character is presented in association with a home assistant device. (Item 39) Item 34. The consumer device of item 33, further comprising a sensory feedback system comprising at least one sensory feedback device configured to provide sensory feedback when the holographic character is presented. (Item 40) 40. The consumer device of claim 39, wherein the sensory feedback comprises tactile feedback, audio feedback, aroma feedback, temperature feedback, or any combination thereof. (Item 41) 40. The consumer device of claim 39, wherein the sensory feedback includes a volumetric tactile projection system configured to provide a tactile surface matching a surface of the holographic character with which the one or more users can interact via touch. (Item 42) a tracking system comprising one or more tracking devices configured to obtain information about the user; Item 34. The consumer device of item 33, wherein the controller is configured to generate the holographic character of the user based on the information obtained by the tracking system. (Item 43) Item 43. The consumer device of item 42, wherein the information obtained by the tracking system includes any of the user's location, the user's movement, the user's gestures, the user's facial expressions, the user's gaze, the user's age, and the user's gender. (Item 44) Item 43. The consumer device of item 42, wherein the tracking information of the user includes the user's gaze, and the LF display assembly is configured to update the eyes of the holographic character to maintain eye contact with the user's gaze. (Item 45) Item 43. The consumer device of item 42, wherein the controller is configured to generate the holographic character using the information acquired by the tracking system and an artificial intelligence model. (Item 46) accessing the information obtained by the tracking system; and processing said information to identify said user; generating a user profile for said user; and a user profiling module configured to: Item 43. The consumer device of item 42, wherein the controller is configured to generate the holographic character of the user based in part on the user profile. (Item 47) Item 47. The consumer device of item 46, wherein the controller is configured to generate the holographic character using the user profile and an artificial intelligence model. (Item 48) The user profiling module comprises: further configured to update the user profile using information from the user's social media account; Item 47. The consumer device of item 46, wherein the controller is configured to generate the holographic character based in part on the updated user profile. (Item 49) Item 49. The consumer device of item 48, wherein the controller is configured to generate the holographic character using the updated user profile and an artificial intelligence model. (Item 50) Item 34. The consumer device of item 33, wherein the controller is further configured to modify the holographic character based on one or more recognized commands from the user. (Item 51) The LF display system comprises: receiving security credentials from a user; authenticating the received security credentials; and enabling secure access to one or more functions of the consumer device; enabling generation of specific holographic content in response to said authentication; Item 1. The consumer device of item 1, further comprising a security module having an authentication module configured to: (Item 52) The security module includes: receiving data captured by an internal sensor within the consumer device; providing the received data to the controller; and 52. The consumer device of item 51, further configured to: (Item 53) Item 53. The consumer device of item 52, wherein the controller is configured to generate the holographic content based on the received data from the internal sensor. (Item 54) Item 54. The consumer device of item 53, wherein the controller is configured to generate the holographic content using, in part, an artificial intelligence model.

Claims

1. 1. A consumer device comprising: A controller configured to generate holographic content; a LF display assembly including one or more LF display modules configured to present the generated holographic content to a user of the consumer device; and a command interface configured to recognize one or more commands from the user; A light field display system (LF display system) having The LF display assembly is configured to direct energy from multiple energy source locations along different propagation paths to multiple converging points in space that form a holographic representation of an object in the holographic content. Consumer devices.

2. The consumer device of claim 1 , wherein the controller is configured to update the generated holographic content in response to changes in an operational state of the consumer device.

3. 3. The consumer device of claim 1 or 2, wherein the generated holographic content comprises a holographic user interface (UI) with one or more holographic features for controlling operation of the consumer device, and the command interface is further configured to recognize the one or more commands received from the user via the generated holographic UI.

4. 4. The consumer device of claim 3, wherein the command interface is further configured to receive commands from the user via the holographic UI based on a determination that the user is within a threshold distance of the one or more holographic features in the presented holographic UI.

5. The consumer device of claim 1 , wherein the one or more recognized commands cause a change in an operational state of the consumer device.

6. The LF display system comprises:

6. The consumer device of claim 1 , further comprising a sensory feedback system comprising at least one sensory feedback device configured to provide sensory feedback when the holographic content is presented.

7. The consumer device of claim 6 , wherein the command interface is further configured to recognize the provided sensory feedback as one or more commands from the user.

8. 8. The consumer device of claim 6 or 7, wherein the sensory feedback system comprises haptic feedback, audio feedback, aroma feedback, temperature feedback, pressure feedback, force feedback, or any combination thereof.

9. The consumer device of claim 1 , wherein the controller is further configured to modify holographic content based on one or more recognized commands from the user.

10. The controller: Retrieving at least a portion of the holographic content in a vectorized format from an online system via a network interface; 10. The consumer device of claim 1, further configured to convert the vectorized format of the holographic content into a rasterized format for presentation on the LF display assembly.

11. 11. The consumer device of claim 1, wherein the LF display system is configured to modify the presentation of the holographic content based on environmental characteristics at a location where the holographic content is presented to the user.

12. 12. The consumer device of claim 1, wherein the LF display system is configured to modify the presentation of the holographic content based on values ​​of parameters received at the LF display system.

13. 13. The consumer device of claim 1 , further comprising an ultrasonic energy projection device configured to generate a volumetric tactile surface within a threshold distance of or matching a surface of the presented holographic object.

14. The consumer device of claim 13 , further comprising: adjusting a resistance of the generated volumetric tactile surface to a user touch based on a value of a parameter received at the controller.

15. a tracking system including one or more tracking devices configured to obtain information regarding the user of the consumer device; an ultrasonic energy projection device configured to generate a volumetric tactile surface that is generated based in part on the information acquired by the tracking system; 15. The consumer device of claim 1 , further comprising:

16. The consumer device of claim 15 , wherein the ultrasonic energy projection device is further configured to update the generated volumetric tactile surface in response to information obtained by the tracking system.

17. 17. The consumer device of claim 15 or 16, wherein the ultrasonic energy projection device is further configured to generate the volumetric tactile surface using the information obtained by the tracking system and an artificial intelligence model.

18. 18. The consumer device of claim 1, wherein the controller is further configured to augment the generated holographic content with sensory content including tactile stimuli, acoustic stimuli, thermal stimuli, olfactory stimuli, or any combination thereof.

19. 19. The consumer device of claim 1, wherein the holographic content is presented in conjunction with an application running on the consumer device.

20. 20. The consumer device of claim 19, wherein the presented holographic content is based on content received from an online source.

21. 21. The consumer device of claim 20, wherein the presented holographic content is based on content received from an online source in response to operational selections made by the user.

22. 22. The consumer device of claim 1, wherein the presented holographic content comprises a holographic character.

23. 23. The consumer device of claim 22, wherein characteristics associated with the presented holographic character are configured, at least in part, by the user.

24. 24. The consumer device of claim 23, wherein the characteristics include visual, audio, and personality characteristics of the presented holographic character.

25. 25. A consumer device as claimed in claim 23 or 24, wherein the characteristics associated with the presented holographic character are retrieved from an online system over a network.

26. 26. The consumer device of claim 25, wherein the features are retrieved from the online system over the network in exchange for a transaction fee.

27. 27. The consumer device of claim 23, wherein the holographic character is presented in association with a home assistant device.

28. 28. The consumer device of any one of claims 22 to 27, further comprising a sensory feedback system comprising at least one sensory feedback device configured to provide sensory feedback when the holographic character is presented.

29. 30. The consumer device of claim 28, wherein the sensory feedback comprises haptic feedback, audio feedback, aroma feedback, temperature feedback, or any combination thereof.

30. 30. The consumer device of claim 28 or 29, wherein the sensory feedback includes a volumetric tactile projection system configured to provide a tactile surface matching a surface of the holographic character with which one or more of the users may interact via touch.

31. 31. The consumer device of any one of claims 22 to 30, wherein the controller is further configured to modify the holographic character based on one or more recognized commands from the user.

32. The LF display system comprises: receiving security credentials from a user; authenticating the received security credentials; and enabling secure access to one or more functions of the consumer device; enabling generation of specific holographic content in response to said authentication; 32. The consumer device of claim 1, further comprising a security module comprising an authentication module configured to:

33. The security module includes: receiving data captured by an internal sensor within the consumer device; providing the received data to the controller; and 33. The consumer device of claim 32, further configured to:

34. 34. The consumer device of claim 33, wherein the controller is configured to generate the holographic content based on the received data from the internal sensor.

35. 35. The consumer device of claim 34, wherein the controller is configured to generate the holographic content using, in part, an artificial intelligence model.

Images