Reducing current drain in AR / VR display systems

Abstract

To provide power management in virtual reality and augmented reality systems.SOLUTION: In some embodiments, eye tracking is used on an AR display system or VR display system to determine if a user of the display system is blinking or otherwise cannot see. In response, current drain or power usage of a display associated with the display system may be reduced, for example, by dimming or turning off a light source associated with the display, or by configuring a graphics driver to skip a designated number of frames or reducing a refresh rate for a designated period of time.SELECTED DRAWING: None

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 304,098, filed Mar. 4, 2016, which is hereby incorporated by reference in its entirety.

[0002] (Field) The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to power management within virtual reality and augmented reality systems.

Background Art

[0003] (Description of Related Art) Modern computing and display technologies have facilitated the development of systems for so - called “virtual reality” or “augmented reality” experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner such that they appear or can be perceived as if they were real. Virtual reality, i.e., “VR” scenarios, typically involve the presentation of digital or virtual image information without transparency to other actual real - world visual inputs, and augmented reality, i.e., “AR” scenarios, typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user.

Summary of the Invention

Means for Solving the Problems

[0004] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to the desirable attributes disclosed herein. Various exemplary systems and methods are provided below.

[0005] Embodiment 1: A display system with reduced power usage, an inward - facing sensor, a display, A processing electronic device that communicates with inward-facing sensors and a display, Using inward-facing sensors, changes in the user's eye status are detected, Based on the detection of changes in the user's eye status, the current drain of the display system is reduced. Processing electronic equipment configured to perform A display system equipped with these features.

[0006] Embodiment 2: The display system according to Embodiment 1, wherein the change in the user's eye status is blinking or saccading.

[0007] Embodiment 3: A display system according to any one of Embodiments 1-2, wherein the display comprises a light source, and reducing the current drain of the display includes dimming the light source of the display.

[0008] Embodiment 4: A display system according to any one of Embodiments 1-2, wherein the display comprises a light source, and reducing the current drain of the display includes turning off the light source.

[0009] Embodiment 5: A display system according to any one of Embodiments 1-4, wherein reducing the current drain of the display includes configuring a graphics driver associated with the display to reduce the amount of power consumed by the display.

[0010] Embodiment 6: The display system according to Embodiment 5, wherein the graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the length of time the eye blinks or saccades.

[0011] Embodiment 7: The display system according to any one of Embodiments 1-6, wherein the display comprises an LCD display.

[0012] Embodiment 8: The display system is the display system according to any one of Embodiments 1-7, comprising an augmented reality or virtual reality display.

[0013] Embodiment 9: A display system according to any one of Embodiments 1-8, wherein the inward-facing sensor comprises a camera.

[0014] Embodiment 10: The display system according to any one of Embodiments 1-9, wherein the inward-facing sensor comprises an eye-tracking camera.

[0015] Embodiment 11: A display system according to any one of Embodiments 1-10, wherein the processing electronic equipment is configured to reduce the current drain of the display by reducing the refresh rate associated with the display.

[0016] Embodiment 12: A display system according to any one of Embodiments 1-11, further comprising a graphics driver, which reduces the current drain of the display system, thereby reducing the power consumption of the graphics driver.

[0017] Embodiment 13: A method for reducing the power consumption of a display system, A step of using an inward-facing sensor to detect changes in the user's eye status, A step of reducing the current drain of the display system based on when a change in the user's eye status is detected. Methods that include...

[0018] Embodiment 14: The method according to Embodiment 13, wherein the change in the user's eye status is blinking or saccadic.

[0019] Embodiment 15: The display system includes a light source, and the step of reducing the current drain of the display system includes the step of dimming the light source of the display system, according to the method described in any of Embodiments 13-14.

[0020] Embodiment 16: The display system includes a light source, and the step of reducing the current drain of the display system includes the step of shutting off the light source of the display, according to the method described in any of Embodiments 13-14.

[0021] Embodiment 17: The step of reducing the current drain of the display system includes the step of configuring the graphics driver associated with the display system to reduce the amount of power consumed by the display system, according to the method described in any of Embodiments 13-16.

[0022] Embodiment 18: The graphics driver is configured to skip a specified number of frames, and the specified number of frames is based on the length of a blink or the length of time the eye is not visible, according to the method described in Embodiment 17.

[0023] Embodiment 19: The graphics driver is configured to reduce the amount of power consumed by the display system over a specified period based on the length of a blink or the length of time the eye is not visible, according to the method described in Embodiment 17.

[0024] Embodiment 20: The display system includes an LCD display, according to the method described in any of Embodiments 13-19.

[0025] Embodiment 21: The display system includes an augmented reality or virtual reality display, according to the method described in any of Embodiments 13-20.

[0026] Embodiment 22: The inward-facing sensor includes an eye-tracking camera, according to the method described in any of Embodiments 13-21.

[0027] Embodiment 23: The method according to any one of Embodiments 13-22, wherein the step of reducing the current drain of the display system includes the step of reducing the refresh rate associated with the display.

[0028] Embodiment 24: The method according to any one of Embodiments 13-23, wherein the step of reducing the current drain of the display system includes the step of reducing the power consumption of the graphics driver.

[0029] Embodiment 25: A display system, A camera facing inward, The display and A hardware processing electronic device that communicates with an inward-facing camera and display, Using the camera to determine when the user of the display is blinking, In response to the detection that the user is blinking, the current drain of the display system is reduced. Hardware processing electronics programmed to perform A display system equipped with these features.

[0030] Embodiment 26: The display system according to Embodiment 25, wherein the display comprises a light source, and reducing the current drain of the display includes dimming the light source of the display.

[0031] Embodiment 27: A display system according to any one of Embodiments 25-26, wherein the light source is a backlight.

[0032] Embodiment 28: A display system according to any of Embodiments 25-27, wherein reducing the current drain of the display includes configuring a graphics driver associated with the display to reduce the amount of power consumed by the display.

[0033] Embodiment 29: The graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the blink length, as described in Embodiment 28 of the display system.

[0034] Embodiment 30: The display system according to Embodiment 28, wherein the graphics driver is configured to reduce the amount of power consumed by the display over a specified period based on the blink length.

[0035] Embodiment 31: A display system according to any one of embodiments 25-30, wherein the display comprises an LCD display.

[0036] Embodiment 32: A display system according to any one of embodiments 25-31, wherein the display comprises an augmented reality or virtual reality display.

[0037] Embodiment 33: A method for reducing current drain in a display, A step of using an inward-facing camera to determine when the user of the display system is blinking, A step of reducing the current drain of the display in response to the detection that the user is blinking. Methods that include...

[0038] Embodiment 34: The method according to Embodiment 33, wherein the display comprises a light source, and the step of reducing the current drain of the display includes the step of dimming the light source of the display.

[0039] Embodiment 35: The method according to Embodiment 34, wherein the light source is a backlight.

[0040] Embodiment 36: The method according to any one of embodiments 33-35, wherein the step of reducing the current drain of the display includes configuring a graphics driver associated with the display to reduce the amount of power consumed by the display.

[0041] Embodiment 37: The graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the blink length, as described in Embodiment 36.

[0042] Embodiment 38: The method according to Embodiment 36, wherein the graphics driver is configured to reduce the amount of power consumed by the display over a specified period of time based on the blink duration.

[0043] Embodiment 39: The method according to any one of embodiments 33-38, wherein the display comprises an LCD display.

[0044] Embodiment 40: The method according to any one of embodiments 33-39, wherein the display comprises an augmented reality or virtual reality display.

[0045] Embodiment 41: The method according to any one of embodiments 33-40, wherein the camera comprises an eye-tracking camera.

[0046] Embodiment 42: A display system according to any one of embodiments 25-32, wherein the camera comprises an eye-tracking camera.

[0047] Embodiment 43: The display system according to any one of Embodiments 1-12, wherein the display comprises a head-mounted display.

[0048] Embodiment 44: A display system according to any one of embodiments 1-12 or 43, further comprising a frame configured to support the display in front of the user's eyes.

[0049] Embodiment 45: The display system according to any one of Embodiments 1-12 or 43-44, comprising an AR system or VR system configured to provide image content to a user with different divergence amounts such that the image content appears to the user at different depths.

[0050] Embodiment 46: The display system is the method according to any one of embodiments 13-23, comprising a head-mounted display.

[0051] Embodiment 47: The method according to any one of embodiments 13-23 or 46, wherein the display system further comprises a frame configured to support the display in front of the user's eyes.

[0052] Embodiment 48: The method according to any one of Embodiments 13-23 or 46-47, wherein the display system comprises an AR system or a VR system configured to provide image content to the user with different amounts of divergence such that the image content appears to the user at different depths. The present invention provides, for example, the following: (Item 1) A display system with reduced power consumption, A sensor facing inward, The display and A processing electronic device that communicates with the inward-facing sensor and the display, wherein the processing electronic device is The inward-facing sensor is used to detect changes in the user's eye status, Based on the detection of a change in the user's eye status, the current drain of the display system is reduced. Processing electronic equipment configured to perform A display system equipped with these features. (Item 2) The display system described in item 1, wherein the change in the user's eye status is blinking or saccadic. (Item 3) The display system according to item 1, wherein the display comprises a light source, and reducing the current drain of the display includes dimming the light source of the display. (Item 4) The display system according to item 1, wherein the display comprises a light source, and reducing the current drain of the display includes turning off the light source. (Item 5) The display system according to item 1, wherein reducing the current drain of the display includes configuring a graphics driver associated with the display to reduce the amount of power consumed by the display. (Item 6) The graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the length of time the eye blinks or saccades, as described in item 5 of the display system. (Item 7) The display system according to item 1, wherein the display comprises an LCD display. (Item 8) The inward-facing sensor is a display system according to item 1, comprising a camera. (Item 9) The display system according to item 1, wherein the processing electronic equipment is configured to reduce the current drain of the display by reducing the refresh rate associated with the display. (Item 10) The display system according to item 1, further comprising a graphics driver, and reducing the current drain of the display system, which includes reducing the power consumption of the graphics driver. (Item 11) A method for reducing the power consumption of a display system, A step of using an inward-facing sensor to detect changes in the user's eye status, A step of reducing the current drain of the display system based on the detection of a change in the user's eye status. Methods that include... (Item 12) The change in the user's eye status is blinking or saccadic, as described in item 11. (Item 13) The method according to item 11, wherein the display system comprises a light source, and the step of reducing the current drain of the display system includes the step of dimming the light source of the display system. (Item 14) The method according to item 11, wherein the display system comprises a light source, and the step of reducing the current drain of the display system includes the step of shutting off the light source of the display system. (Item 15) The method according to item 11, wherein the step of reducing the current drain of the display system includes configuring a graphics driver associated with the display system to reduce the amount of power consumed by the display system. (Item 16) The graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the length of a blink or the length of time the eye is not visible, as described in item 15. (Item 17) The method according to item 15, wherein the graphics driver is configured to reduce the amount of power consumed by the display system over a specified period based on the length of blink or the length of time the eye is not visible. (Item 18) The display system, comprising an LCD display, is the method described in item 11. (Item 19) The method according to item 11, wherein the step of reducing the current drain of the display system includes the step of reducing the refresh rate associated with the display. (Item 20) The method according to item 11, wherein the step of reducing the current drain of the display system includes the step of reducing the power consumption of the graphics driver. [Brief explanation of the drawing]

[0053] [Figure 1] Figure 1 illustrates the user's view of augmented reality (AR) through an AR device.

[0054] [Figure 2] Figure 2 illustrates an embodiment of a wearable display system.

[0055] [Figure 3] Figure 3 illustrates a conventional display system for simulating a three-dimensional image for the user.

[0056] [Figure 4] Figure 4 illustrates aspects of an approach to simulating a 3D image using multiple depth planes.

[0057] [Figure 5] Figures 5A-5C illustrate the relationship between the radius of curvature and the radius of focus.

[0058] [Figure 6] Figure 6 illustrates an example of a waveguide stack for outputting image information to the user.

[0059] [Figure 7] Figure 7 illustrates an example of an output beam produced by a waveguide.

[0060] [Figure 8]Figure 8 illustrates a flowchart of the process for reducing the current drain of the display system.

[0061] It should be understood that the drawings are provided to illustrate exemplary embodiments and are not intended to limit the scope of this disclosure. Similar reference numbers refer to similar features throughout. [Modes for carrying out the invention]

[0062] (Example display system) Referring to Figure 1, an augmented reality scene 100 is depicted. Modern computing and display technologies are accelerating the development of systems for so-called “virtual reality” or “augmented reality” experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears to be real, or can be perceived as such. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs, while augmented reality or “AR” scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Figure 1 shows an example of such a scene in which a user of AR technology can see a real-world park setting 110 featuring people, trees, and buildings in the background, and a concrete platform 120. In addition to these items, users of AR technology also perceive that they are "seeing" a robotic figure 130 standing on a real-world platform 120 and a flying, cartoonish avatar character 140 that looks like a personification of a bumblebee, although these elements 130 and 150 do not exist in the real world. The human visual perception system is complex, making it difficult to generate VR or AR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0063] Figure 2 illustrates an embodiment of a wearable display system 200. The display system 200 includes a display 208 and various mechanical and electronic modules and systems to support the functions of the display 208. The display 208 may be coupled to a frame 212, which is wearable by a display system user or viewer 201 and is configured to position the display 208 in front of the user 201's eyes. In some embodiments, the display 208 may be considered eyewear. In some embodiments, a speaker 216 is coupled to the frame 212 and positioned adjacent to the user 201's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). In some embodiments, the display system may also include one or more microphones (not shown) or other devices to detect sound. In some embodiments, the microphone may be configured to allow the user to provide input or commands to the system 200 (e.g., selection of voice menu commands, natural language questions, etc.) and / or enable audio communication with other persons (e.g., other users of a similar display system).

[0064] Continuing to refer to Figure 2, the display 208 is operably coupled to the local data processing module 224 by wired connections or wireless connectivity, which may be mounted in various configurations, such as being fixedly attached to the frame 212, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removable by the user 201 (e.g., in a backpack configuration, a belt-mounted configuration). The local processing and data module 224 may include a hardware processor or processing electronic equipment or circuitry, as well as digital memory such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be used to assist in data processing, caching, and storage. The data includes a) data captured from sensors such as image capture devices (camera, etc.), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes (e.g., operably coupled to frame 212 or otherwise attached to user 201), and / or b) data acquired and / or processed using the remote processing module 228 and / or remote data repository 232 for passage to display 208 after processing or reading, as possible. The local processing and data module 224 may be operably coupled to the remote processing module 228 and the remote data repository 232 by communication links 236, 240, such as via wired or wireless communication links, so that these remote modules 228, 232 are operably coupled to each other and available as resources for the local processing and data module 224. In some embodiments, the location processing and data module 224 may include one or more of the following: an image acquisition device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope.In some other embodiments, one or more of these sensors may be mounted on the frame 212, or they may be independent structures communicating with the local processing and data module 224 via a wired or wireless communication path.

[0065] Continuing with Figure 2, in some embodiments, the remote processing module 228 may comprise one or more processors or processing electronic devices or circuits configured to analyze and process data and / or image information. In some embodiments, the remote data repository 232 may comprise digital data storage facilities that may be available through the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 232 may comprise one or more remote servers that provide information, for example, information for generating augmented reality content, to the local processing and data module 224 and / or the remote processing module 228. In some embodiments, all data is stored, and all calculations are performed within the local processing and data modules, enabling fully autonomous use from the remote modules.

[0066] The perception of an image as "three-dimensional" or "3-D" can be achieved by providing slightly different presentations of the image to each eye of the viewer. Figure 3 illustrates a conventional display system for simulating a three-dimensional image with respect to a user. Two distinctly different images 306, 308 (one for each eye 302, 304) are output to the user. Images 306, 308 are spaced 310 units apart from eyes 302, 304 along an optical axis or z-axis parallel to the viewer's line of sight. Images 306, 308 are flat, and eyes 302, 304 can focus on the image by taking a single, focused state. Such a system relies on the human visual system, combines images 306, 308, and provides a perception of depth of the combined image.

[0067] However, it should be understood that the human visual system is more complex and provides a more realistic perception of depth. For example, although not limited by theory, it is thought that a viewer of an object may perceive it as "three-dimensional" due to a combination of convergence-divergence and accommodation. The convergence-divergence movement of two eyes relative to each other (i.e., the rolling movement of the pupils toward or away from each other to converge the lines of sight and fix the eyes on an object) is closely related to the focusing of the eye's lens (or "accommodation"). Under normal conditions, when shifting attenuation from one object to another at a different distance, changes in the convergence-divergence movement of the eyes will automatically produce a consistent change in the focusing of the eye's lens or in the eye's accommodation under a relationship known as the "accommodation-convergence-divergence reflex." Similarly, changes in accommodation will, under normal conditions, induce a consistent change in convergence-divergence movement. As described herein, many stereoscopic or "3-D" display systems display a scene using slightly different presentations (and therefore slightly different images) to each eye so that a three-dimensional viewpoint is perceived by the human visual system. However, such systems, above all, simply provide different presentations of the scene, but can be uncomfortable for many viewers because they function against the "accommodation-convergence-divergence reflex" when the eye views all image information in a single accommodated state. A display system that provides better coordination between accommodation and convergence-divergence movements can form a more realistic and comfortable simulation of a three-dimensional image.

[0068] Figure 4 illustrates aspects of an approach to simulating a three-dimensional image using multiple depth planes. Objects at various distances from eyes 302, 304 on the z-axis are accommodated by eyes 302, 304 so that those objects are in focus. Eyes (302 and 304) take on specific accommodated states to focus on objects at different distances along the z-axis. As a result, a specific accommodated state can be associated with one of the specific depth planes 402, having an associated focal length such that an object or part of an object in a particular depth plane is in focus when the eye is in the accommodated state for that depth plane. In some embodiments, the three-dimensional image may be simulated by providing a different presentation of the image for each eye 302, 304, and by providing a different presentation of the image corresponding to each depth plane. For the sake of clarity in the illustration, it should be understood that the fields of view of eyes 302, 304 may overlap, for example, as the distance along the z-axis increases, although they are shown as separate. Furthermore, for the sake of illustration, although shown as flat, it should be understood that the contour of the depth plane can be curved in physical space so that all features within the depth plane are in focus with the eye in a particular state of perspective adjustment.

[0069] The distance between an object and the eye 302 or 304 can also change the amount of light diverging from that object as it is visible to that eye. Figures 5A-5C illustrate the relationship between distance and ray divergence. The distances between the object and the eye 302 are expressed in the order of decreasing distances R1, R2, and R3. As shown in Figures 5A-5C, the rays diverge more as the distance to the object decreases. As the distance increases, the rays become more collimated. In other words, the light field produced by a point (object or part of an object) can be said to have a spherical wavefront curvature, which is a function of the distance the point is from the user's eye. The curvature increases with decreasing distance between the object and the eye 302. Consequently, in different depth planes, the ray divergence is also different, and the divergence increases with decreasing distance between the depth plane and the viewer's eye 302. Only monocular 302 is illustrated in Figures 5A–5C and other figures herein for illustrative purposes, but it should be understood that the discussion relating to eye 302 may apply to both eyes 302 and 304 of the viewer.

[0070] While not limited by theory, the human eye is typically thought to be capable of interpreting a finite number of depth planes and providing depth perception. Consequently, a highly realistic simulation of perceived depth can be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. These different presentations may be used to provide depth cues to the user based on the eye's accommodation required to focus on different image features for scenes located on different depth planes, and / or based on the observation of different image features on different depth planes that are out of focus.

[0071] Figure 6 illustrates an embodiment of a waveguide stack for outputting image information to a user. The display system 600 includes a waveguide stack or a stacked waveguide assembly 605, which may be used to provide three-dimensional perception to the eyes / brain using a plurality of waveguides 620, 622, 624, 626, 628. In some embodiments, the display system 600 is the system 200 of Figure 2, and Figure 6 shows some parts of that system 200 in more detail. For example, the waveguide assembly 605 may be part of the display 208 of Figure 2.

[0072] Continuing with Figure 6, the waveguide assembly 1240 may also include several features 630, 632, 634, and 636 between the waveguides. In some embodiments, features 630, 632, 634, and 636 may be lenses. The waveguides 620, 622, 624, 626, and 628 and / or the several lenses 630, 632, 634, and 636 may be configured to transmit image information to the eye using varying levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 640, 642, 644, 646, and 648 may function as light sources for the waveguides and may be used to input image information into waveguides 620, 622, 624, 626, and 628, and each may be configured to disperse incident light across each individual waveguide for output toward the eye 302, as described herein. By using different sources, the light sources themselves act to switch depth planes by switching illumination on or off for each depth plane as desired. Light exits from the output surfaces 650, 652, 654, 656, and 658 of image input devices 640, 642, 644, 646, and 648 and is input into the corresponding input surfaces 670, 672, 674, 676, and 678 of waveguides 620, 622, 624, 626, and 628. In some embodiments, the input surfaces 670, 672, 674, 676, and 678 may each be the edge of the corresponding waveguide or a portion of the main surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 610 or the viewer's eye 302). In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide to output a whole field of cloned collimated beams, which are directed toward the eye 302 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, one of the image input devices 640, 642, 644, 646, and 648 may be associated with a plurality (e.g., three) of waveguides 620, 622, 624, 626, and 628 to injected light into them.

[0073] In some embodiments, the image input devices 640, 642, 644, 646, and 648 are discrete displays that generate image information for input into the corresponding waveguides 620, 622, 624, 626, and 628, respectively. In some other embodiments, for example, the image input devices 640, 642, 644, 646, and 648 comprise a scanning fiber or a scanning fiber display device. In some other embodiments, the image input devices 640, 642, 644, 646, and 648 are output terminals of a single multiplexed display that can send image information to each of the image input devices 640, 642, 644, 646, and 648 via one or more optical conduits (such as optical fiber cables). It should be understood that the image information provided by the image input devices 640, 642, 644, 646, and 648 may include light of different wavelengths or colors (e.g., different primary colors).

[0074] In some embodiments, the light introduced into waveguides 620, 622, 624, 626, and 628 is provided by an optical output module 614, which may include a light source such as a backlight 614b. The backlight 614b may comprise one or more emitters, such as one or more light-emitting diodes (LEDs). The light from the backlight 614b may be modified by an optical modulator 614a, for example, a spatial light modulator. The optical modulator 614a may be configured to change the perceived intensity of the light introduced into waveguides 620, 622, 624, 626, and 628. Embodiments of the spatial light modulator include liquid crystal displays (LCDs) and digital light processing (DLP) displays. In some embodiments, the optical output module may include one or more optical guides, optical pallets, or reflectors, which are configured to output the light from the emitters to the optical modulator 614a (for example, by transmitting and / or reflecting the light).

[0075] The controller 612 controls the operation of one or more of the stacked waveguide assemblies 1240, including the operation of the image input devices 640, 642, 644, 646, 648, the optical emitter 614b, and / or the optical modulator 614a. In some embodiments, the controller 612 is part of the local data processing module 224. The controller 612 includes programming (e.g., instructions in a non-transient medium) to coordinate the timing and provisioning of image information to waveguides 620, 622, 624, 626, 628, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller 612 may be configured to control the operation of one or more cameras or sensors (e.g., inward-facing cameras) that image the user's eyes and / or the input received therefrom, and the operation of the optical emitter 614b and / or optical modulator 614a may be at least in part based on the image of the eyes and / or associated image data such as a determination of whether the eyes are blinking or moving. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 612 may be part of a processing module or electronic equipment 224 or 228 (Figure 2) and / or other processing electronic equipment and circuitry.

[0076] Continuing with Figure 6, waveguides 620, 622, 624, 626, 628, and 190 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 620, 622, 624, 626, and 628 may each be planar or have another shape (e.g., curved), with major upper and lower surfaces and edges extending between their major upper and lower surfaces. In the illustrated configuration, waveguides 620, 622, 624, 626, and 628 may each include external coupling optical elements 660, 662, 664, 666, and 628, respectively, configured to extract light from the waveguide by redirecting the light propagating within each individual waveguide and outputting image information to eye 4. The extracted light may also be referred to as externally coupled light, and the external coupling optical elements may also be referred to as light extraction optical elements. The extracted beam of light can be output by the waveguide at the point where light propagating within the waveguide strikes the light extraction optical element. The external coupling optical elements 660, 662, 664, 666, 628 may be gratings, for example, including diffractive optical features as further discussed herein. For ease of explanation and clarity of the drawings, they are shown as being located on the bottom main surface of the waveguides 620, 622, 624, 626, 628, but in some embodiments, the external coupling optical elements 660, 662, 664, 666, 628 may be located on the top and / or bottom main surface, as further discussed herein, and / or directly within the volume of the waveguides 620, 622, 624, 626, 628. In some embodiments, the external coupling optical elements 660, 662, 664, 666, and 628 may be mounted on a transparent substrate and formed within a layer of material that forms the waveguides 620, 622, 624, 626, and 628. In some other embodiments, the waveguides 620, 622, 624, 626, and 628 may be monolithic components of the material, and the external coupling optical elements 660, 662, 664, 666, and 628 may be formed on and / or inside the surface of that component of the material.

[0077] Continuing with reference to Figure 6, as discussed herein, each waveguide 620, 622, 624, 626, 628 is configured to emit light and form an image corresponding to a particular depth plane. For example, the waveguide 620 closest to the eye may be configured to deliver collimated light to the eye 302 as it is fed into such waveguide 620. The collimated light may represent the optical infinity focal plane. The next upper waveguide 622 may be configured to emit collimated light that passes through a first lens 630 (e.g., a negative lens) before it can reach the eye 302. Such a first lens 630 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 622 as originating from a first focal plane closer inward from optical infinity toward the eye 302. Similarly, the third upper waveguide 624 passes its output light through both the first lens 630 and the second lens 632 before reaching the eye 302. The combined refractive power of the first lens 630 and the second lens 632 may be configured to produce a different, gradually increasing wavefront curvature so that the eye / brain interprets the light emanating from the third waveguide 624 as emanating from a second focal plane that is closer inward toward the person from optical infinity than the light from the next upper waveguide 622.

[0078] Other waveguide layers 626, 628 and lenses 634, 636 are configured similarly, with the highest waveguide 628 in the stack emitting its output through all the lenses between it and the eye for a convergent focusing force representing the focal plane closest to the person. When viewing / interpreting light originating from the other side world 610 of the stacked waveguide assembly 605, a compensating lens layer 638 may be positioned on top of the stack to compensate for the convergent force of the lower lens stacks 630, 632, 634, 636 to compensate for the stack of lenses 630, 632, 634, 636. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the external coupling optical elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0079] In some embodiments, two or more of the waveguides 620, 622, 624, 626, and 628 may have the same associated depth plane. For example, a plurality of waveguides 620, 622, 624, 626, and 628 may be configured to output images set in the same depth plane, or a plurality of subsets of waveguides 620, 622, 624, 626, and 628 may be configured to output images set in the same plurality of depth planes, with one set for each depth plane. This can provide the advantage of forming tiled images to provide an extended field of view in those depth planes.

[0080] Continuing with Figure 6, the external coupling optical elements 660, 662, 664, 666, and 628 may be configured to both redirect light from their respective waveguides for a specific depth plane associated with the waveguide and to output this light with an appropriate amount of divergence or collimation. As a result, waveguides with different associated depth planes may have different configurations of the external coupling optical elements 660, 662, 664, 666, and 628, which will output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 660, 662, 664, 666, and 628 may be three-dimensional or surface features, which may be configured to output light at a specific angle. For example, the light extraction optical elements 660, 662, 664, 666, and 628 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, features 630, 632, 634, and 636 may not be lenses. Rather, they may simply be spacers (e.g., structures for forming cladding layers and / or voids).

[0081] In some embodiments, the external coupling optical elements 660, 662, 664, 666, and 628 are diffractive features that form a diffraction pattern or “diffractive optical element” (also referred to herein as “DOE”). In various embodiments, the DOE has a sufficiently low diffraction efficiency such that only a portion of the beam light is deflected toward the eye 302 through each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is therefore split into several associated emitted beams that exit the waveguide at various locations, resulting in a very uniform pattern of emitted beam toward the eye 302 with respect to this particular collimated beam bouncing within the waveguide.

[0082] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal, in which microdroplets have a diffraction pattern in the host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0083] Figure 7 shows an embodiment of an outgoing beam output by a waveguide. Although one waveguide is shown, other waveguides within the waveguide assembly 605 may function similarly, and it should be understood that the waveguide assembly 605 includes multiple waveguides. Light 700 is introduced into the waveguide 620 at its input surface 670 and propagates through the waveguide 620 by TIR. At the point where the light 700 collides on the DOE 660, a portion of the light exits the waveguide as an outgoing beam 702. The outgoing beam 702 is shown as substantially parallel, but may be redirected to propagate to the eye 302 at a certain angle (e.g., forming a divergent outgoing beam), depending on the depth plane associated with the waveguide 620, as discussed herein. It should be understood that a nearly parallel emitted beam may represent a waveguide with an externally coupled optical element that externally couples the light to form an image that appears to be set in the depth plane at a distance from the eye 302 (e.g., optical infinity). Other waveguides or other sets of externally coupled optical elements may output a more divergent emitted beam pattern, which would require the eye 302 to adjust to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 302 than optical infinity.

[0084] (Reduction of current drain) In some embodiments, the display system 600 described above may be powered by a battery. Current drain reduction or power reduction may be desirable to provide a longer runtime from the battery or to reduce device overheating. In some embodiments, current in the display system 200 may be drawn to light the display of the display system 620 (e.g., backlight 614b, possibly using one or more image input devices 640, 642, 644, 646, 648 such as scanning fiber or scanning fiber display devices). In addition, current is also used to control the display (e.g., the graphics processor or driver of the controller 612).

[0085] As described herein, a current drain reduction or power reduction can be achieved, for example, by dimming or turning off the display (e.g., dimming or turning off the display backlight), reducing the display update or refresh rate, or by dimming or shutting off the display after a timeout period based on the lack of user interaction.

[0086] In some embodiments of augmented reality or virtual reality devices as described herein, a camera (or other means) may be used to track eye movements. The display system 600 may include an inward-facing camera 616 that is directed inward relative to the user's face, and in particular toward the user's eyes (e.g., eyes 302). In some cases, this eye tracking may be performed, for example, to adjust the view displayed by the display system 600. For example, the camera 616 may be used to capture an image of eyes 302, from which the state or position of the pupil or iris of the eye can be tracked. The state or position of the pupil or iris of the eye may be used to determine where the user of the device is looking and to allow the display to adjust accordingly.

[0087] In some embodiments, eye tracking can be used to determine whether the user's eyes are temporarily blind. For example, a user may be blind when they are blinking. In addition, a user may be blind when their eyes are undergoing saccadic vision (e.g., rapid eye movement between fixed points).

[0088] In some embodiments, an eye-tracking camera or an inward-facing camera (or other sensor or sensor system) can be used to determine whether a user is blinking by determining whether the user's pupil or iris is partially or completely blocked from view. For example, the camera may track the user's iris as a dark circle in the background (e.g., the user's whites of the eyes). Alternatively, the camera may track the user's pupil as a darker circle within the iris. When a user is blinking, part or all of the circle defined by the iris or pupil may be obscured or interrupted. The controller 612 may "graphically" detect blinking in response to the partial or complete absence of the circular pattern corresponding to the user's iris or pupil. For example, in some embodiments, the amount of the circular pattern that is visible may be compared against a threshold, and if the amount of visible (e.g., circular) pattern does not meet the threshold, the user is determined to be blinking. In some embodiments, the threshold may be pre-configured based on user testing.

[0089] In some embodiments, the controller 612 may detect whether the user is blinking based on the amount of contrast calculated from the view of the camera 616. For example, a determination may be made regarding whether the contrast meets a threshold. In some embodiments, when the user's eyes are open and the user's iris or pupil is visible, a high contrast amount may exist in the image reflected (e.g., from the eyes or a combination of eyes and eyelids) and captured by the camera. On the other hand, when the user's eyes are closed (e.g., the user's eyelids cover the eyes), the contrast amount may be much lower than when the user's eyes are open (e.g., at least partially open). Therefore, the controller 612 may detect blinking when the contrast is below a threshold.

[0090] In some embodiments, the controller 612 may be unable to detect the position of the user's iris or pupil. For example, if the controller 612 cannot detect the user's iris or pupil, it may generate an "error" state, which can also serve as blink detection.

[0091] In some embodiments, the controller 612 may detect saccades by the user. When a user's eyes are in a saccade state, the user may not perceive any visual information, even though their eyes are open. In some embodiments, the controller 612 may detect saccades by using an inward-facing camera 616 to track the location of the user's iris or pupil (for example, as a dark circle, as described above). If movement of the user's iris or pupil above a certain rate is detected, the user may be considered to be in a saccade state.

[0092] In some embodiments, the duration of blinking or saccading may be a predetermined period. This predetermined period may be determined based on experimental data from user studies. In some embodiments, the duration of blinking or saccading may be measured by one or more sensors in the display system 600 (e.g., an inward-facing camera 616) based on an eye open / closed criterion or an eye movement criterion, as described above. If the eye is closed or saccading for a period of time, the system may be set to a lower energy state to conserve power.

[0093] The above discussion primarily refers to using a camera to determine when the user is not seeing (e.g., due to blinking or saccades), but any type of hardware, such as other types of sensor systems, that can be used to detect the state of the user's eyes may be used. In some cases, it may be desirable to utilize hardware already integrated with the display system 600 (e.g., hardware designed to perform other purposes within the display system 600) in order to reduce the power consumption that would be consumed by adding new hardware. The camera or other types of sensor systems are not limited to the use of visible light and may employ infrared (IR) light.

[0094] In some embodiments, the display system 600 may reduce its current or power drain during periods when the user is not looking at it (e.g., due to blinking or saccades). For example, the current drain or power consumption of a display can be reduced by employing one or more current drain or power reduction techniques, which may include dimming or turning off a light source for the display associated with the display (e.g., a backlight). In some embodiments, the light source (e.g., a backlight) 614b of the display system 600 may be dimmed or turned off. In other embodiments (e.g., a display system using an OLED display without a backlight), the current drain or power consumption may be reduced by dimming or turning off one or more active pixels of the display. Other types of display components or displays may be turned off, dimmed, or set to a lower power consumption mode when the eye is not looking at them (e.g., during blinking or saccades).

[0095] Alternatively, or in combination, the graphics driver or processor or processing electronics associated with the display may “skip” some frames, or wait for a specified period of time while the graphics driver is in a state that consumes less power than when providing or refreshing a new image. For example, the graphics driver may cause the graphics processor to pause refreshing the displayed image or reduce the display's refresh rate, thus consuming less power compared to normal operation. In some implementations, the number of frames or the duration may be configured to correspond to the length of a blink or saccade, where current drain is reduced. The duration of a blink is typically, for example, 100 to 400 milliseconds.

[0096] It should be understood that any of the current drain reduction techniques discussed herein may be implemented independently or in combination with each other. For example, in some embodiments, in response to the detection of a blink or saccade, the controller 612 may dim the backlight 614b and cause the graphics driver to skip a specified number of frames. In other embodiments, the controller 612 may cause the graphics driver to skip a specified number of frames without dimming the backlight 614b, or vice versa.

[0097] Figure 8 illustrates a flowchart of an exemplary process for reducing current drain or power consumption according to several embodiments. Any part of this flowchart may be performed by electronic equipment such as processing electronics or circuits. In block 802, a determination is made as to whether a state in which the user of the display system is not visible (e.g., blinking or saccading by the user) is detected. In some embodiments, this may be done using an eye-tracking or inward-facing camera or other sensor or sensor system that determines whether the user's pupil or iris is blocked from view or subjected to rapid movement. If blinking or saccading is detected, the process may proceed to block 804. Otherwise, the process may continue eye monitoring, for example, to detect blinking or saccading by the user of the display system.

[0098] In block 804, the light source associated with the display is dimmed or turned off. For example, the light source may be configured to enter a low-power mode or be disabled. In some embodiments, the light source may include a backlight 614b. In other embodiments, the light source may include multiple active pixels of the display (e.g., an OLED display). Other light source and display configurations are also possible.

[0099] In block 806, the graphics driver associated with the display system may reduce the amount of power consumed. For example, the graphics driver may skip X frames or wait for a period Y, where X and Y are determined based on the duration of a blink (e.g., 100-400 milliseconds) or a saccade. In some embodiments, the graphics driver may reduce the refresh rate of the display.

[0100] In block 808, the light sources associated with the display (e.g., the backlight 614b, the active pixels of the display, and / or equivalent) or other components of the display are turned back on or undimmed, and the display system resumes normal operation. It should be understood that the process illustrated in this flowchart is an embodiment, and its steps may be excluded, added, and / or rearranged.

[0101] Figure 8 illustrates both dimming / turning off the light source associated with the display (blocks 804, 808) and reducing the power consumption of the graphics driver or processor (block 806), but it should be understood that in other embodiments, the display system 600 may implement any combination of current draining or power reduction techniques. For example, in some embodiments, the display system 600 may implement only dimming / turning off the display's light source, only reducing the power consumption of the graphics driver or processor (e.g., skipping frames, reducing the refresh rate, and / or equivalent), or both. Power savings may also result from other components. For example, setting the spatial light modulator or one or more scanning fibers or scanning fiber display devices to a lower power state can also reduce power consumption.

[0102] The average person blinks approximately once every 2 to 10 seconds, for a period of 100 to 400 milliseconds. Therefore, in a less frequent scenario, the eye is closed for about 1% of the time. For a more typical scenario, the eye would be closed for 2% to 5% of the time. Thus, a reduction of several percent can potentially be achieved in the current drain associated with lighting the display using a light source (e.g., backlight or active pixels) and / or graphics driver / processor.

[0103] Various exemplary embodiments of the present invention are described herein. These embodiments are provided to illustrate broader applicable aspects of the invention. Various modifications may be made to the described invention, and equivalents may be substituted without departing from the spirit and scope of the invention. For example, advantageously, they are used with AR displays that provide images across multiple depth planes, but the augmented reality content disclosed herein may also be displayed by systems that provide images on a single depth plane.

[0104] Numerous modifications may be made to adapt specific situations, materials, compositions, processes, actions or steps of a process to the object, spirit, or scope of the present invention. Furthermore, as will be understood by those skilled in the art, each of the individual modifications described and illustrated herein has discrete components and features that can be readily separated from or combined with features of any of several other embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0105] The present invention includes methods that may be performed using the device. The methods may include the act of providing such a suitable device. Such provision may be performed by a user. In other words, the act of “providing” is merely requiring the user to acquire, access, approach, position, configure, activate, power on, or otherwise operate in order to provide the necessary device in the method. The methods described herein may be performed in any logically possible order of the described events, as well as in the order in which the events are described.

[0106] Exemplary aspects of the present invention, along with details relating to material selection and manufacturing, are described above. Other details of the present invention are understood in connection with the above-referenced patents and publications and, generally, can be grasped or understood by those skilled in the art. The same may apply to the method-based aspects of the present invention in terms of additional actions that may be adopted generally or theoretically.

[0107] In addition, although the present invention has been described with reference to several embodiments that optionally incorporate various features, the present invention is not limited to those described or indicated to be considered with respect to each modification of the present invention. Various modifications may be made to the described invention, and equivalents may be substituted (whether described herein or not for the sake of some brevity) without departing from the true spirit and scope of the invention. In addition, if a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of that range, and any other provisions or intervening values ​​within that defined range, are encompassed within the present invention.

[0108] Furthermore, it should be considered that any optional feature of a variation of the invention described herein may be described and claimed independently or in combination with any one or more features of those described herein. References to singular items include the possibility of multiple identical items existing. More specifically, as used herein and in the claims associated therewith, the singular forms “a, an,” “said,” and “the” include multiple referents unless otherwise specifically stated. In other words, the use of articles allows for “at least one” of the subject items in the above description and in the claims associated with this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended to function as an antecedent for the use of such exclusive terms, or “negative” restrictions, such as “only,” “only,” and “equivalents,” relating to the description of claim elements.

[0109] Without using such exclusive terms, the term “equipped with” in the claims associated with this disclosure shall allow for the inclusion of any additional elements, whether a given number of elements are enumerated in such claims or whether the addition of features can be considered to transform the nature of the elements described in such claims. Unless otherwise specifically defined herein, all technical and scientific terms used herein are given the broadest possible generally understood meaning while maintaining the validity of the claims.

[0110] The scope of the present invention is not limited to the provided examples and / or specification, but rather is limited only to the claims associated with this disclosure.

Claims

1. A display system with reduced power consumption, A sensor facing inward, The display and A processing electronic device that communicates with the inward-facing sensor and the display, wherein the processing electronic device is The inward-facing sensor is used to detect changes in the user's eye status, Based on the detection of the change in the user's eye status, the current drain of the display system is reduced. Processing electronic equipment configured to perform Reducing current drain by providing a light source does not involve turning off the light source. The change in the user's eye status is blinking or saccadic, The processing electronic device is configured to reduce the current drain of the display system by reducing the refresh rate associated with the display, A display system in which the period during which the refresh rate is reduced corresponds to the length of the saccade.

2. The display system according to claim 1, wherein the change in the user's eye status is blinking.

3. The display system according to claim 1, wherein the display comprises a light source, and reducing the current drain of the display system includes dimming the light source of the display.

4. The display system according to claim 1, wherein reducing the current drain of the display system includes configuring a graphics driver associated with the display to reduce the amount of power consumed by the display system.

5. The display system according to claim 4, wherein the graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the length of time the eye blinks or saccades.

6. The display system according to claim 1, wherein the display comprises a liquid crystal display (LCD).

7. The display system according to claim 1, wherein the inward-facing sensor comprises a camera.

8. The display system according to claim 1, further comprising a graphics driver, wherein reducing the current drain of the display system includes reducing the power consumption of the graphics driver.

9. A method for reducing the power consumption of a display system, Using inward-facing sensors to detect changes in the user's eye status, Based on the detection of the change in the user's eye status, the current drain of the display system is reduced. This includes, and the change in the user's eye status is blinking or saccadic, Reducing current drain does not involve turning off the light source. Reducing the current drain of the display system includes reducing the refresh rate associated with the display system. A method in which the period during which the refresh rate is reduced corresponds to the length of the saccade.

10. The method according to claim 9, wherein the change in the user's eye status is blinking.

11. The method according to claim 9, wherein the display system includes a light source, and reducing the current drain of the display system includes dimming the light source of the display system.

12. The method according to claim 9, wherein reducing the current drain of the display system includes configuring a graphics driver associated with the display system to reduce the amount of power consumed by the display system.

13. The method according to claim 12, wherein the graphics driver is configured to skip a specified number of frames, the specified number of frames being based on the length of a blink or the length of time the eye is not visible.

14. The method according to claim 12, wherein the graphics driver is configured to reduce the amount of power consumed by the display system over a specified period of time based on the length of blink or the length of time the eye is not seeing.

15. The method according to claim 9, wherein the display system comprises a liquid crystal display (LCD).

16. The method according to claim 9, wherein reducing the current drain of the display system includes reducing the power consumption of the graphics driver.

17. The display system according to claim 1, wherein the change in the user's eye status is a saccade.

18. The method according to claim 9, wherein the change in the user's eye status is a saccade.

Images