Ambient light management systems and methods for wearable devices

Abstract

Techniques for operating an optical system are described. In some embodiments, light associated with a world object is received at an optical system. The virtual image light is projected onto an eyepiece of the optical system. A portion of a system field of view of the optical system to be at least partially dimmed is determined based on information detected by the optical system. A plurality of spatially resolved dimming values for the portion of the system field of view may be determined based on the detected information. The detected information may include light information, gaze information, and / or image information. A dimmer of the optical system may be adjusted according to the plurality of dimming values to reduce an intensity of light associated with a world object in the portion of the system field of view.

Description

[0001] This application is a divisional application of a patent application with international application number PCT / US2020 / 060254, international filing date November 12, 2020, Chinese national application number 202080078724.1, and title "Environmental Light Management Systems and Methods for Wearable Devices". BACKGROUND OF THE INVENTION

[0002] Modern computing and display technologies have contributed to the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that they appear to be real or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information and are opaque to other actual real-world visual inputs; augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world around the user.

[0003] Despite progress in these display technologies, there is a need in the art for improved methods, systems, and devices related to augmented reality systems, particularly display systems. SUMMARY OF THE INVENTION

[0004] The present disclosure generally relates to techniques for improving optical systems under varying ambient light conditions. More particularly, embodiments of the present invention provide systems and methods for operating an augmented reality (AR) device that includes a dimming element. Although the present invention is described with reference to an AR device, the present disclosure is applicable to various applications in computer vision and image display systems.

[0005] A review of the present invention is provided below with reference to a plurality of examples. As used herein, any reference to a series of examples will be understood to refer separately to each of those examples (e.g., "Examples 1-4" will be understood to mean "Examples 1, 2, 3, or 4").

[0006] Example 1 is a method of operating an optical system, the method comprising: receiving, at the optical system, light associated with a world object; projecting virtual image light onto an eyepiece; determining, based on information detected by the optical system, a portion of a system field of view of the optical system to be at least partially dimmed; and adjusting a dimming device to reduce the intensity of light associated with the world object in the portion of the system field of view.

[0007] Example 2 is the method according to Example 1, wherein the optical system includes a light sensor configured to detect light information corresponding to light associated with the world object, and wherein the detected information includes the light information.

[0008] Example 3 is the method according to Examples 1-2, wherein the light information includes a plurality of spatially resolved light values.

[0009] Example 4 is the method according to Examples 1-3, wherein the light information includes a global light value.

[0010] Example 5 is the method according to Examples 1-4, wherein the optical system includes an eye tracker configured to detect gaze information of a user's eye corresponding to the optical system. Wherein the detected information includes the gaze information.

[0011] Example 6 is the method according to Examples 1-5, wherein the gaze information includes pixel positions intersecting a gaze vector of the user's eye.

[0012] Example 7 is the method according to Examples 1-6, wherein the gaze information includes one or more of a pupil position of the user's eye, a center of rotation of the user's eye, a pupil size of the user's eye, a pupil diameter of the user's eye, and cone and rod positions of the user's eye.

[0013] Example 8 is the method according to Examples 1-7, further comprising: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.

[0014] Example 9 is the method according to Examples 1-8, wherein the image information includes a plurality of spatially resolved image luminance values.

[0015] Example 10 is the method according to Examples 1-9, wherein the image information includes a global image luminance value.

[0016] Example 11 is the method according to Examples 1-10, further comprising: determining a plurality of spatially resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimming device is adjusted according to the plurality of dimming values.

[0017] Example 12 is the method according to Examples 1-11, wherein the dimming device includes a plurality of pixels.

[0018] Example 13 is the method according to Examples 1-12, wherein the dimming device is adjusted to completely block the intensity of the light associated with all world objects in the system field of view.

[0019] Example 14 is the method according to Examples 1-13, further comprising: adjusting the luminance associated with the virtual image light.

[0020] Example 15 is the method according to Examples 1 - 14, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view of the system.

[0021] Example 16 is the method according to Examples 1 - 15, wherein the portion of the system field of view of the optical system to be at least partially dimmed is determined at least in part based on the at least one world object.

[0022] Example 17 is the method according to Examples 1 - 16, wherein the portion of the system field of view of the optical system to be at least partially dimmed is determined at least in part based on at least one object included in the virtual image.

[0023] Example 18 is a non - transitory computer - readable medium including instructions that, when executed by a processor, cause the processor to perform operations including: receiving light associated with a world object at an optical system; projecting virtual image light onto an eyepiece; determining, based on information detected by the optical system, a portion of the system field of view of the optical system to be at least partially dimmed; and adjusting a dimming device to reduce the intensity of light associated with world objects in the portion of the system field of view.

[0024] Example 19 is the non - transitory computer - readable medium according to Example 18, wherein the optical system includes a light sensor configured to detect light information corresponding to the light associated with the world object, and wherein the detected information includes the light information.

[0025] Example 20 is the non - transitory computer - readable medium according to Example 19, wherein the light information includes a plurality of spatially - resolved light values.

[0026] Example 21 is the non - transitory computer - readable medium according to Example 19, wherein the light information includes a global light value.

[0027] Example 22 is the non - transitory computer - readable medium according to Example 18, wherein the optical system includes an eye tracker configured to detect gaze information corresponding to the user's eyes of the optical system. Wherein the detected information includes the gaze information.

[0028] Example 23 is the non - transitory computer - readable medium according to Example 22, wherein the gaze information includes pixel positions intersecting a gaze vector of the user's eyes.

[0029] Example 24 is the non-transitory computer-readable medium according to Example 22, wherein the gaze information includes one or more of the pupil position of the user's eye, the center of rotation of the user's eye, the pupil size of the user's eye, the pupil diameter of the user's eye, and the cone and rod positions of the user's eye.

[0030] Example 25 is the non-transitory computer-readable medium according to Example 18, wherein the operation further includes: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.

[0031] Example 26 is the non-transitory computer-readable medium according to Example 25, wherein the image information includes a plurality of spatially resolved image luminance values.

[0032] Example 27 is the non-transitory computer-readable medium according to Example 25, wherein the image information includes a global image luminance value.

[0033] Example 28 is the non-transitory computer-readable medium according to Example 18, wherein the operation further includes: determining a plurality of spatially resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.

[0034] Example 29 is the non-transitory computer-readable medium according to Example 18, wherein the dimmer includes a plurality of pixels.

[0035] Example 30 is the non-transitory computer-readable medium according to Example 18, wherein the dimmer is adjusted to completely block the intensity of the light associated with all world objects in the system field of view.

[0036] Example 31 is the non-transitory computer-readable medium according to Example 18, wherein the operation further includes: adjusting the luminance associated with the virtual image light.

[0037] Example 32 is the non-transitory computer-readable medium according to Example 18, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view.

[0038] Example 33 is an optical system, comprising: a projector configured to project virtual image light onto an eyepiece; a dimming device configured to dim light associated with a world object; a processor communicatively coupled to the projector and the dimming device, wherein the processor is configured to perform operations including: based on information detected by the optical system, determining a portion of a system field of view of the optical system to be at least partially dimmed; adjusting the dimming device to reduce the intensity of light associated with world objects in the portion of the system field of view.

[0039] Example 34 is the optical system according to Example 33, further comprising: a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.

[0040] Example 35 is the optical system according to Example 34, wherein the light information includes a plurality of spatially resolved light values.

[0041] Example 36 is the optical system according to Example 34, wherein the light information includes a global light value.

[0042] Example 37 is the optical system according to Example 33, further comprising: an eye tracker configured to detect gaze information corresponding to a user's eyes of the optical system. Wherein the detected information includes the gaze information.

[0043] Example 38 is the optical system according to Example 37, wherein the gaze information includes pixel positions intersecting a gaze vector of the user's eyes.

[0044] Example 39 is the optical system according to Example 37, wherein the gaze information includes one or more of a pupil position of the user's eyes, a center of rotation of the user's eyes, a pupil size of the user's eyes, a pupil diameter of the user's eyes, and cone and rod positions of the user's eyes.

[0045] Example 40 is the optical system according to Example 33, wherein the operations further include: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.

[0046] Example 41 is the optical system according to Example 39, wherein the image information includes a plurality of spatially resolved image luminance values.

[0047] Example 42 is the optical system according to Example 40, wherein the image information includes a global image luminance value.

[0048] Example 43 is the optical system according to Example 33, wherein the operation further includes: determining a plurality of spatially resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.

[0049] Example 44 is the optical system according to Example 33, wherein the dimmer includes a plurality of pixels.

[0050] Example 45 is the optical system according to Example 33, wherein the dimmer is adjusted to completely block the intensity of the light associated with world objects in all of the system field of view.

[0051] Example 46 is the optical system according to Example 33, wherein the operation further includes: adjusting the brightness associated with the virtual image light.

[0052] Example 47 is the optical system according to Example 33, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view.

[0053] Example 48 is an optical system, including: a frame configured to be worn around the head of a user of the optical system; a dimming component carried by the frame and configured to be positioned between the user's eyes and the user's environment; an eye tracker configured to monitor the positioning of the user's eyes; and a control circuit communicatively coupled to the dimming component and the eye tracker, the control circuit being configured to: receive data from the eye tracker; determine, based on the data received from the eye tracker, a position along the optical axis of the user's eyes where a specific anatomical region of the user's eyes is located; identify one or more points in a three-dimensional space within the user's environment; and for each of the one or more identified points in the user's environment: identify a set of one or more pixels of the dimming component at least partially based on the determined position of the specific anatomical region of the user's eyes and the corresponding point in the three-dimensional space within the user's environment; control the dimming component to dim the identified set of one or more pixels.

[0054] Example 49 is the optical system according to Example 48, wherein the specific anatomical region of the user's eyes includes the center of rotation of the user's eyes.

[0055] Example 50 is the optical system according to Example 48, wherein the specific anatomical region of the user's eyes includes the pupil center of the user's eyes.

[0056] Example 51 is the optical system according to Example 48, further comprising: a projector configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the user's eyes and the dimming component, wherein the waveguide is configured to receive light from the projector and direct it to the user's eyes.

[0057] Example 52 is the optical system according to Example 51, wherein the control circuit is communicatively coupled to the projector, and the control circuit is further configured to control the projector to emit light representing one or more pixels of virtual content.

[0058] Example 53 is the optical system according to Example 52, wherein one or more points in the three-dimensional space within the user's environment respectively correspond to one or more positions in the three-dimensional space where the one or more pixels of the virtual content will be perceived by the user.

[0059] Example 54 is the optical system according to Example 52, wherein the one or more pixels of the virtual content include a plurality of pixels of a virtual object.

[0060] Example 55 is the optical system according to Example 54, wherein one or more points in the three-dimensional space within the user's environment respectively correspond to one or more positions in the three-dimensional space where the one or more pixels of a virtual shadow associated with the virtual object will be perceived by the user.

[0061] Example 56 is the optical system according to Example 48, wherein one or more points in the three-dimensional space within the user's environment correspond to one or more points in the three-dimensional space occupied by real-world objects in the user's environment.

[0062] Example 57 is the optical system according to Example 48, wherein, in order to identify the set of one or more pixels of the dimming component, the control circuit is configured to: project a set of one or more light rays from corresponding points in the three-dimensional space within the user's environment to a determined position in a specific anatomical region of the user's eyes; and identify the set of one or more intersections between the set of one or more light rays and the dimming component.

[0063] Example 58 is the optical system according to Example 48, wherein the dimming component is curved in shape.

[0064] Example 59 is the optical system according to Example 48, wherein the control circuit is further configured to determine a set of one or more dimming values for a set of one or more identified pixels of the dimming component, respectively, and wherein the control circuit is configured to control the dimming component to dim the set of one or more identified pixels according to the determined set of one or more dimming values.

[0065] Example 60 is the optical system according to Example 59, wherein the control circuit is further configured to determine one or more characteristics of the user's eyes based on data received from the eye tracker, and wherein the control circuit is configured to determine the set of one or more dimming values for a set of one or more identified pixels of the dimming component, respectively, at least in part based on the one or more determined characteristics of the user's eyes.

[0066] Example 61 is the optical system according to Example 60, wherein the one or more characteristics of the user's eyes include one or more of the pupil size of the user's eyes, the pupil diameter of the user's eyes, the cone and rod positions of the user's eyes, and the accommodation state of the lens of the user's eyes.

[0067] Example 62 is the optical system according to Example 61, wherein the control circuit is configured to identify the set of one or more pixels of the dimming component at least in part based on the one or more determined characteristics of the user's eyes.

[0068] Example 63 is the optical system according to Example 59, further comprising: a projector communicatively coupled to the control circuit and configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the user's eyes and the dimming component, wherein the waveguide is configured to receive light from the projector and direct it to the user's eyes, wherein the control circuit is further configured to control the projector to emit light representing one or more pixels of virtual content at one or more brightness levels, respectively, and wherein the control circuit is configured to determine the set of one or more dimming values for a set of one or more identified pixels of the dimming component, respectively, at least in part based on the one or more brightness levels of the one or more pixels of the virtual content.

[0069] Example 64 is the optical system according to Example 63, wherein the control circuit is configured to determine the set of one or more dimming values for a set of one or more identified pixels of the dimming component, respectively, at least in part based on one or more of a predetermined contrast and a predetermined visibility level specified for the virtual content.

[0070] Example 65 is the optical system according to Example 63, wherein the virtual content includes a virtual object, and wherein the control circuit is configured to identify the set of one or more pixels of the dimming component based at least in part on one or more characteristics of the virtual object.

[0071] Example 66 is the optical system according to Example 65, wherein the one or more characteristics of the virtual object include one or more of the size of the virtual object, the shape of the virtual object, the position in the user's environment where the virtual object will be perceived by the user, and the depth where the virtual object will be perceived by the user.

[0072] Example 67 is the optical system according to Example 59, further comprising: an optical sensor communicatively coupled to the control circuit and configured to respectively monitor one or more brightness levels of light associated with one or more parts of the user's environment, and wherein the control circuit is further configured to determine the set of one or more dimming values for the identified set of one or more pixels of the dimming component based at least in part on the one or more brightness levels associated with the one or more parts of the user's environment.

[0073] Example 68 is the optical system according to Example 67, wherein the optical sensor includes a camera.

[0074] Example 69 is the optical system according to Example 67, wherein the optical sensor includes one or more photodiodes.

[0075] Example 7 is the optical system according to Example 67, further comprising: a projector communicatively coupled to the control circuit and configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the user's eyes and the dimming component, wherein the waveguide is configured to receive light from the projector and direct it to the user's eyes, and wherein the control circuit is further configured to control the projector to emit light representing one or more pixels of virtual content.

[0076] Example 71 is the optical system according to Example 70, wherein the virtual content includes a virtual object, and wherein the one or more parts of the user's environment associated with the one or more brightness levels include a specific part of the user's environment that will be perceived by the user as being occluded by the virtual object.

[0077] Example 72 is the optical system according to Example 70, wherein the control circuit is further configured to control the projector to emit light representing one or more pixels of virtual content at least partially based on the one or more luminance levels associated with the one or more portions of the user's environment.

[0078] Example 73 is the optical system according to Example 48, wherein the eye tracker is configured to monitor the positioning of the user's eyes relative to the dimming component.

[0079] Example 74 is an optical system, including: a frame configured to be worn around the head of a user of the optical system; a left dimming component carried by the frame and configured to be positioned between the user's left eye and the user's environment; a right dimming component carried by the frame and configured to be positioned between the user's right eye and the user's environment; and a control circuit communicatively coupled to the left dimming component and the right dimming component, the control circuit being configured to: identify one or more points in a three-dimensional space within the user's environment; and for each of the one or more identified points in the user's environment: identify a set of one or more pixels of the left dimming component at least partially based on the corresponding point in the three-dimensional space within the user's environment; identify a set of one or more pixels of the right dimming component at least partially based on the corresponding point in the three-dimensional space within the user's environment; control the left dimming component to dim the identified set of one or more pixels of the left dimming component; and control the right dimming component to dim the identified set of one or more pixels of the right dimming component.

[0080] Example 75 is the optical system according to Example 74, further including: a left eye tracker communicatively coupled to the control circuit and configured to monitor the positioning of the user's left eye; and a right eye tracker communicatively coupled to the control circuit and configured to monitor the positioning of the user's right eye; wherein the control circuit is further configured to: receive data from the left eye tracker and the right eye tracker; determine, based on the data received from the left eye tracker, the position along the optical axis of the user's left eye where a particular anatomical region of the user's left eye is located; determine, based on the data received from the right eye tracker, the position along the optical axis of the user's right eye where a particular anatomical region of the user's right eye is located;

[0081] Example 76 is the optical system according to Example 75, wherein the control circuit is configured to: identify a set of one or more pixels of the left dimming component at least in part based on the determined position of the specific anatomical region of the user's left eye and a corresponding point in the three-dimensional space within the user's environment; and identify a set of one or more pixels of the right dimming component at least in part based on the determined position of the specific anatomical region of the user's right eye and a corresponding point in the three-dimensional space within the user's environment.

[0082] In one embodiment, the optical device includes a plastic substrate whose shape exhibits curvature in at least one dimension, including a plurality of components laminated to the plastic substrate. The plurality of components includes an external stack of components including at least one of a reflective film and a colored film, a stack of birefringent films located between the external stack of components and the plastic substrate, and a spatial light modulator assembly located between the stack of birefringent films and the plastic substrate.

[0083] In one or more embodiments, the plurality of components further includes a first polarizer located between the spatial light modulator assembly and the plastic substrate, and a second polarizer located between the stack of birefringent films and the spatial light modulator assembly. The stack of birefringent films may include a first quarter-wave plate (QWP), a second QWP, and a C-plate located between the first QWP and the second QWP. The plastic substrate, the shape of the plastic substrate may exhibit curvature in two dimensions. The plurality of components may further include one or more quarter-wave plates (QWP), achromatic QWPs (AQWP), retarders, anti-reflection layers, multilayer reflective polarizers (MLP), lamination layers, or combinations thereof.

[0084] In one or more embodiments, the spatial light modulator assembly includes a pixel electrode layer, a planar electrode, and a liquid crystal (LC) layer located between the pixel electrode layer and the planar electrode. The spatial light modulator assembly may include a first plastic layer located adjacent to the electrode layer and a second plastic layer located adjacent to the planar electrode. The pixel electrode layer may include segmented electrodes and a circuit electrically coupled to the segmented electrodes. The circuit may include an organic thin-film transistor (O-TFT). The shape of each component in the plurality of components may conform to the curvature of the plastic substrate. The external stack of components may include at least one of a wire-grid polarizer and a cholesteric liquid crystal (CLC) reflector.

[0085] In another embodiment, a wearable display system includes an eyepiece and an optical device. The optical device includes a plastic substrate that exhibits curvature in at least one dimension and includes a plurality of components laminated to the plastic substrate. The plurality of components includes an outer stack of components that includes at least one of a reflective film and a colored film, a stack of birefringent films located between the outer stack of components and the plastic substrate, and a spatial light modulator assembly located in a space between the stack of birefringent films and the plastic substrate.

[0086] In one or more embodiments, the wearable display system further includes a frame configured to be worn around a user's head. The eyepiece may be configured to be located between the user's eyes and the optical device when the frame is worn by the user. The eyepiece may include a plurality of waveguides. The shape of the eyepiece may exhibit curvature in at least one dimension. The wearable display system may further include a control circuit communicatively coupled to the spatial light modulator assembly.

[0087] In yet another embodiment, a display device includes an eyepiece stack and an ambient light management module (ALMM) that includes a dimming component. The ALMM may include a plastic substrate, an angular attenuator component, and one or more of at least one world-facing film.

[0088] Many advantages over the conventional art are achieved through the present disclosure. For example, the augmented reality (AR) devices described herein can be used at different light levels, from dark indoor to bright outdoor, by globally dimming and / or selectively dimming the ambient light reaching the user's eyes. By using a pixelated dimming component to attenuate world light by greater than 99%, embodiments of the present invention allow for AR and virtual reality (VR) capabilities in a single device. Embodiments of the present invention also use a variable focus element with discrete or continuously variable depth plane switching technology to mitigate vergence-accommodation conflict. Embodiments of the present invention improve the battery life of AR devices by optimizing projector brightness based on the detected amount of ambient light. Other advantages of the present disclosure will be readily apparent to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 An augmented reality (AR) scene as viewed through a wearable AR device according to some embodiments described herein is shown.

[0090] Figure 2A One or more general features of an AR device according to the present invention are shown.

[0091] Figure 2B An example of an AR device that determines a dimmed region based on detected light information is illustrated.

[0092] Figure 2C An example of an AR device that determines a dimmed area based on a virtual image is shown.

[0093] Figure 2D An example of an AR device that determines a dimmed area based on gaze information is shown.

[0094] Figure 3 A schematic diagram of a wearable AR device according to the present invention is shown.

[0095] Figure 4 A method for operating an optical system is shown.

[0096] Figure 5 An AR device having an eyepiece and a pixelated dimming element is shown.

[0097] Figure 6 A technique for determining a gaze vector based on the pupil position of a user's eye is shown.

[0098] Figure 7 A technique for determining a gaze vector based on the center of rotation of a user's eye is shown.

[0099] Figure 8 A technique for determining a gaze vector based on detected light information and the positions of cones and rods within the eye is shown.

[0100] Figure 9 The determined gaze vector under high-light conditions of pupil constriction is shown.

[0101] Figure 10 The determined gaze vector under low-light conditions of pupil dilation is shown.

[0102] Figure 11 Three techniques for determining a gaze vector and a corresponding dimmed area under high-light conditions are shown.

[0103] Figure 12 Three techniques for determining a gaze vector and a corresponding dimmed area under low-light conditions are shown.

[0104] Figure 13 A dimmer that has been adjusted to produce a dimmed area determined using a gaze vector calculated using pupil position is shown.

[0105] Figure 14 A dimmer that has been adjusted to produce a dimmed area determined using a gaze vector calculated using cone and rod positions under high-light conditions is shown.

[0106] Figure 15Shows a dimmer that has been adjusted to produce a dimmed area determined using a gaze vector calculated using cone and rod positions for use in low light conditions.

[0107] Figure 16 Shows an example where the dimmed area includes a central portion within an annular area.

[0108] Figure 17 Shows a dimmer that has been adjusted to produce a dimmed area determined using a gaze vector calculated using the center of rotation of the eye.

[0109] Figure 18A and Figure 18B Shows a method for determining a portion of the system field of view to be dimmed based on image information.

[0110] Figure 19A and Figure 19B Shows a method for determining a portion of the system field of view to be dimmed based on image information.

[0111] Figure 20 Shows an example of improving the opacity of virtual content by adjusting a dimmer and / or adjusting a projector.

[0112] Figure 21 Shows an example of improving the opacity of virtual content by dimming a portion of the system field of view corresponding to a virtual object.

[0113] Figure 22 Shows a graph depicting the relationship between virtual image brightness and ambient light level.

[0114] Figure 23A and Figure 23B Shows a graph depicting the effect of a small occlusion on the world scene.

[0115] Figure 24 Shows a graph depicting the effect of changing the diameter of a light shutter on the transmission of a dimming element as a function of the angular range.

[0116] Figure 25 Shows an example of using a single light shutter for dimming.

[0117] Figure 26 Shows an example of the architecture of an optical see-through (OST) head-mounted display (HMD).

[0118] Figure 27 Shows additional examples of the architecture of an OST-HMD.

[0119] Figure 28 Shows additional examples of the architecture of an OST-HMD.

[0120] Figure 29Shows a schematic diagram of an AR device according to the present invention.

[0121] Figure 30 and Figure 31 Shows an example of an AR HMD including an ambient light management module (ALMM) according to some embodiments.

[0122] Figure 32 Shows a simplified computer system according to some embodiments described herein. Detailed Description

[0123] A current technical challenge for optical see-through (OST) augmented reality (AR) devices is the variation in opacity and / or visibility of virtual content under changing ambient light conditions. This problem is exacerbated under extreme lighting conditions such as a completely dark room or outdoors in full bright sunlight. Embodiments of the present invention address these and other problems by dimming the world light at different spatial locations within the field of view of the AR device. The portion of the field of view to which dimming is applied and the amount of dimming applied are each determined based on various information detected by the AR device. This information can include detected ambient light, detected gaze information, and / or the brightness of the projected virtual content. For example, the functionality of the AR device is further improved by detecting the direction associated with the ambient light by detecting a plurality of spatially resolved light values. This allows the AR device to improve its battery life by only dimming the portions of the field of view that need to be dimmed and / or increasing the brightness of the projectors in certain portions of the field of view. Thus, embodiments of the present invention enable AR devices to be used under a wider variety of conditions than has traditionally been possible.

[0124] Figure 1 Shows an AR scene 100 as viewed through a wearable AR device according to some embodiments of the present invention. The AR scene 100 is depicted, where a user of AR technology sees a real-world park-like setting 106 featuring various real-world objects 130 such as people, trees, buildings in the background, and a real-world concrete platform 120. In addition to these items, the user of AR technology likewise perceives that they "see" various virtual objects 102 such as a robotic statue 102-2 standing on the real-world concrete platform 120, and a flying cartoonish avatar character 102-1 that appears to be an avatar of a bumblebee, even though these elements (character 102-1 and statue 102-2) do not exist in the real world. Due to the extremely complex nature of human visual perception and the nervous system, it is challenging to produce virtual reality (VR) or AR technologies that contribute to a comfortable, natural, and rich presentation of virtual image elements together with other virtual or real-world image elements.

[0125] Figure 2AShows one or more general features of the AR device 200 according to the present invention. In some embodiments, the AR device 200 may include an eyepiece 202 and a dynamic dimming device 203, which are configured to be transparent or translucent when the AR device 200 is in an inactive mode or a shutdown mode, so that a user can see one or more world objects 230 when viewing through the eyepiece 202 and the dynamic dimming device 203. As shown, the eyepiece 202 and the dynamic dimming device 203 may be arranged in a side-by-side configuration and may form a system field of view that the user sees when viewing through the eyepiece 202 and the dynamic dimming device 203. In some embodiments, the system field of view is defined as the entire two-dimensional region occupied by one or both of the eyepiece 202 and the dynamic dimming device 203. Although Figure 2A a single eyepiece 202 and a single dynamic dimming device 203 are shown (for illustrative purposes), the AR device 200 may include two eyepieces and two dynamic dimming devices, one for each eye of the user.

[0126] During operation, the dynamic dimming device 203 can be adjusted to reduce the intensity of world light 232 associated with the world object 230 incident on the dynamic dimming device 203, thereby creating a dimmed area 236 within the system field of view. The dimmed area 236 can be a part or a subset of the system field of view and can be dimmed partially or completely. The dynamic dimming device 203 can be adjusted according to a plurality of spatially resolved dimming values for the dimmed area 236. In addition, during operation of the AR device 200, the projector 214 can project virtual image light 222 (i.e., light associated with virtual content) onto the eyepiece 202, and the virtual image light 222 and the world light 232 are observed by the user together.

[0127] Projecting the virtual image light 222 onto the eyepiece 202 can cause a light field (i.e., an angular representation of virtual content) to be projected onto the user's retina so that the user perceives the corresponding virtual content as being located at a certain position within the user's environment. For example, the virtual image light 222 coupled out by the eyepiece 202 can cause the user to perceive the character 202-1 as being located at the first virtual depth plane 210-1 and the statue 202-2 as being located at the second virtual depth plane 210-2. The user perceives the virtual content as well as the world light 232 corresponding to one or more world objects 230 (such as the platform 120).

[0128] In some embodiments, the AR device 200 may include an ambient light sensor 234 configured to detect world light 232. The ambient light sensor 234 may be positioned such that the world light 232 detected by the ambient light sensor 234 is similar to and / or representative of the world light 232 incident on the dynamic dimming device 203 and / or the eyepiece 202. In some embodiments, the ambient light sensor 234 may be configured to detect a plurality of spatially resolved light values corresponding to different pixels of the dynamic dimming device 203. In these embodiments, the ambient light sensor 234 may correspond, for example, to an imaging sensor (e.g., CMOS, CCD, etc.) or a plurality of photodiodes (e.g., in an array or another spatially distributed arrangement). In some embodiments, or in the same embodiment, the ambient light sensor 234 may be configured to detect a global light value corresponding to the average light intensity or a single light intensity of the world light 232. In these embodiments, the ambient light sensor 234 may correspond, for example, to a collection of one or more photodiodes. Other possibilities are contemplated.

[0129] Figure 2B An example of the AR device 200 is shown, where a dimmed region 236 is determined based on the detected light information corresponding to the world light 232. Specifically, the ambient light sensor 234 may detect the world light 232 associated with the sun 250 and may further detect the direction and / or portion of the world light 232 associated with the sun 250 passing through the AR device 200 in the system field of view. In response, the dynamic dimming device 203 may be adjusted to set the dimmed region 236 to cover the portion of the system field of view corresponding to the detected world light. As shown, the dynamic dimming device 203 may be adjusted so as to reduce the intensity of the world light 232 at the center of the dimmed region 236 by a greater amount than at the edges of the dimmed region 236.

[0130] Figure 2CAn example of an AR device 200 that determines a dimmed area 236 based on virtual image light 222 is shown. Specifically, the dimmed area 236 can be determined based on the virtual content perceived by the user caused by the user observing the virtual image light 222. In some embodiments, the AR device 200 can detect image information including the position of the virtual image light 222 (e.g., the position where the user perceives virtual content within the dynamic dimmer 203) and / or the brightness of the virtual image light 222 (e.g., the brightness of the perceived virtual content and / or the light generated at the projector 214), as well as other possibilities. As shown, the dynamic dimmer 203 can be adjusted to set the dimmed area 236 to cover the portion of the system field of view corresponding to the virtual image light 222, or alternatively, in some embodiments, the dimmed area 236 can cover the portion of the system field of view that is not aligned with the virtual image light 222. In some embodiments, the dimming value of the dimmed area 236 can be determined based on the world light 232 detected by the ambient light sensor 234 and / or the brightness of the virtual image light 222.

[0131] Figure 2D An example of an AR device 200 that determines a dimmed area 236 based on gaze information corresponding to the user's eyes is shown. In some embodiments, the gaze information includes the user's gaze vector 238 and / or the pixel position at which the gaze vector 238 of the dynamic dimmer 203 intersects the dynamic dimmer 203. As shown, the dynamic dimmer 203 can be adjusted to set the dimmed area 236 to cover the portion of the system field of view corresponding to the intersection (or intersection area) between the gaze vector 238 and the dynamic dimmer 203, or alternatively, in some embodiments, the dimmed area 236 can cover the portion of the system field of view that does not correspond to the intersection (or intersection area) between the gaze vector 238 and the dynamic dimmer 203. In some embodiments, the dimming value of the dimmed area 236 can be determined based on the world light 232 detected by the ambient light sensor 234 and / or the brightness of the virtual image light 222. In some embodiments, the gaze information can be detected by an eye tracker 240 installed on the AR device 200.

[0132] Figure 3FIG. shows a schematic diagram of a wearable AR device 300 according to the present invention. The AR device 300 may include a left eyepiece 302A and a left dynamic dimming device 303A arranged in a side-by-side configuration, and a right eyepiece 302B and a right dynamic dimming device 303B also arranged in a side-by-side configuration. In some embodiments, the AR device 300 includes one or more sensors, including but not limited to: a left forward-facing world camera 306A directly attached to or near the left eyepiece 302A; a right forward-facing world camera 306B directly attached to or near the right eyepiece 302B; a left-side-facing world camera 306C directly attached to or near the left eyepiece 302A; a right-side-facing world camera 306D directly attached to or near the right eyepiece 302B; a left eye tracker 340A positioned to observe the user's left eye; a right eye tracker 340B positioned to observe the user's right eye; and an ambient light sensor 334. In some embodiments, the AR device 300 includes one or more image projection devices, such as a left projector 314A optically linked to the left eyepiece 302A and a right projector 314B optically linked to the right eyepiece 302B.

[0133] Some or all of the components of the AR device 300 may be worn on the head such that the projected images can be viewed by the user. In a particular implementation, Figure 3 all of the components of the AR device 300 shown in are mounted on a single device wearable by the user (e.g., a single headset). In another implementation, the processing module 350 is physically separated from the other components of the AR device 300 by one or more wired and / or wireless connections and is communicatively coupled to the other components of the AR device 300. For example, the processing module 350 may be mounted in various configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, embedded in a headset, or otherwise removably attached to the user (e.g., in a backpack configuration, in a belt-coupled configuration, etc.).

[0134] The processing module 350 may include a processor 352 and an associated digital memory 356, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data may include data captured from sensors (e.g., which may be operatively coupled to the AR device 300) or otherwise attached to the user, such as a camera 306, an ambient light sensor 334, an eye tracker 340, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope. For example, the processing module 350 may receive one or more images 320 from the camera 306. Specifically, the processing module 350 may receive one or more left front images 320A from the left forward world camera 306A, one or more right front images 320B from the right forward world camera 306B, one or more left side images 320C from the left side world camera 306C, and one or more right side images 320D from the right side world camera 306D. In some embodiments, the one or more images 320 may include a single image, a pair of images, a video including an image stream, a video including a pair of image streams, etc. The images 320 may be periodically generated and sent to the processing module 350 when the AR device 300 is powered on, or may be generated in response to an instruction sent by the processing module 350 to one or more cameras. As another example, the processing module 350 may receive light information from the ambient light sensor 334. In some embodiments, some or all of the functions of the ambient light sensor 334 may be provided by one or more of the world cameras 306A - 306D. As another example, the processing module 350 may receive gaze information from one or both of the eye trackers 340. As another example, the processing module 350 may receive image information (e.g., image brightness values) from one or both of the projectors 314.

[0135] The eyepieces 302A and 302B may include transparent or translucent waveguides configured to direct light from projectors 314A and 314B, respectively. Specifically, the processing module 350 may cause the left projector 314A to output left virtual image light 322A onto the left eyepiece 302A (such that the corresponding light field associated with the left virtual image light 322A will be projected onto the user's retina), and may cause the right projector 314B to output right virtual image light 322B onto the right eyepiece 302B (such that the corresponding light field associated with the right virtual image light 322B will be projected onto the user's retina). In some embodiments, each of the eyepieces 302 may include multiple waveguides corresponding to different colors and / or different depth planes. In some embodiments, the dynamic dimmers 303 may be coupled to and / or integrated with the eyepieces 302. For example, one of the dynamic dimmers 303 may be incorporated into a multi-layer eyepiece and may form one or more layers that make up one of the eyepieces 302.

[0136] The cameras 306A and 306B may be positioned to capture images that substantially overlap the fields of view of the user's left and right eyes, respectively. Thus, the placement of the cameras 306 may be near the user's eyes, but not so close as to block the user's field of view. Alternatively or additionally, the cameras 306A and 306B may be positioned to be aligned with the light-coupling-in positions of the virtual image lights 322A and 322B, respectively. The cameras 306C and 306D may be positioned to capture images of the user's side, for example, in or outside the user's peripheral vision. The image(s) 320C and 320D captured using the cameras 306C and 306D do not necessarily need to overlap with the image(s) 320A and 320B captured using the cameras 306A and 306B.

[0137] One or more components of the AR device 300 may be similar to one or more components described in reference Figures 2A - 2D For example, in some embodiments, the functions of the eyepieces 302, the dynamic dimmers 303, the projectors 314, the ambient light sensors 334, and the eye trackers 340 may be similar to those of the eyepieces 202, the dynamic dimmers 203, the projectors 214, the ambient light sensors 234, and the eye trackers 240, respectively. In some embodiments, the function of the processing module 350 may be implemented by two or more sets of electronic hardware components that are separately housed but communicatively coupled. For example, the function of the processing module 350 may be performed by electronic hardware components housed within the headset in combination with electronic hardware components housed within a computing device physically tethered to the headset, one or more electronic devices within the headset environment (e.g., a smart phone, a computer, a peripheral device, a smart appliance, etc.), one or more remotely located computing devices (e.g., a server, a cloud computing device, etc.), or a combination thereof. Reference is made below toFigure 29 An example of this configuration is described in further detail.

[0138] Figure 4 A method 400 for operating an optical system (e.g., AR device 200 or 300) is shown. The steps of method 400 may be performed in an order different from the order shown in Figure 4 and not all steps need to be performed. For example, in some embodiments, one or more of steps 406, 408, and 410 may be omitted during the execution of method 400, and one or more steps of method 400 may be performed by a processor (e.g., processor 352) or by some other component within the optical system.

[0139] At step 402, light (e.g., world light 232) associated with a world object (e.g., world object 230) is received at the optical system. The world object may be any number of real-world objects viewed by a user of the optical system, such as a tree, a person, a house, a building, the sun, etc. In some embodiments, the light associated with the world object is first received by a dynamic dimming device (e.g., dynamic dimming device 203 or 303) or by an external decorative lens of the optical system. In some embodiments, when the light reaches one or more components of the optical system (e.g., when the light reaches the dynamic dimming device), the light associated with the world object is considered to be received at the optical system.

[0140] At step 404, virtual image light (e.g., virtual image light 222 or 322) is projected onto an eyepiece (e.g., eyepiece 202 or 302). The virtual image light may be projected onto the eyepiece by a projector (e.g., projector 214 or 314) of the optical system. The virtual image light may correspond to a single image, a pair of images, a video including an image stream, a video including a stream of paired images, etc. In some embodiments, the virtual image light is considered to be projected onto the eyepiece when any light associated with the virtual image light reaches the eyepiece. In some embodiments, projecting the virtual image light onto the eyepiece causes a light field (i.e., an angular representation of the virtual content) to be projected onto the user's retina such that the user perceives the corresponding virtual content as being located at a certain position within the user's environment.

[0141] During steps 406, 408, and 410, information can be detected by an optical system using, for example, one or more sensors of the optical system. At step 406, light information corresponding to light associated with a world object is detected. The light information can be detected using a light sensor (e.g., ambient light sensor 234 or 334) mounted to the optical system. In some embodiments, the light information includes a plurality of spatially resolved light values. Each of the plurality of spatially resolved light values can correspond to a two-dimensional position within the field of view of the system. For example, each light value can be associated with a pixel of the dynamic dimming device. In other embodiments, or in the same embodiment, the light information can include a global light value. The global light value can be associated with the entire field of view of the system (e.g., the average light value of light incident on all pixels of the dynamic dimming device).

[0142] At step 408, gaze information corresponding to the eyes of a user of the optical system is detected. The gaze information can be detected using an eye tracker (e.g., eye tracker 240 or 340) mounted to the optical system. In some embodiments, the gaze information includes a gaze vector of the user's eyes (e.g., gaze vector 238). In some embodiments, the gaze information includes one or more of the pupil position of the user's eyes, the center of rotation of the user's eyes, the pupil size of the user's eyes, the pupil diameter of the user's eyes, and the cone and rod positions of the user's eyes. The gaze vector can be determined based on one or more components of the gaze information, such as the pupil position, the center of rotation of the eyes, the pupil size, the pupil diameter, and / or the cone and rod positions. When the gaze vector is determined based on the cone and rod positions, the gaze vector can also be determined based on the light information (e.g., the global light value) to determine the origin of the gaze vector within the retinal layer of the eye that includes the cone and rod positions. In some embodiments, the gaze information includes a pixel or group of pixels of the dynamic dimming device at which the gaze vector intersects the dynamic dimming device.

[0143] At step 410, image information corresponding to virtual image light (e.g., virtual image light 222 or 322) projected by a projector onto an eyepiece is detected. The image information can be detected by the projector, by a processor (e.g., processor 352), or by a separate light sensor. In some embodiments, the image information includes one or more positions within a dynamic dimming device through which a user perceives virtual content when the user observes the virtual image light. In some embodiments, the image information includes a plurality of spatially resolved image brightness values (e.g., the brightness of the perceived virtual content). For example, each of the image brightness values can be associated with a pixel of the eyepiece or the dynamic dimming device. In one particular implementation, when the processor sends an instruction to the projector to project virtual image light onto the eyepiece, the processor can determine the spatially resolved image brightness values based on the instruction. In another particular implementation, when the projector receives an instruction from the processor to project virtual image light onto the eyepiece, the projector sends the spatially resolved image brightness values to the processor. In another particular implementation, a light sensor located on or near the eyepiece detects the spatially resolved image brightness values and sends them to the processor. In other embodiments, or in the same embodiment, the image information includes a global image brightness value. The global image brightness value can be associated with the entire system field of view (e.g., the average image brightness value of all virtual image light).

[0144] At step 412, a portion of the system field of view to be at least partially dimmed is determined based on the detected information. The detected information can include the light information detected during step 406, the gaze information detected during step 408, and / or the image information detected during step 410. In some embodiments, the portion of the system field of view is equal to the entire system field of view. In various embodiments, the portion of the system field of view can be equal to 1%, 5%, 10%, 25%, 50%, or 75% etc. of the system field of view. In some embodiments, different types of information can be weighted differently when determining the portion to be at least partially dimmed. For example, gaze information can be weighted more heavily than light information and image information when available in determining the portion to be at least partially dimmed. In one particular implementation, each type of information can be independently used to determine a different portion of the system field of view to be at least partially dimmed, and the different portions can then be combined into a single portion using an AND or OR operation.

[0145] In some embodiments, the information for determining the portion of the system field of view to be at least partially dimmed includes information associated with one or more objects presented within the virtual content. For example, the virtual content may include text, navigation indicators (e.g., arrows), and / or other content. The portion of the field of view presenting such content and / or the field of view adjacent to the content may be dimmed so that the user can more easily read and understand the content and distinguish the content from the world object(s). The dimming unit may selectively dim one or more pixels and / or regions in the pixels, or enhance the viewing of the content. In one example, a portion of the lower portion of the field of view may be selectively and dynamically dimmed to make it easier for the user to see orientation (e.g., navigation) arrows, text messages, etc. Such dimming may be performed while the content is being displayed in response to determining that such content is to be displayed, and the dimming may be removed when the content is no longer being displayed. In some cases, dimming may be performed to mitigate artifacts caused by a pixel structure capable of dimming across the entire field of view.

[0146] At step 414, a plurality of spatially resolved dimming values for the portion of the system field of view are determined based on the detected information. In some embodiments, a formulaic approach based on the desired opacity or visibility of the virtual content is used to determine the dimming values. In one particular implementation, the following equation may be used to calculate the visibility of the virtual content:

[0147]

[0148] where V is the visibility, I max is the luminance of the virtual image light as indicated by the image information, I back is related to the light value associated with the world object as indicated by the light information (which may be modified by the determined dimming value), and C is the desired contrast (e.g., 100:1). For example, the visibility equation may be used at each pixel location of the dimming unit to calculate the dimming value at a particular pixel location using the luminance of the virtual image light at the particular pixel location and the light value associated with the world object at the particular pixel location. In some embodiments, the following equation may be used to define I back :

[0149] I back = T v * I world

[0150] where T v is the percentage of light allowed to pass through one or more pixels of the dimming unit, and I world is the luminance of the ambient light from the world as indicated by the light information. In some examples, T v may represent the dimming value or be related to the dimming value.

[0151] At step 416, the dimmer is adjusted to reduce the intensity of the light associated with the object in a portion of the system field of view. For example, the dimmer can be adjusted such that the intensity of the light associated with the object incident on each pixel position of the dimmer is reduced according to the dimming value determined for that particular pixel position. As used in this disclosure, adjusting the dimmer can include initializing the dimmer, activating the dimmer, powering the dimmer, modifying or changing a previously initialized, activated, and / or powered dimmer, and the like. In some embodiments, the processor can send data indicating both the portion of the system field of view and the plurality of spatially resolved dimming values to the dimmer.

[0152] At step 418, the projector is adjusted to adjust the luminance associated with the virtual image light. For example, in some embodiments, it is difficult to achieve the desired opacity or visibility of virtual content without increasing or decreasing the luminance of the virtual object. In such embodiments, the luminance of the virtual image light can be adjusted before, after, synchronously, or simultaneously with the adjustment of the dimmer.

[0153] Figure 5 An augmented reality (AR) device 500 having an eyepiece 502 and a pixelated dimming element 503 is shown. The pixelated dimming element 503 includes a spatial grid of dimming regions (i.e., pixels) that can have various dimming levels. Each dimming region can have an associated dimension 510 (i.e., width) and an associated spacing 520 (i.e., pitch). As shown, the spatial grid of dimming regions can include one or more dark pixels 506 that provide complete dimming of the incident light and one or more clear pixels 508 that provide complete transmission of the incident light. Adjacent pixels within the pixelated dimming element 503 can be adjacent (e.g., when the pitch is equal to the dimension) or can be separated by a gap (e.g., when the pitch is greater than the dimension). In various embodiments, the pixelated dimming element 503 can employ liquid crystal technology, such as dye-doped or guest-host liquid crystals, twisted nematic (TN) or vertically aligned (VA) liquid crystals, or ferroelectric liquid crystals. In some embodiments, the pixelated dimming element 503 can include electrochromic devices and other possibilities. In some implementations, the pixelated dimming element 503 can employ electro-controlled birefringence (“ECB”) technology, such as an ECB cell.

[0154] Figure 6A technique for determining a gaze vector 610 based on the pupil position 605 of a user's eye is shown. In some cases, an eye tracker is used to detect the pupil position 605 relative to the AR device, and the gaze vector 610 is subsequently defined as a vector orthogonal to the surface of the eye at the pupil position 605. The gaze vector 610 may alternatively or additionally be defined as a vector that intersects the center of rotation of the eye and the pupil position 605. The center of rotation can be estimated using data collected by the eye tracker. The gaze vector 610 may alternatively or additionally be defined as a vector that intersects the geometric center of the eye and the pupil position 605. The geometric center of the eye can be estimated using data collected by the eye tracker. Other possibilities are contemplated.

[0155] One of several inherent problems with using the pupil position 605 to determine the gaze vector 610 is shown in Figure 6 In the figure above, a first distance D1 between the pupil position 605 and the eyepiece 602 is shown when the eye is looking generally towards the center of the eyepiece 602. In the figure below, a second distance D2 between the pupil position and the eyepiece is shown when the eye is looking generally towards the top of the eyepiece. Here, the first distance D1 is less than the second distance D2, resulting in a rendering registration problem due to the vergence distance that varies as the user's eye moves.

[0156] Figure 7 A technique for determining a gaze vector 710 based on the center of rotation 720 of a user's eye is shown. The center of rotation 720 can be estimated using data collected by the eye tracker, and the gaze vector 710 can subsequently be defined as the vector formed by connecting the center of rotation 720 and the pupil position 705. One of the many benefits of using the center of rotation 720 to determine the gaze vector 710 is that the distance between the center of rotation 720 and the eyepiece 702 can be the same regardless of the direction the eye is looking. In Figure 7 the upper figure of, a third distance D3 between the center of rotation 720 and the eyepiece 702 is shown when the eye is looking generally towards the center of the eyepiece 702. In the lower figure, a fourth distance D4 between the center of rotation 720 and the eyepiece 702 is shown when the eye is looking generally towards the top of the eyepiece 702. Here, the third distance D3 is the same as the fourth distance D4, thus improving the rendering registration.

[0157] Figure 8Techniques for determining a fixation vector based on detected light information and the positions of cones 804 and rods 806 within an eye are shown. Since cones 804 are more sensitive to light under high light conditions 854 and rods 806 are more sensitive to light under low light conditions 856, when the detected ambient light decreases 850 (e.g., global light value), the origin of the fixation vector can be adjusted outward from the central position corresponding to the high density of cones 804 in the retinal layer to one or more points along a ring corresponding to the high density of rods 806. Thus, under high light conditions 854, the determined fixation vector can be a single fixation vector 814 formed by connecting the central position of the retinal layer to the pupil position, while under low light conditions 856, the determined fixation vector(s) can be a single or multiple fixation vectors 816 formed by connecting one or more points along a ring around the central position of the retinal layer 825 to the pupil position. Alternatively or additionally, multiple fixation vectors can be described / represented as a cone or "fixation cone" of fixation vectors including an infinite number of possible fixation vectors.

[0158] The positions of cones 804 and rods 806 can be estimated using information collected by an eye tracker, or in some embodiments, the central position of the retinal layer corresponding to the high density of cones 804 can be defined by causing the fixation vector determined using the pupil position to continue through the eye towards the rear of the eye such that the fixation vector determined using the pupil position is collinear with the fixation vector determined using the positions of cones 804 and rods 806 under high light conditions 854. In some embodiments, the AR device is configured such that the positions of cones 804 and rods 806 are used to determine the fixation vector under low light conditions 856 (e.g., "low light mode"), while the center of rotation of the eye is used to determine the fixation vector under high light conditions 854. In such embodiments, a light threshold can be established relative to which the detected light value can be evaluated such that when the detected light value is below the light threshold, the positions of cones 804 and rods 806 are used to determine the fixation vector, and when the detected light value is above the light threshold, the center of rotation of the eye is used to determine the fixation vector.

[0159] In some embodiments where the dimmed area is significantly large and / or the dimming value is significantly high, the ambient light detected by the light sensor of the AR device may not indicate the actual amount of light reaching the eye. In such embodiments, the size of the pupil can be used as a proxy for the amount of light reaching the eye. For example, the AR device can switch to a "low light mode" when the pupil size exceeds a pupil size threshold (resulting in using the cone 804 and rod 806 positions to determine the gaze vector). For example, in some implementations, the pupil size threshold can be set to be 20% higher than the average pupil size of the user under high light conditions 854 (e.g., the pupil size can correspond to the area, diameter, perimeter, etc. of the pupil). In another specific embodiment, the pupil size threshold can be pre-determined based on the average known pupil sizes under low light 856 and high light conditions 854. Other possibilities are contemplated.

[0160] Figure 9 The gaze vector 914 determined under high light conditions with pupil constriction is shown. In some embodiments, the pupil size 930 can be used to estimate the ambient light (e.g., the global light value), such as the reduced ambient light 950, or alternatively or additionally, the origin(s) of the gaze vector(s) 914 can be directly determined using the pupil size 930 without estimating or detecting the ambient light 950. For example, different pupil diameters can be associated with different cone 904 and rod 906 positions within the retina layer 925, at which the origin(s) of the gaze vector(s) can be defined.

[0161] Figure 10 The gaze vector 1016 determined under low light conditions with pupil dilation is shown. Similar to the scenario under high light conditions, under low light conditions, the pupil size 1032 can be used to estimate the ambient light (e.g., the global light value), such as the reduced ambient light 1050, or alternatively or additionally, the origin(s) of the gaze vector(s) can be directly determined using the pupil size. Similar to Figure 9 , example different pupil diameters can be associated with different cone 1004 and rod 1006 positions within the retina layer 1025, at which the origin(s) of the gaze vector(s) can be defined.

[0162] Figure 11Shows three techniques for determining a gaze vector under high - light conditions, such as outdoor ambient light 1155 with 5000 NITS, and the corresponding dimmed regions determined using each of the three techniques. In the first technique, the pupil position is used to determine the gaze vector, resulting in a gaze vector 1110A that orthogonally extends from the surface of the pupil towards the dimmed region A (or in some embodiments, the non - dimmed region). In the second technique, the cone 1104 and rod 1106 positions within the eye are used to determine the gaze vector, resulting in a gaze vector 1110B that extends from the central position of the retinal layer through the pupil position towards the dimmed region (or in some embodiments, the non - dimmed region). The second technique can be further facilitated by one or more of the following: pupil position (for providing a second point for defining the gaze vector), detected ambient light (for determining the origin(s) along the retinal layer of the gaze vector(s)), and pupil size / diameter (for estimating the ambient light and / or for directly determining the origin(s) along the retinal layer 1125 of the gaze vector(s)). In the third technique, the center of rotation of the eye is used to determine the gaze vector, resulting in a gaze vector 1110C that extends from the center of rotation of the eye through the pupil position towards the dimmed region C (or in some embodiments, the non - dimmed region). In Figure 12 the example, the dimmed region A is the same as the dimmed region B.

[0163] Figure 12 Shows Figure 11 the same techniques as shown in Figure 12 but under low - light conditions, such as outdoor ambient light 1257 with 100 NITS. The gaze vectors 1210A and 1210C determined using the first and third techniques (using pupil position and center of rotation respectively) and the corresponding dimmed regions A and C are the same, but the second technique (using cone 1204 and rod 1206 positions) has been modified. In the second technique, the cone 1204 and rod 1206 positions within the eye are used to determine the gaze vector, resulting in a set of gaze vectors 1210B' that extend from various points on an annulus around the central position along the retinal layer 1225 through the pupil position towards the dimmed region B' (or in some embodiments, the non - dimmed region). In the example shown, each of the dimmed regions A, B', and C is different from each other.

[0164] Figure 13 Shows a dimmer 1303 that has been adjusted to produce the dimmed region A determined using a gaze vector calculated using the pupil position.

[0165] Figure 14Shown is a dimmer 1403 that has been adjusted to produce a dimmed region B determined using a gaze vector that is calculated using cone and rod positions under high light conditions, such as an outdoor ambient light of 5000 NITS.

[0166] Figure 15 Shown is a dimmer 1503 that has been adjusted to produce a dimmed region B' determined using a gaze vector that is calculated using cone and rod positions under low light conditions, such as an outdoor ambient light of 100 NITS. In an alternative embodiment, the dimmed region B' may include only Figure 15 a portion of the annular region shown rather than the entire region thereof.

[0167] Figure 16 Shown is a dimmer 1603 that has been adjusted to produce a dimmed region B' under an outdoor ambient light 1657 of 100 NITS, where the dimmed region B' also includes a central portion within the annular region.

[0168] Figure 17 Shown is a dimmer 1703 that has been adjusted to produce a dimmed region C at an outdoor ambient light, such as 5000 NITS. A gaze vector is used to determine the resulting dimmed region C, and the gaze vector is calculated using the center of rotation of the eye.

[0169] Figure 18A and Figure 18B Shown is a method for determining a portion of a system field of view to be dimmed based on image information. For example, Figure 18A and Figure 18BOne or more of the steps shown in FIG. may correspond to steps 410 and / or 412. In some embodiments, the AR device may project light onto the eyepiece 1812 in such a way that virtual content is perceived by the user at various points in the space outside the eyepiece 1812 and the dynamic dimming device 1803, such as points 1802-1, 1802-02, and 1802-03. Points 1802 (e.g., 1802-01, 1802-02, and 1802-03) may correspond, for example, to positions in three-dimensional space, including positions where pixels of virtual content (e.g., one or more virtual objects) will be perceived by the user when presented through the eyepiece 1812, positions where dark virtual content (e.g., virtual "shadows" projected by or otherwise associated with the virtual content presented through the eyepiece 1812) will be perceived by the user, positions physically occupied by one or more real-world objects or persons in the user's environment (e.g., a virtual black "top hat" anchored to someone's head in the user's environment), etc. In some implementations, points 1802 (e.g., 1802-01, 1802-02, and 1802-03) may be randomly sampled from the virtual content, or in some embodiments, points 1802 may be selected based on key features of the virtual content, such as edges, corners, surface centers, and other possibilities. In some embodiments, points 1802 (e.g., 1802-01, 1802-02, and 1802-03) may be sampled from the outer perimeter of the virtual content (as viewed from a reference point). In other embodiments, or in the same embodiment, the image brightness of the virtual content is also determined at each of the points 1802 (e.g., 1802-01, 1802-02, and 1802-03), which can be used to determine the dimming level (i.e., dimming value) at point 1802 to achieve the desired visibility V of the virtual content. The number of points 1802 used may vary based on a speed-accuracy tradeoff.

[0170] To align dimming with the perceived virtual content, vectors 1804 (e.g., 1804-01, 1804-02, and 1804-03) may be defined to intersect each of the points 1802 (e.g., 1802-01, 1802-02, and 1802-03) and the pupil position (i.e., the reference point). Intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03) may then be defined at each position where the vector 1804 intersects the dynamic dimming device 1803. As referenced Figure 18BAs shown, the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be determined based on the intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03). In some embodiments, one or more light or cone projection techniques can be employed to define the vectors 1804 (e.g., 1804-01, 1804-02, and 1804-03) and identify or otherwise determine the intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03). In some embodiments, each of the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be set to include the regions that contain each of the intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03), or specific pixels of the dynamic dimmers 1803 that contain the intersection points 1806. In some embodiments, the size of the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be a function of the number of sampling points 1802 (e.g., 1802-01, 1802-02, and 1802-03) and / or the density of the points 1802. For example, in some instances, the size of the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be inversely proportional to the number of points 1802 (e.g., 1802-01, 1802-02, and 1802-03). In embodiments where the sampling points 1802 (e.g., 1802-01, 1802-02, and 1802-03) are sampled from the outer perimeter of the virtual content, the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be formed by connecting adjacent intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03) and dimming the enclosed regions. In some examples, the size and / or shading of the dimmed portions 1808 (e.g., 1808-01, 1808-02, and 1808-03) can be a function of a determined distance from a reference point to the intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03), a determined distance from the intersection points 1806 (e.g., 1806-01, 1806-02, and 1806-03) to the points 1802 (e.g., 1802-01, 1802-02, and 1802-03), or a combination thereof. In Figure 18A and Figure 18BIn the example of, the pupil position (e.g., pupil center) that defines the starting position (i.e., reference point) of vector 1804 (e.g., 1804-01, 1804-02, and 1804-03) can change over time as the eye 1800 moves. Accordingly, the position of intersection point 1806 (e.g., 1806-01, 1806-02, and 1806-03) and the dimmed portion 1808 (e.g., 1808-01, 1808-02, and 1808-03) can also change over time as the eye 1800 moves.

[0171] Figure 19A and Figure 19B illustrates a method for determining a portion of the system field of view to be dimmed based on image information similar to that shown in Figure 18A and Figure 18B but with a different reference point. In some embodiments, the AR device can project light onto the eyepiece 1912 in such a way that virtual content is perceived by the user at various points in the space outside the eyepiece 1912 and the dynamic dimming device 1903, such as point 1902. Points 1902 (e.g., 1902-01, 1902-02, and 1902-03) can represent different points in the space where the user perceives the virtual content. Vector 1904 (e.g., 1904-01, 1904-02, and 1904-03) can be defined as intersecting each of points 1902 (e.g., 1902-01, 1902-02, and 1902-03) and the center of rotation 1920 of the eye 1900 (i.e., reference point). Then, an intersection point 1906 (e.g., 1906-01, 1906-02, and 1906-03) can be defined at each position where vector 1904 (e.g., 1904-01, 1904-02, and 1904-03) intersects the dynamic dimming device. As referenced in Figure 19BAs shown, the dimmed portions 1908 (e.g., 1908-01, 1908-02, and 1908-03) can be determined based on the intersection points 1906 (e.g., 1906-01, 1906-02, and 1906-03). In some embodiments, each of the dimmed portions 1908 (e.g., 1908-01, 1908-02, and 1908-03) can be set to include an area that contains each of the intersection points 1906 (e.g., 1906-01, 1906-02, and 1906-03) or a specific pixel of a dynamic dimming device that contains the intersection point 1906. In some examples, the size and / or shading of the dimmed portions 1908 (e.g., 1908-01, 1908-02, and 1908-03) can be a function of a determined distance from a reference point to the intersection points 1906 (e.g., 1906-01, 1906-02, and 1906-03), a determined distance from the intersection points 1906 to the points 1902 (e.g., 1902-01, 1902-02, and 1902-03), or a combination thereof. Compared to the pupil position of the reference point in the example of Figure 18A and Figure 18B , when eye 1900 movement occurs, the position of the center of rotation 1920 of the eye 1900, which is the starting position (i.e., the reference point) that defines the vector 1904 (e.g., 1904-01, 1904-02, and 1904-03) in the example of Figure 19A and Figure 19B , may be more stable over time. Thus, in the example of Figure 19A and Figure 19B , when eye 1900 movement occurs, the positions of the intersection points 1906 (e.g., 1906-01, 1906-02, and 1906-03) and the dimmed portions 1908 (e.g., 1908-01, 1908-02, and 1908-03) may remain stationary or change relatively little over time. Although the pupil position of the eye 1900 and the center of rotation 1920 are described above as examples of reference points that can be used to determine the portion to be dimmed of the system's field of view in Figure 18A , Figure 18B , Figure 19A and Figure 19B , it should be understood that examples of such reference points can also include any one of various other positions along the optical axis of the eye.

[0172] Figure 20An example of using any of the techniques described herein to improve the solidity of the displayed virtual content is shown, such as adjusting the dimmer and / or adjusting the projector based on light information, gaze information, and / or image information. Referring to the left and right fields of view, the virtual content 2004 displayed next to the world object 2002 appears washed out except for the portion 2006 of the virtual content 2004 where the virtual content appears denser than the remainder of the virtual content 2004. As shown in the example, the solidity of the virtual content is improved only at the portion of the system field of view where the user is looking.

[0173] Figure 21 An example of improving the solidity of the displayed virtual object 2102 by dimming the portion of the system field of view corresponding to the virtual object is shown. As shown, the opacity and visibility of the portion 2104 of the virtual object 2102 located in the dimmed area are relatively greater than the opacity and visibility of the portion 2106 of the virtual object 2102 located in the non-dimmed area. By dimming the light associated with the world object 2108 at portion 2104, the user can more clearly perceive the virtual content.

[0174] Figure 22 A graph showing the relationship between virtual image luminance (x-axis) and ambient light level for maintaining a visibility equal to 0.7 (i.e., V = 0.7) is shown. For different ambient light level conditions, the solid slanted line is a fixed visibility horizontal line (for V = 0.7). For example, for a projector brightness of 200 nits used in an indoor area of approximately 100 nits, a dimming level of close to 30% can be employed to keep the visibility close to 0.7. Referring again to the visibility equation described above Figure 4 In some examples, Figure 22 the x-axis and y-axis of the graph shown in max can correspond to I v and T world respectively, while the solid slanted line is a fixed visibility horizontal line (for V = 0.7) for different I

[0175] Figure 23A and 23B show a diagram illustrating the effect of a small occlusion on the world scene. Figure 23A A simple case where the user's eye is looking at infinity is shown. The eye includes a retina 2302, a pupil 2304, and a lens 2306. Light from different angles is focused on different positions on the retina 230. Figure 23BAn occlusor 2308 placed in front of the eye at a distance d away from the pupil 2304 is shown. A simple ray geometry can be used to construct the gradient disk at the retina. Ignoring diffraction, the relative transmittance at the center of the gradient disk is t0 = 1 - (h / p) 2 , where h is the diameter of the occlusor and p is the diameter of the pupil. In other words, t0 = 1 - A 遮挡物 / A 瞳孔 , where A 遮挡物 is the area of the occlusor and A 瞳孔 is the area of the pupil.

[0176] Figure 24 A graph is shown that depicts the effect of changing the occlusor diameter on the transmittance of the dimming element as a function of the angular range (in degrees). As shown, a smaller occlusor diameter (e.g., 1 mm) has a very small effect on transmittance but is more stable over the angular range than a larger occlusor diameter (e.g., 4 mm), and a larger occlusor diameter has a higher impact on the transmittance that varies significantly over the angular range.

[0177] Figure 25 An example of dimming using a single occlusor is shown, where d = 17 mm, p = 4 mm, and h = 1 mm. The dimmed region shows the point spread function (PSF) 2502 of a single pixel. Using the shown dimming, the pixel size requirement for the particular dimming element used can be estimated as 200 μm pixels.

[0178] Figure 26 An example of the shutter transmittance 2650 architecture of an OST head-mounted display (HMD) is shown, including a diffractive waveguide eyepiece 2602 that delivers virtual content (e.g., via display light 2636) to the user's eye. In some embodiments, the HMD is a wearable device wearable on the user's head. The diffractive waveguide eyepiece 2602 may include one or more diffractive optical elements (DOEs), such as an input coupling grating (ICG), an ortho-pupil expander (OPE), and / or an output pupil expander (EPE 2640). World light 2632 also reaches the user's eye through the same elements. As shown, a dynamic dimmer 2604 allows for managing the world light level (e.g., dimmed world light 2634) to keep the virtual content at a certain opacity level. In some embodiments, the dimmer 2604 may correspond to a pixelated dimming element that is functionally similar to or equivalent to the pixelated dimming element 503 described above with reference to Figure 5 In other embodiments, the dimmer 2604 may correspond to a global (non-pixelated) dimming element. As Figure 26As shown, in some embodiments, the dimmer 2604 can be shaped and curved independently of the eyepiece to improve the aesthetics and / or functionality of the OST-HMD.

[0179] Figure 27 Additional examples of the architecture of an OST-HMD including a microdisplay (e.g., LCOS, MEMS, or fiber scanner display types) are shown, where the microdisplay uses a relay optical system to deliver light 2736 into the input grating of a diffractive waveguide structure. The waveguide structure can include an output grating (e.g., EPE 2740) that magnifies the input image plane and delivers it to the user's eye 2700. As shown, various elements can be located between the user's eye 2700 and the world object. The eyepiece 2702 can be a diffractive waveguide combiner that delivers virtual light 2736 to the user's eye 2700 and also allows world light 2732 to pass through. The variable focus element 2704 can include a depth plane change / switching element between the eye 2700 and the eyepiece 2702 to act on the virtual display. In some embodiments, the variable focus element 2704 is a back lens assembly (BLA) 2706. The BLA also acts invariantly on the world light, and thus a front lens assembly (FLA) 2708 is added to eliminate the impact on the world display.

[0180] The dynamic dimming element 2710 in this embodiment is mounted externally to the integrated stack. This allows for switching from a transparent display for AR mode to an opaque display for VR mode that completely blocks world light. The dimming element 2710 can correspond to a global dimming element or a pixelated dimming element. The external lens 2712 is positioned separate from the optical stack to provide a protective and / or support structure for the OST-HMD. The external lens 2712 can also provide a dimming amount to the entire system field of view.

[0181] Figure 28 Additional examples of the architecture of an OST-HMD are shown, where a flat dynamic dimmer 2802 is positioned along the inside of a curved external decorative lens 2804 to convert world light 2832 into doubly dimmed world light 2834. The dimmer 2802 can correspond to a global dimming element or a pixelated dimming element. In some embodiments, the external decorative lens 2804 can provide a dimming amount to the entire system field of view, which can be considered when determining the spatially resolved dimming values of the dynamic dimmer 2802. The OST-HMD can also include an eyepiece 2806, an adaptive BLA 2808, an adaptive FLA 2810, and an EPE 2840 to display light 2836 to the user's eye 2800, as described herein.

[0182] Figure 29A schematic diagram of an AR device 2900 according to the present invention is shown. The AR device 2900 generally includes a local module 2910 and a remote module 2912. The division of the components of the AR device 2900 between the local module 2910 and the remote module can allow large and / or high-power components to be separated from those positioned close to the user's head when the AR device 2900 is in use, thereby increasing user comfort and device performance. The local module 2910 can be head-mounted and can include various mechanical and electronic modules to facilitate the control of the pixelated dimming device 2903 and the spatial light modulator 2904. The control of the spatial light modulator 2904 can cause virtual content to be projected onto the eyepiece 2902, which is viewed by the user of the AR device 2900 together with the world light modified by the dimming device 2903. One or more sensors 2934 of the local module 2910 can detect information from the world and / or the user, and send the detected information to the sensor headset processor 2940, which can send the data stream to the display headset processor 2942 of the local module 2910, and send the original or processed image to the perception processing unit 2944 of the remote module 2912.

[0183] In some embodiments, one or more components of the local module 2910 can be similar to one or more components described with reference to Figure 3 For example, in such embodiments, the functions of the eyepiece 2902 and the dimming device 2903 can be similar to the functions of the eyepiece 302 and the dimming device 303, respectively. In some examples, one or more sensors 2934 can include one or more of a world camera, an ambient light sensor, and / or an eye tracker that are respectively similar to one or more of the world camera 306, the ambient light sensor 334, and / or the eye tracker 340. In some embodiments, the functionality of the spatial light modulator 2904 can be similar to the functionality of one or more components included in the projector 314, and the functionality of one or both of the sensor headset processor 2940 and the display headset processor 2942 can be similar to the functionality of one or more components included in the processing module 350.

[0184] In some embodiments, the display headset processor 2942 may receive virtual content data and pixelated dimming data from the graphics processing unit (GPU) 2946 of the remote module 2912, and may perform various correction and warp techniques before controlling the pixelated dimming 2903 and the spatial light modulator 2906. The dimming data generated by the display headset processor 2942 may be passed through one or more drivers, which may modify or generate voltages for controlling the dimming 2903. In some embodiments, the display headset processor 2942 may receive depth images and headset poses from the sensor headset processor 2940, which may be used to improve the accuracy of the dimmed and projected virtual content.

[0185] The remote module 2912 may be electrically coupled to the local module 2910 through one or more wired or wireless connections, and may be fixedly attached to the user or carried by the user, among other possibilities. The remote module 2912 may include a perception processing unit 2944 for performing / generating an ambient lighting map, headset pose, and eye sensing. The perception processing unit 2944 may send data to the CPU 2948, which may be configured to perform / generate the transmissive world geometry and the application scene geometry. The CPU 2948 may send data to the GPU 2946, which may be configured to perform checks for minimum world luminance throughput, dimming pixel alignment, post-frame time warping, and the rendering pipeline, among other operations. In some embodiments, the CPU 2948 may be integrated with the GPU 2946 such that a single processing unit may perform one or more of the functions described with reference to each. In some embodiments, the functionality of one or more of the components included in the remote module 2912 may be similar to the functionality of one or more of the components included in the processing module 350.

[0186] Indeed, as demonstrated by the systems and techniques described herein, dynamic spatial ambient light attenuation can be advantageously exploited in an optical see-through augmented reality (AR) head-mounted display (HMD) with a diffractive waveguide combiner to maintain virtual content realism under varying light conditions. However, different lighting conditions may pose additional challenges beyond those related to virtual content realism. For example, in some cases, overhead lighting conditions in the user's environment of an AR HMD may produce diffraction artifacts in the user's field of view. A variety of other factors and lighting conditions may also be used to introduce diffraction artifacts into the user's field of view. Accordingly, it may be desirable to employ one or more systems and / or techniques in such an AR HMD for managing diffraction artifacts in the user's field of view across a range of lighting environments.

[0187] In some embodiments, one or more components configured to provide a diffraction artifact management function may be incorporated into and / or disposed adjacent to a dimmer assembly (e.g., a dynamic dimmer), such as one or more of the components described above. In some embodiments, one or more such systems and techniques for managing diffraction artifacts in an AR HMD having a dimmer assembly may be used in an AR HMD having a dimmer assembly similar to or equivalent to one or more of the dynamic dimmers described above with reference to Figures 1 to 29 In some embodiments, one or more such systems and techniques for managing diffraction artifacts in an AR HMD may be used in an AR HMD having a dimmer assembly similar to or equivalent to one or more of the above-described dimmer assemblies.

[0188] In some embodiments, one or more of the above-described systems and techniques for managing diffraction artifacts may be used in an AR HMD having a dimmer assembly similar to or equivalent to one or more of the dynamic dimmers 2604 and 2802 described above with reference to Figure 26 and Figure 28 respectively. In such embodiments, it may also be important that the dimmer assembly meet certain aesthetic and industrial design (ID) requirements (the look and feel of the product) and / or provide shatter resistance to effectively protect the eyepiece and other delicate components of the system. Given that many of the above-described systems and techniques for maintaining virtual content realism and / or managing diffraction artifacts in different light conditions involve incorporating additional layers (e.g., films, lenses, and other optical components) into the optical stack of the AR HMD, meeting such aesthetic and physical requirements can be challenging. Additionally, adding layers to the optical stack of the AR HMD also results in increased thickness and mass, and in some cases, may also introduce transmission losses and visible artifacts. As described in further detail below, in some embodiments, an ambient light management module (ALMM) with a novel design that addresses these issues may be used in an AR HMD.

[0189] In some embodiments, the ALMM is a single integrated optical element. The ALMM may be fabricated on a plastic or other suitable substrate and may be incorporated into the AR HMD on the world side of the display combiner. In some examples, the ALMM may be curved in shape in one or two dimensions. The single integrated element is configured to maintain virtual content realism and manage diffraction artifacts in varying light conditions, but adds a relatively small thickness and mass to the optical stack of the AR HMD and introduces little (or no) additional transmission loss and visible artifacts.

[0190] Figure 30 An example of an AR HMD including an ambient light management module (ALMM 3050) is shown in accordance with some embodiments.Figure 31 shows a more detailed example of an Figure 30 AR - HMD and an ALMM 3150 according to some embodiments. As Figure 30 shown, the AR HMD includes an ALMM 3050 located between the user's eyes 3000 and the world 3030, and an eyepiece stack 3005 located between the user's eyes 3000 and the ALMM 3050. In some embodiments, the ALMM 3050 may include one or more of the following: a one - dimensional or two - dimensional (doubly - curved, spherical) curvature plastic substrate 3010 (1D / 2D curved substrate), a dimming component 3003 for spatial light attenuation to improve content solidity, a segmented dimming component, an angular attenuator component 3008 for high - angle light attenuation (rainbow artifact mitigation layer), and / or other world - facing films 3012 for the aesthetic or decorative appearance of the device. In some embodiments, such as in the ​ and ​ examples shown, the ALMM 3050 further includes a polarizer P1 between the plastic substrate and the dimming component 3003, a polarizer P2 between the dimming component 3003 and the angular attenuator component 3008, and a polarizer P3 between the angular attenuator component 3008 and the world - facing film 3012. In some examples, the polarizers P1 - P3 can be linear polarizers. For example, in such an embodiment, the functions of the polarizers P1 - P3 can be similar to or equivalent to the functions of the polarizers 1120a - 1120c. Embodiments also support including more or fewer polarizers in the ALMM 3050.

[0191] The curvature of the plastic substrate 3010 can be used to enhance the aesthetic appearance of the ALMM 3050. In some embodiments, the plastic substrate 3010 of the ALMM3050 can have a one - dimensional or two - dimensional curvature angle between 10 degrees and 30 degrees. In some examples, the plastic substrate 3010 of the ALMM 3050 can have a one - dimensional or two - dimensional curvature radius between 100 and 200 mm (e.g., 120 mm, 140 mm, etc.). Additionally, in some examples, the plastic substrate 3010 of the ALMM 3050 can have a diameter between 20 and 60 mm (e.g., 35 mm, 45 mm, etc.). Additionally, in some embodiments, the plastic substrate 3010 can be robust and / or strong enough to meet the breakage protection requirements for AR HMD products (e.g., can withstand a dropped ball).

[0192] As ​As shown, the AR HMD may include an ALMM 3150 located between the user's eyes 3100 and the world 3130, and an eyepiece stack 3105 located between the user's eyes 3100 and the ALMM 3150. In some embodiments, the dimming component 3003 / 3103 includes a liquid crystal (LC) spatial light modulator based on flexible organic thin-film transistors (O-TFTs) laminated on a 1D / 2D curved plastic substrate. As ​ As shown, in some embodiments, the dimming component 3103 may include a plurality of layers D1-D5. In some examples, layers D1 and D5 correspond to plastic films or substrates configured to hold the dimming component 3103 together and / or protect other layers contained therein. In some examples, layer D2 corresponds to the pixel electrode and thin-film transistor layer, layer D3 corresponds to the liquid crystal (LC) layer, and layer D4 corresponds to the common plane electrode. The electrodes of the dimming component 3103 may be made of, for example, an optically transmissive material that is also conductive, such as indium tin oxide (ITO). The components of the dimming component 3103 may be flexible in nature, such that the dimming component 3103 can be disposed in the ALMM 3150 in a manner that generally conforms to the profile presented by the curved plastic substrate 3110. For example, in some embodiments, the pixel electrodes of layer D2 may have a geometry similar to or equivalent to one or more of the pixel geometries. In at least some of these embodiments, the pixel electrodes may exhibit a geometry similar to or equivalent to one or more of the above-described pixel geometries in the x and y dimensions, and may be further shaped to conform to the curvature of the ALMN in the z dimension. In some examples, the positions of layers D2 and D4 may be swapped.

[0193] In some embodiments, the angular attenuator component 3018 / 3108 includes a birefringent film laminated on a 1D / 2D curved plastic substrate 3110 to form a diffraction artifact / rainbow mitigation film. As ​ As shown, in some embodiments, the angular attenuator component 3108 may include a plurality of layers A1-A3. In some examples, layers A1 and A3 correspond to quarter-wave plates (QWPs), and layer A2 corresponds to a C-plate.

[0194] In some embodiments, the world-facing film(s) 3012 / 3112 includes one or more wire-grid polarizers or cholesteric liquid crystal (CLC) films laminated on a 1D / 2D curved plastic substrate to provide reflection and / or color to the user when viewing the device from the outside. As ​As shown, in some embodiments, the world-facing film(s) 3112 may include multiple layers W1 and W2. In some implementations, layer W1 corresponds to a wire grid polarizer (WGP) or a multilayer reflective polarizer (MLP), and layer W2 corresponds to a super retarder / achromatic quarter-wave plate (AQWP). In other implementations, layer W1 corresponds to an AQWP, and layer W2 corresponds to a cholesteric liquid crystal polarizer (CLCP). In some embodiments, the world-facing film(s) 3012 / 3112 includes more than two layers (e.g., four layers, five layers, etc.).

[0195] Although not shown in ​ and ​ , in some embodiments, a laminate of two or more layers may be disposed within or adjacent to the ALMM 3050 / 3150. In some examples, the ALMM 3050 / 3150 may include one or more additional layers configured to attenuate ghosting. In some implementations, the ALMM 3050 / 3150 may include one or more additional QWPs. For example, in such an implementation, the ALMM 3050 / 3150 may include a QWP between the plastic substrate and the polarizer P1, a QWP between the polarizer P1 and the dimming component, a QWP between the dimming component and the polarizer P2, or a combination thereof. Additionally, in some embodiments, the ALMM 3050 / 3150 may include one or more anti-reflection layers. For example, in such an embodiment, the ALMM 3050 / 3150 may include an anti-reflection layer between the plastic substrate and the polarizer P1, an anti-reflection layer between the world-facing film(s) and the world, or a combination thereof. Further, in some implementations, the space between the ALMM 3050 / 3150 and the eyepiece stack 3005 / 3105 may be at least partially filled with a refractive index matching material.

[0196] In some examples, ​ and ​ the eyepiece stack 3005 / 3105 may be bent in a manner similar to the ALMM 3050 / 3150. Additionally, ​ and ​ the eyepiece stack 3005 / 3105 may include multiple waveguides. In some embodiments, one or more of the multiple waveguides may be made of a polymer-based material.

[0197] ​ FIG. shows a simplified computer system 3200 according to some embodiments described herein. The computer system 3200 shown in ​ may be incorporated into a device such as the AR device 200 or 300 described herein. ​FIG. 3200 provides a schematic illustration of an example of a computer system that can perform some or all of the steps of the methods provided by the various embodiments. It should be noted that ​ is only intended to provide a general illustration of the various components, any one or all of which may be utilized as appropriate. Thus, ​ broadly illustrates how the individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0198] FIG. 3200 shows that the computer system includes hardware elements that may be electrically coupled via a bus 3205 or may be communicatively coupled in other suitable ways. The hardware elements may include one or more processors 3210, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors, such as digital signal processing chips, graphics acceleration processors, and / or the like; one or more input devices 3215, which may include but are not limited to a mouse, a keyboard, a camera, and / or the like; and one or more output devices 3220, which may include but are not limited to a display device, a printer, and / or the like.

[0199] The computer system 3200 may also include and / or communicate with one or more non-transitory storage devices 3225, which may include but are not limited to local and / or network-accessible storage devices, and / or may include but are not limited to disk drives, drive arrays, optical storage devices, solid-state storage devices (such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, updatable), and / or the like. Such storage devices may be configured to implement any suitable data storage, including but not limited to various file systems, database structures, and the like.

[0200] The computer system 3200 may further include a communication subsystem 3219, which may include but is not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or such as Bluetooth TMChip sets for devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication facilities, and / or the like. The communication subsystem 3219 may include one or more input and / or output communication interfaces to allow data exchange with a network, such as the networks described below, by way of example, other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other embodiments, the portable electronic device or the like may transmit images and / or other information via the communication subsystem 3219. In other embodiments, the portable electronic device (e.g., the first electronic device) may be incorporated into the computer system 3300, such as an electronic device of the input device 3215. In some embodiments, as described above, the computer system 3200 will further include a working memory 3235, which may include RAM or ROM devices.

[0201] The computer system 3200 may further include software elements shown as currently residing within the working memory 3235, including an operating system 3240, device drivers, executable libraries, and / or other code, such as one or more applications 3245, as described herein, the application 3245 may include computer programs provided by various embodiments, and / or may be designed to implement methods and / or configure systems provided by other embodiments. By way of example only, one or more of the processes described with respect to the above methods may be implemented as code and / or instructions executable by a computer and / or a processor within the computer, and thus, in one aspect, such code and / or instructions may be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the described methods.

[0202] This set of instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as the storage device 3225 described above. In some cases, the storage medium may be incorporated within a computer system, such as the computer system 3200. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium such as an optical disc), and / or provided in an installation package such that the storage medium can be used to program, configure, and / or adapt a general-purpose computer and the instructions / code stored thereon. These instructions may take the form of executable code executable by the computer system 3200, and / or may take the form of source code and / or installable code, which, for example, when compiled and / or installed on the computer system 3200 using various commonly available compilers, installers, compression / decompression utilities, etc., then take the form of executable code.

[0203] It will be apparent to those skilled in the art that substantial variations can be made in accordance with specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software including portable software such as applets, or both. Additionally, connections to other computing devices such as network input / output devices may be employed.

[0204] As described above, in one aspect, some embodiments may employ a computer system such as computer system 3200 to perform the methods according to various embodiments of the present technology. According to a set of embodiments, some or all of the processes of such methods are performed by computer system 3200 in response to one or more sequences of one or more instructions executed by processor 3210, which may be incorporated into the operating system 3240 and / or other code included in working memory 3235, such as application 3245. Such instructions may be read into working memory 3235 from another computer-readable medium, such as one or more storage devices 3225. By way of example, the execution of the sequence of instructions included in working memory 3235 may cause processor 3210 to perform one or more of the processes of the methods described herein. Additionally or alternatively, portions of the methods described herein may be performed by dedicated hardware.

[0205] As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any medium that participates in providing data that causes a machine to operate in a particular fashion. In embodiments implemented using computer system 3200, various computer-readable media may be involved in providing instructions / code to processor 3210 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, the computer-readable medium is a physical and / or tangible storage medium. Such media may take the form of non-volatile media or volatile media. Non-volatile media includes, for example, optical discs and / or magnetic disks, such as storage device 3225. Volatile media includes, but is not limited to, dynamic memory, such as working memory 3235.

[0206] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0207] Computer-readable media in various forms may be involved in carrying one or more sequences of one or more instructions to the processor 3210 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as a signal through a transmission medium to be received and / or executed by the computer system 3200.

[0208] The communication subsystem 3219 and / or its components will generally receive the signal, and then the bus 3205 may carry the signal and / or the data, instructions, etc. carried by the signal to the working memory 3235, from which the processor 3210 fetches and executes the instructions. The instructions received by the working memory 3235 may optionally be stored on the non-transitory storage device 3225 before or after being executed by the processor 3210.

[0209] The methods, systems, and devices discussed above are examples. Various configurations may appropriately omit, substitute, or add various processes or components. For example, in an alternative configuration, the method may be executed in a different order than that described, and / or various stages may be added, omitted, and / or combined. Additionally, the features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Moreover, technology is evolving, and thus, many of the elements are examples and do not limit the scope of the present disclosure or the claims.

[0210] Specific details are given in the specification to provide a thorough understanding of the exemplary configurations including the embodiments. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the configurations. The description merely provides example configurations and does not limit the scope, applicability, or configuration of the claims. Instead, the previous description of the configurations will provide those skilled in the art with an enabling description for implementing the described techniques. Various changes may be made to the function and arrangement of the elements without departing from the spirit or scope of the present disclosure.

[0211] In addition, a configuration may be described as a process, which is described as a schematic flowchart or block diagram. Although each operation may be described as a sequential process, many operations may be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. The process may have additional steps not included in the figures. Moreover, examples of the method may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. The processor may execute the described tasks.

[0212] Several example configurations have been described and various modifications, alternative constructions, and equivalent forms can be used without departing from the spirit of the present disclosure. For example, the above elements can be components of a larger system where other rules may take precedence over or otherwise modify the application of the techniques. Also, many steps can be taken before, during, or after considering the above elements. Accordingly, the above description does not limit the scope of the claims.

[0213] As used herein and in the appended claims, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to "a user" includes a plurality of such users, and a reference to "a processor" includes a reference to one or more processors and their equivalents known to those skilled in the art, and so forth.

[0214] Furthermore, when used in this specification and the appended claims, the words "comprises", "comprising", "includes", "including", "having included", and "including" are intended to specify the presence of the stated features, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0215] It should also be understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or alterations thereof will be suggested to those skilled in the art and will be included within the spirit and scope of this application and the scope of the appended claims.

Claims

1. A display device, comprising: An eyepiece stack; And An ambient light management module (ALMN) that includes a dimming component, Wherein, the world-facing film includes one or more wire grid polarizers or cholesteric liquid crystal (CLC) films.

2. The display device according to claim 1, wherein The ALMM includes at least one or more of a plastic substrate, an angular attenuator component, and a world-facing film.

3. The display device according to claim 1, wherein, The ALMM further includes a first polarizer disposed between the at least one plastic substrate and the dimming component.

4. The display device according to claim 3, wherein, The ALMM further includes a second polarizer disposed between the dimming component and the at least one angular attenuator component.

5. The display device according to claim 4, wherein, The ALMM further includes a third polarizer disposed between the at least one angular attenuator component and the at least one world-facing film.

6. The display device according to claim 5, wherein, The first polarizer, the second polarizer, and the third polarizer are linear.

7. The display device according to claim 1, wherein, The ALMM includes a curved, arcuate, or spherical configuration.

8. The display device according to claim 1, wherein, The ALMM is configured to maintain the realism of virtual content under different light conditions.

9. The display device according to claim 8, wherein, The ALMM is further configured to manage diffraction artifacts and minimize the transmission loss of visible artifacts.

10. A display device, comprising: An eyepiece stack; And An optical device having a curved plastic substrate and a plurality of components laminated on the curved plastic substrate, the plurality of components including: An external component stack that includes at least one of a reflective film and a colored film; A birefringent film stack located between the external component stack and the plastic substrate; and A spatial light modulator component located between the birefringent film stack and the plastic substrate.

Images