Systems and methods for manipulating light from ambient light sources

Abstract

An optical device includes a variable optical material that changes at least one of the intensity of ambient light incident through the optical device, the spectral content of the incident ambient light, or the direction of the incident ambient light in response to a stimulus provided by the optical device. The device may detect the intensity and / or spectral features of the ambient light, activate the variable optical material, and provide appropriate stimuli to various parts of the optical device to change at least one of the intensity of the incident ambient light, the spectral content of the incident ambient light, or the direction of the incident ambient light.

Description

Technology Field

[0001] The present application claims priority to U.S. provisional patent application No. 62 / 438,325 filed on December 22, 2016, which is incorporated herein by reference in its entirety.

[0002] This application also relates to U.S. Patent Application No. 15 / 841,043 filed on December 13, 2017, which is incorporated herein by reference in its entirety.

[0003] The present disclosure relates to optical devices including virtual reality and augmented reality imaging and visualization systems. Background Technology

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or parts of images are presented to the user in a manner that appears real or can be perceived as real. Virtual reality, or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs; augmented reality, or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality, or "MR" scenarios, are a type of AR scenario and typically involve virtual objects that are integrated into and respond to the natural world. For example, in an MR scenario, AR image content may be blocked by real-world objects, or otherwise may be perceived as interacting with real-world objects.

[0005] Referring to FIG. 1, an augmented reality scene (10) is illustrated, in which a user of AR technology sees a real-world park-like setting (20) featuring people, trees, buildings, and a concrete platform (30) in the background. In addition to these items, the user of AR technology also perceives that they are "seeing" "virtual content," such as a robot statue (40) standing on the real-world platform (30) and a flying cartoon-like avatar character (50) that appears to be an anthropomorphic bumblebee, but these elements (40, 50) do not exist in the real world. Because the human visual perception system is complex, creating AR technology that facilitates a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world imagery elements is a difficult task.

[0006] The systems and methods disclosed herein solve various challenges related to AR and VR technologies.

[0007] Each of the systems, methods, and devices of the present disclosure has various innovative aspects, and no single aspect among such aspects alone possesses the preferred attributes disclosed herein.

[0008] Details of one or more embodiments of the claimed subject matter described herein are described in the accompanying drawings and description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions in the following drawings may not be drawn to actual scale.

[0009] Various examples of optical devices comprising variable optical materials that undergo physical and / or chemical changes in response to stimulation, such as the examples listed below, are described herein.

[0010] Example 1: A user-wearable display device comprises: a frame configured to be mounted on a user; an augmented reality display attached to the frame and configured to orient images toward the user's eyes; a sensor configured to acquire information regarding the ambient light condition of the environment surrounding the user; a variable optical material undergoing physical and / or chemical changes in response to a stimulus; a source configured to provide a stimulus; and a processing electronic device configured to trigger the source to provide the stimulus to the variable optical material to bring about physical and / or chemical changes in the material based on information acquired by the sensor, such that at least one of the intensity of the ambient light, the spectral content of the ambient light, or the direction of the ambient light is changed.

[0011] Example 2: In the user-wearable display device of the first example, the augmented reality display comprises a waveguide, the waveguide being configured to: allow a view of the environment surrounding the user through the waveguide; and to form images by directing light out of the waveguide and toward the user's eyes.

[0012] Example 3: In a user-wearable display device of the first example or the second example, a waveguide is part of a stack of waveguides, and each waveguide of the stack is configured to output light with different amounts of divergence compared to one or more other waveguides of the stack of waveguides.

[0013] Example 4: In any one of the first to third examples, the user-wearable display device comprises at least one of a light sensor, an image capture device, a global positioning sub-system, or an environment sensor.

[0014] Example 5: A user-wearable display device of any one of the first to fourth examples further includes an image capture device configured to track the movement of the user's eyes.

[0015] Example 6: A user-wearable display device of any one of the first to fifth examples further includes a light source configured to generate a projection beam based on data associated with images directed toward the user's eyes.

[0016] Example 7: In any one of the first to sixth examples of a user-wearable display device, the source includes an optical source configured to direct visible light or invisible light to one or more parts of the display.

[0017] Example 8: In any one of the first to sixth examples of a user-wearable display device, the source includes an electric source configured to provide an electric signal to one or more parts of the display.

[0018] Example 9: In any one of the first to sixth examples, a user-wearable display device comprises a heat source configured to provide thermal radiation to one or more parts of the display.

[0019] Example 10: In any one of the first to sixth examples of a user-wearable display device, the source comprises a sonic / ultrasonic system configured to provide sonic / ultrasonic energy to one or more parts of the display.

[0020] Example 11: In any one of the first to ten examples of a user-wearable display device, a variable optical material is embedded in the surface of the display.

[0021] Example 12: In any one of the first to ten examples of a user-wearable display device, a variable optical material is disposed on the surface of the display.

[0022] Example 13: In any one of the first to twelfth examples of a user-wearable display device, the variable optical material comprises organic or inorganic compounds.

[0023] Example 14: In any one of the first to thirteen examples of a user-wearable display device, the variable optical material comprises electroactive proteins.

[0024] Example 15: In any one of the first to fourteen examples of a user-wearable display device, the variable optical material comprises molecules that exhibit a change in size or shape in response to stimulation.

[0025] Example 16: In any one of the first to fifteen examples of a user-wearable display device, the variable optical material comprises molecules that move, rotate, twist, or shift in response to stimulation.

[0026] Example 17: In any one of the first to sixth examples of a user-wearable display device, the variable optical material comprises molecules that move together and / or attach together in response to stimulation.

[0027] Example 18: In any one of the first to sixth examples of a user-wearable display device, the variable optical material comprises molecules that move apart from each other in response to stimulation.

[0028] Example 19: In any one of the first to eighteen examples of a user-wearable display device, the variable optical material comprises molecules that form nanostructures in response to stimulation.

[0029] Example 20: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first ocular region corresponding to the user's first eye and a second ocular region corresponding to the user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to a part of the display to bring about a physical and / or chemical change of a variable optical material based on information acquired by a sensor, such that at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light is changed through the first ocular region as a result of a stimulus from a source triggered by the processing electronic device.

[0030] Example 21: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first eye region corresponding to a user's first eye and a second eye region corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to a part of the display to bring about a physical and / or chemical change of a material based on information acquired by a sensor, such that at least one of the intensity of ambient light through the first eye region, the spectral content of ambient light, or the direction of ambient light is differently changed compared with the intensity of ambient light through the second eye region, the spectral content of ambient light, or the direction of ambient light.

[0031] Example 22: In any one of the first to 19 examples of a user-wearable display device, the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a material based on information acquired by a sensor, such that the attenuation of the intensity of ambient light transmitted through a first part of the display is greater than the attenuation of the intensity of ambient light transmitted through a second part of the display.

[0032] Example 23: In the user-wearable display device of Example 22, the intensity of ambient light incident on the first part of the display is greater than the intensity of ambient light incident on the second part of the display.

[0033] Example 24: In the user-wearable display device of Example 22 or Example 23, the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of the material based on information acquired by a sensor, so that the intensity of ambient light transmitted through a second part of the display is reduced.

[0034] Example 25: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first eye region corresponding to a user's first eye and a second eye region corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a material based on information acquired by a sensor, such that the intensity of ambient light transmitted through a portion of the first eye region is reduced.

[0035] Example 26: In any one of the first to 19 examples of a user-wearable display device, the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a material based on information acquired by a sensor, such that the spectrum of ambient light transmitted through a first part of the display is different from the spectrum of ambient light transmitted through a second part of the display.

[0036] Example 27: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens, based on information acquired by a sensor, such that the intensity of ambient light transmitted only through the first lens is reduced as a result of a stimulus from a source triggered by the processing electronic device.

[0037] Example 28: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens, based on information acquired by a sensor, such that the intensity of ambient light transmitted through a portion of the first lens is reduced by a greater amount than that of another portion of the first lens.

[0038] Example 29: In the user-wearable display device of Example 28, the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the second lens, based on information acquired by a sensor, so that the intensity of ambient light transmitted through a portion of the second lens is reduced.

[0039] Example 30: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens, based on information acquired by a sensor, such that the intensity of ambient light transmitted through the first lens is attenuated more than through the second lens.

[0040] Example 31: In the user-wearable display device of the 30th example, the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the second lens, based on information acquired by a sensor, so that the intensity of ambient light transmitted through the second lens is reduced.

[0041] Example 32: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens or the second lens based on information acquired by a sensor, such that the spectrum of ambient light transmitted through the first lens and the second lens is different.

[0042] Example 33: In any one of the first to 19 examples of a user-wearable display device, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and a processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens or the second lens based on information obtained by a sensor, such that the spectrum of ambient light transmitted through a portion of the first lens is different from that of another portion of the first lens.

[0043] Example 34: In the user-wearable display device of the 33rd example, the display comprises a first lens corresponding to a user's first eye and a second lens corresponding to a user's second eye, and the processing electronic device is configured to trigger a source to provide a stimulus to the display to bring about a physical and / or chemical change of a variable optical material associated with the first lens or the second lens based on information obtained by a sensor such that the spectrum of ambient light transmitted through a portion of the first lens is different from another portion of the second lens.

[0044] Example 35: In any one of the first to 19 examples of a user-wearable display device, an object seen by the wearer's eyes through the display appears to be aligned with at least one part of the display, and a processing electronic device is configured to cause a source to provide stimulation to at least one part of the display where the object appears to be aligned, so as to bring about a physical and / or chemical change of a variable optical material such that at least one of the intensity of light from the object, the spectral content of the light from the object, or the direction of the light from the object is changed.

[0045] Example 36: In the user-wearable display device of the 35th example, the processing electronic device is configured to determine at least one part of the display where an object appears to be aligned based on the movement of the user's head tracked by the sensor.

[0046] Example 37: In the user-wearable display device of Example 35 or Example 36, the processing electronic device is configured to cause a source to provide a stimulus to at least one part of the display to bring about a physical and / or chemical change of a variable optical material so as to reduce the intensity of ambient light.

[0047] Example 38: A user-wearable display device of any one of Examples 1 to 37 further comprises a head pose sensor.

[0048] Example 39: In any one of the first to third eighth examples of a user-wearable display device, the device is further configured to adjust the position of at least one part of the display in which at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light is changed based on feedback from the user.

[0049] Example 40: In any one of the first to 39 examples of a user-wearable display device, the size of at least one part of the display is further configured to be adjusted based on feedback from the user, such that at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light is changed.

[0050] Example 41: In any one of the first to forty examples, a user-wearable display device is further configured to adjust the amount by which at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light is changed based on feedback from the user.

[0051] Example 42: A method for manipulating light transmitted through a user-wearable display device comprising a display surface including a variable optical material that changes at least one of the intensity of ambient light transmitted through the display surface, the spectral content of ambient light, or the direction of ambient light in response to a stimulus, comprises: acquiring a measurement of the ambient light state of the environment surrounding the user using a sensor; determining the intensity of light incident on a first position associated with a first part of the display surface and a second position associated with a second part of the display surface ― said first position is closer to the first part of the display surface than said second part, and said second position is closer to the second part of the display surface than said first part ―; and controlling a source to provide a first stimulus to the first part to bring about a physical and / or chemical change of the material such that at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light incident on the first part of the display surface is changed by a first amount; and includes the step of controlling a source to provide a second stimulus to a second part to bring about a physical and / or chemical change of the material, such that at least one of the intensity of ambient light incident on the second part of the display surface, the spectral content of the ambient light, or the direction of the ambient light is changed by a second amount.

[0052] Example 43: In the method of example 42, the first amount is different from the second amount. Brief explanation of the drawing

[0053] Figure 1 illustrates a user's view of AR through an augmented reality (AR) device.

[0054] FIGS. 2a and 2b illustrate embodiments of a wearable display system.

[0055] FIG. 3 illustrates a conventional display system for simulating a three-dimensional imagery of a user.

[0056] Figure 4 illustrates aspects of an approach to simulating a 3-dimensional imagery using multiple depth planes.

[0057] Figures 5a-5c illustrate the relationships between the radius of curvature and the radius of focus.

[0058] Figure 6 illustrates an example of a waveguide stack for outputting image information to a user.

[0059] Figure 7 illustrates an example of outgoing beams output by a waveguide.

[0060] FIG. 8 illustrates an example of a stacked waveguide assembly in which each depth plane comprises images formed using a plurality of different component colors.

[0061] FIG. 9a illustrates a side cross-sectional view of an example of a set of stacked waveguides, each containing an in-coupling optical element.

[0062] FIG. 9b illustrates a perspective view of an example of multiple stacked waveguides of FIG. 9a.

[0063] FIG. 9c illustrates a top-down plan view of an example of multiple stacked waveguides of FIG. 9a and 9b.

[0064] Figure 10 illustrates a scene including one or more ambient light sources.

[0065] Figure 11 is a flowchart illustrating a method for changing the transmittance of light through a display lens.

[0066] FIG. 12a is a side view of a display lens including a portion having reduced peripheral light transmittance. FIG. 12b is a front view of the display lens exemplified in FIG. 12a as viewed from the side facing the eye. FIG. 12c is a top view of the display lens exemplified in FIG. 12a.

[0067] Similar reference numbers and designations within various drawings indicate similar elements. Specific details for implementing the invention

[0068] The embodiments considered herein include a wearable display device (e.g., augmented reality and / or virtual reality glasses) comprising at least one variable optical material capable of changing at least one of the intensity of ambient light transmitted through the display device, the spectral content of the ambient light transmitted through the display device, or the optical path of the ambient light transmitted through the display device in response to an external stimulus (e.g., optical stimulus, electrical stimulus, thermal stimulus, ultrasonic / sonic stimulus, radiation pressure, etc.) (e.g., by diffraction of the variable optical element or by changing the refractive index of the variable optical element). In various embodiments, at least one variable optical material may be configured to attenuate the intensity of ambient light in one or more wavelength ranges. In some embodiments, at least one variable optical material may be configured to reflect, refract, scatter, diffract, or absorb incoming light. The wearable display device utilizes physical / chemical changes caused in at least one variable optical material by an external stimulus. As a result of external stimulation, at least one variable optical material may modify at least one of the intensity of ambient light transmitted through a display device, the spectral content of ambient light transmitted through a display device, or the optical path of ambient light transmitted through a display device, depending on the intensity and / or spectral characteristics of the incoming light to enhance the user experience. Various studies may be conducted to characterize the photomodification features of the variable optical material. Different studies may also be conducted to characterize the type of photomodification that will result in a desired user experience for different types of ambient light sources. Feedback from various studies may be considered to determine which regions of the display device require light transmittance modification and the amount of photomodification to provide the desired user experience.

[0069] In some embodiments, at least one variable optical material may be embedded in the display surface of a display device. In some other embodiments, at least one variable optical material may be included in an accessory component that may be placed on the display device. At least one variable optical material may include photosensitive, electro-active, and / or radiosensitive materials. In some embodiments, at least one variable optical material may include organic or inorganic compounds. In some embodiments, at least one variable optical material may include photosensitive materials, such as silver-based compounds (e.g., silver chloride or silver halide). In some other embodiments, at least one variable optical material may include organic compounds such as oxazines and / or naphtopyrans. In some embodiments, at least one variable optical material may include one or more layers of molecules.

[0070] At least one variable optical material may be activated by an optical stimulus provided from a light source, for example, on or integrated into eyeglasses. The light source may be monochromatic or polychromatic. In various embodiments, the light source may include an LED, a scanning fiber projector, a UV light source, or a source configured to provide an electron beam. The light source may be controlled by electrical or mechanical devices. For example, in some embodiments, the light source may be controlled by a movable shutter or a variable filter. As another example, the light source may be electrically controlled by a processor.

[0071] The processor is configured to trigger a device providing optical, electrical, thermal, and / or acoustic / ultrasonic stimulation based on information obtained from one or more sensors (e.g., optical sensors, one or more cameras, eye-tracking cameras, position sensing devices, pose sensing devices, environment sensors configured to detect temperature, global positioning system sub-assemblies, accelerometers, color sensors, etc.). For example, the processor may be configured to turn on or turn off, activate or deactivate, or otherwise control a device providing optical, electrical, thermal, and / or acoustic / ultrasonic stimulation to activate or control at least one variable material in different parts of a display device in order to change at least one of the intensity of ambient light transmitted through a display device, the spectral content of ambient light transmitted through a display device, or the optical path of ambient light transmitted through a display device, based on information obtained from one or more sensors.

[0072] In response to stimulation, at least one variable optical material may undergo physical and / or chemical changes. For example, molecules of at least one variable optical material may undergo a change in size (e.g., contraction or expansion) in response to stimulation. As another example, molecules of at least one variable optical material may undergo a change in shape in response to stimulation. As yet another example, the density of molecules of at least one variable optical material may change in response to stimulation. Consequently, stimulation may alter at least one of the intensity of ambient light transmitted through the display device, the spectral content of ambient light transmitted through the display device, or the optical path of ambient light transmitted through the display device.

[0073] In various embodiments, molecules of at least one variable optical material may be configured to move, shift, rotate, twist, or otherwise change or react when a stimulus is provided. The movement, shift, rotation, or twist of the molecules of at least one variable optical material may be configured to be random in some embodiments. However, in some other embodiments, the movement, shift, rotation, or twist of the molecules of at least one variable optical material may be configured to follow a specific direction. In some embodiments, the speed at which the molecules of at least one variable optical material move, shift, rotate, or twist may be changed by changing the characteristics of the provided stimulus. In various embodiments, the molecules of at least one variable optical material may move closer together in response to the stimulus. In some other embodiments, the molecules of at least one variable optical material may move further apart from each other in response to the stimulus. In some embodiments, the molecules of at least one variable optical material may be configured to form nanostructures in response to the stimulus.

[0074] Physical and / or chemical changes in the molecules of at least one variable optical material may be induced by controlling the characteristics of the stimulus. For example, when the stimulus is optical, physical and / or chemical changes in the molecules of at least one variable optical material may be induced by controlling the wavelength and / or intensity of the optical stimulus. As another example, when the stimulus is electrical, physical and / or chemical changes in the molecules of at least one variable optical material may be induced by controlling the voltage and / or current of the electrical stimulus. In various embodiments, physical and / or chemical changes in the molecules of at least one variable optical material may be controlled by modulating the source providing the stimulus. In some embodiments, physical and / or chemical changes in the molecules of at least one variable optical material may be reversible so that when the stimulus is removed, the molecules of at least one variable optical material return to their original state. In these embodiments, the stimulus is provided constantly to maintain the modified state of the molecules of at least one variable optical material. In some other embodiments, physical and / or chemical changes of molecules of at least one variable optical material may be maintained in the absence of stimulation until non-activating energy is provided to return the molecules of at least one variable optical material to their original state. In these embodiments, stimulation may be provided for a short duration to initiate modification of the molecules of at least one variable optical material.

[0075] Various embodiments of a wearable display device are configured to map real-world objects surrounding a user, including objects visible to the user through the display device, using various sensor assemblies and / or imaging devices. In various embodiments, information obtained from various sensor assemblies and / or imaging devices may be used to create a database containing, for example, the position of various real-world objects relative to the display device and / or the user's head / eyes, and potentially other characteristics of the objects, such as the size, shape, and / or brightness of the objects. The database may update information in real-time or near real-time and / or provide updated information when the user moves their head and / or body, or when surrounding real-world objects appear to move relative to the display device and / or the user's head / eyes. The database may update information regarding the position of new objects from the surrounding real world entering the user's field of vision when the user moves their head, and / or provide updated information in real-time or near real-time. The display device may be configured and / or used to locate and identify different ambient light sources in the real world visible to the user through the display device. Different ambient light sources may appear to be aligned with different parts of the visible surface of a display device. These objects may generate glare. Accordingly, the display device may be configured to change, modify, adjust, or manipulate at least one of the intensity of ambient light transmitted through different parts of the visible surface of the display device where different ambient light sources appear to be aligned to reduce glare, the optical path of the ambient light, or the spectral content of the ambient light.

[0076] Various embodiments of a wearable display device are configured to attenuate incoming ambient light incident on various parts of the display surface. Accordingly, the amount of variation of at least one of the intensity of ambient light transmitted through the display device, the spectral content of ambient light transmitted through the display device, or the optical path of ambient light transmitted through the display device may vary across the surface of the display device and does not need to be uniform. This may be advantageous for maintaining the user experience when a part of the display surface introduces more glare than another part. For example, when a user is viewing a scene with the sun or bright light in the background, the incoming light transmitted through the part of the display device aligned with the sun or bright light may be attenuated by a greater amount than the intensity of the incoming light transmitted through other parts of the display device. Additionally, when a user is viewing a display device near a window or using a desk lamp, incoming light transmitted through the part of the display device near the window or desk lamp may be attenuated by a greater amount than incoming light transmitted through the part of the display device further away from the window or desk lamp, because the part of the display device near the window or desk lamp may have more glare.

[0077] Now, references to drawings in which similar reference numbers refer to similar parts throughout will be made. It will be recognized that the embodiments disclosed herein generally include optical systems, including display systems. In some embodiments, the display systems are wearable, which, advantageously, can provide a more immersive VR or AR experience. For example, displays comprising one or more waveguides (e.g., a stack of waveguides) may be configured to be worn positioned in front of the user or viewer's eyes. In some embodiments, two stacks of waveguides (one for each of the viewer's eyes) may be utilized to provide different images to each eye. Exemplary display systems

[0078] FIG. 2a illustrates an example of a wearable display system (60). The display system (60) includes a display (70) and various mechanical and electronic modules and systems to support the function of the display (70). The display (70) can be worn by a display system user or viewer (90) and may be coupled to a frame (80) configured to position the display (70) in front of the user (90)'s eyes. In some embodiments, the display (70) may be considered as eyewear. In some embodiments, a speaker (100) is coupled to the frame (80) and configured to be positioned adjacent to the user (90)'s external auditory canal (in some embodiments, another speaker not shown is positioned adjacent to the user's other external auditory canal to provide stereo / shapeable sound control). In some embodiments, the display system may also include one or more microphones (110) or other devices for detecting sound. In some embodiments, the microphone is configured to allow the user to provide inputs or commands (e.g., selection of voice menu commands, natural language questions, etc.) to the system (60) and / or to allow audio communication with other people (e.g., other users of similar display systems). The microphone may additionally be configured as an ambient sensor for collecting audio data (e.g., sounds from the user and / or environment). In some embodiments, the display system may also include an ambient sensor (120a) that is separate from the frame (80) and may be attached to the user's (90) body (e.g., the user's (90) head, torso, extremities, etc.). In some embodiments, the ambient sensor (120a) may be configured to acquire data characterizing the user's (90) physiological state. For example, the sensor (120a) may be an electrode.

[0079] Referring again to FIG. 2a, the display (70) is operably coupled by a communication link (130), e.g., wired lead or wireless connectivity, to a local data processing module (140) which can be mounted in various configurations, e.g., fixedly attached to a frame (80), fixedly attached to a helmet or hat worn by a user, embedded in headphones, or otherwise attached removablely to the user (90) (e.g., in a backpack-style configuration, in a belt-coupling style configuration). Similarly, the sensor (120a) can be operably coupled to the local processor and data module (140) by a communication link (120b), e.g., wired lead or wireless connectivity. The local processing and data module (140) may include a hardware processor as well as digital memory, e.g., non-volatile memory (e.g., flash memory or hard disk drives), both of which can be utilized to assist in the processing, caching, and storage of data. The data comprises a) captured from sensors (e.g., operably coupled to the frame (80) or otherwise attached to the user (90)), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, gyros, and / or other sensors disclosed herein; and / or b) data acquired and / or processed using a remote processing module (150) and / or a remote data repository (160) (containing data related to virtual content) (possibly for delivery to a display (70) after such processing or retrieval).The local processing and data module (140) can be operably coupled to the remote processing module (150) and the remote data storage (160) by means of communication links (170, 180), for example, through wired or wireless communication links, so that the remote modules (150, 160) can be operably coupled to each other and available as resources for the local processing and data module (140). In some embodiments, the local processing and data module (140) may include one or more of image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices and / or gyroscopes. In some other embodiments, one or more of these sensors may be attached to the frame (80) or may be freestanding structures that communicate with the local processing and data module (140) by wired or wireless communication channels.

[0080] Referring further to FIG. 2a, in some embodiments, the remote processing module (150) may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data storage (160) may include a digital data storage facility that may be available via the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, the remote data storage (160) may include one or more remote servers that provide information, such as information for creating augmented reality content, to the local processing and data module (140) and / or the remote processing module (150). In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for completely autonomous use from the remote module.

[0081] Various embodiments of the display system (60) may include one or more components configured to detect the environment surrounding the user (90) (e.g., cameras, light sensors, color sensors, temperature sensors, motion detectors, accelerometers, gyroscopes, global positioning sub-systems, etc.). One or more components included in the display system (60) may also be configured to monitor the position of the head and / or track the eye movements of the user (90). For example, one or more components included in the display system (60) may be configured to determine pupil constriction in response to bright light, pupil dilation in response to low light, blinking response, etc. As another example, one or more components included in the display system (60) may be configured to monitor and / or track the movement of the user's head. In some embodiments, one or more components included in the display system (60) may be configured to monitor and / or track the position of real-world objects (e.g., trees, the sun, ambient light sources, etc.) in relation to the user's eyes when the user's head moves.

[0082] FIG. 2b illustrates some of the components included in embodiments of the display system (60). Other embodiments may have additional or fewer components depending on the application in which the system is used. Nevertheless, FIG. 2b provides a basic idea of ​​some of the various components that may be included in the display system (60) configured to detect the environment. In the embodiment illustrated in FIG. 2b, the display device (70) includes a display lens (106) that can be mounted on the user's head or eyes by a frame (80). The display lens (106) may be configured to propagate light (124) projected from one or more light projection systems (118) to the eyes (122). The display lens (106) may also be configured to allow the transmission of at least some light from the local environment surrounding the user (90). In various embodiments of a display system (60) configured as an augmented reality device, the projected light (124) may include virtual content that can be superimposed on real-world content seen by the user's eyes (122).

[0083] The display system may include one or more outward-facing cameras (112) configured to image the environment surrounding the user (90). In some embodiments, the cameras (112) may include wide-field-of-view machine vision cameras. In some embodiments, the cameras (112) may be dual-capture visible / invisible (e.g., infrared) light cameras. The cameras (112) may be integrated into a frame (80) as shown in FIG. 2b. However, in some embodiments, the cameras (112) may be positioned elsewhere. For example, the cameras (112) may be configured to be attached to the user's (90) head, arms, neck, or other parts of the body. In various embodiments, the cameras (112) do not need to be attached to the user (90) but may instead be positioned next to the user.

[0084] Referring further to FIG. 2b, the display system (60) may include one or more inward-facing cameras (114) that can be configured to monitor the user's eyes (122). In various embodiments, the inward-facing cameras (114) may be paired with infrared light sources (e.g., light-emitting diodes) configured to track the user's (90) eyes (122). The system (60) may further include one or more light sensors (128) configured to detect ambient light. For example, one or more light sensors (128) may be configured to detect at least one of the intensity, wavelength, or color temperature or range of ambient light. In various embodiments, the light sensors (128) may include silicon photodetectors, phototransistors, photodiodes, LCD sensors, sensors using resistive properties to detect changes in light intensity / spectral features, infrared (IR) light sensors, etc. The system (60) may further include sensor assemblies (126) that may include one or more X, Y, and Z axis accelerometers, as well as a magnetic compass and one or more X, Y, and Z axis gyroscopes, preferably providing data at a relatively high frequency, e.g., 200 Hz. In some embodiments, the sensor assembly (126) may include a GPS (global positioning satellite) subsystem for providing information about the user's environment.

[0085] The local processing and data module (140) and / or remote processing module (150) may include a processor, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or an advanced reduced-instruction-set machine (ARM) processor, which may be configured to calculate a user head pose in real-time or near real-time from information obtained by the inward cameras (114), outward cameras (112), a light sensor (128), and / or a sensor assembly (126). The processor may be configured to provide information regarding the user's environment from information obtained by the inward cameras (114), outward cameras (112), the light sensor (128), and / or the sensor assembly (126). In various embodiments, the display system (60) may be configured to determine ambient light conditions using information obtained from the outward cameras (112), the light sensor (128), and / or the sensor assembly (126). For example, information obtained from outward-facing cameras (112), a light sensor (128), and / or a sensor assembly (126) may be processed using one or more electronic processors of a local processing and data module (140) and / or a remote processing module (150) to determine whether ambient light is diffused. If ambient light is not diffused, the system (60) may use the information obtained from outward-facing cameras (112), a light sensor (128), and / or a sensor assembly (126) to determine the direction in which ambient light is incident on the display (70). The system (60) may be configured to determine the type of light source providing ambient light. For example, the system (60) may be configured to determine whether the light source is sunlight or light from an artificial light source.As another example, the system (60) may be configured to determine the spectral composition and / or intensity of ambient light from information obtained from outward-facing cameras (112), a light sensor (128) and / or a sensor assembly (126).

[0086] As discussed above, the inward-facing cameras (114) can be utilized to track eyes. Thus, the information provided by the inward-facing cameras (114) can be used to determine the object the user is looking at or the direction the user is looking, as well as the depth to which the user's eyes are focusing. The information provided by the inward-facing cameras (114) can also be used to determine ambient light conditions. For example, information obtained by a light sensor (128), sensor assemblies (126), outward-facing cameras (112), and possibly one or more head pose sensors can be combined with the information provided by the inward-facing cameras (114) regarding the size of the pupils of the user's eyes (122) to determine the pose of the user's head (and / or eyes) and to locate and identify different ambient light sources in the real world that the user sees through the display device. The system (60) can be configured to determine the direction in which ambient light is incident, the intensity of the ambient light, and / or the spectral characteristics of the ambient light incident on the display (70). Information obtained by a light sensor (128), a sensor assembly (126), outward-facing cameras (112), and possibly one or more head posture sensors, regarding the position of an object as well as possibly the pose of the user's head, can be combined with information provided by inward-facing cameras (114), regarding the size of the pupils of the user's eyes (122) and possibly the direction in which the user's eyes are pointing, in order to identify parts of the display (70) that match, align, and / or overlap with ambient light sources in a real-world view seen by the user. Information from the light sensor (128), the sensor assembly (126), the outward-facing cameras (112) and / or the inward-facing cameras (114) can be utilized, possibly in conjunction with data from an associated cloud computing resource, to map the positions of objects and the characteristics of the local world in relation to the local world and objects, their characteristics or features, and the user's eyes.

[0087] In the various embodiments discussed above, the display lens (106) may include a variable optical component having at least one material that can be configured to change at least one of the intensity of ambient light transmitted through at least a portion of the display lens (106), the spectral content of ambient light transmitted through at least a portion of the display lens (106), or the optical path of ambient light transmitted through at least a portion of the display lens (106) in response to a stimulus provided by one or more components of the display system (60) to enhance the user experience. For example, if the display system (60) determines, based on information obtained from a light sensor (128), a sensor assembly (126), outward-facing cameras (112) and / or inward-facing cameras (114), that the ambient light conditions for a part of the display lens (106) are bright or that a bright object is in the user's field of vision and is aligned with the part of the display, the display system (60) may be configured to provide a stimulus (e.g., heat, sound / ultrasonic, optical or electrical stimulus) capable of changing at least one of the intensity of the ambient light transmitted through that part of the display lens (106), the spectral content of the ambient light transmitted through that part of the display lens (106), or the optical path of the ambient light transmitted through that part of the display lens (106), in order to reduce the intensity of the ambient light transmitted through that part of the display lens (106) and / or from the bright object and to enhance the visual experience.

[0088] Accordingly, various embodiments of the display system (60) may include a light-emitting module (134) configured to emit ultraviolet, infrared, and / or visible light to provide optical stimulation to at least a portion of the display lens (106); an electric system (132) capable of providing electrical stimulation to at least a portion of the display lens (106); a heat source (136) capable of providing thermal stimulation to at least a portion of the display lens (106); and / or an ultrasonic / ultrasonic transducer (138) for providing sound waves and / or ultrasonic stimulation to at least a portion of the display lens (106). The optical stimulation provided by the light-emitting module (134) may include a directed narrow beam of non-visible light and / or visible light incident on a portion of the display lens (106) configured to have reduced light transmittance. In various embodiments, the display lens (106) may include an arrangement of electrodes (e.g., an electrode array, a two-dimensional grid of electrodes) electrically connected to the electric system (132). The electric system (132) may provide an electric signal (e.g., a voltage signal or a current signal) to the electrodes of a portion of the display lens (106) configured to change the intensity of ambient light incident on the display lens (106), change the spectral content of the ambient light, and / or change the direction of the ambient light. The light-emitting module (134), heat source (136), acoustic / ultrasonic transducer (138), and / or the electric system (132) may be integrated into the frame (80) as shown in FIG. 2B. Alternatively, in some embodiments, one or all of the light-emitting module (134), heat source (136), acoustic / ultrasonic transducer (138), and electric system (132) may be positioned remotely from the display (70).

[0089] Perception of an image as "three-dimensional" or "3-D" can be achieved by providing slightly different presentations of the image to each eye of the viewer. FIG. 3 illustrates a conventional display system for simulating a three-dimensional imagery for a user. Two distinct images (190 and 200) (one for each eye (210 and 220)) are displayed to the user. The images (190, 200) are spaced apart from the eyes (210, 220) by a distance (230) along the optical or z-axis parallel to the viewer's line of sight. The images (190, 200) can be flattened and the eyes (210, 220) can be focused on the images by assuming a single perspective-adjusted state. Such 3-D display systems rely on the human visual system to combine the images (190, 200) to provide a perception of scale and / or depth for the combined image.

[0090] However, it will be recognized that the human visual system finds it more difficult to provide a more complex and realistic perception of depth. For example, many viewers of conventional "3-D" display systems may find such systems uncomfortable or may not perceive any sense of depth at all. Without being limited by theory, it is believed that viewers of an object can perceive the object as "three-dimensional" due to the combination of vergence and accommodation. The vergence movements of two eyes relative to each other (i.e., the rotation of the eyes that causes the pupils to move toward or away from each other to converge the gazes of the eyes in order to gaze at an object) are closely related to the focusing (or "accommodation") of the lenses and pupils of the eyes. Under normal conditions, changing the focus of the lenses of the eyes or accommodating the eyes to change focus from one object to another at a different distance will automatically cause pupil dilation or constriction as well as a matching change in azimuthal movement for the same distance, under a relationship known as the "accommodation-vergence reflex." Likewise, a change in azimuthal movement will, under normal conditions, trigger a matching change in accommodation, in lens shape and pupil size. As mentioned herein, many stereoscopic or "3-D" display systems display a scene using slightly different presentations (and thus slightly different images) to each eye so that three-dimensional perspective is perceived by the human visual system. However, such systems are inconvenient for many viewers, for among other reasons, that while such systems simply provide different presentations of the scene, the eyes view all image information in a single accommodated state and operate against the "accommodation-vergence reflex."Display systems that provide better matching between accommodation and dissociative movement can form more realistic and comfortable simulations of 3-dimensional imagery, which can increase the duration of wear and ultimately contribute to compliance with diagnostic and treatment protocols.

[0091] FIG. 4 illustrates aspects of an approach for simulating a 3-dimensional imagery using multiple depth planes. Referring to FIG. 4, objects at various distances from the eyes (210, 220) along the z-axis are perspective-controlled by the eyes (210, 220) so that these objects are in focus. The eyes (210, 220) assume specific perspective-controlled states that cause objects at different distances along the z-axis to be in focus. Consequently, a specific perspective-controlled state can be said to be associated with a specific depth plane among the depth planes (240) having an associated focal length, so that objects or parts of objects of a specific depth plane are in focus when the eyes are in a perspective-controlled state with respect to that depth plane. In some embodiments, the 3-dimensional imagery may be simulated by providing different presentations of the image for each of the eyes (210, 220) and also by providing different presentations of the image corresponding to each of the depth planes. Although depicted as separate for the sake of clarity of illustration, it will be recognized that the fields of view of the eyes (210, 220) may overlap, for example, as the distance along the z-axis increases. Furthermore, although depicted as flat for the sake of ease of illustration, it will be recognized that the contours of the depth planes may be curved in physical space so that all features of the depth planes are in focus with the eyes in a specific perspective-adjusted state.

[0092] The distance between the object and the eye (210 or 220) can also change the amount of light emitted from the object when viewed by the eye. FIGS. 5a-5c illustrate the relationships between the distance and the emission of light rays. The distance between the object and the eye (210) is represented by R1, R2, and R3 in order of decreasing distance. As illustrated in FIGS. 5a-5c, the light rays emitted more as the distance to the object decreased. As the distance increased, the light rays were collimated more. In other words, it can be said that the light field generated by the point (object or part of the object) has a spherical wavefront curvature, and the spherical wavefront curvature is a function of how far the point is from the user's eye. The curvature increases as the distance between the object and the eye (210) decreases. Consequently, at different depth planes, the degree of emission of the light rays is also different, and the degree of emission increases as the distance between the depth planes and the viewer's eye (210) decreases. Although only one eye (210) is illustrated in FIG. 5a-5c and other drawings of the present invention for clarity of example, it will be recognized that the discussion regarding the eye (210) may apply to both eyes (210 and 220) of the viewer.

[0093] Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception. Consequently, a highly plausible simulation of perceived depth can be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. These different presentations can be focused on separately by the viewer's eyes and thus help provide depth cues to the user based on the eye's accommodation required to focus on different image features of a scene located on different depth planes, and / or based on observing different image features on different depth planes going out of focus.

[0094] FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. A display system (250) includes a stack of waveguides, or a stacked waveguide assembly (260), which can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides (270, 280, 290, 300, 310). In some embodiments, the display system (250) is the system (60) of FIG. 2a and / or FIG. 2b, and FIG. 6 schematically illustrates parts of the system (60) in more detail. For example, the waveguide assembly (260) may be part of the display (70) of FIG. 2a. As another example, the waveguide assembly (260) may be part of the display lens (106) of FIG. 2b. It will be recognized that in some embodiments, the display system (250) may be considered as a light field display.

[0095] Referring again to FIG. 6, the waveguide assembly (260) may also include a plurality of features (320, 330, 340, 350) between the waveguides. In some embodiments, the features (320, 330, 340, 350) may be one or more lenses. The waveguides (270, 280, 290, 300, 310) and / or the plurality of lenses (320, 330, 340, 350) may be configured to transmit image information to the eye with various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image injection devices (360, 370, 380, 390, 400) can function as light sources for waveguides and can be utilized to inject image information into waveguides (270, 280, 290, 300, 310), and each of the waveguides can be configured to disperse incoming light across each individual waveguide to output toward the eye (210) as described herein. Light exits the output surfaces (410, 420, 430, 440, 450) of the image injection devices (360, 370, 380, 390, 400) and is injected into the corresponding input surfaces (460, 470, 480, 490, 500) of the waveguides (270, 280, 290, 300, 310). In some embodiments, each of the input surfaces (460, 470, 480, 490, 500) may be an edge of the corresponding waveguide or part of the main surface of the corresponding waveguide (i.e., one of the waveguide surfaces faces directly toward the world (510) or the viewer's eye (210)). In some embodiments, a single light beam (e.g., a collimated beam) may be injected into each waveguide to output a full field of collimated beams directed toward the eye (210) at specific angles (and divergence amounts) corresponding to the depth plane associated with the specific waveguide.In some embodiments, one of the image injection devices (360, 370, 380, 390, 400) may be associated with a plurality (e.g., 3) of waveguides (270, 280, 290, 300, 310) and may inject light therein.

[0096] In some embodiments, the image injection devices (360, 370, 380, 390, 400) are discrete displays that each generate image information for injection into a corresponding waveguide (270, 280, 290, 300, 310). In some other embodiments, the image injection devices (360, 370, 380, 390, 400) are output ends of a single multiplexed display that can pipe image information to each of the image injection devices (360, 370, 380, 390, 400) through one or more optical conduits (e.g., optical fiber cables). It will be recognized that the image information provided by the image injection devices (360, 370, 380, 390, 400) may include light of different wavelengths or colors (e.g., different component colors as discussed herein). In some embodiments, the image injection devices (360, 370, 380, 390, 400) may be part of the light projection systems (118) of FIG. 2B.

[0097] In some embodiments, light injected into waveguides (270, 280, 290, 300, 310) is provided by a light projector system (520) comprising a light module (530) that may include a light emitter such as an LED (light emitting diode). Light from the light module (530) may be directed through a beam splitter (550) to a light modulator (540), e.g., a spatial light modulator, and modified by it. The light modulator (540) may be configured to change the perceived intensity of the light injected into the waveguides (270, 280, 290, 300, 310). Examples of spatial light modulators include liquid crystal displays (LCS), including liquid crystal on silicon (LCOS) displays. In some embodiments, the light projector system (520) may be part of the light projection systems (118) of FIG. 2B.

[0098] In some embodiments, the display system (250) may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, helical scan, Lissajous pattern, etc.) into one or more waveguides (270, 280, 290, 300, 310) and ultimately to the viewer's eye (210). In some embodiments, the illustrated image injection devices (360, 370, 380, 390, 400) may schematically represent a single scanning optical fiber or a bundle of scanning optical fibers configured to inject light into one or more waveguides (270, 280, 290, 300, 310). In some other embodiments, the illustrated image injection devices (360, 370, 380, 390, 400) may schematically represent a plurality of scanning optical fibers or a plurality of bundles of scanning optical fibers, each of which is configured to inject light into an associated waveguide among the waveguides (270, 280, 290, 300, 310). It will be recognized that one or more optical fibers may be configured to transmit light from the optical module (530) to one or more waveguides (270, 280, 290, 300, 310). For example, it will be recognized that one or more optical structures interposed between the scanning fiber or fibers and one or more waveguides (270, 280, 290, 300, 310) may be provided to redirect light coming from the scanning fiber to one or more waveguides (270, 280, 290, 300, 310).

[0099] The controller (560) controls the operation of one or more waveguides of the stacked waveguide assembly (260), including the operation of image injection devices (360, 370, 380, 390, 400), a light source (530), and an optical modulator (540). In some embodiments, the controller (560) is part of the local data processing module (140). The controller (560) includes programming (e.g., commands of a non-transient medium) that controls the timing and delivery of image information to the waveguides (270, 280, 290, 300, 310) according to any of the various methods disclosed herein, for example. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller (560) may be part of the processing modules (140 or 150) (Fig. 2a).

[0100] Referring again to FIG. 6, waveguides (270, 280, 290, 300, 310) can be configured to propagate light within each individual waveguide by total internal reflection (TIR). Each of the waveguides (270, 280, 290, 300, 310) may be planar or have other shapes (e.g., curved) and has main upper and lower surfaces and edges extending between these main upper and lower surfaces. In the illustrated configuration, each of the waveguides (270, 280, 290, 300, 310) may include out-coupling optical elements (570, 580, 590, 600, 610) configured to extract light out of the waveguide by diverting light propagating within each individual waveguide out of the waveguide to output image information to the eye (210). The extracted light may also be referred to as out-coupled light, and the out-coupling optical elements may also be referred to as light extraction optical elements. The extracted light beam may be output by the waveguide at locations where light propagating within the waveguide strikes the light extraction optical elements. The out-coupling optical elements (570, 580, 590, 600, 610) may be gratings containing diffractive optical features, for example, as further discussed herein. For ease of explanation and drawing clarity, they are illustrated as being placed on the lower main surfaces of the waveguides (270, 280, 290, 300, 310), but in some embodiments, the out-coupling optical elements (570, 580, 590, 600, 610) may be placed on the upper and / or lower main surfaces, and / or directly in the volume of the waveguides (270, 280, 290, 300, 310), as further discussed herein.In some embodiments, out-coupling optical elements (570, 580, 590, 600, 610) may be formed in a layer of material attached to a transparent substrate to form waveguides (270, 280, 290, 300, 310). In some other embodiments, the waveguides (270, 280, 290, 300, 310) may be a monolithic piece of material and the out-coupling optical elements (570, 580, 590, 600, 610) may be formed on the surface of and / or inside the corresponding piece of material.

[0101] Referring again to FIG. 6, as discussed herein, each waveguide (270, 280, 290, 300, 310) is configured to output light to form an image corresponding to a specific depth plane. For example, the waveguide (270) closest to the eye may be configured to transmit collimated light (injected into such waveguide (270)) to the eye (210). The collimated light may represent an optical infinity focus plane. The next waveguide (280) above may be configured to transmit collimated light passing through a first lens (350) (e.g., a negative lens) before the collimated light can reach the eye (210); Such a first lens (350) can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light coming from the next waveguide (280) above as coming from a first focal plane that is closer inward toward the eye (210) from optical infinity. Similarly, the third waveguide (290) above passes its output light through both the first (350) and second (340) lenses before reaching the eye (210); and the combined optical power of the first (350) and second (340) lenses can be configured to produce a different incremental amount of wavefront curvature so that the eye / brain interprets the light coming from the third waveguide (290) as coming from a second focal plane that is much closer inward toward the person from optical infinity than the light from the next waveguide (280) above.

[0102] Other waveguide layers (300, 310) and lenses (330, 320) are configured similarly, such that the highest waveguide (310) in the stack transmits its output through both the lenses between itself and the eye for aggregated focal power representing the focal plane closest to the person. To compensate for the stack of lenses (320, 330, 340, 350) when viewing or interpreting light coming from the world (510) on the other side of the stacked waveguide assembly (260), a compensation lens layer (620) may be placed at the top of the stack to compensate for the aggregated power of the lens stack (320, 330, 340, 350) below. This configuration provides as many perceived focal planes as there are available waveguide / lens pairings. Both the out-coupling optical elements of the waveguides and the focusing aspects of the lenses can be static (i.e., not dynamic or electronically active). In some alternative embodiments, either one or both can be dynamic using electronically active features.

[0103] In some embodiments, two or more of the waveguides (270, 280, 290, 300, 310) may have the same associated depth plane. For example, a plurality of waveguides (270, 280, 290, 300, 310) may be configured to output images set to the same depth plane, or a plurality of subsets of waveguides (270, 280, 290, 300, 310) may be configured to output images set to the same plurality of depth planes (one image is set for each depth plane). This may provide the advantages of forming a tiled image to provide an extended field of view in such depth planes.

[0104] Referring further to FIG. 6, out-coupling optical elements (570, 580, 590, 600, 610) can be configured to divert light out of their respective waveguides and output this light with an appropriate amount of divergence or collimation with respect to a specific depth plane associated with the waveguide. Consequently, waveguides having different associated depth planes may have out-coupling optical elements (570, 580, 590, 600, 610) of different configurations, and these out-coupling optical elements (570, 580, 590, 600, 610) output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements (570, 580, 590, 600, 610) may be volumetric or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements (570, 580, 590, 600, 610) may be volumetric holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features (320, 330, 340, 350) may not be lenses; rather, they may simply be spacers (e.g., cladding layers and / or structures for forming air gaps).

[0105] In some embodiments, out-coupling optical elements (570, 580, 590, 600, 610) are diffraction features or "diffraction optical elements" (also referred to herein as "DOE") that form a diffraction pattern. Preferably, the DOEs have sufficiently low diffraction efficiency so that only a portion of the light of the beam is deflected toward the eye (210) due to each intersection of the DOEs, while the remainder continues to travel through the waveguide via the TIR. Thus, the light carrying image information is split into a number of associated exit beams leaving the waveguide at multiple locations, and the result is a fairly uniform exit emission pattern toward the eye (210) because this particular collimated beam bounces back and forth within the waveguide.

[0106] In some embodiments, one or more DOEs may be switchable between "on" states in which they actively diffract and "off" states in which they do not significantly diffract. For example, the switchable DOE may comprise a polymer-dispersed liquid crystal layer having a diffraction pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host medium (in which case the pattern does not significantly diffract the incident light) or the microdroplets may be switched to an index that does not match the index of the host medium (in which case the pattern actively diffracts the incident light).

[0107] In some embodiments, a camera assembly (630) (e.g., a digital camera including visible light and infrared cameras) may be provided to capture images of the eye (210) and / or tissues surrounding the eye (210) to detect user inputs and / or monitor the user's physiological state. In various embodiments, the camera assembly (630) may be part of the inward-facing cameras (114) of FIG. 2B. As used herein, the camera may be any image capture device. In some embodiments, the camera assembly (630) may include an image capture device and a light source for projecting light (e.g., infrared) onto the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, as discussed herein, the camera assembly (630) may be attached to the frame (80) (Fig. 2a) and may communicate electrically with processing modules (140 and / or 150) capable of processing image information from the camera assembly (630) to make various decisions regarding the user's physiological state, for example. It will be recognized that information regarding the user's physiological state may be used to determine the user's behavior or emotional state. Examples of such information include the user's movements and / or the user's facial expressions. Subsequently, the user's behavior or emotional state may be triangulated with collected environment and / or virtual content data to determine the relationships between the behavior or emotional state, the physiological state, and the environment or virtual content data. In some embodiments, one camera assembly (630) may be utilized for each eye to monitor each eye separately.

[0108] Now, referring to FIG. 7, an example of outgoing beams output by a waveguide is illustrated. Although one waveguide is illustrated, it will be recognized that other waveguides within the waveguide assembly (260) (Fig. 6) may function similarly, where the waveguide assembly (260) includes multiple waveguides. Light (640) is injected into the waveguide (270) from the input surface (460) of the waveguide (270) and propagates within the waveguide (270) by TIR. At points where the light (640) collides with the DOE (570), a portion of the light exits the waveguide as exiting beams (650). Although the emitted beams (650) are exemplified as substantially parallel, as discussed herein, these emitted beams (650) may also be redirected to propagate toward the eye (210) at any angle according to the depth plane associated with the waveguide (270) (e.g., forming diverging emitted beams). It will be recognized that substantially parallel emitted beams may represent a waveguide containing out-coupling optical elements that out-couple light to form images that appear to be set on a depth plane at a distance (e.g., optical infinity) from the eye (210). Other waveguides or other sets of out-coupling optical elements may output a more diverging emitted beam pattern, which will require the eye (210) to focus on a closer distance to the retina and will be interpreted by the brain as light from a distance closer to the eye (210) than optical infinity. In various embodiments, the ejection beams (650) may correspond to the projection beam (124) of FIG. 2b.

[0109] In some embodiments, a full color image may be formed on each depth plane by overlaying images on each of the component colors, e.g., three or more component colors. FIG. 8 illustrates an example of a stacked waveguide assembly in which each depth plane comprises images formed using a plurality of different component colors. The illustrated embodiment shows depth planes (240a-240f), but more or fewer depths may also be considered. Each depth plane may have three or more component color images associated with it, including a first image of a first color (G); a second image of a second color (R); and a third image of a third color (B). Different depth planes are indicated in the drawings by different numbers for diopters (dpt) following the letters G, R, and B. As merely examples, the numbers following each of these letters indicate diopters (1 / m) or the inverse distance of the depth plane from the viewer, and each box in the drawings represents an individual component color image. In some embodiments, the precise arrangement of depth planes for different component colors may be varied to account for differences in the eye's focusing of light of different wavelengths. For example, images of different component colors for a given depth plane may be placed on depth planes corresponding to different distances from the user. This arrangement may increase visual acuity and user comfort and / or reduce chromatic aberrations.

[0110] In some embodiments, light of each component color may be output by a single dedicated waveguide, and consequently, each depth plane may have multiple waveguides associated with it. In these embodiments, each box in the drawings containing the letters G, R, or B may be understood to represent an individual waveguide, and three waveguides may be provided per depth plane, wherein three component color images are provided per depth plane. Although the waveguides associated with each depth plane are depicted as adjacent to each other in this drawing for ease of explanation, it will be recognized that in a physical device, the waveguides may all be arranged in a stack having one waveguide per level. In some other embodiments, multiple component colors may be output by the same waveguide, so, for example, only a single waveguide may be provided per depth plane.

[0111] Referring again to FIG. 8, in some embodiments, G is a green color, R is a red color, and B is a blue color. In some other embodiments, other colors associated with different wavelengths of light, including magenta and cyan, may substitute for or additionally use one or more of red, green, or blue. In some embodiments, the features (320, 330, 340, and 350) may be active or passive optical filters configured to selectively block light from the surrounding environment to the viewer's eyes.

[0112] It will be recognized that throughout this disclosure, references to light of a given color are understood to include light of one or more wavelengths within the range of wavelengths of light perceived by a viewer as the given color. For example, red light may include light of one or more wavelengths in the range of about 620-780 nm, green light may include light of one or more wavelengths in the range of about 492-577 nm, and blue light may include light of one or more wavelengths in the range of about 435-493 nm.

[0113] In some embodiments, the light source (530) (Fig. 6) may be configured to emit light of one or more wavelengths, such as infrared and / or ultraviolet wavelengths, outside the visual perception range of the viewer. Additionally, the in-coupling, out-coupling, and other light diversion structures of the waveguides of the display (250) may be configured to direct and emit this light out of the display toward the user's eye (210) for, for example, imaging and / or user stimulation applications.

[0114] Now, referring to FIG. 9a, in some embodiments, light striking a waveguide may need to be diverted to in-couple the light into the waveguide. An in-coupling optical element may be used to divert and in-couple the light to its corresponding waveguide. FIG. 9a illustrates a side cross-sectional view of an example of a plurality of stacked waveguides or a set of stacked waveguides (660), each comprising an in-coupling optical element. Each waveguide may be configured to output light of one or more different wavelengths, or a range of one or more different wavelengths. It will be recognized that the stack (660) may correspond to the stack (260) (Fig. 6), and the illustrated waveguides of the stack (660) may correspond to parts of the plurality of waveguides (270, 280, 290, 300, 310), except that light from one or more of the image injection devices (360, 370, 380, 390, 400) is injected into the waveguides from a position requiring the light to be diverted for in-coupling.

[0115] An exemplary set (660) of stacked waveguides includes waveguides (670, 680, and 690). Each waveguide includes an associated in-coupling optical element (which may also be referred to as an optical input region on the waveguide), for example, an in-coupling optical element (700) is placed on the main surface (e.g., upper main surface) of waveguide (670), an in-coupling optical element (710) is placed on the main surface (e.g., upper main surface) of waveguide (680), and an in-coupling optical element (720) is placed on the main surface (e.g., upper main surface) of waveguide (690). In some embodiments, one or more of the in-coupling optical elements (700, 710, 720) may be placed on the lower main surface of each waveguide (670, 680, 690) (particularly, where one or more of the in-coupling optical elements are reflective deflection optical elements). As illustrated, the in-coupling optical elements (700, 710, 720) may be placed on the upper main surface of each waveguide (670, 680, 690) (or the uppermost part of the next lower waveguide), particularly, where these in-coupling optical elements are transmissive deflection optical elements. In some embodiments, the in-coupling optical elements (700, 710, 720) may be placed in the body of each waveguide (670, 680, 690). In some embodiments, as discussed herein, the in-coupling optical elements (700, 710, 720) are wavelength-selective, so they selectively redirect one or more optical wavelengths while transmitting other optical wavelengths. Although exemplified on one side or corner of each of its waveguides (670, 680, 690), it will be recognized that in some embodiments, the in-coupling optical elements (700, 710, 720) may be placed in other regions of each of its waveguides (670, 680, 690).

[0116] As illustrated, the in-coupling optical elements (700, 710, 720) may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset so that it receives the light without the light passing through other in-coupling optical elements. For example, each in-coupling optical element (700, 710, 720) may be configured to receive light from different image injection devices (360, 370, 380, 390, 400) as illustrated in FIG. 6, and may be separated (e.g., laterally spaced) from other in-coupling optical elements (700, 710, 720) so that it does not substantially receive light from the other in-coupling optical elements (700, 710, 720).

[0117] Each waveguide also includes associated light distribution elements, for example, light distribution elements (730) are placed on the main surface (e.g., top main surface) of waveguide (670), light distribution elements (740) are placed on the main surface (e.g., top main surface) of waveguide (680), and light distribution elements (750) are placed on the main surface (e.g., top main surface) of waveguide (690). In some other embodiments, the light distribution elements (730, 740, 750) may each be placed on the bottom main surface of the associated waveguides (670, 680, 690). In some other embodiments, the light distribution elements (730, 740, 750) may each be placed on both the uppermost and lowermost main surfaces of the associated waveguides (670, 680, 690); or the light distribution elements (730, 740, 750) may each be placed on different uppermost and lowermost main surfaces of different associated waveguides (670, 680, 690).

[0118] Waveguides (670, 680, 690) may be spaced apart and separated by layers of, for example, gaseous, liquid, and / or solid materials. For example, as illustrated, layer (760a) may separate the waveguides (670, 680); and layer (760b) may separate the waveguides (680 and 690). In some embodiments, the layers (760a and 760b) are formed of low refractive index materials (i.e., materials having a lower refractive index than the material forming the immediately adjacent one of the waveguides (670, 680, 690)). Preferably, the refractive index of the material forming the layers (760a, 760b) is 0.05 or more or 0.10 or less than the refractive index of the material forming the waveguides (670, 680, 690). Advantageously, the lower refractive index layers (760a, 760b) can function as cladding layers that facilitate the TIR of light through the waveguides (670, 680, 690) (e.g., the TIR between the top and bottom main surfaces of each waveguide). In some embodiments, the layers (760a, 760b) are formed of air. Although not illustrated, it will be recognized that the top and bottom of the illustrated set of waveguides (660) may include immediately adjacent cladding layers.

[0119] Preferably, for ease of manufacturing and other considerations, the material forming the waveguides (670, 680, 690) is similar or identical, and the material forming the layers (760a, 760b) is similar or identical. In some embodiments, the material forming the waveguides (670, 680, 690) may differ between one or more waveguides, and / or the material forming the layers (760a, 760b) may differ while still maintaining the various refractive index relationships mentioned above.

[0120] Referring again to FIG. 9a, rays (770, 780, 790) are incident on a set of waveguides (660). It will be recognized that the rays (770, 780, 790) can be injected into the waveguides (670, 680, 690) by one or more image injection devices (360, 370, 380, 390, 400) (Fig. 6).

[0121] In some embodiments, the light rays (770, 780, 790) have different characteristics, e.g., different wavelengths or different ranges of wavelengths, which may correspond to different colors. The in-coupling optical elements (700, 710, 720) each deflect light so that the incident light propagates through each of the waveguides (670, 680, 690) by TIR.

[0122] For example, an in-coupling optical element (700) may be configured to deflect a light beam (770) having a first wavelength or various wavelengths. Similarly, a transmitted light beam (780) strikes an in-coupling optical element (710) configured to deflect light of a second wavelength or a range of wavelengths and is deflected by it. Likewise, a light beam (790) is deflected by an in-coupling optical element (720) configured to selectively deflect light of a third wavelength or a range of wavelengths.

[0123] Referring again to FIG. 9a, the deflected rays (770, 780, 790) are deflected so that they propagate through the corresponding waveguides (670, 680, 690); that is, the in-coupling optical elements (700, 710, 720) of each waveguide deflect the light into the corresponding waveguide to in-couple the light into the corresponding waveguide (670, 680, 690). The rays (770, 780, 790) are deflected at angles that cause the light to propagate through the respective waveguides (670, 680, 690) by the TIR. The light rays (770, 780, 790) are propagated by TIR through each waveguide (670, 680, 690) until they collide with the corresponding light distribution elements (730, 740, 750) of the waveguide.

[0124] Now, referring to FIG. 9b, a perspective view of an example of a plurality of stacked waveguides of FIG. 9a is illustrated. As mentioned above, in-coupled rays (770, 780, 790) are each deflected by in-coupled optical elements (700, 710, 720) and then propagated by TIR within waveguides (670, 680, 690). Then, the rays (770, 780, 790) strike the optical distribution elements (730, 740, 750) respectively. The optical distribution elements (730, 740, 750) deflect the rays (770, 780, 790) so that they propagate toward the out-coupled optical elements (800, 810, 820) respectively.

[0125] In some embodiments, the light distribution elements (730, 740, 750) are orthogonal pupil expanders (OPEs). In some embodiments, both of the OPEs deflect or distribute light to out-coupling optical elements (800, 810, 820) and also increase the beam or spot size of the light as it propagates to the out-coupling optical elements. For example, in some embodiments where the beam size is already a desired size, the light distribution elements (730, 740, 750) may be omitted, and the in-coupling optical elements (700, 710, 720) may be configured to directly deflect light to the out-coupling optical elements (800, 810, 820). For example, referring to FIG. 9a, the light distribution elements (730, 740, 750) can each be replaced with out-coupling optical elements (800, 810, 820). In some embodiments, the out-coupling optical elements (800, 810, 820) are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light toward the viewer's eye (210) (Fig. 7). It will be noted that the OPEs may be configured to increase the dimensions of the eye box along at least one axis, and the EPEs may increase the eye box along an axis that intersects the axis of the OPEs, for example, an orthogonal axis.

[0126] Accordingly, referring to FIGS. 9a and 9b, in some embodiments, a set of waveguides (660) for each component color comprises waveguides (670, 680, 690); in-coupling optical elements (700, 710, 720); light distribution elements (e.g., OPEs) (730, 740, 750); and out-coupling optical elements (e.g., EPs) (800, 810, 820). The waveguides (670, 680, 690) may be stacked to have an air gap / cladding layer between each waveguide. The in-coupling optical elements (700, 710, 720) redirect or deflect incident light toward their waveguides (as different in-coupling optical elements receive light of different wavelengths). Subsequently, light propagates within each waveguide (670, 680, 690) at an angle that causes TIR. In the illustrated example, a light beam (770) (e.g., blue light) is deflected by the first in-coupling optical element (700) and then continues to bounce along the waveguide to interact with the light distribution element (e.g., OPEs) (730) and then the out-coupling optical element (e.g., EPs) (800) in the manner previously described. Light beams (780 and 790) (e.g., green and red light, respectively) will pass through the waveguide (670), and the light beam (780) strikes the in-coupling optical element (710) and is deflected by it. Next, the light beam (780) is bounced along the waveguide (680) through the TIR to its own light distribution element (e.g., OPEs) (740) and then to the out-coupling optical element (e.g., EPs) (810). Finally, the light beam (790) (e.g., red light) passes through the waveguide (690) and strikes the light in-coupling optical elements (720) of the waveguide (690).The optical in-coupling optical elements (720) deflect the light beam (790) so that it is propagated by TIR to an optical distribution element (e.g., OPEs) (750) and then by TIR to an out-coupling optical element (e.g., EPs) (820). Then, the out-coupling optical element (820) finally out-couples the light beam (790) to a viewer, which also receives out-coupled light from other waveguides (670, 680).

[0127] FIG. 9c illustrates a top-down plan view of an example of multiple stacked waveguides of FIG. 9a and 9b. As illustrated, waveguides (670, 680, 690) can be vertically aligned with the associated optical distribution elements (730, 740, 750) and associated out-coupling optical elements (800, 810, 820) of each waveguide. However, as discussed herein, the in-coupling optical elements (700, 710, 720) are not vertically aligned; rather, the in-coupling optical elements are preferably not overlapped (e.g., laterally spaced when viewed in the top-down view). As further discussed herein, such non-overlapping spatial arrangements facilitate the injection of light from different resources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, an arrangement comprising non-overlapping spatially-separated in-coupling optical elements may be referred to as a shifted pupil system, and the in-coupling optical elements of such arrangements may correspond to sub-pupils. Display systems having regions of variable light transmittance

[0129] In embodiments of a display system (60) composed of augmented reality and / or virtual reality devices, the contrast, brightness, and / or clarity of the displayed augmented reality content and / or virtual reality content may be enhanced in dim or dim environments. For example, in bright daylight, a brightly lit room, and / or a glare-heavy rainy / foggy environment, the contrast, brightness, and / or clarity of the augmented reality content and / or virtual reality content may be reduced when viewed from the outside of embodiments of a display system (60) composed of augmented reality and / or virtual reality devices. Thus, when a part of the display (70) has glare and / or when the ambient light conditions for that part of the display (70) are bright to enhance the clarity of the view, it is advantageous if the intensity of the ambient light transmitted through that part of the display (70) can be reduced. In various embodiments, reducing the intensity of the ambient light through the part of the display (70) in an environment having bright ambient light conditions may advantageously enhance the user's visual experience.

[0130] In some embodiments, the display system (60) may be configured to measure the light intensity of bright ambient light sources, such as, for example, desk lamps, ceiling lights, streetlights, automotive headlights, the sun, or combinations thereof—not limited to these, and to attenuate the amount of light transmitted through one or more parts of the display (70) into which light from the bright ambient light sources is incident. The amount of light from the bright ambient light sources transmitted through one or more parts of the display (70) may be reduced by changing the transmittance of one or more parts of the display (70). For example, one or more parts of the display (70) may be darkened to reduce the amount of light from the bright ambient light sources transmitted through one or more parts. In some embodiments, the display (70) may include one or more optical elements, such as switchable light deflectors (e.g., optical zone plates, diffractive optical elements, or refractive optical elements) that can be switched to deflect some of the light from the bright ambient light sources. Light can be deflected to reduce the amount of light incident on the center of the eye or retina (e.g., the fovea) and the center of the viewer's field of vision. As a result of deflecting the light, the brightness of ambient light sources visible to the viewer may be reduced, and the contrast ratio of the virtual reality content may be increased. In various implementations, the light transmittance through one or more parts of the display (70) does not need to be reduced to an amount where bright ambient light sources are not visible through the display. Instead, the light transmittance through one or more parts of the display (70) may be reduced to a level that allows visibility of the virtual reality content with sufficient visual acuity and also allows visibility of bright ambient light sources.

[0131] Various embodiments of the display system (60) may include an omnidirectional camera / ambient light sensor configured to capture an image of a scene in a field of view (FOV) and determine the location and intensity of various bright light sources in the scene. The omnidirectional camera may be associated with the display system (60). For example, the omnidirectional camera may be mounted on the display (70). A relationship between the FOV of the camera and the FOV of the user through the display (70) may be determined. One or more parts of the display (70) corresponding to the determined locations of bright light sources in the scene, configured to have reduced light transmittance, may be determined by determining the locations of one or more bright light sources in the FOV of the image captured by the camera and identifying the locations of the display (70) corresponding to these bright light sources.

[0132] A method for determining the location and / or intensity of bright light sources in a scene may be similar to a method for updating one or more settings of a content capture device using automatic exposure control (AEC), as described in U.S. Patent Application No. 15 / 841,043, filed December 13, 2017, which is incorporated herein by reference in its entirety. Similar to the method illustrated in FIG. 1a and described in paragraphs

[0060] –

[0065] of U.S. Patent Application No. 15 / 841,043—incorporated herein by reference and filed December 13, 2017—an image captured by a camera / ambient light sensor may be divided into a plurality of pixel groups (e.g., 96 pixel groups, 120 pixel groups, 144 pixel groups, etc.). As incorporated herein by reference and described in paragraph

[0065] of U.S. Patent Application No. 15 / 841,043 filed December 13, 2017, an average lumina value for each pixel group may be calculated. In some examples, the average lumina pixel group value may be computed by accumulating the lumina values ​​for each pixel in the pixel group. In these examples, the lumina values ​​may represent the brightness of an image (e.g., an achromatic portion of the image or a grayscale image). Thus, the lumina value may be a representation of an image without color components. As another example, in the YUV color space, the lumina value may be Y. In some examples, the lumina value is a weighted sum of the gamma-compressed RGB components of the image. In these examples, the lumina value may be referred to as gamma-corrected lumina. In some examples, the accumulation may be performed by software or hardware by summing the lumina values ​​for each pixel in the pixel group. In some implementations, once the luminance values ​​for a pixel group are accumulated, the total number can be divided by the number of pixels within the pixel group to compute the average luminance pixel group value for the pixel group.This process can be repeated for each group of pixels in the image.

[0133] If the image captured by the camera is a grayscale image, the pixel value associated with multiple pixel groups of the grayscale image corresponds to the average luminance value. In some implementations, color images captured by an ambient light sensor may be converted to a YUV image format, and a luminance value corresponding to the Y component of the YUV image may be determined.

[0134] In some implementations, one or more bright spots on the display (70) may be identified as corresponding to one or more saturated regions of an image captured by an ambient light sensor. For example, one or more bright spots on the display (70) corresponding to the positions of bright light sources in the scene may be determined based on the maximum allowable difference in luminance values ​​between adjacent pixels or groups of adjacent pixels. The maximum allowable difference in luminance values ​​between adjacent pixels may be calculated in other ways. For example, in one method, pixels having relative pixel values ​​within a specific threshold of each other may be grouped together. Another method of grouping relative pixel values ​​relies on an adaptive k-means clustering algorithm that outputs a set of clusters having luminance values ​​exceeding a specific threshold level. In some implementations, the saturated region may correspond to a portion of the image having luminance values ​​exceeding a threshold. The threshold may be, for example, 220 for an 8-bit image ranging from 0 for black to 255 for white. Parts of the display (70) corresponding to parts of the image having luminance values ​​exceeding a specific threshold can be selectively occluded to reduce the transmittance of light from bright light sources. Other approaches may be used.

[0135] In some embodiments, the display system (60) may include an electronic processor (e.g., a local processing & data module (140) and / or a remote processing module (150)) configured to reduce the amount of light transmitted through parts of the display (70) that receive light from locations in the surrounding environment having a light intensity higher than the average light intensity of the surrounding environment. In this way, the intensity of light transmitted through the display (70) may be reduced in parts of the display (70) that receive most of the surrounding light. Additionally, the electronic processor may be configured to determine parts of the display (70) where virtual reality content is displayed and to reduce the amount of surrounding light transmitted through these parts in order to increase the relative brightness of the virtual reality content.

[0136] To enable selectively reducing the transmittance of light through one or more parts of the display (70), the display (70) may be configured as a pixelated display. For example, the surface of the display (70) may include a plurality of electronically addressable pixels that may be configured to change the amount of light transmitted therethrough. In some embodiments, the plurality of electronically addressable pixels may include a plurality of spatial light modulators. In some embodiments, the display (70) may include an occlusion mask of the plurality of electronically addressable pixels. The occlusion mask may include a plurality of mask elements, each mask element being associated with one or more of the plurality of addressable pixels. The plurality of mask elements may have different values ​​associated with different transmittance values ​​through the plurality of electronically addressable pixels. An electronic processor (e.g., local processing & data module (140) and / or remote processing module (150)) may be configured to selectively reduce the amount of light transmitted through one or more of the pixels to reduce the brightness of ambient light sources and / or improve the contrast ratio of virtual reality content.

[0137] As discussed above, the display (70) may include a display lens (106). In various embodiments, the display lens (106) may be a unitary lens positioned in front of both eyes of the user (90). The unitary lens may have eye regions positioned in front of each eye so that the user can see the surrounding environment. In some embodiments, the display lens (106) may include two lens elements, each lens element being positioned in front of each eye of the user (90). Each lens element may have an eye region that allows the user to see the surroundings.

[0138] The various embodiments described herein are configured to reduce the intensity of light transmitted through one or more parts of the display lens (106) by, for example, absorbing a portion of ambient light incident on a part of the display lens (106) and / or scattering / refracting / diffracting a portion of ambient light incident on the part(s) of the display lens (106) away from the pupil of the eye. Additionally, in embodiments of the display lens (106) comprising two lens elements positioned on each front of the eye, the intensity of ambient light transmitted through only one of the lens elements (or a part or parts thereof) may be reduced. As another example, the intensity of ambient light transmitted through one or both of the eye regions of the display lens (106) is reduced, while the intensity of ambient light transmitted through the remaining part of the display lens (106) is not reduced (or is reduced by a smaller amount). As another example, the intensity of ambient light transmitted through the first portion of the display lens (106) is reduced, whereas the intensity of ambient light transmitted through the second portion of the display lens is not reduced. In contrast to sunglasses that darken uniformly in bright sunlight and brighten uniformly indoors, various embodiments of the display lens (106) are configured to darken or brighten unevenly. For example, the display lens (106) may be partially darkened, for example, only a portion of the lens (106) may be darkened. As another example, the display lens (106) may be darkened by different amounts in different portions of the lens.Additionally, in various embodiments of the system (60), partially darkening parts of the display lens (106) can be achieved in response to a stimulus provided by the display system (e.g., optical stimulus provided by the light-emitting module (134), electrical stimulus provided by the electric system (132), thermal energy provided by the heat source (136), and / or acoustic / ultrasonic energy provided by the acoustic / ultrasonic transducer (138)) based on information obtained by one or more components that detect the user's environment, such as the light sensor (128), sensor assembly (126), outward-facing cameras (112) and / or inward-facing cameras (114), in relation to data from an associated cloud computing resource. In various embodiments of the display system (60), the dimming or brightening of the display lens (106) needs to occur automatically in response to a stimulus provided by the display system (e.g., an optical stimulus provided by a light-emitting module (134), an electrical stimulus provided by an electric system (132), thermal energy provided by a heat source (136), and / or acoustic / ultrasonic energy provided by an acoustic / ultrasonic transducer (138), based on environmental information acquired by one or more cameras / sensors of the system (60), without responding to ambient light conditions and regardless of data from associated cloud computing resources. In various embodiments, at least one part of the display lens (106) may be configured to transmit about 1% to 100% of the incident ambient light. For example, at least one part of the display lens (106) may be configured to transmit about 5%-90% of incident ambient light, about 10%-80% of incident ambient light, about 15%-75% of incident ambient light, about 20%-70% of incident ambient light, about 25%-60% of incident ambient light, about 30%-50% of incident ambient light, or any value of these ranges and / or sub-ranges.

[0139] The display lens (106) may include at least one variable optical material (e.g., organic molecules, proteins, photochromic materials, electrochromic materials, silver compounds, such as silver halide or silver chloride molecules, aerosols, hydrocolloids, etc.) that can be activated using heat, sound waves / ultrasound, optical and / or electrical stimulation to change at least one of the intensity of ambient light transmitted through the display lens (106), the spectral content of ambient light transmitted through the display lens (106), or the optical path of ambient light transmitted through the display lens (106) (e.g., by changing the refractive index of the variable optical element, e.g., by diffraction, scattering, refraction, or other means). The variable optical material may include a layer of molecules or a plurality of layers of molecules. In various embodiments, at least one variable optical material may include protein-based electroactive materials that respond to an electrical stimulus (e.g., a voltage signal and / or a current signal) provided by a display system (60) to change at least one of the intensity of ambient light transmitted through the display lens (106), the spectral content of ambient light transmitted through the display lens (106), or the optical path of ambient light transmitted through the display lens (106). For example, in response to an electrical stimulus provided by the display system (60), the protein-based electroactive materials may move, expand, contract, twist, rotate, attach together, or move apart from each other to change at least one of the intensity of ambient light transmitted through the display lens (106), the spectral content of ambient light transmitted through the display lens (106), or the optical path of ambient light transmitted through the display lens (106).In some embodiments, at least one variable optical material may include organic materials (e.g., oxazines and / or naphthofiranes) that change at least one of the intensity of ambient light transmitted through the display lens (106), the spectral content of ambient light transmitted through the display lens (106), or the optical path of ambient light transmitted through the display lens (106) in response to an optical stimulus provided by the display system (60). The molecules of the organic materials may be configured to change their size and / or shape when irradiated with light of specific frequencies or wavelengths (e.g., UV light). For example, the organic materials may be configured to expand and absorb more light when irradiated with light of specific frequencies (thus reducing the intensity of light transmitted to the user). As another example, molecules of organic materials may be configured to move, contract, twist, rotate, clump together, or move apart from each other to change the intensity of light transmitted through the display lens (106) in response to optical stimulation. Molecules of organic materials may change the intensity of light transmitted through the display lens (106) by absorbing a portion of the light transmitted through the display lens (106), by changing the color of the display lens (106), and / or by diffracting / refracting / scattering a portion of the light transmitted away from the display lens (106). As discussed above, the variable optical material may comprise a layer of molecules or multiple layers of molecules.

[0140] In various embodiments, at least one variable optical material may comprise one or more molecules combined with specific chemicals that can be configured to change the transmittance of light in response to a stimulus provided by the system (60). The chemicals combined with one or more molecules may be configured to change the intensity of the incoming ambient light, the direction of the incoming ambient light, and / or the spectral content of the incoming ambient light when irradiated by light of specific wavelengths (e.g., one or more wavelengths of UV, infrared, and / or the visible spectrum).

[0141] At least one variable optical material (e.g., photoreactive and / or electroactive materials) is configured to change at least one of the intensity of ambient light transmitted through the display lens (106), the spectral content of ambient light transmitted through the display lens (106), or the optical path of ambient light transmitted through the display lens (106) in response to a stimulus provided by the display system (60), the position of a desired part of the display lens (106) where the intensity of the incoming ambient light, the direction of the incoming ambient light, and / or the spectral content of the incoming ambient light are changed (e.g., by absorption at the desired part and / or by changing the color of the desired part and / or by diffracting / refracting / scattering the ambient light away from the desired part), the duration of time during which the desired part of the display lens (106) is configured to change the intensity of the incoming ambient light, the direction of the incoming ambient light, and / or the spectral content of the incoming ambient light, and the rate at which the desired part of the display lens (106) becomes darker or brighter It can be controlled (e.g., precisely controlled).

[0142] Additionally, the distribution of at least one variable optical material across the surface of the display lens (106) can be customized to meet specific requirements / functions. In some embodiments, at least one variable optical material may be distributed uniformly across the surface of the display lens (106). In some other embodiments, at least one variable optical material may be distributed non-uniformly across the surface of the display lens (106) so that parts of the display lens (106) may have a higher density of at least one variable optical material compared to other parts of the display lens (106). In some embodiments, the density of at least one variable optical material in parts of the eye regions of the display lens (106) may be higher than in parts of non-eye regions that the user cannot see (e.g., temples, nose bridge, eye orbitals and other non-eye regions of the user's face). In some embodiments, certain regions of the display lens (106) (e.g., non-ocular regions) may not have at least one variable optical material, because there may be no need to change at least one of the intensity of ambient light, the spectral content of ambient light, or the optical path of ambient light in these regions.

[0143] Various embodiments of the display lens (106) may include a plurality of layers, each layer may include variable optical materials that change at least one of the intensity of ambient light, the spectral content of ambient light, or the optical path of ambient light in response to a stimulus provided by the display system (60). The materials of the plurality of layers may be configured to act on different wavelengths of the incoming ambient light. For example, the materials of the plurality of layers may attenuate different wavelengths of the incoming ambient light by different amounts. As another example, the materials of the plurality of layers may absorb different wavelengths of the incoming ambient light by different amounts. As yet another example, the materials of the plurality of layers may diffract / scatter / refract different wavelengths of the incoming ambient light by different amounts.

[0144] Accordingly, some embodiments of the display lens (106) include a first layer comprising a first variable optical material configured to attenuate red light (e.g., by absorption, diffraction, refraction, reflection, or scattering) in response to a stimulus provided by the display system (60), a second layer comprising a second variable optical material configured to attenuate green light (e.g., by absorption, diffraction, refraction, reflection, or scattering) in response to a stimulus provided by the display system (60), a third layer comprising a third variable optical material configured to attenuate blue light (e.g., by absorption, diffraction, refraction, reflection, or scattering) in response to a stimulus provided by the display system (60), a fourth layer comprising a fourth variable optical material configured to attenuate ultraviolet light (e.g., by absorption, diffraction, refraction, reflection, or scattering) in response to a stimulus provided by the display system (60), and / or infrared light (e.g., by absorption, It may include a fifth layer comprising a fifth variable optical material configured to attenuate (by diffraction, refraction, reflection, or scattering). A subset of these layers may alternatively be included in a display lens or display system. For example, the first, second, and third layers attenuate red, green, and blue light, respectively. In these embodiments, thermal, acoustic / ultrasonic, optical, or electrical stimulation may be provided to one or more of the plurality of layers to attenuate light of specific wavelengths (e.g., by absorption, diffraction, refraction, reflection, or scattering) based on environmental information acquired by one or more cameras / sensors of the system (60), regardless of data from associated cloud computing resources.

[0145] In various embodiments, groups of variable optical materials having the same chemical / physical properties may be individually activated to perform various functions without activating other groups of variable optical materials having different chemical / physical properties. In various embodiments, variable optical materials that change at least one of the intensity of ambient light incident on the display lens (106), the spectral content of the ambient light, or the optical path of the ambient light may be provided only to specific parts of the display lens (106) (e.g., eye portions of the display lens (106), portions of eye portions of the display lens (106), portions of only one of the eye portions of the display lens (106), etc.). In some of these embodiments, portions of the display lens (106) containing the variable optical materials may automatically darken / brighten in the presence / absence of daylight without requiring any additional stimulation from the display system (60).

[0146] In various embodiments, variable optical materials may be incorporated into a display lens (106). However, in some other embodiments, variable optical materials may be included in an add-on device that can be attached to or detached from the display lens (106). Examples of display lenses incorporated into variable optical materials and / or add-on devices including variable optical materials may be configured to be activated by a small amount of activation energy (e.g., thermal, acoustic / ultrasonic, optical and / or electrical energy). In some cases, after activation, physical and / or chemical changes in the molecules of the variable optical materials that alter at least one of the intensity of ambient light, the spectral content of ambient light, or the optical path of ambient light may occur without requiring any additional amount of energy. The physical and / or chemical changes in the variable optical materials may be maintained until the variable energy materials are deactivated by providing deactivation energy (e.g., thermal, acoustic / ultrasonic, optical and / or electrical energy).

[0147] As discussed above, in some embodiments, the display (70) may be configured as a pixelated display. For example, the surface of the display (70) may include a plurality of electronically addressable pixels capable of changing the amount of light transmitted therefrom in response to an electrical or optical stimulus. In some embodiments, the plurality of electronically addressable pixels may include a plurality of spatial light modulators. In some embodiments, the display (70) may include an occlusion mask comprising a plurality of mask elements associated with a plurality of electronically addressable pixels. An electronic processor (e.g., a local processing & data module (140) and / or a remote processing module (150)) may be configured to provide an electrical or optical signal to selectively reduce the amount of light transmitted through one or more of the plurality of pixels so as to reduce the brightness of an ambient light source and / or improve the contrast ratio of virtual reality content.

[0148] The following examples illustrate the advantages and various operational features of an embodiment of a display system (60) configured to modify at least one of the intensity of ambient light incident on the display (70), the spectral content of the ambient light, and / or the direction of the ambient light, as described above. An embodiment of a display system (60) comprising variable optical materials worn by a user (90) (integrated into the display lens (106) of the display system (60) or included in an add-on device) is considered. When the user moves from a low ambient light condition (e.g., indoors) to a bright environment (e.g., outdoors), sensor assemblies of the display system (60) (e.g., light sensors, outward-facing cameras, inward-facing cameras, etc.) will detect a change in the ambient light condition. Sensor assemblies may be configured to detect changes in ambient light conditions by detecting changes in the environment using information regarding the surrounding environment obtained using location-specific information (e.g., information obtained by GPS, a compass, and / or information obtained from associated cloud computing resources), object recognition algorithms for determining trees / parks, buildings, rooms, etc., temperature sensors, etc. In addition to determining changes in the intensity of ambient light conditions, sensor assemblies may be configured to determine the spectral characteristics of incident light. Sensor assemblies may be configured to determine the intensity / spectral characteristics of ambient light incident on different parts of the display lens (106). Sensor assemblies may include filters and / or sensors having specific spectral responses to determine the spectral characteristics of ambient or incident light.Accordingly, in certain embodiments, sensor assemblies may be configured to not only locate and identify the positions of various real-world ambient light sources visible to the user through the display (70), but also to identify parts of the display (70) and / or display lens (106) that are aligned with the ambient light sources for a specific position of the user's head. Once the various parts of the display lens (106) that match the real-world ambient light sources are known, the system (60) may provide optical, electrical, thermal, and / or acoustic / ultrasonic stimulation to the different parts of the display lens (106) to cause parts of the incident ambient light to be absorbed, deflected, refracted, scattered, and / or reflected, so that the amount of ambient light transmitted through the parts of the display lens (106) that match the real-world ambient light sources is reduced or otherwise modified. In this way, the amount of ambient light transmitted through the display lens (106) may change across the surface of the display lens (106) depending on environmental conditions. For example, when sunlight is incident on the display lens from one side of the user so that the amount of ambient light incident on the surface of the display lens (106) is not uniform, it is taken into account that the user (90) is outside during the morning or evening hours. In these embodiments, the system (60) may be configured to transmit a greater amount of light through one part of the display lens (106) than the amount of light transmitted through another part of the display lens (106). In various embodiments, the amount of light transmitted through one part of the display lens (106) may be about 1%-100% (e.g., 2%-95%, 5%-90%, 7%-80%, 10%-75%, 15%-50%, 20%-60%, 30%-85%, etc.) than the amount of light transmitted through another part of the display lens (106).

[0149] In various embodiments, information obtained from various sensors and / or camera assemblies may be transmitted to a local processing & data module (140) and / or a remote processing module (150) for processing. The local processing & data module (140) and / or the remote processing module (150) may determine one or more positions of a display lens (106) aligned with different ambient light sources by processing information obtained from various sensors and / or camera assemblies. In some embodiments, the local processing & data module (140) and / or the remote processing module (150) may store the positions of various real-world objects in relation to the display device and / or the user's head / eyes in a database. The database may update the information in real-time or near real-time when the user moves their head, or when surrounding real-world objects appear to be moving in relation to the display device and / or the user's head / eyes. The database may update or provide information regarding the positions of new objects from the surrounding real world that enter the user's field of vision in real or near real time, in relation to the display device and / or the user's head / eyes. In various embodiments, the local processing & data module (140) and / or the remote processing module (150) may be configured to determine the intensity / spectral characteristics of ambient light sources that appear to be aligned with different parts of the display lens (106) when viewed through the display lens (106). The local processing & data module (140) and / or the remote processing module (150) may be configured to reduce the amount of ambient light transmitted through parts of the display lens (106) when viewed through the display lens (106) that appears to be aligned with the ambient light sources.The local processing and data module (140) and / or the remote processing module (150) may transmit signals capable of triggering the light-emitting module (134), the electrical system (132), the heat source (136), and / or the sound / ultrasonic transducers (138) to provide appropriate stimulation to activate the variable optical elements of the different parts of the display lens (106) to attenuate ambient light by an appropriate amount in those parts. As discussed above, light through the different parts of the display lens (106) may be attenuated by the same or different amounts depending on the intensity / spectral characteristics of the light from ambient light sources that appear to be aligned with these parts. This may be advantageous when light is incident on the user's eyes from one side, such as a desk lamp positioned on one side of the user, daylight during the morning or evening hours, or from real-world objects visible through the different parts of the display lens (106) that generate different amounts of glare.

[0150] In various embodiments, the system (60) may be configured to acquire information about the environment continuously or substantially continuously. For example, the system (60) may be configured to acquire information about the environment from various cameras / sensor assemblies in intervals of 1-30 microseconds, 100-500 microseconds, 400 microseconds to 1 millisecond, 1-30 milliseconds, 20-100 milliseconds, 50-500 milliseconds, 400 milliseconds to 1 second, 1-5 second, or any values ​​of these ranges or sub-ranges or any combination thereof. The local processing & data module (140) and / or remote processing module (150) may be configured to process information obtained from various cameras / sensor assemblies of the system (60) and to transmit signals capable of triggering a light-emitting module (134), an electrical system (132), a heat source (136), and / or acoustic / ultrasonic transducers (138) to provide the necessary stimulus to activate variable optical elements in different parts of the display lens (106) in real time or near real time so as to maintain the user experience when, for example, environmental conditions change.

[0151] For example, in various embodiments, light sensors (128) may be configured to detect the intensity and / or spectral characteristics of ambient light incident on the display lens. Additionally, outward-facing cameras, inward-facing cameras, and other sensor assemblies may be configured to acquire information about the surrounding world visible to the user through the display lens (106), which can help identify the positions of different ambient light sources and / or real-world glare-generating objects, as well as their positions relative to the display (70) and / or display lens (106) and / or the user's eye. In various embodiments, the display system (60) may also be configured to identify the nature of an ambient light source (e.g., daylight, fluorescent light, incandescent light, LED light, candlelight) that appears to be aligned with different parts of the display lens (106). Once the system (60) has identified the positions of various ambient light sources in relation to the display (70) and / or the display lens (106) and / or the user's eye, the system (60) may determine the parts of the display (70) and / or the display lens (106) where light transmission characteristics must be changed to maintain / enhance the user's visual experience. The system (60) may, for example, provide stimuli to the determined parts of the display lens (106) to attenuate the light transmitted through those parts in real-time or near real-time and / or to change the direction or spectral characteristics of the light transmitted through those parts in order to maintain / enhance the user's visual experience. In this way, the user's visual experience is not necessarily substantially impaired as a result of glare or intensity changes across the surface of the display lens (106).

[0152] In various embodiments, the system (60) may be configured to store a map of locations frequently visited by the user in a data repository accessible by the local processing & data module (140) and / or the remote processing module (150). The stored map for one or more locations frequently visited by the user may include the positions of ambient light sources (e.g., streetlights, porch lights, traffic lights, etc.). Information regarding the intensity and / or spectral content of light from ambient light sources at one or more frequently visited locations may also be stored in the data repository. Information regarding how the light transmission characteristics of various parts of the display lens (106) should change during the day, night, and / or multiple times throughout the year may be predetermined for one or more locations frequently visited by the user and, of course, stored in the data repository. For example, for a location frequently visited by the user, information regarding how the light transmission capability of different parts of the display lens (106) that appear to be aligned with different ambient light sources at that location should change during daytime may be stored in the data repository. As another example, for a location frequently visited by a user, information regarding how the light transmission capability of different parts of the display lens (106) that appear to be aligned with different ambient light sources at that location should be changed during the night (or any other time) may be stored in a data repository. As yet another example, for a location frequently visited by a user, information regarding how the light transmission capability of different parts of the display lens (106) that appear to be aligned with different ambient light sources at that location should be changed during the summer day may be stored in a data repository.As another example, for locations frequently visited by the user, information regarding how the light transmission capability of different parts of the display lens (106) that appear to be aligned with different ambient light sources at those locations should be changed during winter days may be stored in a data repository. The locations and characteristics of different light sources at different locations (e.g., size, shape, brightness, color, etc.) may also be recorded and stored for later access and use.

[0153] The local processing and data module (140) and / or the remote processing module (150) can identify a location from sensor information and access information from a data repository regarding the location and other characteristics of light sources (e.g., size, shape, brightness, color, etc.), as well as potentially, how the light transmission ability of different parts of the display lens (106) that appear to be aligned with different ambient light sources at that location should be changed for that specific time of day and year. This information can be used to instruct stimulus-providing sources to activate variable optical materials in different parts of the display lens to change the intensity, spectral content, and / or direction of ambient light according to predetermined information.

[0154] This can advantageously save processing time. For example, information regarding various ambient light sources (e.g., lamps, windows, ceiling lights, etc.) in the user's home or office (e.g., location, intensity, spectral content, etc.) can be stored in a data storage. Information regarding the position of the sun and the direction of sunlight at various times of the day can also be stored in a data storage. When the system (60) detects from information acquired by a sensor that the user is in an office or home, the local processing & data module (140) and / or the remote processing module (150) can transmit signals appropriate for darkening and / or brightening various parts of the display lens (70) to various stimulus-providing sources based on the stored information regarding various ambient light sources in the user's home or office (e.g., location, intensity, spectral content, etc.).

[0155] FIG. 10 illustrates a scene (1000) viewed by a user at night through a display lens (1006) of an embodiment of a display system. The display system may have features similar to the display system (60) described above. For example, the display system may include one or more sensors configured to acquire information about the scenes, and the information may include the positions of various ambient light sources related to the display lens (1006), the brightness of various ambient light sources, and / or the types of various ambient light sources (e.g., fluorescent lamps, LED lights, incandescent lamps, etc.). The display system may also include electronic processing systems configured to process information acquired by one or more sensors. Processing information acquired by one or more sensors may include identifying parts of the display lens (1006) that appear to be aligned (or matched) with various ambient light sources in the scene (1000) viewed by the user, and determining the light transmission characteristics of one or more parts of the display lens (1006) in order to enhance / maintain the user's visual experience. The display lens (1006) comprises one or more variable optical materials configured to change the intensity of incident ambient light, the spectral content of incident ambient light, and / or the direction of incident ambient light in response to optical, electrical, thermal, and / or sound / ultrasonic stimulation provided by the display system. The display lens (1006) may have features similar to those of the display lens (106). The scene (1000) includes several ambient light sources (1005a, 1005b, 1005c, 1005d, 1005e, and 1005f) along the porch and walkway of the house (1003). The ambient light sources (1005a-1005f) may include porch lights, streetlights, indoor lights, outdoor lights, path lights, landscape lights, etc.In embodiments of display lenses without one or more variable optical materials, ambient light sources (1005a-1005f) can generate glare and / or degrade the clarity of the field of view when viewed through parts of the display lenses that appear to be aligned with the ambient light sources (1005a-1005f). In contrast, the display lens (1006) is configured to change the intensity, spectral content, and / or direction of the ambient light incident on the display lens (1006) through parts of the display lens (1006) that appear to be aligned with the ambient light sources (1005a-1005f) in order to reduce interference with the user experience caused by glare from the ambient light (1005a-1005f).

[0156] As discussed above, sensors associated with the display system can acquire information about the scene (1000) continuously or intermittently. The information may include the positions of ambient light sources (1005a-1005f) related to the display lens (1006), the direction, intensity, and spectral content of ambient light from the ambient light sources (1005a-1005f). Electronic processing systems can process the information acquired by one or more sensors and determine how the distribution of ambient light across the surface of the display lens (1006) should be changed. For example, in some embodiments, electronic processing systems may determine that areas of a display lens (1006) (e.g., 1010a, 1010b, 1010c, 1010d, 1010e, and 1010f) containing portions of a scene containing ambient light sources (1005a-1005f) should be darkened to reduce the intensity of ambient light transmitted through such portions. As another example, in some embodiments, electronic processing systems may determine that ambient light incident on areas of a display lens (1006) (e.g., 1010a, 1010b, 1010c, 1010d, 1010e, and 1010f) containing portions of a scene containing ambient light sources (1005a-1005f) should be diffused to reduce glare. As other examples, in some embodiments, electronic processing systems may determine that ambient light incident on an area of ​​a display lens (1006) (e.g., 1010a, 1010b, 1010c, 1010d, 1010e and 1010f) containing ambient light sources (1005a-1005f) must be redirected to reduce glare.

[0157] Based on that decision, the electrical processing system transmits signals to activate optical, thermal, acoustic / ultrasonic and / or electrical sources associated with the display system and can provide the desired optical, thermal, acoustic / ultrasonic and / or electrical stimulation to the region of the display lens (1006) containing ambient light sources (1005a-1005f) that causes physical and / or chemical changes to variable optical materials in the region of the display lens (1006) (e.g., 1010a, 1010b, 1010c, 1010d, 1010e and 1010f) that cause the intensity of the incident ambient light, the spectral content of the incident ambient light and / or the direction of the incident ambient light.

[0158] In various embodiments, the system (60) may be configured to track the movement of the user’s eyes and / or head in real or near real time and to determine the relative position between the user’s eyes and real-world objects (e.g., trees, the sun, ambient light sources, etc.) in real or near real time. In these embodiments, the system (60) may be configured to dynamically change the ambient light transmission characteristics through different parts of the display lens as the user’s head and / or eyes move, for example, to maintain / enhance the user’s visual experience. For example, it is considered that the user’s head is in a first position and the ambient light source appears to be aligned with the part of the display lens (106) to the left of the user’s left pupil. When the user remains in the first head position, the part of the display lens (106) to the left of the left pupil may be darkened or otherwise modified to reduce the intensity of the ambient light transmitted through that part. When the user’s head moves to a second position to the left, the ambient light source may now appear to be aligned with the part of the display lens (106) to the right of the left pupil. Accordingly, when the user's head is in the second position, the portion of the display lens (106) to the right of the left pupil may be darkened or otherwise modified to reduce the intensity of ambient light transmitted through that portion in order to maintain the user's visual experience. The portion of the display lens (106) to the left of the left pupil, which was previously darkened when the head is in the first position, may be in a brightened or darkened state. A sensor, such as an external camera capable of imaging the field of view of the front of the spectacle and providing a mapping of the positions of objects including bright light sources in the sensor's field of view in relation to the lenses and the user's eye, may be used to determine the portions of the lens to be modified, for example, to attenuate light from bright objects that generate glare.Similarly, a database containing records of the positions of objects and, for example, their brightness can also be used to determine, for example, parts of the lens to be modified to attenuate light from bright objects that generate glare. A head pose sensor and / or system can be used to determine the movement, position, and / or orientation of the head and / or body. This position can be used in relation to a database of object positions to determine the position of an object relative to the user's eye and lens, and to determine the parts of the lens to be modified as well as the parts of the lens aligned with the object(s).

[0159] FIG. 11 illustrates a flowchart (1100) illustrating a method for modifying ambient light transmission features through a display device to enhance the user's visual experience when using an embodiment of the display system (60). The method includes the step of obtaining information regarding the positions of various ambient light sources and / or glare-generating objects in a scene viewed by the user through the display device using one or more sensors, as illustrated in block (1105). For example, one or more light sensors (128), outward-facing camera(s) (112), and / or other sensor assemblies of the display system (60) may be configured to obtain information regarding the location and properties of various ambient light sources and / or glare-generating objects in a scene viewed by the user. The information obtained by one or more light sensors (128), outward-facing camera(s) (112), and / or other sensor assemblies of the display system (60) may include spectral features of the ambient light and / or other features of the ambient light (e.g., the intensity of the ambient light). As another example, one or more light sensors (128), outward-facing camera(s) (112), and / or other sensor assemblies of the display system (60) may be configured to acquire information regarding the location of objects, or areas or zones of a bright forward field of view and one or more areas or zones of a dark forward field of view. As yet another example, one or more light sensors (128), outward-facing camera(s) (112), and / or other sensor assemblies of the display system (60) may be configured to acquire the location of one or more bright ambient light sources and the intensity of light from the bright ambient light sources. Information acquired by one or more sensor assemblies is transmitted to one or more electronic processing systems (e.g., a local processing & data module (140) and / or a remote processing module (150)) for processing. The electronic processing systems may be local or remote.One or more electronic processing systems, as illustrated in block (1107), can process information acquired by one or more sensor assemblies and determine characteristics of ambient light at one or more locations of the display lens (106). The determined characteristics may include the intensity of ambient light at one or more locations of the display lens (106) and / or spectral characteristics of ambient light at one or more locations of the display lens (106). In some embodiments, one or more electronic processing systems may also be configured to determine whether the ambient light source is the sun, a fluorescent light source, an LED light source, or a combination of these light sources. Additionally, as illustrated in block (1107), one or more electronic processing systems may be configured to identify parts of the display lens that appear to be aligned with various ambient light sources and / or glare-generating objects in the scene viewed by the user through the display lens. One or more electronic processing systems can determine ambient light transmission characteristics at one or more locations of the display lens (106) to enhance the user's visual experience, based on the intensity and / or spectral characteristics of the various ambient light sources and / or glare-generating objects, and the determined portions of the display lens (106) that correspond to various ambient light sources and / or glare-generating objects, as illustrated in block (1109).

[0160] For example, one or more electronic processing systems may determine the amount by which one or more locations of the display lens (106) must be darkened to enhance the user's visual experience. As another example, based on the determination that the ambient light is from the sunset, one or more electronic processing systems may determine that changing the transmission characteristics of the portion of the display lens (106) aligned with the sun when viewed by the eye can reduce glare caused by the sun. Similarly, reducing the amount of light of one or more wavelengths (e.g., red wavelengths) of the received light transmitted through the portion of the display lens (106) aligned with the sun when viewed by the eye can reduce glare and possibly enhance the user's visual experience.

[0161] One or more electronic processing systems may transmit signals to trigger one or more stimulus-providing sources associated with the display system (60) to modify ambient light transmission features at one or more locations of the display lens (106), or to cause the stimulus-providing sources to modify the ambient light transmission features, according to a decision made by the one or more electronic processing systems as illustrated in block (1111). For example, one or more electronic processing systems may transmit signals to turn on one or more of the optical, electric, thermal and / or acoustic / ultrasonic sources associated with the display system (60), and may provide optical, electric, thermal and / or acoustic / ultrasonic signals to change the physical / chemical features of the molecules of the variable optical material in that part to modify the ambient light transmission features of at least a part of the display lens (106). As another example, one or more electronic processing systems may turn on an optical source or system associated with the display system (60) to provide an optical signal to modify the physical / chemical characteristics of molecules of a variable optical material in that portion to modify the ambient light transmission characteristics of at least a portion of the display lens (106), or otherwise transmit signals to cause the optical source or system to provide an optical signal. The optical signal may have a predetermined intensity and wavelength. For example, the optical signal may be a visible or invisible light beam of a specific wavelength. The molecules of the variable optical material may expand, contract, move, twist, or rotate in response to stimulation provided by signals from optical, electrical, thermal, and / or acoustic / ultrasonic sources associated with the display system (60), and may provide desired ambient light modification characteristics (e.g., attenuation, light deflection, diffusion, etc. in one or more wavelength regions).

[0162] FIG. 12a schematically illustrates a side view of a display lens (1206) positioned in front of the user's eye (1201). FIG. 12b schematically illustrates a front view of the display lens (1206) as viewed from the side facing the eye. FIG. 12c schematically illustrates a top view of the display lens (1206). An ambient light source (1210) of the scene viewed by the user through the display lens (1206) appears to be aligned with the area (1207) of the display lens (1206). As illustrated in FIG. 12a-12c, the ambient light source (1210) appears to be aligned with the area (1207) of the display lens (1206) in both the x- and y- directions. Likewise, the area (1207) of the display lens (1206) appears to be aligned with the light source (1210) when viewed by the user's eye in both the x- and y- directions. As discussed in this application, an electronic processing system associated with a display lens (1206) may be configured to modify / change the transmittance of ambient light through a region (1207) of the display lens (1206) to enhance the user's visual experience. For example, in some embodiments, the region (1207) may be darkened relative to other parts of the display lens (1206) to reduce the intensity of ambient light transmitted through the region. In some other embodiments, ambient light incident through the region (1207) may be directed away from the user's eye (1201). Other features of the display may be modified.

[0163] Various studies may be performed to characterize the optical modification features of variable optical materials. Different studies may also be performed to characterize the type of optical modification that will result in a desired user experience for different types of ambient light sources. For example, different embodiments of the display system (60) may be tested before being used by a user to characterize the optical modification features of the variable optical material. The tests may include analyses such as the stimulation intensity required to achieve a specific modification in a desired part of the display (70) or display lens (106), the time interval between providing the stimulation and achieving the modification in the desired part of the display (70), and the modification that will provide an enhanced visual experience to an average user for different ambient light sources. The results of the various studies may be stored in a database accessible by the local processing & data module (140) and / or the remote processing module (150). The local processing & data module (140) and / or the remote processing module (150) can access the results of various studies when determining the properties of the optical correction ability of a specific part of the display lens and the signals to be transmitted to various stimulus-providing sources.

[0164] In various embodiments, the display system (60) may be configured to obtain from the user feedback regarding the size and / or location of parts of the display (70) and / or display lens (106) having modified light transmission performance and to what extent the light transmittance should be modified to enhance the user's visual experience in various parts of the display (70) and / or display lens (106). In these embodiments, the local processing & data module (140) and / or remote processing module (150) may initially determine the size and / or location of parts of the display (70) and / or display lens (106) having modified light transmission capability based on information obtained from imaging systems and / or various sensors associated with the system (60). The local processing & data module (140) and / or remote processing module (150) may also initially determine to what extent the light transmittance through various parts of the display (70) and / or display lens (106) should be modified based on the results of initial tests and studies. Subsequently, the system (60) may prompt the user using visual and / or auditory signals and request feedback from the user regarding the size and / or position of parts of the display (70) and / or display lens (106) having modified ambient light transmittance and the extent to which light transmittance through various parts is modified. The local processing & data module (140) and / or remote processing module (150) may, based on feedback from the user, adjust the size and / or position of parts of the display (70) and / or display lens (106) having modified light transmittance and the extent to which light transmittance in various parts of the display (70) and / or display lens (106) should be changed. In this way, the visual experience can be enhanced based on the user's preferences. The user may provide feedback in various ways.For example, the user may provide feedback using voice commands. As another example, the user may use one or more buttons or knobs, a joystick, a touchpad, or a trackball to provide feedback. As yet another example, the user may use gestures (e.g., hand gestures, facial gestures, blinking responses, etc.) to provide feedback. An example of a display device configured to adjust the size and / or position of parts of the display (70) and / or display lens (106) that have altered light transmittance capabilities and the extent to which light transmittance should be modified in various parts of the display (70) and / or display lens (106) based on feedback from the user is discussed below.

[0165] Embodiments of a display system are considered for determining one or more parts of a display lens that appear to be aligned with one or more ambient light sources in a scene viewed by a user through the display lens. In response to that determination, the system may be configured to darken one or more parts of the display lens that appear to be aligned with one or more ambient light sources in the scene. Subsequently, the system may request feedback from the user regarding the size and / or locations of the one or more darkened parts of the display lens and the amount of darkening these parts. The user may provide feedback that the system can use to adjust the size and / or locations of the one or more darkened parts of the display lens and the amount of darkening these parts.

[0166] The variable optical materials discussed herein may be configured to function as filters to filter out specific wavelengths of incoming light, such as blue light, red light, green light, or some other wavelengths of light, in order to enhance the user experience. In various embodiments, the variable optical materials may be configured to direct the incoming light toward or away from specific areas of the eye. In such embodiments, inward-facing cameras (114) may be used to track eye movements, and the chemical / physical properties of the variable optical materials may be controlled by providing a stimulus from the system (60) so that the incoming light continues to be directed toward or away from specific areas of the eye despite eye movements. In various embodiments, the variable optical materials may be configured to partially or completely attenuate incoming light from the environment (e.g., to prevent sensory overload in certain environments).

[0167] Although the attenuation, diffusion, refraction, diversion, filtering, and / or scattering of ambient light through parts of the display lens (106) is discussed above, in any such case, different lenses in specific embodiments may attenuate, diffuse, refract, divert, filter, and / or scatter the incident ambient light. For example, the left and right lenses may attenuate, diffuse, refract, divert, filter, and / or scatter the incident ambient light by different amounts. Additionally, different parts of the left and right lenses may attenuate, diffuse, refract, divert, filter, and / or scatter the incident ambient light differently. Direct control over the degree of attenuation and the parts of the lenses being attenuated enables attenuation of different sizes and distributions as well as attenuation of different parts of the left and right lenses having different shapes and / or sizes. Other features, such as the spectral characteristics of the left and right lenses and their attenuation, may differ.

[0168] It is considered that various embodiments may be implemented or associated with various applications, such as imaging systems and devices, display systems and devices, spatial light modulators, liquid crystal-based devices, polarizers, waveguide plates, etc. The structures, devices, and methods described herein may find particular use in displays, such as wearable displays (e.g., head-mounted displays) that can be used in augmented and / or virtual reality. More generally, the described embodiments may be implemented in any device, apparatus, or system configured to display images, whether dynamic (e.g., video) or static (e.g., still images), and whether textures, graphics, or pictures.However, the described embodiments are various electronic devices, such as: mobile phones, multimedia internet-enabled cellular phones, mobile television receivers, wireless devices, smartphones, Bluetooth® devices, PDAs (personal data assistants), wireless email receivers, handheld or portable computers, netbooks, laptops, smartbooks, tablets, printers, photocopiers, scanners, fax devices, GPS (global positioning system) receivers / navigators, cameras, digital media players (e.g., MP3 players), camcorders, game consoles, wristwatches, clocks, calculators, television monitors, flat-panel displays, electronic reading devices (e.g., e-readers), computer monitors, automotive displays (including odometer and speedometer displays, etc.), cockpit controls and / or displays, camera view displays (e.g., displays of a vehicle's rear-view camera), electronic photographs, electronic It is considered that the teachings may be included in or associated with billboards or signals, projectors, architectural structures, microwaves, refrigerators, stereo systems, cassette recorders or players, DVD players, CD players, VCRs, radios, portable memory chips, washers, dryers, washer / dryers, parking meters, head-mounted displays, and various imaging systems (but not limited thereto). Accordingly, the teachings are not intended to be limited to the embodiments depicted only in the drawings, but instead have a wide applicability that will be readily apparent to those skilled in the art.

[0169] Various modifications to the embodiments described in this disclosure may be obvious to those skilled in the art, and general principles defined herein may be applied to other embodiments without departing from the spirit or scope of this disclosure. Various changes may be made to the described invention, and equivalents may be substituted without departing from the actual spirit and scope of the invention. Additionally, a number of modifications may be made to adapt a particular situation, material, composition of material, process, process operation(s), or step(s) to the purpose(s), spirit, or scope of the invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0170] The word “exemplary” is used exclusively herein to mean “functioning as an example, case, or example.” No embodiment described herein as “exemplary” is to be interpreted as being more desirable or advantageous than other embodiments. Additionally, those skilled in the art will recognize that terms such as “top” and “bottom,” “up” and “bottom,” etc., are sometimes used to facilitate describing the drawings and to indicate relative positions corresponding to the orientation of the drawings on a suitably oriented page, and that when these structures are implemented, they may not reflect the proper orientation of the structures described herein.

[0171] Certain features described herein in the context of separate embodiments may also be implemented in combination of a single embodiment. In contrast, various features described in the context of a single embodiment may also be implemented separately in a number of embodiments or in any suitable subcombination. Furthermore, although features are described above as operating in certain combinations and may even be claimed as such initially, one or more features from the claimed combination may be omitted from the combination in some cases, and the claimed combination may be led to a subcombination or a variation of the subcombination.

[0172] Similarly, while operations may be illustrated in the drawings in a specific order, it should be understood that, to achieve desired results, such operations do not need to be performed in the specific order illustrated or sequential order, nor do all illustrated operations need to be performed. Additionally, the drawings may schematically illustrate one or more exemplary processes in the form of flowcharts. However, other unillustrated operations may be incorporated into the schematically illustrated exemplary processes. For example, one or more additional operations may be performed before, after, simultaneously with, or in between any of the illustrated operations. In certain environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments fall within the scope of the following claims. In some cases, the actions listed in the claims may be performed in a different order, and nevertheless, desired results may be achieved.

[0173] The present invention includes methods that can be performed using the devices of the present invention. The methods may include an operation of providing such a suitable device. Such provision may be performed by an end user. In other words, the “providing” operation merely requires the end user to acquire, access, access, position, set up, activate, power up, or otherwise operate to provide the device essential in the method of the present invention. The methods cited herein may be performed with any logically possible order of cited events as well as with events of the cited order.

[0174] Exemplary aspects of the present invention have been described above, along with details regarding material selection and manufacturing. Other details of the present invention are known or may be generally known to those skilled in the art, as well as recognized in connection with the patents and publications referenced above. These may be applied to the method-based aspects of the present invention in terms of additional operations as commonly or logically utilized.

[0175] Additionally, although the present invention has been described with reference to several examples optionally including various features, the invention is not to be limited to what is described or shown as considered for each variation of the invention. Various variations may be made to the described invention, and equivalents may be substituted without departing from the actual spirit and scope of the invention (whether cited herein or not included for the sake of some simplification). Additionally, where various values ​​are provided, every intervening value between the upper and lower limits of the range and any other mentioned or intervening value within the mentioned range are to be interpreted as being included within the invention.

[0176] Additionally, it is considered that any optional feature of the variations of the described invention may be described and claimed independently or in combination with any one or more of the features described herein. References to singular items include the possibility that multiple identical items exist. More specifically, as used in this and the claims associated with this, singular forms include multiple references unless explicitly stated otherwise. In other words, the use of singulars allows for "at least one" of the claimed items described above, as well as the claims associated with this disclosure. Further note is that these claims may be drafted to exclude any optional element. Accordingly, such description is intended to serve as a basis for the use of such exclusive terms, such as "only" or "only," or "negative" limitations in relation to the listing of claim elements.

[0177] Without the use of such exclusive terms, the term “comprising” in the claims associated with this disclosure may allow for the inclusion of any additional elements, regardless of whether a given number of elements are enumerated in such claims or whether the addition of a feature can be considered to transform the nature of the elements described in those claims. Except as specifically defined herein, all technical and scientific terms used herein should be provided in the generally understood sense as much as possible while maintaining the validity of the claims.

[0178] The scope of the invention is not limited to the examples provided and / or the specification, but rather is limited only by the scope of the claim wording associated with the disclosure.

Claims

Claim 1 A user-wearable display device comprising: a frame configured to be mounted on a user; an augmented reality display attached to the frame and configured to orient images toward the user’s eyes; a variable optical material undergoing physical and / or chemical changes in response to a stimulus; a source configured to provide said stimulus; and a processing electronic device, wherein the processing electronic device stores one or more locations of ambient light sources in a three-dimensional physical space of a real-world environment surrounding the user, determines the location of the user within said real-world environment, and is configured to trigger said source to provide said stimulus to the variable optical material to cause physical and / or chemical changes of said material based on the location of the user in said real-world environment such that at least one of the intensity of the ambient light, the spectral content of the ambient light, or the direction of the ambient light is changed, said stimulus is provided based on the location of the user relative to said one or more locations of said ambient light sources. Claim 2 A user-wearable display device according to claim 1, wherein the device is further configured to change the stimulus based on the time of day. Claim 3 A user-wearable display device according to claim 1, wherein the augmented reality display comprises a waveguide, and the waveguide is configured to form images by directing light out of the waveguide and toward the user's eyes so as to show the real-world environment surrounding the user through the waveguide. Claim 4 A user-wearable display device according to claim 1, further comprising one or more sensors configured to acquire information regarding the ambient light conditions of the environment surrounding the user, and the processing electronic device further configured to trigger the source to provide the stimulus to the variable optical material based on the information acquired by the sensors. Claim 5 In paragraph 4, the sensor comprises at least one of a light sensor, a global positioning sub-system, or an environment sensor, in a user-wearable display device. Claim 6 A user-wearable display device according to claim 1, wherein the source comprises an optical source, an electric source, a thermal source, or an acoustic / ultrasonic source. Claim 7 A user-wearable display device according to claim 1, wherein the variable optical material is embedded in or disposed on the surface of the display. Claim 8 A user-wearable display device according to claim 1, wherein the variable optical material comprises one or more of organic or inorganic compounds, electroactive proteins, molecules that exhibit a change in size or shape in response to the stimulus, molecules that move, rotate, twist, or shift in response to the stimulus, molecules that move together and / or attach together in response to the stimulus, molecules that move away from each other in response to the stimulus, or molecules that form nanostructures in response to the stimulus. Claim 9 A user-wearable display device according to claim 1, wherein the display comprises a first ocular region corresponding to the user's first eye and a second ocular region corresponding to the user's second eye, and the processing electronic device is configured to trigger the source to provide the stimulus to a portion of the display to cause a physical change and / or chemical change of the material such that at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light through the first ocular region is changed differently compared to the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light through the second ocular region. Claim 10 A user-wearable display device according to claim 1, wherein the processing electronic device is configured to trigger the source to provide the stimulus to the display to cause a physical change and / or chemical change of the material such that the attenuation of the intensity of ambient light transmitted through a first part of the display is greater than the attenuation of the intensity of ambient light transmitted through a second part of the display. Claim 11 A user-wearable display device according to claim 1, further comprising a head pose sensor, wherein the processing electronic device is configured to trigger the source to transmit the stimulus to the variable optical material based on the movement or position of the user's head determined by the head pose sensor. Claim 12 A user-wearable display device according to claim 1, wherein the processing electronic device is further configured to adjust the position of the display to which the stimulus is applied based on feedback from the user. Claim 13 A user-wearable display device according to claim 1, wherein the processing electronic device is further configured to adjust the size of the area of ​​the display to which the stimulus is applied based on feedback from the user. Claim 14 A user-wearable display device according to claim 1, wherein the processing electronic device is further configured to adjust the amount by which at least one of the intensity of ambient light, the spectral content of ambient light, or the direction of ambient light is changed based on feedback from the user.

Images