Active dimming control using parallax camera images

Abstract

Active dimming control using parallax camera images is disclosed. A waveguide display includes a waveguide, a projector configured to project images (e.g., images to be displayed to a user and / or parallax test images) onto the waveguide, an input coupler configured to couple the images into the waveguide, and one or more output couplers located on the waveguide. At least one of the one or more output couplers is configured to direct the projected images toward an eye of the user. At least one of the one or more output couplers is configured to direct the projected images toward a parallax camera. The waveguide display includes an active dimming layer on a side of the waveguide opposite the user's eye. The active dimming layer is electrically controllable to attenuate ambient light that passes through the waveguide to the user's eye based on an effect of the ambient light on visibility of the projected images as measured using the parallax camera.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. nonprovisional patent application No. 18 / 990,580, filed on December 20, 2024. Technical Field

[0002] This disclosure generally relates to near-eye display systems. Background Technology

[0003] Extended reality (XR) systems, such as head-mounted display (HMD) or heads-up display (HUD) systems, typically include a near-eye display (e.g., in the form of a headset or a pair of glasses) that presents content to the user via an electronic or optical display, for example, within approximately 12 mm to 20 mm in front of the user's eyes. In virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, the near-eye display can display virtual objects or combine images of real objects with virtual objects. For example, in an AR system, a user can view images of virtual objects (e.g., computer-generated images (CGI)) and images of the surrounding environment, for example, through transparent display glasses or lenses (often called optical see-through).

[0004] An example of an optical see-through AR system may include a waveguide-based optical display, in which light projecting an image can be coupled into a waveguide (e.g., a transparent substrate), propagate within the waveguide, and be coupled out of the waveguide at multiple locations. In some implementations, the light projecting the image may be coupled into or out of the waveguide using a semi-transparent mirror or diffractive optics (e.g., a surface relief grating or a volume Bragg grating). Light from the surrounding environment can pass through the see-through area of ​​the waveguide and reach the user's eyes. Summary of the Invention

[0005] This disclosure generally relates to near-eye display systems. More specifically, the technology disclosed herein relates to optically perceptual near-eye display systems, which include an active dimming layer controlled based on images acquired by a parallax camera. Various inventive embodiments, including devices, components, systems, modules, subsystems, and methods, are described herein.

[0006] According to some embodiments, a near-eye display system may include: a projector configured to generate a display image; a waveguide display including a waveguide and configured to reproduce the display image; an active dimming layer located on the side of the waveguide opposite to the user's eye; a camera configured to receive both a reproduced image of the display image and an image of the surrounding environment seen through the active dimming layer and the waveguide; and a controller configured to control the projector and the active dimming layer based on the reproduced image of the display image and the image of the surrounding environment from the camera. The waveguide display may include an output coupler configured to couple the reproduced image of the display image out from a peripheral region of the waveguide. The active dimming layer may be configured to attenuate ambient light in response to a control signal.

[0007] According to some embodiments, a near-eye display system may include a first waveguide display subsystem comprising: a first projector configured to generate a first display image; a first waveguide display including a first waveguide and configured to reproduce the first display image; and a first active dimming layer located on the side of the first waveguide opposite to the first eye of a user of the near-eye display system. The first waveguide display may include a first output coupler configured to couple a reproduced image of the first display image from a peripheral region of the first waveguide. The first active dimming layer may be configured to variably attenuate ambient light. The near-eye display system may also include a second waveguide display subsystem comprising: a second projector configured to generate a second display image; a second waveguide display including a second waveguide and configured to reproduce the second display image; and a second active dimming layer located on the side of the second waveguide opposite to the second eye of a user of the near-eye display system. The second waveguide display includes a second output coupler configured to couple a copy of the second displayed image from a peripheral region of the second waveguide. The second active dimming layer can be configured to variably attenuate ambient light. The near-eye display system may further include: a camera configured to receive at least a copy of the first displayed image and an image of the surrounding environment seen through the first active dimming layer and the first waveguide; and a controller configured to control at least the first projector and the first active dimming layer based on the copy of the first displayed image and the image of the surrounding environment.

[0008] According to some aspects, a method may include: a controller of a near-eye display system: receiving one or more images from at least one parallax camera of the near-eye display system, the one or more images including a display image and an image of the surrounding environment seen through an active dimming layer; determining a brightness level of an area of ​​the image of the surrounding environment that overlaps with the display image; and generating a control signal based on the determined brightness level to control the dimming level of the active dimming layer.

[0009] This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood through reference to suitable portions of this disclosure, any or all of the accompanying drawings, and each claim. The foregoing and other features and examples will be described in more detail below in the description, claims, and drawings. Attached Figure Description

[0010] The illustrative embodiments are described in detail below with reference to the accompanying drawings.

[0011] Figure 1 This is a simplified block diagram of an example extended reality system environment including a near-eye display according to certain embodiments.

[0012] Figure 2 This is a stereoscopic view of an example of a near-eye display in the form of a head-mounted display (HMD) device used to implement some of the examples disclosed herein.

[0013] Figure 3 This is a stereoscopic view of an example of a near-eye display in the form of glasses, used to implement some of the examples disclosed herein.

[0014] Figure 4 This is a cross-sectional view of an example of a near-eye display according to certain embodiments.

[0015] Figure 5 An example of an optical perspective augmented reality system including a waveguide display, according to certain embodiments, is shown.

[0016] Figure 6 An example of an optical perspective augmented reality system, including a waveguide display for outgoing pupil expansion, is shown according to certain embodiments.

[0017] Figure 7 An example of an optical perceptual near-eye display system, including a waveguide display and an active dimming layer for each eye, is shown according to certain embodiments.

[0018] Figure 8 An example of a waveguide display including one or more parallax gratings according to certain embodiments is shown.

[0019] Figure 9 An example of a waveguide display system including an active dimming layer and a parallax camera according to certain embodiments is shown.

[0020] Figure 10 An example of a waveguide display system including an active dimming layer and a parallax camera according to certain embodiments is shown.

[0021] Figure 11 An example of a near-eye display system according to certain embodiments is shown, the near-eye display system including a waveguide structure for coupling parallax test light and ambient light from two pupil replica waveguides to a common parallax camera for binocular parallax sensing and active dimming.

[0022] Figure 12 Examples of images captured by a parallax camera of a near-eye display system according to certain embodiments are shown.

[0023] Figure 13 The flowchart includes an example of a method for dynamically controlling the active dimming layer of a near-eye display system according to certain embodiments.

[0024] Figure 14 This is a simplified block diagram of an example electronic system for implementing some of the examples of near-eye displays disclosed herein.

[0025] These accompanying drawings depict embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the illustrated structures and methods may be employed without departing from the principles of the present disclosure or the benefits claimed herein.

[0026] In the accompanying drawings, similar parts and / or features may have the same reference numerals. Furthermore, various parts of the same type can be distinguished by a ligature following the reference numeral and a second reference numeral to differentiate multiple similar parts. If only the first reference numeral is used in the specification, the specification applies to any of the similar parts having the same first reference numeral, regardless of the second reference numeral. Detailed Implementation

[0027] This disclosure generally relates to near-eye display systems. More specifically, the technology disclosed herein relates to optically perceptual near-eye display systems, which include an active dimming layer controlled based on images acquired by a parallax camera. Various inventive embodiments, including devices, components, systems, modules, subsystems, and methods, are described herein.

[0028] Augmented reality (AR), virtual reality (VR), mixed reality (MR), and other extended reality applications can use near-eye display systems to present images of virtual and / or real objects to a user's eyes. Near-eye display systems typically include an image source (e.g., a display panel) located close to the user's eyes and capable of generating images for the user to view. Near-eye display systems may also include display optics configured to project the image generated by the image source to create a virtual image that appears distant from the image source and only a few centimeters from the user's eyes. Image sources for near-eye display systems can include, for example, liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, micro-OLED displays, inorganic light-emitting diode (ILED) displays, micro-LED displays, active-matrix OLED (AMOLED) displays, and transparent OLED (TOLED) displays. The display optics of a near-eye display system may include, for example, lenses (e.g., pancake lenses) and / or optical combiners, such as waveguide combiners, partial reflector combiners, prism birdbath combiners, and free-space birdbath combiners.

[0029] For example, in some near-eye display systems, waveguide displays can be used to present images to a user. A waveguide display may include a light source for each of the user's two eyes, a waveguide, and an optical coupler formed on or within the waveguide. The light source may include one or more projectors for projecting two-dimensional color images and may be placed in the peripheral area of ​​the waveguide display so as not to obstruct the user's field of view. The optical coupler may include one or more input couplers configured to couple display light into the waveguide, allowing the display light to propagate within the waveguide via total internal reflection. The waveguide may be transparent to visible light and may guide the display light within the waveguide while also allowing ambient light to pass through. The optical coupler may also include one or more output couplers configured to couple display light out of the waveguide at multiple locations, allowing the display light to be directed to the eyebox of the near-eye display system to fill the eyebox.

[0030] Near-eye display systems are typically expected to have small size, low weight, large field of view, large eye-fitting frame, high power efficiency, high brightness, high resolution, high refresh rate, low cost, long battery life, and long lifespan. Achieving these characteristics in near-eye display systems can be challenging. For example, near-eye display systems are usually worn on the user's head (called head-mounted displays), so weight constraints may be more stringent than for other battery-powered portable electronic devices (e.g., mobile phones, tablets, or laptops). Therefore, near-eye display systems may be limited in the amount of power available to the display device, and low power consumption is desired to improve battery life and / or reduce the weight of the near-eye display system. However, the illumination source of a near-eye display system may have limited power efficiency, for example, due to the limited coupling efficiency of the optical couplers used to couple light from the image source into and out of the waveguide, and the display optics may also have limited optical efficiency. Therefore, display projectors (e.g., display panels) in near-eye display systems may typically need to have high power consumption in order to deliver bright, high-resolution, and high-refresh-rate images with a wide color gamut and a large field of view (FOV), thereby improving the immersive experience when using head-mounted displays. Achieving high output image brightness and low power consumption in near-eye display systems can be challenging.

[0031] The power consumption of a near-eye display system can depend on several factors, including the brightness and color gamut of the displayed image. Reducing the brightness of the displayed image can significantly reduce the power consumption of the display system. Therefore, one way to reduce the power consumption of a near-eye display system is to globally reduce the brightness of each pixel. Reducing the brightness of pixels can also increase their lifespan (e.g., reduce pixel burn-in), thereby increasing the overall lifespan of the near-eye display system. However, from the perspective of contrast and image quality, reducing the overall brightness of pixels can negatively impact the user experience. In particular, when using a near-eye display system outdoors in sunlight, the brightness of the displayed image may need to be high enough compared to the bright sunlight in the surrounding environment so that the user can view the displayed image against a bright background. In some near-eye display systems, the display projector may not be able to provide the high output power and brightness required for outdoor use due to the limited maximum output power of the illumination source. Even if the display projector can provide the required high output power, the near-eye display system may be unsafe for the user's eyes, especially when the user's pupils are dilated. Furthermore, the power efficiency and battery life of a near-eye display system may be lower.

[0032] In some near-eye display systems, an active dimming layer can be used to dim ambient light, allowing the brightness of the image displayed from the image source to be reduced to lower power consumption, while maintaining a sufficiently high relative brightness (e.g., contrast) between the displayed image and the ambient light for comfortable viewing even in the presence of ambient light. It may be necessary to dim the active dimming layer based on the brightness of the surrounding environment to achieve the desired contrast between the displayed image and the surrounding environment, thus correctly rendering augmented reality scenes. For example, when the ambient light is bright, the active dimming layer can attenuate the ambient light more. The active dimming layer can be located on the side of the waveguide display opposite the user's eyes (e.g., between the surrounding environment and the waveguide display), allowing it to dim ambient light but not the image displayed to the user's eyes by the waveguide display.

[0033] To automatically and dynamically control the dimming of the active dimming layer, near-eye display systems may need to measure the brightness or intensity of ambient light and adjust the transparency of the active dimming layer accordingly. Some near-eye display systems can use a front-facing Simultaneous Localization and Mapping (SLAM) or Point-of-View (POV) camera for ambient light measurement. However, adaptive dimming using ambient light measured by a front-facing camera can present several challenges. For example, since bright areas of the surrounding environment behind the displayed image may have the greatest impact on the user's viewing experience, while bright areas outside the displayed image area may have little or no impact, it may be necessary to register (e.g., map) the image of the surrounding environment captured by the front-facing camera into the display coordinate space, and vice versa, to determine the bright areas of the surrounding environment behind the displayed image and the appropriate attenuation of ambient light. Furthermore, collecting images that can identify faces and locations raises privacy concerns. Augmented reality devices can have privacy warning lights to indicate the active front-facing camera. Enabling a high-resolution front-facing camera in all dimming scenarios may not be socially acceptable. Furthermore, the voltage level used to control the active dimming layer to attenuate ambient light by a certain amount (e.g., a certain percentage) may change over time, for example, due to aging and degradation caused by ultraviolet (UV) radiation. Measuring the brightness of the ambient light passing through the active dimming layer may be beneficial and desirable for better estimating and controlling the amount of ambient light that passes through the active dimming layer and reaches the waveguide display.

[0034] In some near-eye display systems, misalignment (e.g., relative tilt, slant, and rotation) between the waveguide and the light source (e.g., including the display panel and projector) or relative movement of the components including the waveguide and light source for one eye relative to another component for the other eye can cause the displayed images presented to the user's left and right eyes to differ from the desired two-dimensional (2D) display image. Binocular displays are designed to display objects with appropriate binocular parallax so that the user's brain can extract appropriate depth information from the two 2D images acquired by both eyes through stereoscopic vision. Therefore, users may see two images of the same object, as well as other undesirable optical artifacts, such as brightness and / or color mismatches or inhomogeneities, which can degrade the user experience of near-eye display systems.

[0035] In some near-eye display systems, to reduce or otherwise mitigate unwanted optical artifacts caused by the aforementioned misalignment or relative movement, and to facilitate dynamic uniformity correction and de-distortion of the display, one or more parallax couplers (on one or both sides of the waveguide) can be used on the nose side of the waveguide for each eye to couple some light from the light source (e.g., light displaying an image or a parallax test light pattern) from the nose side of the waveguide to the outside of the waveguide for monitoring the displayed image. One or two parallax sensors can detect parallax test light from two waveguides for both eyes to determine the alignment error between the left and right images in a binocular near-eye display system. The display panel (e.g., a liquid crystal display panel, an organic light-emitting diode display, or a light-emitting diode display) can be adjusted to compensate for misalignment between the waveguide and the light source in the components of the binocular display system, to compensate for relative movement between the two components of the binocular display system, and / or brightness and color non-uniformity between the two components.

[0036] According to some embodiments, a near-eye display system may include a projector, a waveguide display, an active dimming layer located between the waveguide display and the surrounding environment, and a parallax camera (e.g., located on the side of the waveguide display opposite the active dimming layer) for acquiring images of the displayed image and the surrounding environment as seen through the active dimming layer and the waveguide display. The active dimming layer can be dynamically controlled to attenuate ambient light based on measured ambient brightness. The parallax camera can be controlled to measure ambient light passing through the active dimming layer and the waveguide display during periods when the display is inactive between frames, or it can monitor the displayed image and simultaneously measure ambient light passing through the active dimming layer and the waveguide display. In one example, a controller can control the projector to periodically stop displaying an image, and when the projector is not displaying an image, the parallax camera can acquire images of the surrounding environment to determine the brightness of the surrounding environment. The controller can also control the parallax camera to acquire images of the displayed image and the surrounding environment in the same image frame to determine areas of the surrounding environment that may overlap with the displayed image in the user's field of vision. The measured intensity of ambient light in the overlapping region and passing through the active dimming layer and waveguide display can be used to control the dimming of the active dimming layer.

[0037] In some examples, the near-eye display system may include two parallax cameras, one for capturing ambient and display light for the waveguide display for the left eye, and the other for capturing ambient and display light for the waveguide display for the right eye. In some examples, the near-eye display system may include another light combiner that directs the display and ambient light from both the waveguide displays for the left and right eyes to the same parallax camera. In one example, the light combiner may include a smaller waveguide including an input coupler and an output coupler formed thereon or therein. In another example, the light combiner may include multiple reflectors (e.g., mirrors or prisms).

[0038] In this way, one or two parallax cameras can be used to monitor the displayed image and measure ambient light passing through the active dimming layer and waveguide display. Additional cameras to measure ambient light may not be necessary. Because parallax cameras have low effective resolution, the resulting images can be processed by a local sensor processor that cannot make the acquired images usable for storage, thus mitigating potential privacy concerns.

[0039] Furthermore, the parallax camera behind the active dimming layer and the waveguide display can measure the intensity of ambient light that has been attenuated by the active dimming layer. This allows any variation in the active dimming layer's response to electrical control signals to be taken into account by the active dimming layer's controller, compensating for such variations. Therefore, the dimming level of the active dimming layer can be controlled more precisely to allow the appropriate amount of ambient light to pass through and mix with the display light. Additionally, since the same camera (e.g., within the same image frame) captures images of the surrounding environment and the display image, it may not be necessary to map or align the image of the surrounding environment to the display image, or vice versa, which can further reduce the power consumption of the near-eye display system.

[0040] In the following description, specific details are set forth for illustrative purposes to provide a thorough understanding of the various examples of this disclosure. However, it will be apparent, however, that various examples may be practiced without these specific details. For example, devices, systems, structures, components, methods, and other parts may be shown as parts in block diagram form to avoid obscuring these examples with unnecessary detail. In other instances, well-known devices, processes, systems, structures, and techniques may be shown without necessary detail to avoid obscuring the examples. These figures and descriptions are not intended to be limiting. The terms and expressions used in this disclosure are used as descriptive terms rather than limiting terms, and the use of these terms and expressions is not intended to exclude any equivalents or portions thereof of the features shown and described. The word “example” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as an “example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0041] Figure 1 A simplified block diagram of an example of an extended reality system environment 100 including a near-eye display 120 according to certain embodiments. Figure 1 The extended reality system environment 100 shown may include a near-eye display 120, an optional external imaging device 150, and an optional input / output interface 140, each of which can be coupled to an optional console 110. Although Figure 1An example of an extended reality system environment 100 is shown, including a near-eye display 120, an external imaging device 150, and an input / output interface 140. However, the extended reality system environment 100 may include any number of these components, or any of these components may be omitted. For example, multiple near-eye displays 120 may be present, monitored by one or more external imaging devices 150 communicating with a console 110. In some configurations, the extended reality system environment 100 may not include the external imaging device 150, the optional input / output interface 140, and the optional console 110. In alternative configurations, different or additional components may be included in the extended reality system environment 100.

[0042] The near-eye display 120 may be a head-mounted display that presents content to a user. Examples of content presented by the near-eye display 120 include one or more of the following: images; video; audio; or any combination thereof. In some embodiments, audio may be presented via an external device (e.g., speakers and / or headphones) that receives audio information from the near-eye display 120, the console 110, or both, and presents audio data based on that audio information. The near-eye display 120 may include one or more rigid bodies that may be rigidly or non-rigidly coupled to each other. Rigid coupling between multiple rigid bodies allows these coupled rigid bodies to function as a single rigid entity. Non-rigid coupling between multiple rigid bodies allows these rigid bodies to move relative to each other. In various embodiments, the near-eye display 120 may be implemented in any suitable shape element, including a pair of glasses. The following describes... Figure 2 and Figure 3 Some embodiments of the near-eye display 120 are further described. Furthermore, in various embodiments, the functionality described herein can be used in head-mounted devices that combine images of the environment outside the near-eye display 120 with augmented reality content (e.g., computer-generated images). Thus, the near-eye display 120 can utilize generated content (e.g., images, videos, sounds, etc.) to enhance images of the physically real-world environment outside the near-eye display 120 to present augmented reality to the user.

[0043] In various embodiments, the near-eye display 120 may include one or more of display electronics 122, display optics 124, and eye-tracking unit 130. In some embodiments, the near-eye display 120 may also include one or more locators 126, one or more position sensors 128, and an inertial measurement unit (IMU) 132. The near-eye display 120 may omit any of the eye-tracking unit 130, locator 126, position sensor 128, and IMU 132, or in various embodiments, the near-eye display may include additional elements. Furthermore, in some embodiments, the near-eye display 120 may include a combination of... Figure 1 Multiple elements describing the functions of various components.

[0044] Display electronics 122 can display images to a user or facilitate the display of images to a user based on data received from, for example, console 110. In various embodiments, display electronics 122 may include one or more display panels, such as liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, inorganic light-emitting diode (ILED) displays, micro light-emitting diode (μLED) displays, active-matrix OLED (AMOLED) displays, transparent OLED (TOLED) displays, or some other type of display. For example, in one embodiment of the near-eye display 120, display electronics 122 may include a front TOLED panel, a rear display panel, and optical components (e.g., attenuators, polarizers, or diffraction or spectral coatings) located between the front TOLED panel and the rear display panel. Display electronics 122 may include multiple pixels that emit light of dominant colors such as red, green, blue, white, or yellow. In some embodiments, display electronics 122 can display three-dimensional (3D) images through the stereoscopic effect produced by a two-dimensional panel to create a subjective perception of image depth. For example, display electronics 122 may include a left display positioned in front of the user's left eye and a right display positioned in front of the user's right eye. The left and right displays may present horizontally shifted copies of the image relative to each other to create a stereoscopic effect (i.e., the user's perception of image depth when viewing the image).

[0045] In some embodiments, display optics 124 may display image content optically (e.g., using optical waveguides and couplers), or amplify received image light from display electronics 122, correct optical errors associated with the image light, and present corrected image light to a user of near-eye display 120. In various embodiments, display optics 124 may include one or more optical elements, such as a substrate, optical waveguide, aperture, Fresnel lens, convex lens, concave lens, filter, input / output coupler, or any other suitable optical element that can influence the image light emitted from display electronics 122. Display optics 124 may include combinations of different optical elements and mechanical couplings for maintaining the relative spacing and orientation of the optical elements in the combination. One or more optical elements in display optics 124 may have optical coatings, such as anti-reflective coatings, reflective coatings, filtering coatings, or combinations of different optical coatings.

[0046] The amplification of image light by the display optics 124 allows the display electronics 122 to be physically smaller, lighter, and consume less power compared to a larger display. Additionally, the amplification can increase the field of view of the displayed content. The amplification of image light by the display optics 124 can be varied by adjusting, adding, or removing optical elements. In some embodiments, the display optics 124 can project a display image onto one or more image planes that are further away from the user's eye than the near-eye display 120.

[0047] Display optics 124 can also be designed to correct one or more types of optical errors, such as two-dimensional optical errors, three-dimensional optical errors, or any combination thereof. Two-dimensional errors can include optical aberrations occurring in two dimensions. Example types of two-dimensional errors can include barrel distortion, pincushion distortion, longitudinal chromatic aberration, and lateral chromatic aberration. Three-dimensional errors can include optical errors occurring in three dimensions. Example types of three-dimensional errors can include spherical aberration, coma, field curvature, and astigmatism.

[0048] Multiple locators 126 may be objects located at specific positions on the near-eye display 120 that are correlated with each other and associated with a reference point on the near-eye display 120. In some embodiments, the console 110 may identify locators 126 in an image acquired by an external imaging device 150 to determine the position, orientation, or both of the extended reality head-mounted device. Locators 126 may be light-emitting diodes (LEDs), corner cube reflectors, reflective markers, a type of light source that contrasts with the environment in which the near-eye display 120 operates, or any combination thereof. In embodiments where locators 126 are active components (e.g., LEDs or other types of light-emitting devices), locators 126 may emit light in the visible light band (e.g., about 380 nm to 750 nm), light in the infrared (IR) band (e.g., about 750 nm to 1 mm), light in the ultraviolet band (e.g., about 12 nm to about 380 nm), or light in another portion of the electromagnetic spectrum, or light in any combination of multiple portions of the electromagnetic spectrum.

[0049] The external imaging device 150 may include one or more cameras, one or more video cameras, any other device capable of acquiring images including one or more of the plurality of positioners 126, or any combination thereof. Additionally, the external imaging device 150 may include one or more filters (e.g., to improve the signal-to-noise ratio). The external imaging device 150 may be configured to detect light emitted or reflected from or from the positioners 126 in the field of view of the external imaging device 150. In embodiments where the positioners 126 include passive elements (e.g., retroreflectors), the external imaging device 150 may include a light source illuminating some or all of the plurality of positioners 126, which may reflect light back to the light source in the external imaging device 150. Slow calibration data from the external imaging device 150 may be transmitted to the console 110, and the external imaging device 150 may receive one or more calibration parameters from the console 110 to adjust one or more imaging parameters (e.g., focal length, focus, frame rate, sensor temperature, shutter speed, aperture, etc.).

[0050] Position sensor 128 can generate one or more measurement signals in response to movement of near-eye display 120. Examples of position sensor 128 may include accelerometers, gyroscopes, magnetometers, other motion detection sensors or error correction sensors, or any combination thereof. For example, in some embodiments, position sensor 128 may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). In some embodiments, the various position sensors may be orthogonally oriented to each other.

[0051] IMU 132 can be an electronic device that generates rapid calibration data based on received measurement signals from one or more of a plurality of position sensors 128. The position sensors 128 can be located external to IMU 132, internal to IMU 132, or any combination thereof. Based on one or more measurement signals from one or more position sensors 128, IMU 132 can generate rapid calibration data indicating an estimated position of the near-eye display 120 relative to an initial position of the near-eye display 120. For example, IMU 132 can integrate received measurement signals from accelerometers over time to estimate a velocity vector, and integrate that velocity vector over time to determine an estimated position of a reference point on the near-eye display 120. Alternatively, IMU 132 can provide sampled measurement signals to console 110, which can determine the rapid calibration data. While a reference point can generally be defined as a point in space, in various embodiments, a reference point can also be defined as a point within the near-eye display 120 (e.g., the center of IMU 132).

[0052] Eye-tracking unit 130 may include one or more eye-tracking systems. Eye tracking may refer to determining the position of the eyes relative to near-eye display 120, including pupil position and gaze angle. The eye-tracking system may include an imaging system for imaging one or more eyes, and may optionally include a light emitter that generates light directed towards the eyes so that the light reflected from the eyes can be captured by the imaging system. For example, eye-tracking unit 130 may include an incoherent or coherent light source (e.g., a laser diode) and a camera that emits light in the visible or infrared spectrum, and that captures the light reflected from the user's eyes. As another example, eye-tracking unit 130 may capture reflected radio waves emitted by a miniature radar unit. Eye-tracking unit 130 may use multiple low-power light emitters that emit light at frequencies and intensities that will not harm the eyes or cause physical discomfort. The eye-tracking unit 130 can be configured to increase the contrast of the eye image acquired by the eye-tracking unit 130 while reducing the total power consumed by the eye-tracking unit 130 (e.g., reducing the power consumed by the light emitter and imaging system included in the eye-tracking unit 130).

[0053] The near-eye display 120 can use the eye-tracking unit 130 to, for example, determine the user's inter-pupillary distance (IPD), determine the gaze direction, collect heuristic information about user interactions in VR media (e.g., the time spent on any particular subject, object, or frame based on experienced stimuli), some other function based in part on the orientation of at least one of the user's eyes, or any combination thereof. Because the orientation of the user's eyes can be determined, the eye-tracking unit 130 can be able to determine where the user is looking. For example, determining the user's gaze direction can include determining a convergence point based on the determined orientation of the user's left and right eyes. The convergence point can be the point where the two foveal axes of the user's eyes intersect. The direction of the user's gaze can be determined by measuring the direction of a line passing through the convergence point and the midpoint between the user's pupils.

[0054] Input / output interface 140 may be a device that allows a user to send action requests to console 110. An action request may be a request to perform a specific action. For example, an action request may launch or terminate an application, or perform a specific action within an application. Input / output interface 140 may include one or more input devices. Example input devices may include a keyboard, mouse, game controller, gloves, buttons, touchscreen, or any other suitable device for receiving action requests and transmitting the received action requests to console 110. Action requests received by input / output interface 140 may be transmitted to console 110, which may perform an action corresponding to the requested action. In some embodiments, input / output interface 140 may provide haptic feedback to the user based on instructions received from console 110. For example, input / output interface 140 may provide haptic feedback when an action request is received, or when console 110 has performed the requested action and transmitted instructions to input / output interface 140. In some embodiments, the external imaging device 150 may be used to track the input / output interface 140, such as the location or position of a controller (which may include, for example, an IR light source), or the location or position of a user's hand, to determine the user's actions. In some embodiments, the near-eye display 120 may include one or more imaging devices to track the input / output interface 140, such as the location or position of a controller, or the location or position of a user's hand, to determine the user's actions.

[0055] The console 110 can provide content to the near-eye display 120 for presentation to the user based on information received from one or more of the external imaging device 150, near-eye display 120, and input / output interface 140. Figure 1 In the example shown, console 110 may include an application repository 112, a head-mounted device tracking subsystem 114, an extended reality engine 116, and an eye-tracking subsystem 118. Some embodiments of console 110 may include combinations with... Figure 1 The modules described are different modules or, in addition to combinations Figure 1 Additional modules beyond those described herein. The functionality further described below can be distributed among the components of console 110 in a manner different from that described herein.

[0056] In some embodiments, console 110 may include a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor. The processor may include multiple processing units that execute the instructions in parallel. The non-transitory computer-readable storage medium may be any memory, such as a hard disk drive, removable memory, solid-state drive (e.g., flash memory), or dynamic random access memory (DRAM). In various embodiments, in combination with... Figure 1 The device or subsystem of the described console 110 may be encoded as instructions in a non-transitory computer-readable storage medium, which, when executed by a processor, cause the processor to perform the functions described further below.

[0057] Application repository 112 may store one or more applications for execution by console 110. An application may include a set of instructions that, when executed by the processor, generate content to be presented to the user. The content generated by the application may be in response to input received from the user via eye movements or in response to input received from input / output interface 140. Examples of applications may include game applications, conferencing applications, video playback applications, or other suitable applications.

[0058] The head-mounted device tracking subsystem 114 can use slow calibration information from an external imaging device 150 to track the movement of the near-eye display 120. For example, the head-mounted device tracking subsystem 114 can use the observed locator from the slow calibration information and a model of the near-eye display 120 to determine the position of a reference point for the near-eye display 120. The head-mounted device tracking subsystem 114 can also use position information from fast calibration information to determine the position of the reference point for the near-eye display 120. Additionally, in some embodiments, the head-mounted device tracking subsystem 114 can use multiple components of fast calibration information, slow calibration information, or any combination thereof to predict the future position of the near-eye display 120. The head-mounted device tracking subsystem 114 can provide the estimated or predicted future position of the near-eye display 120 to the extended reality engine 116.

[0059] The Extended Reality Engine 116 can execute applications within the Extended Reality System Environment 100 and receive position information, acceleration information, velocity information, predicted future position of the near-eye display 120, or any combination thereof from the head-mounted device tracking subsystem 114. The Extended Reality Engine 116 can also receive estimated pupil position and gaze direction information from the eye-tracking subsystem 118. Based on the received information, the Extended Reality Engine 116 can determine the content to be presented to the near-eye display 120 for the user. For example, if the received information indicates that the user has looked to the left, the Extended Reality Engine 116 can generate content for the near-eye display 120 that reflects the user's eye movements in the virtual environment. Furthermore, the Extended Reality Engine 116 can respond to a received action request from the input / output interface 140, execute an in-application action on the console 110, and provide feedback to the user indicating that the action has been performed. The feedback can be visual or auditory feedback via the near-eye display 120, or tactile feedback via the input / output interface 140.

[0060] The eye-tracking subsystem 118 can receive eye-tracking data from the eye-tracking unit 130 and determine the position of the user's eyes based on the eye-tracking data. The position of the eyes can include the orientation, position, or both of the eyes relative to the near-eye display 120 or any element of the near-eye display. Because the axis of rotation of the eye changes with the position of the eye in its socket, determining the position of the eye in its socket allows the eye-tracking subsystem 118 to more accurately determine the orientation of the eyes.

[0061] Figure 2 This is a perspective view of an example of a near-eye display in the form of an HMD device 200 for implementing some of the examples disclosed herein. The HMD device 200 may be part of, for example, a VR system, an AR system, a MR system, or any combination thereof. The HMD device 200 may include a body 220 and a headband 230. Figure 2 The bottom side 223, front side 225, and left side 227 of the body 220 are shown in a perspective view. The headband 230 may have an adjustable or extendable length. Sufficient space may exist between the body 220 of the HMD device 200 and the headband 230 to allow a user to wear the HMD device 200 on their head. In various embodiments, the HMD device 200 may include additional, fewer, or different components. For example, in some embodiments, the HMD device 200 may include, for example, the following Figure 3 The eyeglass temple and templetip shown are not the headband 230.

[0062] HMD device 200 can present media to a user, including virtual and / or augmented views of a physically real-world environment with computer-generated elements. Examples of media presented by HMD device 200 may include images (e.g., two-dimensional (2D) images or three-dimensional (3D) images), videos (e.g., 2D videos or 3D videos), audio, or any combination thereof. Images and videos may be displayed through one or more display components encapsulated in the body 220 of HMD device 200. Figure 2 (Not shown) Presented to each of the user's eyes. In various embodiments, one or more display components may include a single electronic display panel or multiple electronic display panels (e.g., one display panel for each of the user's eyes). Examples of one or more electronic display panels may include, for example, an LCD display, an OLED display, an ILED display, a µLED display, an AMOLED display, a TOLED display, some other display, or any combination thereof. The HMD device 200 may include two eye-fitting frame regions.

[0063] In some embodiments, the HMD device 200 may include various sensors (not shown), such as depth sensors, motion sensors, position sensors, and eye-tracking sensors. Some of these sensors may use structured light patterns for sensing. In some embodiments, the HMD device 200 may include an input / output interface for communicating with a console. In some embodiments, the HMD device 200 may include a virtual reality engine (not shown) that can execute applications within the HMD device 200 and receive depth information, position information, acceleration information, velocity information, predicted future position, or any combination thereof from various sensors of the HMD device 200. In some embodiments, the information received by the virtual reality engine may be used to generate signals (e.g., display instructions) to one or more display components. In some embodiments, the HMD device 200 may include locators (not shown, such as locator 126) located at fixed positions on the body 220 relative to each other and relative to a reference point. Each of these locators may emit light detectable by an external imaging device.

[0064] Figure 3 This is a perspective view of an example of a near-eye display 300 in the form of eyeglasses, used to implement some of the examples disclosed herein. The near-eye display 300 may be... Figure 1This describes a specific embodiment of the near-eye display 120, which can be configured as a virtual reality display, an augmented reality display, and / or a mixed reality display. The near-eye display 300 may include a frame 305 and a display 310. The display 310 may be configured to present content to a user. In some embodiments, the display 310 may include display electronics and / or display optics. For example, as described above regarding... Figure 1 As described in the near-eye display 120, the display 310 may include an LCD display panel, an LED display panel, or an optical display panel (e.g., a waveguide display assembly).

[0065] The near-eye display 300 may also include various sensors 350a, 350b, 350c, 350d, and 350e located on or within the frame 305. In some embodiments, sensors 350a to 350e may include one or more depth sensors, one or more motion sensors, one or more position sensors, one or more inertial sensors, or one or more ambient light sensors. In some embodiments, sensors 350a to 350e may include one or more image sensors configured to generate image data representing different fields of view in different directions. In some embodiments, sensors 350a to 350e may be used as input devices to control or influence the display content of the near-eye display 300 and / or provide an interactive VR / AR / MR experience to the user of the near-eye display 300. In some embodiments, sensors 350a to 350e may also be used for stereoscopic imaging.

[0066] In some embodiments, the near-eye display 300 may further include one or more illuminators 330 to project light into a physical environment. The projected light may be associated with different frequency bands (e.g., visible light, infrared light, ultraviolet light, etc.) and may be used for various purposes. For example, one or more illuminators 330 may project light into a dark environment (or an environment with low intensity infrared light, ultraviolet light, etc.) to help sensors 350a to 350e acquire images of different objects in a dark environment. In some embodiments, one or more illuminators 330 may be used to project certain light patterns onto objects within the environment. In some embodiments, one or more illuminators 330 may function as locators, such as those described above. Figure 1 The locator 126 is described.

[0067] In some embodiments, the near-eye display 300 may further include a high-resolution camera 340. The high-resolution camera 340 can capture images of the physical environment within the field of view. The captured images can be generated, for example, by a virtual reality engine (e.g., Figure 1The extended reality engine 1) is used to process the image to add virtual objects to the captured image or modify the physical objects in the captured image, and the processed image can be displayed to the user by the display 310 for use in AR or MR applications.

[0068] Figure 4 This is a cross-sectional view of an example of a near-eye display 400 according to certain embodiments. The near-eye display 400 may include at least one display component 410. The display component 410 may be configured to direct image light (e.g., display light) to an eye-fitting frame located at an exit pupil 420 and to the user's eye 490. It is worth noting that, although for illustrative purposes, Figure 4 The other accompanying drawings in this disclosure show the eye of a user of a near-eye display, but the user's eye is not part of the corresponding near-eye display.

[0069] Similar to HMD device 200 and near-eye display 300, near-eye display 400 may include a frame 405 and a display assembly 410, which may include a display 412 coupled to or embedded in the frame 405 and / or display optics 414. As described above, display 412 may display images to a user electrically (e.g., using LCD, LED, OLED) or optically (e.g., using waveguide displays and optical couplers) based on data received from a processing unit such as console 110. In some embodiments, display 412 may include a display panel comprising pixels made of LCD, LED, and OLED. Display 412 may include multiple pixels emitting light of dominant colors such as red, green, blue, white, or yellow. In some embodiments, display assembly 410 may include a stack of one or more waveguide displays, including but not limited to stacked waveguide displays and zoom waveguide displays. A stacked waveguide display is a multicolor display (e.g., a red-green-blue (RGB) display) created by stacking multiple waveguide displays (each with a different monochromatic source).

[0070] Display optics 414 may be similar to display optics 124 and may display image content optically (e.g., using optical waveguides and optical couplers), correct optical errors associated with image light, combine images of virtual and real objects, and present the corrected image light to the exit pupil 420 of the near-eye display 400, at which the user's eye 490 may be located. In some embodiments, display optics 414 may also relay images to create virtual images that appear to be far from the display 412 and only a few centimeters away from the user's eye. For example, display optics 414 may collimate an image source to create virtual images that may appear to be far away (e.g., greater than about 0.3 m, such as about 0.5 m, 1 m, or 3 m away) and convert spatial information of the displayed virtual objects into angular information. In some embodiments, display optics 414 may also magnify the source image to make the image appear larger than the actual size of the source image. Further details of the display 412 and display optics 414 will be described below.

[0071] Figure 5An example of an optical perspective augmented reality system 500 including a waveguide display according to certain embodiments is shown. The augmented reality system 500 may include a projector 510 and a combiner 515. The projector 510 may include a light source or image source 512 and projector optics 514. In some embodiments, the light source or image source 512 may include one or more microLED devices as described above. In some embodiments, the image source 512 may include multiple pixels displaying virtual objects, such as an LCD display panel or an LED display panel. In some embodiments, the image source 512 may include a light source that generates coherent or partially coherent light. For example, the image source 512 may include a laser diode, a vertical-cavity surface-emitting laser, an LED, and / or a microLED as described above. In some embodiments, the image source 512 may include multiple light sources (e.g., the microLED array described above), each emitting monochromatic image light corresponding to a primary color (e.g., red, green, or blue). In some embodiments, the image source 512 may include three two-dimensional microLED arrays, wherein each two-dimensional microLED array may include a microLED configured to emit primary color light (e.g., red, green, or blue). In some embodiments, image source 512 may include an optical pattern generator, such as a spatial light modulator. Projector optics 514 may include one or more optical components that can modulate light from image source 512, such as expanding, collimating, scanning, or projecting light from image source 512 onto combiner 515. One or more optical components may include, for example, one or more lenses, one or more liquid lenses, one or more mirrors, one or more apertures, and / or one or more gratings. For example, in some embodiments, image source 512 may include one or more one-dimensional micro-LED arrays or elongated two-dimensional micro-LED arrays, and projector optics 514 may include one or more one-dimensional scanners (e.g., micromirrors or prisms) configured to scan the one-dimensional micro-LED array or elongated two-dimensional micro-LED array to generate image frames. In some embodiments, projector optics 514 may include a liquid lens (e.g., a liquid crystal lens) having multiple electrodes that allow scanning of light from image source 512.

[0072] Combiner 515 may include an input coupler 530 for coupling light from projector 510 to substrate 520 of combiner 515. Input coupler 530 may include a volume holographic grating, a diffractive optical element (DOE) (e.g., a surface relief grating), a tilted surface of substrate 520, or a refractive coupler (e.g., a wedge or prism). For example, input coupler 530 may include a reflective or transmissive volume Bragg grating. Input coupler 530 may have high coupling efficiency for visible light. Light coupled to substrate 520 may propagate within substrate 520 via, for example, total internal reflection (TIR). Substrate 520 may be in the form of a lens of an eyeglass. The substrate 520 may be planar or curved and may comprise one or more types of dielectric materials, such as glass, quartz, plastic, polymer, poly(methyl methacrylate) (PMMA), crystal, silicon nitride (SiN), silicon carbide (SiC), or ceramic. The thickness of the substrate may range, for example, from less than about 1 mm to about 12 mm or greater. The substrate 520 may be transparent to visible light.

[0073] The substrate 520 may include or be coupled to a plurality of output couplers 540, each output coupler being configured to extract at least a portion of light guided by and propagating within the substrate 520 and to direct the extracted light 560 to an eye-fitting frame 595, wherein the user's eye 590 of the augmented reality system 500 may be located when the augmented reality system 500 is used. The plurality of output couplers 540 may replicate the outgoing pupil to increase the size of the eye-fitting frame 595, allowing the displayed image to be visible over a larger area. Like the input coupler 530, the output couplers 540 may include grating couplers (e.g., volume holographic gratings or surface relief gratings), other diffractive optical elements, prisms, etc. For example, the output coupler 540 may include a reflective volume Bragg grating or a transmissive volume Bragg grating. The output couplers 540 may have different coupling (e.g., diffraction) efficiencies at different locations. The substrate 520 may also allow light 550 from the environment in front of the combiner 515 to pass through with little or no loss. The output coupler 540 can also allow light 550 to pass through with almost no loss. For example, in some embodiments, the output coupler 540 can have a very low diffraction efficiency for light 550, allowing light 550 to be refracted or otherwise pass through the output coupler 540 with very little loss, thus achieving a higher intensity than the extracted light 560. In some embodiments, the output coupler 540 can have a high diffraction efficiency for light 550 and can diffract light 550 with almost no loss along certain desired directions (i.e., diffraction angles). Therefore, the user can be able to view a combined image of the environment in front of the combiner 515 and an image of a virtual object projected by the projector 510.

[0074] In some embodiments, the projector 510, input coupler 530, and output coupler 540 may be located on either side of the substrate 520. The input coupler 530 and output coupler 540 may be reflective gratings (also known as reflective gratings) or transmissive gratings (also known as transmissive gratings) to couple display light into or out of the substrate 520.

[0075] Figure 6 An example of an optical perspective augmented reality system 600, according to certain embodiments, including a waveguide display for expanding the exit pupil, is shown. Augmented reality system 600 may be similar to augmented reality system 500 and may include a waveguide display and a projector, the projector including a light source or image source 610 and projector optics 620. The waveguide display may include a substrate 630, an input coupler 640, and a plurality of output couplers 650, as described above with respect to augmented reality system 600. Although Figure 6 Only the propagation of light from a single field of view is shown, but Figure 6 The propagation of light from multiple fields of view is shown.

[0076] Figure 6 The diagram illustrates how an exit pupil is replicated by an output coupler 650 to form a converged exit pupil or eye-fitting frame. Within this frame, different regions of the field of view (e.g., different pixels on image source 610) can be associated with different corresponding propagation directions toward the eye-fitting frame, and light from the same field of view (e.g., the same pixel on image source 610) can have the same propagation direction for different individual exit pupils. Therefore, a single image of image source 610 can be formed regardless of the user's eye position within the eye-fitting frame, where light from different individual exit pupils propagating in the same direction can originate from the same pixel on image source 610 and be focused onto the same location on the user's retina. In other words, the user's eye can convert angular information in the eye-fitting frame or exit pupil (e.g., corresponding to the Fourier plane) into spatial information in the form of an image on the retina. Figure 6 This demonstrates that even when the user's eye moves to different positions within the eye-fitting frame, the user's eye can still see the image from the image source.

[0077] Figure 5 and Figure 6 The light projecting the image is shown to be coupled into a waveguide (e.g., a transparent substrate), propagating within the waveguide via total internal reflection, and coupled out of the waveguide at multiple locations to replicate the exit pupil and expand the eye-fitting frame in one dimension. In some embodiments, multiple waveguides and / or multiple couplers (e.g., gratings or semi-transparent mirrors) can be used to replicate the exit pupil in two dimensions to fill a large eye-fitting frame with a 2D array of pupils, thereby expanding the eye-fitting frame so that the user's eye can view the displayed image even if the user's eye moves over a large area. For example, two or more gratings can be used to expand the display light in two dimensions or along two axes. The two gratings can have different grating parameters, such that one grating can be used to replicate the exit pupil in one direction while the other grating can be used to replicate the exit pupil in another direction.

[0078] As mentioned above, near-eye display systems are generally expected to have small size, low weight, large field of view, large eye-fitting frame, high power efficiency, high brightness, high resolution, high refresh rate, low cost, long battery life, and long lifespan. Achieving these characteristics in near-eye display systems can be challenging. For example, near-eye display systems are typically worn on the user's head (called head-mounted displays), so weight constraints may be more stringent than for other battery-powered portable electronic devices (e.g., mobile phones or touchpads). Therefore, near-eye display systems may be limited in terms of the amount of power available to the display device, and it is desirable for near-eye display systems to have low power consumption to improve battery life and / or reduce the weight of the near-eye display system. However, the image source of a near-eye display system may have limited power efficiency, and the display optics may also have limited optical efficiency, for example, due to the limited coupling efficiency of the optical couplers used to couple light from the image source into and out of the waveguide. Therefore, the image source (e.g., the display panel) of a near-eye display system may typically need to have high power consumption in order to provide bright, high-resolution, high-refresh-rate images with a large color gamut and a large field of view (FOV), thereby enhancing the immersive experience when using head-mounted displays. Achieving high output image brightness and low power consumption in near-eye display systems can be challenging.

[0079] The power consumption of a near-eye display system can depend on several factors, including the brightness of the displayed image. Reducing the brightness of the displayed image can significantly reduce the power consumption of the display system. Therefore, one way to reduce the power consumption of a near-eye display system is to globally reduce the brightness of each pixel. Reducing the brightness of pixels can also increase their lifespan (e.g., reduce pixel burn-in), thereby increasing the overall lifespan of the near-eye display system. However, from an image quality perspective, reducing the overall brightness of pixels can negatively impact the user experience. In particular, when using a near-eye display system outdoors in sunlight, the brightness of the displayed image may need to be high enough compared to the bright sunlight in the surrounding environment so that the user can view the displayed image against a bright ambient background. In some near-eye display systems, the image source may not be able to provide the high output power and brightness required for outdoor use due to the limited maximum output power of the image source. Even if the image source can provide the required high output power, the near-eye display system may still be unsafe for the user's eyes, especially when the user's pupils are dilated. Furthermore, the electron-to-photon conversion efficiency of LED illuminators decreases with increasing current, thus gradually affecting the battery life of the near-eye display system.

[0080] In some near-eye display systems, an active dimming layer can be used to dim ambient light, allowing the brightness of the image displayed by the image source to be reduced to lower power consumption, while maintaining a sufficiently high relative brightness (e.g., contrast) between the displayed image and the ambient light for comfortable viewing of content superimposed on the dimmed ambient light image. The active dimming layer may need to dim according to the brightness of the surrounding environment to achieve the desired contrast between the displayed image and the surrounding environment in augmented or mixed reality images. For example, when the ambient light is bright, the active dimming layer can attenuate the ambient light more. The active dimming layer can be located on the side of the waveguide display opposite the user's eyes (e.g., between the surrounding environment and the waveguide display), allowing it to dim ambient light but not the image displayed to the user's eyes by the waveguide display.

[0081] Figure 7 An example of an optical perspective near-eye display system 700 according to certain embodiments is shown. The near-eye display system 700 includes a waveguide display and an active dimming layer for each eye. In the illustrated example, the near-eye display system 700 may be in the form of a pair of glasses, which may include a frame 705, a left waveguide display 720, and a right waveguide display (…). Figure 7 (Not shown in the image). A first group of one or more projectors 710 may be at least partially embedded in the left temple of frame 705, and a second group of one or more projectors may be at least partially embedded in the right temple of frame 705. Waveguide display 720 may include a group of one or more input couplers and a group of one or more output couplers (e.g., a flat or curved glass substrate, such as a lens) formed on or in a waveguide. The group of one or more input couplers may couple display light from one or more projectors 710 into the waveguide, and the group of one or more output couplers may replicate the display light in one or both directions and guide the display light to the eye-fit frame to fill the eye-fit frame, as described above.

[0082] The near-eye display system 700 may further include an active dimming layer 730 located between the waveguide display 720 and the surrounding environment (e.g., on the side of the waveguide display 720 opposite to the user's eyes), such that ambient light can be attenuated by the active dimming layer 730 before it combines (e.g., mixes or superimposes) with display light through the waveguide display 720, which can be used as a light combiner. The active dimming layer 730 can have different transparency or opacity when different voltage signal levels are applied to it. For example, if the intensity of the ambient light is high, the active dimming layer 730 can be controlled to reduce its transmittance, allowing the ambient light to attenuate more before combining with the display light, thus not affecting the user's viewing of the displayed image. The active dimming layer 730 may include, for example, an electrochromic material (e.g., including tungsten trioxide (WO3)), a photochromic material, a liquid crystal (LC) layer, or a material having suspended nanoscale particles. In one example, the active dimming layer 730 may include a liquid crystal material layer that can be adjusted by applying an electric field to change the orientation of LC molecules, thereby altering the transmittance of the active dimming layer 730 to ambient light. The LC-based active dimming layer can be implemented using, for example, polymer-dispersed liquid crystal (PDLC) dimming devices, guest-host liquid crystal (GHLC) dimming devices, or polymer-stabilized cholesteric texture (PSCT) liquid crystal dimming devices.

[0083] To automatically control the dimming of the active dimming layer 730, the near-eye display system 700 may need to measure the brightness or intensity of ambient light and adjust the transparency of the active dimming layer 730 accordingly. Some near-eye display systems may use a front-facing simultaneous localization and mapping (SLAM) or point-of-view (POV) camera for ambient light measurement. For example, one or more of sensors 350b, 350c, 350d, and 350e may be a front-facing camera used to acquire images of the surrounding environment to determine the brightness of areas of the surrounding environment. However, using ambient light brightness measured by a front-facing camera for adaptive dimming may present several challenges. For example, since bright areas of the surrounding environment behind the displayed image may have the greatest impact on the user's viewing of the displayed image, while bright areas outside the displayed image area may have little or no impact on the user's viewing of the displayed image, it may be necessary to register (e.g., map) the image of the surrounding environment acquired by the front-facing camera into the display coordinate space, and vice versa, in order to determine the impact of bright areas of the surrounding environment behind the displayed image and the appropriate amount of attenuation of ambient light. Furthermore, enabling the front-facing camera might require turning on a privacy light, which could draw attention and may not be socially acceptable in all situations where dimming is necessary. Additionally, due to aging and degradation of the dimming layer, such as from UV radiation, the voltage level used to control the active dimming layer to attenuate ambient light by a certain amount (e.g., a certain percentage) may change over time. Measuring the brightness of the ambient light passing through the active dimming layer is likely beneficial and desirable for better estimating and controlling the amount of ambient light reaching the waveguide display.

[0084] Furthermore, misalignment (e.g., relative tilt, tilt, and rotation) between the waveguide display (e.g., the substrate of waveguide display 720) and the light source (e.g., one or more projectors 710), and misalignment or relative movement of components of the waveguide display for the left eye relative to another component of the waveguide display for the right eye, may cause the displayed images presented to the user's left and right eyes to differ from the desired two-dimensional (2D) display images: images that show objects with correct binocular parallax, allowing the user's brain to extract appropriate depth information from two 2D images captured by both eyes through stereoscopic vision. Therefore, the user may see two images of the same object, as well as other undesirable optical artifacts, such as brightness and / or color mismatch or inhomogeneity, which also degrades the user experience when using the near-eye display system. For example, due to misalignment of the display components, non-uniformity of output power between the projectors for the left and right eyes, and non-uniformity of optical efficiency between the waveguide displays for the left and right eyes, the displayed image for the first eye and the displayed image for the second eye may have different brightness and / or different colors.

[0085] In some implementations, to reduce or otherwise mitigate undesirable optical artifacts caused by the aforementioned misalignment or relative movement, and to facilitate dynamic uniformity correction and dedistortion of the display, one or more parallax couplers (on one or both sides of the waveguide) can be used on the nose side of the waveguide for each eye to couple some light from the light source (e.g., light from the displayed image or a parallax test light pattern) away from the waveguide on the nose side for monitoring the displayed image. One or two parallax sensors can detect light from two waveguides for parallax testing for both eyes to determine the visual axis and alignment errors between the left and right images in a binocular near-eye display system, such that one or more projectors and / or display panels (e.g., liquid crystal display panels, organic light-emitting diode displays, or light-emitting diode displays) can be adjusted to compensate for misalignment between the waveguide and the light source, relative movement between the two components of the binocular display system, and / or brightness and color nonuniformity between the two components in the binocular display system.

[0086] Figure 8 An example of a waveguide display 800 according to certain embodiments, including one or more parallax couplers (e.g., gratings), is shown. The waveguide display 800 can be an example of the waveguide display 720 described above. The waveguide display 800 may include a substrate 810, one or more input gratings, a first output grating 830, a second output grating 840, and one or more parallax gratings 850. Figure 8 In the example shown, one or more input gratings may include three input gratings for coupling three different colors of display light and / or parallax test light. For example, input grating 820 may be used to couple red light to substrate 810, input grating 822 may be used to couple green light to substrate 810, and input grating 824 may be used to couple blue light to substrate 810.

[0087] Light coupled into substrate 810 via one or more input gratings can be guided by total internal reflection to propagate within substrate 810 in a first direction and can reach first output grating 830. First output grating 830 can redirect display light and / or parallax test light at multiple locations along the first direction to a second direction toward second output grating 840. Second output grating 840 can couple multiple portions of display light and / or parallax test light from substrate 810 at multiple locations along the first and second directions to generate a two-dimensional exit pupil array to fill the eye-fitting frame as described above. At least a portion of the light used for parallax testing can reach one or more parallax gratings 850. One or more parallax gratings 850 may include parallax gratings having the same or similar grating vectors as second output grating 840, and thus can at least partially couple a portion of the light used for parallax testing from substrate 810 toward the parallax camera.

[0088] According to certain embodiments disclosed herein, a near-eye display system may include a projector, a waveguide display, an active dimming layer between the waveguide display and the surrounding environment, and a parallax camera (e.g., located on the side of the waveguide display opposite the active dimming layer) for acquiring images of the displayed image and the surrounding environment as seen through the active dimming layer and the waveguide display. The active dimming layer can be dynamically controlled to attenuate ambient light based on measured ambient brightness. The parallax camera can be controlled to measure ambient light passing through the active dimming layer and the waveguide display over a period of time, or to measure the displayed image superimposed on ambient light passing through both the active dimming layer and the waveguide display simultaneously over a period of time. In one example, a controller may control the projector to periodically stop displaying an image or parallax light pattern (e.g., during blanking periods), and when the projector is not displaying an image or parallax light pattern, the parallax camera may acquire images of the surrounding environment to determine the brightness of the surrounding environment. The controller acquires images of the displayed image and the surrounding environment in the same image frame to determine areas of the surrounding environment that may overlap with the displayed image in the user's field of vision. The intensity of ambient light measured in the overlapping area by the active dimming layer and the waveguide display can be used to control the dimming of the active dimming layer.

[0089] Figure 9 An example of a waveguide display system 900, including an active dimming layer 930 and a parallax camera, according to certain embodiments is shown. The waveguide display system 900 may be a subsystem of a near-eye display system 700 and may be used for one eye in a binocular display system. In the illustrated example, the waveguide display system 900 may include a projector comprising an image source 910 and projector optics 912. The image source 910 may include a display panel (e.g., the LCD, OLED, or micro-LED-based display panel described above) and may be controlled by a controller 950 to generate an image to be displayed to a user and / or to generate a display test light pattern for parallax detection and correction. The projector optics 912 may collimate the light emitted by the image source 910 from the displayed image and guide the display light to a waveguide 920 to illuminate the entrance pupil of the waveguide display system 900. In some embodiments, the projector may alternatively or additionally generate a parallax test light pattern and project the parallax test light pattern onto the waveguide 920. In some embodiments, the parallax test light pattern may include infrared light that is invisible to the user's eye and therefore may not affect the user's viewing of the displayed image.

[0090] Waveguide 920 may include a substrate transparent to visible light, such as a glass substrate (e.g., a planar or curved lens). Input coupler 922 may be formed on or within waveguide 920 and may receive display light from a projector. Input coupler 922 may include, for example, a grating coupler (e.g., a holographic grating coupler or a surface relief grating coupler) or multiple transflective mirrors. Input coupler 922 may couple display light and / or parallax test light into waveguide 920. Display light coupled into waveguide 920 may be guided by waveguide 920 by, for example, total internal reflection at multiple surfaces of waveguide 920. During propagation within waveguide 920, multiple portions of the display light may be coupled out of waveguide 920 toward an eye-fitting frame via multiple output couplers 924. In the example shown, output couplers 924 may include transflective mirrors, and waveguide 920 may be referred to as a geometric waveguide. At least a portion of the display light or parallax test light can be coupled out of the waveguide 920 via one or more parallax output couplers 926 (e.g., one or more semi-transparent mirrors) located in the peripheral region of the waveguide 920. This portion of the display light or parallax test light coupled out of the waveguide 920 via the one or more parallax output couplers 926 can be collected by a parallax camera, which may include camera optics 942 and an image sensor 940. The display camera can be electrically coupled to a controller 950 or another controller coupled to the controller 950, such that the controller 950 can receive the image acquired by the parallax camera and adjust the display image based on the acquired image for parallax correction.

[0091] like Figure 9As shown, the waveguide display system 900 may include an active dimming layer 930 located on the side of the waveguide 920 opposite to the user's eye and the parallax camera. An air gap (e.g., formed by a spacer) may exist between the active dimming layer 930 and the waveguide 920, allowing light coupled into the waveguide 920 to be reflected at the interface between the waveguide 920 and the air via total internal reflection. The active dimming layer 930 may include, for example, an electrochromic active dimming layer, a photochromic active dimming layer, a liquid crystal (LC)-based active dimming layer, or an active dimming layer comprising suspended nanoscale particles. The active dimming layer 930 may be controlled by a controller 950 to alter the attenuation of ambient light 990 (e.g., sunlight) by the active dimming layer 930. Ambient light passing through the active dimming layer 930 can be combined with display light via the waveguide 920, allowing the ambient light to be superimposed on the display light to form an augmented reality or mixed reality image. The ambient light can pass through the waveguide 920 and is then partially collected by the parallax camera. In some examples, ambient light can pass through waveguide 920 and be partially coupled into and guided by waveguide 920 to parallax output coupler 926. Parallax output coupler 926 can couple the ambient light out of waveguide 920 and guide it to parallax camera. Therefore, parallax camera can also acquire images of the surrounding environment. The acquired images of the surrounding environment can be sent to controller 950, which can determine the brightness or intensity of the ambient light and generate appropriate control signals to control the dimming of active dimming layer 930.

[0092] In this way, one or two parallax cameras can be used to monitor the parallax between the left and right display images and measure the ambient light passing through the active dimming layer and the waveguide display, eliminating the need for additional cameras to measure ambient light. Since the parallax camera images are processed locally by the controller 950 and are not available externally to the device via wired or wireless access, privacy issues are effectively addressed. Furthermore, the parallax camera behind the active dimming layer and the waveguide display can measure the intensity of ambient light attenuated by the active dimming layer, allowing any changes in the active dimming layer's response to electrical control signals to be taken into account by the controller of the active dimming layer to compensate for these variations. Therefore, the dimming level of the active dimming layer can be controlled more precisely to allow the appropriate amount of ambient light to pass through and mix with the display light. Moreover, since the same camera (e.g., in the same image frame) is used to capture images of the surrounding environment and the display image, it may not be necessary to map or align the image of the surrounding environment to the display image and vice versa, which can further reduce the power consumption of the near-eye display system.

[0093] In some examples, the near-eye display system may include two parallax cameras, one for capturing ambient and display light from the waveguide display for the left eye, and the other for capturing ambient and display light from the waveguide display for the right eye. In some examples, the near-eye display system may include another light combiner that directs the display and ambient light from the waveguide displays for the left and right eyes to the same parallax camera. In one example, the light combiner may include a smaller waveguide including an input coupler and an output coupler formed thereon or therein. In another example, the display and ambient light from the waveguide displays for the left and right eyes may be directed to the parallax camera via a reflector assembly. For example, the reflector assembly may include a pair of prisms for each of the left and right sides of the near-eye display system. In another example, the reflector assembly may include a pair of mirrors located on each of the left and right sides of the near-eye display system, respectively.

[0094] Figure 10 An example of a waveguide display system 1000, including an active dimming layer 1040 and a parallax camera 1030, is shown according to certain embodiments. The waveguide display system 1000 may be a subsystem of a binocular near-eye display system. In the illustrated example, the waveguide display system 1000 may include a projector 1010 that can provide display light and / or parallax test light 1016 to a waveguide 1020. The parallax test light 1016 may carry a specific parallax test pattern, such as a structured light pattern (e.g., a stripe pattern or a dot array). In one example, the projector 1010 may include a display panel 1012 and projector optics 1014. The display panel 1012 may include, for example, an LCD, OLED, or micro-LED display panel. In some embodiments, the projector 1010 may be of a different type, such as a scanning projector including one or more lasers and a scanner. In some examples, projector 1010 may be a dual-function projector that can provide a parallax test pattern for tilt / parallax sensing and also provide a display image for presentation to the user's eye 1090. In some embodiments, projector 1010 may be used only to provide a parallax test pattern for waveguide tilt / binocular parallax sensing, and another projector may be used to provide a display image for presentation to the user's eye 1090. In some embodiments, the parallax test pattern may be made invisible to the user, for example, by providing the parallax test pattern in invisible light (e.g., near-infrared (NIR) light), presenting the parallax test pattern outside the display's field of view (FOV), or presenting the parallax test pattern when the user blinks.

[0095] The parallax test pattern and / or display light can be coupled into the waveguide 1020 via an input coupler 1022 (e.g., a grating coupler) located at the first end of the waveguide 1020. The waveguide 1020 can be a one-dimensional or two-dimensional pupil replication waveguide as described above, and the parallax test light 1016 and / or display light can be guided by a series of total internal reflections (TIR) ​​at opposite surfaces of the waveguide 1020. The waveguide 1020 may include at least one output coupler 1024 (e.g., a diffraction grating) for outwardly coupling multiple portions 1018 of the display light to the user's eye 1090. The waveguide 1020 may also include at least one output coupler 1026 at its end for outwardly coupling at least a portion of the parallax test light or display light to the parallax camera 1030. The output coupler 1026 may be part of the output coupler 1024, or it may be separate from the output coupler 1024, but it may have the same grating vector as the output coupler 1024. As described above, the input coupler 1022, the output coupler 1024, and the output coupler 1026 may include, for example, holographic gratings, surface relief gratings, semi-transparent mirrors, and polarizer gratings.

[0096] The parallax camera 1030 of the waveguide display system 1000 can be configured to receive parallax test light and / or display light from the waveguide 1020. In the example shown, the parallax camera 1030 may include a focusing element 1032 (e.g., a lens) that can focus the parallax test light and / or display light from the output coupler 1026 to form a copy of the parallax test pattern and / or display image on the image sensor 1034 of the parallax camera 1030. The parallax camera 1030 may include a two-dimensional (2D) pixel array and can generate an output signal representing the image.

[0097] The waveguide display system 1000 may include a controller 1038 coupled to a projector 1010 and an image sensor 1034. The controller 1038 may be configured to cause the projector 1010 to provide display light and / or parallax test light, receive output signals from the image sensor 1034, and determine tilt or other misalignment of the waveguide 1020 relative to the projector 1010 based on the output signals generated by the image sensor 1034. For example, tilt or other misalignment may be determined based on the position and / or orientation of the displayed image and / or the parallax test light pattern in the image acquired by the image sensor 1034. In some examples, the controller 1038 may also receive output signals from the image sensor of the waveguide display system for the user's other eye and determine the parallax between the two waveguide display systems for the user's eyes, such as misalignment between the two waveguide display systems, differences in brightness and / or color of the displayed image, and / or the parallax test light patterns output by the two waveguide display systems. Then, the controller 1038 can control the projectors of the two waveguide display systems to compensate for parallax by, for example, changing the field of view, orientation, and / or brightness of the displayed images generated by the image sources of one or both projectors.

[0098] The waveguide display system 1000 may include an active dimming layer 1040 located on the side of the waveguide 1020 opposite to the user's eye and parallax camera 1030. An air gap (e.g., formed by a spacer) may exist between the active dimming layer 1040 and the waveguide 1020, allowing light coupled into the waveguide to be reflected at the interface between the waveguide 1020 and the air via total internal reflection. The active dimming layer 1040 may include, for example, an electrochromic active dimming layer, a photochromic active dimming layer, a liquid crystal-based active dimming layer, or an active dimming layer comprising suspended nanoscale particles. The active dimming layer 1040 may be controlled by a controller 1038 to alter the attenuation of ambient light (e.g., sunlight). Ambient light passing through the active dimming layer 1040 may be combined with display light through the waveguide 1020, allowing the display light to be superimposed on the ambient light to form an augmented reality or mixed reality image. Ambient light passing through waveguide 1020 can be partially collected by parallax camera 1030. In some examples, ambient light passing through active dimming layer 1040 can be partially coupled into waveguide 1020 and guided by waveguide 1020 to output coupler 1026, which can couple the ambient light out of waveguide 1020 and guide it to parallax camera 1030. Therefore, parallax camera 1030 can also acquire images of the surrounding environment. The acquired images of the surrounding environment can be sent to controller 1038, which can determine the brightness or intensity of the ambient light and generate appropriate control signals to control the dimming of active dimming layer 1040. For example, as described above, controller 1038 can determine the average, maximum, and / or minimum brightness of the area of ​​the surrounding environment that overlaps with the display image in the image acquired by parallax camera 1030, and control the dimming of active dimming layer 1040 based on the determined average, maximum, and / or minimum brightness.

[0099] Figure 11 An example of a near-eye display system 1100 according to certain embodiments is shown. The near-eye display system 1100 includes a waveguide structure 1130 for coupling parallax test light and ambient light from two pupil-replicated waveguide displays to a common parallax camera for binocular parallax sensing and active dimming. Figure 11 In the example shown, the near-eye display system 1100 may include a first pupil-replicating waveguide display 1110, a second pupil-replicating waveguide display 1120, a waveguide structure 1130, and a parallax camera including a focusing element 1140 and an image sensor 1150. Although Figure 11The parallax camera is shown on the same side of the waveguide structure 1130 as the first pupil replica waveguide display 1110 and the second pupil replica waveguide display 1120. However, in some other examples, the parallax camera may be on the side of the waveguide structure 1130 opposite to the first pupil replica waveguide display 1110 and the second pupil replica waveguide display 1120.

[0100] The first pupil replication waveguide display 1110 may be similar to the waveguide display system 1000 described above, and may include a projector (not shown), a waveguide display, and an active dimming layer 1160. The projector may be similar to the projector 1010 described above. The waveguide display may include a waveguide 1112 (e.g., a substrate transparent to visible light), an input coupler 1114 for coupling display light and / or parallax test light into the waveguide 1112, one or more output couplers 1116 configured to couple multiple portions of the display light from the waveguide 1112 to an eye-fitting frame for the user's left eye, and one or more parallax couplers 1118 configured to couple the parallax test light (or display light) from the waveguide 1112. The parallax test light (or display light) coupled from waveguide 1112 can be coupled into the waveguide of waveguide structure 1130 via input coupler 1132, and coupled out of waveguide structure 1130 towards focusing element 1140 and image sensor 1150 via output coupler 1136. Waveguide structure 1130 may be a one-dimensional waveguide and may guide the coupled parallax test light (or display light) to propagate in one direction (e.g., the x-direction). In some examples, output coupler 1136 may be on the opposite side (e.g., the bottom side) of waveguide structure 1130, and in some other examples, focusing element 1140 and image sensor 1150 may be on the side of waveguide structure 1130 opposite to the first pupil replica waveguide display 1110 and the second pupil replica waveguide display 1120. As described above, the input coupler 1114, output coupler 1116, parallax coupler 1118, input coupler 1132, and output coupler 1136 may include, for example, holographic gratings, surface relief gratings, semi-transparent mirrors, and polarizing gratings.

[0101] The active dimming layer 1160 may be located on the side of the waveguide 1112 opposite to the user's eye. An air gap (e.g., formed by a spacer) may exist between the active dimming layer 1160 and the waveguide 1112, allowing light coupled into the waveguide 1112 to be reflected at the interface between the waveguide 1112 and the air via total internal reflection. The active dimming layer 1160 may include, for example, an electrochromic active dimming layer, a photochromic active dimming layer, a liquid crystal-based active dimming layer, or an active dimming layer comprising suspended nanoscale particles. The active dimming layer 1160 may be controlled by a controller (e.g., controller 1038). Figure 11(Not shown) The active dimming layer 1160 is controlled to change the attenuation of ambient light (e.g., sunlight) by the active dimming layer 1160. Ambient light passing through the active dimming layer 1160 can be combined with display light via waveguide 1112, allowing the ambient light to be superimposed on the display light to form an augmented reality or mixed reality image. Ambient light passing through waveguide 1112 can be partially coupled into waveguide structure 1130. In some examples, ambient light passing through the active dimming layer 1160 can be partially coupled into waveguide 1112 and guided by waveguide 1112 to parallax coupler 1118, which can couple the ambient light out of waveguide 1112 and guide it into waveguide structure 1130. Ambient light can be partially coupled into waveguide structure 1130 via input coupler 1132 and coupled out of waveguide structure 1130 towards focusing element 1140 and image sensor 1150 via output coupler 1136. Therefore, the parallax camera can also acquire images of the surrounding environment. The acquired images of the surrounding environment can be sent to a controller, which can determine the brightness or intensity of the ambient light and generate appropriate control signals to control the dimming of the active dimming layer 1160.

[0102] The second pupil replica waveguide display 1120 may be similar to the first pupil replica waveguide display 1110 described above, and may include a projector (not shown), a waveguide display, and an active dimming layer 1162. The projector may be similar to the projector 1010 described above. The waveguide display may include a waveguide 1122 (e.g., a substrate transparent to visible light), an input coupler 1124 for coupling display light and / or parallax test light into the waveguide 1122, one or more output couplers 1126 configured to couple multiple portions of the display light from the waveguide 1122 to an eye-fitting frame for the user's right eye, and one or more parallax couplers 1128 configured to couple the parallax test light out of the waveguide 1122. The parallax test light coupled out of the waveguide 1122 may be coupled into the waveguide structure 1130 via the input coupler 1134, and coupled out of the waveguide structure 1130 towards the focusing element 1140 and the image sensor 1150 via the output coupler 1136. The input coupler 1124, output coupler 1126, parallax coupler 1128 and input coupler 1132 may include, for example, holographic gratings, surface relief gratings, semi-transparent mirrors and polarizing gratings.

[0103] The active dimming layer 1162 may be located on the side of the waveguide 1122 opposite to the user's eyes. An air gap (e.g., formed by a spacer) may exist between the active dimming layer 1162 and the waveguide 1122, allowing light coupled into the waveguide 1122 to be reflected at the interface between the waveguide 1122 and the air via total internal reflection. The active dimming layer 1162 may be similar to the active dimming layer 1160. The active dimming layer 1162 may be controlled by a controller to alter its attenuation of ambient light (e.g., sunlight). Ambient light passing through the active dimming layer 1162 can be combined with display light via the waveguide 1122, allowing the ambient light to be superimposed on the display light to form an augmented reality or mixed reality image. Ambient light passing through the waveguide 1122 may be partially coupled into the waveguide structure 1130. In some examples, ambient light passing through the active dimming layer 1162 can be partially coupled into waveguide 1122 and guided by waveguide 1122 to parallax coupler 1128, which couples the ambient light out of waveguide 1112 towards waveguide structure 1130. Ambient light can also be partially coupled into waveguide structure 1130 via input coupler 1134 and out of waveguide structure 1130 towards focusing element 1140 and image sensor 1150 via output coupler 1136. Therefore, the parallax camera can also acquire images of the surrounding environment. The acquired images of the surrounding environment can be sent to a controller that determines the brightness or intensity of the ambient light and generates appropriate control signals to control the dimming of the active dimming layer 1162.

[0104] Display light (and / or parallax test light) and ambient light coupled from the second pupil replica waveguide display 1120 and waveguide structure 1130, as well as display light (and / or parallax test light) and ambient light coupled from the first pupil replica waveguide display 1110 and waveguide structure 1130, can be focused by focusing element 1140 to form an image on image sensor 1150. The relative position of the parallax test light pattern (or display image) from the first pupil replica waveguide display 1110 and the second pupil replica waveguide display 1120 can indicate the tilt angle or other misalignment or relative movement of the first pupil replica waveguide display 1110 relative to the second pupil replica waveguide display 1120. When relative tilt, displacement, or other misalignment is detected, the display image generated by the left projector and / or the display image generated by the right projector can be digitally adjusted to compensate for the relative tilt, displacement, or other misalignment. Furthermore, the difference in intensity and / or color between the displayed image (or parallax test light pattern) from the first pupil replica waveguide display 1110 and the displayed image (or parallax test light pattern) from the second pupil replica waveguide display 1120 can be determined based on the image acquired by the parallax camera and can be used to control the projector to compensate for the difference. Additionally, images of the surrounding environment can be acquired and used to determine the brightness of areas of the surrounding environment, thereby controlling the dimming of the active dimming layers 1160 and 1162.

[0105] Figure 12An example of an image 1200 captured by a parallax camera in a near-eye display system (e.g., waveguide display system 900 or 1000, or near-eye display system 1100) according to certain embodiments is shown. Image 1200 includes an image of the surrounding environment (e.g., a parking lot), which may include bright and dark areas. Image 1200 also includes a display image (or parallax test light pattern) comprising a two-dimensional dot array 1210. Because the image captured by the parallax camera includes both an image of the surrounding environment and a display image, the relative position of the display image to the bright areas of the surrounding environment as seen by the user's eye can be determined without transforming or aligning the images of the surrounding environment and the display image. By temporally offsetting or aligning the exposure time of the parallax camera relative to the exposure time of the display (e.g., the time when the projector is on), the parallax camera can capture an image that includes only the image of the surrounding environment and excludes the display image, or an image that includes both the image of the surrounding environment and the display image (or a parallax test light pattern). Based on the brightness of the surrounding environment areas that may overlap with the displayed image in the image captured by the parallax camera, appropriate control signals (e.g., regional or global active dimming control voltage levels) can be generated to control the dimming of the active dimming layer. This allows bright areas of the surrounding environment viewed by the user to be dimmed to a brightness level comparable to that of the displayed image, thereby improving the user experience of viewing the displayed image and reducing the overall power consumption of the near-eye display system.

[0106] Figure 13 This includes a flowchart 1300 illustrating an example of a method for dynamically controlling the active dimming layer of a near-eye display system according to certain embodiments. The various operations described in flowchart 1300 are for illustrative purposes only and are not intended to be limiting. In various embodiments, flowchart 1300 may be modified to add additional operations or omit some operations. The operations described in flowchart 1300 may be performed by, for example, an HMD device 200, a near-eye display 300, a near-eye display system 700, a waveguide display system 900 (e.g., controller 950), a waveguide display system 1000 (e.g., controller 1038), a near-eye display system 1100, or the electronic system 1400 described below.

[0107] Operation at block 1310 may include: receiving one or more images from a camera (e.g., a parallax camera) of the near-eye display system, the one or more images including a display image and an image of the surrounding environment as seen through the active dimming layer and the waveguide display. In some examples, the one or more images may include: a first image including the display image; and a second image including an image of the surrounding environment as seen through the active dimming layer and the waveguide display. In some other examples, the image from the camera may include a display image superimposed on the image of the surrounding environment as seen through the active dimming layer and the waveguide display. Figure 12 Examples of such images are shown. In some examples, the displayed image may include an image to be displayed to a user, a parallax test light pattern including infrared light, or a parallax test light pattern displayed during the user's blink. In some examples, the one or more images may include a display image for the first eye of a user of the near-eye display system and a display image for the second eye of a user of the near-eye display system. The image of the surrounding environment may include bright areas and dark areas. The waveguide display may include: one or more input couplers configured to couple a display image generated by a projector into the waveguide; and one or more output couplers configured to couple the display image out of the waveguide in one or more regions of the waveguide. At least one of the one or more output couplers may be located in the peripheral region of the waveguide and may be configured to couple a copy of the display image from the waveguide to a camera.

[0108] The operation at box 1320 may include determining the brightness level of an area of ​​the image of the surrounding environment that overlaps with the displayed image. For example, the controller may determine the average or maximum brightness level of an area of ​​the image of the surrounding environment that may overlap with the displayed image in one or more received images. The brightness level of this area of ​​the image of the surrounding environment may have been attenuated by an active dimming layer.

[0109] The operation at box 1330 may include generating a control signal based on a determined brightness level to control the dimming level of the active dimming layer. The active dimming layer may include, for example, an electrochromic material, a photochromic material, a liquid crystal layer, or a material comprising suspended nanoscale particles. The active dimming layer may have variable transparency or opacity, which can be controlled by a voltage level applied to the active dimming layer. For example, when the brightness level of that area of ​​the image of the surrounding environment is high, the active dimming layer can be controlled to have low transparency and high attenuation, so that the brightness level of the ambient light seen by the user's eye is comparable to the brightness level of the image displayed to the user's eye, thus the displayed image can have good quality (e.g., contrast) and can be easily viewed by the user.

[0110] Optional operations at box 1340 may include determining the parallax of the near-eye display based on the displayed image in one or more received images. As described above, misalignment in the near-eye display system may be determined based on the position and / or orientation of the displayed image (or parallax test light pattern) in the one or more images. In some examples, the one or more images may include a display image for a first eye of the user of the near-eye display system and a display image for a second eye of the user of the near-eye display system. Due to, for example, misalignment of components of the waveguide display for the left eye and / or for the right eye, non-uniformity of output power of the projector for the left eye and the projector for the right eye, non-uniformity of optical efficiency of the waveguide display for the left eye and the waveguide display for the right eye, the display image for the first eye and the display image for the second eye may be misaligned and / or may have different brightness.

[0111] Optional operations at box 1350 may include controlling at least one projector of the near-eye display system to compensate for parallax in the near-eye display. For example, the controller may control the output power level of the projector, the field of view of the image generated by the projector, or a combination thereof, such that the display image for the first eye and the display image for the second eye can be aligned and have similar brightness.

[0112] Embodiments of the present invention may include, or may combine with, an extended reality system. Extended reality is a form of reality that has been adapted in some way before being presented to a user. This form of reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Extended reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Extended reality content may include video, audio, haptic feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect for the viewer). Additionally, in some examples, extended reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in extended reality and / or otherwise used in extended reality (e.g., to perform activities in extended reality). Extended reality systems that provide extended reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, mobile devices or computing systems, or any other hardware platform capable of providing extended reality content to one or more viewers.

[0113] Figure 14 This is a simplified block diagram of an example electronic system 1400 for implementing some of the examples of the various examples disclosed herein, such as an HMD device. Electronic system 1400 can be used as the electronic system for the HMD device or other near-eye displays described above. In this example, electronic system 1400 may include one or more processors 1410 and memory 1420. The one or more processors 1410 may be configured to execute instructions for performing operations at multiple components and may be, for example, a general-purpose processor or microprocessor suitable for implementation within a portable electronic device. The one or more processors 1410 may be communicatively coupled to multiple components within electronic system 1400. To achieve this communicative coupling, the one or more processors 1410 may communicate with other illustrated components across bus 1440. Bus 1440 may be any subsystem suitable for transmitting data within electronic system 1400. Bus 1440 may include multiple computer buses and additional circuitry for transmitting data.

[0114] Memory 1420 may be coupled to one or more processors 1410. In some embodiments, memory 1420 may provide both short-term and long-term storage and may be divided into several units. Memory 1420 may be volatile, such as static random access memory (SRAM) and / or dynamic random access memory (DRAM), and / or memory 1420 may be non-volatile, such as read-only memory (ROM) and flash memory. Additionally, memory 1420 may include a removable storage device, such as a secure digital (SD) card. Memory 1420 may provide storage for computer-readable instructions, data structures, program modules, and other data for electronic system 1400. In some embodiments, memory 1420 may be distributed across different hardware modules. A set of instructions and / or code may be stored on memory 1420. These instructions may take the form of executable code that can be executed by the electronic system 1400, and / or may take the form of source code and / or installable code. The source code and / or installable code may take the form of executable code when compiled and / or installed on the electronic system 1400 (e.g., using any of the various commonly used compilers, installers, compression / decompression utilities, etc.).

[0115] In some embodiments, memory 1420 may store multiple applications 1422 to 1424, which may include any number of applications. Examples of applications may include game applications, conferencing applications, video playback applications, or other suitable applications. These applications may include depth sensing functionality or eye-tracking functionality. Applications 1422 to 1424 may include specific instructions to be executed by one or more processors 1410. In some embodiments, certain applications or portions of applications 1422 to 1424 may be executable by other hardware subsystems 1480. In some embodiments, memory 1420 may also include secure memory, which may include additional security controls to prevent copying of security information or other unauthorized access to security information.

[0116] In some embodiments, memory 1420 may include an operating system 1425 loaded therein. Operating system 1425 may be operable to initiate and execute instructions provided by applications 1422 to 1424 and / or manage other hardware subsystems 1480 and interact with a wireless communication subsystem 1430, which may include one or more wireless transceivers. Operating system 1425 may be adapted to perform other operations across components of electronic system 1400, including threading, resource management, data storage control, and other similar functions.

[0117] The wireless communication subsystem 1430 may include, for example, infrared communication devices, wireless communication devices and / or chipsets (e.g., Bluetooth® devices, IEEE 802.11 devices, Wi-Fi devices, WiMax devices, cellular communication facilities, etc.) and / or similar communication interfaces. The electronic system 1400 may include one or more antennas 1434 for wireless communication, which may be part of the wireless communication subsystem 1430 or as separate components coupled to any part of the system. Depending on the desired functionality, the wireless communication subsystem 1430 may include separate transceivers for communicating with base transceiver stations and other wireless devices and access points. This may include communication with different data networks and / or network types, such as wireless wide-area networks (WWAN), wireless local area networks (WLAN), or wireless personal area networks (WPAN). A WWAN may be, for example, a WiMax (IEEE 802.16) network. A WLAN may be, for example, an IEEE 802.11x network. WPAN can be, for example, a Bluetooth network, IEEE 802.15x, or some other type of network. The technologies described herein can also be used in any combination of WWAN, WLAN, and / or WPAN. The wireless communication subsystem 1430 can allow the exchange of data with networks, other computer systems, and / or any other devices described herein. The wireless communication subsystem 1430 may include means for transmitting or receiving data (e.g., identifiers of HMD devices, location data, geographic maps, heat maps, photographs, or videos) using one or more antennas 1434 and one or more wireless links 1432.

[0118] Embodiments of the electronic system 1400 may also include one or more sensors 1490. The one or more sensors 1490 may include, for example, an image sensor, accelerometer, pressure sensor, temperature sensor, proximity sensor, magnetometer, gyroscope, inertial sensor (e.g., a subsystem combining an accelerometer and a gyroscope), ambient light sensor, or any other similar device or subsystem operable to provide sensory output and / or receive sensory input (e.g., a depth sensor or position sensor). For example, in some embodiments, the one or more sensors 1490 may include one or more inertial measurement units (IMUs) and / or one or more position sensors. The IMU may generate calibration data indicating an estimated position of the HMD device relative to an initial position of the HMD device based on multiple measurement signals received from one or more of the position sensors. The position sensors may generate one or more measurement signals in response to motion of the HMD device. Examples of position sensors may include, but are not limited to, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting motion, a class of sensors for error correction of the IMU, or some combination thereof. The position sensors may be located external to the IMU, internal to the IMU, or some combination thereof. At least some sensors can use structured light patterns for sensing.

[0119] Electronic system 1400 may include display 1460. Display 1460 may be a near-eye display and may graphically present information from electronic system 1400 to a user, such as images, videos, and various instructions. Such information may be obtained from one or more applications 1422 to 1424, virtual reality engine 1426, one or more other hardware subsystems 1480, combinations thereof, or any other suitable means for resolving graphical content for a user (e.g., via operating system 1425). Display 1460 may use liquid crystal display (LCD) technology, light-emitting diode (LED) technology (including, for example, OLED, ILED, μLED, AMOLED, TOLED, etc.), light-emitting polymer display (LPD) technology, or some other display technology.

[0120] Electronic system 1400 may include user input / output interface 1470. User input / output interface 1470 allows a user to send action requests to electronic system 1400. An action request may be a request to perform a specific action. For example, an action request may be to start or stop an application or to perform a specific action within an application. Input / output interface 1470 may include one or more input devices. Example input devices may include a touchscreen, touchpad, one or more microphones, one or more buttons, one or more knobs, one or more switches, keyboard, mouse, game controller, or any other suitable device for receiving action requests and transmitting the received action requests to electronic system 1400. In some embodiments, user input / output interface 1470 may provide haptic feedback to a user based on instructions received from electronic system 1400. For example, haptic feedback may be provided upon receiving the action request or upon the execution of the action request.

[0121] Electronic system 1400 may include camera 1450, which can be used to capture photos or videos of a user, for example, to track the position of the user's eyes. Camera 1450 may also be used to capture photos or videos of the environment, for example, for VR, AR, or MR applications. Camera 1450 may include, for example, a complementary metal-oxide-semiconductor (CMOS) image sensor with millions or tens of millions of pixels. In some embodiments, camera 1450 may include two or more cameras that can be used to acquire 3D images.

[0122] In some embodiments, the electronic system 1400 may include a plurality of other hardware modules 1480. Each of the other hardware modules 1480 may be a physical module within the electronic system 1400. While each hardware module in each of the other hardware subsystems 1480 may be permanently configured as a structure, some hardware modules in each of the other hardware subsystems 1480 may be temporarily configured to perform a specific function or be temporarily activated. Examples of other hardware subsystems 1480 may include, for example, audio output and / or input interfaces (e.g., microphones or speakers), near field communication (NFC) devices, rechargeable batteries, battery management systems, wired / wireless battery charging systems, etc. In some embodiments, one or more functions of the other hardware subsystems 1480 may be implemented in software.

[0123] In some embodiments, the memory 1420 of the electronic system 1400 may also store a virtual reality engine 1426. The virtual reality engine 1426 can execute applications within the electronic system 1400 and receive position information, acceleration information, velocity information, predicted future position, or combinations thereof from various sensors of the HMD device. In some embodiments, the information received by the virtual reality engine 1426 can be used to generate signals (e.g., display instructions) to the display 1460. For example, if the received information indicates that the user has looked to the left, the virtual reality engine 1426 can generate content for the HMD device reflecting the user's movement in the virtual environment. Furthermore, the virtual reality engine 1426 can perform in-application actions in response to received action requests from the user input / output interface 1470 and provide feedback to the user. The provided feedback may be visual, auditory, or haptic feedback. In some embodiments, one or more processors 1410 may include one or more graphics processing units (GPUs) capable of executing the virtual reality engine 1426.

[0124] In various implementations, the aforementioned hardware and subsystems can be implemented on a single device or on multiple devices that can communicate with each other using wired or wireless connections. For example, in some implementations, components or subsystems, such as the GPU, virtual reality engine 1426, and applications (e.g., tracking applications), can be implemented on a console separate from the head-mounted display device. In some implementations, a console can connect to or support more than one HMD.

[0125] In alternative configurations, the electronic system 1400 may include different and / or additional components. Similarly, the functionality of one or more of these components may be distributed among them in a manner different from that described above. For example, in some embodiments, the electronic system 1400 may be modified to include other system environments, such as AR system environments and / or MR environments.

[0126] The methods, systems, and devices described above are all examples. Various procedures or components may be appropriately omitted, substituted, or added in various embodiments. For example, in alternative configurations, the described methods may be performed in a different order than that described, and / or various stages may be added, omitted, and / or combined. Furthermore, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the various embodiments may be combined in a similar manner. Moreover, technology is evolving, and therefore many elements are examples, which do not limit the scope of this disclosure to these specific examples.

[0127] Specific details are set forth in the description to provide a thorough understanding of the embodiments. However, embodiments may be practiced without these specific details. For example, well-known circuits, processes, systems, structures, and techniques have been shown without unnecessary details to avoid obscuring the embodiments. This description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the invention. Rather, the foregoing description of the various embodiments will provide those skilled in the art with an enabling description for implementing the various embodiments. Various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this disclosure.

[0128] Furthermore, some embodiments are described as processes depicted as flowcharts or block diagrams. Although each process can describe multiple operations as a sequential process, many of these operations can be performed in parallel or simultaneously. Moreover, the order of these operations can be rearranged. Processes may have additional steps not included in the figures. Furthermore, various embodiments of the methods can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the associated tasks can be stored in a computer-readable medium such as a storage medium. Each processor can execute the associated tasks.

[0129] It will be apparent to those skilled in the art that substantial variations can be made to suit a number of specific requirements. For example, custom or dedicated hardware can be used, and / or specific elements can be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity to other computing devices, such as network input / output devices, can be employed.

[0130] Referring to the accompanying drawings, multiple components that may include a memory may include non-transitory machine-readable media. As used herein, the terms "machine-readable media" and "computer-readable media" refer to any storage medium that participates in providing data that enables a machine to operate in a particular manner. In the embodiments provided above, various machine-readable media may relate to providing instructions / code to multiple processing units and / or one or more other devices for execution. Additionally or alternatively, machine-readable media may be used to store and / or carry such instructions / code. In many embodiments, computer-readable media are physical and / or tangible storage media. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Common forms of computer-readable media include, for example, RAM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash-EPROM, any other memory chip or cartridge, solid-state drive (SSD), or any other medium from which a computer can read instructions and / or code. Computer program products may include code and / or machine-executable instructions that can represent procedures, functions, subroutines, programs, routines, applications (Apps), subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements.

[0131] Those skilled in the art will recognize that the information and signals used to transmit the messages described herein can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the foregoing description can be represented using voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof. As used herein, the terms “and” and “or” can have a variety of meanings, which are also anticipated at least in part based on the context in which these terms are used. Generally, “or” when used in relation to a list, such as A, B, or C, is intended to mean A, B, and C (in an inclusive sense) and A, B, or C (in an exclusive sense). Furthermore, the term “one or more” as used herein can be used to describe any feature, structure, or characteristic in the singular, or can be used to describe some combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example, and the claimed subject matter is not limited to this example. Additionally, the term “at least one of…” when used in relation to a list, such as A, B, or C, can be interpreted as meaning: A; B; C; or any combination of A, B, and / or C, such as AB, AC, BC, AA, ABC, AAB, or AABBCCC, etc.

[0132] In this description, the expression "based on" means "at least partially based on". Therefore, if X is based on Y, then X can be a function of at least a part of Y and any number of other factors. If action X is "based on" Y, then action X can be at least partially based on at least a part of Y.

[0133] Furthermore, although certain embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also feasible. Some embodiments may be implemented solely in hardware, or solely in software, or using a combination thereof. In one example, the software may be implemented using a computer program product containing computer program code or instructions that can be executed by one or more processors to perform any or all of the steps, operations, or processes described in this disclosure, wherein the computer program may be stored on a non-transitory computer-readable medium. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0134] When a device, system, component, or module is described as being configured to perform certain operations or functions, such configuration can be accomplished, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor) to perform the operation (e.g., by executing computer instructions or code), or by programming a processor or core programmed to execute code or instructions stored on a non-transitory storage medium, or any combination thereof. Processes can communicate using various technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0135] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that various additions, deletions, omissions, and other modifications and changes can be made without departing from the broader spirit and scope set forth in the claims. Thus, although several specific embodiments have been described, these embodiments are not intended to be limiting. Various modifications and equivalents fall within the scope of the appended claims.

Claims

1. A near-eye display system, comprising: A projector configured to generate a display image; A waveguide display configured to reproduce the displayed image, the waveguide display including an output coupler configured to couple a reproduced image of the displayed image out from a peripheral region of the waveguide of the waveguide display; An active dimming layer is located on the side of the waveguide opposite to the user's eyes in the near-eye display system, and the active dimming layer can be configured to attenuate ambient light in response to a control signal; A camera configured to receive both a copy of the displayed image and an image of the surrounding environment seen through the active dimming layer and the waveguide; as well as A controller configured to control the projector and the active dimming layer based on a copy of the displayed image and an image of the surrounding environment from the camera.

2. The near-eye display system according to claim 1, wherein, The controller is configured to turn the projector on or off such that the image captured by the camera includes an image of the surrounding environment but excludes a copy of the displayed image, or includes both a copy of the displayed image and an image of the surrounding environment.

3. The near-eye display system according to claim 1, wherein, The controller is configured to control the active dimming layer based on the brightness level of the area where the image of the surrounding environment overlaps with a copy of the displayed image.

4. The near-eye display system according to claim 1, wherein, The waveguide display includes: One or more input couplers configured to couple the display image generated by the projector to the waveguide; and One or more output couplers are configured to couple a copy of the displayed image out of the waveguide from multiple regions of the waveguide.

5. The near-eye display system according to claim 4, wherein, The one or more input couplers and the one or more output couplers include surface relief gratings, holographic gratings, polarizing holograms, semi-transparent mirrors, or combinations thereof.

6. The near-eye display system according to claim 1, wherein, The active dimming layer includes electrochromic materials, photochromic materials, liquid crystal layers, or materials including suspended nanoscale particles.

7. The near-eye display system according to claim 1, wherein, The projector includes collimating optics configured to collimate the displayed image.

8. The near-eye display system according to claim 1, wherein, The displayed image includes: The image to be displayed to the user; Including parallax test light patterns for infrared light; The parallax test light pattern displayed during the user's blink; or Their combination.

9. The near-eye display system according to claim 1, further comprising: A second projector, configured to generate a second display image; A second waveguide display, configured to replicate the second display image, includes a second output coupler configured to couple a replicated image of the second display image out from a peripheral region of the second waveguide of the second waveguide display. A second active dimming layer is located on the side of the second waveguide opposite to the user's second eye of the near-eye display system. The second active dimming layer can be configured to attenuate the ambient light in response to a second control signal. as well as A light combiner configured to direct a copy of the displayed image, a copy of the second displayed image, ambient light attenuated by the active dimming layer, and ambient light attenuated by the second active dimming layer to the camera.

10. The near-eye display system according to claim 9, wherein, The optical combiner includes a third waveguide and a plurality of optical couplers formed on or in the third waveguide.

11. The near-eye display system according to claim 9, wherein, The optical combiner includes multiple reflectors.

12. The near-eye display system according to claim 1, wherein: The near-eye display system is in the form of a pair of glasses; and The projector is positioned on or inside the temple of the pair of glasses.

13. The near-eye display system according to claim 1, wherein, The waveguide display is configured to replicate the displayed image along two dimensions.

14. A near-eye display system, comprising: A first waveguide display subsystem, comprising: A first projector, configured to generate a first display image; A first waveguide display, configured to replicate the first displayed image, includes a first output coupler configured to couple a replicated image of the first displayed image from a peripheral region of the first waveguide of the first waveguide display; and A first active dimming layer is located on the side of the first waveguide opposite to the first eye of the user of the near-eye display system. The first active dimming layer can be configured to variably attenuate ambient light. The second waveguide display subsystem includes: A second projector, configured to generate a second display image; A second waveguide display, configured to replicate the second displayed image, includes a second output coupler configured to couple a replicated image of the second displayed image from a peripheral region of the second waveguide of the second waveguide display; and A second active dimming layer is located on the side of the second waveguide opposite to the user's second eye of the near-eye display system. The second active dimming layer can be configured to variably attenuate the ambient light. A camera, configured to receive at least a copy of the first displayed image and an image of the surrounding environment seen through the first active dimming layer and the first waveguide; and A controller configured to control at least the first projector and the first active dimming layer based on a copy of the first displayed image and an image of the surrounding environment.

15. The near-eye display system of claim 14, further comprising a light combiner configured to guide a copy of the first displayed image, a copy of the second displayed image, ambient light attenuated by the first active dimming layer, and ambient light attenuated by the second active dimming layer to the camera. in, The camera is also configured to receive both a copy of the second displayed image and an image of the surrounding environment as seen through the second active dimming layer and the second waveguide.

16. The near-eye display system according to claim 14, wherein, The controller is configured to turn the first projector on or off such that the image captured by the camera includes an image of the surrounding environment seen through the first active dimming layer but excluding a copy of the first display image, or includes both a copy of the first display image and an image of the surrounding environment seen through the first active dimming layer.

17. The near-eye display system according to claim 14, wherein, The controller is configured to control the first active dimming layer based on the brightness level of the area where the image of the surrounding environment overlaps with a copy of the first displayed image.

18. The near-eye display system according to claim 14, wherein, The first displayed image includes: The image to be displayed to the user; Including parallax test light patterns for infrared light; The parallax test light pattern displayed during the blink of the user's first eye; or Their combination.

19. A method comprising: Through the controller of the near-eye display system: Receive one or more images from at least one parallax camera of the near-eye display system, the one or more images including a display image and an image of the surrounding environment seen through an active dimming layer; Determine the brightness level of the area where the image of the surrounding environment overlaps with the displayed image; as well as A control signal is generated based on the determined brightness level to control the dimming level of the active dimming layer.

20. The method of claim 19, further comprising: The parallax of the near-eye display system is determined based on the received one or more images; as well as Control at least one projector of the near-eye display system to compensate for the parallax of the near-eye display system.

Images