Augmented and virtual reality display systems with associated in-coupling and out-coupling optical regions

Abstract

Augmented reality and virtual reality display systems and devices are configured to efficiently use projected light. In some aspects, the display system includes a light projection system and a head-mounted display, the head-mounted display configured to project light into the user's eyes to display virtual image content. The head-mounted display includes at least one waveguide, the waveguide including a plurality of incoupling regions, each of which is configured to receive light corresponding to a portion of the user's field of view from the light projection system and couple the light into the waveguide; and a plurality of outcoupling regions, which are configured to couple light out of the waveguide to display virtual content, wherein each of the outcoupling regions is configured to receive light from a different incoupling region. In some implementations, each incoupling region has a one-to-one correspondence with a uniquely corresponding outcoupling region.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] Any and all applications for which foreign or domestic priority claims are identified in the Application Data Sheet filed with the present application are incorporated herein by reference pursuant to 37 CFR 1.57.

[0003] This application claims priority to U.S. Provisional Patent Application No. 63 / 029312, filed on May 22, 2020, entitled “METHOD AND SYSTEM FOR DUAL PROJECTOR WAVEGUIDE DISPLAYS WITH WIDE FIELD OF VIEW,” and U.S. Provisional Patent Application No. 63 / 050635, filed on July 10, 2020, entitled “AUGMENTED AND VIRTUAL REALITY DISPLAY SYSTEMS WITH CORRELATED IN-COUPLING AND OUT-COUPLING OPTICAL REGIONS FOR EFFICIENT LIGHT UTILIZATION,” the contents of which are incorporated herein by reference in their entirety.

[0004] This application is related to the following patent applications: U.S. Patent Application Publication No. 2018 / 0275410, entitled “DEPTH BASED FOVEATE DRENDERING FOR DISPLAY SYSTEMS”, published on September 27, 2018; U.S. Patent No. 10573042, entitled “PERIOCULAR TEST FOR MIXEDREALITY CALIBRATION”, published on February 25, 2020; and U.S. Patent Application Publication No. 2019 / 0222830, entitled “DISPLAY SYSTEMS AND METHODS FOR DETERMININGREGISTRATION BETWEEN A DISPLAY AND A USER'S EYES”, published on July 18, 2019; and U.S. Patent Application Publication No. 2019 / 0222830, entitled “DISPLAY SYSTEMS AND METHODS FOR DETERMININGREGISTRATION BETWEEN A DISPLAY AND A USER'S EYES”, filed and published on May 21, 2019; and U.S. Patent Application Publication No. 2018 / 0275410, entitled “DEPTH BASED FOVEATE DRENDERING FOR DISPLAY SYSTEMS”, published on September 27, 2018; and U.S. Patent Application Publication No. 10573042, entitled “PERIOCULAR TEST FOR MIXEDREALITY CALIBRATION”, published on February 25, 2020; and U.S. Patent Application Publication No. 2019 / 0222830, entitled “DISPLAY SYSTEMS AND METHODS FOR DETERMININGREGISTRATION BETWEEN A DISPLAY AND A USER'S EYES”, filed and published on May 21, 2019 US Patent No. 10296792 entitled “Iris Boundary Estimation Using Cornea Curvature”; US Patent Publication No. 2017 / 0053165 entitled “EYELID SHAPE ESTIMATION USING EYE POSE MEASUREMENT” published on February 23, 2017; US Patent Publication No. 2017 / 0053166 entitled “EYELID SHAPE ESTIMATION” published on February 23, 2017, and US Patent No. 2017 / 0053166 entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN U.S. Patent Application Publication No. 2019 / 0243558, entitled “Eye Center of Rotation Determination with One or More Eye Tracking Cameras for Display Systems”; International Publication No. WO 2021 / 01166, published on January 21, 2021, entitled “EYE CENTER OF ROTATION DETERMINATION WITH ONE OR MORE EYE TRACKING CAMERAS”;U.S. Patent Application Publication No. 2018 / 0113311, entitled “SYSTEM AND METHOD FOR PRESENTING IMAGE CONTENT ON MULIPLE DEPTH PLANES BY PROVIDING MULTIPLE INTRA-PUPIL PARALLAX VIEWS,” published on April 26, 2018; International Patent Publication No. WO 2019 / 173158, entitled “DISPLAY SYSTEM WITH LOW-LATENCY PUPIL TRACKER,” published on September 12, 2019; and AUGMENTED REALITY SYSTEMS AND METHODS WITH VARIABLE FOCUS LENS, published on October 12, 2017. U.S. Patent Application Publication No. 2017 / 0293145, entitled “Augmented Reality System and Method with Variable Focus Lens Elements”; and U.S. Patent Application Publication No. 2018 / 0113311, entitled “SYSTEM AND METHOD FOR PRESENTING IMAGE CONTENT ON MULIPLE DEPTH PLANES BY PROVIDING MULTIPLE INTRA-PUPIL PARALLAX VIEWS”, published on April 26, 2018, each of which is hereby incorporated by reference in its entirety.; Technical Field

[0005] The present disclosure relates to display systems, and more particularly, to augmented and virtual reality display systems. Background Art

[0006] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in such a way that they appear to be real or may be perceived as real. Virtual reality or "VR" scenes generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenes generally involve the presentation of digital or virtual image information to enhance the visualization of the real world around the user. Mixed reality or "MR" scenes are a type of AR scene and generally involve virtual objects that are integrated into and responsive to the natural world. For example, a MR scene may include AR image content that appears to be blocked by objects in the real world or is perceived to interact with objects in the real world.

[0007] refer to Figure 1 , an augmented reality scene 10 is shown. A user of AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. The user also perceives that he / she "sees" "virtual content" such as a robot statue 40 standing on the real-world platform 30, and a cartoon-like avatar character 50 that appears to be an avatar of a bumblebee flying around. These elements 50, 40 are "virtual" because they do not exist in the real world. Because the human visual perception system is complex, it is challenging to develop an AR technology that promotes comfortable, natural-feeling, rich presentation of virtual image elements as well as other virtual or real image elements.

[0008] The systems and methods disclosed herein address various challenges associated with display technologies, including AR and VR technologies. Summary of the invention

[0009] Various aspects of the present disclosure will now be described in detail with reference to the accompanying drawings and various examples. However, it will be appreciated by those skilled in the art that other configurations of the apparatus and methods disclosed herein will fall within the scope of the present disclosure even if not described in the same detail. The aspects of the various configurations discussed do not limit the scope of the present disclosure, but are defined by the claims following this description.

[0010] Various examples of display systems that project light into one or more eyes of a user to display virtual image content in the user's field of view are described herein, such as the following examples:

[0011] Example 1: A head-mounted display system configured to project light to display virtual image content, the display system comprising: a light projection system; and a head-mounted display configured to display virtual image content, the head-mounted display comprising at least one waveguide, the at least one waveguide comprising: a plurality of coupling-in regions, each coupling-in region being configured to receive light having image information for forming the virtual image content from the light projection system and couple the light into the waveguide; and a plurality of coupling-out regions being configured to couple the light out of the waveguide to display the virtual content, wherein each of the plurality of coupling-out regions is configured to receive light from a different coupling-in region among the coupling-in regions.

[0012] Example 2: The display system of Example 1, wherein each of the plurality of incoupling regions corresponds to and is configured to selectively direct light to a unique outcoupling region of the plurality of outcoupling regions.

[0013] Example 3: A display system according to Example 1 or 2, wherein each coupling-in region is a diffraction grating having at least one of a tilt, a grating orientation, a grating structure size, or a grating spacing, and the at least one of the tilt, the grating orientation, the grating structure size, or the grating spacing is selected so that light coupled in by the coupling-in region is preferentially coupled out by the outcoupling region corresponding to the coupling-in region.

[0014] Example 4: A display system according to any one of Examples 1 to 3, wherein the light coupling region is configured to redirect incident light from the light projection system along a propagation direction at an angle for total internal reflection through the waveguide, and the display system also includes a light distribution structure configured to redirect a portion of the light to propagate along an axis that intersects the propagation direction.

[0015] Example 5: A display system according to any one of Examples 1 to 4, wherein the light distribution structure and the plurality of outcoupling regions are arranged on opposite main surfaces of the at least one waveguide.

[0016] Example 6: A display system according to any of Examples 1 to 5, wherein the plurality of coupling regions include a diffraction grating.

[0017] Example 7: A display system according to any of Examples 1 to 6, wherein each of the plurality of coupling regions has a different grating pitch than at least one other coupling region of the waveguide.

[0018] Example 8: A display system according to any one of Examples 1 to 7, wherein the different grating spacings change the total internal reflection angle, and light coupled into the waveguide at the total internal reflection angle is totally internally reflected toward the waveguide toward an associated outcoupling region among the multiple outcoupling regions.

[0019] Example 9: A display system according to any of Examples 1 to 6, wherein each of the plurality of coupling regions has a different grating orientation in the plane of the waveguide than at least one other coupling region of the waveguide.

[0020] Example 10: A display system according to any one of Examples 1 to 9, wherein the different grating orientations change the angle at which light is coupled into the waveguide so that the light propagates toward a desired outcoupling region among the multiple outcoupling regions.

[0021] Example 11: A display system according to any of Examples 1 to 10, wherein each of the plurality of coupling regions has a different inclination relative to the plane of the waveguide than at least one other coupling region of the waveguide.

[0022] Example 12: The display system of any one of Examples 1 to 11, wherein the coupling-in regions are spaced apart from each other.

[0023] Example 13: A display system according to any one of Examples 1 to 12, wherein the light incoupling regions are arranged in a row around at least one side of the plurality of outcoupling regions.

[0024] Example 14: A display system according to any of Examples 1 to 13, wherein each of the out-coupling regions defines a strip spanning the waveguide, the strip being elongated along an axis that intersects the row of in-coupling regions.

[0025] Example 15: A display system according to any of Examples 1 to 14, wherein the outcoupling region defines a grid pattern across the waveguide.

[0026] Example 16: A display system according to any one of Examples 1 to 15, wherein each of the multiple coupling regions is arranged to couple light and guide the light to propagate within the waveguide through total internal reflection along a direction different from at least one other coupling region of the waveguide.

[0027] Example 17: A display system according to any one of Examples 1 to 16, wherein at least one coupling-in region near a lateral edge of the at least one coupling-out region is configured to couple light along an angle range symmetrically arranged around an inwardly biased axis.

[0028] Example 18: The display system of any one of Examples 1 to 17, wherein at least two of the plurality of coupling-in regions are disposed along different sides of at least one of the plurality of coupling-out regions.

[0029] Example 19: A display system according to any one of Examples 1 to 18, wherein the plurality of coupling-in regions include coupling-in optical elements surrounding at least one of the plurality of outcoupling regions on at least three sides.

[0030] Example 20: A display system according to any one of Examples 1 to 19, wherein the light projection system includes a light source and a movable light injector, and the movable light injector is configured to guide image light into each of the coupling-in regions.

[0031] Example 21: A display system according to any one of Examples 1 to 20, wherein the head-mounted display includes a plurality of waveguides forming a waveguide stack, each of the waveguides including a plurality of the coupling-in regions and the coupling-out regions, wherein the coupling-in region of each of the waveguides is laterally shifted in a front plan view when viewed from a direction of incident light incident on the coupling-in region.

[0032] Example 22: A display system according to any one of Examples 1 to 21, wherein outcoupling regions of the same waveguide are configured to output light having the same wavefront divergence, wherein outcoupling regions of different waveguides are configured to output light having a different wavefront divergence than an outcoupling region of at least one other waveguide, wherein the different wavefront divergence amounts correspond to different depth planes.

[0033] Example 23: A display system according to any one of Examples 1 to 22, wherein the coupling region of each waveguide is configured to couple in light of a wavelength range corresponding to the same component color, and wherein the coupling regions of some waveguides are configured to couple in light of a wavelength range corresponding to a component color different from the coupling regions of other waveguides.

[0034] Example 24: A display system according to any one of Examples 1 to 23, wherein the light projection system includes: a light source comprising an array of light emitters; a spatial light modulator configured to modulate light from the light source; and a projection optical device mounted on a frame and configured to guide light from the spatial light modulator to the coupling-in area, wherein the display system is configured to activate different multiple light emitters to provide the virtual image content on different depth planes, wherein the display system is configured to: provide a relatively large distance between each of the multiple light emitters for providing the virtual image content on a relatively close depth plane; and provide a relatively small distance between each of the multiple light emitters for providing the virtual image content on a relatively far depth plane.

[0035] Example 25: A display system according to any one of Examples 1 to 24, configured to sequentially activate each of a plurality of light emitters at different times, wherein the display system is configured to synchronize the activation of each light emitter with the display of different intra-pupil images with different parallaxes.

[0036] Example 26: A display system according to any of Examples 1 to 25, wherein the light source is a microdisplay and the light emitter is a light emitting diode.

[0037] Example 27: A display system according to any one of Examples 1 to 26, further comprising a variable focus lens element located on an opposite side of the waveguide.

[0038] Example 28: A display system according to any of Examples 1 to 27, wherein the light projection system includes a plurality of projectors, and wherein a projector of the plurality of projectors is configured to provide image content for a subdivided portion not exceeding a FOV of the head mounted display.

[0039] Example 29: A display system comprising: a light projection system; a head-mounted display configured to project light into a user's eyes to display virtual image content, the head-mounted display comprising: at least one waveguide comprising: a plurality of light incoupling regions configured to receive light from the light projection system; and a plurality of light outcoupling regions configured to output light to the user's eyes; and processing electronics communicating with the display and an inward-facing imaging system, the processing electronics comprising one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: determining a light outcoupling region associated with an eye posture of the eye; and providing instructions to direct light from the light projection system to a light incoupling region associated with the determined light outcoupling region.

[0040] Example 30: A display system according to Example 29, wherein determining the light outcoupling area includes: determining an intersection of the line of sight of the eye and one of the multiple light outcoupling areas.

[0041] Example 31: A display system according to Example 29 or 30, wherein the light coupling-in region is arranged in a row around at least one side of the plurality of outcoupling regions.

[0042] Example 32: A display system according to Example 29, wherein the light projection system includes at least one light source and at least one movable light injector, and the at least one movable light injector is configured to output image light.

[0043] Example 33: A display system according to any of Examples 29 to 32, wherein the at least one movable light injector includes a scanning mirror.

[0044] Example 34: A display system according to any of Examples 29 to 33, wherein the at least one movable light injector includes a scanning fiber injector.

[0045] Example 35: A display system according to any one of Examples 29 to 34, wherein the light projection system includes at least one light source and at least one spatial light modulator, and the spatial light modulator is configured to modulate light from the light source.

[0046] Example 36: A display system according to any of Examples 29 to 35, wherein the head-mounted display is configured to display virtual objects on multiple depth planes.

[0047] Example 37: A display system according to any one of Examples 29 to 36, wherein the operation includes: injecting a group of intra-pupil images of the virtual object with different parallaxes into the eye to display the virtual object on one of the depth planes.

[0048] Example 38: A display system according to any of Examples 29 to 37, wherein the operation includes injecting each of the groups of intra-pupil images with different parallax within a flicker fusion threshold.

[0049] Example 39: A display system according to any one of Examples 29 to 39, wherein the operation includes: providing instructions to direct light to multiple coupling elements in the light coupling region of the multiple coupling regions associated with the determined light outcoupling region.

[0050] Example 40: A display system according to any of Examples 29 to 39, further comprising at least one variable focus lens element on a user side of the at least one waveguide, wherein the first variable focus lens element is configured to modify the projected light to correct ametropia of the user's eye.

[0051] Example 41: A display system according to any one of Examples 29 to 40, wherein the operation further includes: determining a gaze point of the eye and applying a correction corresponding to the gaze point.

[0052] Example 42: A display system according to any one of Examples 29 to 41, further comprising an inward-facing imaging system configured to capture one or more images of the user's eyes, wherein the operation comprises: receiving the images of the user's eyes captured by the inward-facing imaging system; and recognizing the eye gesture based on the received images.

[0053] Example 43: A display system according to any one of Examples 29 to 42, wherein the at least one waveguide is one of a plurality of waveguides forming a waveguide stack, each of the waveguides comprising a plurality of the coupling-in regions and the coupling-out regions, wherein the coupling-in regions of each of the waveguides are laterally shifted in a front plan view when viewed from a direction of incident light onto the coupling-in regions.

[0054] Example 44: A display system according to any of Examples 29 to 43, wherein outcoupling regions of the same waveguide are configured to output light having the same wavefront divergence, wherein outcoupling regions of different waveguides are configured to output light having a different amount of wavefront divergence than an outcoupling region of at least one other waveguide, and wherein the different amounts of wavefront divergence correspond to different depth planes.

[0055] Example 45: A display system according to any one of Examples 29 to 44, wherein the coupling region of each waveguide is configured to couple in light of a wavelength range corresponding to the same component color, and wherein the coupling regions of some waveguides are configured to couple in light of a wavelength range corresponding to a component color different from the coupling regions of other waveguides.

[0056] Example 46: A display system according to any of Examples 29 to 45, wherein the light projection system includes a plurality of projectors, and wherein a projector of the plurality of projectors is configured to provide image content for a subdivided portion not exceeding a FOV of the head mounted display.

[0057] Example 47: A display system comprising: a light projection system; a head-mounted display configured to project light into a user's eyes to display virtual image content, the head-mounted display comprising: at least one waveguide comprising: a plurality of light incoupling regions configured to receive light from the light projection system; and a plurality of light outcoupling regions configured to output light to the user's eyes; and processing electronics in communication with the display, the processing electronics comprising one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: determining a position of a virtual object in the user's field of view; identifying an outcoupling region corresponding to the position; and providing instructions to direct light from the light projection system to a light incoupling region associated with the determined light outcoupling region.

[0058] Example 48: A display system according to Example 47, wherein identifying the outcoupling area includes: determining an eye posture of the eye; and determining the outcoupling area overlapping with a position that the virtual object is to occupy in the field of view.

[0059] Example 49: A display system according to Example 47 or 48, wherein the operation includes: providing movement for the virtual object.

[0060] Example 50: A display system according to any one of Examples 47 to 49, wherein the operation includes: providing temporal changes in visual characteristics of the virtual object over time, wherein the visual characteristics include one or more of the following: color, size, and brightness.

[0061] Example 51: A display system according to any one of Examples 47 to 50, wherein the light coupling-in regions are arranged in a row around at least one side of the plurality of coupling-out regions.

[0062] Example 52: A display system according to any one of Examples 47 to 52, wherein the light projection system includes at least one light source and at least one movable light injector, and the at least one movable light injector is configured to output image light.

[0063] Example 53: A display system according to any of Examples 47 to 52, wherein the at least one movable light injector includes a scanning mirror.

[0064] Example 54: A display system according to any of Examples 47 to 52, wherein the at least one movable light injector includes a scanning fiber injector.

[0065] Example 55: A display system according to any one of Examples 47 to 51, wherein the light projection system includes at least one light source and at least one spatial light modulator, and the spatial light modulator is configured to modulate light from the light source.

[0066] Example 56: A display system according to any of Examples 47 to 55, wherein the head-mounted display is configured to display virtual objects on multiple depth planes.

[0067] Example 57: A display system according to any one of Examples 47 to 56, wherein the operation includes: injecting a group of intra-pupil images of the virtual object with different parallaxes into the eye to display the virtual object on one of the depth planes.

[0068] Example 58: A display system according to any of Examples 47 to 56, wherein the operation includes: providing instructions to direct light to multiple coupling elements in the light coupling region of the multiple coupling regions associated with the determined light outcoupling region.

[0069] Example 59: A display system according to any of Examples 47 to 56, further comprising at least one variable focus lens element on a user side of the at least one waveguide, wherein a first variable focus lens element is configured to modify the projected light to correct ametropia of the eye of the user.

[0070] Example 60: A display system according to any one of Examples 47 to 59, further comprising an inward-facing imaging system configured to capture one or more images of the user's eyes, wherein the operation comprises: receiving images of the user's eyes captured by the inward-facing imaging system; and recognizing the eye gesture based on the received images.

[0071] Example 61: A display system according to any one of Examples 47 to 60, wherein the at least one waveguide is one of a plurality of waveguides forming a waveguide stack, each of the waveguides comprising a plurality of the coupling-in regions and the coupling-out regions, wherein the coupling-in regions of each of the waveguides are laterally shifted in a front plan view when viewed from a direction of incident light incident on the coupling-in regions.

[0072] Example 62: A display system according to any one of Examples 47 to 61, wherein outcoupling regions of the same waveguide are configured to output light having the same wavefront divergence, wherein outcoupling regions of different waveguides are configured to output light having a different amount of wavefront divergence than an outcoupling region of at least one other waveguide, and wherein the different amounts of wavefront divergence correspond to different depth planes.

[0073] Example 63: A display system according to any one of Examples 47 to 62, wherein the coupling region of each waveguide is configured to couple in light of a wavelength range corresponding to the same component color, and wherein the coupling regions of some waveguides are configured to couple in light of a wavelength range corresponding to a component color different from the coupling regions of other waveguides.

[0074] Example 64: A display system according to any of Examples 47 to 63, wherein the coupling-in region includes a diffraction grating.

[0075] Example 65: A display system according to any of Examples 47 to 64, wherein the outcoupling region includes a diffraction grating.

[0076] Example 66: A display system according to any of Examples 47 to 65, wherein the light projection system includes a plurality of projectors, and wherein projectors of the plurality of projectors are configured to provide image content for a subdivided portion not exceeding a FOV of the head-mounted display.

[0077] Example 67: A head-mounted display system configured to project light to display virtual image content, the display system comprising: a light projection system; and a head-mounted display configured to display virtual image content, the head-mounted display comprising at least one waveguide, the at least one waveguide comprising: a plurality of coupling regions proximate at least one lateral edge of the at least one waveguide, wherein each waveguide is configured to receive light having image information for forming the virtual image content from the light projection system and couple the light into the waveguide.

[0078] Example 68: The display system according to Example 67 also includes a plurality of outcoupling regions, which are configured to couple the light out of the waveguide to display the virtual content, wherein each of the plurality of outcoupling regions is configured to receive light from a different incoupling region among the incoupling regions.

[0079] Example 69: A display system according to Example 67 or 68, wherein at least two of the plurality of coupling-in regions are arranged along different lateral edges of at least one of the plurality of coupling-out regions.

[0080] Example 70: A display system according to any of Examples 67 to 69, wherein the plurality of coupling-in regions surround the plurality of coupling-out regions.

[0081] Example 71: A display system according to any of Examples 67 to 70, wherein each of the plurality of coupling-in regions corresponds to and is configured to selectively direct light to a unique coupling-out region of the plurality of coupling-out regions.

[0082] Example 72: A display system according to any one of Examples 67 to 71, wherein the light coupling region is configured to redirect incident light from the light projection system at an angle of total internal reflection through the waveguide along a propagation direction, and the display system also includes a light distribution structure configured to redirect a portion of the light to propagate along an axis that intersects the propagation direction.

[0083] Example 73: A display system according to any one of Examples 67 to 72, wherein the light distribution structure and the plurality of outcoupling regions are arranged on relative major surfaces of the at least one waveguide.

[0084] Example 74: A display system according to any of Examples 67 to 73, wherein the outcoupling region defines a grid pattern across the waveguide.

[0085] Example 75: A display system according to any one of Examples 67 to 74, wherein the head-mounted display includes a plurality of waveguides forming a waveguide stack, each of the waveguides including a plurality of the coupling-in regions and the coupling-out regions, wherein the coupling-in region of each of the waveguides is laterally shifted in a front plan view when viewed from a direction of incident light incident on the coupling-in region.

[0086] Example 76: A display system according to any one of Examples 67 to 75, wherein outcoupling regions of the same waveguide are configured to output light having the same wavefront divergence, wherein outcoupling regions of different waveguides are configured to output light having a different amount of wavefront divergence than an outcoupling region of at least one other waveguide, wherein the different amounts of wavefront divergence correspond to different depth planes.

[0087] Example 77: A display system according to any one of Examples 67 to 76, wherein the coupling region of each waveguide is configured to couple in light of a wavelength range corresponding to the same component color, and wherein the coupling regions of some waveguides are configured to couple in light of a wavelength range corresponding to a component color different from the coupling regions of other waveguides.

[0088] Example 78: A display system according to any of Examples 67 to 77, wherein the plurality of coupling regions include a diffraction grating.

[0089] Example 79: A display system according to any of Examples 67 to 78, wherein each of the plurality of coupling regions has a different grating pitch than at least one other coupling region of the waveguide.

[0090] Example 80: A display system according to any of Examples 67 to 79, wherein each of the plurality of coupling regions has a different grating orientation within the plane of the waveguide than at least one other coupling region of the waveguide.

[0091] Example 81: A display system according to any of Examples 67 to 80, wherein each of the plurality of coupling regions has a different inclination relative to the plane of the waveguide than at least one other coupling region of the waveguide.

[0092] Example 82: A display system according to any of Examples 67 to 81, wherein the coupling-in areas are spaced apart from each other.

[0093] Example 83: A display system according to any one of Examples 67 to 82, wherein each of the multiple coupling regions is arranged to couple light and guide the light to propagate within the waveguide through total internal reflection along a direction different from at least one other coupling region of the waveguide.

[0094] Example 84: A display system according to any one of Examples 67 to 83, wherein the light projection system includes at least one light source and at least one movable light injector, and the at least one movable light injector is configured to guide image light into each of the coupling-in areas.

[0095] Example 85: A display system according to any one of Examples 67 to 84, wherein the light projection system includes: at least one light source comprising an array of light emitters; at least one spatial light modulator configured to modulate light from the light source; and a projection optical device mounted on a frame and configured to guide light from the spatial light modulator to the coupling-in area, wherein the display system is configured to activate different multiple light emitters to provide the virtual image content on different depth planes, wherein the display system is configured to: provide a relatively large distance between each of the multiple light emitters for providing the virtual image content on a relatively close depth plane; and provide a relatively small distance between each of the multiple light emitters for providing the virtual image content on a relatively far depth plane.

[0096] Example 86: A display system according to any one of Examples 67 to 86, configured to sequentially activate each of a plurality of light emitters at different times, wherein the display system is configured to synchronize the activation of each light emitter with the display of different intra-pupil images with different parallaxes.

[0097] Example 87: A display system according to any of Examples 67 to 86, further comprising variable focus lens elements on relative lateral edges of the waveguide.

[0098] Example 88: A display system according to any of Examples 67 to 87, wherein the light projection system includes a plurality of projectors, and wherein projectors of the plurality of projectors are configured to provide image content for a subdivided portion not exceeding a FOV of the head-mounted display.

[0099] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither the present disclosure nor the following detailed description limit the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 An augmented reality (AR) scene view of a user through an AR device is shown.

[0101] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown.

[0102] Figures 3A-3C The relationship between the radius of curvature and the focal radius is shown.

[0103] Figure 4A A representation of the accommodation-convergence response of the human visual system is shown.

[0104] Figure 4B Examples of different accommodation and convergence states of a user's eyes are shown.

[0105] Figure 4C An example of a representation of a top plan view of a user viewing content via a display system is shown.

[0106] Figure 4D Another example of a representation of a top plan view of a user viewing content via a display system is shown.

[0107] Figure 5 Aspects of a method for simulating three-dimensional images by modifying wavefront divergence are shown.

[0108] Figure 6 An example of a waveguide stack for outputting image information to a user is shown.

[0109] Figure 7 An example of an outgoing light beam output by a waveguide is shown.

[0110] Figure 8 An example of a stacked waveguide assembly is shown where each depth plane includes an image formed using multiple different component colors.

[0111] Fig.9A A cross-sectional side view of an example of a set of stacked waveguides, each stacked waveguide including an in-coupling optical element, is shown.

[0112] Fig. 9B Shows Fig.9A A perspective of an example of multiple stacked waveguides.

[0113] Fig. 9C Shows Fig.9A and Fig. 9B A top plan view of an example of multiple stacked waveguides.

[0114] Fig.9D An example of a wearable display system is shown.

[0115] Fig. 10A An example outcoupling element is shown that couples light out of the waveguide towards the user's eye.

[0116] Fig. 10B Shows Fig. 10A An example configuration of an example outcoupling element split into different outcoupling regions.

[0117] Fig. 10C Example paths of light beams that may be coupled out of a waveguide of an example wearable display system are shown.

[0118] Fig. 10DExample principal rays of light that may be coupled out of a waveguide of an example wearable display system are shown.

[0119] Fig.11A An example outcoupling element having an array of light distributing structures is shown.

[0120] Fig. 11B An example split configuration of an example outcoupling element is shown.

[0121] Fig. 11C Shows Fig. 11B Example cross-sectional side view of a waveguide.

[0122] Fig. 12A An example display system including an outcoupling element having a plurality of segments is shown.

[0123] Fig. 12B An example arrangement of in-coupling elements in conjunction with an example non-rectangular out-coupling element is shown.

[0124] Fig.13 An example coupling-in element configuration surrounding an example coupling-out element is shown.

[0125] Fig.14A Another example coupling-in element configuration in conjunction with the example coupling-out element is shown.

[0126] Fig. 14B yes Fig.14A Example of a cross-sectional side view of a waveguide.

[0127] Fig. 14C A cross-sectional side view of an example of a set of stacked waveguides each including an in-coupling optical element configuration is shown.

[0128] Fig.15 A flow chart showing an example of a process for selecting an outcoupling region based on the direction of light to a user's eyes.

[0129] Fig.16 A flow chart showing an example of a process for selecting an outcoupling region based on eye gesture.

[0130] Fig.17A An example of an eye is schematically shown.

[0131] Figure 17A-1 An example coordinate system for determining the eye pose of an eye is schematically shown.

[0132] Fig. 17B is a schematic diagram of a wearable system including an eye tracking system.

[0133] Fig. 17C is a block diagram of a wearable system that may include an eye tracking system.

[0134] Fig.18 A flow chart of an example process for operating a display system having multiple in-coupling areas is shown.

[0135] Fig.19 An example display having a light injection system and an array of incoupling areas for directing light to selected areas is shown.

[0136] Figures 20A-20D An example movable light injection system including a scanning fiber injector is shown.

[0137] Figures 21A-21B An example movable light injection system including a scanning mirror is shown.

[0138] Fig. 22 An example fixed light injection system including a spatial light modulator is shown.

[0139] Fig.23 A top plan view of a light module having multiple light emitters is shown.

[0140] Fig.24 A variable focus lens system is shown.

[0141] Fig.25A The pre- and post-adaptation conditions of the eye for a continuous incident wavefront are shown.

[0142] Fig.25B The pre- and post-adaptation conditions of the eye for a piece-wise approximation of a continuous incident wavefront are shown.

[0143] Fig.25C shows the components used to form Fig.25B Example of a display system with a divergent wavefront approximation for a projection system.

[0144] Fig.25D An example of a display system including an eye tracking system and a pupil rely combiner eyepiece having a pupil expander configured to produce a non-infinite depth plane is shown. DETAILED DESCRIPTION

[0145] AR and / or VR systems can display virtual content to a user or viewer. For example, the content can be displayed on a head-mounted display, such as part of glasses, which projects image information to the user's eyes. In addition, in the case where the system is an AR system, the display can also transmit light from the surrounding environment to the user's eyes to allow viewing of the surrounding environment. As used herein, it should be understood that a "head-mounted" or "head-mountable" display is a display that can be mounted on the head of a viewer or user. Such a display can be understood to form part of a display system.

[0146] In various augmented reality and virtual reality display systems, a waveguide may extend through an associated eye of a user. The waveguide may be configured to output image light having image information to form an image in the eye. For example, the light may be output using an outcoupling element (which may be a grating) that redirects the light from the waveguide to the eye.

[0147] It should be understood that the eye can move relative to the waveguide. To ensure that the eye continues to receive image light even as it moves, the waveguide can output image light with similar image information across the entire viewing area of ​​the waveguide. This allows the eye to receive image light even as it moves and aligns with different areas of the waveguide. A portion of the light coupled out by the outcoupling element can be usefully directed into the pupil of the eye (where the light will pass into the eye to form an image), and other light coupled out by the outcoupling element will not be incident on the pupil of the eye and therefore will not contribute to the image seen by the eye. Therefore, although energy is used to produce all of the output light, only a small portion of that light reaches the eye to form an image. Unpleasantly, light that does not enter the pupil of the eye and the energy used to produce that light is somewhat considered "wasted" because it cannot be perceived by the viewer.

[0148] Advantageously, in some implementations, the display systems disclosed herein allow light to be used efficiently. Rather than outputting light over the entire viewing area of ​​the waveguide, light is selectively output from areas of the waveguide that are aligned with the eye, thereby increasing the proportion of output light that enters the pupil of the eye. Preferably, these areas occupy an area that is less than the entire viewing area of ​​the waveguide, and in some implementations, larger than the area of ​​a typical eye pupil, which may result in dividing the viewing area into two or more, three or more, four or more, or nine or more different light outcoupling areas. Thus, light is not output and wasted over the entire viewing area, while the output area is large enough to provide the desired field of view and provide some tolerance for eye movement.

[0149] In some implementations, an outcoupling region may have a uniquely associated or related incoupling region. It will be appreciated that the incoupling region receives image light (e.g., light modulated by a spatial light modulator) and couples the image light into the waveguide such that the image light propagates toward and is incident on the outcoupling region, from which the light is coupled out and propagates toward the user's eye. In some implementations, the light distribution feature may also be used to distribute light from the incoupling region along an axis passing through the desired outcoupling region. Because the incoupling region selectively directs light toward an associated or related outcoupling region, an outcoupling region for outputting light may be selected by directing light toward one or more associated incoupling regions. Advantageously, in addition to energy efficiency, flexibility in the arrangement of the incoupling regions may be provided by associating specific incoupling regions with specific outcoupling regions. For example, the incoupling regions may be distributed in rows across and / or around the perimeter of the waveguide, which may have advantages for providing a thinner form factor or a wider range of waveguide shapes than in the incoupling regions (which must be located in the same portion of the waveguide).

[0150] It should be understood that the display system may include a light projection system that is configured to selectively direct light into one or more desired coupling regions. In some examples, the light projection system may include a projector or multiple projectors, and / or in some implementations, the projector may be used as a light source. Since the light projection system inputs light into the coupling region, it may also be referred to as a light injection system. The light projection system may include a single projection device or multiple projection devices (e.g., a light projector). In some implementations, the field of view (FOV) of the display system may be subdivided into multiple portions, and each light projection device may be configured to project image content for an associated portion of the entire FOV. Therefore, in some implementations, each projector provides image content that does not exceed a subdivided portion of the field of view of the display system. It should be understood that in order to provide acceptable image quality, the optical devices and associated systems used to form the entire FOV may be complex and large in size, especially when the projection system collimates light for input into a waveguide. On the other hand, the optics and associated systems for a projector that forms only a portion of the FOV can be simpler, and in some implementations, the overall size of a light projection system having multiple light projectors, each providing image content for a portion of the entire field of view, is smaller than the size of a single light projection system providing image content for the entire FOV. Thus, advantageously, the use of multiple light projectors helps achieve a smaller form factor than a system having a single light projector for the entire field of view.

[0151] In some implementations, the light projection system may include a mechanical actuation structure to physically adjust the direction of the output image light toward the desired coupling region. For example, the light projection system may include one or more switchable reflectors that can, for example, rotate to direct light from one or more light sources into the desired coupling region. In some other implementations, the light projection system may include one or more scanning optical fibers that scan over the coupling region and are configured to output light when the output end of the scanning optical fiber is aligned with a position corresponding to the desired coupling region.

[0152] In some other implementations, the light projection system can electrically switch the light output to different coupling-in regions without mechanical actuation. For example, the light projection system can include multiple individually activated light sources that are arranged to provide unique light paths to individually corresponding coupling-in regions of the coupling-in regions. As a result, activation of the appropriate light sources can be used to direct light into the desired coupling-in regions. In some implementations, a spatial light modulator can be set in the light path between the light source and the coupling-in region to encode the light from the light source with image information that forms an image.

[0153] In some implementations, an appropriate outcoupling region for outputting light can be selected based on a determination of an orientation of the eye (e.g., an orientation of a pupil of the eye). For example, a display system can include an inward-facing camera that images the eye to determine its orientation. In some implementations, the line of sight of the eye can be extrapolated to determine the intersection of the line of sight with the outcoupling region. The intersecting light outcoupling region can be understood as a desired region for receiving image light to be output to the eye.

[0154] In some implementations, in conjunction with eye gestures, the location of a virtual object to be displayed can be used to select an outcoupling region to receive image light to be output. For example, a virtual object can be understood to be located in a specific portion of the user's field of view that overlaps with a specific outcoupling region. As a result, the outcoupling region can be designated to receive image light. In some implementations, the outcoupling region can also intersect with the eye's line of sight, particularly in cases where the virtual object is an object that is expected to attract the user's attention (e.g., the virtual object exhibits motion and / or temporal changes in visual characteristics, such as changes in color, size, brightness, etc.).

[0155] Advantageously, display systems according to some implementations can provide a three-dimensional presentation of virtual objects with a high degree of viewing comfort. It should be understood that the user's left and right eyes can be presented with different views of the virtual object to provide convergence cues, thereby providing a sense of depth through stereoscopic vision. In addition, the display system can be configured to output light with different levels of wavefront divergence to elicit an appropriate adaptation response for the desired depth. As a result, an adaptation-convergence match can be achieved, which can provide a comfortable and realistic viewing experience.

[0156] In some implementations, different levels of wavefront divergence can be achieved using multiple waveguides forming a waveguide stack. The outcoupling region of each waveguide can include an outcoupling feature (e.g., a diffraction grating) that is configured to couple out light having a specific amount of wavefront divergence. Thus, a desired amount of wavefront divergence can be selected by directing image light into an incoupling region of a waveguide having outcoupling regions having outcoupling features configured to output light having a desired amount of wavefront divergence. In some implementations, the incoupling regions of different waveguides can be laterally shifted, as shown in a top plan view. Thus, different waveguides can be selected by directing light to different incoupling regions. For example, a specific incoupling region can be associated with a specific waveguide and a specific outcoupling region within the waveguide.

[0157] In some implementations, in addition to or in lieu of multiple waveguides that output light with different amounts of wavefront divergence, one or more variable focus lens elements can be provided to modify the wavefront divergence of light propagating from the waveguides to the eye. In some implementations, the variable focus lens element can alternatively or additionally be used to provide refractive error correction for light propagating toward the eye.

[0158] In some other implementations, the light projection system can include an array of discrete light sources that direct light through the projection optics from different angles to provide parallactically-disparate intra-pupil images, thereby approximating different amounts of wavefront divergence. In some implementations, the wavefront divergence can be modulated using optional separation between discrete light sources. For example, a relatively large distance between the light sources can provide a wavefront divergence corresponding to a relatively close depth plane, while a relatively small distance between the light sources can provide a wavefront divergence corresponding to a relatively far depth plane. Image light for the parallax-disparate intra-pupil image can be directed to an appropriate coupling-in region as described herein. For example, a mechanically actuated structure such as a scanning mirror or a scanning optical fiber can be used to direct the image light for the parallax-disparate intra-pupil image to an appropriate coupling-in region.

[0159] Implementations of the present technology can provide various advantages. For example, the implementations disclosed herein can advantageously improve the efficiency of the display system by increasing the proportion of light generated by the light projection system that is used to form the image perceived by the user, while reducing the proportion of light that falls on the user's eyes or other parts of the face, or does not contribute to the image perceived by the user. By selectively coupling light toward the pupil, such a system can reduce the amount of light energy generated by the light projection system or other display light source to produce an image of a given brightness in the user's eye. In addition, since a larger proportion of the generated light is directed into the eye, the image brightness can be increased. In addition, less energy can be used to generate light that reaches the desired brightness, allowing energy usage to be reallocated to other uses, such as to computing processes, which has advantages for image rendering, reducing latency, and the like.

[0160] It should be noted that the coupling element may be referred to herein as an input coupling element, an incoupling grating (or ICG), a diffractive input coupling element, etc. Similarly, the coupling element may be referred to herein as an output coupling element, a diffractive output coupling element, etc.

[0161] Reference will now be made to the drawings, wherein like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale.

[0162] Example Display System

[0163] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown. It should be understood that the user's eyes are spaced apart, and when viewing a real object in space, each eye will have a slightly different view of the object, and an image of the object may be formed at a different location on the retina of each eye. This may be referred to as binocular parallax, and may be used by the human visual system to provide a perception of depth. Conventional display systems simulate binocular parallax by presenting two different images 190, 200 (one for each eye 210, 220) having slightly different views of the same virtual object, corresponding to the view of the virtual object that each eye would see if the virtual object were a real object of the desired depth. These images provide binocular cues that the user's visual system may interpret as obtaining a perception of depth.

[0164] Continue to refer Figure 2, the images 190, 200 are spaced a distance 230 from the eyes 210, 220 on the z-axis. The z-axis is parallel to the optical axis of the viewer, whose eyes are fixed on an object at optical infinity directly in front of the viewer. The images 190, 200 are flat and at a fixed distance from the eyes 210, 220. Based on slightly different views of the virtual objects in the images presented to the eyes 210, 220, respectively, the eyes can naturally rotate so that the image of the object falls on corresponding points on the retina of each eye to maintain a single binocular vision. The rotation can cause the line of sight of each of the eyes 210, 220 to converge to a point in space at which the virtual object is perceived to exist. As a result, providing a three-dimensional image typically involves providing binocular cues that can manipulate the convergence of the user's eyes 210, 220, and the human visual system interprets it as providing a perception of depth.

[0165] However, generating a realistic and comfortable perception of depth is challenging.It will be appreciated that light from objects at different distances from the eye have wavefronts with different amounts of divergence. Figures 3A-3C The relationship between distance and divergence of light is shown. The distance between the object and the eye 210 is represented by R1, R2 and R3 in order of decreasing distance. Figures 3A-3C As shown in , as the distance from the object decreases, the light becomes more divergent. Conversely, as the distance increases, the light becomes more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of how far the point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. Although for clarity in Figures 3A-3C Only a single eye 210 is shown in FIG. 2 and other figures herein, but the discussion regarding eye 210 may apply to both eyes 210 and 220 of a viewer.

[0166] Continue to refer Figures 3A-3C, the light emitted by the object on which the viewer's eye is focused may have different degrees of wavefront divergence. Due to different amounts of wavefront divergence, the lens of the eye may focus the light differently, which in turn may require the lens to take different shapes to form a focused image on the retina of the eye. In the case where a focused image is not formed on the retina, the resulting retinal blur may serve as a cue for adaptation, which causes a change in the shape of the lens of the eye until a focused image is formed on the retina. For example, the cue of adaptation may trigger the ciliary muscles around the lens of the eye to relax or contract, thereby adjusting the force applied to the suspensory ligaments that hold the lens, thereby causing the shape of the lens of the eye to change until the retinal blur of the focused object is eliminated or minimized, thereby forming a focused image of the focused object on the retina of the eye (e.g., the fovea). The process by which the lens of the eye changes shape may be referred to as adaptation, and the shape of the lens of the eye required to form a focused image of the focused object on the retina of the eye (e.g., the fovea) may be referred to as an adapted state.

[0167] Reference now Figure 4A , shows a representation of the accommodation-convergence response of the human visual system. Movement of the eyes in fixation on an object causes the eyes to receive light from the object, which forms an image on each retina of the eyes. The presence of retinal blur in the image formed on the retina can provide a cue for accommodation, and the relative position of the images on the retina can provide a cue for convergence. The cue of accommodation causes accommodation to occur, resulting in the lenses of the eyes each assuming a specific state of accommodation, which forms a focused image of the object on the retina of the eye (e.g., the fovea). On the other hand, the cue of convergence causes a vergence movement (rotation of the eyes) to occur, so that the images formed on each retina of each eye are located at corresponding retinal points that maintain a single binocular vision. In these positions, the eyes can be said to have assumed a specific state of convergence. Continue with reference to Figure 4A , accommodation can be understood as the process by which the eye reaches a specific state of accommodation, and vergence can be understood as the process by which the eye reaches a specific state of vergence. Figure 4A As shown in , if the user looks at another object, the accommodation and convergence states of the eyes may change. For example, if the user looks at a new object at a different depth on the z-axis, the accommodation state may change.

[0168] Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of convergence and accommodation. As described above, the convergence movement of the two eyes relative to each other (e.g., rotation of the eyes so that the pupils move toward or away from each other to converge the eyes' line of sight to fixate on an object) is closely related to the accommodation of the lens of the eye. Under normal circumstances, changing the shape of the lens of the eye to change the focus from one object to another object at a different distance will automatically result in a matching change in vergence from the same distance under a relationship called the "accommodation-convergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in the shape of the lens.

[0169] Reference now Figure 4B , examples of different accommodation and convergence states of eyes are shown. A pair of eyes 222a is fixated on an object at optical infinity, while a pair of eyes 222b is fixated on an object 221 at less than optical infinity. Notably, the convergence state of each pair of eyes is different, with the pair of eyes 222a pointing straight ahead, while the pair of eyes 222 converge on the object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of the lenses 210a, 220a.

[0170] Unexpectedly, many users of conventional "3D" display systems find such conventional systems uncomfortable or unable to perceive a sense of depth at all due to the mismatch between the accommodation state and the convergence state in these displays. As described above, many stereoscopic or "3D" display systems display a scene by providing a slightly different image to each eye. Such systems are uncomfortable for many viewers because they, among other things, provide different presentations of the scene and cause changes in the convergence state of the eyes, but do not change the accommodation state of those eyes accordingly. Instead, the image is shown at a fixed distance from the eye by the display so that the eye views all image information in a single accommodation state. Such an arrangement counteracts the "accommodation-convergence reflex" by causing changes in the convergence state without a matching change in the accommodation state. It is believed that this mismatch can cause discomfort to the viewer. A display system that provides a better match between accommodation and convergence may form a more realistic and more comfortable three-dimensional image simulation.

[0171] Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Therefore, by providing the eye with different presentations of images corresponding to each of these limited number of depth planes, a highly credible simulation of perceived depth can be achieved. In some implementations, the different presentations can provide both cues for convergence and matching cues for accommodation, thereby providing a physiologically correct accommodation-vergence match.

[0172] Continue to refer Figure 4B, two depth planes 240 are shown, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, a vergence cue may be provided by displaying an image of appropriately different perspectives for each eye 210, 220. In addition, for a given depth plane 240, the light forming the image provided to each eye 210, 220 may have a wavefront divergence corresponding to the light field produced by a point at the distance of the depth plane 240.

[0173] In the illustrated implementation, the depth plane 240 containing point 221 is at a distance of 1 m along the z-axis. As used herein, a zero point located at the pupil of the user's eye may be used to measure the distance or depth along the z-axis. Thus, a depth plane 240 located at a depth of 1 m corresponds to a distance of 1 m from the pupil of the user's eye on the optical axis of these eyes, with the eyes directed toward optical infinity. As an approximation, the depth or distance along the z-axis may be measured from a display in front of the user's eye (e.g., from the surface of a waveguide), plus a distance value between the device and the pupil of the user's eye. This value may be referred to as interocular distance, and corresponds to the distance between the pupil of the user's eye and the display that the user wears in front of the eye. In practice, the value of interocular distance may be a normalized value commonly used for all viewers. For example, it may be assumed that the interocular distance is 20 mm, and the distance of the depth plane at a depth of 1 m in front of the display may be 980 mm.

[0174] Reference now Figure 4C and 4D , respectively show examples of matched adaptation-convergence distance and mismatched adaptation-convergence distance. Figure 4C As shown in , the display system can provide an image of a virtual object to each eye 210, 220. The image can cause the eyes 210, 220 to present a state of convergence, in which the eyes converge on a point 15 on a depth plane 240. In addition, the image can be formed by light having a wavefront curvature corresponding to a real object at the depth plane 240. As a result, the eyes 210, 220 present an accommodation state, in which the image is focused on the retinas of those eyes. Therefore, the user can perceive the virtual object as being at a point 15 on the depth plane 240.

[0175] It should be understood that each of the accommodation state and the convergence state of the eyes 210, 220 is associated with a specific distance on the z-axis. For example, an object at a specific distance from the eyes 210, 220 causes those eyes to exhibit a specific accommodation state based on the distance of the object. The distance associated with a specific accommodation state may be referred to as the accommodation distance A. d Similarly, in a particular state of convergence or position relative to each other, there is a particular vergence distance V associated with the eyes. dIn the case where the adaptation distance and the vergence distance match, the relationship between adaptation and vergence can be said to be physiologically correct. This is considered the most comfortable scenario for the viewer.

[0176] However, in stereoscopic displays, the accommodation distance and the vergence distance may not always match. Figure 4D As shown in , the image displayed to the eyes 210, 220 can be displayed with a wavefront divergence corresponding to the depth plane 240, and the eyes 210, 220 can present a particular accommodation state in which points 15a, 15b on the depth plane are in focus. However, the image displayed to the eyes 210, 220 may provide a convergence cue that causes the eyes 210, 220 to converge on a point 15 that is not located on the depth plane 240. As a result, in some implementations, the accommodation distance corresponds to the distance from the pupil of the eyes 210, 220 to the depth plane 240, while the convergence distance corresponds to the larger distance from the pupil of the eyes 210, 220 to the point 15. The accommodation distance is different from the convergence distance. Therefore, there is an accommodation-convergence mismatch. Such a mismatch is considered undesirable and may cause discomfort to the user. It should be understood that the mismatch corresponds to the distance (e.g., V d -A d ) and can be characterized using diopters.

[0177] In some implementations, it should be appreciated that reference points other than the pupils of the eyes 210, 220 may be used to determine the distance used to determine the accommodation-convergence mismatch, as long as the same reference point is used for the accommodation distance and the convergence distance. For example, the distance from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., a waveguide of a display device) to the depth plane, etc. may be measured.

[0178] Without being limited by theory, it is believed that users may still perceive accommodation-convergence mismatches of up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters as physiologically correct, without the mismatch itself causing significant discomfort. In some implementations, a display system (e.g., Figure 6 The display system 250 of the embodiment of the present invention presents an image with an accommodation-convergence mismatch of about 0.5 diopters or less to a viewer. In some other implementations, the accommodation-convergence mismatch of the image provided by the display system is about 0.33 diopters or less. In other implementations, the accommodation-convergence mismatch of the image provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0179] Figure 5Aspects of a method for simulating a three-dimensional image by modifying wavefront divergence are shown. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to an eye 210 of a user. The waveguide 270 can output light 650 having a defined amount of wavefront divergence that corresponds to the wavefront divergence of a light field generated by a point on a desired depth plane 240. In some implementations, the same amount of wavefront divergence is provided for all objects presented on the depth plane. Additionally, it will be described that image information from a similar waveguide can be provided to the user's other eye.

[0180] In some implementations, a single waveguide can be configured to output light having a set amount of wavefront divergence corresponding to a single or limited number of depth planes, and / or the waveguide can be configured to output light of a limited wavelength range. Thus, in some implementations, multiple waveguides or stacked waveguides can be utilized to provide different amounts of wavefront divergence for different depth planes and / or output light of different wavelength ranges. As used herein, it should be understood that at a depth plane can be a flat surface or can follow the contour of a curved surface.

[0181] Figure 6 An example of a waveguide stack for outputting image information to a user is shown. The display system 250 includes a stack of waveguides, or a stacked waveguide assembly 260, which can be used to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. It will be appreciated that in some implementations, the display system 250 can be considered a light field display. Additionally, the waveguide assembly 260 can also be referred to as an eyepiece.

[0182] In some implementations, the display system 250 can be configured to provide a substantially continuous cue of convergence and multiple discrete cues of accommodation. The cues of convergence can be provided by displaying a different image to each eye of the user, and the cues of accommodation can be provided by outputting image-forming light with selectable discrete amounts of wavefront divergence. In other words, the display system 250 can be configured to output light with variable levels of wavefront divergence. In some implementations, each discrete level of wavefront divergence corresponds to a specific depth plane and can be provided by a specific waveguide of the waveguides 270, 280, 290, 300, 310.

[0183] Continue to refer Figure 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between the waveguides. In some implementations, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image injection device 360, 370, 380, 390, 400 may serve as a light source for the waveguides and may be used to inject image information into the waveguides 270, 280, 290, 300, 310, as described herein, which may be configured to distribute incident light on each respective waveguide for output toward the eye 210. Light exits the output surfaces 410, 420, 430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 and is injected into the corresponding input surfaces 460, 470, 480, 490, 500 of the waveguides 270, 280, 290, 300, 310. In some implementations, each of the input surfaces 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide, or may be a portion of a major surface of the corresponding waveguide (that is, one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some implementations, a single beam (e.g., a collimated beam) may be injected into each waveguide to output an entire field of cloned collimated beams directed toward the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some implementations, a single one of the image injection devices 360, 370, 380, 390, 400 may be associated with multiple (eg, three) of the waveguides 270, 280, 290, 300, 310 and inject light therein.

[0184] In some implementations, the image injection devices 360, 370, 380, 390, 400 are discrete displays that each generate image information for injection into a corresponding waveguide 270, 280, 290, 300, 310, respectively. In some other implementations, the image injection devices 360, 370, 380, 390, 400 are outputs of a single multiplexed display that can deliver image information to each of the image injection devices 360, 370, 380, 390, 400, for example, via one or more light conduits (such as fiber optic cables). It should be understood that the image information provided by the image injection devices 360, 370, 380, 390, 400 can include light of different wavelengths or colors (e.g., different component colors as described herein).

[0185] In some implementations, the light injected into the waveguides 270, 280, 290, 300, 310 is encoded with image information and provided by the light projection system 1010, as further described herein. In some implementations, the light projection system 1010 may include one or more arrays of light-emitting pixels. It should be understood that the arrays of light-emitting pixels may include a plurality of light-emitting pixels, respectively, which may be configured to emit light of varying intensity and color. It should be understood that the image injection devices 360, 370, 380, 390, 400 are schematically shown, and in some implementations, these image injection devices may represent different optical paths and positions in a common projection system, which is configured to output light to an associated one of the waveguides 270, 280, 290, 300, 310. In some implementations, the waveguide in the waveguide assembly 260 may be used as an ideal lens, while relaying the light injected into the waveguide out of the waveguide to the user's eyes. In this concept, the object may be an array of pixels of the light projection system 1010, and the image may be an image on a depth plane.

[0186] The controller 560 controls the operation of one or more stacked waveguide assemblies 260, including the operation of the image injection devices 360, 370, 380, 390, 400, and the light projection system 1010. In some implementations, the controller 560 is part of the local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transitory medium) that regulates the timing and provision of image information to the waveguides 270, 280, 290, 300, 310 according to, for example, any of the various schemes disclosed herein. In some implementations, the controller can be a single integral device, or a distributed system connected by wired or wireless communication channels. In some implementations, the controller 560 can be a processing module 140 or 150 ( Fig.9D ) part.

[0187] Continue to refer Figure 6, the waveguides 270, 280, 290, 300, 310 can be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 270, 280, 290, 300, 310 can each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between those major top and bottom surfaces. In the configuration shown, the waveguides 270, 280, 290, 300, 310 can each include an outcoupling optical element 570, 580, 590, 600, 610, which is configured to extract light from the waveguide by redirecting the light propagating within each respective waveguide outside the waveguide to output image information to the eye 210. The extracted light can also be referred to as outcoupled light, and the outcoupling optical element can also be referred to as a light extraction optical element. The extracted light beam can be output by the waveguide at the location where the light propagating in the waveguide hits the light extraction optical element. As further discussed herein, the outcoupling optical element 570, 580, 590, 600, 610 can be, for example, a grating including a diffractive optical feature. Although shown as being disposed on the bottom major surface of the waveguide 270, 280, 290, 300, 310, for ease of description and clarity of drawing, in some implementations, as further discussed herein, the outcoupling optical element 570, 580, 590, 600, 610 can be disposed on the top and / or bottom major surface, and / or can be disposed directly in the volume of the waveguide 270, 280, 290, 300, 310. In some implementations, the outcoupling optical element 570, 580, 590, 600, 610 can be formed as a material layer that is attached to a transparent substrate to form the waveguide 270, 280, 290, 300, 310. In some other implementations, the waveguides 270, 280, 290, 300, 310 may be a single piece of material and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on a surface and / or within the interior of the piece of material.

[0188] Continue to refer Figure 6As described herein, each waveguide 270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye can be configured to deliver collimated light (which is injected into such waveguide 270) to the eye 210. The collimated light can represent an optical infinity focal plane. The next waveguide up 280 can be configured to send the collimated light through a first lens 350 (e.g., a negative lens) before the collimated light can reach the eye 210; such first lens 350 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next waveguide up 280 as coming from a first focal plane closer inward toward the eye 210 from optical infinity. Similarly, the third upward waveguide 290 has its output light pass through both the first lens 350 and the second lens 340 before reaching the eye 210; the combined optical power of the first 350 and second 340 lenses can be configured to produce another incremental wavefront curvature so that the eye / brain interprets the light from the third waveguide 290 as coming from a second focal plane that is closer inward toward the person from optical infinity than the light from the next waveguide upward 280.

[0189] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all the lenses between it and the eye to represent the aggregate focal power closest to the person's focal plane. To compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 can be provided on top of the stack to compensate for the aggregate focal power of the underlying lens stack 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the outcoupling optics of the waveguide and the focusing aspects of the lens can be static (i.e., not dynamic or electrically activated). In some alternative implementations, one or both of the above can be dynamic using electrically activated features.

[0190] In some implementations, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same depth plane, or multiple subsets of the waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same multiple depth planes, with one set for each depth plane. This may provide advantages for forming tiled images to provide an expanded field of view at those depth planes.

[0191] Continue to refer Figure 6 , the out-coupling optical elements 570, 580, 590, 600, 610 can be configured to both redirect light out of its corresponding waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of out-coupling optical elements 570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some implementations, the light extraction optical elements 570, 580, 590, 600, 610 can be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, 610 can be volume holograms, surface holograms, and / or diffraction gratings. In some implementations, the features 320, 330, 340, 350 may not be lenses; instead, they may simply be spacers (e.g., cladding and / or structures used to form air gaps).

[0192] In some implementations, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features that form a diffraction pattern, or are "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a sufficiently low diffraction efficiency that only a portion of the beam is deflected out toward the eye 210 at each intersection of the DOE, while the remainder continues to move through the waveguide via TIR. The light carrying the image information is thus split into multiple related exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of exit toward the eye 210 for that particular collimated beam that bounces within the waveguide.

[0193] In some implementations, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which a droplet includes a diffraction pattern in a host medium, and the refractive index of the droplet can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the droplet can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0194] In some implementations, a camera assembly 630 (e.g., a digital camera including visible light and infrared light cameras) can be provided to capture images of the eye 210 and / or tissue surrounding the eye 210, for example, to detect user input and / or monitor a physiological state of the user. As used herein, a camera can be any image capture device. In some implementations, the camera assembly 630 can include an image capture device and a light source to project light (e.g., infrared light) into the eye, which can then be reflected by the eye and detected by the image capture device. In some implementations, the camera assembly 630 can be attached to the frame 80 ( Fig.9D ), and can be in electrical communication with processing modules 140 and / or 150, which can process image information from camera assembly 630. In some implementations, one camera assembly 630 can be used for each eye to monitor each eye separately.

[0195] Reference now Figure 7 , shows an example of an outgoing beam output by a waveguide. One waveguide is shown, but it should be understood that the waveguide assembly 260 ( Figure 6 ) can function similarly, where the waveguide assembly 260 includes multiple waveguides. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 is incident on the DOE 570, a portion of the light exits the waveguide as an exit beam 650. The exit beams 650 are shown as being substantially parallel, but, as described herein, they may also be redirected to propagate to the eye 210 at a certain angle depending on the depth plane associated with the waveguide 270 (e.g., forming diverging exit beams). It should be understood that substantially parallel exit beams can be indicative of a waveguide having an out-coupling optical element that couples light out to form an image that appears to be disposed at a depth plane at a large distance from the eye 210 (e.g., optical infinity). Other waveguides or other group outcoupling optical elements may output a more divergent exit beam pattern that will require the eye 210 to adapt to a closer distance to focus it on the retina, and will be interpreted by the brain as light coming from a distance closer to the eye 210 than optical infinity.

[0196] In some implementations, a full-color image can be formed at each depth plane by overlaying an image in each component color, such as three or more component colors. Figure 8An example of a stacked waveguide assembly is shown, where each depth plane includes an image formed using multiple different component colors. The illustrated implementation shows depth planes 240a-240f, but more or fewer depths may also be considered. Each depth plane may have three or more component color images associated with it, including: a first image of a first color G; a second image of a second color R; and a third image of a third color B. In the figure, different depth planes are represented by different numbers of diopters (dpt) after the letters G, R, and B. As an example, the numbers after each of these letters indicate the diopters (1 / m) or the inverse of the distance of the depth plane from the viewer, and each box in the figure represents a separate component color image. In some implementations, in order to account for differences in the eye's focus on light of different wavelengths, the exact location of the depth planes for different component colors may be different. For example, different component color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or may reduce chromatic aberration. Without being limited by theory, it should be appreciated that not placing different component color images at the same physical focus helps compensate for the natural longitudinal chromatic aberration (LCA) of the human eye, and the effect is significant. For example, the eyepiece can help compensate for the eye's LCA by biasing the focus levels of the component color images so that they focus on the retina after being focused by the eye's high LCA optics. In some implementations, the system can have a sparser set of depth planes for one component color (e.g., 2x blue depth planes) and a denser set of depth planes for another component color (e.g., 6x green depth planes). Advantageously, this approach can help reduce hardware cost and complexity while still providing sufficient depth information to the eye. For example, the eye can prioritize the focus of medium (e.g., "green") wavelengths over short and long wavelengths, so a uniform set of depth planes is not required across images of all component colors. For example, in some implementations, a waveguide that outputs a component color image formed by short or long wavelength light (e.g., a blue or red component color image) can be used to form a full-color image at multiple depth planes, while a waveguide that outputs a component color image formed by medium wavelength light (e.g., a green component color image) can be used to form a full-color image for only a single depth plane, provided that the eye will prioritize medium wavelength light (e.g., for forming a green component color image) when perceiving a depth plane of a full-color image. Thus, there is a one-to-one correspondence between the number of waveguides for medium wavelength light and the total number of available depth planes, while the number of waveguides for another wavelength component color (e.g., red light and / or blue light) is less than the total number of available depth planes.

[0197] In some implementations, light for each component color may be output by a single dedicated waveguide, and therefore, each depth plane may have multiple waveguides associated with it. In such implementations, each box in the figure including the letters G, R, or B may be understood to represent a separate waveguide, and three waveguides may be provided per depth plane, with three component color images provided per depth plane. Although the waveguides associated with each depth plane are shown as being adjacent to each other in the figure for ease of description, it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other implementations, the same waveguide may output multiple component colors, such that, for example, only a single waveguide may be provided per depth plane.

[0198] Continue to refer Figure 8 In some implementations, G is green, R is red, and B is blue. In some other implementations, other colors associated with other wavelengths of light, including magenta and cyan, may be used in addition to or in place of one or more of red, green, or blue.

[0199] It should be understood that throughout this disclosure, references to light of a given color will be understood to encompass light of one or more wavelengths within a range of wavelengths perceived by a viewer as light of the given color. For example, red light may include light of one or more wavelengths within a range of about 620-780 nm, green light may include light of one or more wavelengths within a range of about 492-577 nm, and blue light may include light of one or more wavelengths within a range of about 435-493 nm.

[0200] In some implementations, the light projection system 1010 ( Figure 6 ) can be configured to emit one or more wavelengths of light outside the visual range of a viewer, such as infrared and / or ultraviolet wavelengths. Additionally, the incoupling, outcoupling, and other light redirection structures of the waveguide of the display 250 can be configured to direct this light toward the user's eye 210 and emit it out of the display, for example, for imaging and / or user stimulation applications.

[0201] Reference now Fig.9A In some implementations, it may be desirable to redirect light impinging on a waveguide in order to couple the light into the waveguide. Incoupling optical elements may be used to redirect and couple light into its corresponding waveguide. Fig.9AA cross-sectional side view of an example of a plurality of stacked waveguides or stacked waveguide groups 660 is shown, each stacked waveguide including an incoupling optical element. The waveguides can each be configured to output light of one or more different wavelengths or one or more different wavelength ranges. It should be understood that stack 660 can correspond to stack 260 (or stacks 260) except that light from one or more of the image injection devices 360, 370, 380, 390, 400 is injected into the waveguide from a location where light needs to be redirected for incoupling. Figure 6 ), and the waveguide shown in stack 660 can correspond to a portion of multiple waveguides 270, 280, 290, 300, 310.

[0202] The stacked waveguide group 660 shown includes waveguides 670, 680, and 690. Each waveguide includes an associated coupling optical element (which may also be referred to as a light input region on the waveguide), for example, a coupling optical element 700 disposed on a major surface (e.g., an upper major surface) of the waveguide 670, a coupling optical element 710 disposed on a major surface (e.g., an upper major surface) of the waveguide 680, and a coupling optical element 720 disposed on a major surface (e.g., an upper major surface) of the waveguide 690. In some implementations, one or more of the coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the corresponding waveguide 670, 680, 690 (particularly where one or more of the coupling optical elements are reflective deflection optical elements). As shown, the coupling optical elements 700, 710, 720 may be disposed on the upper major surface of their corresponding waveguides 670, 680, 690 (or the top of the next lower waveguide), particularly where those coupling optical elements are transmissive deflection optical elements. In some implementations, the coupling-in optical elements 700, 710, 720 may be disposed in the body of the respective waveguides 670, 680, 690. In some implementations, as discussed herein, the coupling-in optical elements 700, 710, 720 are wavelength selective such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths. Although shown on one side or corner of their respective waveguides 670, 680, 690, it should be understood that in some implementations, the coupling-in optical elements 700, 710, 720 may be disposed in other regions of their respective waveguides 670, 680, 690. It should be understood that the coupling-in optical elements 700, 710, 720 may be understood as coupling-in regions, with each coupling-in element corresponding to a coupling-in region. In some other implementations, as discussed herein, the coupling-in optical elements of each waveguide are each subdivided into a plurality of regions.

[0203] As shown, the coupling-in optical elements 700, 710, 720 can be laterally offset from each other. In some implementations, each coupling-in optical element can be offset so that it receives light without the light passing through another coupling-in optical element. Figure 6 As shown in the figure, each coupling-in optical element 700, 710, 720 can be configured to receive light from different image injection devices 360, 370, 380, 390 and 400, and can be separated from other coupling-in optical elements 700, 710, 720 (for example, laterally spaced apart) so that it substantially does not receive light from other coupling-in optical elements in the coupling-in optical elements 700, 710, 720.

[0204] Each waveguide also includes an associated light distribution element, such as a light distribution element 730 disposed on a major surface (e.g., top major surface) of the waveguide 670, a light distribution element 740 disposed on a major surface (e.g., top major surface) of the waveguide 680, and a light distribution element 750 disposed on a major surface (e.g., top major surface) of the waveguide 690. In some other implementations, the light distribution elements 730, 740, 750 may be disposed on the bottom major surface of the associated waveguides 670, 680, 690, respectively. In some other implementations, the light distribution elements 730, 740, 750 may be disposed on both the top major surface and the bottom major surface of the associated waveguides 670, 680, 690, respectively; or the light distribution elements 730, 740, 750 may be disposed on different major surfaces of the top major surface and the bottom major surface in different associated waveguides 670, 680, 690, respectively.

[0205] The waveguides 670, 680, 690 may be spaced apart and separated, for example, by layers of gas, liquid, and / or solid material. For example, as shown, layer 760a may separate waveguides 670 and 680; and layer 760b may separate waveguides 680 and 690. In some implementations, layers 760a and 760b are formed of a low refractive index material (that is, a material having a lower refractive index than the material forming the waveguides 670, 680, 690 that is immediately adjacent to the waveguide). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or greater, or 0.10 or less, of the refractive index of the material forming the waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may serve as cladding that facilitates total internal reflection (TIR) ​​of light passing through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some implementations, the layers 760a, 760b are formed of air. Although not shown, it is understood that the top and bottom of the illustrated waveguide set 660 can include an adjacent cladding layer.

[0206] Preferably, for ease of manufacturing and other considerations, the materials forming the waveguides 670, 680, 690 are similar or the same, and the materials forming the layers 760a, 760b are similar or the same. In some implementations, the materials forming the waveguides 670, 680, 690 can be different, and / or the materials forming the layers 760a, 760b can be different between one or more waveguides while still maintaining the various refractive index relationships described above.

[0207] Continue to refer Fig.9A , light rays 770, 780, 790 are incident on the waveguide group 660. It should be understood that the image can be injected by one or more image injection devices 360, 370, 380, 390, 400 ( Figure 6 )Inject light 770, 780, 790 into waveguides 670, 680, 690.

[0208] In some implementations, the light rays 770, 780, 790 are intended for different waveguides (e.g., waveguides configured to output light having different amounts of wavefront divergence, and / or configured to output light having different characteristics, such as different wavelengths or colors). Thus, in some implementations, the light rays 770, 780, 790 have different characteristics, such as different wavelengths or different ranges of wavelengths, which may correspond to different colors. The incoupling optical elements 700, 710, 720 each deflect incident light such that the light propagates through a corresponding waveguide of the waveguides 670, 680, 690 by TIR. In some implementations, the incoupling optical elements 700, 710, 720 each selectively deflect one or more specific wavelengths of light while transmitting other wavelengths to the underlying waveguide and associated incoupling optical element.

[0209] For example, the coupling-in optical element 700 can be configured to deflect light 770 having a first wavelength or wavelength range, while transmitting light 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light 780 impinges upon and is deflected by the coupling-in optical element 710, which is configured to deflect light of the second wavelength or wavelength range. The light 790 is deflected by the coupling-in optical element 720, which is configured to selectively deflect light of a third wavelength or wavelength range.

[0210] Continue to refer Fig.9A, the deflected light rays 770, 780, 790 are deflected so that they propagate through the corresponding waveguides 670, 680, 690; that is, the coupling optical element 700, 710, 720 of each waveguide deflects light into the corresponding waveguide 670, 680, 690 to couple the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at a certain angle to propagate through the corresponding waveguide 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the corresponding waveguide 670, 680, 690 by TIR until they impinge on the corresponding light distribution element 730, 740, 750 of the waveguide.

[0211] Reference now Fig. 9B , showing Fig.9A 6. As described above, the incoupled light rays 770, 780, 790 are deflected by the incoupling optical elements 700, 710, 720, respectively, and then propagate through TIR within the waveguides 670, 680, 690, respectively. The light rays 770, 780, 790 then impinge on the light distributing elements 730, 740, 750, respectively. The light distributing elements 730, 740, 750 deflect the light rays 770, 780, 790 so that they propagate toward the outcoupling optical elements 800, 810, 820, respectively.

[0212] In some implementations, the light distribution elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some implementations, the OPEs deflect or distribute light to the outcoupling optical elements 800, 810, 820, and in some implementations, may also increase the beam or spot size of the light as it propagates to the outcoupling optical elements. In some implementations, the light distribution elements 730, 740, 750 may be omitted, and the incoupling optical elements 700, 710, 720 may be configured to deflect light directly to the outcoupling optical elements 800, 810, 820. For example, referring to Fig.9A , the light distribution elements 730, 740, 750 can be replaced by outcoupling optical elements 800, 810, 820, respectively. In some implementations, the outcoupling optical elements 800, 810, 820 are exit pupils (EP) or exit pupil expanders (EPE) that direct light into the viewer's eye 210 ( Figure 7). It should be understood that the OPE may be configured to increase the size of the eyebox in at least one axis, while the EPE may be configured to increase the eyebox in an axis that intersects (e.g., is orthogonal) to the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light that strikes the OPE to the EPE of the same waveguide, while allowing the remainder of the light to continue to propagate down the waveguide. Upon striking the OPE again, another portion of the remaining light is redirected to the EPE, and the remainder of that portion continues to propagate further down the waveguide, and so on. Similarly, upon striking the EPE, a portion of the light that strikes is directed outward from the waveguide toward the user, and the remainder of that light continues to propagate through the waveguide until it strikes the EP again, at which time another portion of the light that strikes is directed out of the waveguide, and so on. Therefore, each time a portion of the light is redirected by an OPE or EPE, a single coupled-in beam can be "copied," thereby forming a field of cloned beams, such as Figure 6 As shown in . In some implementations, the OPE and / or EPE can be configured to modify the size of the beam.

[0213] Accordingly, refer to Fig.9A and Fig. 9BIn some implementations, the waveguide set 660 includes waveguides 670, 680, 690; coupling-in optical elements 700, 710, 720; light distribution elements (e.g., OPEs) 730, 740, 750; and outcoupling optical elements (e.g., EPs) 800, 810, 820 for each component color. The waveguides 670, 680, 690 can be stacked with an air gap / cladding between each waveguide. The coupling-in optical elements 700, 710, 720 redirect or deflect the incident light (with different coupling-in optical elements receiving light of different wavelengths) into its waveguide. The light then propagates at an angle that will cause TIR within the corresponding waveguide 670, 680, 690. In the example shown, light ray 770 (e.g., blue light) is deflected by the first incoupling optical element 700 and then continues to bounce down the waveguide in the manner previously described, interacting with the light distribution element (e.g., OPE) 730 and then the outcoupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red, respectively) will pass through the waveguide 670, with light ray 780 impinging on and being deflected by the incoupling optical element 710. Light ray 780 then bounces down the waveguide 680 via TIR, continues to its light distribution element (e.g., OPE) 740, and then to the outcoupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the light incoupling optical element 720 of the waveguide 690. The light in-coupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light distribution element (e.g., OPE) 750 and then propagates by TIR to the out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally couples the light ray 790 out to the viewer, which also receives the out-coupled light from the other waveguides 670, 680.

[0214] Fig. 9C Shows Fig.9A and Fig. 9BA top plan view of an example of multiple stacked waveguides. As shown, the waveguides 670, 680, 690 and the associated light distribution elements 730, 740, 750 and associated coupling-out optical elements 800, 810, 820 of each waveguide can be vertically aligned. However, as described herein, the coupling-in optical elements 700, 710, 720 are not vertically aligned; instead, the coupling-in optical elements are preferably non-overlapping (e.g., laterally spaced apart as seen in the top plan view). As further discussed herein, this non-overlapping spatial arrangement facilitates one-to-one injection of light from different resources into different waveguides, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some implementations, arrangements including non-overlapping spatially separated coupling-in optical elements can be referred to as shifted pupil systems, and the coupling-in optical elements within these arrangements can correspond to sub-pupils. It should be understood that the top view shown can be understood as a front plan view when viewed from the direction of incident light beams (e.g., light 770, 780, 790) incident on the incoupling elements 700, 710, 720, respectively.

[0215] Fig.9D An example of a wearable display system 60 is shown into which the various waveguides and related systems disclosed herein may be integrated. In some implementations, the display system 60 is Figure 6 System 250, Figure 6 Some parts of the system 60 are schematically shown in more detail. For example, Figure 6 The waveguide assembly 260 can be part of the display 70.

[0216] Continue to refer Fig.9D, the display system 60 includes a display 70, and various mechanical and electronic modules and systems that support the functions of the display 70. The display 70 can be coupled to a frame 80, which can be worn by a display system user or viewer 90, and is configured to position the display 70 in front of the eyes of the user 90. In some implementations, the display 70 can be considered as glasses. In some implementations, a speaker 100 is coupled to the frame 80 and is configured to be positioned near the ear canal of the user 90 (in some implementations, another speaker not shown can be optionally positioned near the other ear canal of the user to provide stereo / shaped sound control). The display system 60 may also include one or more microphones 110 or other devices that detect sound. In some implementations, the microphone is configured to allow the user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.), and / or may allow audio communication with other people (e.g., other users of similar display systems). The microphone can be further configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or the environment). In some implementations, the display system may further include a peripheral sensor 120a, which may be separated from the frame 80 and attached to the body of the user 90 (e.g., the head, torso, limbs, etc. of the user 90). In some implementations, the peripheral sensor 120a may be configured to obtain data representing a physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0217] Continue to refer Fig.9D, the display 70 is operably coupled to a local data processing module 140 via a communication link 130 (such as via a wired conductor or a wireless connection), which can be mounted in various configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration, in a belt-coupled configuration). Similarly, the sensor 120a can be operably coupled to the local processor and data module 140 via a communication link 120b (e.g., a wired conductor or a wireless connection). The local processing and data module 140 can include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard drive), both of which can be used to assist in the processing, caching, and storage of data. Optionally, the local processor and data module 140 can include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include data that is: a) captured from a sensor (e.g., that may be operably coupled to the frame 80 or otherwise attached to the user 90), such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, a gyroscope, and / or other sensors disclosed herein; and / or b) obtained and / or processed using the remote processing module 150 and / or the remote data repository 160 (including data related to virtual content), possibly communicated to the display 70 after such processing or retrieval. The local processing and data module 140 may be operably coupled to the remote processing module 150 and the remote data repository 160 by communication links 170, 180, such as via a wired or wireless communication link, so that these remote modules 150, 160 are operably coupled to each other and can be used as a resource for the local processing and data module 140. In some implementations, the local processing and data module 140 may include one or more image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. In some other implementations, one or more of these sensors may be attached to the frame 80, or may be a stand-alone structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0218] Continue to refer Fig.9DIn some implementations, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some implementations, the remote data repository 160 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some implementations, the remote data repository 160 may include one or more remote servers that provide information to the local processing and data module 140 and / or the remote processing module 150, such as information for generating augmented reality content. In some implementations, all data is stored in the local processing and data module, and all computations are performed in the local processing and data module, allowing for fully autonomous use from the remote module. Optionally, an external system including a CPU, GPU, etc. (e.g., a system of one or more processors, one or more computers) can perform at least a portion of the processing (e.g., generating image information, processing data) and provide information to modules 140, 150, 160, and receive information from modules 140, 150, 160, for example via a wireless or wired connection.

[0219] Example display system with multiple outcoupling regions

[0220] The display systems described herein (e.g., display system 60, Fig.9D ) can be used to present augmented or virtual reality content (referred to herein as virtual content). To present virtual content, the display system can direct light out of the one or more waveguides using one or more outcoupling elements that couple the light out in a direction that causes the light to propagate into the eyes of a user (e.g., a wearer of a head mounted display).

[0221] like Fig. 10AAs shown, the example outcoupling element 830 couples light propagating within the waveguide (e.g., by total internal reflection) toward the user's eye 1010, as described herein. However, a typical outcoupling element can be constructed to output light in various directions at all areas across the area of ​​the outcoupling optical element. Therefore, a subset of the light coupled out by the outcoupling optical element 830 can be effectively directed to the pupil 1012 of the user's eye 1010, where the light will enter the eye 1010 to form an image. Ideally directed light is shown by the example beam 1015, which can extend perpendicular or substantially perpendicular to the outcoupling element 830 at a position near the center of the outcoupling element 830 (and can extend at an inward angle at a position near the periphery of the outcoupling element 830). Other light coupled out by the outcoupling element 830 (e.g., light beam 1017 coupled out from the periphery of the outcoupling element 830 in the normal direction, or coupled out at various angles that are not incident on the eye 1010 at or near the pupil 1012) does not contribute to the image formed in the eye 1010. Such light coupled out from the outcoupling optical element 830 but not incident at or near the pupil can be considered to be wasted energy. It should be understood that the drawings are schematic and not necessarily drawn to scale, so that the lateral distance between the light beams 115 and 117 is large relative to the eye 1010 shown.

[0222] Advantageously, some implementations increase the fraction of light that is outcoupled toward the pupil 1012 of the user's eye 1010 (e.g., light that is outcoupled in concert with light beam 1015), and reduce the amount of light that is outcoupled in other directions that are not aligned with the eye 1010 (e.g., light that is outcoupled in concert with light beam 1017). For example, such a system can increase the fraction of incoupled light that ultimately enters the pupil, and can correspondingly reduce the amount of light energy that must be generated by a light projection system or other display light source in order to produce an image of a given brightness in the user's eye. Various implementations of the present technology provide systems including outcoupling elements, incoupling elements, and / or light projection systems that are configured to selectively direct image light toward the pupil 1012 of the user's eye 1010. Such systems can thereby advantageously improve the efficiency of the display systems disclosed herein because they can increase the fraction of a given amount of light generated by the light projection system that reaches the eye 1010 to form an image perceived by the user, and can reduce the fraction of light that falls on the user's eye or other parts of the face, or that does not contribute to the image perceived by the user.

[0223] like Fig. 10B and 10CAs schematically shown, some implementations may include dividing the outcoupling element 1000 of the waveguide 900 into different outcoupling regions 1000a, 1000b, 1000c. Different outcoupling regions may be configured to outcouple light differently. For example, the outcoupling region 1000b may guide outcoupled light 1015 while reducing or not guiding light 1017 output from other outcoupling regions 1000a, 1000c. As a result, light 1017 that does not enter the eye and is not perceived by the user is not "wasted" by being output from the outcoupling regions 1000a, 1000c.

[0224] It should be understood that different outcoupling regions may direct the output light at angles other than perpendicular to the outcoupling element 1000 . Fig. 10C Example beam directions 1019 that may be associated with different outcoupling regions 1000a, 1000b, 1000c are shown. In some implementations, different outcoupling regions may be utilized to outcouple light for different portions of the field of view. For example, a central region 1000b of the outcoupling element 1000 may be desired to outcouple light along a set of angles generally arranged around directions extending vertically outward toward the eye 1010, while peripheral regions 1000a, 1000c of the outcoupling element 1000 may be desired to outcouple light along a set of angles generally arranged around directions angled inward toward the eye 1010, and these lights also enter the pupil of the eye. Thus, in some implementations, as discussed herein, the field of view and / or virtual content available in different portions of the field of view provided by the display system is changed as desired by appropriately selecting the outcoupling regions 1000a, 1000b, 1000c. Advantageously, this ability to select different portions of the field of view (e.g., portions containing desired virtual content) promotes high energy efficiency by allowing light from certain other directions not to be emitted.

[0225] In some implementations, it should be understood that a light projection system can be utilized to direct light into the waveguide 900, and the light projection system can utilize collimated light to input into the projection optics in the waveguide 900. The image tiling method disclosed herein advantageously allows the use of separate light projectors for each or multiple portions of different portions of the field of view, wherein each projector forms image content for an associated portion of the field of view. It should be understood that if a single light projector is used, the collimated light used to provide image content over the entire field of view may require large and complex projection optics. On the other hand, projection optics for providing a relatively small field of view (e.g., a portion of the field of view of a display system) are advantageously simpler and smaller than projection optics for collimated light for a larger field of view. As discussed herein, therefore, for the entire field of view, the use of multiple light projectors may occupy a smaller total volume than using a single light projector.

[0226] Fig. 10DAn example principal ray 1015 that may be coupled out of a waveguide of an example wearable display system is shown. In some implementations, a single one of the outcoupling regions 1000a, 1000b, 1000c may be large enough to produce a desired field of view for the eye 1010. For example, the light rays 1015 that may be coupled out of the outcoupling region 1000b may be spread out in a region near and including the limit of vision of the eye, or may provide a range of angles for light incident on the eye corresponding to a desired field of view.

[0227] Reference now Fig.11A , the waveguide 900 can have a decoupling element, such as decoupling element 1000, which can include an array of other light distributing structures 1020 or diffractions that cause light to diffract and propagate on the decoupling element 1000. In some implementations, the light distributing structures 1020 and the decoupling element 1000 can be disposed on the same side of the waveguide 900, such as on the same major surface of the waveguide 900. In such an implementation, the area occupied by the decoupling element 1000 can include structures that couple light out of the waveguide (e.g., diffractive structures) as well as structures that distribute incident light on the waveguide by total internal reflection (e.g., diffractive structures).

[0228] In some other implementations, in the top view shown, the outcoupling element 1000 may be understood to be located below the light distribution element 1020. In such a configuration, the outcoupling element 1000 and the light distribution element 1020 may be disposed on different opposing major surfaces of the waveguide 900.

[0229] Continue to refer Fig.11A When a light beam 1025 (e.g., corresponding to image light) impinges on the coupling element 1005, the light beam 1025 is coupled into the waveguide 900, propagates in a propagation direction along one axis (e.g., the vertical axis) until it impinges on the light distribution structure 1020, which causes a portion of the light beam 1025 to be split into small light beams 1027 that can generally cascade and propagate along different axes (e.g., the orthogonal axis or the horizontal axis as shown). Preferably, the coupling element 1005 redirects the light into the waveguide 900 at a certain angle so that the light beam 1025 propagates within the waveguide by total internal reflection. It should be understood that the light distribution structure 1020 has a sufficiently low light redirection efficiency to allow most of the light beam 1025 to continue to propagate (e.g., propagate downward along the vertical axis). Therefore, the light beam 1025 can be replicated and substantially fill the entire area of ​​the outcoupling element 1000. Although such an arrangement can provide uniform propagation and high coverage on the outcoupling element 1000, it may make it difficult to aim the propagation of the small light beams 1027 to effectively utilize the light.

[0230] In some implementations, such as Fig. 11BAs shown, the waveguide 900 may have a coupling element 1000, which may be divided into a left region 1000l and a right region 1000r. The left region 1000l may include only a light distribution structure 1020l, which is configured to cause the small beam 1027l to propagate approximately to the left and downward. Similarly, the right region 1000r may include only a light distribution structure 1020r, which is configured to cause the small beam 1027r to propagate approximately to the right and downward. Therefore, each light beam 1025 entering each region 1000l, 1000r is coupled out only from the same region in the regions 1000l and 1000r, and does not propagate to another region in the regions 1000l, 1000r. This configuration can therefore improve the efficiency of the coupling element 1000, because the area of ​​the coupling element 1000 on which each light beam 1025 is propagated is reduced by approximately half. It should be understood that in various implementations, further segmentation may be implemented to provide efficient use of the coupled-in light, for example by including additional segmentation regions and / or segmentation regions of different shapes. The boundary between the left region 1000l and the right region 1000r may correspond to a physical segmentation structure (e.g., an optically opaque and / or absorptive partition), or may simply be an interface between regions of light distribution structures 1020r, 1020l of different orientations.

[0231] Fig. 11C Shows the implementation Fig. 11B 900, wherein the outcoupling region and the light distribution structure can be located on different sides of the waveguide. For example, the outcoupling region 1000r can be disposed on a first major surface of the waveguide 900, and the light distribution structure 1020r can be disposed on a second opposing major surface of the waveguide 900. In some implementations, as shown, the light distribution structure 1020r can be vertically aligned and occupy a similar area as the outcoupling region 1000r. As discussed herein, the outcoupling region and the light distribution structure can be diffraction structures, such as diffraction gratings in some implementations.

[0232] Fig. 12A and 12B An example configuration is shown in which some aspects of the coupling element may be varied to improve the efficiency of light use in the imaging system described herein. Fig. 12A and 12B , waveguide 900 includes a plurality of discrete incoupling regions.

[0233] exist Fig. 12A, the example waveguide 900 includes an outcoupling element 1000, which includes regions 1000l, 1000c, and 1000r. Regions 1000l, 1000c, and 1000r may be physically divided regions of the outcoupling element 1000, or may be regions of the outcoupling element 1000 that are not physically divided by any dividing structure, change in light distribution structure, or other physical structures.

[0234] The example display system also includes corresponding coupling elements 1005l, 1005c, 005r, which are configured to couple light incident on the coupling elements 1005l, 1005c, 1005r toward the outcoupling element 1000. It should be understood that Fig. 11C The light distributing structure shown may be disposed on a major surface of the waveguide 900 opposite to the major surface having the light outcoupling element 1000. In some other implementations, the light distributing structure and the light outcoupling element 1000 may be located on the same major surface.

[0235] Continue to refer Fig. 12A , the coupling element 1005c couples and directs light into a central region 1000c of the outcoupling element 1000, and it may be desirable that the light be evenly distributed in both lateral directions. In some implementations, it may be desirable that the coupling element 1005c directs the incoupled light within an angular range that is approximately symmetrically distributed around the axis 1025c. In addition, as shown, the coupling elements 1005l and 1005r can couple light into associated regions 1000l, 1000r of the outcoupling element 1000, where it may be more desirable that the light be distributed inwardly toward the center of the outcoupling element 1000 rather than toward the lateral edges of the outcoupling element 1000 (e.g., because light reaching the lateral edges of the outcoupling element 1000 may be absorbed, may be coupled out away from the eye, or may not be effectively coupled out). Therefore, it may be desirable that the coupling elements 1005l and 1005r direct the incoupled light within an angular range around the inwardly biased axes 1025l, 1025r to reduce the amount of light reaching the lateral edges of the outcoupling element 1000. In some implementations, regions 10001, 1000c, and 1000r may be understood as outcoupling regions defining a strip on waveguide 900, where the strip extends along an axis that intersects the row defined by incoupling regions 10051, 1005c, and 1005r.

[0236] In some implementations, such as implementations where the coupling elements 10051, 1005c, 1005r include diffraction gratings, the coupling elements 10051, 1005c, 1005r can achieve a desired coupling directivity based on the orientation of the coupling structure 1007. Fig. 12AIn the example implementation shown, the coupling structures 1007 of the lateral coupling elements 10051 and 1005r are tilted or offset in a particular direction relative to the orientation of the coupling structures 1007 of the central coupling element 10005c.

[0237] Continue to refer Fig. 12A , regions 1000l, 1000c, 1000r and / or associated light distribution structures 1020 ( Fig. 11C ) may be separated by optically opaque and / or absorbing partitions to limit the propagation of light between different regions in the associated regions 1000l, 1000c, 1000r. In some other implementations, no partitions are provided between different regions or their associated light distribution structures. Instead, light directed to one outcoupling region may be allowed to propagate freely to other outcoupling regions. However, such a configuration provides an advantage for efficient use of light, because the incoupling region directs light primarily to a specific outcoupling region, so that the entire waveguide receives less light than the outcoupling region.

[0238] Fig. 12B Another example arrangement of an in-coupling element in combination with a non-rectangular out-coupling element 1000 is shown. Fig. 12B In an example implementation of the invention, the waveguide 900 includes an even number of coupling-in elements 10051, 10052, 10053, 10054. In some implementations, the coupling-in elements 10051, 10052, 10053, 10054 are arranged in a substantially symmetrical distribution around the middle of the outcoupling element 1000. Therefore, since none of the coupling-in elements 10051, 10052, 10053, 10054 is centered on the central portion of the outcoupling element 1000, all four coupling-in elements 10051, 10052, 10053, 10054 can be configured to have an inward bias. Since the coupling elements 10052 and 10053 are located at a more central position relative to the coupling elements 10051 and 10054, the coupling structures 1007 of the coupling elements 10052 and 10053 have a smaller tilt relative to the coupling structures 1007 of the coupling elements 10051 and 10054, so that the coupling elements 10052 and 10053 couple light into a wider central area of ​​the outcoupling element 1000. The coupling elements 10051 and 10054 are closer to the lateral edges of the outcoupling element 1000 and therefore have a more tilted coupling structure 1007, so that the light coupled by the coupling elements 10051 and 10054 has a larger inward bias and is less likely to be incident on the lateral edges of the outcoupling element 1000.

[0239] although Fig. 12A and 12BThe illustrated implementation shows multiple discrete, velocity-separated incoupling regions, but the waveguide 900 disclosed herein may also be implemented using one or more incoupling regions in which successive portions of individual light incoupling elements have light redirecting structures of different configurations that define their different incoupling regions. For example, Fig.13 A waveguide having coupling elements extending along two or more sides or lateral edges of the waveguide is shown. For example, as shown, coupling elements 1005 can be arranged along multiple sides or edges of the waveguide to surround the coupling element 1000. In some implementations, as shown, the coupling element 1005 substantially surrounds the coupling element 1000. The coupling element 1005 includes coupling structures 1007 that are arranged to couple light and direct the light in a specific desired direction toward the coupling element 1000. For example, the coupling structure 1007 located near the top of the coupling element 1000 can be arranged to couple light generally downward, and the coupling structure 1007 located near the bottom of the coupling element 1000 can be arranged to couple light generally upward, and the coupling structure 1007 located near the lateral edge of the coupling element 1000 can be arranged to couple light generally inward toward the center of the coupling element 1000. It should be understood that different regions of the coupling element 1005 having different configurations of the coupling structure 1007 may be understood to constitute different coupling regions.

[0240] In some examples, the light injection system can be configured to project light toward the coupling element 1005. The light injection system can include multiple projectors, which has the advantage of simplifying the provision of light to the coupling element on different sides of the waveguide. As discussed herein, the entire FOV of the display system can be subdivided into multiple sections, and the light injection system can include multiple projectors that are configured to provide image content for each associated section by directing light to different areas of the coupling element 1005. Additionally or alternatively, the projectors can be oriented to direct light to portions of the coupling element 1005 on different sides of the waveguide 900. In some examples, the light injection system can include a first projector configured to direct light into a portion of the coupling-in element 1005 along a top edge of the waveguide 900, a second projector configured to direct light into a portion of the coupling-in element 1005 along a right edge of the waveguide 900, a third projector configured to direct light into a portion of the coupling-in element 1005 along a bottom edge of the waveguide 900, and a fourth projector configured to direct light into a portion of the coupling-in optical element 1005 along a left edge of the waveguide 900. It should be understood that each projector can be configured to provide image content for different portions of the coupling-out element 1000 that are associated with different portions of the coupling-in optical element 1005, as discussed herein.

[0241] Fig.14A Another example configuration of a waveguide 900 is shown having an incoupling element for coupling light into an outcoupling element 1000. The incoupling element 1005 may include a separate incoupling region 1005. a1 ...1005 d4 Each coupling region 1005 a1 ...1005 d4 The outcoupling regions a1 ... d4 may be configured to couple in light that will be coupled out from the corresponding or associated outcoupling regions a1 ... d4 of the outcoupling element 1000. The outcoupling regions a1 ... d4 may define a grid pattern across the waveguide 900. Based on the above techniques, for example, based on one or more features such as tilt, grating pitch, grating structure size (e.g., size of protrusions on a surface forming the grating), or each incoupling region 1005 may be configured to couple in light that will be coupled out from the corresponding or associated outcoupling regions a1 ... d4 of the outcoupling element 1000. The outcoupling regions a1 ... d4 may define a grid pattern across the waveguide 900. Based on the above techniques, for example, based on one or more features such as tilt, grating pitch, grating structure size (e.g., size of protrusions on a surface forming the a1 ...1005 d4 Other characteristics of each coupling region 1005 a1 ...1005 d4 The light coupled in at each of the coupling regions 1005a2, 1005b2, 1005c2, 1005d2 may be guided to be coupled out only or primarily from the corresponding coupling regions a1 ... d4 of the coupling element 1000. For example, the coupling regions 1005a2, 1005b2, 1005c2, 1005d2 may differ from each other in terms of tilt, grating, grating spacing, grating structure size or another characteristic, so that the light coupled in at each of the coupling regions 1005a2, 1005b2, 1005c2, 1005d2 is selectively or primarily guided to a specific associated coupling region to be coupled out by the coupling region. a2 can be configured so that the light coupled in from it has a total internal reflection angle in the outcoupling element 1000 that causes the light to be incident on the area a2. b2 It can be configured so that light coupled in therefrom has different total internal reflection angles within the waveguide 900 causing most of the light to travel further along the outcoupling element 1000 and be incident on the outcoupling region b2.

[0242] As discussed herein, the light injection system can be configured to direct light toward different incoupling regions 1005. a1 ...1005 d4 Projecting light. The light injection system may include multiple projectors configured to direct light to different incoupling areas 1005 a1 ...1005 d4 In some examples, the light injection system may include different projectors configured to emit subsets of the FOV into one or more incoupling regions 1005. a1 ...1005 d4For example, a first projector may be configured to emit light toward a subset of in-coupling regions that are closer to a first lateral edge of the waveguide, while a second projector may be configured to emit light toward a subset of in-coupling regions that are closer to a second lateral edge of the waveguide.

[0243] Fig. 14B yes Fig.14A Example of a schematic cross-sectional side view of a waveguide, the cross section being taken along plane 14B ( Fig.14A ) intercepted. As shown, the coupling-in regions 1005a2, 1005b2, 1005c2, 1005d2 can receive an incident light beam 1025, which can be coupled in and redirected toward the outcoupling regions a1, b1, c1, d1 of the outcoupling element 1000. It should be understood that the coupling-in regions 1005a2, 1005b2, 1005c2, 1005d2 can be configured to couple in light so that the light propagates through the waveguide 900 by total internal reflection. In addition, when coupling in light, the coupling-in regions 1005a2, 1005b2, 1005c2, 1005d2 can be configured to deflect the light at different angles, as shown. For example, the light beam 1025 can be deflected by the coupling-in regions 1005a2, 1005b2, 1005c2, 1005d2 to provide coupled-in light beams 1205, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, a2 ,1025 b2 ,1025 c2 ,1025 d2 . Incoupling beam 1205 a2 ,1025 b2 ,1025 c2 ,1025 d2 The light is then transmitted to the waveguide through the total internal reflection region 1005a2, 1005b2, 1005c2, 1005d2, and the light is then transmitted to the waveguide through the total internal reflection region 1005a2, 1005b2, 1005c2, 1005d2. ...

[0244] In some implementations, the selectivity of the correspondence between a particular incoupling region and a particular outcoupling region can be improved by configuring the outcoupling region so that the outcoupling region is most effective at coupling out light received from an associated incoupling region. For example, the outcoupling region can be configured to most effectively couple out light incident on the outcoupling region from an angle corresponding to the angle of the incoupling light propagating from the associated incoupling region. For example, the outcoupling region can most effectively couple out light impinging thereon at a TIR angle provided by the associated incoupling region, and / or the outcoupling region can most effectively couple out light impinging thereon from a lateral direction corresponding to the associated incoupling region.

[0245] In some implementations, you can combine Fig.14A The example configuration uses an image tiling method. It should be understood that in some contexts, virtual content may be displayed in only a portion of the user's field of view. In some implementations, regions a1...d4 may each be associated with a particular portion of the field of view, and the image content output by a particular region a1...d4 may correspond to the virtual content of that portion of the field of view. In some cases, an image may span multiple regions a1...d4. Projected onto each coupling element 1005 a1 ...1005 d4 The image light on the coupling element 1005 can represent the partial image that will appear on the corresponding area a1...d4 of the coupling element 1000. Therefore, when the individual image portions are projected onto the corresponding coupling element 1005 a1 ...1005 d4 When the image light for each image portion is coupled out at the corresponding region a1 ... d4 of the outcoupling element 1000, the intensity thereof is greater than that from the other coupling-in regions 1005. a1 ...1005 d4 In some implementations, the image light can be time-multiplexed (directed to different in-coupling regions at different times) so that a user viewing the out-coupling element 1000 perceives a complete image. When time-multiplexed, the image light is preferably provided to each out-coupling region that desires to output image light within a flicker fusion threshold.

[0246] It should be understood that, for example Figures 11A-14B As shown, waveguide 900 can be one of a plurality of similar waveguides forming a waveguide stack. As discussed herein, such a waveguide stack can be advantageously used to provide virtual content at multiple depth planes.

[0247] Fig. 14C A set 660A of stacked waveguides 670A, 680A, and 690A are shown, each of which may be similar to waveguide 900 ( Figures 11A-14B). Each waveguide includes one or more associated incoupling regions 700A, 700B, 710A, 710B, 720A, and 720B (also referred to as light input regions on the waveguide, and may correspond to reference Figures 12A-14C One or more incoupling regions may include one or more incoupling optical elements and may be configured to direct light into one or more outcoupling elements 1000A, 1000B, and 1000C in its associated waveguide. As discussed herein, the outcoupling elements may be segmented into different regions. The configuration of the incoupling regions and outcoupling regions in each stacked waveguide may correspond to any of the example waveguide configurations disclosed, such as those described with reference to FIG. Figures 11A-14A Any of the waveguide configurations discussed.

[0248] As shown, the coupling optical regions 700A, 700B, 710A, 710B, 720A, and 720B can be laterally offset from each other. In some implementations, each coupling optical region can be offset so that it receives light without passing through another coupling optical region. For example, each coupling region 700A, 700B, 710A, 710B, 720A, and 720B can be separated (e.g., laterally spaced) from other coupling optical regions 700A, 700B, 710A, 710B, 720A, and 720B so that it does not substantially receive light passing through other coupling optical elements in the coupling optical elements 700A, 700B, 710A, 710B, 720A, and 720B. In some examples, one or more coupling optical regions can share an image injection device, or can receive light from different image injection devices.

[0249] The waveguide can be similar to the reference Fig.9A The discussed waveguide stacks are spaced or separated in a manner. In some implementations, the light beams 770, 780, 790 are targeted to different waveguides (e.g., waveguides configured to output light with different amounts of wavefront divergence, and / or configured to output light with different characteristics, such as different wavelengths or colors). Advantageously, different waveguides can be selected by directing light to different incoupling regions, thereby providing image light outputs with different amounts of wavefront divergence. Therefore, in some implementations, a particular incoupling region can be associated with a particular waveguide (to couple light into the waveguide) and can also be associated with a particular outcoupling region within the waveguide (to direct light toward the associated outcoupling region).

[0250] Directing light to the outcoupling region

[0251] The display system may selectively direct light to one or more outcoupling regions of the display by identifying regions where image light is output and directing the image light to the outcoupling regions. Fig.15An example light directing process 3000 for directing light to an outcoupling region of a display is shown.

[0252] Continue to refer Fig.15 , in box 3002, the display system can determine an expected position of a virtual object on a display, the display can include a waveguide having a viewing area, and the viewing area can include a coupling element divided into multiple coupling areas. In the case where the display system is an AR or VR system, the system can identify an expected and / or perceived location in the user's 3D environment to display the virtual object and determine an associated location in the area of ​​the display. For example, an application of the display system can send an instruction to the display system to display a virtual object, such as a butterfly, at a specified location in the user's physical environment. The display system can determine that the specified location corresponds to a projected location of the virtual object on the display, such as the upper right corner of the display. For example, the area on the display that provides image light for forming the virtual object can correspond to an expected intersection between a viewing area of ​​the display and an expected light path from the virtual object to the pupil of the eye.

[0253] The projection location on the display can include the entire area of ​​the display on which the virtual object can be projected. For example, if the virtual object is a large object that may occupy most of the user's field of view, such as a tree, the desired location may include a large percentage of the display. In another example, if the virtual object is a small object that occupies a small portion of the user's field of view, such as a butterfly, the desired location may include a small percentage of the display. In another example, the display system can display more than one virtual object. In examples with more than one virtual object, the display system can determine multiple locations, areas, or portions of the display where the virtual objects are displayed.

[0254] It should be understood that the representation of virtual objects in 3D space can take into account the user's eye posture, and the area of ​​the display used to display the virtual object can change over time as the user's eye posture changes. For example, the position of the virtual object can be anchored to a physical object, and the position of the virtual object within the user's field of view can change as the user's eyes move relative to the physical object used as an anchor, which can result in a corresponding change in the area of ​​the display used to output image light. As discussed herein, the display system can recognize the user's eye posture based on one or more eye tracking processes. In some implementations, as discussed herein, in order to provide a perception of the location of the virtual object in the user's 3D environment, the display can output light from a location on an outcoupling element associated with the location so that the light propagates to the eye from a direction corresponding to the location.

[0255] At block 3004, the display system may identify one or more outcoupling regions associated with the determined locations. For example, the display may include multiple outcoupling regions. One or more determined locations or regions of the display may fall on the one or more outcoupling regions. The display system may identify which outcoupling regions are associated with locations or regions of the display associated with one or more displayed virtual objects.

[0256] At block 3006, the display system may send instructions to output light to the identified one or more outcoupling regions. For example, the display system may send instructions to the controller to project light to an incoupling region associated with the identified outcoupling region. For example, the incoupling region may include one or more light redirection structures, such as one or more diffraction gratings. The display system may selectively direct light to the incoupling region using a light injection system, as discussed with reference to FIGS. 20-22.

[0257] In some other implementations, the display system can display different representations of virtual objects based on the user's eye posture. For example, the display system can use foveate rendering techniques to reduce the resolution of virtual content based on, for example, the distance of the virtual object from the center of the user's field of view (e.g., the gaze point), thereby reducing the computational workload involved in displaying the virtual content. For example, the display system can be configured to display the virtual object at a relatively high (e.g., maximum) resolution when the virtual object coincides with the gaze point of the user's eye, and to display the virtual object at a relatively low resolution when the virtual object is located at the periphery of the user's field of view. Foveate rendering techniques are discussed in U.S. Patent Application Publication No. 2018 / 0275410, entitled "DEPTH BASED FOVEATED RENDERING FOR DISPLAY SYSTEMS," published on September 27, 2018, the entire disclosure of which is incorporated herein by reference.

[0258] Example of region determination using eye pose determination

[0259] As mentioned above Fig.15 As described, a desired position of a display that outputs image light may be determined based at least in part on the user's eye gesture. For example, as described above, the desired position may correspond to an intersection in a viewing area of ​​the display (e.g., on a waveguide) of light from a 3D position of a virtual object in the user's environment to a pupil of the user's eye.

[0260] Alternatively, for at least some content (e.g., displaying system menu items and alerts, or other content that is not anchored to a location in the physical world), the area of ​​the display that outputs image light may be selected based simply on the user's eye gesture. For example, displaying a system menu may be simply tied to the user's eye gesture so that the menu remains within the user's field of view regardless of the eye gesture.

[0261] Fig.16 3100 is an example of a flow chart of an out-coupling position determination process 3100 using eye gestures. At block 3102, a display system may receive one or more eye images from an imaging system. The imaging system may include an inward-facing imaging system, such as one or more cameras associated with a display of the display system. The imaging system may be configured to image the left eye and / or right eye of a user. As described below with reference to Fig.17A As described in detail, an eye image may include an image of an eye including one or more features of the eye, such as an eyelid, sclera, iris, and pupil.

[0262] At block 3104, the display system may determine an eye pose of the user's eyes. For example, the display system may analyze one or more eye images to determine the eye pose. In some examples, the eye pose may include a rotation center of the user's eyes, a gaze direction, other orientation of the user's eyes, or some combination thereof. Fig. 17B As described in detail in 32C, the display system can utilize one or more eye tracking modules to analyze images to recognize eye gestures.

[0263] At block 3106, the display system may apply the eye gesture to determine a display area associated with the eye gesture to render virtual content. For example, the display system may utilize one or more engines to calculate a display position on a display to render virtual content at a different position relative to the pupil of the eye, for example, along a vector that intersects the rotation center or the viewing angle center of the user's eye, as referenced by Fig. 17B and 32C described in detail.

[0264] Example eye image for eye tracking

[0265] Fig.17AAn image of an eye 3200 is shown having an eyelid 3204, a sclera 3208 (the "white" of the eye), an iris 3212, and a pupil 3216. Curve 3216a shows the pupil boundary between the pupil 3216 and the iris 3212, and curve 3212a shows the marginal boundary between the iris 3212 and the sclera 3208. The eyelid 3204 includes an upper eyelid 3204a and a lower eyelid 3204b. The eye 3200 is shown in a natural resting posture (e.g., where the user's face and gaze are both directed toward a distant object directly in front of the user). The natural resting posture of the eye 3200 may be indicated by a natural resting direction 3200, which is a direction orthogonal to the surface of the eye 3200 when in the natural resting posture (e.g., directly in front of the user). Fig.17A ) outside the plane of the eye 3200 shown in , and in this example, located at the center of the pupil 3216.

[0266] As the eye 3200 moves to look toward different objects, the eye pose will change relative to the natural resting direction 3200. The current eye pose can be determined with reference to the eye pose direction 3204, which is oriented orthogonal to the surface of the eye (and centered at the pupil 3216) but in the direction of the object the eye is currently pointing at. Fig.17A , the pose of the eye 3200 can be expressed as two angular parameters indicating the azimuthal deflection and zenithal deflection of the eye pose direction 3204 of the eye, both relative to the natural resting direction 3200 of the eye. For purposes of illustration, these angular parameters may be represented as θ (azimuthal deflection determined from a fiducial azimuth) and φ (zenithal deflection, sometimes also referred to as polar deflection). In some embodiments, the angular roll of the eye around the eye pose direction 3204 may be included in the determination of the eye pose, and the angular roll may be included in the following analysis. In other embodiments, other techniques for determining eye pose may use, for example, a pitch, yaw, and optionally a roll system.

[0267] The eye image may be obtained from the video using any suitable process, for example, using a video processing algorithm that can extract an image from one or more consecutive frames. The pose of the eye may be determined from the eye image using a variety of eye tracking techniques. For example, the eye pose may be determined by considering the lens effect of the cornea on a provided light source. Any suitable eye tracking technique may be used to determine the eye pose.

[0268] Eye Tracking System Example

[0269] Fig. 17B A schematic diagram of a wearable system 3230 including an eye tracking system is shown. In at least some implementations, the wearable system 3230 may include components located in a head-mounted unit 3232 and components located in a non-head-mounted unit 3234. The non-head-mounted unit 3234 may be, for example, a belt-mounted component, a hand-held component, a component in a backpack, a remote component, etc. Incorporating some components of the wearable system 3230 into the non-head-mounted unit 3234 may help reduce the size, weight, complexity, and cost of the head-mounted unit 3232. In some embodiments, some or all of the functions described as being performed by one or more components of the head-mounted unit 3232 and / or the non-head-mounted unit 3234 may be provided by one or more components included elsewhere in the wearable system 3230. For example, some or all of the functions described below associated with the CPU 3242 of the head-mounted unit 3232 may be provided by the CPU 3246 of the non-head-mounted unit 3234, and vice versa. In some examples, some or all of such functionality may be provided by peripheral devices of the wearable system 3230. Additionally, in some embodiments, the wearable system 3230 may be provided by one or more cloud computing devices or other remote computing devices in a manner similar to that described above with reference to FIG. Fig.9D In addition, it should be understood that the wearable system 3230 can correspond to the display system 60 ( Fig.9D ), and in some implementations, the head-mounted unit 3232 and the non-head-mounted unit 3234 may correspond to the local data processing module 140 and the remote processing module 150, respectively.

[0270] like Fig. 17BAs shown, the wearable system 3230 may include an eye tracking system that includes a camera 3270 that captures an image of a user's eye 3240. If desired, the eye tracking system may also include light sources 3260a and 3260b (such as light emitting diodes "LEDs" that may emit visible light in some implementations or invisible light, such as infrared light, in other implementations). The light sources 3260a and 3260b may produce flashes (e.g., reflected off the user's eyes that appear in the image of the eyes captured by the camera 3270). The position of the light sources 3260a and 3260b relative to the camera 3270 may be known, and therefore, the position of the flashes within the image captured by the camera 3270 may be used to track the user's eyes (as will be discussed in more detail below). In at least one implementation, there may be one light source 326 and one camera 3270 associated with one eye 3240 of the user. In another implementation, there may be one light source 326 and one camera 3270 associated with both eyes 3240 of the user. In other implementations, there may be one or more cameras 3270 and one or more light sources 326 associated with one or each eye 3240 of the user. As one specific example, there may be two light sources 3260a and 3260b associated with each eye 3240 of the user and one or more cameras 3270. As another example, there may be three or more light sources (e.g., light sources 3260a and 3260b) associated with each eye 3240 of the user and one or more cameras 3270. In some implementations described herein, two or more cameras may be used to image a given eye.

[0271] The eye tracking module 3244 may receive images from the eye tracking camera 3270 and may analyze the images to extract various pieces of information. As examples, the eye tracking module 3244 may detect the user's eye pose, the three-dimensional position of the user's eyes relative to the eye tracking camera 3270 (and relative to the head mounted unit 3232), the direction in which one or both of the user's eyes 3240 are focused, the user's convergence depth (e.g., the depth at which the user is focused relative to the user), the position of the user's pupil, the position of the user's cornea and / or corneal sphere, the center of rotation of one or each of the user's eyes, and the center of view of one or each of the user's eyes, or any combination thereof. The eye tracking module 3244 may use the following in combination with Fig. 17C The techniques described in this paper are used to extract such information. Fig. 17B As shown, in some implementations, the eye tracking module 3244 can be a software module implemented using the CPU 3242 in the head mounted unit 3232.

[0272] although Fig. 17BOne camera 3270 is shown imaging the eye, but in some implementations, as discussed herein, multiple cameras may image the eye and be used for measurements, such as corneal center and / or rotation center measurements, or for eye tracking or other purposes.

[0273] Data from the eye tracking module 3244 can be provided to other components in the wearable system. For example, such data can be sent to components in the non-head-mounted unit 3234, such as the CPU 3246 including software modules for the light field rendering controller 3248 and the registration observer 3250.

[0274] The rendering controller 3248 may use information from the eye tracking module 3244 to adjust images displayed to the user by a rendering engine 3252 (e.g., a rendering engine that may be a software module in the GPU 3250 and that may provide images to the display 220). As an example, the rendering controller 3248 may adjust images displayed to the user based on the user's center of rotation or the center of view. In particular, the rendering controller 3248 may use information about the user's center of view to simulate a rendering camera (e.g., simulate collecting images from the user's view), and may adjust images displayed to the user based on the simulated rendering camera.

[0275] A "rendering camera", sometimes also referred to as a "pinhole perspective camera" (or simply "perspective camera") or a "virtual pinhole camera" (or simply "virtual camera"), is a simulated camera used in rendering virtual image content that may come from a database of objects in a virtual world. Objects may have positions and orientations relative to a user or wearer and possibly relative to real objects in the environment surrounding the user or wearer. In other words, the rendering camera may represent a perspective view within a rendering space from which the user or wearer will view the 3D virtual content (e.g., virtual objects) of the rendering space. The rendering camera may be managed by a rendering engine to render virtual images based on a database of virtual objects to be presented to the eye. The virtual image may be rendered as if it were taken from the user's or wearer's perspective. For example, a virtual image may be rendered as if it were captured by a pinhole camera (corresponding to a "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, skew / distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components relative to the virtual world). A virtual image is captured from the perspective of a rendering camera having a position and orientation of such camera (e.g., extrinsic parameters of the rendering camera). Thus, the system may define and / or adjust intrinsic and extrinsic rendering camera parameters. For example, the system may define a particular set of extrinsic rendering camera parameters such that a virtual image may be rendered as if it were captured from the perspective of a camera having a particular position relative to the eyes of a user or wearer to provide an image that appears to be from the perspective of the user or wearer. The system may then dynamically adjust the extrinsic rendering camera parameters on-the-fly to maintain registration with the particular position. Similarly, intrinsic rendering camera parameters may be defined and dynamically adjusted over time. In some embodiments, an image is rendered as if it were captured from the perspective of a camera having an aperture (e.g., a pinhole) at a particular position relative to the eyes of the user or wearer (e.g., a center of view or a center of rotation or elsewhere).

[0276] In some implementations, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and another rendering camera for the user's right eye because the user's eyes are physically separated from each other and, therefore, consistently positioned at different locations. Thus, in at least some embodiments, virtual content rendered from the perspective of a rendering camera associated with the viewer's left eye may be presented to the user through an eyepiece on the left side of a head-mounted display (e.g., head-mounted unit 3232), and virtual content rendered from the perspective of a rendering camera associated with the user's right eye may be presented to the user through an eyepiece on the right side of such head-mounted display. Further details discussing the creation, adjustment, and use of rendering cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled “METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION,” which is expressly incorporated herein by reference in its entirety for all purposes.

[0277] In some examples, one or more modules (or components) of the system 3230 (e.g., light field rendering controller 3248, rendering engine 3250, etc.) may determine the position and orientation of the rendering camera within the rendering space based on the position and orientation of the user's head and eyes (e.g., determined based on head pose and eye tracking data, respectively). That is, the system 3230 may effectively map the position and orientation of the user's head and eyes to a specific position and angular position within the 3D virtual environment, place and orient the rendering camera at the specific position and angular position within the 3D virtual environment, and render virtual content for the user as captured by the rendering camera. Further details discussing the real world to virtual world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE," which is expressly incorporated herein by reference in its entirety for all purposes. As an example, the rendering controller 3248 may adjust the depth of a displayed image by selecting which depth plane (or multiple depth planes) to utilize to display the image at any given time. In some embodiments, such depth plane switching can be performed by adjusting one or more intrinsic rendering camera parameters. For example, when performing depth plane switching or adjustment, the light field rendering controller 3248 can adjust the focal length of the rendering camera. As described in further detail below, the depth plane can be switched based on the user's determined convergence or gaze depth.

[0278] The registration observer 3250 may use information from the eye tracking module 3244 to identify whether the head mounted unit 3232 is correctly positioned on the user's head. As an example, the eye tracking module 3244 may provide eye position information, such as the location of the center of rotation of the user's eyes, which indicates the three-dimensional location of the user's eyes relative to the camera 3270 and the head mounted unit 3232, and the eye tracking module 3244 may use the position information to determine whether the display 220 is correctly aligned in the user's field of view, or whether the head mounted unit 3232 (or headset) has slipped or is misaligned with the user's eyes. As an example, the registration observer 3250 can determine: whether the head mounted unit 3232 has slid down along the bridge of the user's nose, thereby moving the display 220 away from and below the user's eyes (which may be undesirable); whether the head mounted unit 3232 has moved up along the bridge of the user's nose, thereby moving the display 220 closer to and up from the user's eyes; whether the head mounted unit 3232 has moved left or right relative to the bridge of the user's nose, whether the head mounted unit 3232 has been raised above the bridge of the user's nose, or whether the head mounted unit 3232 has moved away from a desired position or range of positions in these or other ways. In general, the registration observer 3250 can determine whether the head mounted unit 3232 in general, and the display 220 in particular, are properly positioned in front of the user's eyes. In other words, the registration observer 3250 can determine whether the left display in the display system 220 is properly aligned with the user's left eye, and whether the right display in the display system 220 is properly aligned with the user's right eye. The registration observer 3250 can determine whether the head mounted unit 3232 is properly positioned by determining whether the head mounted unit 3232 is positioned and oriented within a desired range of positions and / or orientations relative to the user's eyes.

[0279] In at least some implementations, the registration viewer 3250 can generate user feedback in the form of an alert, message, or other content. Such feedback can be provided to the user to inform the user of any misalignment of the head mounted unit 3232, as well as optional feedback on how to correct the misalignment (such as suggestions to adjust the head mounted unit 3232 in a particular way).

[0280] Examples of registration observation and feedback techniques that may be used by the registration observer 3250 are described in U.S. Patent No. 10,573,042, entitled “PERIOCULAR TEST FOR MIXED REALITY CALIBRATION,” issued on February 25, 2020, and U.S. Patent Application Publication No. 2019 / 0222830, entitled “DISPLAY SYSTEMS AND METHODSFOR DETERMINING REGISTRATION BETWEEN A DISPLAY AND A USER'S EYES,” published on July 18, 2019, the entire contents of which are incorporated herein by reference.

[0281] Example of an eye tracking module for eye pose determination

[0282] Fig. 17C 614 is a block diagram of an example eye tracking module 614 that can be used to determine eye gestures. Fig. 17C As shown, the eye tracking module 614 may include various submodules, may provide various outputs, and may utilize various available data to track the user's eyes. As an example, the eye tracking module 614 may utilize available data, including eye tracking extrinsics and intrinsics, such as the geometric arrangement of the eye tracking camera 3270 relative to the light source 326 and the head mounted unit 602; assumed eye size 3304, such as the typical distance between the center of corneal curvature of the user and the average rotation center of the user's eye is about 4.7 mm, or the typical distance between the rotation center of the user and the center of visual angle; and per-user calibration data 3306, such as the interpupillary distance of a particular user. Other examples of extrinsic parameters, intrinsic parameters, and other information that may be used by the eye tracking module 614 are described in U.S. patent application Ser. No. 15 / 497,726, filed on April 26, 2017 (attorney docket No. MLEAP.023A7), and entitled “IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE,” issued on May 21, 2019 as U.S. Patent No. 10,296,792, the entire contents of which are incorporated herein by reference.

[0283] The image pre-processing module 3280 may receive an image from an eye camera, such as the eye camera 3270, and may perform one or more pre-processing (e.g., conditioning) operations on the received image. As an example, the image pre-processing module 3280 may apply a Gaussian blur to the image, may downsample the image to a lower resolution, may apply an unsharp mask, may apply an edge sharpening algorithm, or may apply other suitable filters to aid in the post-detection, localization, and marking of glints, pupils, or other features in the image from the eye camera 3270. The image pre-processing module 3280 may apply a low-pass filter or a morphological filter (e.g., an opening filter) that may remove artifacts such as from the pupil boundary 516a (see Figure 5 ) high frequency noise, thereby removing noise that may hinder pupil and flash determination. The image preprocessing module 3280 can output the preprocessed image to the pupil identification module 3282 and the flash detection and marking module 3284.

[0284] The pupil identification module 3282 may receive pre-processed images from the image pre-processing module 3280 and may identify regions in those images that include the user's pupils. In some implementations, the pupil identification module 3282 may determine coordinates of a location of the user's pupils or coordinates of a center or centroid in the eye tracking images from the camera 3270. In at least some implementations, the pupil identification module 3282 may identify contours in the eye tracking images (e.g., contours of a pupil-iris boundary), identify contour moments (e.g., centroids), apply starburst pupil detection and / or Canny edge detection algorithms, reject outliers based on intensity values, identify sub-pixel boundary points, correct eye camera distortions (e.g., distortions in images captured by the eye camera 3270), apply a random sample consensus (RANSAC) iterative algorithm to fit an ellipse to a boundary in the eye tracking images, apply a tracking filter to the images, and identify sub-pixel image coordinates of the centroid of the user's pupils. The pupil identification module 3282 may output pupil identification data, which may indicate which areas of the pre-processed image module 3282 are identified as showing the user's pupils, to the flash detection and labeling module 3284. The pupil identification module 3282 may provide the 2D coordinates of the user's pupils within each eye tracking image (e.g., the 2D coordinates of the centroid of the user's pupils) to the flash detection module 3284. In at least some implementations, the pupil identification module 3282 may also provide the same kind of pupil identification data to the coordinate system normalization module 3288.

[0285] The pupil detection technology that can be utilized by the pupil identification module 3282 is described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, the entire contents of each of which are incorporated herein by reference in their entirety.

[0286] The flash detection and marking module 3284 may receive the preprocessed image from module 3280 and the pupil identification data from module 3282. The flash detection module 3284 may use this data to detect and / or identify flashes (e.g., light from light source 326 reflecting off the user's eye) in the area of ​​the preprocessed image showing the user's pupil. As an example, the flash detection module 3284 may search for bright areas (sometimes referred to herein as "blobs" or local intensity maxima) within the eye tracking image that are near the user's pupil. In at least some implementations, the flash detection module 3284 may rescale (e.g., enlarge) the pupil ellipse to include the additional flashes. The flash detection module 3284 may filter the flashes by size and / or intensity. The flash detection module 3284 may also determine the 2D location of each flash within the eye tracking image. In at least some examples, the flash detection module 3284 may determine the 2D location of the flash relative to the user's pupil, which may also be referred to as a pupil flash vector. The flash detection and marking module 3284 can mark the flashes and output the pre-processed image with the marked flashes to the 3D cornea center estimation module 3286. The flash detection and marking module 3284 can also pass data, such as the pre-processed image from module 3280 and the pupil identification data from module 3282. In some implementations, the flash detection and marking module 3284 can determine which light source (e.g., multiple light sources from a system including infrared light sources 326a and 326b) produced each identified flash. In these examples, the flash detection and marking module 3284 can mark the flashes with information identifying the associated light source and output the pre-processed image with the marked flashes to the 3D cornea center estimation module 3286.

[0287] Pupil and glint detection performed by modules such as modules 3282 and 3284 may use any suitable technique. As an example, edge detection may be applied to an eye image to identify glints and pupils. Edge detection may be applied by various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector may be applied to an image to detect edges such as in lines of the image. Edges may include points located along the line corresponding to local maximum derivatives. For example, a Canny edge detector may be used to locate pupil boundary 516a (see Figure 5). Once the position of the pupil has been determined, various image processing techniques may be used to detect the "pose" of the pupil 116. Determining the eye pose of an eye image may also be referred to as detecting the eye pose of an eye image. A pose may also be referred to as a gaze, pointing direction, or orientation of the eye. For example, the pupil may be looking to the left at an object, and the pose of the pupil may be classified as a left pose. Other methods may be used to detect the position of the pupil or glint. For example, a Canny edge detector may be used to place concentric rings in an eye image. As another example, an integral differential operator may be used to find the pupil or limbal boundary of the iris. For example, a Daugman integral differential operator, a Hough transform, or other iris segmentation techniques may be used to return a curve that estimates the boundary of the pupil or iris.

[0288] The 3D cornea center estimation module 3286 may receive pre-processed images including detected flash data and pupil identification data from modules 3280, 3282, 3284. The 3D cornea center estimation module 3286 may use these data to estimate the 3D position of the user's cornea. In some implementations, the 3D cornea center estimation module 3286 may estimate the 3D position of the center of corneal curvature of the eye or the user's corneal sphere, for example, the center of an imaginary sphere having a surface portion generally coextensive with the user's cornea. The 3D cornea center estimation module 3286 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 3288, the optical axis determination module 3292, and / or the light field rendering controller 618. More details of the operation of the 3D cornea center estimation module 3286 are provided herein in conjunction with Figures 11-16C. U.S. patent application Ser. No. 15 / 497,3296 (attorney docket number MLEAP.023A7), filed on April 26, 2017 (incorporated herein by reference in its entirety), discusses example techniques for estimating the position of eye features (e.g., the cornea or corneal sphere), which may be used by the 3D cornea center estimation module 3286 and other modules in the wearable system of the present disclosure.

[0289] A coordinate system normalization module 3288 may optionally (as shown in dashed outline) be included in the eye tracking module 614. The coordinate system normalization module 3288 may receive an estimated 3D coordinate indicating the center of the user's cornea (and / or the center of the user's corneal sphere) from the 3D cornea center estimation module 3286, and may also receive data from other modules. The coordinate system normalization module 3288 may normalize the eye camera coordinate system, which may help compensate for slippage of the wearable device (e.g., slippage of the head mounted component from its normal resting position on the user's head, which may be identified by the registration observer 620). The coordinate system normalization module 3288 may rotate the coordinate system to align the z-axis (e.g., vergence depth axis) of the coordinate system with the center of the cornea (e.g., as indicated by the 3D cornea center estimation module 3286), and may translate the camera center (e.g., the origin of the coordinate system) to a predetermined distance (such as 30 mm) from the center of the cornea (e.g., the module 3288 may zoom in or out of the eye tracking image depending on whether the eye camera 3270 is determined to be closer or farther than the predetermined distance). Through this normalization process, the eye tracking module 614 is able to establish consistent orientation and distance in the eye tracking data, relatively independent of changes in the positioning of the headset on the user's head. The coordinate system normalization module 3288 may provide the 3D coordinates of the center of the cornea (and / or cornea sphere), pupil identification data, and pre-processed eye tracking images to the 3D pupil center locator module 3290.

[0290] The 3D pupil center locator module 3290 may receive data in a normalized or non-normalized coordinate system, including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil position data, and pre-processed eye tracking images. The 3D pupil center locator module 3290 may analyze such data to determine the 3D coordinates of the center of the user's pupil in a normalized or non-normalized eye camera coordinate system. The 3D pupil center locator module 3290 can determine the location of the user's pupil in three dimensions based on the 2D location of the pupil centroid (as determined by module 3282), the 3D location of the corneal center (as determined by module 3286), the assumed eye size 3304 (e.g., the size of the corneal sphere of a typical user and the typical distance from the corneal center to the pupil center), and the optical properties of the eye (e.g., the refractive index of the cornea relative to the refractive index of air), or any combination of these. Techniques for estimating the location of eye features such as the pupil that can be utilized by the 3D pupil center locator module 3290 and other modules in the wearable system of the present disclosure are discussed in U.S. patent application Ser. No. 15 / 497,3296 filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0291] The optical axis determination module 3292 may receive data from modules 3286 and 3290 indicating 3D coordinates of the center of the user's cornea and the user's pupil. Based on such data, the optical axis determination module 3292 may identify a vector from the location of the center of the cornea (e.g., from the center of the corneal sphere) to the center of the user's pupil, which may define the optical axis of the user's eye. As an example, the optical axis determination module 3292 may provide output specifying the user's optical axis to modules 3294, 3298, 3310, and 3312.

[0292] The center of rotation (CoR) estimation module 3294 may receive data from module 3292, the data including parameters of the optical axis of the user's eye (e.g., data indicating the direction of the optical axis in a coordinate system having a known relationship with the head mounted unit 602). For example, the CoR estimation module 3294 may estimate the center of rotation of the user's eye. The center of rotation may indicate the point about which the user's eye rotates when the user's eye rotates left, right, up, and / or down. Although the eye may not rotate perfectly about a singular point, assuming a singular point may be sufficient. In at least some implementations, the CoR estimation module 3294 may estimate the center of rotation of the eye by moving a specific distance from the center of the pupil (identified by module 3290) or the center of curvature of the cornea (identified by module 3286) along the optical axis (identified by module 3292) toward the retina. The specific distance may be the assumed eye size 3304. As an example, the specific distance between the center of curvature of the cornea and the CoR may be approximately 4.7 mm. This distance may be altered for a particular user based on any relevant data including the user's age, gender, vision prescription, other relevant characteristics, etc.

[0293] In at least some implementations, the CoR estimation module 3294 may refine its estimate of the center of rotation of each of the user's eyes over time. For example, over time, the user will eventually rotate their eyes (look elsewhere, look at something closer, look at something farther away, or look left, right, up, or down at some point), causing the optical axis of each of their eyes to shift. The CoR estimation module 3294 may then analyze the two (or more) optical axes identified by module 3292 and locate the 3D intersection point of those optical axes. The CoR estimation module 3294 may then determine that the center of rotation is located at that 3D intersection point. Such techniques may provide an estimate of the center of rotation whose accuracy improves over time.

[0294] Various techniques may be employed to improve the accuracy of the CoR estimation module 3294 and the determined CoR positions for the left and right eyes. As an example, the CoR estimation module 3294 may estimate the CoR by finding the average intersection of the optical axes determined over time for various different eye poses. As additional examples, the module 3294 may filter or average the estimated CoR positions over time, may calculate a moving average of the estimated CoR positions over time, and / or may apply a Kalman filter and the known dynamics of the eyes and eye tracking system to estimate the CoR positions over time. In some implementations, a least squares method may be employed to determine one or more intersections of the optical axes. In such an implementation, the system may identify the positions at a given point in time where the sum of the squared distances to a given set of optical axes is reduced or minimized as optical axis intersections. As a specific example, module 3294 may calculate a weighted average of the determined optical axis intersection point and an assumed CoR position (e.g., 4.7 mm from the center of corneal curvature of the eye) such that the determined CoR may slowly drift over time from the assumed CoR position (e.g., 4.7 mm behind the center of corneal curvature of the eye) to a slightly different position within the user's eye as eye tracking data for the user is obtained, thereby enabling refinement of the CoR position per user.

[0295] Under ideal conditions, the 3D position of the user's eyes relative to the true CoR of the HMD should change over time by a negligible or minimal amount as the user moves their eyes (e.g., as the user's eyes rotate about their center of rotation). In other words, for a given set of eye movements, the 3D position of the user's eyes' true CoR (relative to the HMD) should be assumed to change less over time than any other point along the optical axis of the user's eyes. Thus, it can be seen that the farther a point along the optical axis is from the true CoR of the user's eyes, the greater the variation or variance its 3D position exhibits over time as the user moves his eyes. In some implementations, the CoR estimation module 3294 and / or other submodules of the eye tracking module 614 can exploit this statistical relationship to improve CoR estimation accuracy. In such an implementation, the CoR estimation module 3294 and / or other submodules of the eye tracking module 614 can refine their estimates of the CoR 3D position over time by identifying changes in CoR estimates that have small variations (e.g., low variance or standard deviation).

[0296] As a first example, and in implementations where the CoR estimation module 3294 estimates the CoR based on the intersection of multiple different optical axes (each optical axis is associated with a different direction of user viewing), the CoR estimation module 3294 can exploit this statistical relationship by introducing a common offset to the direction of each optical axis (e.g., offsetting each axis by a uniform amount) and determining whether the offset optical axes intersect each other in an intersection point with small variation (e.g., low variance or standard deviation) (the true CoR should have low variance). This can correct for minor systematic errors in calculating the optical axis directions and help refine the estimated position of the CoR to be closer to the true CoR.

[0297] As a second example, and in implementations where the CoR estimation module 3294 estimates the CoR by moving a specific distance (e.g., the distance between the CoR and the center of corneal curvature) along the optical axis (or other axis), the system can change, optimize, adjust, or otherwise adjust the specific distance between the CoR and the center of corneal curvature over a period of time (e.g., for a large number of eye images captured at different times) in a manner that reduces or minimizes variation (e.g., variance and / or standard deviation) in the estimated CoR position. For example, if the CoR estimation module 3294 initially uses a specific distance value of 4.7 mm (from the center of corneal curvature and along the optical axis) to obtain CoR position estimates, but the true CoR for a given user's eye is located 4.9 mm behind the center of corneal curvature of the eye (along the optical axis), then the initial set of CoR position estimates obtained by the CoR estimation module 3294 may exhibit a relatively large amount of variation (e.g., variance or standard deviation). In response to detecting such a relatively large amount of variation (e.g., variance or standard deviation), the CoR estimation module 3294 can seek and identify one or more points along the optical axis having a smaller amount of variation (e.g., variance or standard deviation), can identify 4.9 mm as the distance having the smallest variation (e.g., variance or standard deviation), and thereby adjust the particular distance value used to 4.9 mm.

[0298] The CoR estimation module 3294 may seek an alternative CoR estimate with a smaller amount of variation (e.g., variance and / or standard deviation) in response to detecting that the current CoR estimate has a relatively large amount of variation (e.g., variance or standard deviation), or may seek an alternative CoR estimate with a minimum amount of variation (e.g., variance or standard deviation) as a matter of course after obtaining the initial CoR estimate. In some examples, such optimization / adjustment may occur gradually over a period of time, while in other examples, such optimization / adjustment may be performed during an initial user calibration session. In examples where such a process is performed during a calibration process, the CoR estimation module 3294 does not initially subscribe to / adhere to any assumed specific distance, but may instead collect a set of eye tracking data over a period of time, perform statistical analysis on the set of eye tracking data, and determine, based on the statistical analysis, a specific distance value that produces a CoR position estimate with a minimum possible amount of variation (e.g., variance or standard deviation) (e.g., a global minimum).

[0299] The interpupillary distance (IPD) estimation module 3296 may receive data from the CoR estimation module 3294 indicating the estimated 3D position of the center of rotation of the user's left and right eyes. The IPD estimation module 3296 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Typically, when the user is looking at optical infinity (e.g., the optical axes of the user's eyes are substantially parallel to each other), the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils, which is a typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules in the wearable system. For example, the user's IPD may be provided to the registration observer 620 and used to assess the degree to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are properly separated according to the user's IPD, based on the user's IPD). As another example, the user's IPD may be provided to the vergence depth estimation module 3298 and used to determine the user's vergence depth. Module 3296 may employ various techniques, such as those discussed in conjunction with CoR estimation module 3294, to improve the accuracy of the estimated IPD. As an example, IPD estimation module 3294 may apply filtering, averaging over time, weighted averaging including assumed IPD distances, Kalman filters, etc. as part of estimating the user's IPD in an accurate manner.

[0300] The vergence depth estimation module 3298 may receive data from various modules and submodules in the eye tracking module 614 (e.g., in conjunction with Fig. 17C). Specifically, the vergence depth estimation module 3298 may employ data indicating an estimated 3D position of a pupil center (e.g., as provided by module 3290 described above), one or more determined optical axis parameters (e.g., as provided by module 3292 described above), an estimated 3D position of a rotation center (e.g., as provided by module 3294 described above), an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the rotation centers) (e.g., as provided by module 3296 described above), and / or one or more determined optical axis and / or visual axis parameters (e.g., as provided by module 3292 and / or module 3310 described below). The vergence depth estimation module 3298 may detect or otherwise obtain a measure of the user's vergence depth, which may be a distance from the user at which the user's eyes are focusing. For example, when the user is looking at an object three feet in front of them, the vergence depth of the user's left and right eyes is three feet; and when the user is looking at a distant landscape (e.g., the optical axes of the user's eyes are substantially parallel to each other, so that the distance between the centers of the user's pupils may be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the vergence depth of the user's left and right eyes is infinite. In some embodiments, the vergence depth estimation module 3298 may utilize data indicating the estimated centers of the user's pupils (e.g., as provided by module 3290) to determine the 3D distance between the estimated centers of the user's pupils. The vergence depth estimation module 3298 may obtain a measure of the vergence depth by comparing this determined 3D distance between the pupil centers with an estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as indicated by module 3296 described above). In addition to the 3D distance between the pupil centers and the estimated IPD, the vergence depth estimation module 3298 may utilize known, assumed, estimated, and / or determined geometry to calculate the vergence depth. As an example, the module 3298 may combine the 3D distance between the centers of the pupils, the estimated IPD, and the 3D CoR position in a triangulation calculation to estimate (e.g., determine) the vergence depth of the user. In practice, evaluating the 3D distance between the centers of the pupils so determined according to the estimated IPD may be used to indicate a measure of the user's current depth of focus relative to optical infinity. In some examples, the vergence depth estimation module 3298 may simply receive or access data indicating an estimated 3D distance between the estimated centers of the user's pupils in order to obtain such a measure of the vergence depth. In some implementations, the vergence depth estimation module 3298 may estimate the vergence depth by comparing the user's left and right optical axes. In particular, the vergence depth estimation module 3298 may estimate the vergence depth by locating a distance from the user at which the user's left and right optical axes intersect (or the projection of the user's left and right optical axes on a plane, such as a horizontal plane, intersect).Module 3298 can utilize the user's IPD in this calculation by setting the zero depth to the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some implementations, vergence depth estimation module 3298 can determine vergence depth by triangulating eye tracking data with known or derived spatial relationships.

[0301] In some implementations, the vergence depth estimation module 3298 may estimate the vergence depth of a user based on the intersection of the user's visual axes (rather than their optical axes), which may provide a more accurate indication of the distance at which the user is focused. In at least some implementations, the eye tracking module 614 may include an optical axis to visual axis mapping module 3310. As discussed in further detail in conjunction with FIG. 10, the user's optical axis and visual axis are typically not aligned. The visual axis is the axis along which a person looks, while the optical axis is defined by the center of the person's lens and pupil and may pass through the center of the person's retina. In particular, the user's visual axis is typically defined by the location of the user's fovea, which may be offset from the center of the user's retina, causing the optical axis and the visual axis to be different. In at least some of these implementations, the eye tracking module 614 may include an optical axis to visual axis mapping module 3310. The optical axis to visual axis mapping module 3310 can correct for differences between the user's optical axis and the visual axis and provide information about the user's visual axis to other components in the wearable system (e.g., the vergence depth estimation module 3298 and the light field rendering controller 618). In some examples, the module 3310 can use an assumed eye size 3304, including a typical offset of about 5.2° inward (nasally, toward the user's nose) between the optical axis and the visual axis. In other words, the module 3310 can shift the user's left optical axis (nasally) by 5.2° to the right of the nose, and the user's right optical axis (nasally) by 5.2° to the left of the nose, in order to estimate the direction of the user's left and right optical axes. In other examples, the module 3310 can utilize per-user calibration data 3306 when mapping the optical axis (e.g., as indicated by the module 3292 described above) into the visual axis. As additional examples, module 3310 may shift the user's optical axis nasally between 4.0° and 6.5°, between 4.5° and 6.0°, between 5.0° and 5.4°, and so on, or any range formed by any of these values. In some arrangements, module 3310 may apply the shift based at least in part on characteristics of a particular user (e.g., their age, gender, vision prescription, or other relevant characteristics), and / or may apply the shift based at least in part on a calibration process for a particular user (e.g., to determine an optical-optical axis offset for a particular user). In at least some implementations, module 3310 may also shift the origin of the left and right optical axes to correspond to the user's CoP (determined by module 3312) rather than to the user's CoR.

[0302] The optional center of view (CoP) estimation module 3312 (when provided) can estimate the location of the user's left and right centers of view (CoP). The CoP can be a useful location for a wearable system, and in at least some implementations, is a location directly in front of the pupil. In at least some implementations, the CoP estimation module 3312 can estimate the location of the user's left and right centers of view based on the 3D location of the user's pupil center, the 3D location of the user's corneal curvature center, or such suitable data or any combination. As an example, the user's CoP can be approximately 5.01 mm in front of the center of the corneal curvature (e.g., 5.01 mm from the center of the corneal sphere in a direction toward the cornea of ​​the eye and along the optical axis) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical axis or visual axis. The user's center of view can be directly in front of the center of their pupil. For example, the user's CoP can be less than approximately 2.0 mm from the user's pupil, less than approximately 1.0 mm from the user's pupil, or less than approximately 0.5 mm from the user's pupil, or any range between these values. As another example, the center of view may correspond to a location within the anterior chamber of the eye. As other examples, the CoP may be between 1.0 mm and 2.0 mm, approximately 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm from the user's pupil.

[0303] The center of view described herein (as a potential desired location for the pinhole of the rendering camera and an anatomical location in the user's eye) can be a location for reducing and / or eliminating undesirable parallax shifts. In particular, the optical system of the user's eye closely approximates a theoretical system formed by a pinhole in front of a lens, projected onto a screen, with the pinhole, lens, and screen roughly corresponding to the user's pupil / iris, lens, and retina, respectively. In addition, it can be expected that when two point light sources (or objects) at different distances from the user's eye are rigidly rotated around the opening of the pinhole (e.g., rotated along a radius of curvature equal to their distance from the opening of the pinhole), there is little or no parallax shift. Therefore, it seems that the CoP should be located at the center of the pupil of the eye (and such a CoP can be used in some implementations). However, the human eye, in addition to the pinhole of the lens and pupil, also includes a cornea that imparts additional optical power to light propagating to the retina. Therefore, in the theoretical system described in this paragraph, the anatomical equivalent of the pinhole can be the area of ​​the user's eye located between the outer surface of the cornea of ​​the user's eye and the center of the pupil or iris of the user's eye. For example, the anatomical equivalent of the pinhole may correspond to an area within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP to such a location within the anterior chamber of the user's eye.

[0304] As described above, the eye tracking module 614 can provide data such as the estimated 3D position of the left and right eye rotation centers (CoR), the vergence depth, the left and right eye optical axes, the 3D position of the user's eyes, the 3D position of the user's left and right corneal curvature centers, the 3D position of the user's left and right pupil centers, the 3D position of the user's left and right visual angle centers, the user's IPD, etc. to other components in the wearable system (such as the light field rendering controller 618 and the registration observer 620). The eye tracking module 614 may also include other submodules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 may include a blink detection module that provides a sign or other alert whenever the user blinks, and a glance detection module that provides a sign or other alert whenever the user's eyes glance (e.g., quickly move the focus to another point).

[0305] Other methods of eye tracking and determining the center of rotation are possible. Thus, the eye tracking module 614 can be different. For example, in various implementations of the eye tracking module described below, an estimate of the center of rotation is determined based on multiple corneal curvature center values. In some implementations, for example, as described in reference Figure 17A-19 As discussed, the eye tracking module 614 may estimate the rotation center of the eye by determining the convergence or intersection between surface normal vectors that may fit to a surface of multiple corneal curvature centers for different eye poses. However, one or more features from the eye tracking module 614 described above or elsewhere herein may be included in other implementations of the eye tracking module.

[0306] Eye tracking technology is discussed in U.S. patent application publication No. 2019 / 0243558, entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS,” published on August 8, 2019, and international publication No. WO 2021 / 01166, entitled “EYE CENTER OF ROTATION DETERMINATION WITH ONE OR MORE EYE TRACKING CAMERAS,” published on January 21, 2021 (attorney case number MLEAP.256WO; prior to U.S. Provisional Application No. 62 / 874867 filed on July 16, 2019), and the entire disclosure of each of the above patent applications is incorporated herein by reference.

[0307] Example Light Injection System

[0308] As mentioned above Fig.15 As described, the display system may identify one or more outcoupling regions for receiving image light to be output. In addition, as described above, eye gestures may be determined to facilitate determining an appropriate outcoupling region for selecting to couple out image light. Once the appropriate one or more outcoupling regions are determined, the display system may be configured to determine an appropriate path for light to the one or more outcoupling regions. For example, functionally directing light to one or more specific light outcoupling regions may involve directing image light into incoupling regions uniquely associated with those outcoupling regions.

[0309] Fig.18 33 is a flow chart of an example light directing process 3320 that can be used to direct light to a determined outcoupling region of a display. At block 3322, the display system can determine an incoupling region to direct light to the display. For example, the incoupling region can include an incoupling region associated with a desired rendering location on the display. The desired rendering location can be based on, for example, a reference image. Fig.16 In some examples, the coupling region can be a region, group, array, or other combination of coupling optical features (such as coupling gratings). Fig.19 As shown, an incoupling region of a display may include an arrangement of incoupling features arranged along a portion of the display. At block 3322, the display system may determine to direct light to the incoupling region so that the light is directed to an appropriate associated outcoupling region to form at least a portion of a perceived image on the display.

[0310] In box 3324, the display system can determine an image to be displayed to the user. The image can include a representation of virtual content to be displayed in the user's environment, and the image can include a portion of the virtual content or all of the virtual content. For example, the display system can receive an indication of a display tree. The display system can determine a representation of a tree to be projected into a determined coupling area or a representation of a portion of a tree to be projected into a determined coupling area. In some examples, the display system can break up an image or a representation of a virtual object that falls on multiple coupling-out areas of the viewing area of ​​the display. In order to show the full effect of the virtual object in this case, the display system can break up the image so that the light of a portion of the image projected into the first area of ​​the display includes an image of a portion associated with the area during rendering. In some examples, the image can include multiple portions of a representation of a virtual object that overlap by a certain amount to prevent rendering gaps from occurring when large objects are displayed across areas of the display.

[0311] At block 3326, the display system may send instructions to direct light to the incoupling region. As described above, the directed light may include image information for forming an image displayed from the outcoupling region of the display associated with the incoupling region. The display system may utilize one or more light injection systems to direct light to the incoupling region. For example, Fig.19 As shown, the display system may include a controller 3402, a light injection system 3404, a waveguide 3412, and a camera 3408. The controller 3402 may send instructions to the light injection system 3404 to direct or inject light 3410 into a selected coupling region, such as coupling regions 3414A, 3414B, or 3414C on the waveguide 3412. As described above, reference Figure 16-17C , the display system can determine which of the coupling-in regions 3414A, 3414B, 3414C or coupling-in optical elements 3416A, 3416B, 3416C associated with the coupling-in regions to inject light into based on, for example, the user's eye posture. The user's eye posture can be determined by the display system using images from one or more cameras 3408, which are configured to image at least one eye of the user. The injected light can propagate through one or more waveguides associated with the display to the coupling-out regions 3418A, 3418B, 3418C. One or more coupling-out optical elements in the coupling-out regions 3418A, 3418B, 3418C can output light toward the user's eye 3406. It should be understood that the coupling-in regions 3414A, 3414B, 3414C can correspond to any coupling-in region identified by reference numerals 1005, 700, 710, and 720 and various suffixes (e.g., see FIG. 1005 ). Fig.11A and 14B ).

[0312] The controller 3402 may include one or more software engines for performing the processes and functions described herein. The software engine may include programming instructions for performing the processes as discussed herein (and shown in the flow chart) to direct the light injection system to inject light into areas of the display. The engine may be executed by one or more hardware processors associated with the display system. The programming instructions may be stored in a memory associated with the display system. The programming instructions may be implemented in C, C++, JAVA, or any other suitable programming language. In some implementations, some or all of the portions of the controller 3402 including the engine may be implemented in dedicated circuits such as ASICs and FPGAs. Some aspects of the functionality of the controller 3402 may be performed remotely on a server over a network. Thus, the controller 3402 may be implemented using the instructions described above with reference to Figure 6 and 9D The described hardware components (eg, controller 560, processing module 140, and / or processing module 150) are implemented.

[0313] The light injection system may be any suitable system for selectively directing light to a desired area. Figures 20A-20D As shown, the light injection system may include a movable injection mechanism, such as a scanning fiber injector. Figure 21A-2 As shown in FIG1D , the light injection system may include one or more movable mirrors to guide the light. Additionally or alternatively, the light injection system may use one or more fixed injection systems, such as a system that forms images of different coupling regions at different locations of the SLM, as described in reference to FIG1D . Fig. 22 However, while details of an example light injection system may be discussed, the display system may utilize any number or combination of light injection systems to direct light to the determined incoupling regions.

[0314] Example of a mechanically driven light injection system

[0315] In some implementations, the light injection system 3404 ( Fig.19 ) can take the form of a movable injector system. The movable injector system can include one or more components that are configured to physically move the optical component to direct light to a desired location, such as the coupling-in regions 3414A, 3414B, 3414C on the waveguide 3412. Figures 20A-20D and 21A-21B show two examples of movable injection systems.

[0316] Figures 20A-20D An implementation of an example movable injection system 3500 is shown including at least one scanning fiber injector 3527. It should be understood that the movable injection system 3500 may correspond to the optical injection system 3404 ( Fig.19 ). The scanning fiber injector 3527 may include at least one light emitter 3528 and at least one actuator 3540 that moves the optical fiber 3542. Light 3501 from the end of the optical fiber 3542 may propagate through the optical device 3534 to the coupling region 3510A' or 3510B' in the waveguide 3526'. In some examples, such as Fig. 20A As shown, the optical device 3534 can be a collimating optical device configured to receive light fanned out from the optical fiber 3542 and collimate the light before the light propagates to the coupling region 3510A' or 3510B'.

[0317] The collimating optics can take various forms and can include one or more lens structures. In some examples, such as Fig. 20BAs shown, the optical device 3534 may include a lenticular lens including a plurality of lenses 3534L', 3534L" (eg, a lenslet array). Each of the plurality of lenses may be configured to provide collimated light to an associated incoupling region 3510A' or 3510B'. In some examples, such as Fig. 20C As shown, the optical device 3534 may include a collimating lens structure 3534C for collimating incident light and a projection lens structure 3534P for propagating the collimated light to the coupling region 3510A' or 3510B'. More generally, it will be appreciated that the optical device 3534 may include a combination of various lens structures to perform the function of providing collimated light to multiple coupling regions (e.g., coupling regions 3510A' or 3510B'). In some examples, such as Fig.20D As shown, the projection optical device 3534 may include a collimating lens structure 3534C and a projection lens structure 3534P, which includes a convex lens 3534P' and a bi-convex lens, which includes multiple lenses 3534L', 3534L". It should be understood that other combinations of lenses may also be considered.

[0318] Reference again Figures 20A-20D , the actuator 3540 can move the optical fiber 3542 along a predefined path (e.g., a circular path) at a known speed. Thus, the processing module 3638 can be configured to synchronize the propagation of light out of the end of the optical fiber 3542 with the movement of the optical fiber 3542 so that the image light propagates out of the optical fiber 3542 at a desired light output position, which in turn is synchronized with the image to be displayed.

[0319] In some implementations, the light source 3528 can be replaced by a virtual light source formed on the image plane of the light projection system. The light projection system may include an actuator that enables a light beam to be scanned on an area corresponding to the virtual light source on the image plane. As discussed herein, in order to simulate the ability to activate discrete light emitting areas of the light source 3528, the light output by the projection system is synchronized with the movement of the actuator so that the light is output to a desired position on the image plane at a specific time. Preferably, the actuator is capable of scanning the light beam on the image plane at a rate high enough so that all desired light output positions on the image plane can be accessed during the time period (timeframe) of displaying any given intra-pupillary image. For example, during the amount of time that a particular image is displayed, the actuator is preferably capable of scanning the light beam on the image plane at least once, preferably multiple times, on an area corresponding to the virtual 2D light source.

[0320] Fig.21A and 21BAnother example movable injector system 3600 is shown that includes at least one scanning mirror system for directing light propagation to different locations. Light injection system 3626 can include an image source 3628 that can output image light and mirrors 3630 and 3632 that are moved by actuators 3631 and 3633, respectively. Examples of image source 3628 include one or more light sources and a spatial light modulator that encodes image information into light that passes through the spatial light modulator from the light source to the mirror 3632.

[0321] As shown, light 3501 can propagate from image source 3628 to mirror 3632, which reflects the light to mirror 3630, which then reflects the light so that it propagates through projection optics 3634 toward an incoupling optical region 3510A' or 3510B' of waveguide 3526'. Mirrors 3630 and 3632 can be part of a dual-axis galvanometer, where actuators 3631 and 3633 rotate the mirrors along different axes (e.g., orthogonal axes), thereby allowing light to be directed to a region defined by two axes along waveguide 3526'. In some implementations, actuators 3631, 3633 can be motors. Projection optics 3634 can be a linear transmission lens, such as an F-theta (F-θ or F-tanθ) lens, and can be configured to project light onto waveguide 3526' at an incoupling optical region (e.g., 3510A' or 3510B'). In some implementations, the light injection system 3626 may also include a collimating lens 3636 to collimate the light emitted by the right light emitter 3628 before the light reaches the reflector 3632. Fig.21A As shown, projection optics 3634 may include various lens structures. In some examples, such as Fig. 21B As shown, the projection optical device 3634 may include a convex lens structure 3634P' and a biconvex lens, which includes a plurality of lenses 3634L', 3634L", each lens being configured to direct collimated light to an associated coupling optical region, such as 3510A' or 3510B'.

[0322] Continue to refer Fig.21A and 21B, the light injection system 3626 preferably also includes or is in communication with a processing module 3638 that controls and synchronizes the output of light from the light emitter 3628 with the movement of the actuators 3631, 3633 and the image to be formed. For example, the processing module 3638 can coordinate the movement of the reflectors 3632, 3630 with the emission of light from the image source 3628. In some implementations, the reflectors 3632, 3630 are continuously rotated or oscillated back and forth on the axis on which the reflectors are designed to move by the actuators 3631, 3633. The light emitted by the image source 3628 (e.g., light pulses) is synchronized (timed with) this movement so that the light is directed to the desired location on the waveguide 3526' at a given time, and the location and time are also determined based on the image to be displayed (e.g., the coupling region is consistent in time with the display of the image for that particular coupling region). In some implementations, the emission of light from image source 3628 is controlled by switching image source 3628 between on and off states (e.g., by powering or not powering the light emitter, respectively). In some other implementations, the emission of light from image source 3628 can be mechanically controlled using a physical switch that selectively allows or blocks light from reaching waveguide 3526'.

[0323] In some other implementations, instead of including two reflectors 3632 and 3630, the movable light injection system 3600 includes only a single reflector configured to direct light along a single axis. For example, such a configuration can be utilized in an arrangement where the incoupling regions such as 3510A' or 3510B' are arranged in a row along a single axis.

[0324] Some movable injector technologies are discussed in U.S. patent application publication No. 2018 / 0113311, entitled “SYSTEM AND METHOD FOR PRESENTING IMAGE CONTENT ON MULIPLE DEPTH PLANES BY PROVIDING MULTIPLE INTRA-PUPIL PARALLAX VIEWS,” published on April 26, 2018, the entire disclosure of which is incorporated herein by reference.

[0325] Example of electronically switchable light output without mechanical actuation

[0326] The light injection system 3404 may include an injector system in which the position of the output light is switched electronically without mechanical actuation. The fixed injector system may include one or more components configured to selectively emit light so as to direct the light to a desired location (e.g., incoupling regions 3414A, 3414B, 3414C on the waveguide 3412).

[0327] Fig. 22 An example fixed injection system 3700 is shown, which includes a spatial light modulator (SLM) 3736 configured to project light 3501 to one or more coupling optical regions 3710A', 3710B' on a waveguide 3526'. The fixed injection system 3700 may include a light transmitter 3728 and a spatial light modulator (SLM) 3736. Examples of light transmitters 3528 include LEDs and lasers. In some implementations, a fiber optic cable may transmit light from a remote light transmitter. As shown, light 3501 may propagate from the light transmitter 3728 to the SLM 3736, which modulates the light to propagate through an optical device 3734 to the coupling optical region 3710A' or 3710B' in the waveguide 3526'. In some implementations, the SLM 3736 may be configured to modulate the light transmitted through the SLM. In some other implementations, SLM 3736 can be configured to reflect and modulate light. Examples of SLMs include liquid crystal display panels, DLP mirror arrays, LCoS, etc. Optics 3734 can be a linear transmission lens, such as an F-theta (F-θ or F-tanθ) lens, and can be configured to focus light onto an incoupling optical region (e.g., 3710A' or 3710B') on waveguide 3526'. However, other configurations of optics 3734 are also contemplated, such as a lens system including some combination of collimating lenses, biconvex lenses, or other lens structures.

[0328] SLM 3736 may include a panel display that may include an array of light emitters such as LEDs or lasers. In order to select which coupling optical region to propagate to, such as 3710A' or 3710B', controller 3738 may communicate with light injection system 3700 to cause SLM 3736 to activate certain pixels, arrays, or pixel groups in SLM 3736 that may be associated with the desired coupling optical region or coupling region. In some implementations, different portions of the SLM may be used to form different images for different associated regions in coupling regions 3710A', 3710B'. In some implementations, the size of the activated light-emitting pixel group on light source SLM 1020 may provide a smaller or larger image. In some other implementations, the size of the light-emitting region on light source SLM 3736 may be modified to control the depth of focus of the display system.

[0329] Some removable injector technologies are discussed in International Patent Publication No. WO 2019 / 173158, entitled “DISPLAY SYSTEM WITH LOW-LATENCY PUPIL TRACKER,” published on September 12, 2019, the entire disclosure of which is incorporated herein by reference.

[0330] Depth Plane Projection Example

[0331] Some implementations of display devices such as those described above may be configured to adjust the wavefront divergence of light (including light of image information projected from the display system and incident light from objects in the surrounding real world) by adjusting the focal length of a variable focus lens element included in the augmented reality system. As described above, an augmented reality system may include a display device that may include a display device that displays a video image to a user or viewer (e.g., Figure 2 Multiple stacked waveguides (e.g., corresponding to Fig.9A and 9B A plurality or a group of stacked waveguides 660, or corresponding to Figure 6 In some other implementations, the display device may include only a single waveguide. Therefore, although multiple waveguides are mentioned in various parts of the disclosure of this article, it should be understood that the multiple waveguides can be replaced by a single waveguide.

[0332] As described herein, projected light from a waveguide can be used to provide virtual, augmented reality image information to a viewer. The light can be projected so that the user perceives the light as coming from one or more different depths or from one or more distances from the viewer. The display device can be optically transmissive so that the user can see real-world objects in the surrounding environment through the display device. In some implementations, the waveguide can be configured to have a fixed optical focal length. In order to provide the appearance that the projected light comes from different depths, the waveguide can be configured to output a divergent light beam with different divergence amounts corresponding to different depth planes.

[0333] In order to provide a multi-dimensional sense of the displayed virtual objects, the display system can display images in multiple depth planes. In the case of an array of coupled elements, there are a variety of methods to project images in multiple depth planes. These systems include Fig.23 The separated pupil system shown in Fig.24 The variable focus lens system shown, and Fig.25C and 25D A system configured to approximate a continuous wavefront using multiple intra-pupil images is shown.

[0334] Separate pupil depth plane projection example

[0335] In some examples, the depth plane projection can be achieved by projecting an image onto an in-coupling optical element in a waveguide stack, as described above with reference to Figures 9A-9C In case of an array of incoupling regions in a waveguide, depth plane projection may be achieved, for example, by providing a plurality of incoupling optical regions for each incoupling element corresponding to a plurality of depth planes.

[0336] Fig.23 A top view of a coupling region 3800 is shown, which includes a plurality of coupling optical elements 3810A, 3810B, 3810C. A light emitter (not shown) can emit light corresponding to a wavelength of one of a plurality of colors to the coupling elements 3810A, 3810B, 3810C. The coupling optical elements 3810A, 3810B, 3810C can be arranged spaced apart in a manner similar to the light emitters. For example, a single optical element of the coupling optical elements 3810A, 3810B, 3810C can be located in a direct optical path of an associated one of the light emitters. Multiple coupling optical elements (such as elements 3810A, 3810B, 3810C) can be disposed in a waveguide stack to provide multi-depth planar projections for images projected toward the region 3800 containing the coupling optical elements.

[0337] In the example shown, there are nine coupling-in optical elements 3810. The nine coupling-in optical elements 3810 can be configured to project a single image having multiple colors on multiple depth planes. The nine coupling-in optical elements 3810 shown can include, for example, three elements configured to emit light of a first color (e.g., red), three elements configured to emit light of a second color (e.g., green), and three elements configured to send light of a third color (e.g., blue). However, other numbers of coupling-in optical elements with other groupings are possible. The groups can each include coupling-in optical elements of different colors (e.g., each group can have one coupling-in optical element for the first color, one coupling-in optical element for the second color, and one coupling-in optical element for the third color), and each group can be configured to form an image on a specific depth plane. For example, the coupling-in optical elements 3810 form three groups, each group projecting an image on an associated one of the three depth planes. However, other numbers, arrangements and configurations of coupling optical elements are also possible, such as six coupling optical elements configured to project light of three component colors onto two depth planes, twelve coupling optical elements configured to project light of four colors onto three depth planes, etc.

[0338] In some examples, the coupling-in regions can be configured to project light in multiple colors at multiple depth planes using time multiplexing. For example, the coupling-in element can include multiple coupling-in regions that are configured to project light at multiple depth planes. One or more light emitters can be coupled to the coupling-in element to project light of a specific color to the coupling-in optical element at a specific time. If the time between color projections is fast enough, the resulting perceived image can be an image with all the colors projected to the coupling-in optical element.

[0339] Example of variable focus lens configuration

[0340] In some examples, a variable focus lens system can be used to achieve depth planar projection. Fig.24 An example variable focus lens system 3900 that can be used to implement deep planar projection is shown.

[0341] refer to Fig.24In some implementations, a first variable focus lens element may be disposed between the waveguide and the viewer's eye to appropriately adjust the wavefront of the light output by the waveguide, thereby allowing the viewer's eye to properly focus the light. However, this first lens element is also in the path of light propagating from the surrounding environment to the viewer's eye. As a result, the first lens element may change the wavefront of the light from the surrounding environment, thereby causing aberrations in the world seen by the viewer. In order to correct such aberrations, a second variable focus lens element may be disposed on a side of the plurality of stacked waveguides opposite to the first variable focus lens element; that is, the second variable focus lens element may be located between the plurality of stacked waveguides and the surrounding real world to adjust the wavefront of light from real-world objects in the surrounding environment. The second variable focus lens element may be configured to compensate for aberrations caused by the first variable focus lens element. In some implementations, the second variable focus lens may also be configured to compensate for aberrations caused by the waveguide.

[0342] In some implementations, the focus of the second variable focus lens element can be inverse or opposite to the focus of the first variable focus lens element. For example, if the first variable focus lens element has a positive optical power, the second variable focus lens element can have a negative optical power of similar magnitude. In some other implementations, the optical power of the second lens element can be opposite to the total optical power of the first lens element and the waveguide and have a similar magnitude to the total optical power of the first lens element and the waveguide in order to compensate for the optical power of the first variable focus lens element and the optical power of the intermediate waveguide.

[0343] In some other implementations, the waveguide may have no optical power (e.g., the waveguide may be configured to output collimated light), and the first variable focus lens element may be configured to modify the wavefront of the light emitted from the waveguide to provide an appropriate amount of divergence for image information to be interpreted by a viewer as being located at a particular depth plane. It should be understood that the appropriate amount of divergence may vary for different viewers because the optical power used to place the image information at a particular depth plane will be adjusted by a particular differential to take into account the viewer's optical prescription for that depth plane. In such an implementation, the waveguide stack between the first and second variable focus lens elements may simply be formed of a single waveguide.

[0344] It will be appreciated that first and second variable focus lens elements may be provided to one eye of a viewer, while third and fourth variable focus lens elements, respectively similar to the first and second variable focus lens elements, may be provided to the other eye of the viewer.

[0345] Fig.24 Schematic diagram of an example display system (e.g., augmented reality display system) with a variable focus lens element and a waveguide stack is shown. It should be understood that display system 3910 can correspond to display system 250 ( Figure 6). The first variable focus lens element 3907a and the second variable focus lens element 3907b are arranged on either side of the waveguide stack 3905, and the third variable focus lens element 3908a and the fourth variable focus lens element 3908b are arranged on either side of the waveguide stack 3906.

[0346] The various waveguides 3905a, 3905b, 3906a, 3906b shown may have Figure 6 The waveguides 270, 280, 290, 300, 310 and / or Fig.9A and 9B The waveguide stacks 3905, 3906 may have properties and / or characteristics similar to those of a single waveguide in waveguides 670, 680, and 690. Fig.9A and 9B A plurality or a group of 660 stacked waveguides or Figure 6 The stacked waveguide assembly 260 of FIG. 20 can be similar in properties and / or features to the stacked waveguide assembly 260 of FIG. 20. In some implementations, the waveguides 3905a, 3905b, 3906a, 3906b can include optical elements, such as diffractive optical elements, that provide waveguides with optical power (e.g., fixed optical power). For example, one or more of the waveguides can have an optical power in a range between 0 diopters and about 5.0 diopters; between about 0.5 diopters and about 4.5 diopters; between about 1.0 diopters and about 4.0 diopters; between about 1.5 diopters and about 3.5 diopters, and between about 2.0 diopters and about 3.0 diopters, or any value within these ranges or sub-ranges. As another example, in a particular implementation, each waveguide can have an optical power of 1.5 diopters.

[0347] As described above, light from light source 3903 or 3904 providing image information (e.g., virtual content) can be injected into waveguide 3905a or 3906a, respectively, so that the light propagates through each of these waveguides by total internal reflection. The propagated light can be coupled out through an outcoupling element (e.g., corresponding to Fig.9A and 9B800, 810, 839) are projected from the waveguide 3905a (or waveguide 3905b) to the user's eye 3901. In some implementations, the light sources 3903, 3904 can be fiber scanning devices (FSDs) that utilize moving optical fibers to create 2D image patterns, as disclosed herein. The FSD can create 2D image patterns by projecting light in various patterns (e.g., raster scans, spiral scans, Lissajous, etc.). In some other implementations, also as disclosed herein, the light source 3903a (and / or 3903b) can be, for example, an image projection system in which a complete image is projected onto a waveguide. It should be understood that light from the light source 3903a (and / or 3903b) can be injected into the waveguide stack 3905 through the edge of the waveguide or through the main surface of the waveguide. Where the waveguide stack comprises a plurality of waveguides, light sources 3903 and / or 3904 may be configured to inject light into a plurality of these waveguides, or additional light sources may be provided, for example one light source per waveguide.

[0348] The first variable focus lens element 3907a may be disposed between the waveguide stack 3905 and the user's eye 3901, and the second variable focus lens element 3907b may be disposed between the waveguide stack 3905 and the real world around the user. It should be understood that the eye 3901 may correspond to Figure 6 of the viewer's eye 210. Similarly, a third variable focus lens element 3908a may be disposed between the waveguide stack 3906 and the user's eye 3902, and a fourth variable focus lens element 3908b may be disposed between the waveguide stack 3906 and the real world around the user.

[0349] In some implementations, the first and second variable focus lens elements 3907a and 3907b and the third and fourth variable focus lens elements 3908a and 3908b can be adaptive optical elements. The adaptive optical element can be dynamically changed, for example, by applying an electrical signal to it, to change the shape of the wavefront incident thereon. In some implementations, the adaptive optical element can include a transmissive optical element, such as a dynamic lens (e.g., a liquid crystal lens, an electro-active lens, a conventional refractive lens with a moving element, a lens based on mechanical deformation, an electro-wetting lens, an elastic lens, or a plurality of fluids with different refractive indices). By changing the shape, refractive index, or other properties of the adaptive optical device, the wavefront incident thereon can be changed, for example, to change the focus of light of a viewer's eye, as described herein.

[0350] In some implementations, the variable focus lens elements 3907a, 3907b, 3908a, 3908b can include a liquid crystal layer sandwiched between two substrates. The substrates can include an optically transmissive material such as glass, plastic, acrylic, etc. In some implementations, the substrates can be flat. In some implementations, the substrates can have curved portions such that various portions of the substrate can have a fixed optical power.

[0351] In some implementations, the optical power of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can be changed by adjusting an electrical signal (e.g., current and / or voltage) applied to the liquid crystal layer through, for example, one or more thin film transistors (TFTs) and / or electrodes integrated with the liquid crystal layer and / or substrate. It should be understood that the orientation of the liquid crystal material in the liquid crystal layer determines the refractive index of the layer. The applied electrical signal sets the orientation of the liquid crystal material, thereby allowing the refractive index of the liquid crystal layer to be changed as desired by changing the applied electrical signal. In some implementations, the optical power of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can vary between approximately ±5.0 diopters (e.g., between approximately -4.0 diopters and +4.0 diopters; between approximately -3.5 diopters and approximately +3.5 diopters, between approximately -3.0 diopters and approximately +3.0 diopters; between approximately -2.0 diopters and approximately +2.0 diopters; between approximately -1.5 diopters and approximately +1.5 diopters, including any values ​​in these ranges or sub-ranges).

[0352] Advantageously, the variable focus lens elements 3907a, 3907b, 3908a, 3908b can have wide apertures that substantially match the apertures of the waveguides of their respective associated waveguide stacks 3905, 3906. In some implementations, the apertures of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can be substantially equal (e.g., within about ±39%, about ±15%, or about ±10%) to the surface areas of the waveguides of the waveguide stacks 3905, 3906. Thus, the areas over which the variable focus lens elements 3907a, 3907b, 3908a, 3908b and the waveguide stacks 3905, 2396 transmit light to the associated eyes 3901, 3902 can be substantially equal.

[0353] Continue to refer Fig.24, the first and third variable focus lens elements 3907a, 3908a can each change their optical power to adjust the wavefront of light projected from the waveguides of the waveguide stacks 3905, 3906, respectively, so as to appropriately focus the light onto the retinas of the eyes 3901, 3902, respectively. As described herein, the first and third variable focus lens elements 3907a, 3908a can induce aberrations in the wavefront of incident light from an object 3909 in the surrounding environment, thereby reducing the quality of the optical image of the real world object 3909 viewed through the first variable focus lens element 3907a. The second and fourth variable focus lens elements 3907b, 3908b can advantageously compensate for the aberrations introduced by the first and third variable focus lens elements 3907a, 3908a, respectively, and any waveguides when viewing the object 3909. In some implementations, the second and fourth variable focus lens elements 3907b, 3908b can be configured to provide an optical power that is opposite to the optical power provided by the first and third variable focus lens elements 3907a, 3908a and the associated waveguide stacks 3905, 3906, respectively. In some implementations, the magnitude of the opposite optical power is such that for each eye 3901, 3902, the net optical power of the display system 3910 is equal to the optical prescription of the eye at the depth plane to which the eye is proximate. The optical power provided by the first and second variable focus lens elements 3907a and 3907b can be varied and controlled by the electronic hardware control system 3911. In some implementations, the electronic hardware control system 3911 can correspond to Figure 2 local processing and data module 140 and / or remote processing module 150.

[0354] In some implementations, the augmented reality display system 3910 can be configured to determine the vergence of the user's eyes. The optical power of the first and second variable focus lens elements 3907a, 3907b can be set based on the convergence point of the eyes 3901, 3902. The optical power of the third and fourth variable focus lens elements 3908a, 3908b can also be set based on the convergence point. It should be understood that the convergence point is a point in space where the sight lines of the eyes 3901, 3902 converge, and can correspond to the physiological adaptation target of these eyes. In some implementations, the distance of the point from the eyes 3901, 3902 can be calculated, for example, based on the known amount of separation between the eyes 3901, 3902 and the angle formed by each eye. Once the distance is calculated, the appropriate correction for the viewer for the distance can be determined. For example, the display system 3910 can be programmed with one or more optical prescriptions. In some implementations, the optical prescription can be stored in the local processing and data module 140 and / or the remote database 160. The distance between the eyes 3901, 3902 and the convergence point can be matched with an appropriate correction for that distance, and the variable focus lens elements 3907a, 3907b, 3908a, 3908b can be adjusted to provide the correction. In some implementations, the eyes 3901, 3902 can have different prescription corrections, and as a result, multiple pairs of variable focus lens elements 3907a, 3907b and 3908a, 3908b can provide different optical powers.

[0355] Advantageously, the variable focus lens elements 3907a, 3907b, 3908a, 3908b provide a large number of possible corrections because their optical power can be adjusted as needed by, for example, applying different voltages. In some implementations, the total number of corrections for each eye may be 1, 2, 3, or more than 4. In some implementations, the total number of corrections for each eye may be equal to the number of depth planes on which the display system 3910 is configured to display image information. It should be understood that these corrections may correspond to optical prescriptions, which may be determined for objects at different distances from the eyes 3901, 3902. For example, four prescriptions may be obtained by determining the correction of refractive error at four progressively farther distances from the eyes 3901, 3902 (e.g., near distance, intermediate near distance, intermediate far distance, and far distance). In some implementations, the number of possible corrections for viewing image content output by the waveguide stack 3905 may be different from the number of possible corrections for viewing objects 3909 in the surrounding environment.

[0356] In some implementations, the focus or optical power of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can each be set based on the determined vergence of the user's eyes 3901, 3904. For example, the optical power of the first and second variable focus lens elements 3907a and 3907b can be changed based on the vergence of the user's eye 3901 without specific reference to the optical power of the other lens elements.

[0357] In some implementations, one of the first and second variable focus lens elements 3907a, 3907b or one of the third and fourth variable focus elements 3908a, 3908b can be designated as a master element, and the other of the first and second variable focus lens elements 3907a, 3907b or the third and fourth variable focus elements 3908a, 3908b can be designated as a slave element. The variable focus lens element designated as a slave element can be configured to follow the master variable focus lens element. In some other implementations, the second and fourth variable focus lens elements 3907b, 3908b can be subordinate to the first and third variable focus lens elements 3907a, 3908a, and the focus of the first and third variable focus lens elements 3907a, 3908a can be set based on the determined convergence point of the user's eyes 3901, 3902. For example, if waveguide 3905a (and / or waveguide 3905b) has an optical power of approximately 1.5 diopters and the user converges at 2.0 diopters, the first variable focus lens element 3907a can have an optical power of +0.5 diopters and the second variable focus lens element 3907b can have an optical power of -0.5 diopters.

[0358] The optical power of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can be changed in real time, and can preferably be changed at a rate equal to or greater than the rate at which the human eye changes its adaptation state. Preferably, the first and second variable focus lens elements can change their optical power before the human eye changes its adaptation state, so that the user does not experience a delay in receiving the appropriate correction for a given convergence point. In some implementations, the optical power of the first and second variable focus lens elements can be changed in less than about 300ms, less than about 275ms, or less than about 250ms. The electronic hardware control system 3911 can drive the variable focus lens elements 3907a, 3907b, 3908a, 3908b so that the optical power of the variable focus lens elements 3907a, 3907b, 3908a, 3908b can be changed simultaneously.

[0359] Variable focus lens technology is discussed in U.S. Patent Application Publication No. 2017 / 0293145, published on October 12, 2017, entitled “AUGMENTED REALITY SYSTEMS AND METHODS WITH VARIABLE FOCUS LENS ELEMENTS,” the entire public contents of which are incorporated herein by reference.

[0360] Example of wavefront approximation using multiple intrapupil images

[0361] It has been found that a continuous wavefront can be approximated using multiple wavefronts, such as Fig.25A Wavefront 4000. In some examples, depth plane selection can be achieved by approximating a continuous wavefront at one or more desired depth planes using multiple discrete intra-pupil images directed into the eye within a flicker fusion threshold.

[0362] Reference now Fig.25A , showing the pre-adaptation and post-adaptation states of the eye 210 when receiving a continuous input wavefront 4000. Figure a) shows the pre-adaptation state before the visual system focuses the wavefront 4000 on the retina 211. It should be noted that the focal point 212 is not on the retina 211. For example, as shown in the figure, the focal point 212 can be in front of the retina 211. Figure b) shows the post-adaptation state after the human visual system flexes the pupil muscle tissue of the viewer's eye 210 to focus the wavefront 4000 on the retina 211. As shown in the figure, the focal point 212 can be on the retina 211.

[0363] Fig.25B Shows the receiving Fig.25A The pre-adaptation and post-adaptation states of the eye 210 when performing a segment-by-segment approximation of the continuous wavefront 4000. Fig.25B Figure a) shows the pre-adaptation state of the eye 210, and figure b) shows the post-adaptation state of the eye 210. The approximation can be formed using multiple constituent wavefronts 1010a, 1010b, and 1010c, each constituent wavefront being associated with a separate light beam. As used herein, reference numerals 1010a, 1010b, and 1010c may indicate a light beam and an associated wavefront of the light beam. In some implementations, the constituent wavefronts 1010a and 1010b may be plane wavefronts, such as formed by collimated light beams. As shown in figure b), the wavefront approximation 1010 formed by the constituent wavefronts 1010a and 1010b is focused by the eye 210 onto the retina 211, with the focus 212 being on the retina 211. Advantageously, the pre-adaptation and post-adaptation states are similar to those formed by Fig.25A The state caused by 4000 continuous wave fronts is shown.

[0364] It should be understood that a continuous diverging wavefront can be approximated using an optical projection system that outputs multiple intra-pupillary images with different parallaxes into the eye. Fig.25C The eye is shown adjusting a diverging wavefront emitted by a finite focal length virtual image provided by a projection system. The system includes a projection optical device 1020 having a focal length "F" and an external aperture and a spatial light modulator 4018. An image can be formed by the spatial light modulator 1018, and light containing image information from the spatial light modulator 4018 can be directed to the eye 210 through the projection optical device 4020. The spacing between the spatial light modulator 4018 and the projection optical device 4020 can be selected (less than F) so that the diverging wavefront 4000 is output toward the eye 210. As described above with respect to Fig.25A As described, the eye 210 can then focus the wavefront 4000 onto the retina 211.

[0365] Continue to refer Fig.25C , showing the components used to form Fig.25B 4010 . The projection system 4003 includes a light source 4026 configured to output light 4010a' and 4010b' to a spatial light modulator 4018, which modulates the light to form an image showing slightly different parallax views of the same object or scene. The modulated light with the image information then propagates through a relay / projection optical device 4020 and is output by the relay / projection optical device 4020 into the eye 210 as light beams 4010a and 4010b. The projection system 4003 may also include a lens structure 4014, which may be configured to convert spatial differences in the emission of light 4010a' and 4010b' into angular differences in the light as it propagates toward the spatial light modulator 4018. The projection system 4003 may also include a polarizing beam splitter 4016 configured to 1) direct light from the light source 4026 to the spatial light modulator 4018; and 2) allow modulated light from the spatial light modulator 4018 to propagate back through the beam splitter 4016 to the relay / projection optics 4020. In some implementations, the display system 4001 may include an eye tracking device 4022, e.g., a camera, configured to monitor eye gaze. As discussed herein, such monitoring may be used to determine the direction in which the viewer is looking, which may be used to select image content appropriate for that direction, and for eye pose determination. Preferably, the eye tracking device 4022 tracks both eyes of the viewer, or each eye includes its own associated eye tracking device. As a result, the convergence of the viewer's eyes may be tracked, and the convergence point of the eyes may be determined to determine in what direction and at what distance the eyes are pointing.

[0366] It should be understood that the light source 4026 can output light 4010a' and 4010b' at different times, the spatial light modulator 4018 can form different parallax views with the light 4010a' and 4010b' at different times, and as discussed herein, the final light beams 4010a and 4010b can be injected into the eye 210 at different times.

[0367] Continue to refer Fig.25C , the light source 4026 can be a 2D light source having a plurality of selectively activated light output positions arranged substantially on a plane. In some implementations, the selectively activated light output position can be a selectively activated light emitting area. For example, the light source 4026 can be a light emitting diode (LED) array, or a spatial light modulator (e.g., a digital micromirror device, such as a digital light processing (DLP) device, an LCOS device, etc.), which contains an array of discrete units or light emitters that output light. Examples of LED arrays include organic light emitting diode (OLED) arrays and inorganic light emitting diode (ILED) arrays, each of which can be part of a microdisplay. In some implementations, the individual light emitting diodes and / or light modulators in the light source 4026 can constitute a light emitting area. In some other implementations, multiple groups of light emitting diodes and / or light modulators can form a light emitting area. In such an implementation, there may be some overlap between the light emitting diodes and / or light modulators of different light emitting areas, although these areas are considered different because they do not completely overlap.

[0368] In some other implementations, light source 4026 can be configured to focus light onto an image plane to effectively provide a virtual 2D light source on the image plane. Different locations on the image plane can be considered different light output locations, and these locations can be activated by directing light from a light emitter through these locations on the image plane using an actuated mirror or fiber scanner. More details on this virtual 2D light source are below Fig.19 and 20 are provided in the discussion.

[0369] In some implementations, an example of the spatial light modulator 4018 includes a liquid crystal on silicon (LCOS) panel. As another example, in some other implementations, the spatial light modulator 4018 can include a transmissive liquid crystal panel or a MEMs device, such as a DLP.

[0370] Continue to refer Fig.25C, the display system 4001 may also include a control system 4024 for determining the timing and type of image content provided by the display system. In some implementations, the control system 4024 includes one or more hardware processors having memory storing programs for controlling the display system 4001. For example, the system 4024 may be configured to control the activation of light emitting regions of the light source 4026, the actuation of individual pixel elements of the spatial light modulator 4018, and / or the interpretation and reaction of the display system 4001 to data received from the eye tracking device 4022. Preferably, the system 4024 includes a calculation module 4024a configured to receive input regarding a desired depth plane or wavefront divergence and calculate appropriate light emitting regions to be activated in order to form a parallax view with an appropriate amount of parallax for the desired depth plane or wavefront divergence. In addition, the calculation module 4024a may be configured to determine the appropriate actuation of the pixels of the spatial light modulator 4018 to form an image of the desired parallax view. The system 4024 may also include a synchronization module 4024b configured to synchronize activation of specific light emitting areas of the light source 4026 with light modulation of the spatial light modulator 4018 to form a parallax view corresponding to these activated light emitting areas. In addition, the system 4024 may include an eye tracking module 4024c that receives input from the eye tracking device 4022. For example, the eye tracking device 4022 may be a camera configured to image the eye 210. Based on the images captured by the eye tracking device 4022, the eye tracking module 4024c may be configured to determine the orientation of the pupil and extrapolate the line of sight of the eye 210. This information may be transmitted electronically to the computing module 4024a. The computing module 4024a may be configured to select image content based on the line of sight or gaze of the eye 210 (preferably also based on the line of sight or gaze of the viewer's other eye).

[0371] Because light source 4026 may include an array of discrete light emitters, the shape and size of the light emitting area formed by the light emitters can be changed as needed by activating selected light emitters. For example, different light emitting areas can be activated for different parallax views. In one example, the elongated light emitting areas can be spaced horizontally, which is ideal for adaptation to horizontal parallax drive only. In another example, the circular light emitting areas can have horizontal and vertical displacements. In another example, the brightness of the light emitting areas may be reduced. In another example, the light emitting areas can overlap. In another example, the light emitting areas can form an array. Thus, light source 4026 ( Fig.25C) can include binary (simply switched on and off) light emitters and / or light emitters combined with grayscale (emitting light of selectively variable intensity). In some implementations, light source 4026 can include elements that switch at very high rates, including rates that exceed the parallax switching rate of system 4001. For example, light source 4026 can have a light output element that switches light output on and off at a rate that is higher than the rate at which parallax (intra-pupil) images are switched in implementations where different intra-pupil images are displayed at different times.

[0372] It should be understood that different amounts of wavefront divergence can be approximated by appropriately selecting the physical distances separating the light emitters that are activated to form different intra-pupillary images. The light emitters of light source 4026 can direct light from different angles through the projection optics to provide intra-pupillary images with different parallaxes for approximating different amounts of wavefront divergence. In some implementations, a relatively large distance between the light sources can provide a wavefront divergence corresponding to a relatively close depth plane, while a relatively small distance between the light sources can provide a wavefront divergence corresponding to a relatively far depth plane.

[0373] Reference again Fig.25C In some implementations, the control system 4024 may include two parts: 1) light field generation and 2) factored light field optimization. As discussed herein, in order to approximate the wavefront, an appropriate image is displayed on the spatial light modulator 4018 for each activated light emitting region of the light source 4026. It should be understood that these images are created during the light field generation step, where the 3D scene is rendered from multiple slightly offset viewpoints that correspond to the slight offsets in the activated light emitting regions. For example, to display a 5x5 light field, the 3D scene will be rendered 25 times from 25 different viewpoints arranged in a grid pattern. The positions of the viewpoints in the grid pattern will correspond to the positions of the activated light source regions, and the rendered images will correspond to the images formed by the spatial light modulator.

[0374] It may be desirable to increase the brightness of the image formed by the spatial light modulator 4018. Advantageously, utilizing a light source 4026 comprising an array of light emitters allows the formation of light emitting areas of various shapes and sizes, which can be used to increase brightness. In some implementations, the brightness can be increased by increasing the size of the activated light emitting area without significantly changing the image formed by the spatial light modulator 4018. The computational module 4024a may be configured to determine the size and shape of the activated light emitting area using factored light field optimization. The module 4024a may be configured to acquire an input focus stack and create a series of patterns to be displayed on the spatial light modulator 4018 and the light source 4026, wherein the configured pattern creates a desired approximation to the focus stack in a least squares sense. The optimization takes advantage of the fact that a small shift in the viewpoint does not significantly change the perceived image, and is able to generate a light emitting area pattern using illumination from a larger area on the light source 4026 while displaying the same image on the spatial light modulator 4018.

[0375] The optimization problem can be formulated as a non-convex optimization problem as follows:

[0376]

[0377] Wherein, the projection operator p performs a linear transformation from a 4D light field to a 3D focal stack (using a shift and add algorithm). This problem is a non-negative matrix factorization embedded in a deconvolution problem. The algorithm to solve this problem uses the alternating direction method of multipliers (ADMM). Additional details on an example method for solving this problem are discussed in Appendix I. It should be understood that module 4024a is configured to actively calculate the appropriate size and shape of the light emitting area in real time based on the parallax view to be formed by the spatial light modulator 4018.

[0378] In some other implementations, the optimization problem can be formulated as a slightly different non-convex optimization problem as follows:

[0379]

[0380] Where A and B represent the patterns displayed on the spatial light modulators (e.g., light source 4026 and spatial light modulator 4018 used to form an image), y is the target 4D light field that is the desired output of the algorithm, and AB' is an operator that combines the spatial light modulator patterns to simulate the 4D light field emitted by the physical display when A and B are displayed on the modulator. This problem is a non-negative matrix factorization. The algorithm that solves this problem uses an iterative optimization technique to refine A and B from a random initial guess.

[0381] Continue to refer Fig.25CIt should be understood that the flicker fusion threshold of the human visual system imposes a time limit on the number of images that may be injected into the eye 210 while still being considered to be injected simultaneously. For example, the processing bandwidth of the control system 4024 and the ability to switch the light emitting area of ​​the light source 4026 and the light modulator of the spatial light modulator 4018 can limit the number of images injected into the eye 210 within the time period allowed by the flicker fusion threshold. Given this limited number of images, the control system 4024 can be configured to make a selection regarding the image displayed. For example, within the flicker fusion threshold, the display system is required to inject a set of intra-pupillary images with different parallax into the eye, and in turn, each parallax view requires images of various component colors to form a full-color image. In some implementations, the formation of a full-color image using component color images is separated from the elucidation of the desired adaptation response. For example, without being limited by theory, a desired adaptation response can be induced with a single color of light. In this case, the intra-pupillary images with different parallax used to induce the adaptation response will only be monochrome. As a result, there is no need to use other colors of light to form parallax-different intrapupillary images, freeing up time for other types of images to be displayed within the flicker fusion threshold. For example, to better approximate the wavefront, a larger set of parallax-different intrapupillary images can be generated.

[0382] In some other implementations, the control system 4024 can be configured to spend less time within the flicker fusion threshold to display images of color light to which the human visual system is less sensitive. For example, the human visual system is less sensitive to blue light than green light. As a result, the display system can be configured to generate a higher number of images formed with green light than with blue light.

[0383] Reference now Fig.25D , shows an example of a projection system 4003 that includes an eye tracking system 4022 and a combined eyepiece 1005 having a pupil expander 1035 configured to produce a non-infinite depth plane. In some implementations, the non-infinite depth plane may be 3 meters, which provides an in-budget adaptation from about 2.5 meters to infinity. For example, given the tolerance of the human visual system to accommodation-convergence mismatch, virtual content at a distance from the viewer of about 2.5 meters to infinity can be placed on the 3 meter depth plane with little discomfort. In such a system, parallax-variant intra-pupillary images can be used to drive adaptation of a narrower range of depth planes, which may all be closer to the viewer than a fixed "default" focal plane. In some implementations, the system may also include an eye tracking system 4022 to determine the viewer's gaze distance, for example based on the convergence angle of the viewer's eyes.

[0384] In some implementations, light source 4026 can be replaced by a virtual light source formed on the image plane of the light projection system. The light projection system may include an actuator that enables a light beam to scan an area on the image plane corresponding to the virtual light source. In order to simulate the ability to activate discrete light emitting areas of light source 4026, the light output of the projection system is synchronized with the movement of the actuator so that light is output to a desired position on the image plane at a specific time. Preferably, the actuator is capable of scanning the light beam on the image plane at a rate high enough that all desired light output positions on the image plane can be accessed during the time period in which any given intra-pupillary image is displayed. For example, during the time period in which a particular intra-pupillary image is displayed, the actuator is preferably capable of scanning the light beam at least once, preferably multiple times, on the area on the image plane corresponding to the virtual 2D light source.

[0385] It should be understood that, as discussed herein, parallax-variant intra-pupil images can be directed into a waveguide having multiple in-coupling regions and associated out-coupling regions. Image light for each of these intra-pupil images can be routed to the desired in-coupling region by a variety of schemes, including: a scanning fiber system, as discussed with respect to FIG. 20; a movable mirror system, as discussed with respect to FIG. 21; and a system for forming images for different in-coupling regions of different portions of a spatial light modulator, as discussed with respect to FIG. Fig. 22 discussed.

[0386] Some wavefront approximation techniques are discussed in U.S. patent application publication No. 2018 / 0113311, entitled “SYSTEM AND METHOD FOR PRESENTING IMAGE CONTENT ON MULIPLE DEPTH PLANES BY PROVIDING MULTIPLE INTRA-PUPIL PARALLAX VIEWS,” published on April 26, 2018, the entire disclosure of which is incorporated herein by reference.

[0387] Other Implementations

[0388] Various aspects, implementations, embodiments or features of the described implementations may be used alone or in any combination. Various aspects of the described implementations may be implemented by software, hardware, or a combination of hardware and software. The described implementations may also be embodied as computer-readable code on a computer-readable medium for controlling manufacturing operations, or as computer-readable code on a computer-readable medium for controlling a manufacturing line. A computer-readable medium is any data storage device that can store data that can subsequently be read by a computer system. Examples of computer-readable media include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. Computer-readable media may also be distributed on network-coupled computer systems so that computer-readable code is stored and executed in a distributed manner.

[0389] For the purpose of explanation, the above description uses specific terms to provide a thorough understanding of the described implementations. However, it will be apparent to those skilled in the art that the described implementations may be practiced without the specific details. Therefore, for the purpose of illustration and description, the above descriptions of specific implementations are provided. They are not intended to be exhaustive or to limit the described implementations to the precise forms disclosed. For those of ordinary skill in the art, many modifications and variations are possible in view of the above teachings.

[0390] It should also be understood that each process, method, and algorithm described herein and / or depicted in the accompanying drawings may be embodied in and fully or partially automated by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute specialized and specific computer instructions. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, etc., programmed with specific computer instructions. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some implementations, specific operations and methods may be performed by circuits specific to a given function.

[0391] Furthermore, certain embodiments of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that specialized hardware or one or more physical computing devices (with appropriate specialized executable instructions) may be required to perform the functionality, e.g., due to the amount or complexity of the computations involved, or in order to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0392] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer memory, including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of these and / or similar memories, etc. In some implementations, the non-transitory computer-readable medium may be part of one or more of the local processing and data module (140), the remote processing module (150), and the remote data repository (160). The methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media) and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored permanently or otherwise in any type of non-transitory tangible computer memory, or may be transmitted via a computer-readable transmission medium.

[0393] Any process, box, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to potentially represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (such as logic or arithmetic) or step in the process. Various processes, boxes, states, steps or functions can be combined, rearranged, added to the illustrative examples provided herein, deleted, modified or otherwise changed from the illustrative examples provided herein. In some implementations, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the boxes, steps or states associated therewith can be executed in other appropriate orders (such as serial, parallel or in some other way). Tasks or events can be added to or deleted from the disclosed example implementations. In addition, the separation of various system components in the implementation described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products.

[0394] In the foregoing description, the present invention has been described with reference to specific implementations of the present invention. However, it is apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive. For example, although advantageously combined with an AR display that provides images across multiple depth planes, the augmented reality content disclosed herein may also be displayed by a system that provides images on a single depth plane.

[0395] In fact, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, none of which bears sole responsibility or claim for the desired properties disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure.

[0396] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable subcombination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features in the claimed combination may be excised from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination. No single feature or set of features is necessary or essential to every implementation.

[0397] It should be understood that the conditional language used here, especially such as "can", "will", "may", "can", "for example", etc., unless otherwise explicitly stated, is understood in the context of use as generally intended to convey that certain implementations include and other implementations do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements and / or steps are necessary for one or more implementations in any way, or that one or more implementations must include logic for determining whether these features, elements and / or steps are included or will be executed in any particular implementation with or without author input or prompts. The terms "include", "comprise", "have", etc. are synonyms, used inclusively in an open manner, and do not exclude additional elements, features, actions, operations, etc. Moreover, the term "or" is used in its inclusive meaning (rather than in its exclusive meaning), so for example, when used to connect a list of elements, the term "or" represents one, some or all of the elements in the list. In addition, "one", "an" and "the" used in the present application and the appended claims should be interpreted as representing "one or more" or "at least one", unless otherwise stated. Similarly, although operations can be depicted in the drawings in a particular order, it is to be appreciated that such operations do not need to be performed in the particular order or sequential order shown, or that all operations shown need not be performed to achieve the desired result. In addition, the drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be combined in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, in other implementations, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated in a single software product or packaged into multiple software products. In addition, other implementations are within the scope of the appended claims. In some cases, the actions recorded in the claims may be performed in different orders, and the desired results may still be achieved.

Claims

1. A head mounted display system configured to project light to display virtual image content, the display system comprising: Light projection system; as well as A head mounted display configured to display virtual image content, the head mounted display comprising at least one waveguide, each of the at least one waveguide comprising: a plurality of incoupling regions, each incoupling region being configured to receive light having image information for forming the virtual image content from the light projection system and to couple the light into the waveguide; and a plurality of outcoupling regions configured to couple the light out of the waveguide to display the virtual image content, wherein each of the plurality of outcoupling regions is configured to receive light from a different one of the incoupling regions, The multiple outcoupling regions do not overlap with each other.

2. The display system according to claim 1, wherein: Each incoupling region of the plurality of incoupling regions corresponds to and is configured to selectively direct light to a unique outcoupling region of the plurality of outcoupling regions.

3. The display system according to claim 2, wherein: Each coupling-in region is a diffraction grating having at least one of a tilt, a grating orientation, a grating structure size, or a grating spacing, wherein the at least one of the tilt, the grating orientation, the grating structure size, or the grating spacing is selected so that light coupled in by the coupling-in region is preferentially coupled out by the outcoupling region corresponding to the coupling-in region.

4. The display system according to claim 1, wherein: Each coupling region is configured to redirect incident light from the light projection system along a propagation direction at an angle for total internal reflection through the waveguide, and the display system also includes a light distribution structure configured to redirect a portion of the light to propagate along an axis that intersects the propagation direction.

5. The display system according to claim 4, wherein: The light distributing structure and the plurality of outcoupling regions are disposed on opposing major surfaces of the at least one waveguide.

6. The display system according to claim 1, wherein: The plurality of incoupling regions comprises a diffraction grating.

7. The display system according to claim 6, wherein: Each incoupling region of the plurality of incoupling regions has a different grating pitch than at least one other incoupling region of the waveguide.

8. The display system according to claim 7, wherein: The different grating pitches vary the total internal reflection angle at which light coupled into the waveguide is totally internally reflected toward the waveguide toward an associated outcoupling region of the plurality of outcoupling regions.

9. The display system according to claim 6, wherein: Each incoupling region of the plurality of incoupling regions has a different grating orientation in the plane of the waveguide than at least one other incoupling region of the waveguide.

10. The display system according to claim 9, wherein: The different grating orientations change the angle at which light is coupled into the waveguide so that the light propagates toward a desired outcoupling region among the plurality of outcoupling regions.

11. The display system according to claim 1, wherein: Each coupling-in region of the plurality of coupling-in regions has a different inclination relative to the plane of the waveguide than at least one other coupling-in region of the waveguide.

12. The display system according to claim 1, wherein: The plurality of coupling regions are spaced apart from one another.

13. The display system according to claim 1, wherein: The plurality of incoupling regions are arranged in a row around at least one side of the plurality of outcoupling regions.

14. The display system according to claim 13, wherein: Each of the plurality of outcoupling regions defines a stripe spanning the waveguide, the stripe being elongated along an axis intersecting the row of incoupling regions.

15. The display system according to claim 1, wherein: The plurality of outcoupling regions define a grid pattern across the waveguide.

16. The display system according to claim 1, wherein: Each of the plurality of incoupling regions is arranged to incouple light and guide the light to propagate within the waveguide by total internal reflection in a direction different from at least one other incoupling region of the waveguide.

17. The display system according to claim 16, wherein: At least one incoupling region proximate a lateral edge of the at least one outcoupling region is configured to incouple light along an angular range arranged symmetrically about an inwardly biased axis.

18. The display system according to claim 1, wherein: At least two of the plurality of coupling-in regions are arranged along different sides of at least one of the plurality of coupling-out regions.

19. The display system according to claim 18, wherein: The plurality of incoupling regions include an incoupling optical element surrounding the at least one outcoupling region among the plurality of outcoupling regions on at least three sides.

20. The display system according to claim 1, wherein: The light projection system includes a light source and a movable light injector configured to direct image light into respective ones of the incoupling regions.

21. The display system according to claim 1, wherein: The head mounted display comprises a plurality of waveguides forming a waveguide stack, each of the waveguides comprising a plurality of the incoupling regions and the outcoupling regions, Wherein, the coupling region of each of the waveguides is laterally displaced in a front plan view when viewed from the direction of incident light incident on the coupling region.

22. The display system according to claim 21, wherein: The outcoupling regions of the same waveguide are configured to output light with the same wavefront divergence, wherein the outcoupling regions of different waveguides are configured to output light having a different wavefront divergence than the outcoupling regions of at least one other waveguide, The different wavefront divergences correspond to different depth planes.

23. The display system according to claim 21, wherein: The incoupling region of each waveguide is configured to couple in light of a wavelength range corresponding to the same component color, Therein, the coupling-in regions of some of the waveguides are configured to couple in light of a wavelength range corresponding to a different component color than the coupling-in regions of other waveguides.

24. The display system of claim 1 further comprising a variable focus lens element located on an opposite side of the waveguide.

25. The display system according to claim 1, wherein: The light projection system includes a plurality of projectors, and wherein projectors of the plurality of projectors are configured to provide image content for subdivided portions that do not exceed a FOV of the head mounted display.

26. A head mounted display system configured to project light to display virtual image content, the display system comprising: Light projection system; as well as A head mounted display configured to display virtual image content, the head mounted display comprising at least one waveguide, each of the at least one waveguide comprising: a plurality of incoupling regions, each incoupling region being configured to receive light having image information for forming the virtual image content from the light projection system and to couple the light into the waveguide; and a plurality of outcoupling regions configured to couple the light out of the waveguide to display the virtual image content, wherein each of the plurality of outcoupling regions is configured to receive light from a different one of the incoupling regions, Wherein, the light projection system comprises: a light source comprising an array of light emitters; a spatial light modulator configured to modulate light from the light source; and projection optics mounted on a frame and configured to direct light from the spatial light modulator to the incoupling region, The display system is configured to activate different pluralities of the light emitters to provide the virtual image content at different depth planes, wherein the display system is configured to: providing a relatively large distance between respective light emitters of the plurality of light emitters for providing the virtual image content on a relatively close depth plane; and A relatively small distance is provided between respective light emitters of the plurality of light emitters for providing the virtual image content on a relatively far depth plane.

27. The display system of claim 26, configured to sequentially activate each of a plurality of light emitters at different times, wherein: The display system is configured to synchronize activation of each light emitter with display of a different parallax-dependent intra-pupillary image.

28. The display system according to claim 26, wherein: The light source is a microdisplay and the light emitters are light emitting diodes.

Images