Depth based central fovea rendering for display systems

Abstract

To provide favorable depth based central fovea rendering for display systems.SOLUTION: A display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include determining a fixation point of user's eyes. Location information associated with a first virtual object to be presented to the user via a display device is obtained. A resolution-modifying parameter of the first virtual object is obtained. A particular resolution at which the first virtual object should be rendered is identified based on the location information and the resolution-modifying parameter of the first virtual object. The particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point. The first virtual object rendered at the identified resolution is presented to the user via the display system.SELECTED DRAWING: Figure 10A

Description

[Technical Field]

[0001] (Priority Claim) This application claims the benefit of priority to U.S. Provisional Application No. 62 / 644,366, filed March 16, 2018, which is incorporated herein by reference in its entirety. (Incorporated by Reference)

[0002] This application incorporates by reference the following patent applications and publications: U.S. Patent Application No. 14 / 555,585, filed November 27, 2014, published July 23, 2015 as U.S. Patent Publication No. 2015 / 0205126; U.S. Patent Application No. 14 / 690,401, filed April 18, 2015, published October 22, 2015 as U.S. Patent Publication No. 2015 / 0302652; U.S. Patent Application No. 9,411,412, filed March 14, 2014, published August 16, 2016; This application incorporates in its entirety each of U.S. Patent Application No. 14 / 212,961, filed July 14, 2014, published as U.S. Patent Publication No. 2015 / 0309263 on October 29, 2015, U.S. Patent Application No. 14 / 331,218, filed February 22, 2018, U.S. Provisional Patent Application No. 62 / 475,012, filed March 22, 2017, and U.S. Provisional Patent Application No. 62 / 539,934, filed August 1, 2017.

[0003] The present disclosure relates to display systems, including augmented reality imaging and visualization systems. [Background technology]

[0004] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears, or can be perceived as, real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual image information as an augmentation to the user's visualization of the real world around them. Mixed reality, or "MR," scenarios are a type of AR scenario that typically involve virtual objects integrated into and responsive to the natural world. For example, an MR scenario may include AR image content that appears blocked by or is perceived to otherwise interact with objects in the real world.

[0005] Referring to FIG. 1 , an augmented reality scene 10 is depicted. A user of the AR technology sees a real-world park-like setting 20 featuring people, trees, and buildings in the background, and a concrete platform 30. The user also perceives that they are "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30 and a flying, cartoonish avatar character 50 that appears to be an anthropomorphic bumblebee. These elements 50, 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex, making it difficult to produce AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] The systems and methods disclosed herein address various challenges associated with AR or VR technology. Summary of the Invention [Means for solving the problem]

[0007] According to some embodiments, a system includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include monitoring eye movement of a user based on information detected via one or more sensors. A fixation point at which the user's eye is fixating is determined based on the eye movement, the fixation point being a three-dimensional location within the user's field of view. The operations include obtaining location information associated with one or more virtual objects and presenting the location information indicating the three-dimensional positions of the virtual objects to the user. The operations also include adjusting a resolution of the at least one virtual object based, at least in part, on a proximity of the at least one virtual object to the fixation point. The operations also include causing presentation of the virtual object to the user via a display, wherein the at least one virtual object is rendered according to the adjusted resolution.

[0008] According to some embodiments, a display system includes a display device configured to present virtual content to a user, one or more processors, and one or more computer storage media storing instructions that, when executed by the system, cause the system to perform operations. The operations include monitoring information associated with a user's eye movements. A fixation point within a display frustum of the display device is determined based on the monitored information, the fixation point indicating a three-dimensional location fixated by the user's eye. The operations also include presenting virtual content at a three-dimensional location within the display frustum based on the determined fixation point, the virtual content being adjusted in resolution based on the proximity of the virtual content from the fixation point.

[0009] According to another embodiment, a method includes monitoring eye movement of a user based on information detected via one or more sensors. A fixation point at which the user's eye is fixating is determined based on the eye movement, the fixation point being a three-dimensional location within the user's field of view. Location information associated with one or more virtual objects for presentation to the user is obtained, the location information indicating the three-dimensional positions of the virtual objects. A resolution of the at least one virtual object is adjusted, at least in part, based on a proximity of the at least one virtual object to the fixation point. The method also includes causing presentation of the virtual objects to the user via a display, the at least one virtual object being rendered according to the adjusted resolution.

[0010] According to some embodiments, a display system includes a frame configured to mount on a user's head, a light modulation system configured to output light and form an image, and one or more waveguides attached to the frame and configured to receive light from the light modulation system and output light across a surface of the one or more waveguides. The system also includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The operations include determining an amount of light reaching a retina of a user's eye and adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina.

[0011] According to certain other embodiments, a display system includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations, including determining an amount of light reaching a retina of an eye of a user of the display system and adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina.

[0012] According to some embodiments, a method is implemented by a display system comprising one or more processors and a head-mountable display, the method including determining an amount of light reaching a retina of an eye of a user of the display system, and adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina.

[0013] In another embodiment, a display system includes a frame configured to be mounted on a user's head, a light modulation system, one or more waveguides, one or more processors, and one or more computer storage media storing instructions. The light modulation system is configured to output light and form an image. The one or more waveguides are attached to the frame and configured to receive light from the light modulation system and output light across a surface of the one or more waveguides. The one or more computer storage media store instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The operations include adjusting resolution of primary color images forming the virtual content based on proximity of the virtual content from a user's fixation point and colors of the primary color images. At least one of the primary color images differs in resolution from a primary color image of another color.

[0014] According to yet another embodiment, a display system includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations, including adjusting a resolution of primary color images forming the virtual content based on a proximity of the virtual content from a user fixation point and a color of the primary color images, at least one of the primary color images differing in resolution from a primary color image of another color.

[0015] According to another embodiment, a method is implemented by a display system including one or more processors and a head-mountable display, the method including adjusting resolution of primary color images forming the virtual content based on proximity of the virtual content from a user fixation point and colors of the primary color images, at least one of the primary color images differing in resolution from another color primary color image.

[0016] According to yet another embodiment, a display system includes an image source including a spatial light modulator for providing a first image stream and a second image stream, a viewing assembly, one or more processors in communication with the image source, and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The viewing assembly includes light guidance optics for receiving the first and second image streams from the image source and outputting the first and second image streams to a user. The various operations performed by the one or more processors include causing the image source to output the first image stream to the viewing assembly, where an image formed by the first image stream has a first pixel density, and causing the image source to output the second image stream to the viewing assembly. The image formed by the second image stream has a second pixel density greater than the first pixel density and corresponds to a portion of the image provided by the first image stream. The image formed by the second image stream covers a corresponding portion of the field of view provided by the first image stream.

[0017] According to some embodiments, a wearable display system may include an afocal magnifying lens with circular polarization handedness-dependent magnification. The afocal magnifying lens may include a first fixed focal length lens element, a first geometric phase lens that exhibits positive refractive power for a first handedness of incident circularly polarized light and a second geometric phase lens that exhibits negative refractive power for a second handedness of incident circularly polarized light.

[0018] According to certain other embodiments, an optical subsystem for a wearable image projector may include a polarization-selective reflector and a set of four lens elements centered around the polarization-selective reflector.

[0019] According to another embodiment, a display system for projecting an image to a user's eye may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include a first image source configured to project a first light beam associated with a first image stream. The first image stream may have a first field of view and may be incident on a first surface of the internal coupling grating. A portion of the first light beam may be coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to the user's eye. The display system may further include a second image source configured to project a second light beam associated with a second image stream. The second image stream may have a second field of view that is narrower than the first field of view. The display system may further include a scanning mirror configured to receive and reflect the second light beam so that the second light beam is incident on a second surface of the internal coupling grating as opposed to the first surface. A portion of the second light beam may be coupled into the waveguide by the internal coupling grating. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

[0020] According to another embodiment, a display system for projecting an image to a user's eye may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view, and the second image stream may have a second field of view narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system may further include a polarizing beam splitter configured to receive the first light beam and reflect it along a first optical path and receive the second light beam and transmit it along a second optical path. The display system may further include a first optical reflector positioned along the first optical path and configured to receive and reflect the first light beam so that the first light beam is incident on a first surface of the internal coupling grating. A portion of the first light beam may be coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to the user's eye. The display system may further include a scanning mirror positioned along the second optical path and configured to receive and reflect the second light beam, and a second optical reflector positioned along the second optical path downstream of the scanning mirror. The second optical reflector may be configured to receive and reflect the second light beam so that the second light beam is incident on a second surface of the internal coupling grating opposite the first surface of the scanning mirror. A portion of the second light beam may be coupled into the waveguide by the internal coupling grating. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuitry may be configured to position the scanning mirror such that the position of the second image stream is moved according to detected movements of the user's eye.

[0021] According to another embodiment, a display system for projecting an image to a user's eye may include a waveguide and an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view, and the second image stream may have a second field of view narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system may further include a polarizing beam splitter configured to receive the first light beam and reflect it along a first optical path and receive the second light beam and transmit it along a second optical path. The display system may further include a first incoupling prism positioned along the first optical path and adjacent to a first surface of the waveguide. The first incoupling prism may be configured to couple a portion of the first light beam into the waveguide to position the first image stream at a fixed position relative to the user's eye. The display system may further include a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam. The display system may further include a second incoupling prism positioned along the second optical path and downstream of the scanning mirror, adjacent to a second surface of the waveguide as opposed to a first surface of the waveguide. The second incoupling prism may be configured to couple a portion of the second light beam into the waveguide. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuitry may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

[0022] According to an embodiment, a display system for projecting an image to a user's eye includes an image source. The image source can be configured to project a first light beam associated with a first image stream at a first polarization and a second light beam associated with a second image stream at a second polarization different from the first polarization. The first image stream can have a first field of view, and the second image stream can have a second field of view narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system can further include a polarizing beam splitter. The polarizing beam splitter can be configured to receive and reflect the first light beam along a first optical path toward a viewing assembly and receive and transmit the second light beam along a second optical path to position the first image stream at a fixed position relative to the user's eye. The display system can further include a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam toward the viewing assembly. The display system can further include an eye gaze tracker configured to detect movement of the user's eye, and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit can be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

[0023] According to another embodiment, a display system for projecting an image to a user's eye includes an image source. The image source can be configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream. The first image stream can have a first field of view, and the second image stream can have a second field of view narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system can further include a scanning mirror configured to receive and reflect the first light beam and the second light beam toward a viewing assembly to project the first image stream and the second image stream. The display system can further include an eye gaze tracker configured to detect movement of the user's eye and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuitry can be configured to position the scanning mirror such that the position of the first image stream and the position of the second image stream are moved according to the detected movement of the user's eye. The display system can further include a switchable optical element disposed in the optical path of the first light beam and the second light beam. The switchable optical element can be configured to be switched to a first state for a first light beam such that the first light beam is angularly expanded by a first angular expansion, and to be switched to a second state for a second light beam such that the second light beam is angularly expanded by a second angular expansion that is less than the first angular expansion.

[0024] In some embodiments, a display system comprises one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including determining an eye fixation point of a user, obtaining location information associated with a first virtual object to be presented to the user via a display device, obtaining a resolution modification parameter for the first virtual object, identifying a particular resolution at which to render the first virtual object based on the location information and the resolution modification parameter of the first virtual object, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point, and causing presentation of the first virtual object rendered at the identified resolution to the user via the display device.

[0025] In some embodiments, a computer-implemented method is provided. The method is performed by a display system of one or more processors. The method includes determining a fixation point of an eye of a user, acquiring location information associated with a first virtual object to be presented to the user via a display device, acquiring a resolution modification parameter for the first virtual object, identifying a specific resolution at which to render the first virtual object based on the location information and the resolution modification parameter of the first virtual object, the specific resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point, and causing presentation of the first virtual object rendered at the identified resolution to the user via the display device.

[0026] In some embodiments, a non-transitory computer storage medium is provided. The computer storage medium stores instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations including determining an eye fixation point of a user, acquiring location information associated with a first virtual object to be presented to the user via a display device, acquiring a resolution modification parameter for the first virtual object, identifying a particular resolution at which to render the first virtual object based on the location information and the resolution modification parameter for the first virtual object, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point, and causing presentation of the first virtual object rendered at the identified resolution to the user via the display device.

[0027] Additional examples of embodiments are provided below.

[0028] 1. A display system comprising: one or more processors; When executed by one or more processors, the one or more processors determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of a first virtual object; identifying a particular resolution at which to render the first virtual object based on the location information of the first virtual object and the resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from a fixation point; causing presentation to the user via the display device of the first virtual object rendered at the identified resolution; one or more computer storage media storing instructions for performing operations, including: A display system comprising:

[0029] 2. The resolution modification parameters include a content type associated with the first virtual object, and the operations further include: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from a plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution comprises a specific resolution distribution; 2. The display system of Example 1, comprising:

[0030] 3. The display system of example 3, wherein the virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

[0031] 4. The display system of example 3, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, and the values ​​of the roll-offs are different for content with different frequency spectra.

[0032] 5. The display system of example 1, wherein the resolution modification parameter is a user-selectable value.

[0033] 6. The display device is configured to adjust a specific resolution, and the step of adjusting the specific resolution includes: causing presentation of a second virtual object to the user via a display device, the second virtual object being rendered in the resolution distribution identified for the first virtual object; receiving a response from the user indicating user detection of a reduction in resolution of the second virtual object, the user response being a user-selectable value; adjusting a particular resolution distribution; 6. The display system of Example 5, comprising:

[0034] 7. The step of adjusting a specific resolution distribution is A display system as described in Example 6, comprising a step of adjusting a roll-off associated with a particular resolution distribution, wherein the step of adjusting the roll-off varies the amount of resolution reduction based on angular distance from the center of the user's field of view.

[0035] 8. The display system of example 1, wherein the fixation point is within a volume at the center of the user's visual field.

[0036] 9. The display system of Example 1, wherein the user's field of view is separated into a plurality of portions based on a resolution distribution, the plurality of portions including a first portion, each portion encompassing a distinct range of angular distances from a center of the field of view, and each portion being assigned an associated resolution at which virtual content should be rendered.

[0037] 10. The operation is further determining a proximity of a boundary of one of the plurality of portions to a first virtual object; modifying the presentation of the first virtual object based on the determined proximity; 10. The display system of Example 9, comprising:

[0038] 11. The display system of Example 9, wherein modifying the presentation of the first virtual object based on the determined proximity includes applying a blurring process to the virtual object.

[0039] 12. Identifying a particular resolution at which to render the first virtual object comprises: identifying a second portion of the plurality of portions that encompasses the first virtual object; identifying a resolution based on the second portion; 10. The display system of Example 9, comprising:

[0040] 13. A computer-implemented method, performed by a display system of one or more processors, comprising: determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of a first virtual object; identifying a particular resolution at which to render the first virtual object based on the location information of the first virtual object and the resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from a fixation point; causing presentation to the user via the display device of the first virtual object rendered at the identified resolution; A method comprising:

[0041] 14. The resolution modification parameters include a content type associated with the first virtual object, and the method further comprises: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from a plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution comprises a specific resolution distribution; 14. The computer-implemented method of example 13, comprising:

[0042] 15. The computer-implemented method of example 14, wherein the virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

[0043] 16. The computer-implemented method of Example 14, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, and the values ​​of the roll-offs are different for content with different frequency spectra.

[0044] 17. A non-transitory computer storage medium that, when executed by a display system of one or more processors, causes the one or more processors to: determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of a first virtual object; identifying a particular resolution at which to render the first virtual object based on the location information of the first virtual object and the resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from a fixation point; causing presentation to the user via the display device of the first virtual object rendered at the identified resolution; 2. A non-transitory computer storage medium for storing instructions for performing operations, including:

[0045] 18. The resolution modification parameters include a content type associated with the first virtual object, and the operations further include: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from a plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution comprises a specific resolution distribution; 18. The computer storage medium of Example 17, comprising:

[0046] 19. The computer storage medium of Example 18, wherein the virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object.

[0047] 20. The computer storage medium of Example 18, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, and the values ​​of the roll-offs are different for content with different frequency spectra. The present specification also provides, for example, the following items: (Item 1) 1. A display system comprising: one or more processors; one or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of the first virtual object; identifying a particular resolution at which to render the first virtual object based on the location information of the first virtual object and a resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point; causing presentation to a user via the display device of the first virtual object rendered at the identified resolution; and one or more computer storage media for performing operations including: A display system comprising: (Item 2) The resolution modification parameters include a content type associated with the first virtual object, and the operation further comprises: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from the plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution distribution includes the specific resolution distribution; Item 1. The display system of item 1, comprising: (Item 3) Item 4. The display system of item 3, wherein a virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object. (Item 4) Item 4. The display system of item 3, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, the values ​​of the roll-offs being different for content with different frequency spectra. (Item 5) Item 10. The display system of item 1, wherein the resolution correction parameter is a user-selectable value. (Item 6) The display device is configured to adjust the particular resolution, and adjusting the particular resolution includes: causing presentation of a second virtual object to the user via the display device, the second virtual object being rendered in the resolution distribution identified for the first virtual object; and receiving a response from the user indicating user detection of a reduction in resolution of the second virtual object, the user response being the user selectable value; adjusting the particular resolution distribution; Item 6. The display system of item 5, comprising: (Item 7) Adjusting the specific resolution distribution comprises: adjusting a roll-off associated with the particular resolution distribution, adjusting the roll-off varying the amount of resolution reduction based on angular distance from a center of the user's field of view. Item 7. The display system of item 6, comprising: (Item 8) Item 1. A display system as described in item 1, wherein the fixation point is within a volume at the center of the user's field of view. (Item 9) Item 1, a display system in which the user's field of view is separated into multiple portions based on the resolution distribution, the multiple portions including the first portion, each portion encompassing a distinct range of angular distances from a center of the field of view, and each portion being assigned an associated resolution at which virtual content should be rendered. (Item 10) The operation further comprises: determining a proximity of the first virtual object to a boundary of one of the plurality of portions; modifying the presentation of the first virtual object based on the determined proximity; and Item 10. The display system of item 9, comprising: (Item 11) 10. The display system of claim 9, wherein modifying the presentation of the first virtual object based on the determined proximity includes applying a blurring process to the virtual object. (Item 12) Identifying a particular resolution at which to render the first virtual object comprises: identifying a second portion of the plurality of portions that includes the first virtual object; identifying the resolution based on the second portion; and Item 10. The display system of item 9, comprising: (Item 13) 1. A computer-implemented method, the method being performed by a display system of one or more processors, the method comprising: determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of the first virtual object; identifying a particular resolution at which to render the first virtual object based on location information of the first virtual object and a resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point; causing presentation to a user via the display device of the first virtual object rendered at the identified resolution; and 11. A computer-implemented method comprising: (Item 14) The resolution modification parameters include a content type associated with the first virtual object, and the method further comprises: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from the plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution distribution includes the specific resolution distribution; Item 14. The computer-implemented method of item 13, comprising: (Item 15) Item 15. The computer-implemented method of item 14, wherein a virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object. (Item 16) Item 15. The computer-implemented method of item 14, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, the values ​​of the roll-offs being different for content with different frequency spectra. (Item 17) A non-transitory computer storage medium having stored thereon instructions that, when executed by a display system of one or more processors, cause the one or more processors to: determining a fixation point of a user's eye; obtaining location information associated with a first virtual object to be presented to a user via a display device; obtaining a resolution modification parameter of the first virtual object; identifying a particular resolution at which to render the first virtual object based on location information of the first virtual object and a resolution modification parameter, the particular resolution being based on a resolution distribution that defines a resolution for a corresponding distance from the fixation point; causing presentation to a user via the display device of the first virtual object rendered at the identified resolution; and A non-transitory computer storage medium for performing operations including: (Item 18) The resolution modification parameters include a content type associated with the first virtual object, and the operation further comprises: accessing a plurality of resolution distributions, each resolution distribution associated with a distinct virtual content type; selecting a specific resolution distribution from the plurality of resolution distributions based on a content type of the first virtual object, wherein the specific resolution distribution includes the specific resolution distribution; Item 18. The computer storage medium of item 17, comprising: (Item 19) Item 19. The computer storage medium of item 18, wherein the virtual content type associated with the first virtual object is identified based on a frequency spectrum associated with the first virtual object. (Item 20) 20. The computer storage medium of claim 18, wherein the multiple resolution distributions are associated with distinct roll-offs in resolution away from the fixation point, the values ​​of the roll-offs being different for content with different frequency spectra. [Brief explanation of the drawings]

[0048] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) device.

[0049] [Figure 2] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.

[0050] [Figure 3] 3A-3C illustrate the relationship between the radius of curvature and the radius of focus.

[0051] [Figure 4A] Figure 4A illustrates a representation of the accommodation-vergence response of the human visual system.

[0052] [Figure 4B] FIG. 4B illustrates an example of different accommodation and convergence states of a pair of a user's eyes.

[0053] [Figure 4C] FIG. 4C illustrates an example of a top-down view representation of a user viewing content through a display system.

[0054] [Figure 4D]FIG. 4D illustrates another example of a top-down view representation of a user viewing content through a display system.

[0055] [Figure 5] FIG. 5 illustrates aspects of an approach for simulating three-dimensional images by correcting for wavefront divergence.

[0056] [Figure 6] FIG. 6 illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0057] [Figure 7] FIG. 7 illustrates an example of an output beam output by a waveguide.

[0058] [Figure 8] FIG. 8 illustrates an example of a stacked waveguide assembly where each depth plane contains an image formed using multiple different primary colors.

[0059] [Figure 9A] FIG. 9A illustrates a cross-sectional side view of an example of a set of stacked waveguides, each including an internal coupling optical element.

[0060] [Figure 9B] FIG. 9B illustrates a perspective view of the multiple stacked waveguide embodiment of FIG. 9A.

[0061] [Figure 9C] FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B.

[0062] [Figure 9D] FIG. 9D illustrates an example of a wearable display system.

[0063] [Figure 10A]FIG. 10A illustrates an example of a top-down view representation of a user viewing content through a display system.

[0064] [Figure 10B] FIG. 10B illustrates another example of a top-down view representation of a user viewing content through a display system.

[0065] [Figure 10C] FIG. 10C illustrates yet another example of a top-down view representation of a user viewing content through a display system.

[0066] [Figure 10D] FIG. 10D is a block diagram of an exemplary display system.

[0067] [Figure 11A1] FIG. 11A1 illustrates an example of a top-bottom view representation of accommodation at resolution within different resolution accommodation zones based on 3D fixation tracking.

[0068] [Figure 11A2] FIG. 11A2 illustrates an example of a top-down view representation of the resolution adjustment zones at different times as the size and number of zones are varied.

[0069] [Figure 11B] FIG. 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone of FIG. 11A1.

[0070] [Figure 11C] FIG. 11C illustrates another example of a configuration for a resolution adjustment zone.

[0071] [Figure 11D] FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

[0072] [Figure 11E]FIG. 11E illustrates another example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

[0073] [Figure 12A] 12A-12C show schematic diagrams of an embodiment of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point. [Figure 12B] 12A-12C show schematic diagrams of an embodiment of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point. [Figure 12C] 12A-12C show schematic diagrams of an embodiment of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point.

[0074] [Figure 13] FIG. 13 illustrates an example of a representation of a user viewing multiple virtual objects aligned with the user's line of sight.

[0075] [Figure 14] FIG. 14 is a diagram of an example process for adjusting virtual content based on angular proximity to a user's line of sight.

[0076] [Figure 15] FIG. 15 illustrates an example of a representation of the retina of a user's eye.

[0077] [Figure 16] FIG. 16 illustrates diagrammatically an example of resolution and rod and cone density across the retina of FIG.

[0078] [Figure 17] FIG. 17 graphically illustrates an example of the relationship between pupil size and the amount of light incident on a user's eye.

[0079] [Figure 18] FIG. 18 is a schematic diagram of an example process for adjusting virtual content based on the amount of light incident on a user's eye.

[0080] [Figure 19] FIG. 19 graphically illustrates an example of the change in resolution detectable by a user's eye as the amount of light incident on the eye changes.

[0081] [Figure 20] FIG. 20 graphically illustrates an example of the differences in the eye's sensitivity to light of different colors at different levels of illumination.

[0082] [Figure 21] FIG. 21 is a schematic diagram of an example process for adjusting virtual content created using multiple primary color images, where the resolution adjustment is based on the color of the primary color images.

[0083] [Figure 22] 22A-22C illustrate examples of contrast sensitivity changing as the amount of light incident on a user's eye decreases.

[0084] [Figure 23] FIG. 23 illustrates an example of a representation of the optic nerve and peripheral blind spot of a user's eye.

[0085] [Figure 24] FIG. 24 shows an exemplary monocular field of view for the human eye.

[0086] [Figure 25A] FIG. 25A shows an exemplary wearable display device configured to provide virtual content to a user.

[0087] [Figure 25B] FIG. 25B is a block diagram depicting an augmented reality system.

[0088] [Figure 25C]FIG. 25C illustrates diagrammatically the light paths within a viewing optical assembly (VOA), which may be used to present a digital or virtual image to a viewer.

[0089] [Figure 26A] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26B] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26C] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26D] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations.

[0090] [Figure 26E] 26E-26F diagrammatically illustrate exemplary configurations of images that may be presented to a user. [Figure 26F] 26E-26F diagrammatically illustrate exemplary configurations of images that may be presented to a user.

[0091] [Figure 26G] 26G-26H diagrammatically illustrate exemplary configurations of images that may be presented to a user. [Figure 26H] 26G-26H diagrammatically illustrate exemplary configurations of images that may be presented to a user.

[0092] [Figure 27] FIG. 27 illustrates the field of view and oculomotor field of view as shown in FIG. 24 overlaid on one of the displays in a wearable display device as shown in FIG.

[0093] [Figure 28A] Figures 28A-28B illustrate some of the principles described in Figures 26A-26D. [Figure 28B] Figures 28A-28B illustrate some of the principles described in Figures 26A-26D.

[0094] [Figure 28C] 28C-28D illustrate some example images that may be presented to the user. [Figure 28D] 28C-28D illustrate some example images that may be presented to the user.

[0095] [Figure 28E] FIG. 28E illustrates an exemplary high FOV low resolution image frame.

[0096] [Figure 28F] FIG. 28F illustrates an exemplary low FOV high resolution image frame.

[0097] [Figure 29A] FIG. 29A shows a simplified block diagram of a display system.

[0098] [Figure 29B] FIG. 29B diagrammatically illustrates a cross-sectional view of an augmented reality (AR) system.

[0099] [Figure 30A] 30A-30B diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 30B] 30A-30B diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0100] [Figure 30C]FIG. 30C schematically illustrates a cross-sectional view of an augmented reality (AR) system.

[0101] [Figure 30D] FIG. 30D shows a simplified block diagram of a display system.

[0102] [Figure 31A] FIG. 31A diagrammatically illustrates the operating principle of the first relay lens assembly in the display system illustrated in FIGS. 30A-30B.

[0103] [Figure 31B] FIG. 31B diagrammatically illustrates the operating principle of the second relay lens assembly in the display system illustrated in FIGS. 30A-30B.

[0104] [Figure 31C] 31C-31D illustrate a schematic representation of the display system. [Figure 31D] 31C-31D illustrate a schematic representation of the display system.

[0105] [Figure 32A] 32A-32C illustrate a schematic representation of the display system. [Figure 32B] 32A-32C illustrate a schematic representation of the display system. [Figure 32C] 32A-32C illustrate a schematic representation of the display system.

[0106] [Figure 33A] 33A-33B illustrate a schematic representation of the display system. [Figure 33B] 33A-33B illustrate a schematic representation of the display system.

[0107] [Figure 34A] 34A-34B illustrate a schematic representation of the display system. [Figure 34B] 34A-34B illustrate a schematic representation of the display system.

[0108] [Figure 35] FIG. 35 illustrates a schematic of the display system.

[0109] [Figure 36] FIG. 36 illustrates diagrammatically an augmented reality eyepiece display system.

[0110] [Figure 37A] FIG. 37A is a schematic diagram of a 2x afocal magnifying lens.

[0111] [Figure 37B] FIG. 37B is a schematic diagram of a bifocal magnifying afocal magnifying lens.

[0112] [Figure 38] 38A-38B diagrammatically illustrate exemplary configurations of images that may be presented to a user.

[0113] [Figure 39A] 39A-39B illustrate some example images that may be presented to the user. [Figure 39B] 39A-39B illustrate some example images that may be presented to the user.

[0114] [Figure 40A] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40B] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40C] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40D]40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0115] [Figure 41A] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41B] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41C] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41D] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0116] [Figure 42] FIG. 42 illustrates an exemplary frame structure for time-division multiplexed high FOV low-resolution image streams and low FOV high-resolution image streams.

[0117] [Figure 43] FIG. 43 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0118] [Figure 44] FIG. 44 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0119] [Figure 45] FIG. 45 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0120] [Figure 46] FIG. 46 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0121] [Figure 47] FIG. 47 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0122] [Figure 48] FIG. 48 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0123] [Figure 49] FIG. 49 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0124] [Figure 50] FIG. 50 diagrammatically illustrates a display system for projecting an image stream to a user's eyes, according to some embodiments.

[0125] [Figure 51] FIG. 51 diagrammatically illustrates a display system for projecting an image stream to a user's eye, according to some embodiments.

[0126] [Figure 52A] 52A-52B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments. [Figure 52B] 52A-52B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments.

[0127] [Figure 53A] 53A-53B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments. [Figure 53B]53A-53B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments.

[0128] [Figure 54] FIG. 54 illustrates a representation of a user's viewing angle along with an example resolution distribution.

[0129] [Figure 55A] 55A-55B illustrate an example scheme for identifying roll-off for a resolution distribution based on the type of virtual content. [Figure 55B] 55A-55B illustrate an example scheme for identifying roll-offs for resolution distributions based on the type of virtual content.

[0130] [Figure 55C] 55C-55D illustrate graphs of the average roll-off determined for different types of image content. [Figure 55D] 55C-55D illustrate graphs of the average roll-off determined for different types of image content.

[0131] [Figure 56] FIG. 56 illustrates a flowchart of an example process 5600 for determining the roll-off to be utilized in the resolution distribution.

[0132] [Figure 57] FIG. 57 illustrates an example flowchart of a process for presenting virtual content according to the type of virtual content.

[0133] [Figure 58A] FIG. 58A illustrates two example blur regions.

[0134] [Figure 58B] FIG. 58B illustrates two additional example blur regions.

[0135] [Figure 59] FIG. 59 illustrates an example of different resolution adjustment zones according to the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0136] Rendering virtual content for augmented and virtual display systems is computationally intensive, particularly computationally intensive, which may undesirably use large amounts of memory, cause long latency, and / or require the use of powerful processing units that may have high cost and / or high energy consumption.

[0137] In some embodiments, methods and systems conserve computational resources, such as memory and processing time, by reducing the resolution of virtual content located away from a user's eye fixation point. For example, the system may render virtual content at or near the user's eye fixation point at a relatively high (e.g., highest) resolution, while utilizing one or more lower resolutions for virtual content away from the fixation point. The virtual content is presented by a display system that can display virtual content at multiple different depths (e.g., multiple different depth planes, such as two or more depth planes), and the reduction in resolution preferably occurs at least along the z-axis, which is the depth axis (corresponding to distance away from the user). In some embodiments, the resolution reduction occurs along the z-axis and one or both of the x- and y-axes, where the x-axis is the lateral axis and the y-axis is the vertical axis.

[0138] Determining an appropriate resolution for the virtual content may include determining a fixation point for the user's eyes in three-dimensional space. For example, the fixation point may be an x, y, z coordinate in the user's field of view on which the user's eyes are fixated. The display system may be configured to present virtual objects with differences in resolution, where the resolution decreases with decreasing proximity of the virtual object to the fixation point, or in other words, the resolution decreases with increasing distance from the fixation point.

[0139] As discussed herein, a display system may present virtual objects within a display frustum of the display system, where the virtual objects can be presented at different depth planes. In some embodiments, the display frustum is a field of view provided by the display system over which the display system is configured to present virtual content to a user of the display system. The display system may be a head-mounted display system including one or more waveguides that may present virtual content (e.g., virtual objects, graphics, text, etc.), where the one or more waveguides are configured to output light with different wavefront divergences and / or different binocular disparities corresponding to different depth planes (e.g., corresponding to particular distances from the user). It should be understood that each eye may have one or more associated waveguides. Using the different wavefront divergences and / or different binocular disparities, the display system may cause a first virtual object to appear to be located at a first depth within the user's field of view, while a second virtual object to appear to be located at a second depth within the user's field of view. In some embodiments, the depth plane of the fixation point or a nearby depth plane may be determined, and the resolution of content on other depth planes may be reduced based on the distance from those depth planes to the depth plane where the fixation point is located. It should be understood that references herein to the depth of virtual content (the distance of the virtual content from the user on the z-axis) refer to the apparent depth of the virtual content as intended to be seen by the user. In some embodiments, the depth of a virtual object may be understood as the distance from the user of a real object that has wavefront divergence and / or binocular disparity similar to that of the virtual object.

[0140] It should be understood that the proximity of a virtual object to a fixation point may be determined by various measures, non-limiting examples of which include determining the distance between the fixation point and the virtual object, determining the resolution adjustment zone occupied by the virtual object relative to the resolution adjustment zone occupied by the fixation point (in embodiments in which the user's field of view is subdivided into resolution adjustment zones, as described below), and determining the angular proximity of the virtual object to the user's fixation point. Proximity may also be determined using a combination of the above techniques. For example, the distance and / or angular proximity of a first zone (in which the virtual object is located) and a second zone (in which the fixation point is located) may be used to determine proximity. These various measures are discussed further below.

[0141] In some embodiments, determining a fixation point may include anticipating a fixation point of a user's eyes and utilizing the anticipated fixation point as the fixation point for determining the resolution of the virtual content. For example, a display system may render particular content at a relatively high resolution with the anticipation that the user's eyes will fixate on that content. As an example, it should be understood that the human visual system may be sensitive to sudden changes in a scene (e.g., sudden motion, changes in brightness, etc.). In some embodiments, a display system may determine that virtual content is of a certain type (e.g., involving motion in a scene where other virtual and real objects are stationary) that will cause the user's eyes to fixate on it, and then render the virtual content at a high resolution with the anticipation that the user's eyes will subsequently focus on the virtual content.

[0142] As noted above, in some embodiments, the distance from the determined fixation point to the virtual object may correspond to a distance extending in three dimensions. As an example, a first virtual object located at the same depth (e.g., the same depth plane) from the user as the determined fixation point, but located horizontally or vertically from the fixation point, may similarly be reduced in resolution because a second virtual object is located at a farther depth (e.g., a farther depth plane) from the determined fixation point. As a result, different resolutions may be associated with different distances from the fixation point.

[0143] In some embodiments, the environment around the user may be divided into spatial volumes (also referred to herein as resolution adjustment zones), and the resolution of virtual objects within the same resolution adjustment zone may be similar. The resolution adjustment zones may have arbitrary three-dimensional shapes, such as cubes, or other three-dimensional polygonal shapes, or curved three-dimensional shapes, as described herein. In some embodiments, all resolution adjustment zones have similar shapes, such as rectangular prisms or spheres. In some other embodiments, different resolution adjustment zones may have different shapes or sizes (e.g., the shape and / or size of the volumes may change with distance from the fixation point).

[0144] In some embodiments, the resolution adjustment zone is a portion of the user's field of view. For example, the user's field of view may be separated into spatial volumes that form the resolution adjustment zones. In some embodiments, each depth plane may be subdivided into one or more contiguous spatial volumes, i.e., one or more resolution adjustment zones. In some embodiments, each resolution adjustment zone may encompass a specific range of depth from the user (e.g., the depth plane value + / - a variance amount, with example variance amounts including 0.66 dpt, 0.50 dpt, 0.33 dpt, or 0.25 dpt) and a specific lateral and vertical distance. Virtual objects located within the same resolution adjustment zone as the determined fixation point may be presented (e.g., rendered) at high (e.g., full) resolution, while virtual objects located in a spatial volume outside the resolution adjustment zone of the fixation point may be rendered at a lower resolution according to the distance of the volume from the spatial volume of the fixation point. In some embodiments, each resolution adjustment zone may be assigned a particular resolution (e.g., a particular reduction in resolution relative to full resolution), and virtual content that falls within a given zone may be rendered at the associated resolution for that zone. In some embodiments, the distance between a volume and a volume occupied by a fixation point may be determined, and the resolution may be set based on this distance.

[0145] Advantageously, the number and size of the resolution adjustment zones utilized to divide the user's field of view may be modified according to the confidence in the user's determined fixation point. For example, the size associated with each spatial volume may be increased or decreased based on the confidence that the user's gaze is converged on a precise point in three-dimensional space. If the confidence in the fixation point is high, the display system may present only virtual objects within a compact resolution adjustment zone at a relatively high resolution (a compact resolution adjustment zone that includes the fixation point), while reducing the resolution of other virtual objects, thus saving processing power. However, if the confidence is low, the display system may increase the size of each spatial volume (e.g., reduce the overall number of volumes) so that each spatial volume encompasses a larger number of virtual objects within the spatial volume of the fixation point. It should be understood that the size and shape of the volumes may be fixed during production of the display system, for example, based on expected tolerances in the system for determining the fixation point, and / or may be adjusted or set in the field in response to user characteristics, the user's environment, and / or software changes that alter tolerances for the system for determining the fixation point.

[0146] It should be understood that a user's sensitivity to resolution may decrease with distance from the fixation point. Consequently, by ensuring that full-resolution content is presented at the fixation point and by allowing for a tolerance of error as to where the fixation point is located, the perceptibility of the reduction in resolution may be reduced or eliminated, thereby providing the perception of a high-resolution display without utilizing the computational resources typically required to present content for such a high-resolution display.

[0147] In some embodiments, the proximity of a virtual object to a fixation point may be determined based on the angular proximity of the virtual object to the user's line of sight, and the resolution of the virtual object may decrease as the angular proximity decreases. In some embodiments, this may result in virtual objects located at different depths from the user being presented with similar resolution. For example, a first virtual object at a location corresponding to the user's determined fixation point may be located in front of a second virtual object (e.g., closer to the user in depth). The second virtual object may optionally be presented with similar (e.g., the same) resolution as the first virtual object because the second virtual object is along the user's line of sight and therefore will similarly fall on the user's fovea, where the user's eyes are most sensitive to changes in resolution. Optionally, the second virtual object may be further adjusted via a blurring process (e.g., a Gaussian blur kernel may be convolved with the second virtual object), which may reduce in resolution and represent the second virtual object being farther from the user (e.g., located on a more distant depth plane).

[0148] The reduction in resolution may vary based on how the virtual content is presented by the display system. In some embodiments, a first exemplary display system, referred to herein as a variable-focus display system, may present virtual content on different depth planes, with all content (e.g., virtual objects) being presented at the same depth plane (e.g., via the same waveguide) at one time for each frame presented to the user. That is, the variable-focus display system may utilize a single depth plane at a time (e.g., selected from multiple depth planes based on the user's fixation point, or selected based on the depth of a particular presented virtual object) to present content, or may change the depth plane (e.g., select a different depth plane) in subsequent frames. In some other embodiments, a second exemplary display system, referred to herein as a multi-focal display system, may present virtual content on different depth planes, with content being displayed on multiple depth planes simultaneously. As will be described further herein, a variable-focus display system may optionally utilize a single frame buffer, and with respect to the above example regarding blurring the second virtual object, the second virtual object may be blurred prior to presentation to the user from the single frame buffer. In contrast, a multi-focal display system may optionally present the second virtual object at a greater depth (e.g., on a greater depth plane) from the first virtual object, and at a reduced resolution, and the second virtual object may appear to the user as blurred (e.g., the second virtual object will be blurred based on the natural physics of the user's eye without further processing).

[0149] As disclosed herein, the display system may present virtual objects at or near a determined fixation point at a relatively high (e.g., full) resolution and may present virtual objects further from the fixation point at a reduced resolution. Preferably, the relatively high resolution is the highest resolution for presentation of virtual objects within the user's field of view. The relatively high resolution may be the maximum resolution of the display system, a user-selectable resolution, a resolution based on the specific computing hardware presenting the virtual objects, etc.

[0150] It should be understood that adjusting the resolution of a virtual object may include any modification to the virtual object to alter the quality of the presentation of the virtual object, including adjusting quality at one or more points in the graphics pipeline of a graphics processing unit (GPU), and may include one or more of: adjusting the number of polygons of the virtual object, adjusting the primitives utilized to generate the virtual object (e.g., adjusting the shape of the primitives, e.g., adjusting the primitives from a triangular mesh to a quadrilateral mesh, etc.), adjusting operations performed on the virtual object (e.g., shader operations), adjusting texture information, color resolution, or depth, adjusting the number of rendering cycles or frame rate, etc.

[0151] In some embodiments, on the x- and y-axes, changes in resolution of the virtual content away from the fixation point may generally track changes in the distribution of photoreceptors within the retina of the user's eye. For example, it should be understood that views of the world and virtual content may be imaged onto the retina such that different portions of the retina may be mapped to different portions of the user's field of view. Advantageously, the resolution of the virtual content across the user's field of view may generally track the density of corresponding photoreceptors (rods or cones) across the retina. In some embodiments, the resolution reduction away from the fixation point may generally track the reduction in cone density across the retina. In some other embodiments, the resolution reduction away from the fixation point may generally track the reduction in rod density across the retina. In some embodiments, the trend of resolution reduction away from the fixation point may be within ±50%, ±30%, ±20%, or ±10% of the trend of the reduction in rod and / or cone density across the retina.

[0152] Rods and cones are active at different levels of incident light. For example, cones are active under relatively bright conditions, while rods are active under relatively low-light conditions. As a result, in some embodiments, the display system may be configured to determine the amount of light incident on the retina, where the reduction in resolution generally tracks the density of rods or cones across the retina. Based on this amount of light, an appropriate adjustment in resolution may be made. For example, the reduction in resolution may generally track changes in rod density across the retina under low-light conditions, while the reduction in resolution may generally track changes in cone density under bright conditions. As a result, in some embodiments, the display system may be configured to vary the profile of the reduction in image resolution based on the amount of light incident on the retina.

[0153] It should be understood that the ability of the human eye to resolve fine detail may not be directly proportional to the density of rods or cones in the retina. In some embodiments, changes in the resolution of virtual content across a user's field of view generally track changes in the eye's ability to resolve fine detail. As noted above, the rate at which the resolution of the virtual content changes may vary with the amount of light reaching the retina.

[0154] In some embodiments, the amount of light reaching the retina may be determined by detecting the amount of ambient light incident on a sensor mounted on the display device. In some embodiments, determining the amount of light reaching the retina may also include determining it by the amount of light output to the user by the display device. In yet other embodiments, the amount of light reaching the retina may be determined by imaging the user's eye and determining pupil size. Because pupil size is related to the amount of light reaching the retina, determining pupil size allows the amount of light reaching the retina to be extrapolated.

[0155] It should be understood that full-color virtual content may be formed by multiple primary color images that, in the aggregate, provide a full-color perception. The human eye may have different sensitivities to different wavelengths or colors of light. In some embodiments, in addition to varying based on proximity to a fixation point, changes in resolution of the virtual content may vary based on the color of the primary color images presented by the display system. For example, if the primary color images comprise red, green, and blue images, the green primary color image may have a higher resolution than the blue primary color image, which may have a higher resolution than the red primary color image. In some embodiments, to account for changes in the eye's sensitivity to different colors at different levels of incident light, the amount of light reaching the retina may be determined, and the resolution adjustment for a given primary color image may also vary based on a determination of the amount of light reaching the retina.

[0156] It should be understood that the contrast sensitivity of the eye may also vary based on the amount of light incident on the retina. In some embodiments, the size or total number of gradations in contrast within the virtual content may vary based on the amount of light reaching the retina. In some embodiments, the contrast ratio of the images forming the virtual content may vary based on the amount of light incident on the retina, with the contrast ratio decreasing as the amount of light decreases.

[0157] In some embodiments, certain portions of a user's field of view may not be provided with any virtual content. For example, the display system may be configured to not provide virtual content within the blind spot caused by the optic nerve and / or peripheral blind spot of a given eye.

[0158] As discussed herein, a display system may be configured to display high-resolution content in one portion of a user's field of view and lower-resolution content in another portion of the user's field of view. It should be understood that the high-resolution content may have a higher pixel density than the lower-resolution content. In some environments, a display system may be configured to provide such high- and low-resolution content by effectively overlaying high-resolution and low-resolution images. For example, the system may display a low-resolution image that spans the entire field of view and then display a high-resolution image that spans a small portion of the field of view, with the high-resolution image co-located with the corresponding portion of the low-resolution image. The high- and low-resolution images may be routed through different optics that output light at the appropriate angles and determine the amount of the field of view that the images occupy.

[0159] In some embodiments, a single spatial light modulator (SLM) may be used to encode light with image information, and a beam splitter or optical switch may be used to split the single light stream from the SLM into two streams, one stream propagating through an optical system for a low-resolution image and the second stream propagating through an optical system for a high-resolution image. In some other embodiments, the polarization of the light encoded with image information may be selectively switched and passed through an optical system that effectively provides different angular magnifications for the different polarizations of light, thereby providing high and low resolution images.

[0160] Advantageously, various embodiments disclosed herein reduce the processing power requirements for providing content on a display system. Because a larger allocation of processing power may be devoted to virtual objects closer to a user's three-dimensional fixation point, while processing power for more distant virtual objects may be reduced, the overall required processing power for the display system may be reduced, thus reducing one or more of the size of the processing components, the heat generated by the processing components, and the energy requirements for the display system (e.g., the display system may optionally be battery-powered, require a lower-capacity battery, and / or operate for a longer duration for a given battery). Thus, the embodiments described herein address technical issues arising from augmented or virtual reality display systems. Additionally, the described techniques manipulate graphical content such that the graphical content may appear to the user to be identical, while being presented fundamentally differently (e.g., the resolution may be modified) depending on the presentation to the user. Thus, the display system preserves visual fidelity and conserves processing power while transforming the graphical content as the user looks around their surroundings.

[0161] It should be understood that the display system may be part of an augmented reality display system or a virtual reality display system. As one example, the display of the display system may be transparent, allowing a user to view the real world while providing the user with virtual content in the form of images, videos, interactions, etc. As another example, the display system may block the user's view of the real world, and virtual reality images, videos, interactions, etc. may be presented to the user.

[0162] Reference is now made to the drawings, wherein like reference numerals refer to like parts throughout.

[0163] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. It should be understood that when a user's eyes are spaced apart and viewing a real object in space, each eye may have a slightly different view of the object, forming an image of the object at a different location on each eye's retina. This may be referred to as binocular disparity and may be utilized by the human visual system to provide the perception of depth. Conventional display systems simulate binocular disparity by presenting two distinct images 190, 200, one for each eye 210, 220, with slightly different views of the same virtual object, corresponding to the view of the virtual object as it would appear by each eye as if the virtual object were a real object at a desired depth. These images provide binocular cues that the user's visual system may interpret to derive the perception of depth.

[0164] Continuing with reference to FIG. 2 , images 190 and 200 are spaced apart from eyes 210 and 220 by a distance 230 on the z-axis. The z-axis is parallel to the optical axis of a viewer fixating on an object at optical infinity directly in front of the viewer. Images 190 and 200 are flat and at a fixed distance from eyes 210 and 220. Based on slightly different views of the virtual object in the images presented to eyes 210 and 220, respectively, the eyes may naturally rotate so that the image of the object falls on a corresponding point on each eye's retina, maintaining single binocular vision. This rotation may cause the gaze of each eye 210 and 220 to converge on a point in space where the virtual object is perceived to reside. As a result, providing three-dimensional images traditionally involves manipulating the convergence and divergence of the user's eyes 210 and 220 and providing binocular cues that the human visual system interprets to provide the perception of depth.

[0165] However, creating a realistic and comfortable perception of depth is challenging. It should be understood that light from an object at different distances from the eye has a wavefront with different amounts of divergence. Figures 3A-3C illustrate the relationship between distance and light ray divergence. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in Figures 3A-3C, light rays become more divergent as the distance to the object decreases. Conversely, as the distance increases, light rays become more collimated. In other words, the light field generated by a point (an object or portion of an object) can be said to have a spherical wavefront curvature that is a function of the distance the point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. While only a single eye 210 is illustrated in Figures 3A-3C and various other figures herein for clarity of illustration, the discussion regarding the eye 210 may apply to both eyes 210 and 220 of the viewer.

[0166] Continuing with reference to Figures 3A-3C, light from an object at which a viewer's eye is fixating may have different wavefront divergences. Due to the different wavefront divergences, the light may be focused differently by the eye's lens, which in turn may require the lens to assume a different shape to form a focused image on the eye's retina. If a focused image is not formed on the retina, the resulting retinal blur acts as an accommodative cue, causing the shape of the eye's lens to change until a focused image is formed on the retina. For example, an accommodative cue may induce relaxation or contraction of the ciliary muscles surrounding the eye's lens, thereby modulating the force applied to the suspensory ligaments that hold the lens, thus changing the shape of the eye's lens and forming a focused image of the fixated object on the eye's retina (e.g., the fovea) until retinal blur of the fixated image is eliminated or minimized. The process by which the eye's lens changes shape can be referred to as accommodation, and the shape of the eye's lens required to form a focused image of a fixated object on the eye's retina (e.g., the fovea) can be referred to as the state of accommodation.

[0167] Referring now to Figure 4A, a representation of the accommodation-vergence response of the human visual system is illustrated. Eye movement to fixate an object causes the eye to receive light from the object, which forms an image on each of the eye's retinas. The presence of retinal blur in the image formed on the retina can provide a cue for accommodation, and the relative location of the image on the retina can provide a cue for vergence. Accommodation cues cause accommodation, resulting in the eye's lens adopting a specific accommodation state in which a focused image of the object is formed on the eye's retina (e.g., the fovea). Conversely, vergence cues cause vergence movements (eye rotations) so that the images formed on each retina of each eye are at corresponding retinal points, maintaining single binocular vision. In these positions, the eyes can be said to adopt a specific vergence state. Continuing with reference to FIG. 4A , accommodation can be understood as the process by which the eyes achieve a particular accommodation state, and convergence can be understood as the process by which the eyes achieve a particular convergence state. As shown in FIG. 4A , the accommodation and convergence states of the eyes can change when the user fixates on a different object. For example, the accommodated state can change when the user fixates on a new object at a different depth on the z-axis.

[0168] Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of convergence-divergence and accommodation. As previously mentioned, vergence-divergence movements of the two eyes relative to one another (e.g., eye rotation such that the pupils move toward or away from one another, converging the eyes' lines of sight and fixating on an object) are closely linked to accommodation of the eye's lenses. Under normal conditions, changing the shape of the eye's lenses and shifting focus from one object to another at a different distance will automatically produce a corresponding change in vergence-divergence to the same distance, a relationship known as the "accommodation-divergence reflex." Similarly, a change in vergence-divergence will induce a corresponding change in lens shape under normal conditions.

[0169] 4B, an example of different accommodation and convergence states of the eyes is illustrated. Paired eye 222a fixates an object at optical infinity, while paired eye 222b fixates an object 221 at less than optical infinity. Notably, the convergence states of each pair of eyes are different: paired eye 222a points straight ahead, while paired eye 222 converges on object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of lenses 210a, 220a.

[0170] Unfortunately, many users of conventional "3-D" display systems may find such systems uncomfortable or may not perceive any depth perception due to a mismatch between accommodation and convergence states in these displays. As previously mentioned, many stereoscopic or "3-D" display systems display a scene by providing a slightly different image to each eye. Such systems are uncomfortable for many viewers because, among other things, they simply provide different presentations of a scene, causing changes in the eyes' convergence states without corresponding changes in the eyes' accommodation states. Rather, images are presented by the display at a fixed distance from the eyes so that the eyes view all image information in a single accommodation state. Such an arrangement counters the "accommodation-vergence-divergence reflex" by causing changes in the convergence states without a corresponding change in the accommodation state. This mismatch is believed to cause viewer discomfort. Display systems that offer better alignment between accommodation and convergence-divergence movements may create a more realistic and comfortable simulation of three-dimensional images.

[0171] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. In some embodiments, the different presentations may provide both vergence cues and matching cues for accommodation, thereby providing physiologically correct accommodation-vergence divergence matching.

[0172] 4B , two depth planes 240 are illustrated, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, vergence-divergence cues may be provided by displaying appropriately different perspective images for each eye 210, 220. Additionally, for a given depth plane 240, the light forming the image provided to each eye 210, 220 may have a wavefront divergence corresponding to the light field generated by a point at the distance of that depth plane 240.

[0173] In the illustrated embodiment, the distance along the z-axis of depth plane 240 containing point 221 is 1 meter. As used herein, distance or depth along the z-axis may be measured with a zero point located at the exit pupil of the user's eye. Thus, depth plane 240 located at a depth of 1 meter corresponds to a distance of 1 meter away from the exit pupil of the user's eye on the optical axis of those eyes. As an approximation, the depth or distance along the z-axis may be measured from a display (e.g., the surface of a waveguide) in front of the user's eye, and a value related to the distance between the device and the exit pupil of the user's eye with the eye pointed toward optical infinity may be added. That value may be referred to as pupil distance and may correspond to the distance between the exit pupil of the user's eye and a display worn by the user in front of the eye. In practice, the value related to pupil distance may generally be a normalized value used for all viewers. For example, pupil distance may be assumed to be 20 mm, and the depth plane at a depth of 1 meter may be at a distance of 980 mm in front of the display.

[0174] 4C and 4D, examples of matched accommodation-vergence-divergence distances and mismatched accommodation-vergence-divergence distances are illustrated, respectively. As illustrated in FIG. 4C, the display system may provide an image of a virtual object to each eye 210, 220. The image may cause the eyes 210, 220 to assume a convergence-divergence state in which the eyes converge on point 15 on the depth plane 240. In addition, the image may be formed by light having a wavefront curvature corresponding to the real object on that depth plane 240. As a result, the eyes 210, 220 assume an accommodation state in which the image is focused on the retinas of those eyes. Thus, the user may perceive the virtual object as being at point 15 on the depth plane 240.

[0175] It should be understood that each of the accommodation and convergence states of the eyes 210, 220 is associated with a particular distance on the z-axis. For example, an object at a particular distance from the eyes 210, 220 will cause those eyes to assume a particular accommodation state based on the distance of the object. The distance associated with a particular accommodation state is referred to as the accommodation distance A. d Similarly, a particular convergence-divergence distance V associated with the eyes in a particular convergence-divergence state or position relative to one another may be referred to as d However, if the accommodation distance and the convergence distance are consistent, the relationship between accommodation and convergence can be said to be physiologically correct. This is considered to be the most comfortable scenario for the viewer.

[0176] However, in a stereoscopic display, accommodation distance and vergence distance may not always match. For example, as illustrated in FIG. 4D , images displayed to eyes 210, 220 may be displayed with a wavefront divergence corresponding to depth plane 240, and eyes 210, 220 may be in a particular accommodation state in which points 15a, 15b on that depth plane are in focus. However, images displayed to eyes 210, 220 may provide convergence cues that cause eyes 210, 220 to converge on point 15 that is not located on depth plane 240. As a result, accommodation distance, in some embodiments, corresponds to the distance from a particular reference point of the user (e.g., the exit pupils of eyes 210, 220) to depth plane 240, while vergence distance corresponds to a greater distance from that reference point to point 15. Therefore, the accommodation distance differs from the convergence distance, and an accommodation-vergence mismatch exists. Such a mismatch may be considered undesirable and may cause discomfort to the user. The mismatch may be due to the distance (e.g., V d -A d ) and can be characterized using diopters (units of reciprocal length 1 / m). For example, a V of 1.75 diopters d and 1.25 diopters of A d or 1.25 diopters V d and 1.75 diopters of A d would provide an accommodation-vergence-divergence mismatch of 0.5 diopters.

[0177] It should be understood that in some embodiments, a reference point other than the exit pupil of the eye 210, 220 may be used to determine the distance for determining accommodation-vergence mismatch, so long as the same reference point is used for accommodation distance and vergence distance. For example, the distance may be measured at a depth plane from the cornea, a depth plane from the retina, a depth plane from the eyepiece (e.g., a waveguide in a display device), etc.

[0178] Without being limited by theory, it is believed that a user may still perceive an accommodation-vergence-divergence mismatch of up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters as physiologically correct, without the mismatch itself causing significant discomfort. In some embodiments, a display system disclosed herein (e.g., display system 250, FIG. 6 ) presents a viewer with an accommodation-vergence-divergence mismatch of about 0.5 diopters or less. In some other embodiments, the accommodation-vergence-divergence mismatch of images provided by the display system is about 0.33 diopters or less. In still other embodiments, the accommodation-vergence-divergence mismatch of images provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0179] FIG. 5 illustrates aspects of an approach for simulating a three-dimensional image by modifying wavefront divergence. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to a user's eye 210. The waveguide 270 may output light 650 with a defined amount of wavefront divergence corresponding to the wavefront divergence of a light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on that depth plane. In addition, the user's other eye will be illustrated as being provided with image information from a similar waveguide.

[0180] In some embodiments, a single waveguide may be configured to output light with a set wavefront divergence corresponding to a single or limited number of depth planes, and / or the waveguide may be configured to output light of a limited range of wavelengths. As a result, in some embodiments, multiple or stacked waveguides may be utilized to provide different wavefront divergences for different depth planes and / or output light of different ranges of wavelengths. As used herein, it should be understood that a depth plane may follow the contour of a flat or curved surface. In some embodiments, for convenience, a depth plane may follow the contour of a flat surface.

[0181] 6 illustrates an example of a waveguide stack for outputting image information to a user. Display system 250 includes a stack of waveguides or stacked waveguide assembly 260 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. It should be understood that display system 250 may be considered a light field display in some embodiments. Additionally, waveguide assembly 260 may also be referred to as an eyepiece.

[0182] In some embodiments, display system 250 may be configured to provide a substantially continuous cue for convergence and multiple discrete cues for accommodation. The cues for convergence may be provided by displaying different images to each of the user's eyes, and the cues for accommodation may be provided by outputting light that forms images with selectable discrete amounts of wavefront divergence. In other words, display system 250 may be configured to output light with variable levels of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a particular depth plane and may be provided by a particular one of waveguides 270, 280, 290, 300, and 310.

[0183] Continuing with reference to FIG. 6 , the waveguide assembly 260 may also include multiple features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or multiple lenses 320, 330, 340, 350 may be configured to transmit image information to the eye with various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. The image injection devices 360, 370, 380, 390, 400 may act as light sources for the waveguides and may be utilized to inject image information into the waveguides 270, 280, 290, 300, 310, each configured to disperse incident light across each individual waveguide for output toward the eye 210, as described herein. Light exits output surfaces 410, 420, 430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 and is injected into corresponding input surfaces 460, 470, 480, 490, 500 of the waveguides 270, 280, 290, 300, 310. In some embodiments, each input surface 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide or may be a portion of a major surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide, outputting a total field of cloned collimated beams directed toward the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of the image launch devices 360, 370, 380, 390, 400 may be associated with and launch light into multiple (e.g., three) waveguides 270, 280, 290, 300, 310.

[0184] In some embodiments, each of the image input devices 360, 370, 380, 390, 400 is a discrete display that generates image information for input into a corresponding waveguide 270, 280, 290, 300, 310. In some other embodiments, the image input devices 360, 370, 380, 390, 400 are the output of a single multiplexed display that may, for example, send image information via one or more optical conduits (such as fiber optic cables) to each of the image input devices 360, 370, 380, 390, 400. It should be understood that the image information provided by the image input devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different primary colors as discussed herein).

[0185] In some embodiments, light injected into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520, which includes a light module 530, which may include a light emitter such as a light emitting diode (LED). Light from the light module 530 may be directed through a beam splitter 550 and modified by a light modulator 540, e.g., a spatial light modulator. The light modulator 540 may be configured to change the perceived intensity of the light injected into the waveguides 270, 280, 290, 300, 310 and encode the light with image information. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays. It should be understood that image injection devices 360, 370, 380, 390, 400 are illustrated diagrammatically, and in some embodiments, these image injection devices may represent different light paths and locations within a common projection system configured to output light into associated ones of waveguides 270, 280, 290, 300, 310. In some embodiments, the waveguides of waveguide assembly 260 may function as ideal lenses, relaying light injected into the waveguides to the user's eye. In this concept, the object may be a spatial light modulator 540, and the image may be an image on a depth plane.

[0186] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, spiral scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately to the viewer's eye 210. In some embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent a single scanning fiber or a bundle of scanning fibers configured to inject light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent multiple scanning fibers or multiple bundles of scanning fibers, each configured to inject light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or fibers and one or more waveguides 270, 280, 290, 300, 310, for example, to redirect light exiting the scanning fiber into one or more waveguides 270, 280, 290, 300, 310.

[0187] Controller 560 controls the operation of one or more of stacked waveguide assemblies 260, including the operation of image input devices 360, 370, 380, 390, 400, light source 530, and light module 540. In some embodiments, controller 560 is part of local data processing module 140. Controller 560 contains programming (e.g., instructions in a non-transitory medium) that coordinates the timing and provision of image information to waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 560 may, in some embodiments, be part of processing module 140 or 150 (FIG. 9D).

[0188] Continuing with reference to FIG. 6 , the waveguides 270, 280, 290, 300, and 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Each of the waveguides 270, 280, 290, 300, and 310 may be planar or have another shape (e.g., curved), with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 270, 280, 290, 300, and 310 may each include outcoupling optical elements 570, 580, 590, 600, and 610 configured to extract light from the waveguide by redirecting light propagating within each individual waveguide out of the waveguide and outputting image information to the eye 210. The extracted light may also be referred to as outcoupling light, and the outcoupling optical elements may also be referred to as light extraction optical elements. The extracted beam of light may be output by the waveguide at a location where light propagating within the waveguide strikes the light extraction optical element. The outcoupling optical element 570, 580, 590, 600, 610 may be, for example, a grating including diffractive optical features as discussed further herein. While shown disposed on the bottom major surface of the waveguides 270, 280, 290, 300, 310 for ease of explanation and clarity of drawing, in some embodiments, the outcoupling optical element 570, 580, 590, 600, 610 may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides 270, 280, 290, 300, 310, as discussed further herein. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 may be formed within a layer of material attached to a transparent substrate and forming the waveguides 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be a monolithic piece of material, and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on and / or within that piece of material.

[0189] Continuing with reference to FIG. 6 , as discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (injected into such waveguide 270) to the eye 210. The collimated light may represent an optical infinity focal plane. The next upper waveguide 280 may be configured to send collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210. Such first lens 350 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets light emerging from the next upper waveguide 280 as emerging from a first focal plane closer inward from optical infinity toward the eye 210. Similarly, the third upper waveguide 290 passes its output light through both the first lens 350 and the second lens 340 before reaching the eye 210. The combined refractive power of the first lens 350 and the second lens 340 may be configured to produce another, increasing amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 290 as originating from a second focal plane that is closer inward from optical infinity towards the person than was the light from the next upper waveguide 280.

[0190] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all of the lenses between it and the eye for a collective focal power representing the focal plane closest to the person. To compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light originating from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 may be placed on top of the stack to compensate for the collective power of the lower lens stacks 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the waveguide outcoupling optical elements and the focusing sides of the lenses may be static (i.e., not dynamic or electro-active). In some alternative embodiments, one or both may be dynamic using electro-active features.

[0191] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same depth plane, or multiple subsets of the waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same multiple depth planes, with one set per depth plane. This may provide the advantage of forming tiled images to provide an extended field of view at those depth planes.

[0192] Continuing with reference to FIG. 6 , the outcoupling optical elements 570, 580, 590, 600, 610 may be configured to redirect light from its respective waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with that waveguide. As a result, waveguides with different associated depth planes may have different configurations of outcoupling optical elements 570, 580, 590, 600, 610, which output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 may be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (eg, cladding layers and / or structures for forming air gaps).

[0193] In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a sufficiently low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 210 at each intersection of the DOE, while the remainder continues traveling through the waveguide via TIR. The light carrying the image information is thus split into several related output beams that exit the waveguide at various locations, resulting in a very uniform pattern of output emission toward the eye 210 for this particular collimated beam bouncing within the waveguide.

[0194] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0195] In some embodiments, a camera assembly 630 (e.g., a digital camera, including a visible light and infrared light camera) may be provided to capture images of the eye 210 and / or tissue surrounding the eye 210, for example, to detect user input and / or monitor the physiological state of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source that projects light (e.g., infrared light) onto the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 630 may be mounted on the frame 80 (FIG. 9D) and may be in electrical communication with processing modules 140 and / or 150, which may process image information from the camera assembly 630. In some embodiments, one camera assembly 630 may be utilized per eye, monitoring each eye separately.

[0196] 7, an example of an output beam output by a waveguide is shown. While one waveguide is illustrated, it should be understood that other waveguides in waveguide assembly 260 (FIG. 6) may function similarly, and that waveguide assembly 260 includes multiple waveguides. Light 640 is launched into waveguide 270 at input surface 460 of waveguide 270 and propagates within waveguide 270 by TIR. At the point where light 640 impinges on DOE 570, a portion of the light exits the waveguide as output beam 650. Output beam 650 is illustrated as being approximately parallel, but may be redirected to propagate to eye 210 at an angle (e.g., forming a diverging output beam), as discussed herein and depending on the depth plane associated with waveguide 270. It should be understood that a nearly collimated exit beam may refer to a waveguide with outcoupling optics that outcouples light to form an image that appears to be set at a depth plane at a large distance (e.g., optical infinity) from the eye 210. Other waveguides or other sets of outcoupling optics may output a more divergent exit beam pattern, which would require the eye 210 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0197] In some embodiments, a full-color image may be formed at each depth plane by overlaying an image in each of the primary colors, for example, three or more primary colors. FIG. 8 illustrates an example of a stacked waveguide assembly in which each depth plane includes an image formed using multiple different primary colors. The illustrated embodiment shows depth planes 240a-240f, but more or fewer depths are also contemplated. Each depth plane may have three or more primary color images associated with it, including a first image in a first color G, a second image in a second color R, and a third image in a third color B. The different depth planes are illustrated by different numbers for diopters (dpt) following the letters G, R, and B. By way of example only, the number following each of these letters indicates the diopter (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the diagram represents an individual primary color image. In some embodiments, the exact locations of the depth planes for different primary colors may be varied to account for differences in the eye's focusing of light of different wavelengths. For example, different primary color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or reduce chromatic aberrations.

[0198] In some embodiments, light for each primary color may be output by a single dedicated waveguide, such that each depth plane may have multiple waveguides associated with it. In such embodiments, each box in the diagram containing the letter G, R, or B may be understood to represent an individual waveguide, and three waveguides may be provided per depth plane, with three primary color images provided per depth plane. While the waveguides associated with each depth plane are shown adjacent to each other in this drawing for ease of illustration, it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other embodiments, multiple primary colors may be output by the same waveguide, such that, for example, only a single waveguide may be provided per depth plane.

[0199] 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may be used in addition to or replace one or more of red, green, or blue.

[0200] It should be understood that references throughout this disclosure to a given color of light are understood to encompass light of one or more wavelengths within the range of wavelengths of light that are perceived by a viewer as that given color. For example, red light may include one or more wavelengths of light within a range of about 620-780 nm, green light may include one or more wavelengths of light within a range of about 492-577 nm, and blue light may include one or more wavelengths of light within a range of about 435-493 nm.

[0201] In some embodiments, light source 530 (FIG. 6) may be configured to emit light at one or more wavelengths outside the range of a viewer's visual perception, e.g., infrared and / or ultraviolet wavelengths. Additionally, the waveguide incoupling, outcoupling, and other light redirecting structures of display 250 may be configured to direct and emit this light from the display toward the user's eye 210, e.g., for imaging and / or user stimulation applications.

[0202] Referring now to FIG. 9A , in some embodiments, light impinging on a waveguide may need to be redirected to incoupling the light into the waveguide. An incoupling optical element may be used to redirect and incoupling the light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an example of a plurality or set 660 of stacked waveguides, each including an incoupling optical element. The waveguides may each be configured to output light of one or more different wavelengths or one or more different wavelength ranges. While stack 660 may correspond to stack 260 ( FIG. 6 ), and the illustrated waveguides of stack 660 may correspond to a portion of multiple waveguides 270, 280, 290, 300, 310, it should be understood that light from one or more of image injection devices 360, 370, 380, 390, 400 is injected into the waveguide from a location requiring the light to be redirected for incoupling.

[0203] The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as the light input area on the waveguide), for example, internal coupling optical element 700 is disposed on a major surface (e.g., the top major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some embodiments, one or more of internal coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of an individual waveguide 670, 680, 690 (particularly, one or more internal coupling optical elements are reflective polarizing optical elements). As shown, the internal coupling optical elements 700, 710, 720 may be disposed on the upper major surfaces of their respective waveguides 670, 680, 690 (or on top of the next lower waveguide), and in particular, the internal coupling optical elements are transmissive polarizing optical elements. In some embodiments, the internal coupling optical elements 700, 710, 720 may be disposed within the bodies of the respective waveguides 670, 680, 690. In some embodiments, as discussed herein, the internal coupling optical elements 700, 710, 720 are wavelength selective so as to selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. While illustrated on one side or corner of the respective waveguides 670, 680, 690, it should be understood that the internal coupling optical elements 700, 710, 720 may be disposed within other areas of the respective waveguides 670, 680, 690 in some embodiments.

[0204] As shown, the in-coupling optical elements 700, 710, 720 may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset to receive light without that light passing through another in-coupling optical element. For example, each in-coupling optical element 700, 710, 720 may be configured to receive light from a different image input device 360, 370, 380, 390, and 400, as shown in FIG. 6 , and may be separated (e.g., laterally spaced) from the other in-coupling optical elements 700, 710, 720 so as to substantially not receive light from others of the in-coupling optical elements 700, 710, 720.

[0205] Each waveguide also includes an associated optically dispersive element, for example, optically dispersive element 730 is disposed on a major surface (e.g., the top major surface) of waveguide 670, optically dispersive element 740 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and optically dispersive element 750 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on the bottom major surfaces of associated waveguides 670, 680, 690, respectively. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of associated waveguides 670, 680, 690, respectively, or optically dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

[0206] Waveguides 670, 680, 690 may be spaced apart and separated, for example, by gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediately adjacent ones of waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more or 0.10 or less than the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote total internal reflection (TIR) ​​of light through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include immediate cladding layers.

[0207] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or the same, and the materials forming layers 760a, 760b are similar or the same. In some embodiments, the materials forming waveguides 670, 680, 690 may vary between one or more waveguides, and / or the materials forming layers 760a, 760b may vary while still maintaining the various refractive index relationships described above.

[0208] 9A, light rays 770, 780, 790 enter the set of waveguides 660. It should be understood that light rays 770, 780, 790 may be injected into the waveguides 670, 680, 690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG. 6).

[0209] In some embodiments, light rays 770, 780, 790 have different properties, e.g., different wavelengths or different wavelength ranges, which may correspond to different colors. Each of the incoupling optical elements 700, 710, 720 deflects incident light such that the light propagates through a respective one of the waveguides 670, 680, 690 by TIR. In some embodiments, each of the incoupling optical elements 700, 710, 720 selectively deflects one or more particular wavelengths of light while transmitting other wavelengths to the underlying waveguide and associated incoupling optical element.

[0210] For example, in-coupling optical element 700 may be configured to deflect light ray 770 having a first wavelength or wavelength range while transmitting light rays 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. Transmitted light ray 780 impinges on and is deflected by in-coupling optical element 710, which is configured to deflect light of the second wavelength or wavelength range. Light ray 790 is deflected by in-coupling optical element 720, which is configured to selectively deflect light of the third wavelength or wavelength range.

[0211] 9A , the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the in-coupling optical element 700, 710, 720 of each waveguide deflects the light into its corresponding waveguide 670, 680, 690 and in-couples the light into its corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the respective waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the respective waveguides 670, 680, 690 by TIR until they impinge on the waveguide's corresponding optical dispersive element 730, 740, 750.

[0212] 9B, a perspective view of the multiple stacked waveguide embodiment of FIG. 9A is illustrated. As described above, in-coupled light rays 770, 780, 790 are deflected by in-coupling optical elements 700, 710, 720, respectively, and then propagate by TIR within waveguides 670, 680, 690, respectively. Light rays 770, 780, 790 then impinge on optically dispersive elements 730, 740, 750, respectively. Optically dispersive elements 730, 740, 750 deflect light rays 770, 780, 790 to propagate toward out-coupling optical elements 800, 810, 820, respectively.

[0213] In some embodiments, the optically dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse light into the out-coupling optical elements 800, 810, 820, and in some embodiments, may also increase the beam or spot size of this light as it propagates into the out-coupling optical elements. In some embodiments, the optically dispersive elements 730, 740, 750 may be omitted, and the in-coupling optical elements 700, 710, 720 may be configured to deflect light directly into the out-coupling optical elements 800, 810, 820. For example, with reference to FIG. 9A , the optically dispersive elements 730, 740, 750 may be replaced with the out-coupling optical elements 800, 810, 820, respectively. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light toward the viewer's eye 210 ( FIG. 7 ). It should be understood that an OPE can be configured to increase the size of the eyebox in at least one axis, and that the EPE may increase the eyebox in an axis that intersects the axis of the OPE, e.g., orthogonal to the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light striking the OPE to an EPE of the same waveguide while allowing the remaining portion of the light to continue propagating down the waveguide. In response to striking the OPE again, another portion of the remaining light is redirected to the EPE, and the remainder of that portion continues to propagate further down the waveguide, and so on. Similarly, in response to striking an EPE, a portion of the impinging light is directed out of the waveguide toward the user, and the remainder of that light continues to propagate through the waveguide until it again strikes an EPE, at which point another portion of the impinging light is directed out of the waveguide, and so on. As a result, a single beam of internally coupled light may be "replicated" each time a portion of that light is redirected by the OPE or EPE, thereby forming a cloned beam field of light, as shown in FIG. 6. In some embodiments, the OPE and / or EPE may be configured to modify the size of the beam of light.

[0214] 9A and 9B, in some embodiments, a waveguide set 660 includes, for each primary color, waveguides 670, 680, 690, in-coupling optical elements 700, 710, 720, optically dispersive elements (e.g., OPEs) 730, 740, 750, and out-coupling optical elements (e.g., EPs) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer between each one. The in-coupling optical elements 700, 710, 720 redirect or deflect incident light into that waveguide (with different in-coupling optical elements receiving different wavelengths of light). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the example shown, light ray 770 (e.g., blue light) is deflected by the first in-coupling optical element 700 in the manner described above, then continues bouncing down the waveguide, interacting with the optically dispersive element (e.g., OPE) 730 and then the out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through the waveguide 670, with light ray 780 impinging on and being deflected by the in-coupling optical element 710. Light ray 780 will then, via TIR, bounce down the waveguide 680, to its optically dispersive element (e.g., OPE) 740 and then the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the optically in-coupling optical element 720 of the waveguide 690. The light in-coupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light dispersive element (e.g., OPE) 750 and then by TIR to the out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light ray 790 to a viewer, who also receives the out-coupled light from the other waveguides 670, 680.

[0215] FIG. 9C illustrates a top-down plan view of an example of the multiple stacked waveguides of FIGS. 9A and 9B. As shown, waveguides 670, 680, 690 may be vertically aligned, along with each waveguide's associated optically dispersive element 730, 740, 750 and associated out-coupling optical elements 800, 810, 820. However, as discussed herein, the in-coupling optical elements 700, 710, 720 are not vertically aligned. Rather, the in-coupling optical elements are preferably non-overlapping (e.g., laterally spaced apart, as seen in the top-down view). As discussed further herein, this non-overlapping spatial arrangement facilitates the injection of light from different sources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, arrays including non-overlapping, spatially separated in-coupling optical elements may be referred to as shifted-pupil systems, and the in-coupling optical elements in these arrays may correspond to sub-pupils.

[0216] 9D illustrates an example of a wearable display system 60 into which the various waveguide and associated systems disclosed herein may be integrated. In some embodiments, the display system 60 is the system 250 of FIG. 6, which diagrammatically illustrates some portions of the system 60 in greater detail. For example, the waveguide assembly 260 of FIG. 6 may be part of the display 70.

[0217] Continuing with reference to FIG. 9D , display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functionality of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and configured to position the display 70 directly in front of the user's 90's eye. The display 70, in some embodiments, may be considered an eyepiece. In some embodiments, a speaker 100 is coupled to frame 80 and configured to be positioned adjacent to the user's 90 ear canal (in some embodiments, another speaker, not shown, may optionally be positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). Display system 60 may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones are configured to allow a user to provide input or commands to system 60 (e.g., voice menu command selections, natural language queries, etc.) and / or enable audio communication with other persons (e.g., other users of similar display systems). The microphone may further be configured as an ambient sensor and collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system 60 may further include one or more outwardly directed environmental sensors 112 configured to detect lights, objects, stimuli, people, animals, places, or other aspects of the world around the user. For example, the environmental sensors 112 may include, for example, one or more cameras that may be positioned facing outward to capture images similar to at least a portion of the user 90's normal field of view. In some embodiments, the display system may also include an ambient sensor 120a, which may be separate from the frame 80 and mounted on the user 90's body (e.g., the user's 90's head, torso, limbs, etc.). The ambient sensor 120a, in some embodiments, may be configured to obtain data characterizing the user 90's physiological state. For example, the sensor 120a may be an electrode.

[0218] 9D , display 70 is operably coupled to local data processing module 140 by a communication link 130, such as wired or wireless connectivity, which may be mounted in a variety of configurations, such as fixedly attached to frame 80, fixedly attached to a helmet or hat worn by the user, embedded within headphones, or otherwise removably attached to user 90 (e.g., in a backpack-style configuration, in a belt-linked configuration). Similarly, sensor 120a may be operably coupled to local data processing module 140 by a communication link 120b, such as wired or wireless connectivity. Local processing and data module 140 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. Optionally, local processing and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include a) data captured from sensors (such as image capture devices (cameras, etc.), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein (e.g., which may be operatively coupled to frame 80 or otherwise attached to user 90)) and / or b) data obtained and / or processed using remote processing module 150 and / or remote data repository 160 (including data related to virtual content), possibly for passage to display 70 after processing or retrieval. Local processing and data module 140 may be operatively coupled to remote processing module 150 and remote data repository 160 by communication links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to local processing and data module 140.In some embodiments, local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted to frame 80 or may be a freestanding structure that communicates with local processing and data module 140 by a wired or wireless communication path.

[0219] 9D , in some embodiments, remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, remote data repository 160 may comprise digital data storage facilities that may be available through the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, remote data repository 160 may include one or more remote servers that provide information, for example, information for generating augmented reality content, to local processing and data module 140 and / or remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote module. Optionally, an external system (e.g., one or more processors, one or more computer systems), including a CPU, GPU, etc., may perform at least a portion of the processing (e.g., generating image information, processing data) and provide information to and receive information from modules 140, 150, 160, e.g., via a wireless or wired connection. (I. Quality Adjustment Based on Depth Information)

[0220] As described herein, a display system (e.g., an augmented reality display system such as display system 60 of FIG. 9D ) according to various embodiments may determine a user's three-dimensional fixation point, for example, by monitoring the user's eyes. The fixation point may indicate the location of a point in space along (1) the x-axis (e.g., the lateral axis), (2) the y-axis (e.g., the vertical axis), and (3) the z-axis (e.g., the depth of the point, e.g., depth from the user). In some embodiments, the display system may utilize cameras, sensors, etc. to monitor the user's eyes (e.g., the pupils, cornea, etc. of each eye) and determine the line of sight of each eye. The line of sight of each eye may generally be understood to be a vector extending from the center of the retina of that eye through the eye's lens. For example, the vector may generally extend from the center of the macula (e.g., the fovea) through the eye's lens. The display system may be configured to determine where vectors associated with the eyes intersect, and this intersection point may be understood to be the eye's fixation point. In other words, the fixation point may be a location in three-dimensional space where the user's eyes are converged. In some embodiments, the display system may filter slight movements of the user's eyes, for example, during fast movements (e.g., saccades, microsaccades), and may update the fixation point in response to determining that the eyes are fixating at a location in three-dimensional space. For example, the display system may be configured to ignore movements of the eyes that fixate at a point for less than a threshold duration.

[0221] The resolution of content, such as a virtual object or content, presented by a display system may be adjusted based on its proximity to a fixation point, as discussed herein. It should be understood that the display system may store or have access to information regarding the location of the virtual object in three-dimensional space. Based on the known location of the virtual object, the proximity of a given virtual object to the fixation point may be determined. For example, the proximity of the virtual object to the fixation point may be determined by determining one or more of: (1) the three-dimensional distance of the virtual object from the user's fixation point; (2) if the display frustum of the display system is divided into resolution adjustment zones, the resolution adjustment zone in which the virtual object is located relative to the resolution adjustment zone in which the fixation point is located; and (3) the angular separation between the virtual object and the user's line of sight. Virtual content closer to the fixation point may be presented with a higher resolution than content farther from the fixation point. In some embodiments, the resolution of the virtual content varies depending on the depth plane in which the virtual content is located relative to the fixation point or the proximity of the depth plane in which the fixation point is located. In some embodiments, the adjustment to the resolution may be made by one or more graphics processing units, e.g., a rendering engine, such as a rendering engine included in one or more of modules 140, 150 (FIG. 9D).

[0222] FIG. 10A illustrates an example of a top-down representation of a user viewing content (e.g., content contained within a display frustum 1004) presented by a display system (e.g., display system 60, FIG. 9D). The representation includes the user's eyes 210, 220 and a determination of a fixation point 1006 for the eyes 210, 220. As shown, the line of sight of each eye is represented as a vector (e.g., vectors 1003A, 1003B), and the display system detects the fixation point 1006 by, for example, determining where the vectors converge in front of the eyes 210, 220. In the illustrated example, the fixation point 1006 coincides with the location of a first virtual object 1008A presented by the display system. Examples of systems and methods for eye tracking may be found in U.S. patent application Ser. No. 14 / 690,401, filed April 18, 2015 (incorporated by reference for all purposes) and the accompanying appendices. For example, eye tracking systems and methods may be utilized for eye tracking and / or for determining fixation points as described at least in Figures 25-27 of the Appendix and at least in part as described herein.

[0223] 10A , a second virtual object 1008B is also presented by the display system within the display frustum 1004. A view of these virtual objects 1008A, 1008B as seen by a viewer is shown in a rendered frame 1010. The rendered frame 1010 may include the first virtual object 1008A rendered at a first resolution, while the second virtual object 1008B, located away from the fixation point 1006, is rendered at a second, lower resolution. Specifically, the second virtual object 1008B may be determined to be at a greater depth than the first virtual object 1008A and to be located toward its side. For example, the display system may determine the depth of the second virtual object 1008B as discussed herein, or optionally, a content provider associated with the virtual content may indicate a virtual object depth that the display system may utilize to render that virtual object. Thus, the fixation point 1006 describes the three-dimensional location in space where the user is looking, as described above, and the second virtual object 1008B can be determined to be located at a depth farther from the user and displaced laterally from the fixation point 1006.

[0224] Without being limited by theory, it is believed that when the user's eyes 210, 220 are viewing the first virtual object 1008A, the image of the first virtual object 1008A may fall within the user's fovea, while the image of the second virtual object 1008B does not. As a result, the resolution of the second virtual object 1008B may be reduced without significantly affecting the perceived image quality of the display system due to the lower sensitivity of the human visual system to the second virtual object 1008B. In addition, the lower resolution advantageously reduces the computational load required to provide the image. As discussed herein, the resolution at which the second virtual object 1008B is rendered may be based on its proximity to the fixation point 1006, and the reduction in resolution (e.g., relative to the resolution of the first virtual object 1008A) may increase with decreasing proximity (or increasing distance) between the fixation point 1006 and the virtual object 1008A. In some embodiments, the rate of decrease in resolution may be based on the rate of decrease in cone density in the human eye or the decrease in visual acuity with distance from the fovea.

[0225] It should be understood that the resolution of various virtual objects presented by the display system may vary dynamically as the fixation point changes location. For example, FIG. 10B illustrates another example of a top-down representation of a user viewing content presented by a display system. As illustrated in FIG. 10B, the user is now focused on a second virtual object 1008B, as compared to FIG. 10A, where the user was focused on a first virtual object 1008A. By monitoring the user's gazes 1003A, 1003B, the display system determines that the eyes 210, 220 are convergent on the second virtual object 1008B and sets that location as the new fixation point 1006.

[0226] In response to detecting this change in location of fixation point 1006, the display system now renders second virtual object 1008B at a higher resolution than first virtual object 1008A, as shown in rendered frame 1010. Preferably, the display system monitors the user's line of sight 1003A, 1003B with a sufficiently high frequency to change the resolution of the virtual objects sufficiently rapidly so that the transition in resolution of first virtual object 1008A and second virtual object 1008B is substantially imperceptible to the user.

[0227] 10C illustrates another example of a top-down representation of a user viewing content through a display system (e.g., display system 60, FIG. 9D). In the example, the user's field of view 1004 is illustrated along with a fixation point 1006. Three virtual objects are illustrated, with a first virtual object 1012A approaching closer to the fixation point 1006 than either a second virtual object 1012B or a third virtual object 1012C. Similarly, the second virtual object 1012B is illustrated approaching closer to the fixation point 1006 than the third virtual object 1012C. Thus, when virtual objects 1012A-1012C are presented to a user, the display system may allocate resources such that rendering the first virtual object 1012A is given a greater resource allocation than the second virtual object 1012B (e.g., object 1012A is rendered at a higher resolution), which in turn receives a greater resource allocation than the third virtual object 1012C, which may optionally not be rendered at all because it is outside the field of view 1004.

[0228] 10C , where the zones are elliptical (e.g., circular) shapes that are interpreted along depth and lateral axes. As shown, a fixation point 1006 is within a central zone 1014A, and a first virtual object 1012A extends between zones 1014B, 1014C and within the user's foveal cone of vision 1004a. The first virtual object 1012A may therefore be presented to the user at a resolution associated with zone 1014B or 1014C, or, optionally, a portion of the object 1012A within zone 1014B may be presented according to the resolution of zone 1014B, and the remaining portion within zone 1014C may be presented according to the resolution of zone 1014C. For example, in an embodiment where zones are assigned a resolution reduced from a maximum (e.g., highest) resolution, the first virtual object 1012A may be presented at the assigned resolution. Optionally, the first virtual object 1012A may be presented either by resolution (e.g., the display system may be programmed to display at the highest resolution associated with any zone across which the first virtual object 1012A extends) or by a measure of central tendency of the resolution (e.g., the measure may be weighted according to the extent to which the object 1012A is located within zones 1014B, 1014C). With continued reference to FIG. 10C , it should be understood that the resolution adjustment zones at different distances from the fixation point 1006 may have different shapes. For example, zone 1014C may have a different shape than zones 1014A-1014C and conform to the contours of the field of view 1004. In some other embodiments, one or more of the zones 1014A-1014C may have a different shape than one or more others of the zones 1014A-1014C.

[0229] 10D is a block diagram of an exemplary display system. An exemplary display system (e.g., display system 60, FIG. 9D) may be an augmented reality display system and / or a mixed reality display system, which may adjust the use of rendering hardware resources according to a user's point of fixation, as described herein. For example, as described above with respect to FIG. 10C, rendering hardware resources 1021 may be adjusted according to a user's point of fixation. A resource arbiter 1020 may be implemented to adjust the use of such resources 1021; for example, the arbiter 1020 may allocate the resources 1021 to a particular application process 1022 associated with presenting virtual objects to a user. The resource arbiter 1020 and / or the rendering hardware resources 1021 may optionally be included within the local processing and data module 140 (e.g., as illustrated in FIG. 9D) and / or the remote processing module 150 of the display system 60. For example, rendering hardware resource 1021 may comprise a graphics processing unit (GPU), which may be included within module 140 and / or module 150, as described above with respect to FIG. 9D.

[0230] As an example of adjusting resources 1021, with reference to FIG. 10C , a first virtual object 1012A associated with a first application process may be allocated a larger share of resources 1021 than a second virtual object 1012B associated with a second application process. A virtual object associated with an application process 1022 may be included in a frame buffer 1024 that is rendered based on the allocated resources 1021 and composited (e.g., by a compositor 1026) into a final frame buffer 1028. The final frame buffer 1028 may then be presented by display hardware 1030, e.g., display 70 illustrated in FIG. 9D , with the rendered virtual object adjusted in resolution.

[0231] As disclosed herein, the resolution of a virtual object may be determined based on the proximity of the virtual object to a fixation point. In some embodiments, the resolution may be modified as a function of the distance between the virtual object and the fixation point. In some embodiments, the modification may occur in discrete steps. That is, a similar modification may be applied to all virtual objects located within a particular volume or zone. FIG. 11A1 illustrates an example of a top-down representation of adjustments in resolution within different resolution adjustment zones based on three-dimensional fixation point tracking. The display system may divide the display frustum into multiple volumes or resolution adjustment zones and modify the resolution in discrete steps corresponding to these zones. Thus, in some embodiments, to determine adjustments in the resolution of virtual content, the display system may utilize information describing spatial volumes (hereinafter referred to as resolution adjustment zones) and the assignment of resolution adjustments to each spatial volume. As shown, the field of view provided by the display system (e.g., the display frustum of the display) is separated into multiple different zones, each encompassing a range of depths from the user (e.g., depth ranges 1102A-1102E). In some embodiments, each depth range 1102A-1102E has a single associated depth plane that can be presented by the display system. Continuing with reference to FIG. 11A1 , five zones encompass each identified depth range from the user, contiguous along the lateral direction. In the exemplary top-down view shown, the field of view is divided into a grid 1100 of 25 zones. Each zone represents a volume of real-world space in which virtual content can be placed for the user.

[0232] It should be understood that the zones may also extend vertically (e.g., along the y-axis, not shown), such that the illustrated grid 1100 can be understood to represent one cross-section along the vertical direction. In some embodiments, multiple zones are also provided in the vertical direction. For example, there may be five vertical zones per depth range, i.e., a total of 125 resolution adjustment zones. An example of such zones extending in three dimensions is illustrated in FIG. 11B and described below.

[0233] Continuing with reference to FIG. 11A1 , a user's eyes 210, 220 fixate on a particular fixation point 1006 within a grid 1100. The display system may determine the location of the fixation point 1006 and the zone in which the fixation point 1006 is located. The display system may adjust the resolution of the content based on the proximity of the virtual content to the fixation point 1006, which may include determining the proximity of the virtual content to the zone in which the fixation point 1006 is located. As an example, for content contained within the zone in which the fixation point 1006 is located, the resolution may be set to a particular number of polygons, in an example embodiment, 10,000 polygons. Based on the distance from the fixation point 1006, content contained within the remaining zones may be adjusted accordingly. For example, content contained within zones adjacent to the zone containing the fixation point 1006 may be rendered at a lower resolution (e.g., 1,000 polygons). 11A1 illustrates, as an example, adjusting the number of polygons, as described herein, adjusting the resolution may encompass making other modifications to the resolution of the presented content. For example, adjustments in resolution may include one or more of adjusting the number of polygons, adjusting primitives utilized to generate the virtual object (e.g., adjusting the shape of the primitives, e.g., adjusting the primitives from a triangular mesh to a quadrilateral mesh, etc.), adjusting operations performed on the virtual object (e.g., shader operations), adjusting texture information, color resolution, or depth, adjusting the number of rendering cycles or frame rate, and adjusting quality at one or more points in the graphics pipeline of a graphics processing unit (GPU).

[0234] Additionally, while the example of FIG. 11A1 provides a specific example of differences in the number of polygons within different resolution adjustment zones, other absolute numbers of polygons and other rates of change of resolution with distance from fixation point 1006 are also contemplated. For example, the drop in resolution from fixation point 1006 may be based on a drop rate that is symmetric about depth and lateral distance from fixation point 1006, although other drop relationships may also be utilized. For example, lateral distance from fixation point 1006 may be associated with a greater drop in resolution relative to depth distance from fixation point 1006. Furthermore, the size of each zone included within the grid (e.g., the size of the zone's spatial volume) may optionally vary (e.g., the zones may vary radially from the foveal axis). In some embodiments, the drop may be continuous from fixation point 1006, such that a discrete zone with an assigned resolution or a resolution relationship with the zone containing fixation point 1006 is not utilized. For example, the drop-off from the fixation point 1006 to a particular zone 1108 (e.g., a zone whose contents are rendered at a resolution of 100 polygons) may be modified to be a continuous drop-off from the fixation point 1006 to the edge of the grid (e.g., the edge of the particular zone 1108). As described in further detail below with reference to Figures 54-59, in some embodiments, such a drop-off in resolution may be associated with a "roll-off" attribute of the resolution distribution. It should be understood that each of the above considerations also applies to vertically extending zones.

[0235] In some embodiments, the number and size of zones included within the grid may be based on a confidence level associated with the determination of the user's fixation point 1006. For example, the confidence level may be based on the amount of time the user's eyes fixate on the fixation point 1006, with shorter amounts of time being associated with lower confidence levels. For example, the display system may monitor the user's eyes at a particular sampling rate (e.g., 30 Hz, 60 Hz, 120 Hz, 1 kHz) and increase its confidence level in the fixation point 1006 because successive samples indicate that the user is generally maintaining the fixation point 1006. Optionally, a particular threshold for fixation may be utilized; for example, fixations at the same or similar fixation point for a particular duration (e.g., 100-300 milliseconds) may be associated with high confidence levels, while fixations of less than a particular duration may be associated with lower confidence levels. Similarly, intraocular variations, such as mydriasis, that may affect the determination of the user's fixation point may cause the display system to reduce its confidence level. It should be understood that the display system may monitor the eyes using sensors such as camera imaging devices (e.g., camera assembly 630, FIG. 6 ). Optionally, the display system may utilize a combination of sensors to determine the user's eye gaze (e.g., different eye-gaze determination processes may be utilized, such as an infrared sensor utilized to detect infrared reflection from the eye and identify the pupil, a visible light imaging device utilized to detect the eye's iris, etc.). The display system may increase the confidence level when multiple eye-gaze determination processes agree and decrease the confidence level when they disagree. Similarly, for display systems that perform only one eye-gaze determination process, each eye-gaze determination process may be associated with a particular confidence level (e.g., one determination process may be considered more accurate than the other), and the size of the resolution accommodation zone may be selected, at least in part, based on the process being implemented.

[0236] In some embodiments, the display system may increase or decrease the number of zones with each update of the fixation point 1006. For example, as the confidence associated with the fixation point 1006 increases, more zones may be utilized, and as the confidence decreases, fewer zones may be utilized. FIG. 11A2 illustrates an example of a top-down representation of the resolution adjustment zones at different times as the size and number of zones change. At time t=1, the user's field of view may be divided into an initial set of zones, as seen in the top-down view. At time t=2, the confidence in the location of the fixation point 1006 increases, and the display system may also decrease the size of the zone occupied by the fixation point 1006 and rendered at high resolution. Optionally, the size of the other zones may also decrease, as shown. At time t=3, the confidence in the location of the fixation point 1006 decreases, and the display system may also increase the size of the zone occupied by the fixation point 1006 and rendered at high resolution. Optionally, the size of the other zones may also increase, as shown. It should be understood that multiple zones may also extend on the y-axis, and that a similar increase or decrease in the size and number of zones may also be established on that axis. For example, the size of zones extending vertically on the y-axis may decrease with increasing confidence, while the size may increase with decreasing confidence. Optionally, the display system may determine the confidence of the fixation point 1006 for each frame presented to the user by the display system, where t=1, t=2, and t=3 may represent different frames. Because allocating more zones may require an increase in computational power (e.g., the display system may need to adjust the resolution of more content, identify zones in which content is contained, etc.), the display system may balance the increase in required computational power resulting from an increase in the number of zones with the savings in computational power resulting from a potential decrease in content resolution.

[0237] 11A1, the grid may be dynamically changing in the sense that fixation point 1006 may be set to be located at the center (e.g., center of gravity) of the grid. Thus, the display system may avoid edge cases where fixation point 1006 is determined to be located on a vertex of the grid. For example, as a user's eyes rotate and then fixate at different three-dimensional locations in space, the grid may similarly move with the user's gaze.

[0238] 11B-11E illustrate examples of various resolution adjustment zone configurations. Additional shapes and configurations of resolution adjustment zones, not shown, may be utilized, and the examples should not be considered comprehensive. Additionally, in some figures, the user's eyes 210, 220 may be shown spaced apart from the various resolution adjustment zones for ease and clarity of illustration. It should be understood that with all these figures, the eyes 210, 220 may be positioned at or within the boundaries of the zones (see, e.g., FIG. 11A1).

[0239] FIG. 11B illustrates an example of a three-dimensional representation of some of the resolution adjustment zones of FIG. 11A1. FIG. 11A1 can be understood to illustrate a cross-sectional view taken along plane 11A1-11A1 of the three-dimensional representation of FIG. 11B, and it should be understood that FIG. 11B omits some of the resolution adjustment zones of FIG. 11A1 for clarity of illustration. Continuing with FIG. 11A1, the field of view provided by the display system is separated into 27 zones. That is, the field of view is separated into three depth ranges 1102B-1102D, and in each depth range, a 3×3 grid of zones is included that extend laterally and vertically through the depth range.

[0240] The determined fixation point 1006 is illustrated as being within a zone located in the center of the field of view. Virtual objects located within zones outside the zone containing the fixation point 1006 may have their resolution reduced according to their distance from the fixation point 1006 zone, as discussed herein. Because the zones extend laterally and vertically, the reduction in resolution can occur based on the distance on the lateral, vertical, and depth axes (x-, y-, and z-axes, respectively) from the fixation point's resolution adjustment zone. For example, in some embodiments, a virtual object located in zone 1108 can have its resolution reduced according to lateral distance, as shown in FIG. 11A1 (e.g., zone 1108 may include the same vertical portion of the user's field of view and be on the same depth plane as the zone containing the fixation point 1006).

[0241] As above, and as with the zones described in Figures 11C-11E below, the user's fixation point can optionally be maintained at the center (e.g., center of gravity) of the zone, or the zone can be fixed relative to the user's field of view and the user's fixation point can be located within any of the zones.

[0242] FIG. 11C illustrates another example of a configuration for resolution adjustment zones. In the example, the field of view provided by the display system is illustrated as being separated into elliptical zones, each encompassing a specific three-dimensional volume of space. Similar to FIG. 11A1 , each zone (e.g., zones 1112A-112D) extends along the lateral and depth dimensions. In some embodiments, each zone also extends to encompass at least a portion of the user's vertical field of view. Fixation point 1006 is illustrated as being at the center of the zone (e.g., within zone 1112A). Virtual objects located within zones outside zone 1112A may have their resolution reduced according to their distance from zone 1112A, e.g., according to techniques described herein. For example, each zone outside zone 1112A can be assigned a specific resolution, or a drop rate can be utilized to determine the reduction in resolution. Zone 1112D is illustrated as the zone farthest from zone 1110A, and the reduction in resolution may be greatest within zone 1112D.

[0243] FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zones of FIG. 11C, with FIG. 11C showing a cross-sectional view taken along plane 11C-11C. In this example, the field of view provided by the display system is illustrated as being separated into ellipsoidal zones, each encompassing a three-dimensional volume of space. The user's fixation point 1006 is illustrated at the center of gravity of the user's field of view and is located within zone 1112A. Optionally, FIG. 11D may represent each ellipse of FIG. 11C transformed into an ellipsoid. In some embodiments, the size of zone 1112A of FIG. 11C along the depth and lateral directions can define the size of the major axes of zone 1112A of FIG. 11D along the X and Z axes. The various zones may form concentric spheres or ellipsoids.

[0244] FIG. 11E illustrates another example of a three-dimensional representation of the resolution adjustment zones of FIG. 11C, where FIG. 11C shows a cross-sectional view taken along plane 11C-11C. The field of view provided by the display system is illustrated as being separated into similar concentric zones of stacked levels. For example, FIG. 11E may show how the ellipse of FIG. 11C is extended along the vertical direction to create a cylinder. The cylinders can then be separated vertically so that each cylinder encompasses a portion of the user's vertical field of view. Thus, FIG. 11E illustrates nine zones of the cylinder. Each zone additionally excludes any interior zones (e.g., ellipsoid 1112B would encompass a volume of space that excludes the volume of space encompassed by ellipsoid 1112A). In an example, fixation point 1006 is illustrated as being within central zone 1110A, and virtual objects located outside central zone 1110A can have their resolution reduced according to the techniques described herein.

[0245] 12A illustrates a flowchart of an example process 1200 for adjusting resolution of content according to proximity to a three-dimensional fixation point. For convenience, process 1200 may be described as being performed by a display system (e.g., wearable display system 60, which may include processing hardware and software and, optionally, provide information to one or more computers or other external processing systems, e.g., offload processing to and receive information from the external systems).

[0246] In block 1202, the display system determines the user's three-dimensional fixation point. As described above, the display system may include sensors to monitor information associated with the user's eyes (e.g., eye orientation). A non-exhaustive list of sensors includes infrared sensors, ultraviolet sensors, and visible wavelength light sensors. The sensors may optionally output infrared, ultraviolet, and / or visible light onto the user's eyes and determine reflections of the output light from the user's eyes. As an example, infrared light may be output by an infrared light emitter and an infrared light sensor. It should be understood that the sensors, which may include light emitters, may correspond to imaging device 630 of FIG. 6.

[0247] The display system may utilize sensors to determine a line of sight associated with each eye (e.g., a vector extending from the user's eye, such as extending from the fovea through the eye's lens) and the intersection of each eye's line of sight. For example, the display system may output infrared light onto the user's eye, and a reflection from the eye (e.g., a corneal reflection) may be monitored. A vector between the eye's pupil center (e.g., the display system may determine the pupil's centroid, for example, through infrared imaging) and the reflection from the eye may be used to determine the eye's line of sight. The intersection of the line of sight may be determined and assigned as a three-dimensional fixation point. The fixation point may therefore indicate a location where content will be rendered at full or full resolution. For example, based on the determined line of sight, the display system may triangulate the three-dimensional location in space where the user is fixating. Optionally, the display system may utilize orientation information associated with the display system (e.g., information describing the orientation of the display system in three-dimensional space) when determining the fixation point.

[0248] In block 1204, the display system obtains location information associated with content being or to be presented to the user by the display system. Prior to rendering the content for presentation to the user (e.g., via a waveguide output, as described above), the display system may obtain location information associated with the content to be presented to the user. For example, as described above, virtual content may be presented to the user such that the content appears to be located in the real world (e.g., the content may be located at different depths within the user's field of view). It should be understood that the display system may include or have access to a three-dimensional map of the surrounding environment that may indicate the location of any virtual content within the surrounding environment. With reference to this map, the display system may access and provide information defining the three-dimensional location of the virtual content within the user's field of view (e.g., location within the display frustum, as illustrated in FIGS. 10A-10B ).

[0249] In block 1206, the display system adjusts the resolution of the virtual content to be displayed to the user. The display system adjusts the resolution of the content based on its proximity to the three-dimensional fixation point. For example, a rendering engine, such as a rendering engine implemented by a processing device (e.g., a central processing unit, a graphics processing unit), that renders the content for presentation to the user may adjust resources invested in rendering the content (e.g., the rendering engine may adjust the resolution of the content).

[0250] The display system may determine the distance in three-dimensional space between the content to be presented to the user and the user's point of fixation and may reduce the resolution of the content based on the determined distance. The reduction may be determined according to a rate of descent, e.g., a continuous function correlating distance and content resolution, and the display system may obtain the resolution and render the content based on the continuous function. Optionally, the display system may determine the distance from the center of gravity of the content to the point of fixation and may render the content at a certain resolution based on the distance. Optionally, the display system may render portions of the same content at different resolutions according to the distance of the various portions to the point of fixation (e.g., the display system may separate the content into portions and render more distant portions at a reduced resolution compared to closer portions).

[0251] In some embodiments, the display system may access information that can be used to separate a user's field of view (e.g., corresponding to a display frustum) into zones, with each zone representing a volume of space in which content may be contained. The accessed information, for example, the grid illustrated in FIG. 11A1, may indicate a particular resolution to utilize when rendering content that will be contained within each zone, with a three-dimensional fixation point set at the center of the grid. In addition, the grid may indicate a rate of descent in resolution to utilize when rendering the content. For content contained within multiple zones (e.g., content located within the three-dimensional space required by two zones), the display system may optionally adjust the resolution of the content to correspond to a single zone, or may optionally adjust the resolution of the content according to the corresponding zone in which that portion is located.

[0252] When setting the content resolution, the display system renders content located at the fixation point (e.g., within the same zone as the fixation point) at full or maximum resolution. The maximum resolution may be based on the maximum value that the display system hardware and / or software is capable of rendering while ensuring that the content is presented to the user at a refresh rate above a threshold (e.g., 60 Hz, 120 Hz) and, optionally, ensuring that the content is updated at a rate that exceeds the vergence-divergence rate (e.g., above 60 ms) and the accommodation time (e.g., 20 ms to 100 ms) to reduce the perceptibility of changes in resolution. The display system may dynamically modify the maximum resolution based on the display system's available resources, for example, prior to the display system rendering each frame. For example, as more content is to be presented to the user, the maximum resolution of the content may be reduced, ensuring that the display system can present frames of rendered content above a desired threshold rate to reduce the perceptibility of changes in resolution. The display system may optionally monitor the frames per second at which content is presented and adjust the maximum resolution and / or the rate of resolution reduction based on the distance from the fixation point to ensure that the presented frames per second do not fall below a threshold rate. As an example, the display system may render content, such as a first virtual object located in the zone of the fixation point, at full resolution. Instead of reducing the maximum resolution of the first virtual object and ensuring that the frames per second remain above a certain threshold, the display system may dynamically increase the rate of resolution reduction based on distance. In this manner, the display system may adjust the resolution assigned to each zone outside the zone of the fixation point.Optionally, the display system may set a minimum resolution that can be used within each zone outside the zone of the fixation point, and may adjust the maximum resolution if the minimum resolution would be exceeded (e.g., the display system may reduce the maximum resolution if it is necessary to reduce the resolution of content below the minimum and maintain a threshold ratio). Similarly, the display system may reduce the maximum resolution without reducing the resolution of content within zones outside the zone of the fixation point. Optionally, a user of the display system may indicate whether they prefer content located close to the fixation point to be given preference over other content.

[0253] In some embodiments, as described in more detail below with respect to FIGS. 13-14 , the display system may optionally adjust the resolution of the content using the angular proximity of the content to the user's line of sight. For example, if certain content is located outside the zone in which the fixation point is located but within a threshold proximity of the user's line of sight, such that the certain content would be in the fovea of ​​the user's eye, the display system may cause the certain content to be rendered at a higher resolution (e.g., full resolution or a resolution greater than that shown in the grid illustrated in FIG. 11A1 ). Optionally, the display system may reduce the resolution of certain content and apply a blurring process (e.g., Gaussian blurring) to the certain content. In this way, certain content may be rendered at a lower resolution while being blurred, indicating that the certain content is, for example, farther from the user than the fixation point. Additionally, the blurring may reduce the perceptibility of the lower resolution (e.g., the blurring may reduce the perceptibility of the increased pixel size resulting from the lower resolution).

[0254] Exemplary operations associated with presenting virtual content are illustrated in FIGS. 12B-12C (e.g., a rendering pipeline). In the example of FIG. 12B, a three-dimensional scene is presented to a user without adjustments to resolution made as described herein. In FIG. 12C, adjustments to resolution are made according to fixation point information as described herein. For example, one or more of the following adjustments can be made: reducing vertex motion complexity, reducing level of detail, reducing geometry generation, reducing pixel motion complexity / aggregation of multiple pixels, etc. The adjustments can be advantageously made at different steps in the pipeline for presenting virtual content, as illustrated, and can be optimized according to the particular software and / or hardware utilized to present the virtual content. It should be understood that the fidelity zones described in FIG. 12C are resolution adjustment zones.

[0255] 12A , the display system presents the adjusted content to the user in block 1208. As described above, the display system adjusts the resolution of the content based on its proximity to the three-dimensional fixation point. The display system then presents the rendered content to the user at the associated location. In some embodiments, the display system may perform process 1200 for each frame of content to be rendered, or may adjust the resolution of the content as the user adjusts their fixation point.

[0256] As mentioned above, in some embodiments, virtual objects may be presented at different depths while still being within a user's line of sight. Figure 13 illustrates an example of a representation of a user viewing multiple virtual objects aligned with the user's line of sight. The exemplary representation includes the user's field of view (e.g., the display frustum 1004 of a display system) along with the lines of sight 1003A, 1003B of the user's eyes 210, 220, which are fixated on a fixation point on a first virtual object 1008A.

[0257] As shown, the second virtual object 1008B is within angular proximity of the user's line of sight (e.g., one or both of line of sight vectors 1003A, 1003B) such that the second virtual object 1008B would fall within the user's fovea (e.g., would fall within at least one fovea of ​​either of the eyes). For example, in response to rendering frame 1110, the second virtual object 1008B is located behind (e.g., at a deeper perceived depth from) the first virtual object 1008A. It should be understood that the fovea is the portion of the retina with the best visual acuity. If the resolution of the second virtual object 1008B is reduced (e.g., reduced as described above with respect to at least FIG. 11A1 ) because the second virtual object 1008B would fall within the user's fovea, the user may perceive a reduction in resolution. To avoid a perceptible reduction in resolution, the display system may (1) render the second virtual object 1008B at the same resolution as the first virtual object 1008A or within a threshold resolution of the first virtual object 1008A, and / or (2) render the second virtual object 1008B at a reduced resolution (e.g., as shown in FIG. 11A1) and apply blurring to the second virtual object prior to presentation to the user. Without being limited by theory, the blurring may mask the reduction in resolution while providing depth cues.

[0258] 14 is a flowchart of an example process 1400 for adjusting virtual content based on angular distance from a user's line of sight. For convenience, process 1400 will be described as being performed by a display system (e.g., wearable display system 60, which may include processing hardware and software and may optionally provide information to one or more computers or other processing units external to the system, e.g., offload processing to and receive information from the external system). In example process 1400, the display system is a variable-focus display system in which each frame is presented on the same depth plane and, optionally, collapses all content to be presented into a single frame buffer, i.e., the variable-focus display system presents virtual content on one depth plane at a time.

[0259] The display system determines the user's three-dimensional fixation point (block 1402) and obtains location information associated with the content to be presented (block 1404). Blocks 1402 and 1404 may correspond to blocks 1202 and 1204, respectively, of FIG. 12A. As described above with reference to FIG. 12A, the display system monitors the user's eye movement (e.g., eye orientation) and determines the user's fixation point. The display system may obtain the location information of the content to be presented (e.g., in the next frame) and may subsequently adjust the resolution of the content.

[0260] Continuing with reference to FIG. 14, the display system determines content that will have reduced resolution and that is located within a threshold angular distance from the user's line of sight (block 1406). The display system identifies content that will have reduced resolution due to the content's proximity from the fixation point (e.g., the content is located at a greater depth than the fixation point), but that would fall within the user's fovea (e.g., within a threshold angle from the user's line of sight). Because the content will fall within the user's fovea, the user may be able to perceive a reduction in resolution, such as with three-dimensional fixation-centered rendering described herein. It should be understood that content block 1406 may include steps implementing the blocks illustrated in FIG. 12C, particularly the blocks identified in the section "GPU."

[0261] As a result, in block 1408, the display system may optionally cause the determined content to be rendered at a higher resolution. The display system may adjust the resolution of the determined content to be full resolution (e.g., the same resolution as the fixation point or content located within the same zone or volume of space as the fixation point) or above the reduced resolution that would otherwise be assigned to the content (e.g., as described in block 1406).

[0262] In block 1410, the display system may optionally reduce the resolution of the content or blur the content prior to presentation to the user. As described above, a variable focus display system may present content to a user using a single display buffer. Because the variable focus display system presents all content at the same depth plane, the variable focus display system may use the same display buffer to output content, for example, from a rendering engine.

[0263] Optionally, the display system may utilize initial depth buffers, each assigned one or more depth planes, and may combine the initial depth buffers to obtain a display buffer. Referring to the illustration of FIG. 13, a first depth buffer may include a first virtual object 1306, while a second depth buffer may include a second virtual object 1308. The display system may then apply a blurring process to the second depth buffer or to particular content contained within the second depth buffer (e.g., the display system may apply a blurring process to the second virtual content 1308 but not to other content on the same depth plane but located at a greater angular distance from the user's line of sight). After performing the blurring process, the display system may combine the first and second depth buffers (e.g., the display system may add occlusion, e.g., remove portions of the second virtual object 1308 that are invisible due to occlusion by the first virtual object 1306) to obtain a display buffer.

[0264] An exemplary blurring process may include the display system performing a convolution of the content with a kernel associated with the blur (e.g., a Gaussian kernel, a circular kernel such as to recreate a disruptive effect, a box blur, etc.). In this way, the reduction in resolution may be masked while the processing savings resulting from reducing the resolution may be preserved. Optionally, the strength associated with the blurring process (e.g., the degree to which the content is blurred) may be based on the difference in depth between the user's point of fixation and the content and / or the angular proximity of the content to the user's gaze. For example, the degree of blurring may increase with increasing proximity to the user's gaze.

[0265] In some embodiments, the display system may utilize the features of blocks 1408 or 1410 according to the display system's hardware and / or software. For example, certain hardware (e.g., a graphics processing unit) may be able to implement the blurring process in hardware without reaching a threshold for the hardware's performance. For this certain hardware, the display system may be configured to reduce the resolution of the content and then blur the content. However, other hardware may be slow to implement the blurring process, and rendering the content at a higher resolution may result in better performance. For this other hardware, the display system may be configured to render the content at a higher resolution. Furthermore, the decision of whether to render the content at a higher resolution or at a lower resolution with blurring may depend on the type of content to be displayed. For example, the display system may be configured to render text at a higher resolution while rendering shapes at a lower resolution and blurring them.

[0266] 14, in block 1412, the display system presents the content to the user. The display system may present the adjusted content to the user from the same display buffer, for example, as described above. II. Varying Resolution Based on Ambient Lighting Level

[0267] In addition to, or as an alternative to, a reduction in resolution along the z-axis, various other schemes for presenting virtual content with a reduction in resolution may be implemented in some embodiments. Advantageously, as described herein, some aspects of the virtual content may be presented at a relatively high resolution and certain other aspects may be presented at a relatively low resolution, which may reduce the use of computational and energy resources by the display system, while preferably having little effect on the perceived image quality of the virtual content.

[0268] 15, an example of a representation of the retina of a user's eye is illustrated. The illustrated view shows the retina 1500 as seen when viewing head-up along the visual axis of the retina. The retina 1500 includes a fovea centralis 1510 surrounded by a peripheral area 1530. Within the fovea centralis 1510 is a foveola 1520, which intersects the visual axis.

[0269] It should be understood that the retina contains two types of photoreceptors: rods and cones. In addition, the distribution of these photoreceptors across the retina varies, providing different rod and cone densities across the retina.

[0270] Referring now to FIG. 16 , an example of the resolution of the retina 1500 of FIG. 15 and the rod and cone densities across it is diagrammatically illustrated. The x-axis indicates eccentricity relative to the point where the visual axis intersects the retina. The right direction on the page is toward the nose, and the left direction on the page is toward the temple. As illustrated, the resolution of the human eye roughly correlates with the density of photoreceptors (rods and cones) in the retina. As a result, in some embodiments, the reduction or taper in resolution (e.g., spatial resolution) of virtual content on the x- and y-axes (e.g., at a given depth plane) may substantially track the reduction in cone density, rod density, or collection of rod and cone densities across the retina. For example, the trend of resolution reduction with distance from the fixation point across the user's field of view may be within ±50%, ±30%, ±20%, or ±10% of the trend in changes in photoreceptor density (e.g., cone density, rod density, or combined rod and cone densities) across a corresponding portion of the retina. In some embodiments, the reduction in resolution with distance from the fixation point is gradual and substantially tracks the density change. In some other embodiments, the reduction in resolution may occur in steps (e.g., one step, two steps, etc.). For example, there may be two steps: a highest resolution region of the field of view correlated with the fovea, a medium resolution region correlated with the fovea, and a lower resolution region correlated with the peripheral area.

[0271] 16 , it should be understood that different photoreceptors have different levels of activity under different lighting conditions, e.g., at different ambient lighting levels. As a result, it is possible that a reduction in resolution that follows photoreceptor density may not be significantly perceptible to a user at some lighting levels, but may be perceptible at other lighting levels. As a result, in some embodiments, a reduction in resolution of virtual content along the x-, y-, or z-axis may be set with reference to external lighting conditions.

[0272] For example, the visual behavior of the eye can be divided into three modes based on light conditions. The three modes are photopic, mesopic, and scotopic. Photopic vision typically occurs in bright conditions, e.g., about 10-10 8 cd / m 2 Including, approximately 3 cd / m 2 Ambient light conditions or lighting levels occur at or above 100°C. In photopic vision, the cones are primarily active. In scotopic vision, the rods are primarily active. In mesopic vision, both rods and cones may be active. As used herein, ambient light conditions or lighting levels refer to the amount of light to which a user's eyes and their retina are exposed.

[0273] Mesopic vision is typically seen in lower light conditions, e.g., around 10 -3 ~10 0.5 cd / m 2 This occurs under illumination levels of 1000-120 ...

[0274] Scotopic vision typically occurs under lighting conditions where the illumination level is below that for photopic vision. For example, scotopic vision occurs at approximately 10 -3 ~10 -6 cd / m 2 Including about 10 -2 cd / m 2 less than or about 10 -3 cd / m 2 This may occur at the following illumination levels: Rods are primarily active in scotopic vision. It should be understood that the illumination levels described herein for photopic, mesopic, and scotopic vision are examples. In some embodiments, the illumination levels associated with each type of vision may be arbitrarily assigned based on user preference and / or customization for a group to which the user belongs (e.g., based on gender, age, ethnicity, presence of visual abnormalities, etc.).

[0275] In some embodiments, the type of vision active in the user (photopic, mesopic, or scotopic) may be determined based on measurements of ambient lighting levels. For example, the display system may be configured to measure ambient lighting levels using a light sensor, such as an outward-facing camera 112 (FIG. 9D). In some embodiments, the display system may communicate with another sensor or device, which provides information regarding ambient lighting levels.

[0276] It should be understood that the head-mounted display system may block or attenuate some of the ambient light, such that the outward-facing camera may not provide a brightness level that accurately reflects the amount of light impinging on the eye. In addition, the display system also projects light onto the eye, acting as a light source that can modify the illumination level to which the eye is exposed when providing virtual content. In some other embodiments, the inward-facing camera may be utilized to determine the brightness level. For example, the brightness level may be roughly correlated with the size of the pupil. FIG. 17 graphically illustrates an example of the relationship between pupil size and the amount of light incident on a user's eye. The x-axis indicates values ​​related to brightness, and the y-axis indicates values ​​related to pupil area. As a result, the display system may be configured to determine the user's pupil area and then extrapolate brightness based on this pupil area. For example, the display system may be configured to capture an image of the user's eye 210 using the inward-facing camera 500 (FIG. 6), then analyze the image to determine the pupil area or other metric indicative of the pupil area (e.g., pupil diameter or width). For example, the area occupied by the pupil of eye 210 in an image captured by the camera may be determined and then corrected for any scaling factor caused by the camera's optics. Advantageously, using the pupil area to determine the brightness level may effectively take into account both the reduction in ambient brightness level caused by the display blocking some ambient light, and also the contribution to brightness level due to the light output of the display itself.

[0277] 17, the display system may be configured to determine whether the user's eye is in photopic, mesopic, or scotopic mode based on the determined pupil area. For example, the display system may have resident in memory a table or other stored information that defines the expected vision mode for a particular pupil area. As an example, consistent with the graph shown in FIG. 17, the display system may determine whether the user's eye is in photopic, mesopic, or scotopic mode. 2 The pupil area below is considered to indicate photopic vision, and 3mm 2 Up to about 38mm 2 Assuming that the pupil area is 38mm, which indicates mesopic vision, 2 A pupil area exceeding 0.05 mm may be categorized as indicative of scotopic vision. It should be understood that these luminance values ​​and associated visual modes are examples, and other values ​​may be substituted. For example, different values ​​may be applied to different users in response to input from the users, or different values ​​may be applied based on particular categories that users may fall into (e.g., gender, age, ethnicity, presence of visual abnormalities, etc.). In addition, it should be understood that the display system does not necessarily identify a specific visual mode. Rather, the display system may simply be configured to associate a particular measured pupil area with a particular resolution level or accommodation.

[0278] In some embodiments, input from both inward-facing camera 510 (FIG. 6) and outward-facing camera 112 (FIG. 9D) may be utilized to determine the brightness level. For example, the display system may be configured to take an average (including a weighted average) of the brightness levels determined using cameras 510 and 112. As noted above, the brightness level determined using camera 510 may be extrapolated from the size of the pupil area of ​​the user's eye based on imaging the user's eye using camera 510.

[0279] It should be understood that rods and cones have different levels of visual acuity and different sensitivity to color and contrast. As a result, because the ambient luminance level affects whether rods and / or cones are active, there are differences in visual acuity and sensitivity to color and contrast at different ambient luminance levels. Advantageously, the light level differences in visual acuity and sensitivity to color and contrast can be applied to provide an additional basis for reducing resolution, which can be used in conjunction with changing resolution based on fixation point, as described above (e.g., with respect to FIGS. 12A and 14), or can be used separately without specifically changing resolution based on fixation point.

[0280] 18, a schematic diagram of an example process 1800 for adjusting virtual content based on the amount of light incident on a user's eye is shown. For convenience, the process may be described as being performed by a display system (e.g., wearable display system 60 (FIG. 9D), which may include processing hardware and software and optionally provide information to, e.g., offload processing to, and receive information from, one or more computers or other processing units external to the system).

[0281] In block 1810, the display system determines the amount of light reaching the retina. Preferably, this determination is an estimate of the amount of light reaching the retina, rather than a direct measurement of the light impinging on the retina. This estimation may be made using methods disclosed for determining brightness levels, as discussed herein. For example, the brightness level may be assumed to correspond to the amount of light reaching the retina. As a result, determining the amount of light reaching the retina may include determining the size of the user's pupils and / or determining the ambient brightness level using a sensor configured to detect light, such as an outward-facing camera on the display device.

[0282] In block 1820, the display system adjusts the resolution of the virtual content to be presented to the user based on the amount of light found to reach the retina in block 1810. In some embodiments, adjusting the resolution of the virtual content includes adjusting one or more of the spatial resolution, color depth, and light intensity resolution of the virtual content. It should be understood that the human visual system has greatest visual acuity and sensitivity with respect to spatial resolution, color, and light intensity under photopic lighting levels. The ability to perceive differences in spatial resolution, color, and light intensity decreases under mesopic lighting levels and further decreases under scotopic lighting levels.

[0283] As a result, in some embodiments, if the amount of light present is found to correspond to a level for photopic vision, the virtual object may be rendered at full or high spatial resolution (compared to the spatial resolution that would be utilized for mesopic or scotopic vision). If the amount of light present is found to correspond to a level for mesopic vision, the virtual object may be rendered at reduced spatial resolution compared to the spatial resolution utilized for virtual objects under photopic lighting levels. If the amount of light is found to correspond to a scotopic vision level, the virtual object may be rendered at a spatial resolution lower than that used under mesopic or photopic lighting levels. The spatial resolution may be adjusted, for example, by reducing the number of polygons, as described herein.

[0284] The color depth or bit depth may also be adjusted according to the illumination level, with the highest color depth being used under photopic illumination levels, a medium color depth being used under mesopic illumination levels, and the lowest color depth being used under scotopic illumination levels. It should be understood that the color depth may be adjusted by varying the number of bits used per color component of a pixel, with fewer bits being equivalent to a lower color depth.

[0285] Similarly, without being limited by theory, it is believed that gradations in light intensity become greater as illumination levels progress from photopic to mesopic to scotopic illumination levels. In other words, it is believed that the human visual system is capable of discriminating smaller differences in light intensity as ambient illumination levels decrease. In some embodiments, the display system may be configured to display fewer gradations in light intensity as illumination levels progress from photopic to mesopic to scotopic illumination levels. As a result, the greatest number of gradations in light intensity levels are presented under photopic illumination levels, fewer gradations are presented under mesopic illumination levels, and even fewer gradations are presented under scotopic illumination levels.

[0286] Additionally, in some embodiments, the display system may be capable of providing a greater number of gradations in light intensity than are perceptible to a user. An example of this is illustrated in FIGS. 22a-22c and discussed further below. For example, the display system may be capable of displaying 256 different levels of intensity for a given image pixel, but the user may be able to perceive a smaller number of levels, e.g., only 64 levels. In this instance, multiple possible light intensity levels are subsumed within a single perceptible light intensity level. For example, the display system may be capable of displaying four different light intensity levels, but the user may perceive all four as similar. In such a situation where multiple possible light intensities are perceived by the user as being the same, the display system may be configured to select the lowest intensity value for display from those values ​​perceived as similar. As a result, the display system may be able to utilize lower intensities, thereby reducing the amount of power used to illuminate the display and achieve the desired light intensity. This may have particular advantages in display systems in which the individual pixels of the spatial light modulator are themselves light emitters, such as organic and inorganic LEDs. In some embodiments, the number of gray levels decreases with decreasing ambient lighting levels, and the display system is configured to group a greater number of possible light intensity levels together and display the lowest light intensity group.

[0287] It should be understood that one, two, or all three of the spatial resolution, color depth, and light intensity resolution of the virtual content to be displayed may be varied based on the light conditions (the amount of light reaching the user's retina) directed at the user. These adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made to the entire virtual content without making adjustments to resolution based on the distance from the fixation point of the user's eye, as disclosed herein. In some other embodiments, adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made in conjunction with adjustments to resolution based on the distance from the fixation point (e.g., see FIGS. 12A and 14). In some embodiments, if resolution decreases with distance from the fixation point, the profile of the decrease in resolution on a given plane (on the x- and y-axes) preferably matches the profile of the change in cone density across the corresponding portion of the retina.

[0288] In some embodiments, as described herein, adjustments to spatial resolution, color depth, and / or light intensity resolution are preferably tied to the mode of vision (photopic, mesopic, or scotopic) that is active at a given time. These adjustments may change dynamically as the mode of vision changes. For example, as a user progresses from photopic to scotopic vision, resolution may decrease, as discussed herein. Conversely, as a user progresses from scotopic to mesopic vision, resolution of the virtual content may increase. It should be understood that tying adjustments to resolution to a particular mode of vision does not require a specific determination that the user is in that particular mode. Rather, the display system may simply be configured to associate a particular ambient lighting level or range of pupil sizes with a particular resolution, regardless of spatial resolution, color depth, or light intensity resolution. In addition, the resolution adjustment is preferably associated with three levels of light conditions (corresponding to three modes of vision) as discussed herein, although in some embodiments the resolution adjustment may be associated with two levels of light conditions or more than three levels of light conditions.

[0289] It should also be understood that resolution adjustments may occur in real time (e.g., as ambient light conditions change) or may be delayed for a set duration to allow the human visual system to adapt to existing light conditions before resolution adjustments are made to the virtual content. Without being limited by theory, it is believed that the human visual system requires a period of time to adapt to different lighting levels, and that the period of time increases as lighting levels decrease. As a result, in some embodiments, adjustments in resolution due to changing lighting levels do not occur until the user has been exposed to a particular lighting level for a set amount of time (e.g., substantially continuously exposed). For example, the set amount of time may be 5 minutes, 10 minutes, 15 minutes, or 20 minutes.

[0290] Continuing with reference to Figure 18, at block 1830, virtual content is presented to the user. This presentation of virtual content may occur as discussed herein, for example, as in block 1208 of Figure 12A or block 1412 of Figure 14.

[0291] Referring now to FIG. 19 , an example of the change in resolution detectable by a user's eye as the amount of light incident on the eye changes is diagrammatically illustrated. The figure illustrates an example of the human visual system's sensitivity to spatial resolution under different vision modes. Scotopic vision occurs in the low-light region 1910, mesopic vision occurs in the mid-light region 1920, and photopic vision occurs in the bright-light region 1930. As shown, the sensitivity to spatial resolution decreases substantially as the ambient illumination level decreases. In some embodiments, the adjustments to spatial resolution discussed above with respect to FIG. 18 correspond to the contours of the illustrated curves. For example, for a given light level in photopic or scotopic mode, virtual content is rendered with sufficient spatial resolution to meet or exceed the resolution value shown on the y-axis.

[0292] Referring now to FIG. 20, it should be understood that different photoreceptors can be used to perceive light of different wavelengths or colors. FIG. 20 graphically illustrates an example of differences in the eye's sensitivity to light of different colors at different levels of illumination. The differences in duration on the x-axis reflect the amount of time typically required for the human visual system to adapt to a particular ambient illumination level so that a particular mode of vision is activated. Notably, at ambient illumination levels corresponding to scotopic and mesopic vision, photoreceptors for red light may no longer be active, while photoreceptors for blue light are active under the lowest light conditions. It should be understood that red, green, and blue light correspond to the colors most typically used as primary colors in display systems to form full-color images (e.g., as discussed herein with respect to FIGS. 8-9B). In some embodiments, the display system may be configured to vary the rendering of images of different colors depending on the ambient illumination level.

[0293] Referring now to FIG. 21 , a schematic diagram of an example process 2100 for adjusting virtual content formed using multiple primary color images is shown, where the resolution adjustment is based on the color of the primary color images. In block 2110, a display system provides the virtual content to be presented using multiple primary color images. These may be different images of different primary colors that are directed to different waveguides, as discussed with respect to FIGS. 8-9B . As a result, in some embodiments, each stream of images for different primary colors may be rendered separately. Providing the virtual content to be presented using multiple primary color images may include utilizing a display system that outputs image streams of different primary colors to form a full-color image.

[0294] In block 2120, the display system may adjust the resolution of the primary color image based on its color. For example, the display system may select one of these primary color images for resolution adjustment. For example, the selection may be made based on a determination of the illumination level, as described above with respect to block 1810 of FIG. 18. As shown in FIG. 19, some primary colors may not be perceived by a user at some illumination levels. The display system may have stored therein information regarding illumination levels and the primary colors that are invisible at those levels. If there is a match between the illumination levels and the primary colors that are invisible at those levels, the image of that primary color may be selected for adjustment. In some environments, one adjustment may simply be to not render or display the primary color image if the ambient illumination levels are such that the user is not expected to perceive that color. For example, under scotopic illumination levels, the display system may be configured not to render or display the primary color red image.

[0295] 21, in block 2130, the virtual content is presented to the user. Presentation of the virtual content may occur as discussed herein, for example, in block 1208 of FIG. 12A or block 1412 of FIG. 14.

[0296] Referring now to Figures 22A-22C, as discussed above, and without being limited by theory, it is believed that the human visual system's ability to perceive gradations in light intensity varies with ambient illumination levels. Figures 22A-22C show examples of contrast sensitivity varying as the amount of light incident on a user's eye decreases. For example, Figure 22A can be understood to show contrast sensitivity under photopic light conditions, Figure 22B can be understood to show contrast sensitivity under mesopic light conditions, and Figure 22C can be understood to show contrast sensitivity under scotopic light conditions. Figure 22A shows contrast sensitivity varying from gradations 21101 to 2110 iSimilarly, FIG. 22B shows a progression 2100 from gray levels 21101 to 2110 i Similarly, FIG. 22C shows a progression 2102 from gray levels 21101 to 2110, going from high light intensity to low light intensity. i 18 illustrates a progression 2104 from high to low light intensity, progressing from high to low light intensity. Boxes 2120, 2130, and 2140 illustrate groups of intensity gradations that are perceived as identical by a user. The size of these groups is expected to increase with decreasing ambient lighting levels, as illustrated. Consequently, as described above with respect to FIG. 18 , in some embodiments, the display system may be configured to use the lowest intensity value within each group (e.g., within each of boxes 2120, 2130, and 2140).

[0297] 23, an example of a representation of the optic nerve and peripheral blind spot of a user's eye is illustrated. In some embodiments, in addition to, or as an alternative to, any of the resolution adjustments disclosed herein, the display system may be configured to refrain from rendering content in various locations where the content is not expected to be perceptible by the user. FIG. 23 illustrates a representation of the optic nerve and peripheral blind spot of a user's eye 210, respectively. L and 210 R Each eye has a separate optical axis 1003A and 1003B and an optic nerve 2300. L and 2300 R and optic nerve 2300L and 2300 R Each of the individual eyes 210 L and 210 R These blind spots cause the viewer to see the rays 2302 L and 2302 R In addition, there is a region around each eye where content cannot be seen by the opposite eye. For example, the left peripheral region P L Contents in left eye 210 L , but the right eye 210 ROn the other hand, the right peripheral area P R The contents of the right eye 210 R , but the left eye 210 L As a result, in some embodiments, the display system L and 210 R content that would be mapped to the blind spot of, for example, ray 2302 L and 2302 R Additionally or alternatively, in some embodiments, the display system may be configured to omit rendering the overlying content. L If it enters the left eye, the content will be L and / or the display system may be configured to omit rendering the content in the left peripheral region P L If it enters the right eye, the content will be R It should be appreciated that the locations of the blind spots and / or peripheral areas may be preset, for example, based on averages for a population of users, and / or may be adjusted and calibrated for a particular user through testing with content displayed in various locations and input from the user indicating whether the virtual object is visible. (Multiple image streams to provide content with different resolutions)

[0298] In some embodiments, a foveated image having high and low spatial resolution regions may be formed by two or more spatially overlapping image streams, each having a different resolution (e.g., different perceived pixel densities). For example, one of the image streams, e.g., a low-resolution image stream, may form an image having a wide field of view, and another of the image streams, e.g., a high-resolution image stream, may form an image having a narrow field of view. The narrow field of view image and the wide field of view image may contain similar content but are viewed by a user at different resolutions or pixel densities. These images may be overlaid on each other (e.g., occupying the same location in space simultaneously or closely in time, such that the viewer perceives the images as being presented simultaneously). Thus, a viewer may receive an aggregate image having a high resolution in a constrained portion of their field of view and a low-resolution image over a larger portion of their field of view. Preferably, as discussed herein, the high-resolution portion is mapped to the foveal visual region of the user's eye, while the low-resolution portion is mapped to the peripheral visual region of the user's eye. Thus, the difference in resolution between the high-resolution and low-resolution portions of the image is preferably not readily perceptible to the user.

[0299] In some environments, display systems for displaying high- and low-resolution images utilize the same spatial light modulator to form both images. Thus, the spatial light modulator has pixels of a fixed size and density. In display systems with fixed pixel sizes and densities, increasing the angular field of view (FOV) comes at the expense of spatial or angular resolution, as governed, for example, by Lagrange invariants. For example, if an SLM with a fixed number of pixels is used to form both high- and low-resolution images, spreading the pixels across the entire field of view will provide images with lower apparent resolution than constraining the pixels to a small portion of the total field of view. That is, the pixel density of high-resolution images is higher than the pixel density of low-resolution images. As a result, there is generally an inverse correlation between FOV and angular resolution. Because FOV and angular resolution affect image visibility and quality, this trade-off imposes constraints on the user experience and the ultimate achievable FOV and angular resolution in an AR or VR system. As will be apparent from the discussion herein, in some embodiments, the term “resolution” may be used to refer to “angular resolution.”

[0300] Head-mounted or wearable display devices can be configured to provide an immersive user experience by projecting virtual content directly into the user's eyes. While providing wide-FOV images with uniformly high resolution across the FOV can be beneficial, physiological limitations of the human visual system can prevent a user from perceiving or even noticing high-resolution images located within the peripheral region of the user's field of view. The inability to perceive high-resolution images within this peripheral region arises from the properties of the retina of the human eye, which contains two types of photoreceptors: rod cells and cone cells. Cones are more responsible for acuity (detail) vision. Rods and cones are distributed differently within the human eye. The highest concentration of cone cells is found within the fovea (i.e., the center of the retina), while the highest concentration of rod cells is found within the region immediately surrounding the fovea (i.e., the periphery of the retina). Because of this non-uniform distribution of rods and cones, the fovea is responsible for acute central vision (also called foveal vision). Visual acuity decreases with increasing distance from the fovea.

[0301] For AR or VR applications, a headset is generally worn by one user at a time. The headset can be configured to take advantage of the user's inability to perceive all details of a wide-field-of-view stream of images at once by limiting the display of high-resolution content to the area within the wide field of view currently focused by the user. In this way, the headset can provide the user with the appearance of a high-resolution, wide-FOV stream of images without the processing power that would otherwise be required to generate high-resolution content across the entire field of view. The stream of images presented to the user can take many forms and will generally be referred to as an image stream. For example, the image stream can indicate a static image by continually displaying the same image to the user, or it can indicate movement by displaying a stream of different images. In some embodiments, the headset can be configured to simultaneously display more than one image stream, and the different image streams can have different angular resolutions and extend across different regions of the user's FOV. It should be noted that the image stream associated with an AR system may not display content entirely across the specific region to which it is assigned, as AR systems are designed to mix virtual and real-world content.

[0302] According to some embodiments, the first image stream and the second image stream can be presented to the user simultaneously or in rapid succession, so that the two image streams appear to be displayed simultaneously. The first image stream can have a wide FOV and low resolution that encompasses the user's visual field and can evoke an immersive experience for the user. The second image stream can have a narrow FOV and high resolution that can be dynamically displayed within the boundaries of the first image stream according to the user's current fixation point as determined in real time using eye-tracking techniques. In other words, the second image stream can be shifted as the user's line of sight changes so that the second image stream continuously covers the user's foveal vision. In some embodiments, the first image stream is presented to the user at a fixed position as the second image stream is shifted relative to the first image stream. In some other embodiments, both the first image stream and the second image stream are shifted according to the user's current fixation point.

[0303] The content of the second image stream can include a subset of the content of the first image stream with higher resolution than the first image stream and can be overlaid on and appropriately aligned with the first image stream. Because the higher-resolution second image stream overlays portions of the first image stream within the user's foveal vision, the lower resolution of the first image stream cannot be perceived or noticed by the user. In some embodiments, the subset of the content of the first image stream overlaid by the second image stream can be turned off or presented at a lower intensity for more uniform brightness and better resolution perception. In this way, the user can perceive the combination of the first image stream and the second image stream as having both a wide FOV and high resolution. Such a display system can provide several advantages. For example, the display system can have a relatively small form factor and conserve computing resources and power while providing a better user experience.

[0304] According to some embodiments, the first light beam associated with the first image stream and the second light beam associated with the second image stream can be multiplexed into a combined light beam using a multiplexing method. For example, time division multiplexing, polarization division multiplexing, wavelength division multiplexing, and the like can be used according to various embodiments. The combined light beam can be directed to one or more optical elements that serve to demultiplex the combined light beam into two separate optical paths. For example, a beam splitter or optical switching element, such as a polarizing beam splitter (PBS) or a dichroic beam splitter, can be used to separate the combined light beam, depending on the multiplexing method used. Once separated, the first light beam associated with the first image stream and the second light beam associated with the second image stream can be routed through their respective optical paths and ultimately provided to a user as an output.

[0305] According to some embodiments, a first light beam associated with a first image stream can be angularly expanded by optical elements in the first optical path such that the first image stream is presented with a wider FOV and lower angular resolution (as governed by Lagrange invariants), while a second light beam associated with a second image stream is angularly unexpanded, de-expanded, or expanded by an amount less than the amount of expansion applied to the first light beam associated with the first image stream. In this way, the second image stream can be presented with a narrower FOV and higher angular resolution (as governed by Lagrange invariants) than the first image stream.

[0306] FIG. 24 shows a visual field diagram depicting the perimeter of an exemplary monocular visual field 3002 for a human eye in two-dimensional angular space. As shown in FIG. 24, the temple-nasal and inferior-superior axes of the visual field diagram serve to define the two-dimensional angular space into which the perimeter of the monocular visual field 3002 is mapped. In this manner, the visual field diagram of FIG. 24 may be considered an equivalent or similar to a "Goldmann" visual field map or plot for a human eye. As indicated by the depicted arrangement of the temple-nasal and inferior-superior axes, the visual field diagram shown in FIG. 24 represents a visual field diagram for a human left eye. While the visual field may vary slightly from person to person, the depicted visual field approximates what most people are able to see with their left eye. A field of view diagram depicting the periphery of an exemplary monocular field of view for the right eye would result in the temple-nasal axis and periphery of the monocular field of view 3002 being similar to that of a version of the field of view diagram of FIG. 24 mirrored about the inferior-superior axis.

[0307] The field of view diagram of FIG. 24 further depicts the periphery of an exemplary ocular field of view 3004 for a human eye, which represents a portion of a monocular field of view 30022 in angular space at which a person may fixate. In addition, the field of view diagram of FIG. 24 also depicts the periphery of an exemplary foveal field 3006 for a human eye, which represents a portion of a monocular field of view 3002 in angular space within a direct view of the fovea of ​​a human eye at a given time. As depicted, a person's foveal field 3006 may move anywhere within the ocular field of view 3004. The portion of the monocular field of view 3002 outside the foveal field 3006 in angular space may be referred to herein as the peripheral region of a person's field of view. Because the human eye's ability to distinguish high levels of detail outside the foveal field 3006 is very limited, displaying reduced-resolution images outside the foveal field 3006 is unlikely to be noticeable and can enable substantial savings in power consumption for processing components involved in generating content for a display.

[0308] 25A shows an exemplary wearable display device 4050 configured to provide virtual content to a user, according to some embodiments. The wearable display device 4050 includes a primary display 4052 supported by a frame 4054. The frame 4054 can be attached to a user's head using attachment members in the form of temple arms 4006.

[0309] Referring now to FIG. 25B , an exemplary embodiment of an AR system configured to provide virtual content to a user will now be described. In some embodiments, the AR system of FIG. 25B may represent a system to which the wearable display device 4050 of FIG. 25A belongs. The AR system of FIG. 25B uses a stacked light guidance optics element assembly 4000 and generally includes an image generation processor 4010, a light source 4020, a controller 4030, a spatial light modulator (“SLM”) 4040, an injection optics system 4060, and at least one set of stacked eyepiece layers or light guidance optics elements (“LOEs,” e.g., planar waveguides) 4000 that function as a multiple planar focusing system. The system may also include an eye tracking subsystem 4070. It should be understood that other embodiments may have multiple sets of stacked LOEs 4000, but the following disclosure will focus on the exemplary embodiment of FIG. 25B .

[0310] The image generation processor 4010 is configured to generate the virtual content to be displayed to the user. The image generation processor may convert images or video associated with the virtual content into a format that can be projected to the user in 3-D. For example, when generating 3-D content, the virtual content may need to be formatted so that portions of certain images are displayed at certain depth planes, while others are displayed at other depth planes. In one embodiment, the images may all be generated at certain depth planes. In another embodiment, the image generation processor may be programmed to provide slightly different images to the right and left eyes 210 so that, when viewed together, the virtual content appears coherent and pleasant to the user's eyes.

[0311] The image generation processor 4010 may further include a memory 4012, a GPU 4014, a CPU 4016, and other circuitry for image generation and processing. The image generation processor 4010 may be programmed with desired virtual content to be presented to a user of the AR system of FIG. 25B . It should be understood that in some embodiments, the image generation processor 4010 may be stored within a wearable AR system. In other embodiments, the image generation processor 4010 and other circuitry may be stored within a belt pack coupled to the wearable optics. The image generation processor 4010 is operably coupled to a light source 4020 and one or more spatial light modulators (described below) that project light associated with the desired virtual content.

[0312] The light source 4020 is compact and has high resolution. The light source 4020 includes multiple spatially separated sub-light sources 4022 operably coupled to a controller 4030 (described below). For example, the light source 4020 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 4020 may include similarly colored LEDs or lasers, each one coupled to a specific region of the display's field of view. In another embodiment, the light source 4020 may comprise a wide-area emitter, such as an incandescent or fluorescent lamp, with a mask overlay to segment the emission area and location. Although the sub-light sources 4022 are directly connected to the AR system of FIG. 2B in FIG. 2B , the sub-light sources 222 may also be connected to the system via optical fibers (not shown), as long as the distal ends of the optical fibers (away from the sub-light sources 4022) are spatially separated from each other. The system may also include a concentrator (not shown) configured to collimate the light from the light source 4020.

[0313] The SLM 4040, in various exemplary embodiments, may be reflective (e.g., DLP DMD, MEMS mirror system, LCOS, or FLCOS), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED). The type of spatial light modulator (e.g., speed, size, etc.) can be selected to improve the creation of the 3-D perception. While DLP DMDs operating at higher refresh rates can be easily incorporated into stationary AR systems, wearable AR systems typically use DLPs of smaller size and power. The power of the DLP changes the way the 3-D depth plane / focal plane is created. The image generation processor 4010 is operably coupled to the SLM 4040, which encodes light from the light source 4020 with desired virtual content. Light from the light source 4020 may be encoded with image information as it reflects from, emits from, or passes through the SLM 4040.

[0314] Referring back to FIG. 25B , the AR system also includes an input optical system 4060 configured to direct light from the light source 4020 (i.e., the multiple spatially separated sub-light sources 4022) and the SLM 4040 into the LOE assembly 4000. The input optical system 4060 may include one or more lenses configured to direct light into the LOE assembly 4000. The input optical system 4060 is configured to form adjacent spatially separated and distinct pupils in the LOE 4000 (at separate focal points of the beams exiting the input optical system 4060) corresponding to the spatially separated and distinct beams from the sub-light sources 4022 of the light source 4020. The input optical system 4060 is configured such that the pupils are spatially displaced from one another. In some embodiments, the input optical system 4060 is configured to spatially displace the beams only in the X and Y directions. In such embodiments, the pupils are formed in a single X,Y plane. In other embodiments, the launch optical system 4060 is configured to spatially displace the beam in the X, Y, and Z directions.

[0315] The spatial separation of the light beams forms distinct beams and pupils, which allows for the placement of internal coupling gratings in distinct beam paths such that each internal coupling grating is primarily addressed (e.g., intersects or impinges) by only one distinct beam (or group of beams). This, in turn, facilitates the injection of spatially separated light beams into individual LOEs 4000 of the LOE assembly 4000 while minimizing the injection of other light beams from other sub-light sources 4022 of the plurality (i.e., crosstalk). A light beam from a particular sub-light source 4022 enters an individual LOE 4000 through an internal coupling grating thereon (not shown in FIG. 25B , see FIGS. 24-26 ). The internal coupling gratings of the individual LOEs 4000 are configured to interact with the spatially separated light beams from the multiple sub-light sources 4022 such that each spatially separated light beam only intersects the internal coupling grating of one LOE 4000. Thus, each spatially separated light beam is primarily incident on one LOE 4000. Thus, image data encoded by SLM 4040 onto the light beams from each of the sub-light sources 4022 can be efficiently propagated along the single LOE 4000 for delivery to the user's eye 210.

[0316] Each LOE 4000 is then configured to project an image or sub-image onto the user's retina that appears to originate from a desired depth plane or FOV angular location. The individual multiple LOEs 4000 and sub-light sources 4022 can thus selectively project images (synchronously encoded by SLM 4040 under control of controller 4030) that appear to originate from different depth planes or locations in space. By using each of the individual multiple LOEs 4000 and sub-light sources 4022 to sequentially project images at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full-volume frame rate of 60 Hz), the system of FIG. 25B can generate 3-D images of virtual objects at different depth planes that appear to exist simultaneously within the 3-D image.

[0317] The controller 4030 communicates with and is operatively coupled to the image generation processor 4010, the light source 4020 (sub-light source 4022), and the SLM 4040, and coordinates the synchronous display of images by instructing the SLM 4040 to encode the light beam from the sub-light source 4022 with appropriate image information from the image generation processor 4010.

[0318] The AR system also includes an optional eye tracking subsystem 4070 configured to track the user's eyes 4002 and determine the user's focus. In one embodiment, only a subset of the sub-light sources 4022 may be activated to illuminate a subset of the LOEs 4000 based on input from the eye tracking subsystem, as discussed below. Based on input from the eye tracking subsystem 4070, one or more sub-light sources 4022 corresponding to a particular LOE 4000 may be activated such that an image is generated at a desired depth plane consistent with the user's focus / accommodation. For example, if the user's eyes 210 are parallel to one another, the AR system of FIG. 25B may activate sub-light sources 4022 corresponding to LOEs 4000 configured to deliver collimated light to the user's eyes so that the image appears to originate from optical infinity. In another example, if the eye tracking subsystem 4070 determines that the user's focus is one meter away, then sub-light sources 4022 corresponding to LOEs 4000 configured to focus approximately within that range may instead be activated. It should be appreciated that in this particular embodiment, only one group of sub-light sources 4022 is activated at any given time, while the other sub-light sources 4020 are deactivated to conserve power.

[0319] FIG. 25C schematically illustrates light paths within an exemplary viewing optical assembly (VOA) that may be used to present digital or virtual images to a viewer, according to some embodiments. In some embodiments, the VOA may be incorporated into a system similar to the wearable display device 4050 as depicted in FIG. 25A. The VOA includes a projector 4001 and an eyepiece 200 that may be worn around the viewer's eye. The eyepiece 4000 may correspond to, for example, the LOE 4000 described above with reference to FIG. 25B. In some embodiments, the projector 4001 may include a group of red LEDs, a group of green LEDs, and a group of blue LEDs. For example, the projector 201 may include two red LEDs, two green LEDs, and two blue LEDs, according to an embodiment. In some examples, projector 4001 and its components (e.g., LED light source, reflective collimator, LCoS SLM, and projector relay) as depicted in FIG. 25C may represent or provide the functionality of one or more of light source 4020, sub-light source 4022, SLM 4040, and injection optical system 4060 as described above with reference to FIG. 25B. Eyepiece 4000 may include one or more eyepiece layers, each of which may represent one of LOEs 4000 as described above with reference to FIG. 25B. Each eyepiece layer of eyepiece 4000 may be configured to project an image or sub-image onto the retina of a viewer's eye that appears to result from a respective desired depth plane or FOV angular position.

[0320] In one embodiment, the eyepiece 4000 includes three eyepiece layers, one for each of the three primary colors, i.e., red, green, and blue. For example, in this embodiment, each eyepiece layer of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from the optical infinity depth plane (0 diopters). In another embodiment, the eyepiece 4000 may include six eyepiece layers, i.e., one set of eyepiece layers for each of the three primary colors configured to form a virtual image at one depth plane, and another set of eyepiece layers for each of the three primary colors configured to form a virtual image at another depth plane. For example, in this embodiment, each ocular lens layer in one set of ocular lens layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from the optical infinity depth plane (0 diopters), while each ocular lens layer in another set of ocular lens layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from a distance of 2 meters (0.5 diopters). In other embodiments, the eyepiece 4000 may include three or more ocular lens layers, one for each of three primary colors for three or more different depth planes. For example, in such an embodiment, yet another set of ocular lens layers may each be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter).

[0321] Each eyepiece layer comprises a planar waveguide and may include an internal coupling grating 4007, an orthogonal pupil expander (OPE) region 4008, and an exit pupil expander (EPE) region 4009. Further details regarding internal coupling gratings, orthogonal pupil expansion, and exit pupil expansion are described in U.S. Patent Application Nos. 14 / 555,585 and 14 / 726,424, the contents of which are expressly and fully incorporated herein by reference in their entirety as if set forth in their entirety. Still referring to FIG. 25C , projector 4001 projects image light onto internal coupling grating 4007 in eyepiece layer 4000. Internal coupling grating 4007 couples the image light from projector 4001 into a waveguide, where it propagates in a direction toward OPE region 4008. The waveguide propagates the image light horizontally by total internal reflection (TIR). The OPE region 4008 of the eyepiece lens layer 4000 also includes diffractive elements that couple a portion of the image light propagating in the waveguide and redirect it toward the EPE region 4009. More specifically, the collimated light propagates horizontally (i.e., relative to the view of FIG. 25C ) along the waveguide via TIR, and in doing so, repeatedly intersects with the diffractive elements of the OPE region 4008. In some embodiments, the diffractive elements of the OPE region 4008 have a relatively low diffraction efficiency. This causes a percentage (e.g., 10%) of the light to be diffracted vertically downward toward the EPE region 4009 at each point of intersection with a diffractive element in the OPE region 4008, and a percentage of the light to continue horizontally along the waveguide on its original trajectory via TIR. Thus, at each point of intersection with a diffractive element in the OPE region 4008, additional light is diffracted downward toward the EPE region 4009. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light is expanded horizontally by the diffractive elements in the OPE region 4008. The expanded light that is outcoupled from the OPE region 4008 enters the EPE region 4009.

[0322] The EPE region 4009 of the eyepiece lens layer 4000 also includes diffractive elements that couple a portion of the image light propagating in the waveguide and redirect it toward the viewer's eye 210. Light entering the EPE region 4009 propagates perpendicularly (i.e., relative to the view of FIG. 25C ) along the waveguide by TIR. At each point of intersection between the propagating light and the diffractive elements of the EPE region 4009, a percentage of the light is diffracted toward the adjacent face of the waveguide, allowing the light to escape the TIR, emerge from the face of the waveguide, and propagate toward the viewer's eye 210. In this manner, the image projected by the projector 4001 can be viewed by the viewer's eye 210. In some embodiments, the diffractive elements of the EPE region 4009 may be designed or configured to have a phase profile that is the sum of a linear diffraction grating and a radially symmetric diffractive lens. The radially symmetric lens sides of the diffractive elements of EPE region 4009 additionally impart a level of focus to the diffracted light, both shaping the optical wavefronts of the individual beams (e.g., imparting curvature) and steering the beams at an angle consistent with the designed level of focus. Each beam of light outcoupled by the diffractive elements of EPE region 4009 may geometrically extend to a discrete focal point located directly in front of the viewer and may be imparted with a convex wavefront profile with a radius centered at the discrete focal point, producing an image or virtual object at a given focal plane.

[0323] Further descriptions of such viewing optical assemblies and other similar setups are provided in U.S. Patent Application Nos. 14 / 331,218, 15 / 146,296, and 14 / 555,585, all of which are incorporated herein by reference in their entireties. It follows that in some embodiments, an exemplary VOA may include and / or take the form of one or more components described above with reference to FIG. 25C and described in any of the patent applications incorporated herein by reference. III. Large Field of View and High Resolution Foveated Displays Using Multiple Optical Paths

[0324] 26A-26D illustrate example rendering viewpoints to be used and light fields to be produced within an AR system for two example eye orientations. In FIG. 26A , the viewer's eye 210 is oriented in a first manner relative to the eyepiece 5000. In some embodiments, the eyepiece 5000 may be similar to the stack of LOEs or the eyepiece 4000, as described above with reference to FIGS. 25B and 25C . More specifically, in this example, the viewer's eye 210 is oriented such that the viewer may be able to view the eyepiece 5000 in a relatively straight-ahead direction. In some examples, the AR system to which the eyepiece 5000 belongs, which may be similar to the AR system described above with reference to FIG. 25B , may perform one or more operations to present virtual content at one or more distances in front of the viewer's eye 210, on one or more depth planes positioned within the viewer's FOV.

[0325] The AR system may determine a viewpoint within the rendering space based on the viewer's head position and orientation, from which the viewer views 3-D virtual content, such as virtual objects, in the rendering space. As described in further detail below with reference to FIG. 29A , in some embodiments, such an AR system may include one or more sensors and utilize data from these one or more sensors to determine the viewer's head position and / or orientation. The AR system may include such one or more sensors in addition to one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to FIG. 25B . Using such data, the AR system may effectively map the viewer's head position and orientation in the real world to a specific location and a specific angular position in the 3D virtual environment, create a virtual camera positioned at the specific location in the 3D virtual environment and oriented at the specific angular position in the 3D virtual environment relative to the specific location in the 3D virtual environment, and render the virtual content for the viewer as would be captured by the virtual camera. Further details discussing the real world to virtual world mapping process are provided in the "Selecting Virtual Objects in a Virtual Environment" section of this document. No. 15 / 296,869, entitled "A Three-Dimensional Space," which is expressly incorporated herein by reference in its entirety for all purposes.

[0326] In some examples, the AR system may create or dynamically reposition and / or reorient one such head-tracking virtual camera for the viewer's left eye or eye socket and another such head-tracking virtual camera for the viewer's right eye or eye socket as the viewer's eyes and / or eye sockets are physically separated from one another and therefore consistently positioned in different locations. It follows that virtual content rendered from the perspective of the head-tracking virtual camera associated with the viewer's left eye or eye socket may be presented to the viewer through an eyepiece on the left side of a wearable display device such as that described above with reference to FIGS. 25A-25C , and virtual content rendered from the perspective of the head-tracking virtual camera associated with the viewer's right eye or eye socket may be presented to the viewer through an eyepiece on the right side of the wearable display device. A head-tracking virtual camera may be created and / or dynamically repositioned for each eye or eye socket based on information about the current position and orientation of the viewer's head, although the position and orientation of such head-tracking virtual cameras may be independent of either the viewer's individual eye sockets or the position or orientation of each of the viewer's eyes relative to the viewer's head. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTUAL FEATURES IN 3D RECONSTRUCTION," which is expressly incorporated herein by reference in its entirety for all purposes.

[0327] The AR system of Figure 26A may create or dynamically reposition and / or reorient such a head-tracked virtual camera, render virtual content from the head-tracked virtual camera's viewpoint (point of view 5010), and project light representing the rendering of the virtual content through the eyepiece 5000 onto the retina of the viewer's eye 210. As shown in Figure 26A, the head-tracked rendering viewpoint 5010 may be angled diagonally, horizontally, and / or vertically at angles ±θ 310The head-tracking rendering viewpoint 5010 may provide a FOV spanning an area of ​​angular units. As described in more detail below, in some embodiments, the head-tracking rendering viewpoint 5010 may provide a relatively wide FOV. In such embodiments, the AR system may also create or dynamically reposition and / or reorient a separate virtual camera for each eye or eye socket, distinct from and in addition to the head-tracking virtual camera. In the example of FIG. 26A , the AR system may render and present virtual content from the viewpoint of the head-tracking virtual camera 5010 as well as virtual content from the viewpoint of another virtual camera in the rendering space.

[0328] For example, in such an embodiment, the AR system of FIG. 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera based on the current line of sight of the viewer's eye 210. As described in further detail below with reference to FIG. 29A, in some examples, such an AR system may include one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to FIG. 25B, to determine the viewer's current line of sight, the current position and / or orientation of the viewer's eye 210 relative to the viewer's head, and the like. Using such data, the AR system of FIG. 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera, render virtual content from the foveal tracking virtual camera's viewpoint (viewpoint 5020A), and project light representing the virtual content as rendered from viewpoint 5020A through eyepiece 5000 onto the fovea of ​​the viewer's eye 210.

[0329] 26A , the foveated tracking rendering viewpoint 5020A may provide a narrower FOV than that of the head tracking rendering viewpoint 5010. In this manner, the FOV of the foveated tracking rendering viewpoint 5020A may be viewed as occupying a conical subspace of the FOV of the head tracking rendering viewpoint 5010. That is, the FOV of the foveated tracking rendering viewpoint 5020A may be a subfield of the FOV of the head tracking rendering viewpoint 5010. For example, as shown in FIG. 26A , the foveated tracking rendering viewpoint 320A may be configured such that the relationship between the FOV of the head tracking rendering viewpoint 5010 and the foveated tracking rendering viewpoint 5020A is −θ 310 ≦-θ 320A ≦θ 320A ≦θ 310 Diagonally, horizontally, and / or vertically, ±θ 320A The head-tracking virtual camera 5010 may provide an FOV spanning an area of ​​angular units. In some examples, the FOV of the head-tracking rendering viewpoint 5010 may be at least as wide as the viewer's ocular field of view, which in this example would be the total cone-shaped space that the viewer's eye 210 may fixate on when the viewer's head is held at a given position and orientation. Thus, in these examples, the head-tracking virtual camera and the foveal-tracking virtual camera may be positioned at substantially the same location in the rendering space, or may be positioned at locations in the rendering space that are a fixed distance from each other so that both virtual cameras can be translated linearly and / or angularly in unison within the rendering space as the position and / or orientation of the viewer's head changes. For example, the head-tracking virtual camera may be positioned at a location in the rendering space that corresponds to the center of rotation of the viewer's eye 210, while the foveal-tracking virtual camera may be positioned at a location in the rendering space that corresponds to the region of the viewer's eye 210 between the center of rotation and the cornea. In fact, the Euclidean distance between two virtual cameras can remain substantially constant when translated in rendering space, in the same way that the Euclidean distance between two concrete regions of a viewer's eye 210 or another rigid body can remain substantially constant at all times.

[0330] The spatial relationship between each virtual camera in such a pair of virtual cameras may remain substantially fixed in rendering space throughout use of the AR system; however, in these examples, the orientation of the foveated tracking virtual camera may vary relative to the head-tracking virtual camera as the viewer rotates their eyes 210. In this manner, the conical subspace of the head-tracking virtual camera's FOV occupied by the foveated tracking virtual camera's FOV may change dynamically as the viewer rotates their eyes 210.

[0331] Furthermore, virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020A may be rendered and presented by the AR system at relatively high resolution. More specifically, the resolution at which virtual content within the FOV of the foveal tracking virtual camera is rendered and presented may be higher than the resolution at which virtual content within the FOV of the head tracking virtual camera is rendered and presented. Thus, the highest resolution subfield of a given light field that is coupled out by the eyepiece 5000 and projected onto the retina of the viewer's eye 210 may be that which reaches the fovea of ​​the viewer's eye 210.

[0332] FIG. 3B illustrates an exemplary light field 5030A that is outcoupled by the eyepiece 5000 and projected onto the retina of the viewer's eye 210 while the viewer's eye 210 is oriented in a first manner as depicted in and described above with reference thereto. The light field 5030A may include various angular light components representing virtual content as would be captured in rendering space by the pair of virtual cameras described above. As described in further detail below with reference to FIG. 26A and thereafter, light representing virtual content as would be captured in rendering space by the head-tracking virtual camera and light representing virtual content as would be captured in rendering space by the foveal-tracking virtual camera may be multiplexed by the AR system according to any of a variety of different multiplexing schemes. Employing such a multiplexing scheme may, at least in some instances, enable the AR system to operate with greater efficiency and / or occupy less physical space.

[0333] Still referring to FIG. 26B, the angular light component of light field 5030A, which represents the virtual content as it would be captured in rendering space by the head-tracking virtual camera (e.g., virtual objects and other content that fall within head-tracking rendering viewpoint 5010), is angled at −θ 310 ~+θ 310 The angular light components of the light field 5030A, which represent the virtual content as it would be captured in rendering space by the foveal tracking virtual camera (e.g., virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020A), may be projected onto the retina of the viewer's eye 210 at angles spanning angular units. Similarly, the angular light components of the light field 5030A, which represent the virtual content as it would be captured in rendering space by the foveal tracking virtual camera (e.g., virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020A), may be projected at angles spanning angular units relative to the viewer's eye 210 at angles spanning angular units. 320A ~+θ 320A The angle components associated with the foveal tracking rendering viewpoint 5020A may include those that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. 320A and +θ320A The spacing between angle units is −θ 310 and +θ 310 The spacing between angular units may be more regular than the spacing between angular units. In this manner, the resolution at which virtual content associated with the foveal tracking rendering viewpoint 5020A may be rendered and presented to the viewer may be higher than the resolution at which virtual content associated with the head tracking rendering viewpoint 5010 may be rendered and presented to the viewer.

[0334] In some embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030A may also be angled relative to the viewer's eyes 210 by −θ 320A ~+θ 320A 5030A. In such an embodiment, the angular light components associated with the head tracking rendering viewpoint 5010 may include those that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. In such an embodiment, the angular light components associated with the head tracking rendering viewpoint 5010 may occur within the light field 5030A, such as -θ 320A and +θ 320A The spacing between angle units is −θ, where the angular light component associated with the foveated tracking rendering viewpoint 5020A occurs within the light field 5030A. 320A and +θ 320A In other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030A are spaced apart by −θ relative to the viewer's eyes 210. 320A ~+θ 320A 5030A may exclude angular components that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. Thus, in these other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030A may be filtered out by −θ 310 ~-θ 320A Angle or θ in degrees 320A ~θ 310 2. The image may be projected onto the retina of the viewer's eye 210 at an angle of .gtoreq..times ...

[0335] In Figure 26C, the viewer's eye 210 is oriented relative to the eyepiece 5000 in a second manner that is different from the first manner in which the viewer's eye 210 is oriented relative to the eyepiece 5000 in Figures 26A-26B. For purposes of this example, the viewer's head position and orientation in Figures 26C-26D may be treated as being identical to the viewer's head position and orientation as described above with reference to Figures 26A-26B. Thus, Figures 26A-26B and Figures 26C-26D may represent the viewer and AR system described above at first and second time-series stages, respectively. More specifically, in this example, the viewer's eye 210 is rotated off-center from a relatively straight-ahead orientation as depicted in Figures 26A-26B.

[0336] During the transition from the first stage to the second stage, the AR system of FIG. 26C may function, for example, to maintain the head-tracking virtual camera in the same position and orientation as described above with reference to FIGS. 26A-26B because the viewer's head pose (e.g., position and orientation) has not changed. Thus, during the second stage depicted in FIGS. 26C-26D, the AR system may render virtual content from the head-tracking virtual camera's perspective (i.e., head-tracking rendering viewpoint 5010) and project light representing the rendering of the virtual content onto the retina of the viewer's eye 210 through the eyepiece 5000. While the head-tracking rendering viewpoint 5010 may remain static or relatively static throughout the first and second time-sequential stages of FIGS. 26A-26D, during the transition from the first stage to the second stage, the AR system may function to adjust the orientation of the foveal-tracking virtual camera within the rendering space based on the change in the line of sight of the viewer's eye 210 from the first stage to the second stage. That is, the AR system may displace or reorient the foveal tracking virtual camera as employed in the first stage to provide the foveal tracking rendering viewpoint 5020A, such that the foveal tracking virtual camera as employed in the second stage provides a foveal tracking rendering viewpoint 5020C that is different from the foveal tracking rendering viewpoint 5020A. In the second stage, the AR system may also render the virtual content from the perspective of the foveal tracking virtual camera viewpoint 5020C and project light representing the rendering of the virtual content through the eyepiece 5000 onto the fovea of ​​the viewer's eye 201.

[0337] 26C-26D, the foveated tracking rendering viewpoint 5020C may occupy a different conical subspace of the head tracking rendering viewpoint 5010 than that of the foveated tracking rendering viewpoint 5020A. For example, as shown in FIG. 26C, the foveated tracking rendering viewpoint 5020C is located θ 320C The angular units are displaced diagonally, horizontally, and / or vertically, ±θ 320AThat is, the foveated tracking rendering viewpoint 5020C may provide a FOV that spans an area in degrees, i.e., diagonally, horizontally, and / or vertically, θ 320C ±θ 320A A FOV may be provided, covering an area in degrees.

[0338] 26D illustrates an example light field 5030C that is coupled out by the eyepiece 5000 and projected onto the retina of the viewer's eye 201 while the viewer's eye 201 is oriented in the second manner as depicted in and described above with reference to FIG. 26C. The light field 5030C may include various angular light components that represent the virtual content as it would be captured in rendering space from the head tracking rendering viewpoint 5010 and the foveated tracking rendering viewpoint 5020C. The angular light components of the light field 5030C that represent the virtual content as it would be captured in rendering space from the head tracking rendering viewpoint 5010 are angled at -θ relative to the viewer's eye 210. 310 ~+θ 310 26A-26B, starting from the first stage, however, the angular light components of the light field 5030C, representing the virtual content as it would be captured in rendering space by the foveal tracking virtual camera (e.g., virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020C), are at angles ranging from θ 320C -θ 320A angle unit ~θ 320C +θ 320A It may include what would be projected onto the retina of the viewer's eye 210 at angles spanning angular units.

[0339] Such angular light components associated with the foveated tracking rendering viewpoint 320C occur within the light field 5030C, θ 320C -θ 320A Angle units and θ 320C +θ 320AThe spacing between angle units is the angular light component associated with the head tracking rendering viewpoint 5010 that occurs within the light field 5030C, −θ 310 and +θ 310 In this manner, the resolution at which virtual content associated with the foveal tracking rendering viewpoint 5020C may be rendered and presented to the viewer is notably −θ 320A ~+θ 320A The resolution may be higher than the resolution at which virtual content associated with the head tracking rendering viewpoint 5010 can be rendered and presented to the viewer, including virtual content represented by angular light components that will be projected onto the retina of the viewer's eye 210 at angles spanning angular units.

[0340] In some embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030C are further angled relative to the viewer's eyes 210 by θ 320C -θ 320A angle unit ~θ 320C +θ 320A Angular light components associated with the head tracking rendering viewpoint 310 may include those that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. In such an embodiment, such angular light components associated with the head tracking rendering viewpoint 310 occur within the light field 5030C, such as -θ 320C -θ 320A Angle units and θ 320C +θ 320A The spacing between angular units is the angular light component associated with the foveated tracking rendering viewpoint 5020C that occurs within the light field 5030C, θ 320C -θ 320A Angle units and θ 320C +θ 320A In other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030C may be spaced apart by θ relative to the viewer's eyes 210. 320C -θ 320A angle unit ~θ 320C +θ 320A5030C. The angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030C may be filtered out by -θ 310 ~θ 320C -θ 320A Angle and θ in degrees 320C +θ 320A Angle~θ 310 It may be what will be projected onto the retina of the viewer's eye 210 in degrees or degrees in degrees.

[0341] 26E-26F diagrammatically illustrate example configurations of images that may be presented to a user, according to some embodiments. Note that the grid squares in FIGS. 26E-26F diagrammatically represent image points where fields of view 3002, 3004, and 3006 are defined in two-dimensional angular space, as described above with reference to FIG. 24. A low-resolution first image stream 5010E with a wide FOV can be displayed at a static location. The low-resolution first image stream 5010E with a wide FOV can represent one or more images of virtual content as would be captured by a first virtual camera having a static position and orientation in the rendering space. For example, the low-resolution first image stream 5010E can represent one or more images of virtual content as would be captured by a head-tracking virtual camera, such as the head-tracking virtual camera described above with reference to FIGS. 26A-26D. The first image stream 5010E can encompass the user's vision and evoke an immersive experience for the user.

[0342] A high-resolution second image stream 5020E having a relatively narrow FOV can be displayed within the boundaries of the first image stream 5010E. In some examples, the second image stream 5020E can represent one or more images of the virtual content as would be captured by a second, different virtual camera having an orientation within the rendering space that can be dynamically adjusted in real time to an angular position consistent with the user's current fixation point based on data acquired using eye-tracking techniques. In these examples, the high-resolution second image stream 5020E can represent one or more images of the virtual content as would be captured by a foveal-tracking virtual camera, such as the foveal-tracking virtual camera described above with reference to FIGS. 26A-26D. In other words, the viewpoint within the rendering space from which one or more images of the virtual content represented by the second image stream 5020E are captured can be reoriented as the user's eye line of sight changes, such that the viewpoint associated with the second image stream 5020E is persistently aligned with the user's foveal vision.

[0343] For example, the second image stream 5020E can encompass virtual content located within a first region of the rendering space when the user's eye line of sight is fixated at a first position, as illustrated in Figure 26E. As the user's eye line of sight moves to a second position different from the first position, the viewpoint associated with the second image stream 5020E can be adjusted, and the second image stream 5020E can encompass virtual content located within the second region of the rendering space, as illustrated in Figure 26F. In some embodiments, the first image stream 5010E has a wide FOV but low angular resolution, as indicated by the sparse grid. The second image stream 5020E has a narrow FOV but high angular resolution, as indicated by the dense grid.

[0344] FIG. 26G diagrammatically illustrates an example configuration of images that may be presented to a user according to some other embodiments. As in FIGS. 26E-26F, the grid squares in FIG. 26G diagrammatically represent image points defined in two-dimensional angular space. Similar to the configuration illustrated in FIGS. 26E-26F, a low-resolution first image stream 5010G with a wide FOV encompasses the virtual content as viewed from a head-tracking rendering viewpoint, while a high-resolution second image stream 5020G with a narrow FOV encompasses the virtual content as viewed from a foveated-tracking rendering viewpoint that may be dynamically reoriented to coincide with the user's current fixation point. Here, the perimeter of the FOV associated with the first image stream 5010G may form a rectangular boundary with rounded corners, and the perimeter of the FOV associated with the second image stream 5020G may form a circular boundary.

[0345] FIG. 26H also schematically illustrates an example configuration of an image that may be presented to a user according to some other embodiments. As in FIGS. 26E-26G, the grid squares in FIG. 26H diagrammatically represent image points defined in two-dimensional angular space. Here, the perimeter of the FOV associated with the first image stream 5010H and the perimeter of the FOV associated with the second image stream 5020H may both form a circular boundary. In some other embodiments, one or both of the perimeters of the FOV associated with the first image stream 5010H and the perimeter of the FOV associated with the second image stream 5020H may form an elliptical boundary or other shape. In some embodiments, the image source of the AR system of FIG. 26H may include a scanning fiber that may be scanned in a predetermined pattern to provide light beams for the first image stream 5010H and the second image stream 5020H with a desired boundary shape.

[0346] 27 illustrates the field of view 3002 and ocular field of view 3004 as shown in FIG. 24 overlaid on one of the displays 4052 in a wearable display device 4050 as shown in FIG. 25A. According to some embodiments, a first image stream 5010E with a wide FOV and low resolution as illustrated in FIGS. 26E-26F may be displayed across the entire area of ​​the display 4052 (the relatively low resolution of the first image stream 5010E is illustrated with a sparse grid), while a second image stream 5020E with a narrow FOV and high resolution may be displayed in the user's current foveated region 3006 (the relatively high resolution of the second image stream 5020E is illustrated with a dense grid). In FIG. 27, the first image stream 5010E and the second image stream 5020E are illustrated as being displayed on the "plane" of the display 4052; however, in a see-through augmented reality (AR) display system, the first image stream 5010E and the second image stream 5020E can also be presented to the user as a light field within an angular field of view. Such an AR display system can produce a display plane that appears to "float" at a distance (e.g., 2 meters) in front of the user. The display plane can appear much larger than the glasses. This floating distance display is used to overlay information about the real world.

[0347] 28A-28B illustrate some of the principles described in FIGS. 26A-26D using example virtual content that may be presented to a user, according to some embodiments. Thus, FIGS. 28A-28B may represent a viewer and an AR system at first and second time-series stages, respectively. Furthermore, some or all of the components shown in FIGS. 28A-28B may be the same as or at least similar to components as described above with reference to FIGS. 26A-26D.

[0348] The AR system of Figures 28A-28B may create or dynamically reposition and / or reorient a head-tracking virtual camera similar to the head-tracking virtual camera described above with reference to Figures 26A-26D, render virtual content from the perspective of the head-tracking virtual camera, and project light representing the rendering of the virtual content through eyepiece 6000 onto the retina of viewer's eye 210. The AR system of Figures 28A-28B may also create or dynamically reposition and / or reorient a foveal-tracking virtual camera similar to the foveal-tracking virtual camera described above with reference to Figures 26A-26D, render virtual content from the perspective of the foveal-tracking virtual camera, and project light representing the rendering of the virtual content through eyepiece 400 onto the fovea of ​​viewer's eye 210. As shown in Figures 28A-28B, such virtual content may include 3-D virtual objects 6011, 6012, and 6013. 28A-28B may simultaneously perform one or more of the operations described immediately above with respect to a head-tracking rendering viewpoint and one or more of the operations described immediately above with respect to a foveated-tracking rendering viewpoint. In other embodiments, the AR system of FIGS. 28A-28B may perform such operations in rapid succession.

[0349] In this example, the FOV of the head-tracking rendering viewpoint adopted by the AR system in Figures 28A-28B may be a sufficiently wide angular space to encompass each of virtual objects 6011, 6012, and 6013 diagonally, horizontally, and / or vertically. For purposes of the example, the viewer's head position and orientation may be treated as static throughout the first and second phases as depicted in Figures 28A and 28B, respectively, such that the position and orientation of the head-tracking rendering viewpoint remains the same throughout the two phases. For the FOV of the head-tracking rendering viewpoint adopted by the AR system to be wide enough to encompass virtual objects 6011-6013, it must span at least α+ζ angular units diagonally, horizontally, and / or vertically. More specifically, in the example of Figures 28A-28B, it can be seen that virtual objects 6011, 6012, and 6013 can span α-β, γ+δ, and ζ-ε angular unit ranges, respectively.

[0350] In FIG. 28A , the viewer's eye 210 is oriented in a first manner relative to the eyepiece 6000 such that the viewer may be able to view the eyepiece 6000 in a relatively straight-on direction. The orientation of the viewer's eye 210 in FIG. 28A may be the same as or similar to the orientation of the viewer's eye 210 as described above with reference to FIGS. 26A-26B , for example, and may be determined by the AR system using sensing components and / or one or more of the techniques described herein. Thus, at the stage depicted in FIG. 28A , the AR system may adopt a head-tracking and foveated-tracking rendering viewpoint at a relative position and orientation similar to that of the head-tracking and foveated-tracking rendering viewpoints 5010 and 5020A, respectively. In the particular example of FIG. 28A , the FOV of the foveated-tracking rendering viewpoint adopted by the AR system may encompass virtual object 6012, but not either of virtual objects 6011 and 6013, for example. 28A , it follows that the AR system may render virtual object 6012 in high resolution as it would be captured from the perspective of the foveal-tracking virtual camera, and may render virtual objects 6011 and 6013 in lower resolution as they would be captured from the perspective of the head-tracking virtual camera. In addition, the AR system may project light representing such renderings of virtual objects 6011, 6012, and 6013 through eyepiece 6000 onto the retina of viewer's eye 210. In some embodiments, the AR system may also render virtual object 6012 in lower resolution as it would be captured from the perspective of the head-tracking virtual camera.

[0351] 28A also illustrates an example light field 6030A that is outcoupled by the eyepiece 6000 and projected onto the retina of the viewer's eye 210. The light field 6030A may include various angular light components that represent one or more of the renderings of the virtual objects 6011, 6012, and 6013 described above. For example, the angular light components of the light field 6030A that represent the virtual object 6011 as it would be captured from the perspective of a head-tracking virtual camera may include those that would be projected onto the retina of the viewer's eye 210 at angles ranging from −α to −β angular units relative to the viewer's eye 210, and the angular light components of the light field 6030A that represent the virtual object 6013 as it would be captured from the perspective of a head-tracking virtual camera may include those that would be projected onto the retina of the viewer's eye 210 at angles ranging from ε to ζ angular units relative to the viewer's eye 210. Similarly, the angular light components of the light field 6030A representing the virtual object 6012 as would be captured from the perspective of the foveal tracking virtual camera may include those that would be projected onto the fovea of ​​the viewer's eye 210 at angles ranging from −γ to δ angular units relative to the viewer's eye 210. Thus, the components of the light field 6030A representing the virtual object 6012 (i.e., components that would be projected at angles ranging from −γ to δ angular units relative to the viewer's eye 210) may be more densely distributed in angular space than the components of the light field 6030A representing the virtual objects 6011 or 6013 (i.e., components that would be projected at angles ranging from −α to −β or ε to ζ angular units relative to the viewer's eye 210). In this way, the resolution at which the virtual object 6012 may be rendered and presented to the viewer may be higher than the resolution at which the virtual objects 6011 or 6013 may be rendered and presented to the viewer.

[0352] In FIG. 28B , the viewer's eye 210 is oriented relative to the eyepiece 6000 in a second manner that differs from the first manner in which the viewer's eye 210 is oriented relative to the eyepiece 6000 in FIG. 28A . The orientation of the viewer's eye 210 in FIG. 28B may be the same as or similar to the orientation of the viewer's eye 210 as described above with reference to FIGS. 26C-26D , for example, and may be determined by the AR system using sensing components and / or one or more of the techniques described herein. Thus, at the stage depicted in FIG. 28B , the AR system may adopt a head-tracking and foveated-tracking rendering viewpoint at a relative position and orientation similar to those of the head-tracking and foveated-tracking rendering viewpoints 5010 and 5020C, respectively. In the particular example of FIG. 28B , the FOV of the foveated-tracking rendering viewpoint adopted by the AR system may encompass, for example, virtual object 6013 but not either of virtual objects 6011 and 6012. 28B , it follows that the AR system may render virtual object 6013 in high resolution as it would be captured from the perspective of the foveal-tracking virtual camera, and may render virtual objects 6011 and 6012 in lower resolution as they would be captured from the perspective of the head-tracking virtual camera. In addition, the AR system may project light representing such renderings of virtual objects 6011, 6012, and 6013 through eyepiece 6000 onto the retina of viewer's eye 210. In some embodiments, the AR system may also render virtual object 6013 in lower resolution as it would be captured from the perspective of the head-tracking virtual camera.

[0353] 28B also illustrates an example light field 6030B that is outcombined by the eyepiece 6000 and projected onto the retina of the viewer's eye 210. The light field 6030B may include various angular light components that represent one or more of the renderings of the virtual objects 6011, 6012, and 6013 described above. For example, the angular light components of the light field 6030B representing the virtual object 6011 as it would be captured from the perspective of the head-tracking virtual camera may include those that would be projected onto the retina of the viewer's eye 210 at angles ranging from −α to −β angular units relative to the viewer's eye 210, and the angular light components of the light field 6030B representing the virtual object 6012 as it would be captured from the perspective of the head-tracking virtual camera may include those that would be projected onto the retina of the viewer's eye 210 at angles ranging from −γ to δ angular units relative to the viewer's eye 210. Similarly, the angular light components of light field 6030B, representing virtual object 6013 as would be captured from the perspective of a foveal tracking virtual camera, may include those that would be projected onto the fovea of ​​the viewer's eye 210 at angles ranging from ε to ζ angular units relative to the viewer's eye 210. Thus, the components of light field 6030B representing virtual object 6013 (i.e., components that would be projected at angles ranging from ε to ζ angular units relative to the viewer's eye 210) may be more densely distributed in angular space than the components of light field 6030A representing virtual objects 6011 or 6012 (i.e., components that would be projected at angles ranging from −α to −β or −γ to δ angular units relative to the viewer's eye 210). In this way, the resolution at which virtual object 6013 may be rendered and presented to the viewer may be higher than the resolution at which virtual objects 6011 or 6012 may be rendered and presented to the viewer. In fact, from the stage of FIG. 28A to the stage of FIG. 28B, the AR system described in this specification with reference thereto effectively reorients the viewpoint from which the virtual content can be viewed in high resolution, in accordance with the change in line of sight of the viewer's eye 402 between the stages.

[0354] 28C-28F illustrate some of the principles described in FIGS. 3E-3F using some example images that may be presented to a user, according to some embodiments. In some examples, one or more of the images and / or image streams depicted in FIGS. 28C-28F may represent two-dimensional images or portions thereof that would be displayed at a particular depth plane, such as one or more of the depth planes described above with reference to FIG. 25B. That is, such images and / or image streams may represent 3-D virtual content projected onto at least one two-dimensional surface at a fixed distance from the user. It should be understood that in such examples, such images and / or image streams may be presented to the user as one or more light fields with an angular field of view similar to those described above with reference to FIGS. 26A-26D and 28A-28B.

[0355] As depicted, the first image stream 6010 includes a tree. During a first time period represented by FIG. 28C, the eye tracking sensor can determine that the user's eye line of sight (i.e., foveal vision) is focused within a first region 6010-1 of the tree, including the tree trunk. In response to determining that the user's eye line of sight is focused on the first region 6010-1, a second image stream 6020, including a high-resolution image associated with the first region 6010-1 of the first image stream 6010, can be positioned within the first region 410-1 concurrently with the display of the first image stream 6010. The first image stream 410 may have a lower resolution than the second image stream 6020, as illustrated in FIG. 28C.

[0356] During a second time period, represented by FIG. 28D, the eye tracking sensor can determine that the user's line of sight has moved to a second region 6010-2 of the tree, including the tree branches, as illustrated in FIG. 28D. Thus, the second image stream 420 can be shifted to the second region 6010-2, changing its content to correspond to the content in the second region 6010-2 of the first image stream 6010. Because the higher-resolution second image stream 6020 overlays the portion of the first image stream 6010 within the user's foveal vision, the lower-resolution portions of the first image stream 6010 surrounding the second image stream 6020 cannot be perceived or noticed by the user. In this way, the user can perceive a combination of the first image stream 6010 and the second image stream 6020 as having both a wide FOV and high resolution. Such a display system can provide several advantages. For example, the display system can maintain a relatively small form factor and relatively low computational resource requirements while providing a superior user experience. The small form factor and low computational resource requirements can result from the device only needing to generate high-resolution images within a limited area of ​​the display.

[0357] The second image stream 6020 can be overlaid on the first image stream 6010 simultaneously or in rapid succession. As discussed above, in some embodiments, the subset of content of the first image stream 6010 overlaid by the second image stream 6020 can be turned off or presented at a lower intensity for a more uniform brightness and better resolution perception. Also, note that in some embodiments, the second image stream associated with the second image stream 6020 may differ in other respects from the first image stream associated with the first image stream 6010. For example, the color resolution of the second image stream may be higher than the color resolution of the first image stream. The refresh rate of the second image stream may also be higher than the refresh rate of the first image stream.

[0358] FIG. 28E illustrates an exemplary high FOV low-resolution image frame (i.e., a first image stream), and FIG. 28F illustrates an exemplary low FOV high-resolution image frame (i.e., a second image stream), according to some embodiments. As illustrated in FIG. 28E, a region 6030 of the high FOV low-resolution image frame that would be overlaid by the low FOV high-resolution image frame may lack virtual content. By omitting the portion of the high FOV image corresponding to region 6030, any image blurring or smearing resulting from slight differences in the two images may be avoided. The content of the low FOV high-resolution image frame (e.g., as illustrated in FIG. 28F) may include a high-resolution version of the content corresponding to region 6030.

[0359] 29A shows a simplified block diagram of a display system 7000A according to some embodiments. The display system 7000A can include one or more sensors 7002 for detecting the position and movement of a user's head and the position and interocular distance of the user. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and the like. In an augmented reality system, the one or more sensors 7002 can be mounted on a head-mounted frame.

[0360] For example, in some implementations, one or more sensors 7002 of display system 7000A may be part of a head-mounted transducer system that includes one or more inertial transducers to capture inertial measurements indicative of movement of a user's head. Thus, in these implementations, one or more sensors 7002 may be used to sense, measure, or collect information about the user's head movement. For example, such may be used to detect the measured movement, velocity, acceleration, and / or position of the user's head.

[0361] In some embodiments, the one or more sensors 7002 may include one or more forward-facing cameras that can be used to capture information about the environment in which the user is located. The forward-facing cameras may be used to capture information indicative of the user's distance and orientation relative to the environment and specific objects within that environment. When worn on the head, the forward-facing cameras are particularly well-suited for capturing information indicative of the environment in which the user is located and the distance and orientation of the user's head relative to specific objects within that environment. The forward-facing cameras can be employed to detect head movement, speed of head movement, and acceleration. The forward-facing cameras can also be employed to detect or infer the user's center of attention, for example, based at least in part on the orientation of the user's head. Orientation may be detected in any direction (e.g., up, down, left, right, relative to the user's frame of reference).

[0362] The one or more sensors 7002 may also include a pair of rear-facing cameras to track the user's eye movements, blinks, and depth of focus. Such eye-tracking information may be determined, for example, by projecting light onto the user's eye and detecting a return or reflection of at least a portion of the projected light. Further details discussing eye tracking devices are provided in U.S. Provisional Patent Application No. 61 / 801,219, entitled "DISPLAY SYSTEM AND METHOD," U.S. Provisional Patent Application No. 62 / 005,834, entitled "METHODS AND SYSTEM FOR CREATING FOCAL PLANES IN VIRTUAL AND AUGMENTED REALITY," U.S. Provisional Patent Application No. 61 / 776,771, entitled "SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY," and U.S. Provisional Patent Application No. 62 / 420,292, entitled "METHOD AND SYSTEM FOR EYE TRACKING USING SPECKLE PATTERNS," which are expressly incorporated herein by reference.

[0363] The display system 7000A may further include a user orientation determination module 7004 communicatively coupled to the one or more sensors 7002. The user orientation determination module 7004 receives data from the one or more sensors 7002 and uses such data to determine the user's head pose, corneal position, interpupillary distance, and the like. The user orientation determination module 7004 may detect the instantaneous position of the user's head and predict the position of the user's head based on the position data received from the one or more sensors 7002. The user orientation determination module 7004 also tracks the user's eyes based on the tracking data received from the one or more sensors 7002.

[0364] The display system 7000A may further include a control subsystem, which may take any of a wide variety of forms, including several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), e.g., field programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs).

[0365] 29A , display system 7000A includes a central processing unit (CPU) 7010, a graphics processing unit (GPU) 7020, and frame buffers 7042 and 7044. In brief, as described in further detail below, CPU 7010 controls overall operation, while GPU 7020 renders frames from three-dimensional data stored in database 7030 (i.e., converts three-dimensional scenes into two-dimensional images) and stores these frames in frame buffers 7042 and 7044. Although not shown, one or more additional integrated circuits may control the loading of frames into and / or from frame buffers 7042 and 7044 and the operation of one or more other components of display system 7000A, such as image multiplexing subsystem 7060, foveated tracking beam-steering component 7080, and the like. The loading of frames into and / or from frame buffers 542 and 544 may employ dynamic addressing, e.g., frames may be over-rendered. Display system 7000A further includes read-only memory (ROM) and random access memory (RAM). Display system 7000A further includes a 3D database 7030, from which GPU 7020 can access 3D data of one or more scenes for rendering frames.

[0366] The CPU 7010 may include a high FOV low resolution rendering viewpoint determination module 7012 and a low FOV high resolution rendering viewpoint determination module 7014. In some embodiments, the user orientation determination module 7004 may be part of the CPU 7010.

[0367] The high FOV low-resolution rendering viewpoint determination module 7012 may include logic for mapping the data output by the user orientation determination module to a location and angle in 3D space from which the high FOV low-resolution image will be perceived. That is, the CPU 7010 determines, at any given time, the viewpoint of a virtual camera that is fixed relative to the user's head based on the data received from the user orientation determination module 7004. Within the context of the example described above with reference to Figures 26A-26D and 28A-28B, the high FOV low-resolution rendering viewpoint determination module 7012 may be responsible for monitoring the head position and orientation as indicated by the user orientation determination module 7004 and, as appropriate, controlling the position and orientation of at least the head-tracked virtual camera in the rendering space.

[0368] The low FOV high-resolution rendering viewpoint determination module 7014 may include logic for mapping data output by the user orientation determination module (e.g., data indicating the user's line of sight and foveal location) to a location and angle in 3D space from which the low FOV high-resolution image will be perceived. That is, the CPU 7010 determines, at any given time, the viewpoint of a virtual camera that is fixed relative to the user's fovea based on the data received from the user orientation determination module 7004. Within the context of the examples described above with reference to Figures 26A-26D and 28A-28B, the low FOV high-resolution rendering viewpoint determination module 7014 may be responsible for monitoring the eye line of sight as indicated by the user orientation determination module 7004 and, as appropriate, controlling the position and orientation of at least the foveal-tracking virtual camera in the rendering space.

[0369] The display system 7000A may further include a graphics processing unit (GPU) 7020 and a database 7030. The database 7030 may store 3D virtual content. The GPU 7020 may access the 3D virtual content stored in the database 7030 to render frames. The GPU 7020 may render frames of the virtual content at a low FOV and high resolution from a virtual camera viewpoint fixed relative to the user's fovea (e.g., a foveated tracking rendering viewpoint), as determined by the CPU 7010 and provided as output. The GPU 7020 may also render frames of the virtual content at a high FOV and low resolution from a virtual camera viewpoint fixed relative to the user's head (e.g., a head-tracking / non-foveated viewpoint), as determined by the CPU 7010 and provided as output. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process are provided in U.S. patent application Ser. No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION," which is expressly incorporated by reference in its entirety for all purposes.

[0370] Frames of virtual content rendered in a high FOV, low resolution may be stored in a buffer of frames rendered in a high FOV, low resolution 7042. Similarly, frames of virtual content rendered in a low FOV, high resolution may be stored in a buffer of frames rendered in a low FOV, high resolution 7044. In some embodiments, the buffer of frames rendered in a high FOV, low resolution 7042 and the buffer of frames rendered in a low FOV, high resolution 7044 may be part of the GPU 7020.

[0371] The display system 7000A may further include an image multiplexing subsystem 7060 and an image multiplexing subsystem controller 7050 communicatively coupled to the image multiplexing subsystem 7060. The image multiplexing subsystem 7060 may include an image source 7062 and a multiplexing component 7064 for multiplexing the high FOV low-resolution image frames and the low FOV high-resolution image frames, substantially as described in further detail below with reference to FIGS. 30A-30B . The image source 7062 may include, for example, a light source in combination with a fiber scanning component, liquid crystal on silicon (LCoS), a MEMS scanning mirror, and the like. The multiplexing component 7064 may include optical elements such as a polarization rotator, switchable optics, a liquid crystal array, a variable focus lens, and the like. The multiplexing component 7064 may be internal or external to the image source 7062.

[0372] The image multiplexing subsystem controller 7050 is communicatively coupled to the image multiplexing subsystem 7060, the buffer of frames rendered at high FOV low resolution 7042, and the buffer of frames rendered at low FOV high resolution 7044. The control circuitry can send control signals to the image source 562 so that appropriate image content is presented from each rendering viewpoint, as discussed above. The image multiplexing subsystem controller 7050 can also control the multiplexing component 7064 in conjunction with the image source 7062 to provide a multiplexed image stream.

[0373] The display system 7000A may further include a foveal tracking beam-steering component 7080 and a foveal tracking controller 7070 communicatively and / or operably coupled to the foveal tracking beam-steering component 7080. The foveal tracking controller 7070 may receive output data from the CPU 7010 regarding the position of the user's fovea (e.g., as determined by the low FOV high-resolution rendering viewpoint determination module 7014 and / or the user orientation determination module 7004) and use such data to control the position of the foveal tracking beam-steering component 7080. The foveal tracking beam-steering component 7080 may serve to dynamically steer or otherwise direct the low FOV high-resolution portion of the multiplexed image stream (produced by the image source 7062 and the multiplexing component 7064) toward the user's fovea. Such a low FOV high-resolution portion of the image stream may represent, for example, virtual content as would be captured from the perspective of a foveal tracking virtual camera.

[0374] Display system 7000A may also include a storage medium for storing computer-readable instructions, databases, and other information usable by CPU 7010, GPU 7020, and / or one or more other modules or controllers of display system 7000A. Display system 7000A may further include an input-output (I / O) interface, such as buttons, that a user may use to interact with the display system. Display system 7000A may also include a wireless antenna for wireless communication with another part of display system 7000A or the Internet.

[0375] FIG. 29B schematically illustrates a cross-sectional view of an AR system 7000B, according to some embodiments. According to some embodiments, the AR system 7000B can incorporate at least some of the components of the display system 7000A, as described above with reference to FIG. 29A, and can be fitted into one of the displays 4052 in the wearable display device 4050, as shown in FIG. 25A. For example, the AR system 7000B can include an image multiplexing subsystem 560, which can include an image source 7062 and one or more multiplexing components. In addition, the AR system 7000B can also include a foveal tracking beam-steering component 7080, which in this example can be an electromechanical optical device such as a MEMS scanning mirror. Like the display system 7000A, the image multiplexing subsystem 7060 can be communicatively and / or operably coupled to an image multiplexing subsystem controller, and the foveal tracking beam-steering component 7080 can be communicatively and / or operably coupled to a foveal tracking controller. The AR system 7000B may further include one or more internal coupling gratings (ICGs) 7007 and one or more eyepieces 7008. Each internal coupling grating 7007 may be configured to couple the first and second light beams into a respective eyepiece 7008. Each eyepiece 7008 may include an external coupling grating for externally coupling the first and second light beams into a user's eye. The internal coupling gratings 7007 and the eyepieces 7008 may be referred to herein as a "viewing assembly." It should be understood that the various internal coupling gratings (ICGs) disclosed herein may correspond to the internal coupling optical elements 700, 710, 720 of FIGS. 9A-9C .

[0376] 30A-30B schematically illustrate a display system 8000 for projecting images to a user's eyes, according to some embodiments. The display system 8000 includes an image source 8010. The image source 8010 can be configured to project a first light beam 8052 associated with a first image stream, as shown in FIG. 30A, and a second light beam 8054 associated with a second image stream, as shown in FIG. 30B. Note that the first light beam 8052 and the second light beam 8054 are depicted in FIGS. 30A-30B as schematic rays, which are not intended to represent precise ray tracings. The first light beam 8052 can be angularly expanded to cover a wider FOV and result in a lower angular resolution image stream. The second light beam 8054 may have a narrower FOV with higher angular resolution, as discussed above with reference to Figures 26A-26F and 28A-28D.

[0377] Image source 8010, according to various embodiments, may include a liquid crystal on silicon (LCoS or LCOS) display (which may also be referred to as a spatial light modulator), a scanning fiber, or a scanning mirror. For example, image source 8010 may include a scanning device that scans an optical fiber in a predetermined pattern in response to a control signal. The predetermined pattern may correspond to a desired image shape, such as a rectangular or circular shape.

[0378] According to some embodiments, a first light beam 8052 associated with a first image stream and a second light beam 8054 associated with a second image stream may be multiplexed and output as a combined light beam by the image source 8010. For example, polarization division multiplexing, time division multiplexing, wavelength division multiplexing, and the like may be used to multiplex the light beam associated with the first image stream and the light beam associated with the second image stream.

[0379] In embodiments in which polarization division multiplexing is used, the first light beam 8052 can be in a first polarization state and the second light beam 8054 can be in a second polarization state that is different from the first polarization state. For example, the first polarization state can be linearly polarized light oriented ...

Claims

1. A wearable display system, comprising: a display configured to output light at multiple depths to present virtual content to a user of the wearable display system; one or more processors communicatively coupled to the display; one or more computer-readable storage media storing instructions; the instructions, when executed by the one or more processors, cause the one or more processors to: determining a three-dimensional (3D) fixation point of the user's eye; determining a 3D virtual object position at which a virtual object should be presented within a field of view (FOV) of the user via the display; determining an angular distance of the virtual object position from the fixation point and a depth difference of the virtual object position from the fixation point; determining a particular resolution at which to render the virtual object based at least in part on a location of the angular distance and the depth disparity on a resolution distribution that specifies the resolution as a function of the angular distance and the depth disparity such that the resolution varies with depth disparity for a particular angular distance, the resolution distribution including: a) a high-resolution foveal zone proximate the fixation point; b) a low-resolution zone surrounding the foveal zone; and c) a roll-off providing a continuous drop-off in resolution between the foveal zone and the low-resolution zone; causing the display to present the virtual object at the virtual object location and at the particular resolution; A wearable display system that performs an operation including:

2. The wearable display system of claim 1, wherein the resolution distribution is a multivariate normal distribution.

3. The wearable display system of claim 1, wherein the resolution distribution provides that the particular resolution decreases as the depth difference increases.

4. The wearable display system of claim 1, wherein the roll-off is a Gaussian roll-off.

5. The wearable display system of claim 1, wherein resolution is substantially constant with angular distance within the foveal zone for a particular depth difference.

6. The operation further comprises: determining a type of the virtual object; adjusting at least one of the shape of the roll-off or the angular distance encompassed by the foveal zone by modifying the resolution distribution based on the type of the virtual object; The wearable display system of claim 1 .

7. The wearable display system of claim 1, wherein input from the user is employed to adjust at least one of the angular distance encompassed by the foveal zone or the shape of the roll-off by modifying the resolution distribution.

8. The wearable display system of claim 1, wherein detected eye movements of the user are employed to adjust at least one of the angular distance encompassed by the foveal zone or the shape of the roll-off by modifying the resolution distribution.

9. The wearable display system of claim 8, wherein the detected velocity of the user's eye saccades is used to adjust the angular distance encompassed by the foveal zone.

10. A computer-readable storage medium storing instructions that, when executed by at least one processor, direct the at least one processor to perform operations for presenting virtual content through a wearable display system, the operations comprising: determining a three-dimensional (3D) fixation point of an eye of a user of the wearable display system, the wearable display system comprising a display configured to output light at multiple depths to present the virtual content to the user; determining a 3D virtual object position at which a virtual object should be presented within a field of view (FOV) of the user via the display; determining an angular distance of the virtual object position from the fixation point and a depth difference of the virtual object position from the fixation point; determining a particular resolution at which to render the virtual object based at least in part on a location of the angular distance and the depth disparity on a resolution distribution that specifies the resolution as a function of the angular distance and the depth disparity such that the resolution varies with depth disparity for a particular angular distance, the resolution distribution including: a) a high-resolution foveal zone proximate the fixation point; b) a low-resolution zone surrounding the foveal zone; and c) a roll-off providing a continuous drop-off in resolution between the foveal zone and the low-resolution zone; causing the display to present the virtual object at the virtual object location and at the particular resolution; 23. A computer-readable storage medium comprising:

11. A method for presenting virtual content through a wearable display system, the method being computer-implemented, the method comprising: determining a three-dimensional (3D) fixation point of an eye of a user of the wearable display system, the wearable display system comprising a display configured to output light at multiple depths to present the virtual content to the user; determining a 3D virtual object position at which a virtual object should be presented within a field of view (FOV) of the user via the display; determining an angular distance of the virtual object position from the fixation point and a depth difference of the virtual object position from the fixation point; determining a particular resolution at which to render the virtual object based at least in part on a location of the angular distance and the depth disparity on a resolution distribution that specifies the resolution as a function of the angular distance and the depth disparity such that the resolution varies with depth disparity for a particular angular distance, the resolution distribution including: a) a high-resolution foveal zone proximate the fixation point; b) a low-resolution zone surrounding the foveal zone; and c) a roll-off providing a continuous drop-off in resolution between the foveal zone and the low-resolution zone; causing the display to present the virtual object at the virtual object location and at the particular resolution; A method comprising:

12. The method of claim 11, wherein the resolution distribution is a multivariate normal distribution.

13. The method described in claim 11, wherein the resolution distribution provides that the particular resolution decreases as the depth difference increases.

14. The method of claim 11, wherein the roll-off is a Gaussian roll-off.

15. The method described in claim 11, wherein resolution is substantially constant with respect to angular distance within the foveal zone for a particular depth difference.

16. The operation further comprises: determining a type of the virtual object; adjusting at least one of the shape of the roll-off or the angular distance encompassed by the foveal zone by modifying the resolution distribution based on the type of the virtual object; The method of claim 11 , comprising:

17. The method described in claim 11, wherein input from the user is employed to adjust at least one of the angular distance encompassed by the foveal zone or the shape of the roll-off by modifying the resolution distribution.

18. The method of claim 11, wherein detected eye movements of the user are employed to adjust at least one of the angular distance encompassed by the foveal zone or the shape of the roll-off by modifying the resolution distribution.

19. The method described in claim 18, wherein the detected velocity of the user's eye saccades is used to adjust the angular distance encompassed by the foveal zone.

Images