Depth-based foveated rendering for display system

Abstract

To provide suitable depth-based foveated rendering for a display system.SOLUTION: A display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes by using different wavefront divergence. Some embodiments include monitoring eye orientations of a user of a display system based on detected sensor information. A fixation point is determined based on the eye orientations, and the fixation point represents a three-dimensional location with respect to a field of view. Location information on virtual objects to present is obtained, and the location information indicates three-dimensional positions of the virtual objects. The resolution of at least one virtual object is adjusted based on the degree of proximity of the at least one virtual object to the fixation point. The virtual objects are presented to the user by the display system, and the at least one virtual object is rendered according to the adjusted resolution.SELECTED DRAWING: Figure 10A

Description

[Technical field]

[0001] (Priority Claim) This application claims priority to U.S. Provisional Application No. 62 / 644,365, filed March 16, 2018, U.S. Provisional Application No. 62 / 475,012, filed March 22, 2017, U.S. Provisional Application No. 62 / 486,407, filed April 17, 2017, and U.S. Provisional Application No. 62 / 539,934, filed August 1, 2017. The above patent applications are incorporated herein by reference in their entireties for all purposes.

[0002] (Incorporated by reference) This application incorporates by reference the entirety of each of the following patent applications and publications: U.S. Application Serial No. 14 / 555,585, filed November 27, 2014, published July 23, 2015 as U.S. Publication No. 2015 / 0205126; U.S. Application Serial No. 14 / 690,401, filed April 18, 2015; No. 9,417,452, filed on August 16, 2016; U.S. Application No. 14 / 331,218, filed on July 14, 2014, published on October 29, 2015 as U.S. Publication No. 2015 / 0309263; U.S. Application No. 15 / 902,927, filed on February 22, 2018; U.S. Provisional Application No. 62 / 475,012, filed on March 22, 2017; and U.S. Provisional Application No. 62 / 539,934, filed on August 1, 2017.

[0003] The present disclosure relates to display systems, including augmented reality imaging and visualization systems. [Background technology]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, while augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality or "MR" scenarios are a type of AR scenario that typically involve virtual objects that are integrated into and responsive to the natural world. For example, MR scenarios may include AR image content that appears blocked by or is otherwise perceived to interact with objects in the real world.

[0005] Referring to FIG. 1, an augmented reality scene 10 is depicted. A user of the AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. The user also perceives that they are "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30 and a flying cartoon-like avatar character 50 that appears to be an anthropomorphic bumblebee. These elements 50, 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex, making it difficult to produce AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] The systems and methods disclosed herein address various challenges associated with AR or VR technology. Summary of the Invention [Means for solving the problem]

[0007] According to some embodiments, a system includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include monitoring eye movements of a user based on information detected via one or more sensors. A fixation point at which an eye of the user is fixated is determined based on the eye movements, the fixation point being a three-dimensional location within the user's field of view. The operations include obtaining location information associated with one or more virtual objects for presentation to the user, the location information indicating a three-dimensional location of the virtual objects. The operations also include adjusting a resolution of the at least one virtual object based, at least in part, on a proximity of the at least one virtual object to the fixation point. The operations also include providing a presentation of the virtual object to the user via the display, the at least one virtual object being rendered according to the adjusted resolution.

[0008] According to some embodiments, a display system includes a display device configured to present virtual content to a user, one or more processors, and one or more computer storage media storing instructions that, when executed by the system, cause the system to perform operations. The operations include monitoring information associated with eye movements of a user. A fixation point within a display frustum of the display device is determined based on the monitored information, the fixation point indicating a three-dimensional location fixated by an eye of the user. The operations also include presenting the virtual content at a three-dimensional location within the display frustum based on the determined fixation point, the virtual content being adjusted in resolution based on a proximity of the virtual content from the fixation point.

[0009] According to some other embodiments, the method includes monitoring an eye orientation of a user of a display device based on information detected via one or more sensors. A fixation point at which an eye of the user is fixated is determined based on the eye orientation, the fixation point being a three-dimensional location within the user's field of view. Location information associated with one or more virtual objects for presentation to the user is obtained, the location information indicating a three-dimensional location of the virtual objects. A resolution of the at least one virtual object is adjusted based, at least in part, on a proximity of the at least one virtual object to the fixation point. The method also includes providing a presentation of the virtual objects to the user via the display, the at least one virtual object being rendered in accordance with the adjusted resolution.

[0010] According to some embodiments, the display system includes a frame configured to be mounted on a user's head, a light modulation system configured to output light and form an image, and one or more waveguides attached to the frame and configured to receive light from the light modulation system and output light across a surface of the one or more waveguides. The system also includes one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The operations include determining an amount of light reaching a retina of a user's eye and adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina.

[0011] According to some other embodiments, a display system includes one or more processors and one or more computer storage media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations, including determining an amount of light that reaches a retina of an eye of a user of the display system and adjusting a resolution of virtual content to be presented to the user based on the amount of light that reaches the retina.

[0012] According to some embodiments, the method is performed by a display system comprising one or more processors and a head-mountable display, the method including determining an amount of light reaching a retina of an eye of a user of the display system and adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina.

[0013] According to some other embodiments, a display system includes a frame configured to be mounted on a user's head, a light modulation system, one or more waveguides, one or more processors, and one or more computer storage media storing instructions. The light modulation system is configured to output light and form an image. The one or more waveguides are attached to the frame and configured to receive light from the light modulation system and output light across a surface of the one or more waveguides. The one or more computer storage media store instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The operations include adjusting a resolution of the primary color images forming the virtual content based on a proximity of the virtual content from a user fixation point and a color of the primary color images. At least one of the primary color images has a different resolution than another color primary color image.

[0014] According to yet another embodiment, a display system includes one or more processors and one or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform various operations, including adjusting a resolution of primary color images forming the virtual content based on a proximity of the virtual content from a user fixation point and a color of the primary color images, at least one of the primary color images having a different resolution than another color primary color image.

[0015] According to some other embodiments, the method is implemented by a display system comprising one or more processors and a head-mountable display, the method including adjusting a resolution of primary color images forming the virtual content based on a proximity of the virtual content from a user fixation point and colors of the primary color images, at least one of the primary color images having a different resolution than another color primary color image.

[0016] According to yet another embodiment, a display system includes an image source including a spatial light modulator for providing a first image stream and a second image stream, a viewing assembly, one or more processors in communication with the image source, and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations. The viewing assembly includes light guidance optics for receiving the first and second image streams from the image source and outputting the first and second image streams to a user. The various operations performed by the one or more processors include causing the image source to output a first image stream to the viewing assembly, where an image formed by the first image stream has a first pixel density, and causing the image source to output a second image stream to the viewing assembly. The image formed by the second image stream has a second pixel density that is greater than the first pixel density and corresponds to a portion of the image provided by the first image stream. The image formed by the second image stream covers a corresponding portion of the field of view provided by the first image stream.

[0017] According to some embodiments, a wearable display system may include an afocal magnification lens with circular polarization handedness-dependent magnification. The afocal magnification lens may include a first fixed focal length lens element, a first geometric phase lens that exhibits positive refractive power for a first handedness of incident circularly polarized light and a second geometric phase lens that exhibits negative refractive power for a second handedness of incident circularly polarized light.

[0018] According to some other embodiments, an optical subsystem for a wearable image projector may include a polarization-selective reflector and a set of four lens elements centered around the polarization-selective reflector.

[0019] A display system for projecting an image to a user's eye, according to some other embodiments, may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include a first image source configured to project a first light beam associated with a first image stream. The first image stream may have a first field of view and may be incident on a first surface of the internal coupling grating. A portion of the first light beam may be coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to the user's eye. The display system may further include a second image source configured to project a second light beam associated with a second image stream. The second image stream may have a second field of view that is narrower than the first field of view. The display system may further include a scanning mirror and may be configured to receive and reflect the second light beam such that the second light beam is incident on a second surface of the internal coupling grating opposite the first surface. A portion of the second light beam may be coupled into the waveguide by the internal coupling grating. The display system may further include an eye gaze tracker configured to detect movements of the user's eye, and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movements of the user's eye.

[0020] According to some other embodiments, a display system for projecting an image to an eye of a user may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view and the second image stream may have a second field of view that is narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system may further include a polarizing beam splitter configured to receive and reflect the first light beam along a first optical path and receive and transmit the second light beam along a second optical path. The display system may further include a first optical reflector positioned along the first optical path and configured to receive and reflect the first light beam so that the first light beam is incident on a first surface of the internal coupling grating. A portion of the first light beam may be coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to the user's eye. The display system may further include a scanning mirror positioned along the second optical path and configured to receive and reflect the second light beam, and a second optical reflector positioned along the second optical path downstream of the scanning mirror. The second optical reflector may be configured to receive and reflect the second light beam so that the second light beam is incident on a second surface of the internal coupling grating opposite the first surface. A portion of the second light beam may be coupled into the waveguide by the internal coupling grating. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuitry may be configured to position the scanning mirror such that the position of the second image stream is moved according to detected movements of the user's eye.

[0021] According to some other embodiments, a display system for projecting an image to an eye of a user may include a waveguide and an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view and the second image stream has a second field of view narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system may further include a polarizing beam splitter configured to receive and reflect the first light beam along a first optical path and receive and transmit the second light beam along a second optical path. The display system may further include a first incoupling prism positioned adjacent to a first surface of the waveguide along the first optical path. The first incoupling prism may be configured to couple a portion of the first light beam into the waveguide to position the first image stream at a fixed position relative to the eye of the user. The display system may further include a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam. The display system may further include a second in-coupling prism positioned along the second optical path downstream of the scanning mirror and adjacent a second surface of the waveguide opposite the first surface of the waveguide. The second in-coupling prism may be configured to couple a portion of the second light beam into the waveguide. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuitry may be configured to position the scanning mirror such that a position of the second image stream is moved according to the detected movement of the user's eye.

[0022] According to an embodiment, a display system for projecting an image to a user's eye includes an image source. The image source can be configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream can have a first field of view and the second image stream can have a second field of view narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system can further include a polarizing beam splitter. The polarizing beam splitter can be configured to receive and reflect the first light beam along a first optical path toward a viewing assembly and receive and transmit the second light beam along a second optical path to position the first image stream at a fixed position relative to the user's eye. The display system can further include a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam toward the viewing assembly. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

[0023] According to another embodiment, a display system for projecting an image to an eye of a user includes an image source. The image source can be configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream. The first image stream can have a first field of view and the second image stream can have a second field of view narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system can further include a scanning mirror configured to receive and reflect the first light beam and the second light beam toward the viewing assembly to project the first image stream and the second image stream. The display system can further include an eye gaze tracker configured to detect movement of the user's eye and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit can be configured to position the scanning mirror such that the position of the first image stream and the position of the second image stream are moved according to the detected movement of the user's eye. The display system can further include a switchable optical element disposed in an optical path of the first light beam and the second light beam. The switchable optical element can be configured to be switched to a first state for a first light beam such that the first light beam is angularly expanded by a first angular expansion factor, and to be switched to a second state for the second light beam such that the second light beam is angularly amplified by a second angular expansion factor that is less than the first angular expansion factor. The present invention provides, for example, the following: (Item 1) 1. A system comprising one or more processors and one or more computer storage media, the one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to: monitoring eye movements of a user based on information detected via one or more sensors; determining a fixation point at which the user's eye is fixated based on the eye movement, the fixation point being a three-dimensional location within the user's field of view; obtaining location information associated with one or more virtual objects for presentation to a user, the location information indicating a three-dimensional position of the virtual objects; adjusting a resolution of at least one virtual object based, at least in part, on a proximity of the at least one virtual object to the fixation point; providing a presentation of the one or more virtual objects to a user via a display device, the at least one virtual object being rendered according to the individual adjusted resolution; and A system that performs an operation including: (Item 2) 2. The system of claim 1, wherein adjusting the resolution of a virtual object includes reducing a presentation quality of the virtual object. (Item 3) 3. The system of claim 2, wherein reducing the presentation quality of the virtual object includes one or more of reducing a polygon count of the virtual object, adjusting primitives utilized to generate the virtual object, adjusting operations performed on the virtual object, adjusting texture information, color resolution, or depth, and adjusting a number of rendering cycles or frame rate. (Item 4) 2. The system of claim 1, wherein the field of view corresponds to a display frustum of the display device. (Item 5) the field of view comprises a plurality of zones, each zone representing a volume of space within the field of view within which a virtual object may be presented by the display device; the zone extends from an initial depth within the field of view to an ending depth within the field of view; a proximity of the one or more virtual objects from the fixation point corresponds to a distance separating a first zone in which the virtual objects will be presented from a second zone in which the fixation point is located. Item 1. The system according to item 1. (Item 6) Item 6. The system of item 5, wherein adjusting the resolution of the at least one virtual object includes reducing the resolution from a resolution initially assigned to the at least one virtual object, and the proximity corresponds to a total number of zones separating the first zone from the second zone. (Item 7) 6. The system of item 5, wherein a second zone in which the fixation point is located is assigned maximum resolution, and the remaining zones are each assigned a resolution reduced from the maximum resolution according to one or more descent rates. (Item 8) The operation further comprises: Providing a user's retinal cone density profile Including, Item 6. The system of item 5, wherein the rate of descent on each depth plane substantially conforms to the rate of descent of cone density in the user's cone density profile. (Item 9) Item 6. The system of item 5, wherein the zones have three-dimensional polygonal shapes. (Item 10) Item 6. The system of item 5, wherein the zones are concentric ellipsoids. (Item 11) Item 6. The system of item 5, wherein the shape of the zone varies with distance to the second zone. (Item 12) The display device further comprises: A plurality of stacked waveguides forming a display area and providing a view of a surrounding environment through the display area, at least some of the plurality of waveguides being configured to output light with a wavefront divergence different from other waveguides. Item 2. The system of item 1, comprising: (Item 13) With respect to a first virtual object to be presented at a greater depth from the fixation point and determined to be within a threshold angular distance of the user's line of sight, the operations further include: adjusting a resolution of the first virtual object based on a proximity of the first virtual object to the fixation point; blurring the first virtual object during presentation of the first virtual object to a user; 2. The system according to item 1, comprising: (Item 14) Blurring the first virtual object includes: performing a convolution of a Gaussian blur kernel, wherein the first virtual object is rendered according to the adjusted resolution; and presenting a result of the convolution to a user via the display device; and Item 14. The system according to item 13, comprising: (Item 15) Item 10. The system of item 1, wherein the one or more sensors comprise one or more of an infrared sensor, an ultraviolet sensor, and a visible wavelength optical imaging device configured to detect a gaze direction of the user's eye. (Item 16) 1. A display system, comprising: a display device configured to present the virtual content to a user; one or more processors; One or more computer storage media having instructions stored thereon that, when executed by the system, cause the system to: monitoring information associated with eye movements of the user; determining a fixation point within a display frustum of the display device based on the monitored information, the fixation point indicating a three-dimensional location fixated by an eye of the user; presenting virtual content at a three-dimensional location within the display frustum based on the determined fixation point, the virtual content having a resolution adjusted based on a proximity of the virtual content from the fixation point; one or more computer storage media for performing operations including: A system comprising: (Item 17) The operation further comprises: providing, by the display device, the determined fixation point to an external system comprising one or more processors; receiving, by the display device, rendered virtual content for presentation from the external system, the virtual content being rendered at a resolution based on a respective proximity of the virtual content from the fixation point; Item 17. The system according to item 16, comprising: (Item 18) Adjusting a resolution of a particular virtual content based on a proximity from the fixation point includes: determining a reduction in maximum resolution based on said proximity, said reduction being based on a rate of descent; and adjusting a resolution of the particular virtual content based on the determined reduction. (Item 19) Item 17. The system of item 16, wherein the display device comprises a plurality of waveguides, each waveguide presenting virtual content associated with a respective depth based on a three-dimensional location of the virtual content. (Item 20) 17. The system of claim 16, wherein the display device presents virtual content based on foveated imaging relative to the fixation point, the foveated imaging incorporating depth information associated with (1) the fixation point and (2) the virtual content. (Item 21) 1. A method comprising: A system of one or more processors monitoring an eye orientation of a user of the display device based on information detected via the one or more sensors; determining a fixation point at which the user's eye is fixated based on the eye orientation, the fixation point representing a three-dimensional location relative to the user's field of view; obtaining location information associated with one or more virtual objects for presentation to a user, the location information indicating a three-dimensional position of the virtual objects; adjusting a resolution of at least one virtual object based, at least in part, on a respective proximity of the at least one virtual object to the fixation point; providing a presentation of the one or more virtual objects to a user via the display device, the at least one virtual object being rendered according to the individual adjusted resolution; and A method comprising: (Item 22) 22. The method of claim 21, wherein adjusting the resolution of a virtual object includes reducing a presentation quality of the virtual object relative to a maximum possible presentation quality. (Item 23) 23. The method of claim 22, wherein reducing the presentation quality of the virtual object includes reducing a polygon count of the virtual object, reducing primitives associated with the virtual object, or reducing texture information associated with the virtual object. (Item 24) 22. The method of claim 21, wherein the fixation point is determined for each frame presented to a user via the display device. (Item 25) the field of view comprises zones, each zone representing a volume of space within the field of view within which a virtual object may be presented; the zone extends from at least an initial depth within the field of view to an ending depth within the field of view; 22. The method of claim 21, wherein the proximity of a virtual object from the fixation point is determined based on a first zone in which the virtual object will be presented relative to a second zone in which the fixation point is located. (Item 26) 22. The method of claim 21, wherein the sensor comprises one or more of an infrared sensor, an ultraviolet sensor, and a visible wavelength optical imaging device. (Item 27) 1. A display system comprising: a frame configured to be mounted on a user's head; a light modulation system configured to output light and form an image; one or more waveguides mounted to the frame and configured to receive light from the light modulation system and output the light across a surface of the one or more waveguides; one or more processors; One or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: determining an amount of light reaching a retina of the user's eye; adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina; one or more computer storage media for performing operations including: A display system comprising: (Item 28) 28. The display system of claim 27, further comprising an outwardly directed camera configured to measure an ambient lighting level, and determining an amount of light reaching the retina includes measuring the ambient lighting level. (Item 29) 30. The display system of claim 28, wherein determining an amount of light reaching the retina further comprises determining an amount of light output by the waveguide to the user's eye. (Item 30) The operation further comprises: determining a user fixation point; obtaining location information of the virtual content, the location information indicating a three-dimensional position of the virtual content; 28. The display system of claim 27, wherein adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on a proximity of the virtual content to the fixation point. (Item 31) Item 31. The display system of item 30, wherein adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on a proximity of the virtual content to the fixation point on a depth axis extending away from the user. (Item 32) 28. The display system of claim 27, wherein the one or more waveguides comprise a stack of waveguides, at least some of the waveguides of the stack of waveguides providing a different amount of wavefront divergence than other waveguides of the stack of waveguides. (Item 33) The one or more waveguides include an internally coupled diffractive optical element; an outcoupling diffractive optical element; a light dispersive element configured to direct light from the in-coupling diffractive optical element to the out-coupling diffractive optical element; 28. The display system of item 27, comprising: (Item 34) 1. A display system comprising: one or more processors; One or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: determining an amount of light reaching a retina of an eye of a user of said display system; adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina; one or more computer storage media for performing operations including: A display system comprising: (Item 35) 35. The display system of claim 34, wherein determining the amount of light reaching the retina includes determining an ambient lighting level. (Item 36) Item 36. The display system of item 35, wherein determining the amount of light that reaches the retina includes determining the amount of light output by the display system to display the virtual content. (Item 37) Item 35. The display system of item 34, wherein determining the amount of light reaching the retina includes determining a size of a pupil of the user's eye. (Item 38) Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes decreasing the resolution of the virtual content as the distance of the virtual content increases from the user's fixation point. (Item 39) Item 39. The display system of item 38, wherein the tendency to reduce resolution is substantially similar to a tendency to reduce cone density in the user's retina. (Item 40) a zone of maximum resolution within the user's field of view is centered over a point corresponding to the fovea of ​​the user's eye; the resolution of the virtual content within the user's field of view is reduced for virtual content located outside of + / - 20° of the point; 40. The display system of item 39. (Item 41) Item 39. The display system of item 38, wherein reducing the resolution includes reducing a polygon count for all levels of light reaching the retina. (Item 42) Item 39. The display system of item 38, wherein adjusting the resolution includes setting the resolution to one of two resolution levels. (Item 43) Item 43. The display system of item 42, wherein adjusting the resolution of the virtual content includes decreasing one or both of color depth and contrast ratio as the amount of light reaching the retina decreases. (Item 44) The operation further comprises: determining whether the amount of light corresponds to a photopic, mesopic, or scotopic illumination level; Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes setting the resolution based on whether the lighting level corresponds to a photopic, mesopic, or scotopic lighting level. (Item 45) Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes displaying the virtual content using a monochrome color at scotopic lighting levels. (Item 46) Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes decreasing a contrast ratio of an image forming the virtual content as the amount of light reaching the retina of the eye decreases. (Item 47) The operation further comprises: Providing virtual content in multiple primary colors Including, Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes providing different resolutions for different primary colors. (Item 48) Item 35. The display system of item 34, wherein the operations further include preventing rendering of virtual content corresponding to the optic nerve blind spot of the eye. (Item 49) Item 35. The display system of item 34, wherein adjusting the resolution of the virtual content includes preventing rendering of virtual content located within an opposite peripheral region for each eye of the user, the opposite peripheral region being on a side of the user opposite the side of the user on which the eye is located. (Item 50) Item 35. The display system of item 34, wherein the operations further include providing a presentation of the virtual content to a user via a display device, the virtual content being rendered according to an associated adjusted resolution. (Item 51) 1. A method implemented by a display system comprising one or more processors and a head-mountable display, the method comprising: determining an amount of light reaching a retina of an eye of a user of said display system; adjusting a resolution of virtual content to be presented to the user based on the amount of light reaching the retina; A method comprising: (Item 52) 1. A display system comprising: a frame configured to be mounted on a user's head; a light modulation system configured to output light and form an image; one or more waveguides mounted to the frame and configured to receive light from the light modulation system and output the light across a surface of the one or more waveguides; one or more processors; One or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: A proximity of the virtual content from a user fixation point; and the colors of the primary color images, at least one of the primary color images having a different resolution than another primary color image; and one or more computer storage media configured to perform operations including adjusting resolution of primary color images forming virtual content based on the A display system comprising: (Item 53) Item 53. The display system of item 52, wherein adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on the proximity of the virtual content to the fixation point on a depth axis extending away from the user. (Item 54) 53. The display system of claim 52, wherein the one or more waveguides comprise a stack of waveguides, at least some of the waveguides of the stack of waveguides providing a different amount of wavefront divergence than other waveguides of the stack of waveguides. (Item 55) The one or more waveguides include an internally coupled diffractive optical element; an outcoupling diffractive optical element; a light dispersive element configured to direct light from the in-coupling diffractive optical element to the out-coupling diffractive optical element; Item 53. A display system as described in item 52, comprising: (Item 56) 1. A display system comprising: one or more processors; One or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: A proximity of the virtual content from a user fixation point; and the colors of the primary color images, at least one of the primary color images having a different resolution than another primary color image; and one or more computer storage media configured to perform operations including adjusting resolution of primary color images forming virtual content based on the A display system comprising: (Item 57) Item 57. The display system of item 56, wherein the resolution corresponds to at least a color depth. (Item 58) Item 57. The display system of item 56, wherein the resolution corresponds to at least a polygon count. (Item 59) the primary color images comprise a red primary color image, a green primary color image, and a blue primary color image; the green primary image has a higher resolution than the red or blue primary images; Item 57. The display system of item 56. (Item 60) Item 60. The display system of item 59, wherein the red primary image has a higher resolution than the blue primary image. (Item 61) Item 57. The display system of item 56, wherein adjusting the resolution of the virtual content includes reducing a contrast ratio of an image forming the virtual content in response to a decrease in one or both of ambient light and light output by the display system to display the virtual content. (Item 62) Item 57. The display system of item 56, wherein the operations further include preventing rendering of virtual content corresponding to an optic blind spot of an eye of a user of the display system. (Item 63) Item 57. The display system of item 56, wherein adjusting the resolution of the virtual content includes preventing rendering of virtual content located within an opposite peripheral region for each eye of the user, the opposite peripheral region being on a side of the user opposite the side of the user on which the eye is located. (Item 64) 57. The display system of claim 56, wherein the operations further include providing a presentation of the virtual content to a user via a display device, the virtual content being rendered according to an associated adjusted resolution. (Item 65) 1. A method implemented by a display system comprising one or more processors and a head-mountable display, the method comprising: A proximity of the virtual content from a user fixation point; and the colors of the primary color images, at least one of the primary color images having a different resolution than another primary color image; and adjusting a resolution of a primary color image forming the virtual content based on the determined resolution. (Item 66) 1. A display system comprising: an image source comprising a spatial light modulator for providing a first image stream and a second image stream; a viewing assembly including light guidance optics for receiving the first and second image streams from the image source and outputting the first and second image streams to a user; one or more processors in communication with the image source; One or more computer storage media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: causing the image source to output the first image stream to the viewing assembly, an image formed by the first image stream having a first pixel density; causing the image source to output the second image stream to the viewing assembly, an image formed by the second image stream having a second pixel density greater than the first pixel density; Performing an operation including an image formed by the second image stream corresponds to a portion of an image provided by the first image stream; one or more computer storage media, wherein the images formed by the second image stream cover a corresponding portion of the field of view provided by the first image stream; A display system comprising: (Item 67) Item 67. The display system of item 66, wherein the operations further include altering a location of the second image relative to the first image. (Item 68) Item 68. The display system of item 67, wherein the operations further include altering a size of the second image over time. (Item 69) an eye gaze tracker configured to detect changes in an orientation of the user's eye; the operations include altering a location of the second image relative to the first image according to a detected change in an orientation of the user's eyes. Item 68. The display system of item 67. (Item 70) a scanning mirror in the path of the second image stream between the image source and the viewing assembly; Further equipped with 70. The display system of claim 69, wherein the operations further include adjusting an orientation of the scanning mirror to alter an image location of the second image relative to the first image. (Item 71) The viewing assembly includes an eyepiece, the eyepiece comprising: Item 67. The display system of item 66, comprising a waveguide comprising an internal coupling grating configured to internally couple light into the waveguide. (Item 72) Item 72. The display system of item 71, wherein the internal coupling grating is a polarization sensitive internal coupling grating of a liquid crystal. (Item 73) Item 72. The display system of item 71, wherein the waveguide comprises another internal coupling grating configured to internally couple light into the waveguide, the other internal coupling grating being disposed along a different surface of the waveguide than the internal coupling grating. (Item 74) the internal coupling grating and the other internal coupling grating are both in an optical path of the second image stream; the internal coupling grating and the further internal coupling grating are transmissive to the second image stream having a first polarization; a mirror spaced apart from and facing the other internal coupling grating, the mirror configured to reflect the second image stream towards the other internal coupling grating while also changing a polarization of the light of the second image stream to a second polarization, the other internal coupling grating configured to incouple light of the second polarization. Item 74. The display system of item 73. (Item 75) 72. The display system of claim 71, wherein the waveguide comprises another internal coupling grating configured to internally couple light into the waveguide, the other internal coupling grating being arranged along the same surface of the waveguide as the internal coupling grating, the internal coupling grating being arranged in the path of the second image stream, and the other internal coupling grating being arranged in the path of the first image stream. (Item 76) Item 67. The display system of item 66, further comprising beam splitting optics for splitting the light into multiple image streams propagating in different directions. (Item 77) Item 77. The display system of item 76, wherein the beam splitting optics is a polarizing beam splitter. Item 77. The display system of item 76, wherein the beam splitting optic is a switchable reflector, the switchable reflector being selectively switchable between a reflective state and a transmissive state. (Item 78) 1. A display system for projecting an image to an eye of a user, the display system comprising: an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, the first image stream having a first field of view and the second image stream having a second field of view narrower than the first field of view, the first light beam and the second light beam being multiplexed; A polarizing beam splitter, the polarizing beam splitter comprising: receiving and reflecting the first light beam along a first optical path toward a viewing assembly to position the first image stream at a fixed position relative to an eye of the user; receiving the second light beam and transmitting it along a second optical path; a polarizing beam splitter configured to a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam towards the viewing assembly; an eye gaze tracker configured to detect eye movements of the user; a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that a position of the second image stream is moved according to detected movement of the user's eye; A display system comprising: (Item 79) Item 79. The display system of item 78, wherein the first image stream has a first angular resolution and the second image stream has a second angular resolution that is higher than the first angular resolution. (Item 80) Item 79. The display system of item 78, wherein the content of the second image stream comprises a portion of the content of the first image stream. (Item 81) Item 81. A display system as described in item 80, wherein the content of the second image stream changes as the second image stream moves relative to the first image stream such that the content of the second image stream corresponds to the portion of the first image stream overlaid by the second image stream. (Item 82) Item 79. The display system of item 78, wherein the first polarization and the second polarization are each linearly polarized light, and the second polarization is orthogonal to the first polarization. (Item 83) Item 79. The display system of item 78, further comprising a first relay lens assembly configured to provide a first angular magnification to the first light beam. (Item 84) a second relay lens assembly configured to provide a second angular magnification to the second light beam, the second angular magnification being different from the first angular magnification; Item 84. The display system of item 83, further comprising: (Item 85) Item 85. The display system of item 84, wherein the first angular magnification ratio is greater than 1 and the second angular magnification ratio is less than or equal to 1. (Item 86) 85. The display system of claim 84, wherein the first relay lens assembly comprises a first optical lens positioned between the image source and the polarizing beam splitter and a second optical lens positioned along the first optical path, wherein a ratio of a focal length of the first optical lens to a focal length of the second optical lens is greater than 1. (Item 87) 87. The display system of claim 86, wherein the second relay lens assembly comprises the first optical lens and a third optical lens positioned along the second optical path, and a ratio of a focal length of the first optical lens to a focal length of the third optical lens is less than or equal to 1. (Item 88) the first light beam and the second light beam are time division multiplexed; Item 79. The display system of item 78, wherein the image source comprises a switching polarization rotator configured to provide a first light beam at the first polarization and a second light beam at the second polarization. (Item 89) Item 79. The display system of item 78, wherein the first light beam and the second light beam are polarization division multiplexed. (Item 90) 1. A display system for projecting an image to an eye of a user, the display system comprising: an image source configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream, the first image stream having a first field of view and the second image stream having a second field of view narrower than the first field of view, the first light beam and the second light beam being multiplexed; a scanning mirror configured to receive and reflect the first light beam and the second light beam toward a viewing assembly to project the first image stream and the second image stream; an eye gaze tracker configured to detect eye movements of the user; a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that a position of the first image stream and a position of the second image stream are moved according to detected movement of an eye of the user; a switchable optical element disposed in an optical path of the first light beam and the second light beam, the switchable optical element configured to be switched to a first state for the first light beam such that the first light beam is angularly expanded by a first angular expansion factor and to a second state for the second light beam such that the second light beam is angularly expanded by a second angular expansion factor that is less than the first angular expansion factor; A display system comprising: (Item 91) Item 91. The display system of item 90, wherein the first light beam and the second light beam are time division multiplexed and the switchable optical element comprises an electrically switchable liquid crystal lens. (Item 92) Item 91. The display system of item 90, wherein the first light beam and the second light beam are polarization division multiplexed and the switchable optical element comprises a multifocal birefringent lens. (Item 93) Item 93. The display system of item 92, wherein the multifocal birefringent lens comprises a birefringent polymer. (Item 94) Item 91. The display system of item 90, wherein the switchable optical element is positioned downstream of the scanning mirror. (Item 95) Item 91. The display system of item 90, wherein the switchable optical element is disposed on a surface of the scanning mirror. (Item 96) Item 91. The display system of item 90, wherein the switchable optical element is disposed between the image source and the scanning mirror. (Item 97) Item 91. The display system of item 90, wherein the first angular magnification ratio is greater than 1 and the second angular magnification ratio is approximately 1. (Item 98) A wearable display system, comprising: 1. An afocal magnifying lens with circular polarization handedness dependent magnification, said afocal magnifying lens comprising: a first fixed focal length lens element; a first geometric phase lens that exhibits a positive refractive power for a first handedness of the incident circularly polarized light and a negative refractive power for a second handedness of the incident circularly polarized light; A second geometric phase lens; An infinity focal magnifying lens comprising: A wearable display system comprising: (Item 99) When transmitting light, the first geometric phase lens reverses the handedness of the circularly polarized light; the second geometric phase lens exhibits a negative refractive power for a first handedness of the circularly polarized light and a positive refractive power for a second handedness of the circularly polarized light. Item 99. The wearable display system of item 98. (Item 100) Item 99. The wearable display system of item 98, further comprising a second fixed focal length lens, the second geometric phase lens exhibiting a positive refractive power for a first handedness of the circularly polarized light and a negative refractive power for a second handedness of the circularly polarized light. (Item 101) Item 99. The wearable display system of item 98, wherein the first geometric phase lens reverses circular polarization handedness when transmitting light. (Item 102) Item 99. The wearable display system of item 98, wherein when the first fixed focal length lens element and the first geometric phase lens are considered as a lens group, for light having the first handedness, the lens group has a positive focal length of a first group, and for light having the second handedness, the lens group has a positive focal length of a second group, the positive focal length of the first group exceeding the positive focal length of the second group. (Item 103) a polarization rotator optically coupled to the afocal magnification lens, the polarization rotator configured to receive polarized light and selectively output light of a first linear polarization or a second linear polarization; a first waveplate between the polarization rotator and the afocal expansion lens, the first waveplate configured to convert light of the first linear polarization to circular polarization of the first handedness and to convert light of the second linear polarization to circular polarization of the second handedness; Item 99. The wearable display system of item 98, further comprising: (Item 104) a first optical path between the polarization rotator and the first waveplate for light of the first linear polarization; a second optical path between the polarization rotator and the first waveplate for light of the second linear polarization; and Item 104. The wearable display system of item 103, comprising: (Item 105) Item 105. The wearable display system of item 104, wherein at least one of the first optical path and the second optical path includes a steerable mirror. (Item 106) Item 105. The wearable display system of item 104, wherein the first optical path and the second optical path each include at least a subset of a plurality of relay lens elements. (Item 107) Item 104. The wearable display system of item 103, wherein within each of the first optical path and the second optical path, consecutive pairs of the multiple relay lens elements form an infinite focal compound lens. (Item 108) Item 105. The wearable display system of item 104, wherein the first optical path intersects a polarization selective reflector and the second optical path intersects the polarization selective reflector. (Item 109) Item 109. The wearable display system of item 108, wherein the first optical path and the second optical path each include at least a subset of four relay lens elements, the four relay lens elements being centered about the polarization selective reflector. (Item 110) Item 109. The wearable display system of item 108, wherein the first optical path intersects the polarization selective reflector in a first direction and the second optical path intersects the polarization selective reflector in a second direction. (Item 111) Item 109. The wearable display system of item 108, wherein at least one of the first optical path and the second optical path includes a second wave plate and a third wave plate. (Item 112) 1. An optical subsystem for a wearable image projector, comprising: a polarization selective reflector; and a set of four lens elements positioned about the polarization selective reflector; An optical subsystem comprising: (Item 113) Item 113. An optical subsystem as described in item 112, wherein each successive pair of the set of four lens elements forms an afocal compound lens along at least one optical path through the subsystem. (Item 114) Item 113. The optical subsystem of item 112, further comprising a polarization rotation switch optically coupled to the polarization selective reflector and the set of four lens elements. (Item 115) Item 115. The optical subsystem of item 114, further comprising a plurality of wave plates positioned about the polarization selective reflector. (Item 116) Item 116. The optical subsystem of item 115, wherein each of the plurality of wave plates is positioned between one of the set of four relay lens elements and the polarization selective reflector. (Item 117) Item 116. The optical subsystem of item 115, wherein the plurality of wave plates includes three wave plates. (Item 118) Item 116. The optical subsystem of item 115, further comprising a foveal tracking mirror positioned on an opposite side of the first of the set of four relay lenses from the polarization selective reflector. (Item 119) Item 119. The optical subsystem of item 118, further comprising an image scanning mirror positioned on an opposite side of the second of the set of four relay lenses from the polarization selective reflector. (Item 120) Item 120. The optical subsystem of item 119, further comprising a light beam source optically coupled to the image scanning mirror. (Item 121) Item 121. The optical subsystem of item 120, wherein the light beam source comprises a dichroic reflector, a first laser diode that emits light of a first color that is reflected by the dichroic reflector, and a second laser diode that emits light of a second color that is transmitted by the dichroic reflector. (Item 122) 1. A display system for projecting an image to an eye of a user, the display system comprising: An eyepiece, the eyepiece comprising: A waveguide; an internal coupling grating optically coupled to the waveguide; an eyepiece including: a first image source configured to project a first light beam associated with a first image stream, the first image stream having a first field of view incident on a first surface of the internal coupling grating, a portion of the first light beam being coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to an eye of the user; a second image source configured to project a second light beam associated with a second image stream, the second image stream having a second field of view narrower than the first field of view; a scanning mirror configured to receive and reflect the second light beam such that the second light beam is incident on a second surface of the internal coupling grating opposite the first surface thereof, and a portion of the second light beam is coupled into the waveguide by the internal coupling grating; and an eye gaze tracker configured to detect eye movements of the user; a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that a position of the second image stream is moved according to detected movement of the user's eye; A display system comprising: (Item 123) Item 123. The display system of item 122, further comprising a first lens assembly positioned along a first optical path of the first light beam and configured to provide a first angular magnification to the first light beam. (Item 124) Item 124. The display system of item 123, further comprising a second lens assembly positioned along a second optical path of the second light beam and configured to provide a second angular magnification ratio to the second light beam, the second angular magnification ratio being different from the first angular magnification ratio. (Item 125) Item 125. The display system of item 124, wherein the first angular magnification ratio is greater than 1 and the second angular magnification ratio is less than or equal to 1. (Item 126) Item 123. The display system of item 122, wherein the first image stream has a first angular resolution and the second image stream has a second angular resolution that is higher than the first angular resolution. (Item 127) 1. A display system for projecting an image to an eye of a user, the display system comprising: An eyepiece, the eyepiece comprising: A waveguide; an internal coupling grating optically coupled to the waveguide; an eyepiece including: an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, the first image stream having a first field of view and the second image stream having a second field of view narrower than the first field of view, the first light beam and the second light beam being multiplexed; A polarizing beam splitter, the polarizing beam splitter comprising: receiving and reflecting the first light beam along a first optical path; receiving the second light beam and transmitting it along a second optical path; a polarizing beam splitter configured to a first optical reflector positioned along the first optical path and configured to receive and reflect the first light beam such that the first light beam is incident on a first surface of the internal coupling grating, a portion of the first light beam being coupled into the waveguide by the internal coupling grating to position the first image stream at a fixed position relative to an eye of the user; and a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam; a second optical reflector positioned along the second optical path downstream of the scanning mirror, the second optical reflector configured to receive and reflect the second light beam such that the second light beam is incident on a second surface of the internal coupling grating opposite the first surface thereof, a portion of the second light beam being coupled into the waveguide by the internal coupling grating; and an eye gaze tracker configured to detect eye movements of the user; a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that a position of the second image stream is moved according to detected movement of the user's eye; A display system comprising: (Item 128) 1. A display system for projecting an image to an eye of a user, the display system comprising: A waveguide; an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, the first image stream having a first field of view and the second image stream having a second field of view narrower than the first field of view, the first light beam and the second light beam being multiplexed; A polarizing beam splitter, the polarizing beam splitter comprising: receiving and reflecting the first light beam along a first optical path; receiving the second light beam and transmitting it along a second optical path; a polarizing beam splitter configured to a first incoupling prism positioned along the first optical path adjacent a first surface of the waveguide, the first incoupling prism configured to couple a portion of the first light beam into the waveguide to position the first image stream at a fixed location relative to an eye of the user; and a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam; a second incoupling prism positioned along the second optical path downstream of the scanning mirror and adjacent a second surface of the waveguide opposite the first surface of the waveguide, the second incoupling prism configured to couple a portion of the second light beam into the waveguide; an eye gaze tracker configured to detect eye movements of the user; a control circuit in communication with the eye gaze tracker and the scanning mirror, the control circuit configured to position the scanning mirror such that a position of the second image stream is moved according to detected movement of the user's eye; A display system comprising: [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) device.

[0025] [Diagram 2] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.

[0026] [Diagram 3] 3A-3C illustrate the relationship between the radius of curvature and the radius of focus.

[0027] [Figure 4A] FIG. 4A illustrates a representation of the accommodation-vergence response of the human visual system.

[0028] [Figure 4B] FIG. 4B illustrates examples of different accommodation and convergence states of a pair of a user's eyes.

[0029] [Figure 4C]FIG. 4C illustrates an example of a top-down view representation of a user viewing content through a display system.

[0030] [Figure 4D] FIG. 4D illustrates another example of a top-down view representation of a user viewing content through a display system.

[0031] [Diagram 5] FIG. 5 illustrates aspects of an approach for simulating a three-dimensional image by modifying wavefront divergence.

[0032] [Figure 6] FIG. 6 illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0033] [Figure 7] FIG. 7 illustrates an example of an output beam output by a waveguide.

[0034] [Figure 8] FIG. 8 illustrates an example of a stacked waveguide assembly where each depth plane contains an image formed using multiple different primary colors.

[0035] [Figure 9A] FIG. 9A illustrates a cross-sectional side view of an example of a set of stacked waveguides, each of which includes an internal coupling optical element.

[0036] [Figure 9B] FIG. 9B illustrates a perspective view of the multiple stacked waveguide embodiment of FIG. 9A.

[0037] [Figure 9C] FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B.

[0038] [Figure 9D]FIG. 9D illustrates an example of a wearable display system.

[0039] [Figure 10A] FIG. 10A illustrates an example of a top-down view representation of a user viewing content through a display system.

[0040] [Figure 10B] FIG. 10B illustrates another example of a top-down view representation of a user viewing content through a display system.

[0041] [Figure 10C] FIG. 10C illustrates yet another example of a top-down view representation of a user viewing content through a display system.

[0042] [Figure 10D] FIG. 10D is a block diagram of an exemplary display system.

[0043] [Figure 11A1] FIG. 11A1 illustrates an example of a top-bottom view representation of accommodation at resolution within different resolution accommodation zones based on 3D fixation tracking.

[0044] [Figure 11A2] FIG. 11A2 illustrates an example of a top-down view representation of the resolution adjustment zones at different times as the size and number of zones are changed.

[0045] [Figure 11B] FIG. 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone of FIG. 11A1.

[0046] [Figure 11C] FIG. 11C illustrates another embodiment of a configuration for a resolution adjustment zone.

[0047] [Figure 11D]FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

[0048] [Figure 11E] FIG. 11E illustrates another example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

[0049] [Figure 12A] 12A-12C show schematic diagrams of example embodiments of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point. [Figure 12B] 12A-12C show schematic diagrams of example embodiments of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point. [Figure 12C] 12A-12C show schematic diagrams of example embodiments of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixation point.

[0050] [Figure 13] FIG. 13 illustrates an example of a representation of a user viewing multiple virtual objects aligned with the user's line of sight.

[0051] [Figure 14] FIG. 14 is a diagram of an example embodiment of a process for adjusting virtual content based on angular proximity to a user's line of sight.

[0052] [Figure 15] FIG. 15 illustrates an example of a representation of the retina of a user's eye.

[0053] [Figure 16] FIG. 16 illustrates diagrammatically an example of resolution and rod and cone densities across the retina of FIG.

[0054] [Figure 17] FIG. 17 graphically illustrates an example of the relationship between pupil size and the amount of light incident on a user's eye.

[0055] [Figure 18] FIG. 18 is a schematic diagram of an example embodiment of a process for adjusting virtual content based on the amount of light incident on a user's eye.

[0056] [Figure 19] FIG. 19 diagrammatically illustrates an example of the change in resolution detectable by a user's eye as the amount of light incident on the eye changes.

[0057] [Figure 20] FIG. 20 graphically illustrates an example of differences in the eye's sensitivity to light of different colors at different levels of illumination.

[0058] [Figure 21] FIG. 21 is a schematic diagram of an example process for adjusting virtual content formed using multiple primary color images, where the resolution adjustment is based on the color of the primary color images.

[0059] [Figure 22] 22A-22C illustrate examples of contrast sensitivity changing as the amount of light incident on a user's eye decreases.

[0060] [Diagram 23] FIG. 23 illustrates an example of a representation of the optic nerve and peripheral blind spot of a user's eye.

[0061] [Figure 24] FIG. 24 shows an exemplary monocular field of view for the human eye.

[0062] [Figure 25A] FIG. 25A shows an exemplary wearable display device configured to provide virtual content to a user.

[0063] [Figure 25B] FIG. 25B is a block diagram depicting an augmented reality system.

[0064] [Figure 25C] FIG. 25C diagrammatically illustrates the light paths within a viewing optical assembly (VOA), which may be used to present a digital or virtual image to a viewer.

[0065] [Figure 26A] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26B] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26C] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations. [Figure 26D] 26A-26D illustrate example rendering viewpoints that will be used and the light fields that will be produced in the AR system for each of two example eye orientations.

[0066] [Figure 26E] 26E-26F diagrammatically illustrate example configurations of images that may be presented to a user. [Figure 26F] 26E-26F diagrammatically illustrate example configurations of images that may be presented to a user.

[0067] [Figure 26G] 26G-26H diagrammatically illustrate example configurations of images that may be presented to a user. [Fig. 26H] 26G-26H diagrammatically illustrate example configurations of images that may be presented to a user.

[0068] [Figure 27]FIG. 27 illustrates the field of view and ocular field of view as shown in FIG. 24 overlaid on one of the displays in a wearable display device as shown in FIG. 25.

[0069] [Figure 28A] 28A-28B illustrate some of the principles described in FIGS. 26A-26D. [Figure 28B] 28A-28B illustrate some of the principles described in FIGS. 26A-26D.

[0070] [Figure 28C] 28C-28D illustrate some example images that may be presented to the user. [Figure 28D] 28C-28D illustrate some example images that may be presented to the user.

[0071] [Figure 28E] FIG. 28E illustrates an exemplary high FOV low resolution image frame.

[0072] [Figure 28F] FIG. 28F illustrates an exemplary low FOV high resolution image frame.

[0073] [Figure 29A] FIG. 29A shows a simplified block diagram of a display system.

[0074] [Figure 29B] FIG. 29B diagrammatically illustrates a cross-sectional view of an augmented reality (AR) system.

[0075] [Figure 30A] 30A-30B diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 30B] 30A-30B diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0076] [Figure 30C] FIG. 30C diagrammatically illustrates a cross-sectional view of an augmented reality (AR) system.

[0077] [Figure 30D] FIG. 30D shows a simplified block diagram of a display system.

[0078] [Figure 31A] FIG. 31A diagrammatically illustrates the operating principle of the first relay lens assembly in the display system illustrated in FIGS. 30A-30B.

[0079] [Figure 31B] FIG. 31B diagrammatically illustrates the operating principle of the second relay lens assembly in the display system illustrated in FIGS. 30A-30B.

[0080] [Figure 31C] 31C-31D illustrate diagrammatically the display system. [Figure 31D] 31C-31D illustrate diagrammatically the display system.

[0081] [Figure 32A] 32A-32C diagrammatically illustrate a display system. [Figure 32B] 32A-32C diagrammatically illustrate a display system. [Figure 32C] 32A-32C diagrammatically illustrate a display system.

[0082] [Figure 33A] 33A-33B diagrammatically illustrate a display system. [Figure 33B] 33A-33B diagrammatically illustrate a display system.

[0083] [Figure 34A] 34A-34B diagrammatically illustrate a display system. [Figure 34B] 34A-34B diagrammatically illustrate a display system.

[0084] [Diagram 35] FIG. 35 illustrates diagrammatically the display system.

[0085] [Figure 36A] FIG. 36A diagrammatically illustrates an augmented reality eyepiece display system.

[0086] [Figure 36B] FIG. 36B diagrammatically illustrates another augmented reality eyepiece display system.

[0087] [Figure 37A] FIG. 37A is a schematic diagram of a 2x afocal magnifying lens.

[0088] [Figure 37B] FIG. 37B is a schematic diagram of a bifocal afocal magnifying lens.

[0089] [Figure 38] 38A-38B diagrammatically illustrate example configurations of images that may be presented to a user.

[0090] [Figure 39A] 39A-39B illustrate some example images that may be presented to a user. [Figure 39B] 39A-39B illustrate some example images that may be presented to a user.

[0091] [Figure 40A] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40B]40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40C] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 40D] 40A-40D diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0092] [Figure 41A] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41B] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41C] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye. [Figure 41D] 41A-41D diagrammatically illustrate a display system for projecting an image stream to a user's eye.

[0093] [Diagram 42] FIG. 42 illustrates an example frame structure for time division multiplexed high FOV low resolution image stream and low FOV high resolution image stream.

[0094] [Diagram 43] FIG. 43 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0095] [Diagram 44] FIG. 44 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0096] [Diagram 45]FIG. 45 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0097] [Figure 46] FIG. 46 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0098] [Figure 47] FIG. 47 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0099] [Figure 48] FIG. 48 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0100] [Figure 49] FIG. 49 illustrates diagrammatically a display system for projecting an image stream to a user's eye.

[0101] [Figure 50] FIG. 50 diagrammatically illustrates a display system for projecting an image stream to a user's eye, according to some embodiments.

[0102] [Figure 51] FIG. 51 diagrammatically illustrates a display system for projecting an image stream to a user's eye, according to some embodiments.

[0103] [Figure 52A] 52A-52B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments. [Figure 52B] 52A-52B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments.

[0104] [Figure 53A] 53A-53B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments. [Figure 53B] 53A-53B diagrammatically illustrate a display system for projecting an image stream to a user's eye, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0105] Rendering virtual content for augmented and virtual display systems is computationally intensive, particularly computationally intensive, which may undesirably use large amounts of memory, cause high latency, and / or require the use of powerful processing units that may have high cost and / or high energy consumption.

[0106] In some embodiments, the methods and systems conserve computational resources, such as memory and processing time, by reducing the resolution of virtual content located away from a user's eye fixation point. For example, the system may render virtual content at or near the user's eye fixation point at a relatively high (e.g., highest) resolution, while utilizing one or more lower resolutions for virtual content away from the fixation point. The virtual content is presented by a display system that may display virtual content at multiple different depths (e.g., multiple different depth planes, such as two or more depth planes), and the reduction in resolution preferably occurs at least along the z-axis, which is the depth axis (corresponding to distance away from the user). In some embodiments, the resolution reduction occurs along the z-axis and / or the x- and y-axes, where the x-axis is the lateral axis and the y-axis is the vertical axis.

[0107] Determining an appropriate resolution for the virtual content may include determining a fixation point of the user's eyes in three-dimensional space. For example, the fixation point may be an x, y, z coordinate in the user's field of view on which the user's eyes are fixated. The display system may be configured to present virtual objects with a difference in resolution, where the resolution decreases with decreasing proximity of the virtual object to the fixation point, in other words, the resolution decreases with increasing distance from the fixation point.

[0108] As discussed herein, the display system may present virtual objects within a display frustum of the display system, where the virtual objects can be presented on different depth planes. In some embodiments, the display frustum is a field of view provided by the display system over which the display system is configured to present virtual content to a user of the display system. The display system may be a head-mounted display system including one or more waveguides that may present virtual content (e.g., virtual objects, graphics, text, etc.), where the one or more waveguides are configured to output light with different wavefront divergences and / or different binocular disparities corresponding to different depth planes (e.g., corresponding to a particular distance from the user). It should be understood that each eye may have one or more waveguides associated therewith. Using the different wavefront divergences and / or different binocular disparities, the display system may cause a first virtual object to appear to be located at a first depth in the user's field of view, while a second virtual object to appear to be located at a second depth in the user's field of view. In some embodiments, the depth plane of the fixation point or its neighboring depth planes may be determined, and the resolution of content on other depth planes may be reduced based on the distance from those depth planes to the depth plane where the fixation point is located. It should be understood that references herein to the depth of a virtual content (the distance of the virtual content from the user on the z-axis) refer to the apparent depth of the virtual content as intended to be seen by the user. In some embodiments, the depth of a virtual object may be understood as the distance from the user of a real object that has wavefront divergence and / or binocular disparity similar to that of the virtual object.

[0109] It should be appreciated that the proximity of the virtual object and the fixation point may be determined by various measurements, non-limiting examples of which include determining the distance between the fixation point and the virtual object, determining the resolution adjustment zone occupied by the virtual object relative to the resolution adjustment zone occupied by the fixation point (in embodiments where the user's field of view is subdivided into resolution adjustment zones, as described below), and determining the angular proximity of the virtual object and the user's fixation point. Proximity may also be determined using a combination of the above techniques. For example, the distance and / or angular proximity of a first zone (where the virtual object is located) and a second zone (where the fixation point is located) may be used to determine the proximity. These various measurements are discussed further below.

[0110] In some embodiments, determining a fixation point may include anticipating a fixation point of a user's eye and utilizing the anticipated fixation point as a fixation point for determining the resolution of the virtual content. For example, the display system may render a particular content at a relatively high resolution with the anticipation that the user's eye will fixate on that content. As an example, it should be appreciated that the human visual system may be sensitive to sudden changes in a scene (e.g., sudden motion, changes in luminance, etc.). In some embodiments, the display system may determine that the virtual content is of a type that will cause the user's eye to fixate on it (e.g., involving motion in a scene where other virtual and real objects are stationary), and then render the virtual content at a high resolution with the anticipation that the user's eye will subsequently focus on the virtual content.

[0111] As mentioned above, in some embodiments, the distance of a virtual object from a determined fixation point may correspond to a distance extending in three dimensions. As an example, a first virtual object located at the same depth (e.g., the same depth plane) from the user as the determined fixation point, but located horizontally or vertically from the fixation point, may also be similarly reduced in resolution because a second virtual object is located at a further depth (e.g., a further depth plane) from the determined fixation point. As a result, different resolutions may be associated with different distances from the fixation point.

[0112] In some embodiments, the environment around the user may be divided into spatial volumes (also referred to herein as resolution adjustment zones), and the resolution of virtual objects within the same resolution adjustment zone is similar. The resolution adjustment zone may have an arbitrary three-dimensional shape, such as a cube, or other three-dimensional polygonal shape, or a curved three-dimensional shape, as described herein. In some embodiments, all the resolution adjustment zones have a similar shape, such as a rectangular prism or a sphere. In some other embodiments, different resolution adjustment zones may have different shapes or sizes (e.g., the shape and / or size of the volume may change with distance from the fixation point).

[0113] In some embodiments, the resolution adjustment zone is a part of the user's field of view. For example, the user's field of view may be separated into volumes of space that form the resolution adjustment zone. In some embodiments, each depth plane may be subdivided into one or more continuous volumes of space, i.e., one or more resolution adjustment zones. In some embodiments, each resolution adjustment zone may encompass a certain range of depth (e.g., depth plane value + / - some variance, examples of variance include 0.66 dpt, 0.50 dpt, 0.33 dpt, or 0.25 dpt) from the user, as well as a certain lateral and vertical distance. Virtual objects located within the same resolution adjustment zone as the determined fixation point may be presented (e.g., rendered) at a high (e.g., full) resolution, while virtual objects located in a volume of space outside the resolution adjustment zone of the fixation point may be rendered at a lower resolution according to the distance of the volume from the volume of space of the fixation point. In some embodiments, each resolution adjustment zone may be assigned a particular resolution (e.g., a particular reduction in resolution relative to full resolution), and virtual content that falls within a given zone may be rendered at the associated resolution for that zone. In some embodiments, a distance between a volume and a volume occupied by a fixation point may be determined, and the resolution may be set based on this distance.

[0114] Advantageously, the number and size of the resolution adjustment zones utilized to divide the user's field of view may be modified according to the confidence in the user's determined fixation point. For example, the size associated with each spatial volume may be increased or decreased based on the confidence that the user's line of sight is converged on a precise point in three-dimensional space. If the confidence in the fixation point is high, the display system may present only virtual objects in one compact resolution adjustment zone at a relatively high resolution (the compact resolution adjustment zone that includes the fixation point), while reducing the resolution of other virtual objects, thus saving processing power. However, if the confidence is low, the display system may increase the size of each spatial volume (e.g., reduce the overall number of volumes) so that each spatial volume contains a larger number of virtual objects within the spatial volume of the fixation point. It should be understood that the size and shape of the volumes may be fixed during production of the display system, for example, based on the tolerances expected in the system for determining the fixation point, and / or may be adjusted or set in the field in response to the user's characteristics, the user's environment, and / or software changes that change the tolerances for the system for determining the fixation point.

[0115] It should be understood that a user's sensitivity to resolution may decrease with distance from the fixation point. As a result, by ensuring that full resolution content is presented at the fixation point, and by allowing for a tolerance of error as to where the fixation point is located, the perceptibility of the reduction in resolution may be reduced or eliminated, thereby providing the perception of a high resolution display without utilizing the computational resources typically required to present content for such a high resolution display.

[0116] In some embodiments, the proximity of the virtual object and the fixation point may be determined based on the angular proximity of the virtual object and the user's line of sight, and the resolution of the virtual object may decrease as the angular proximity decreases. In some embodiments, this may result in virtual objects located at different depths from the user being presented with similar resolution. For example, a first virtual object at a location corresponding to the user's determined fixation point may be located in front of a second virtual object (e.g., closer in depth to the user). The second virtual object may optionally be presented with a similar (e.g., identical) resolution as the first virtual object, since the second virtual object is along the user's line of sight and therefore would similarly fall on the user's fovea, where the user's eyes are most sensitive to changes in resolution. Optionally, the second virtual object may be further adjusted through a blurring process (e.g., a Gaussian blur kernel may be convolved with the second virtual object), which may reduce in resolution and represent the second virtual object being farther from the user (e.g., located on a more distant depth plane).

[0117] The reduction in resolution may vary based on how the virtual content is presented by the display system. In some embodiments, a first exemplary display system, referred to herein as a variable focus display system, may present the virtual content on different depth planes, with all content (e.g., virtual objects) being presented at the same depth plane (e.g., via the same waveguide) at one time, for example, for each frame presented to the user. That is, the variable focus display system may utilize a single depth plane (e.g., selected from multiple depth planes based on the user's fixation point, or selected based on the depth of a particular presented virtual object) at one time to present the content, or may change the depth plane (e.g., select a different depth plane) in subsequent frames. In some other embodiments, a second exemplary display system, referred to herein as a multifocal display system, may present the virtual content on different depth planes, with content being displayed at multiple depth planes at the same time. As will be further described herein, a variable focus display system may optionally utilize a single frame buffer, and with respect to the above example regarding blurring the second virtual object, the second virtual object may be blurred prior to presentation to the user from the single frame buffer. In contrast, a multi-focal display system may present the second virtual object at a further depth (e.g., on a further depth plane) from the first virtual object, optionally at a reduced resolution, and the second virtual object may appear to the user as blurred (e.g., the second virtual object will be blurred based on the natural physics of the user's eye without further processing).

[0118] As disclosed herein, the display system may present virtual objects at or near a determined fixation point in a relatively high (e.g., full) resolution and may present virtual objects further from the fixation point in a reduced resolution. Preferably, the relatively high resolution is the highest resolution for presentation of the virtual objects in the user's field of view. The relatively high resolution may be the maximum resolution of the display system, a user-selectable resolution, a resolution based on the specific computing hardware presenting the virtual objects, etc.

[0119] It should be appreciated that adjusting the resolution of a virtual object may include any modification to the virtual object to alter the quality of the presentation of the virtual object. Such modifications may include adjusting the quality at one or more points in the graphics pipeline of a graphics processing unit (GPU), adjusting the polygon count of the virtual object, adjusting primitives utilized to generate the virtual object (e.g., adjusting the shape of the primitives, e.g., adjusting the primitives from a triangular mesh to a quadrilateral mesh, etc.), adjusting operations performed on the virtual object (e.g., shader operations), adjusting texture information, color resolution, or depth, adjusting the number of rendering cycles or frame rate, etc.

[0120] In some embodiments, on the x- and y-axes, the change in resolution of the virtual content away from the fixation point may generally track the change in the distribution of photoreceptors in the retina of the user's eye. For example, it should be understood that the view of the world and virtual content may be imaged onto the retina such that different parts of the retina may be mapped to different parts of the user's field of view. Advantageously, the resolution of the virtual content across the user's field of view may generally track the density of the corresponding photoreceptors (rods or cones) across the retina. In some embodiments, the resolution reduction away from the fixation point may generally track the reduction in the density of cones across the retina. In some other embodiments, the resolution reduction away from the fixation point may generally track the reduction in the density of rods across the retina. In some embodiments, the trend of the resolution reduction away from the fixation point may be within ±50%, ±30%, ±20%, or ±10% of the trend of the reduction in the density of rods and / or cones across the retina.

[0121] Rods and cones are active at different levels of incidence. For example, cones are active under relatively bright conditions, while rods are active under relatively low light conditions. As a result, in some embodiments, where the reduction in resolution generally tracks the density of rods or cones across the retina, the display system may be configured to determine the amount of light incident on the retina. Based on this amount of light, an appropriate adjustment in resolution may be made. For example, the reduction in resolution may generally track the change in density of rods across the retina under low light conditions, while the reduction in resolution may generally track the change in density of cones under bright conditions. As a result, in some embodiments, the display system may be configured to vary the profile of the reduction in image resolution based on the amount of light incident on the retina.

[0122] It should be understood that the ability of the human eye to resolve fine detail may not be directly proportional to the density of rods or cones in the retina. In some embodiments, the change in resolution of the virtual content across the user's field of view generally tracks the change in the eye's ability to resolve fine detail. As noted above, the rate of change in the resolution of the virtual content may vary with the amount of light reaching the retina.

[0123] In some embodiments, the amount of light reaching the retina may be determined by detecting the amount of ambient light incident on a sensor mounted on the display device. In some embodiments, determining the amount of light reaching the retina may also include determining by the amount of light output by the display device to the user. In yet other embodiments, the amount of light reaching the retina may be determined by imaging the user's eye and determining pupil size. Because pupil size is related to the amount of light reaching the retina, determining pupil size allows the amount of light reaching the retina to be extrapolated.

[0124] It should be understood that full-color virtual content may be formed by multiple primary images that, in the aggregate, provide a full-color perception. The human eye may have different sensitivities to different wavelengths or colors of light. In some embodiments, in addition to varying based on proximity to a fixation point, the change in resolution of the virtual content may vary based on the color of the primary images presented by the display system. For example, if the primary images comprise red, green, and blue images, the green primary image may have a higher resolution than the blue primary image, which may have a higher resolution than the red primary image. In some embodiments, to account for the variation in the eye's sensitivity to different colors at different levels of incident light, the amount of light reaching the retina may be determined, and the resolution adjustment for a given primary image may also vary based on the determination of the amount of light reaching the retina.

[0125] It should be understood that the contrast sensitivity of the eye may also vary based on the amount of light incident on the retina. In some embodiments, the size or total number of gradations in the contrast within the virtual content may vary based on the amount of light reaching the retina. In some embodiments, the contrast ratio of the images forming the virtual content may vary based on the amount of light incident on the retina, with the contrast ratio decreasing as the amount of light decreases.

[0126] In some embodiments, certain portions of a user's visual field may not be provided with any virtual content. For example, the display system may be configured to not provide virtual content within the blind spot caused by the optic nerve and / or peripheral blind spot of a given eye.

[0127] As discussed herein, a display system may be configured to display high resolution content in one portion of a user's field of view and lower resolution content in another portion of the user's field of view. It should be understood that the high resolution content may have a higher pixel density than the lower resolution content. In some environments, a display system may be configured to provide such high and low resolution content by effectively overlapping high and low resolution images. For example, the system may display a low resolution image that spans the entire field of view and then display a high resolution image that spans a small portion of the field of view, with the high resolution image co-located with the corresponding portion of the low resolution image. The high and low resolution images may be routed through different optics that output light at the appropriate angles and determine the amount of the field of view that the images occupy.

[0128] In some embodiments, a single spatial light modulator (SLM) may be used to encode the light with image information, and a beam splitter or optical switch may be used to split the single light stream from the SLM into two streams, one stream propagating through an optical system for a low resolution image and the second stream propagating through an optical system for a high resolution image. In some other embodiments, the polarization of the light encoded with image information may be selectively switched and passed through an optical system that effectively provides different angular magnifications for the light of the different polarizations, thereby providing high and low resolution images.

[0129] Advantageously, various embodiments disclosed herein reduce the processing power requirements for providing content on a display system. Because a larger allocation of processing power may be spent on virtual objects close to the user's three-dimensional fixation point, while processing power for more distant virtual objects may be reduced, the overall required processing power for the display system may be reduced, thus reducing one or more of the size of the processing components, the heat generated by the processing components, and the energy requirements for the display system (e.g., the display system may optionally be battery powered, require a lower capacity battery, and / or operate for a longer duration for a given battery). Thus, the embodiments described herein address technical issues arising from augmented or virtual reality display systems. In addition, the described techniques manipulate the graphical content such that, depending on the presentation to the user, the graphical content may appear to the user as being the same, while being presented fundamentally differently (e.g., the resolution is modified). Thus, the display system preserves visual fidelity and conserves processing power while transforming the graphical content as the user looks around his or her surroundings.

[0130] It should be understood that the display system may be part of an augmented reality display system or a virtual reality display system. As one example, the display of the display system may be transparent, allowing a user to view the real world while providing the user with virtual content in the form of images, videos, interactions, etc. As another example, the display system may block the user's view of the real world, and virtual reality images, videos, interactions, etc. may be presented to the user.

[0131] Reference is now made to the drawings, in which like reference numbers refer to like parts throughout.

[0132] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. It should be understood that when a user's eyes are spaced apart and viewing a real object in space, each eye may have a slightly different view of the object and form an image of the object at a different location on each eye's retina. This may be referred to as binocular disparity and may be exploited by the human visual system to provide a perception of depth. Conventional display systems simulate binocular disparity by presenting two distinctly different images 190, 200 with slightly different views of the same virtual object, one for each eye 210, 220, corresponding to the view of the virtual object as it would appear by each eye as if the virtual object were a real object at a desired depth. These images provide binocular cues that the user's visual system may interpret to derive a perception of depth.

[0133] Continuing with reference to FIG. 2, the images 190, 200 are spaced from the eyes 210, 220 by a distance 230 on the z-axis, which is parallel to the optical axis of the viewer when that eye is fixating on an object at optical infinity directly in front of the viewer. The images 190, 200 are flat and at a fixed distance from the eyes 210, 220. Based on slightly different views of the virtual object in the images presented to the eyes 210, 220, respectively, the eyes may necessarily rotate such that the image of the object falls on a corresponding point on the eye's respective retina to maintain single binocular vision. This rotation may cause the line of sight of each of the eyes 210, 220 to converge on a point in space where the virtual object is perceived to reside. As a result, providing three-dimensional images traditionally involves manipulating the convergence and divergence movements of the user's eyes 210, 220 to provide binocular cues that the human visual system interprets to provide the perception of depth.

[0134] However, creating a realistic and comfortable perception of depth is difficult. It should be understood that light from an object at different distances from the eye has a wavefront with different amounts of divergence. Figures 3A-3C illustrate the relationship between distance and divergence of light rays. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in Figures 3A-3C, the light rays become more divergent as the distance to the object decreases. Conversely, as the distance increases, the light rays become more collimated. In other words, the light field generated by a point (an object or part of an object) can be said to have a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. Although only a monocular 210 is illustrated in Figures 3A-3C and various other figures herein for clarity of illustration, the discussion regarding the eye 210 can be applied to both eyes 210 and 220 of a viewer.

[0135] 3A-3C, light from an object that a viewer's eye is fixating on may have different wavefront divergences. Due to the different wavefront divergences, the light may be focused differently by the eye's lens, which in turn may require the lens to assume a different shape and form a focused image on the eye's retina. If a focused image is not formed on the retina, the resulting retinal blur acts as an accommodation cue, causing the shape of the eye's lens to change until a focused image is formed on the retina. For example, the accommodation cue may trigger the relaxation or contraction of the ciliary muscle surrounding the eye's lens, thereby modulating the force applied to the inferior ligament that holds the lens, thus changing the shape of the eye's lens, thereby forming a focused image of the fixated object on the eye's retina (e.g., the fovea) until retinal blur of the fixated image is eliminated or minimized. The process by which the eye lens changes shape may be referred to as accommodation, and the shape of the eye lens required to form a focused image of a fixated object on the eye's retina (e.g., the fovea) may be referred to as the state of accommodation.

[0136] Referring now to FIG. 4A, a representation of the accommodation-vergence response of the human visual system is illustrated. A movement of the eye to fixate an object causes the eye to receive light from the object, which forms an image on each of the eye's retinas. The presence of retinal blur in the image formed on the retina may provide a cue for accommodation, and the relative location of the image on the retina may provide a cue for vergence. A cue for accommodation causes accommodation, resulting in the eye adopting a particular accommodation state in which the eye's lens forms a focused image of the object on the eye's retina (e.g., the fovea). On the other hand, a cue for vergence causes a vergence movement (eye rotation) such that the image formed on each retina of each eye is at a corresponding retinal point that maintains single binocular vision. In these positions, the eye may be said to adopt a particular vergence state. Continuing with reference to FIG. 4A, accommodation can be understood to be the process by which the eye achieves a particular accommodation state, and convergence can be understood to be the process by which the eye achieves a particular convergence state. As shown in FIG. 4A, the accommodation and convergence state of the eye can change when the user fixates a different object. For example, the accommodated state can change when the user fixates a new object at a different depth on the z-axis.

[0137] Without being limited by theory, it is believed that a viewer of an object may perceive the object as being "three-dimensional" due to a combination of vergence-divergence and accommodation. As previously mentioned, vergence-divergence movements of the two eyes relative to one another (e.g., rotation of the eyes such that the pupils move toward or away from one another, converging the gaze of the eyes to fixate on an object) are closely linked to accommodation of the eye's lenses. Under normal conditions, changing the shape of the eye's lenses and shifting focus from one object to another at a different distance will automatically produce a matching change in vergence to the same distance, in a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in vergence will trigger a matching change in lens shape under normal conditions.

[0138] 4B, an example of different accommodation and convergence-divergence states of the eyes is illustrated. Paired eye 222a fixates an object at optical infinity, while paired eye 222b fixates an object 221 at less than optical infinity. Notably, the convergence-divergence states of each pair of eyes are different, paired eye 222a points straight ahead, while paired eye 222 converges on object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of the lenses 210a, 220a.

[0139] Unfortunately, many users of conventional "3-D" display systems may find such conventional systems uncomfortable or may not perceive any depth sensation due to a mismatch between accommodation and vergence states in these displays. As previously mentioned, many stereoscopic or "3-D" display systems display a scene by providing a slightly different image to each eye. Such systems are uncomfortable for many viewers because, among other things, they simply provide a different presentation of the scene, causing changes in the vergence states of the eyes, but without a corresponding change in the accommodation states of those eyes. Rather, images are presented by the display at a fixed distance from the eyes, such that the eyes view all image information in a single accommodation state. Such an arrangement counters the "accommodation-vergence-divergence reflex" by causing changes in vergence states without a matching change in accommodation state. This mismatch is believed to cause viewer discomfort. By providing better matching between accommodation and convergence-divergence movements, display systems may form a more realistic and comfortable simulation of three-dimensional images.

[0140] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. In some embodiments, the different presentations may provide both vergence cues and matching cues for accommodation, thereby providing physiologically correct accommodation-vergence divergence matching.

[0141] 4B, two depth planes 240 are illustrated, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, vergence-divergence cues may be provided by displaying appropriately different perspective images for each eye 210, 220. Additionally, for a given depth plane 240, the light forming the image provided to each eye 210, 220 may have a wavefront divergence corresponding to the light field generated by a point at the distance of that depth plane 240.

[0142] In the illustrated embodiment, the distance along the z-axis of the depth plane 240 containing the point 221 is 1 m. As used herein, the distance or depth along the z-axis may be measured with the zero point located at the exit pupil of the user's eye. Thus, the depth plane 240 located at a depth of 1 m corresponds to a distance of 1 m away from the exit pupil of the user's eye on the optical axis of those eyes. As an approximation, the depth or distance along the z-axis may be measured from the display (e.g., the surface of the waveguide) in front of the user's eye, and a value for the distance between the device and the exit pupil of the user's eye with the eye pointed towards optical infinity may be added. That value may be called the pupil distance and may correspond to the distance between the exit pupil of the user's eye and the display worn by the user in front of the eye. In practice, the value for the pupil distance may be a normalized value that is generally used for all viewers. For example, the pupil distance may be assumed to be 20 mm, and the depth plane at a depth of 1 m may be at a distance of 980 mm in front of the display.

[0143] 4C and 4D, examples of matched accommodation-vergence-divergence distances and mismatched accommodation-vergence-divergence distances are illustrated, respectively. As illustrated in FIG. 4C, the display system may provide an image of a virtual object to each eye 210, 220. The image may cause the eyes 210, 220 to assume a convergence-divergence state in which the eyes converge on point 15 on the depth plane 240. In addition, the image may be formed by light having a wavefront curvature corresponding to a real object in that depth plane 240. As a result, the eyes 210, 220 assume an accommodation state in which the image is focused on the retinas of those eyes. Thus, the user may perceive the virtual object as being at point 15 on the depth plane 240.

[0144] It should be appreciated that each of the accommodation and convergence states of the eyes 210, 220 is associated with a particular distance on the z-axis. For example, an object at a particular distance from the eyes 210, 220 will cause those eyes to assume a particular accommodation state based on the distance of the object. The distance associated with a particular accommodation state is referred to as the accommodation distance A. d Similarly, a particular convergence distance V associated with the eyes, and in particular their convergence state or position relative to one another, may be referred to as d If the accommodation distance and the vergence distance match, the relationship between accommodation and vergence can be said to be physiologically correct. This is considered the most comfortable scenario for the viewer.

[0145] However, in a stereoscopic display, the accommodation distance and the convergence distance may not always match. For example, as illustrated in FIG. 4D, the images displayed to the eyes 210, 220 may be displayed with a wavefront divergence corresponding to the depth plane 240, and the eyes 210, 220 may be in a particular accommodation state in which the points 15a, 15b on that depth plane are in focus. However, the images displayed to the eyes 210, 220 may provide a convergence cue that causes the eyes 210, 220 to converge on a point 15 that is not located on the depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to a distance from a particular reference point of the user (e.g., the exit pupil of the eyes 210, 220) to the depth plane 240, while the convergence distance corresponds to a larger distance from that reference point to point 15. Thus, the accommodation distance differs from the convergence distance and an accommodation-divergence mismatch exists. Such a mismatch may be considered undesirable and may cause discomfort to the user. The mismatch may be caused by the distance (e.g., V d -A d ), and may be characterized using diopters (units of reciprocal length 1 / m). For example, a V of 1.75 diopters corresponds to d and 1.25 diopters A d or 1.25 diopters V d and 1.75 diopters A d would provide an accommodation-vergence-divergence mismatch of 0.5 diopters.

[0146] It should be understood that in some embodiments, a reference point other than the exit pupil of the eye 210, 220 may be utilized to determine the distance for determining the accommodation-vergence-divergence mismatch, so long as the same reference point is utilized for the accommodation distance and the vergence-divergence distance. For example, the distance may be measured from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., a waveguide in a display device) to the depth plane, etc.

[0147] Without being limited by theory, it is believed that a user may still perceive an accommodation-vergence-divergence mismatch of up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters as physiologically correct without the mismatch itself causing significant discomfort. In some embodiments, a display system disclosed herein (e.g., display system 250, FIG. 6) presents an image to a viewer having an accommodation-vergence-divergence mismatch of about 0.5 diopters or less. In some other embodiments, the accommodation-vergence-divergence mismatch of an image provided by the display system is about 0.33 diopters or less. In still other embodiments, the accommodation-vergence-divergence mismatch of an image provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0148] FIG. 5 illustrates aspects of an approach for simulating a three-dimensional image by modifying wavefront divergence. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to a user's eye 210. The waveguide 270 may output light 650 with a defined amount of wavefront divergence that corresponds to the wavefront divergence of a light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on that depth plane. In addition, the user's other eye will be illustrated as being provided with image information from a similar waveguide.

[0149] In some embodiments, a single waveguide may be configured to output light with a set wavefront divergence corresponding to a single or limited number of depth planes, and / or the waveguide may be configured to output light of a limited range of wavelengths. As a result, in some embodiments, multiple or stacked waveguides may be utilized to provide different wavefront divergence for different depth planes and / or output light of different ranges of wavelengths. As used herein, it should be understood that a depth plane may follow the contour of a flat or curved surface. In some embodiments, for convenience, a depth plane may follow the contour of a flat surface.

[0150] 6 illustrates an example of a waveguide stack for outputting image information to a user. The display system 250 includes a stack of waveguides or a stacked waveguide assembly 260 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. It should be understood that the display system 250 may be considered a light field display in some embodiments. Additionally, the waveguide assembly 260 may also be referred to as an eyepiece.

[0151] In some embodiments, the display system 250 may be configured to provide a substantially continuous cue for convergence and multiple discrete cues for accommodation. The cues for convergence may be provided by displaying different images to each of the user's eyes, and the cues for accommodation may be provided by outputting light forming images with selectable discrete amounts of wavefront divergence. In other words, the display system 250 may be configured to output light with variable levels of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a particular depth plane and may be provided by a particular one of the waveguides 270, 280, 290, 300, 310.

[0152] Continuing with reference to FIG. 6, the waveguide assembly 260 may also include a number of features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the number of lenses 320, 330, 340, 350 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image input devices 360, 370, 380, 390, 400 may act as light sources for the waveguides and may be utilized to input image information into the waveguides 270, 280, 290, 300, 310, each of which may be configured to disperse incident light across each individual waveguide for output towards the eye 210, as described herein. Light exits output surfaces 410, 420, 430, 440, 450 of the image input devices 360, 370, 380, 390, 400 and is input into corresponding input surfaces 460, 470, 480, 490, 500 of the waveguides 270, 280, 290, 300, 310. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide or may be a portion of a major surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide, outputting a total field of cloned collimated beams that are directed toward the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of the image launch devices 360, 370, 380, 390, 400 may be associated with and launch light into multiple (e.g., three) waveguides 270, 280, 290, 300, 310.

[0153] In some embodiments, each of the image input devices 360, 370, 380, 390, 400 is a discrete display that generates image information for input into the corresponding waveguides 270, 280, 290, 300, 310, respectively. In some other embodiments, the image input devices 360, 370, 380, 390, 400 are the output of a single multiplexed display that may, for example, send image information via one or more optical conduits (such as fiber optic cables) to each of the image input devices 360, 370, 380, 390, 400. It should be understood that the image information provided by the image input devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different primary colors as discussed herein).

[0154] In some embodiments, the light injected into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520, which includes a light module 530, which may include a light emitter such as a light emitting diode (LED). The light from the light module 530 may be directed and modified by a light modulator 540, e.g., a spatial light modulator, via a beam splitter 550. The light modulator 540 may be configured to vary the perceived intensity of the light injected into the waveguides 270, 280, 290, 300, 310 and encode the light with image information. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays. It should be understood that image injection devices 360, 370, 380, 390, 400 are illustrated diagrammatically and in some embodiments these image injection devices may represent different light paths and locations within a common projection system configured to output light into associated ones of waveguides 270, 280, 290, 300, 310. In some embodiments, the waveguides of the waveguide assembly 260 may act as ideal lenses, relaying light injected into the waveguides to the user's eye. In this concept, the object may be a spatial light modulator 540 and the image may be an image on a depth plane.

[0155] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, spiral scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately to the viewer's eye 210. In some embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent a single scanning fiber or a bundle of scanning fibers configured to inject light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may diagrammatically represent multiple scanning fibers or multiple bundles of scanning fibers, each configured to inject light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or fibers and one or more of the waveguides 270, 280, 290, 300, 310, for example, to redirect light exiting the scanning fiber into one or more of the waveguides 270, 280, 290, 300, 310.

[0156] The controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image input devices 360, 370, 380, 390, 400, the light source 530, and the light module 540. In some embodiments, the controller 560 is part of the local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transient medium) that coordinates the timing and provisioning of image information to the waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. The controller 560 may be part of the processing module 140 or 150 (FIG. 9D) in some embodiments.

[0157] Continuing with reference to FIG. 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Each of the waveguides 270, 280, 290, 300, 310 may be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between their major top and bottom surfaces. In the illustrated configuration, each of the waveguides 270, 280, 290, 300, 310 may include out-coupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguides by redirecting the light and propagating it within each individual waveguide and outputting image information from the waveguides to the eye 210. The extracted light may also be referred to as out-coupled light, and the out-coupling optical elements may also be referred to as light extraction optical elements. The extracted beam of light may be output by the waveguide at a location where the light propagating within the waveguide strikes the light extraction optical element. The outcoupling optical element 570, 580, 590, 600, 610 may be a grating, for example, including diffractive optical features as further discussed herein. Although shown disposed on the bottom major surface of the waveguide 270, 280, 290, 300, 310 for ease of explanation and clarity of drawing, in some embodiments the outcoupling optical element 570, 580, 590, 600, 610 may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguide 270, 280, 290, 300, 310, as further discussed herein. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 may be formed within a layer of material that is attached to a transparent substrate and forms the waveguide 270, 280, 290, 300, 310. In some other embodiments, the waveguide 270, 280, 290, 300, 310 may be a monolithic piece of material and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on and / or within that piece of material.

[0158] With continued reference to FIG. 6, as discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (injected into such waveguide 270) to the eye 210. The collimated light may represent an optical infinity focal plane. The next upper waveguide 280 may be configured to send collimated light that passes through a first lens 350 (e.g., a negative lens) before it can reach the eye 210. Such first lens 350 may be configured to generate a slight convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 280 as originating from a first focal plane closer inward from optical infinity toward the eye 210. Similarly, the third upper waveguide 290 passes its output light through both the first 350 and second 340 lenses before reaching the eye 210. The combined refractive power of the first 350 and second 340 lenses may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 290 as originating from a second focal plane closer inward toward the person from optical infinity where the light from the next upper waveguide 280 was.

[0159] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate the stack of lenses 320, 330, 340, 350 when viewing / interpreting light originating from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 320, 330, 340, 350 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the outcoupling optical elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0160] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same depth plane, or multiple subsets of the waveguides 270, 280, 290, 300, 310 may be configured to output images set at the same depth planes, with one set per depth plane. This may provide the advantage of forming tiled images to provide an extended field of view at those depth planes.

[0161] Continuing with reference to FIG. 6, the outcoupling optical elements 570, 580, 590, 600, 610 may be configured to redirect light from its respective waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of outcoupling optical elements 570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 may be volume or surface features that may be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers and / or structures for forming air gaps).

[0162] In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a sufficiently low diffraction efficiency such that only a portion of the light in the beam is deflected towards the eye 210 with each intersection point of the DOE, while the remainder continues to travel through the waveguide via TIR. The light carrying the image information is thus split into several related exit beams that exit the waveguide at various locations, resulting in a very uniform pattern of exit emission towards the eye 210 for this particular collimated beam bouncing within the waveguide.

[0163] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0164] In some embodiments, a camera assembly 630 (e.g., a digital camera, including visible and infrared light cameras) may be provided to capture images of the eye 210 and / or tissue surrounding the eye 210, for example, to detect user input and / or monitor a physiological condition of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source that projects light (e.g., infrared light) onto the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 630 may be mounted to the frame 80 (FIG. 9D) and may be in electrical communication with the processing modules 140 and / or 150, which may process image information from the camera assembly 630. In some embodiments, one camera assembly 630 may be utilized per eye, monitoring each eye separately.

[0165] 7, an example of an output beam output by a waveguide is shown. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 260 (FIG. 6) may function similarly and that the waveguide assembly 260 includes multiple waveguides. Light 640 is launched into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 impinges on the DOE 570, a portion of the light exits the waveguide as an output beam 650. The output beam 650 is shown as approximately parallel, but may be redirected to propagate to the eye 210 at an angle (e.g., divergent output beam formation) as discussed herein and depending on the depth plane associated with the waveguide 270. It should be understood that a nearly collimated exit beam may refer to a waveguide with outcoupling optics that outcouples light to form an image that appears to be set at a depth plane at a large distance (e.g., optical infinity) from the eye 210. Other waveguides or other sets of outcoupling optics may output an exit beam pattern that is more divergent, which would require the eye 210 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0166] In some embodiments, a full color image may be formed at each depth plane by overlaying an image on each of the primary colors, for example, three or more primary colors. FIG. 8 illustrates an example of a stacked waveguide assembly, where each depth plane includes an image formed using multiple different primary colors. The illustrated embodiment shows depth planes 240a-240f, but more or less depths are also contemplated. Each depth plane may have three or more primary color images associated with it, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. The different depth planes are illustrated by different numbers for diopters (dpt) following the letters G, R, and B. By way of example only, the numbers following each of these letters indicate the diopters (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the diagram represents an individual primary color image. In some embodiments, the exact location of the depth planes for the different primary colors may be varied to account for differences in the eye's focusing of different wavelengths of light. For example, different primary color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or reduce chromatic aberration.

[0167] In some embodiments, the light of each primary color may be output by a single dedicated waveguide, such that each depth plane may have multiple waveguides associated with it. In such embodiments, each box in the figure containing the letter G, R, or B may be understood to represent an individual waveguide, and three waveguides may be provided per depth plane, with three primary color images provided per depth plane. The waveguides associated with each depth plane are shown adjacent to each other in this drawing for ease of illustration, but it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other embodiments, multiple primary colors may be output by the same waveguide, such that, for example, only a single waveguide may be provided per depth plane.

[0168] 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may be used in addition to or in place of one or more of the red, green, or blue colors.

[0169] It should be understood that references throughout this disclosure to a given color of light are understood to encompass light of one or more wavelengths within the range of wavelengths of light that are perceived by a viewer as that given color. For example, red light may include one or more wavelengths of light that are within the range of about 620-780 nm, green light may include one or more wavelengths of light that are within the range of about 492-577 nm, and blue light may include one or more wavelengths of light that are within the range of about 435-493 nm.

[0170] In some embodiments, the light source 530 (FIG. 6) may be configured to emit light at one or more wavelengths outside the range of visual perception of a viewer, e.g., infrared and / or ultraviolet wavelengths. Additionally, the waveguide in-coupling, out-coupling, and other light redirecting structures of the display 250 may be configured to direct and emit this light from the display toward the user's eye 210, e.g., for imaging and / or user stimulation applications.

[0171] 9A, in some embodiments, light impinging on a waveguide may need to be redirected to in-couple the light into the waveguide. An in-coupling optical element may be used to redirect and in-couple the light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an example of a plurality or set 660 of stacked waveguides, each including an in-coupling optical element. Each of the waveguides may be configured to output light of one or more different wavelengths or one or more different wavelength ranges. The stack 660 may correspond to the stack 260 (FIG. 6), and the illustrated waveguides of the stack 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310, but it should be understood that light from one or more of the image input devices 360, 370, 380, 390, 400 is launched into the waveguide from a position that requires the light to be redirected for in-coupling.

[0172] The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated incoupling optical element (which may also be referred to as the light input area on the waveguide), for example, incoupling optical element 700 is disposed on a major surface (e.g., the top major surface) of waveguide 670, incoupling optical element 710 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and incoupling optical element 720 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be disposed on a bottom major surface of an individual waveguide 670, 680, 690 (particularly, one or more of the incoupling optical elements are reflective polarizing optical elements). As shown, the incoupling optical elements 700, 710, 720 may be disposed on the upper major surface of the respective waveguide 670, 680, 690 (or on top of the next lower waveguide), and in particular, the incoupling optical elements are transmissive deflecting optical elements. In some embodiments, the incoupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguide 670, 680, 690. In some embodiments, as discussed herein, the incoupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of the respective waveguide 670, 680, 690, it should be understood that the incoupling optical elements 700, 710, 720 may be disposed within other areas of the respective waveguide 670, 680, 690 in some embodiments.

[0173] As shown, the in-coupling optical elements 700, 710, 720 may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset to receive light without that light passing through another in-coupling optical element. For example, each in-coupling optical element 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400 as shown in FIG. 6 and may be separated (e.g., laterally spaced) from the other in-coupling optical elements 700, 710, 720 so as to not substantially receive light from others of the in-coupling optical elements 700, 710, 720.

[0174] Each waveguide also includes an associated optically dispersive element, for example, optically dispersive element 730 is disposed on a major surface (e.g., a top major surface) of waveguide 670, optically dispersive element 740 is disposed on a major surface (e.g., a top major surface) of waveguide 680, and optically dispersive element 750 is disposed on a major surface (e.g., a top major surface) of waveguide 690. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on the bottom major surfaces of associated waveguides 670, 680, 690, respectively. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of associated waveguides 670, 680, 690, respectively, or optically dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

[0175] The waveguides 670, 680, 690 may be spaced apart and separated by, for example, gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate the waveguides 670 and 680, and layer 760b may separate the waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low index material (i.e., a material having a lower index of refraction than the material forming the immediate neighbors of the waveguides 670, 680, 690). Preferably, the index of refraction of the material forming layers 760a, 760b is 0.05 or more, or 0.10 or less, than the index of refraction of the material forming the waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may act as cladding layers to promote total internal reflection (TIR) ​​of light through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include immediate cladding layers.

[0176] Preferably, for ease of manufacturing and other considerations, the materials forming the waveguides 670, 680, 690 are similar or the same, and the materials forming the layers 760a, 760b are similar or the same. In some embodiments, the materials forming the waveguides 670, 680, 690 may differ between one or more of the waveguides, and / or the materials forming the layers 760a, 760b may differ while still maintaining the various refractive index relationships discussed above.

[0177] 9A, light rays 770, 780, 790 are incident on the set of waveguides 660. It should be understood that light rays 770, 780, 790 may be injected into the waveguides 670, 680, 690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG. 6).

[0178] In some embodiments, the light beams 770, 780, 790 have different properties, e.g., different wavelengths or different wavelength ranges, that may correspond to different colors. Each of the incoupling optical elements 700, 710, 720 deflects the incident light such that the light propagates through a respective one of the waveguides 670, 680, 690 by TIR. In some embodiments, each of the incoupling optical elements 700, 710, 720 selectively deflects one or more particular wavelengths of light while transmitting other wavelengths to the underlying waveguide and associated incoupling optical element.

[0179] For example, incoupling optical element 700 may be configured to deflect light beam 770 having a first wavelength or wavelength range while transmitting light beams 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light beam 780 impinges on and is deflected by incoupling optical element 710, which is configured to deflect light of the second wavelength or wavelength range. Light beam 790 is deflected by incoupling optical element 720, which is configured to selectively deflect light of the third wavelength or wavelength range.

[0180] 9A, the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the incoupling optical element 700, 710, 720 of each waveguide deflects the light into its corresponding waveguide 670, 680, 690 and incouplings the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the respective waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the respective waveguides 670, 680, 690 by TIR until they impinge on the corresponding optical dispersive element 730, 740, 750 of the waveguide.

[0181] 9B, a perspective view of the multiple stacked waveguide embodiment of FIG. 9A is illustrated. As previously described, the in-coupled light rays 770, 780, 790 are deflected by the in-coupling optical elements 700, 710, 720, respectively, and then propagate by TIR within the waveguides 670, 680, 690, respectively. The light rays 770, 780, 790 then impinge on the optically dispersive elements 730, 740, 750, respectively. The optically dispersive elements 730, 740, 750 deflect the light rays 770, 780, 790 to propagate towards the out-coupling optical elements 800, 810, 820, respectively.

[0182] In some embodiments, the optically dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse the light to the out-coupling optical elements 800, 810, 820, and in some embodiments may also increase the beam or spot size of this light as it propagates to the out-coupling optical elements. In some embodiments, the optically dispersive elements 730, 740, 750 may be omitted and the in-coupling optical elements 700, 710, 720 may be configured to deflect the light directly to the out-coupling optical elements 800, 810, 820. For example, referring to FIG. 9A, the optically dispersive elements 730, 740, 750 may be replaced with the out-coupling optical elements 800, 810, 820, respectively. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light to the viewer's eye 210 (FIG. 7). It should be understood that the OPEs may be configured to increase the size of the eyebox in at least one axis, and the EPEs may increase the eyebox in an axis that intersects, e.g., orthogonal to, the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light striking the OPE to an EPE of the same waveguide, while allowing the remaining portion of the light to continue propagating down the waveguide. In response to striking the OPE, again, another portion of the remaining light is redirected to the EPE, which continues to propagate further down the waveguide, etc. Similarly, in response to striking the EPE, a portion of the striking light is directed out of the waveguide towards the user, and the remaining portion of the light continues to propagate through the waveguide until it strikes the EP again, at which point another portion of the striking light is directed out of the waveguide, etc. As a result, a single beam of internally coupled light may be "replicated" each time a portion of that light is redirected by an OPE or EPE, thereby forming a cloned beam field of light, as shown in Figure 6. In some embodiments, the OPE and / or EPE may be configured to modify the size of the beam of light.

[0183] Thus, referring to Figures 9A and 9B, in some embodiments, a set of waveguides 660 includes, for each primary color, a waveguide 670, 680, 690, an in-coupling optical element 700, 710, 720, an optical dispersion element (e.g., OPE) 730, 740, 750, and an out-coupling optical element (e.g., EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer between each one. The in-coupling optical element 700, 710, 720 redirects or deflects the incoming light into its waveguide (with different in-coupling optical elements receiving different wavelengths of light). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the illustrated embodiment, light ray 770 (e.g., blue light) is deflected by the first in-coupling optical element 700 in the manner described above, then continues to bounce back down the waveguide, interacting with the optically dispersive element (e.g., OPE) 730 and then the out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through the waveguide 670, where light ray 780 impinges on and is deflected by the in-coupling optical element 710. Light ray 780 will then proceed via TIR back down the waveguide 680, to its optically dispersive element (e.g., OPE) 740 and then the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the optically in-coupling optical element 720 of the waveguide 690. The light in-coupling optical element 720 deflects the light beam 790 so that it propagates by TIR to a light dispersive element (e.g., OPE) 750 and then to an out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light beam 790 to a viewer, who also receives the out-coupled light from the other waveguides 670, 680.

[0184] FIG. 9C illustrates a top-down plan view of an example of the multiple stacked waveguides of FIGS. 9A and 9B. As shown, the waveguides 670, 680, 690 may be vertically aligned with each waveguide's associated light dispersive element 730, 740, 750 and associated out-coupling optical elements 800, 810, 820. However, as discussed herein, the in-coupling optical elements 700, 710, 720 are not vertically aligned. Rather, the in-coupling optical elements are preferably non-overlapping (e.g., laterally spaced apart, as seen in the top-down view). As discussed further herein, this non-overlapping spatial arrangement facilitates the injection of light from different sources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, an arrangement including non-overlapping spatially separated in-coupling optical elements may be referred to as a shifted pupil system, and the in-coupling optical elements in these arrangements may correspond to sub-pupils.

[0185] 9D illustrates an example of a wearable display system 60 into which the various waveguide and associated systems disclosed herein may be integrated. In some embodiments, the display system 60 is the system 250 of FIG. 6, which diagrammatically illustrates some portions of the system 60 in greater detail. For example, the waveguide assembly 260 of FIG. 6 may be part of the display 70.

[0186] 9D , the display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functionality of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and configured to position the display 70 in front of the eye of the user 90. The display 70 may be considered an eyepiece in some embodiments. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the ear canal of the user 90 (in some embodiments, another speaker, not shown, may also be optionally positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display system 60 may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones may be configured to allow the user to provide input or commands to the system 60 (e.g., voice menu command selections, natural language queries, etc.) and / or enable audio communication with other persons (e.g., other users of a similar display system). The microphone may further be configured as an ambient sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system 60 may further include one or more outwardly directed environmental sensors 112 configured to detect light, objects, stimuli, people, animals, places, or other aspects of the world around the user. For example, the environmental sensors 112 may include one or more cameras, which may be positioned, for example, facing outward, to capture images similar to at least a portion of the user 90's normal field of view. In some embodiments, the display system may also include an ambient sensor 120a, which may be separate from the frame 80 and mounted on the user 90's body (e.g., the user's 90's head, torso, limbs, etc.). The ambient sensor 120a may, in some embodiments, be configured to obtain data characterizing the user 90's physiological state.For example, the sensor 120a may be an electrode.

[0187] 9D, the display 70 is operably coupled to the local data processing module 140 by a communication link 130, such as wired or wireless connectivity, which may be mounted in a variety of configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded within headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration, in a belt-connected configuration). Similarly, the sensor 120a may be operably coupled to the local data processing module 140 by a communication link 120b, such as wired or wireless connectivity. The local processing and data module 140 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. Optionally, the local processing and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and the like. The data may include a) captured from sensors (such as image capture devices (cameras, etc.), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein (e.g., which may be operatively coupled to the frame 80 or otherwise attached to the user 90)) and / or b) data obtained and / or processed using a remote processing module 150 and / or a remote data repository 160 (including data related to the virtual content), possibly for passage to the display 70 after processing or retrieval. The local processing and data module 140 may be operatively coupled to the remote processing module 150 and the remote data repository 160 by communication links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to the local processing and data module 140.In some embodiments, local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted on frame 80 or may be a stand-alone structure that communicates with local processing and data module 140 by a wired or wireless communication path.

[0188] 9D , in some embodiments, remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, remote data repository 160 may comprise digital data storage facilities that may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, remote data repository 160 may include one or more remote servers, which provide information, for example, information for generating augmented reality content to local processing and data module 140 and / or remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote modules. Optionally, an external system (e.g., one or more processors, one or more computer systems), including a CPU, GPU, etc., may perform at least a portion of the processing (e.g., generating image information, processing data) and provide information to and receive information from modules 140, 150, 160, e.g., via a wireless or wired connection. I. Depth-based quality adjustment

[0189] As described herein, a display system (e.g., an augmented reality display system such as display system 60 of FIG. 9D ), according to various embodiments, may determine a three-dimensional fixation point of a user, for example, by monitoring the user's eyes. The fixation point may indicate the location of a point in space along (1) an x-axis (e.g., a lateral axis), (2) a y-axis (e.g., a vertical axis), and (3) a z-axis (e.g., the depth of the point, e.g., depth from the user). In some embodiments, the display system may utilize cameras, sensors, etc. to monitor the user's eyes (e.g., the pupils, cornea, etc. of each eye) and determine the line of sight of each eye. The line of sight of each eye may generally be understood to be a vector extending from the center of the retina of that eye through the lens of the eye. For example, the vector may generally extend from the center of the macula (e.g., the fovea) through the lens of the eye. The display system may be configured to determine where vectors associated with the eyes intersect, and this intersection point may be understood to be the fixation point of the eye. In other words, the fixation point may be a location in three-dimensional space where the user's eyes are converged. In some embodiments, the display system may filter out slight movements of the user's eyes, for example during fast movements (e.g., saccades, microsaccades), and may update the fixation point in response to determining that the eyes are fixating on a location in three-dimensional space. For example, the display system may be configured to ignore eye movements that fixate on a point for less than a threshold duration.

[0190] The resolution of content, such as a virtual object or content, presented by the display system may be adjusted based on its proximity to a fixation point, as discussed herein. It should be understood that the display system may store therein or have access to information regarding the location of the virtual object in three-dimensional space. Based on the known location of the virtual object, the proximity of a given virtual object to the fixation point may be determined. For example, the proximity of the virtual object to the fixation point may be determined by determining one or more of: (1) the three-dimensional distance of the virtual object from the user's fixation point; (2) if the display frustum of the display system is divided into resolution adjustment zones, the resolution adjustment zone in which the virtual object is located relative to the resolution adjustment zone in which the fixation point is located; and (3) the angular separation between the virtual object and the user's line of sight. Virtual content that is closer to the fixation point may be presented with a higher resolution than content that is farther from the fixation point. In some embodiments, the resolution of the virtual content varies depending on the depth plane in which the virtual content is located relative to the fixation point or the proximity of the depth plane in which the fixation point is located. In some embodiments, the adjustment to the resolution may be made by one or more graphics processing units, e.g., a rendering engine, such as a rendering engine included within one or more of modules 140, 150 (FIG. 9D).

[0191] FIG. 10A illustrates an example of a top-down representation of a user viewing content (e.g., content contained within a display frustum 1004) presented by a display system (e.g., display system 60, FIG. 9D). The representation includes the user's eyes 210, 220 and a determination of a fixation point 1006 for the eyes 210, 220. As shown, the line of sight of each eye is represented as a vector (e.g., vectors 1003A, 1003B), and the display system detects the fixation point 1006, for example, by determining where the vectors converge in front of the eyes 210, 22. In the illustrated example, the fixation point 1006 coincides with the location of a first virtual object 1008A presented by the display system. Examples of systems and methods for eye tracking may be found in U.S. Patent Application Serial No. 14 / 690,401, filed April 18, 2015 (incorporated by reference for all purposes) and the accompanying appendices. For example, eye tracking systems and methods may be utilized for eye tracking and / or to determine fixation points as described at least in Figures 25-27 of the Appendix and at least in part as described herein.

[0192] Continuing with reference to FIG. 10A, a second virtual object 1008B is also presented by the display system within the display frustum 1004. A view of these virtual objects 1008A, 1008B as seen by a viewer is shown in a rendered frame 1010. The rendered frame 1010 may include the first virtual object 1008A rendered at a first resolution, while the second virtual object 1008B, located away from the fixation point 1006, is rendered at a second, lower resolution. Specifically, the second virtual object 1008B may be determined to be at a greater depth than the first virtual object 1008A and toward a side thereof. For example, the display system may determine the depth of the second virtual object 1008B as discussed herein, or optionally, a content provider associated with the virtual content may indicate a virtual object depth that the display system may utilize to render the virtual object. Thus, the fixation point 1006, as described above, describes the three-dimensional location in space where the user is looking, and the second virtual object 1008B can be determined to be located at a depth farther from the user and displaced laterally from the fixation point 1006.

[0193] Without being limited by theory, it is believed that when the user's eyes 210, 220 are looking at the first virtual object 1008A, the image of the first virtual object 1008A may fall into the user's fovea, while the image of the second virtual object 1008B does not fall into the fovea. As a result, the second virtual object 1008B may be reduced in resolution without a significant impact on the perceived image quality of the display system due to the lower sensitivity of the human visual system to the second virtual object 1008B. In addition, the lower resolution advantageously reduces the computational load required to provide the image. As discussed herein, the resolution at which the second virtual object 1008B is rendered may be based on the proximity to the fixation point 1006, and the reduction in resolution (e.g., relative to the resolution of the first virtual object 1008A) may increase with decreasing proximity (or increasing distance) between the fixation point 1006 and the virtual object 1008A. In some embodiments, the rate of decrease in resolution may be based on the rate of decrease in density of cones in the human eye or the drop in visual acuity as one moves away from the fovea.

[0194] It should be understood that the resolution of various virtual objects presented by the display system may vary dynamically as the fixation point changes location. For example, FIG. 10B illustrates another example of a top-down view representation of a user viewing content presented by a display system. As illustrated in FIG. 10B, the user is now focused on a second virtual object 1008B, compared to FIG. 10A, where the user was focused on a first virtual object 1008A. By monitoring the user's gaze 1003A, 1003B, the display system determines that the eyes 210, 220 are converged on the second virtual object 1008B and sets that location as the new fixation point 1006.

[0195] In response to detecting this change in location of the fixation point 1006, the display system now renders the second virtual object 1008B at a higher resolution than the first virtual object 1008A, as shown in rendered frame 1010. Preferably, the display system monitors the user's line of sight 1003A, 1003B with a sufficiently high frequency to change the resolution of the virtual objects rapidly enough that the transition in resolution of the first virtual object 1008A and the second virtual object 1008B is substantially imperceptible to the user.

[0196] 10C illustrates another example of a top-down view representation of a user viewing content via a display system (e.g., display system 60, FIG. 9D). In the example, the user's field of view 1004 is illustrated along with a fixation point 1006. Three virtual objects are illustrated, with a first virtual object 1012A proxied closer to the fixation point 1006 than a second virtual object 1012B or a third virtual object 1012C. Similarly, the second virtual object 1012B is illustrated proxied closer to the fixation point 1006 than the third virtual object 1012C. Thus, when the virtual objects 1012A-1012C are presented to a user, the display system may allocate resources such that rendering the first virtual object 1012A is given a greater resource allocation than the second virtual object 1012B (e.g., object 1012A is rendered at a higher resolution), which in turn receives a greater resource allocation than the third virtual object 1012C, which may optionally not be rendered at all because it is outside the field of view 1004.

[0197] Resolution adjustment zones are illustrated in the example of FIG. 10C, where the zones are elliptical (e.g., circular) shapes described along the depth and lateral axes. As illustrated, the fixation point 1006 is inside the central zone 1014A, and the first virtual object 1012A extends between the zones 1014B, 1014C and within the user's foveal cone of vision 1004a. The first virtual object 1012A may therefore be presented to the user at a resolution associated with the zone 1014B or 1014C, or, optionally, a portion of the object 1012A within the zone 1014B may be presented according to the resolution of the zone 1014B, and the remaining portion within the zone 1014C may be presented according to the resolution of the zone 1014C. For example, in an embodiment where the zones are assigned a resolution reduced from a maximum (e.g., highest) resolution, the first virtual object 1012A may be presented at the assigned resolution. Optionally, the first virtual object 1012A may be presented with either a resolution (e.g., the display system may be programmed to display at the highest resolution associated with any zone across which the first virtual object 1012A extends) or a measure of central tendency of the resolution (e.g., the measure may be weighted according to the extent to which the object 1012A is located within the zones 1014B, 1014C). With continued reference to FIG. 10C, it should be understood that the resolution adjustment zones at different distances from the fixation point 1006 may have different shapes. For example, zone 1014C may have a different shape than zones 1014A-1014C and conform to the contours of the field of view 1004. In some other embodiments, one or more of the zones 1014A-1014C may have a different shape than one or more others of the zones 1014A-1014C.

[0198] FIG. 10D is a block diagram of an exemplary display system. An exemplary display system (e.g., display system 60, FIG. 9D) may be an augmented reality display system and / or a mixed reality display system, which may adjust the use of rendering hardware resources according to a user's fixation point, as described herein. For example, as described above with respect to FIG. 10C, rendering hardware resources 1021 may be adjusted according to a user's fixation point. A resource arbiter 1020 may be implemented to adjust the use of such resources 1021, e.g., the arbiter 1020 may allocate the resources 1021 to a particular application process 1022 associated with presenting a virtual object to a user. The resource arbiter 1020 and / or the rendering hardware resources 1021 may optionally be included within the local processing and data module 140 (e.g., as illustrated in FIG. 9D) and / or the remote processing module 150 of the display system 60. For example, rendering hardware resources 1021 may comprise a graphics processing unit (GPU), which may be included within module 140 and / or module 150, as described above with respect to FIG. 9D.

[0199] As an example of adjusting resources 1021, with reference to FIG. 10C, a first virtual object 1012A associated with a first application process may be allocated a larger share of resources 1021 than a second virtual object 1012B associated with a second application process. A virtual object associated with an application process 1022 may be rendered based on the allocated resources 1021 and contained within a frame buffer 1024 to be composited (e.g., by a compositor 1026) into a final frame buffer 1028. The final frame buffer 1028 may then be presented by display hardware 1030, such as the display 70 illustrated in FIG. 9D, where the rendered virtual object is adjusted in resolution.

[0200] As disclosed herein, the resolution of a virtual object may be determined based on the proximity of the virtual object to a fixation point. In some embodiments, the resolution may be modified as a function of the distance between the virtual object and the fixation point. In some embodiments, the modification may occur in discrete steps. That is, a similar modification may be applied to all virtual objects located within a particular volume or zone. FIG. 11A1 illustrates an example of a top-down view representation of the adjustment in resolution in different resolution adjustment zones based on three-dimensional fixation tracking. The display system may divide the display frustum into multiple volumes or resolution adjustment zones and modify the resolution in discrete steps corresponding to these zones. Thus, in some embodiments, to determine the adjustment in resolution of the virtual content, the display system may utilize information describing the volumes of space (hereinafter referred to as resolution adjustment zones) and the assignment of resolution adjustments to each spatial volume. As illustrated, the field of view provided by the display system (e.g., the display frustum of the display) is separated into multiple different zones, each encompassing a range of depths from the user (e.g., depth ranges 1102A-1102E). In some embodiments, each depth range 1102A-1102E has a single associated depth plane that can be presented by the display system. With continued reference to FIG. 11A1, five zones encompass each identified depth range from the user and are contiguous along the lateral direction. In the illustrated exemplary top-down view, the field of view is divided into a grid 1100 of 25 zones. Each zone represents a volume of real-world space in which virtual content can be placed for the user.

[0201] It should be understood that the zones may also extend in this vertical direction (e.g., along the y-axis, not shown), such that the illustrated grid 1100 may be understood to represent one cross-section along the vertical direction. In some embodiments, multiple zones are also provided in the vertical direction. For example, there may be five vertical zones per depth range, i.e., a total of 125 resolution adjustment zones. An example of such zones extending in three dimensions is illustrated in FIG. 11B and described below.

[0202] 11A1, the user's eyes 210, 220 fixate on a particular fixation point 1006 within the grid 1100. The display system may determine the location of the fixation point 1006 and the zone in which the fixation point 1006 is located. The display system may adjust the resolution of the content based on the proximity of the virtual content to the fixation point 1006, which may include determining the proximity of the virtual content to the zone in which the fixation point 1006 is located. As an example, for the content contained within the zone in which the fixation point 1006 is located, the resolution may be set to a particular polygon count, in an example embodiment, 10,000 polygons. Based on the distance from the fixation point 1006, the content contained within the remaining zones may be adjusted accordingly. For example, content contained within zones adjacent to the zone containing the fixation point 1006 may be rendered at a lower resolution (e.g., 1,000 polygons). 11A1 illustrates, as an example, adjusting a polygon count as described herein, but adjusting the resolution may encompass making other modifications to the resolution of the presented content. For example, adjustments in resolution may include one or more of adjusting a polygon count, adjusting a primitive utilized to generate a virtual object (e.g., adjusting a shape of a primitive, e.g., adjusting a primitive from a triangular mesh to a quadrilateral mesh, etc.), adjusting an operation performed on the virtual object (e.g., a shader operation), adjusting texture information, color resolution, or depth, adjusting the number of rendering cycles or frame rate, and adjusting quality at one or more points in the graphics pipeline of a graphics processing unit (GPU).

[0203] In addition, while the example of FIG. 11A1 provides a particular example of the difference in polygon counts within different resolution adjustment zones, other absolute numbers of polygons and other rates of change of resolution with distance from fixation point 1006 are also contemplated. For example, the drop in resolution from fixation point 1006 may be based on a symmetric rate of drop about depth and lateral distance from fixation point 1006, although other drop relationships may also be utilized. For example, lateral distance from fixation point 1006 may be associated with a greater drop in resolution relative to depth distance from fixation point 1006. Additionally, the size of each zone included within the grid (e.g., the size of the zone's spatial volume) may optionally be different (e.g., the zones may vary radially from the foveal axis). In some embodiments, the drop may be continuous from fixation point 1006 such that a resolution relationship with a discrete zone having an assigned resolution or a zone containing fixation point 1006 is not utilized. For example, the descent from the fixation point 1006 to a particular zone 1108 (e.g., a zone whose content is rendered at a resolution of 100 polygons) may be modified to be a continuous descent from the fixation point 1006 to the edge of the grid (e.g., the edge of the particular zone 1108). It should be understood that each of the above considerations also apply to zones that extend vertically.

[0204] In some embodiments, the number and size of zones included within the grid may be based on a confidence level associated with the determination of the user's fixation point 1006. For example, the confidence level may be based on the amount of time the user's eye has been fixating on the fixation point 1006, with shorter amounts of time being associated with a lower confidence level. For example, the display system may monitor the user's eye at a particular sampling rate (e.g., 30 Hz, 60 Hz, 120 Hz, 1 kHz) and may increase the confidence level in the fixation point 1006 because successive samples indicate that the user is generally maintaining the fixation point 1006. Optionally, a particular threshold of fixation may be utilized, for example, fixations at the same, or similar, fixation point for a particular duration (e.g., 100-300 milliseconds) may be associated with a high confidence level, while anything less than a particular duration may be associated with a lower confidence level. Similarly, intraocular variations, such as pupil dilation, that may affect the determination of the user's fixation point may cause the display system to reduce the confidence level. It should be appreciated that the display system may monitor the eye using sensors such as a camera imaging device (e.g., camera assembly 630, FIG. 6). Optionally, the display system may utilize a combination of sensors to determine the user's eye gaze (e.g., different eye gaze determination processes may be utilized, such as an infrared sensor utilized to detect infrared reflection from the eye and identify the pupil, a visible light imaging device utilized to detect the iris of the eye, etc.). The display system may increase the confidence when multiple eye gaze determination processes are in agreement and may decrease the confidence level if they are in disagreement. Similarly, for a display system that performs only one eye gaze determination process, each eye gaze determination process may be associated with a particular confidence level (e.g., one determination process may be considered more accurate than the other), and the size of the resolution accommodation zone may be selected based, at least in part, on the process being implemented.

[0205] In some embodiments, the display system may increase or decrease the number of zones with each update of the fixation point 1006. For example, as the confidence associated with the fixation point 1006 increases, more zones may be utilized, and as the confidence decreases, fewer zones may be utilized. FIG. 11A2 illustrates an example of a top-down view representation of the resolution adjustment zones at different times as the size and number of zones change. At time t=1, as seen in the top-down view, the user's field of view may be divided into an initial set of zones. At time t=2, the confidence in the location of the fixation point 1006 increases, and the display system may also decrease the size of the zone occupied by the fixation point 1006 and rendered at high resolution. Optionally, as shown, the size of the other zones may also decrease. At time t=3, the confidence in the location of the fixation point 1006 decreases, and the display system may also increase the size of the zone occupied by the fixation point 1006 and rendered at high resolution. Optionally, as shown, the size of the other zones may also increase. It should be understood that the multiple zones may also extend on the y-axis, and similar increases or decreases in size and number of zones may also be enacted on that axis. For example, the size of zones extending vertically on the y-axis may decrease with increasing confidence, while the size may increase with decreasing confidence. Optionally, the display system may determine the confidence of the fixation point 1006 for each frame presented to the user by the display system, where t=1, t=2, and t=3 may represent different frames. Because allocating more zones may require an increase in computational power (e.g., the display system may need to adjust the resolution of more content, identify zones in which content is included, etc.), the display system may balance the increase in required computational power provided by the increase in the number of zones against the computational power savings provided by a potential decrease in the resolution of the content.

[0206] 11A1, the grid may be dynamically changing in the sense that the fixation point 1006 may be set to be located at the center (e.g., center of gravity) of the grid. Thus, the display system may avoid edge cases where the fixation point 1006 is determined to be located on a vertex of the grid. For example, as a user's eyes rotate and then fixate on different three-dimensional locations in space, the grid may similarly move with the user's line of sight.

[0207] 11B-11E illustrate examples of various resolution adjustment zone configurations. Additional shapes and configurations of resolution adjustment zones not shown may be utilized, and the examples should not be considered exhaustive. Additionally, in some figures, the user's eyes 210, 220 may be shown spaced apart from the various resolution adjustment zones for ease and clarity of illustration. It should be understood that with all these figures, the eyes 210, 220 may be located at the boundaries of the zones or within them (see, for example, FIG. 11A1).

[0208] FIG 11B illustrates an example of a three-dimensional representation of some of the resolution adjustment zones of FIG 11A1. It should be understood that FIG 11A1 illustrates a cross-sectional view taken along the plane 11A1-11A1 of the three-dimensional representation of FIG 11B, and FIG 11B omits some of the resolution adjustment zones of FIG 11A1 for clarity of illustration. With continued reference to FIG 11A1, the field of view provided by the display system is separated into 27 zones. That is, the field of view is separated into three depth ranges 1102B-1102D, and in each depth range, a 3x3 grid of zones is included that extend laterally and vertically in the depth range.

[0209] The determined fixation point 1006 is illustrated as being within a zone located in the center of the field of view. Virtual objects located within a zone outside the zone containing the fixation point 1006 may have their resolution reduced according to their distance from the fixation point 1006 zone, as discussed herein. Because the zones extend laterally and vertically, the reduction in resolution may occur based on the distance on the lateral, vertical, and depth axes (x, y, and z-axes, respectively) from the fixation point's resolution adjustment zone. For example, in some embodiments, virtual objects located in zone 1108 may have their resolution reduced according to the lateral distance, as shown in FIG. 11A1 (e.g., zone 1108 may include the same vertical portion of the user's field of view and be on the same depth plane as the zone containing the fixation point 1006).

[0210] Similar to the above, and similar to the zones described in Figures 11C-11E below, the user's fixation point can optionally be maintained at the center (e.g., center of gravity) of the zone, or the zone can be fixed relative to the user's field of view and the user's fixation point can be located within any of the zones.

[0211] FIG. 11C illustrates another example of a configuration for resolution adjustment zones. In an example, the field of view provided by the display system is illustrated as being separated into elliptical zones, each encompassing a particular three-dimensional volume of space. As in FIG. 11A1, each zone (e.g., zones 1112A-112D) extends along the lateral and depth dimensions. In some embodiments, each zone also extends to encompass at least a portion of the user's vertical field of view. A fixation point 1006 is illustrated as being at the center of the zone (e.g., in zone 1112A). Virtual objects located in zones outside of zone 1112A may have their resolution reduced according to their distance from zone 1112A, e.g., according to techniques described herein. For example, each zone outside of zone 1112A may be assigned a particular resolution, or a rate of descent may be utilized to determine the reduction in resolution. Zone 1112D is illustrated as the zone furthest from zone 1110A, and the reduction in resolution may be greatest in zone 1112D.

[0212] FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zones of FIG. 11C, with FIG. 11C showing a cross-sectional view taken along the plane 11C-11C. In this example, the field of view provided by the display system is illustrated as being separated into ellipsoidal zones, each encompassing a three-dimensional volume of space. The user's fixation point 1006 is illustrated at the center of gravity of the user's field of view and is located within zone 1112A. Optionally, FIG. 11D may represent each ellipse of FIG. 11C transformed into an ellipsoid. In some embodiments, the size of zone 1112A of FIG. 11C along the depth and lateral directions may define the size of the major axes of zone 1112A of FIG. 11D along the X and Z axes. The various zones may form concentric spheres or ellipsoids.

[0213] FIG. 11E illustrates another example of a three-dimensional representation of the resolution adjustment zones of FIG. 11C, where FIG. 11C shows a cross-sectional view taken along the plane 11C-11C. The field of view provided by the display system is illustrated as being separated into similar concentric zones of stacked levels. For example, FIG. 11E may show how the ellipse of FIG. 11C is extended along the vertical direction to create a cylinder. The cylinders can then be separated vertically such that each cylinder encompasses a portion of the user's vertical field of view. Thus, FIG. 11E illustrates nine zones of the cylinder. Each zone additionally excludes any interior zones (e.g., ellipsoid 1112B would encompass a volume of space that excludes the volume of space encompassed by ellipsoid 1112A). In an example, fixation point 1006 is illustrated as being within central zone 1110A, and virtual objects located outside central zone 1110A can have their resolution reduced according to the techniques described herein.

[0214] 12A illustrates a flowchart of an example process 1200 for adjusting the resolution of content according to its proximity to a three-dimensional fixation point. For convenience, the process 1200 may be described as being performed by a display system (e.g., a wearable display system 60, which may include processing hardware and software and, optionally, provide information to one or more computers or other external systems for processing, e.g., offload processing to the external systems and receive information from the external systems).

[0215] In block 1202, the display system determines the user's three-dimensional fixation point. As described above, the display system may include sensors to monitor information associated with the user's eye (e.g., eye orientation). A non-exhaustive list of sensors includes infrared sensors, ultraviolet sensors, and visible wavelength light sensors. The sensors may optionally output infrared, ultraviolet, and / or visible light onto the user's eye and determine a reflection of the output light from the user's eye. As an example, the infrared light may be output by an infrared light emitter and an infrared light sensor. It should be understood that the sensor, which may include a light emitter, may correspond to the imaging device 630 of FIG. 6.

[0216] The display system may utilize sensors to determine a line of sight associated with each eye (e.g., a vector extending from the user's eye as extending from the fovea through the eye's lens) and an intersection of the lines of sight for each eye. For example, the display system may output infrared light onto the user's eye, and a reflection from the eye (e.g., a corneal reflection) may be monitored. A vector between the pupil center of the eye (e.g., the display system may determine the pupil centroid, e.g., through infrared imaging) and the reflection from the eye may be used to determine the line of sight of the eye. The intersection of the lines of sight may be determined and assigned as a three-dimensional fixation point. The fixation point may thus indicate a location where content will be rendered at full or full resolution. For example, based on the determined lines of sight, the display system may triangulate a three-dimensional location in space where the user is fixating. Optionally, the display system may utilize orientation information associated with the display system (e.g., information describing the orientation of the display system in three-dimensional space) when determining the fixation point.

[0217] In block 1204, the display system obtains location information associated with content that is or will be presented to the user by the display system. Prior to rendering the content for presentation to the user (e.g., via the output of the waveguide, as described above), the display system may obtain location information associated with the content to be presented to the user. For example, as described above, virtual content may be presented to the user such that the content appears to be located in the real world (e.g., the content may be located at different depths within the user's field of view). It should be appreciated that the display system may include or have access to a three-dimensional map of the surrounding environment that may inform the location of any virtual content within the surrounding environment. With reference to this map, the display system may access and provide information that defines the three-dimensional location of the virtual content within the user's field of view (e.g., location within the display frustum, as illustrated in FIGS. 10A-10B).

[0218] In block 1206, the display system adjusts the resolution of the virtual content to be displayed to the user. The display system adjusts the resolution of the content based on its proximity to the three-dimensional fixation point. For example, a rendering engine, such as a rendering engine implemented by a processing device (e.g., a central processing unit, a graphics processing unit) that renders the content for presentation to the user, may adjust resources invested in rendering the content (e.g., the rendering engine may adjust the resolution of the content).

[0219] The display system may determine a distance in three-dimensional space between the content to be presented to the user and the user's fixation point, and may reduce the resolution of the content based on the determined distance. The reduction may be determined according to a rate of descent, e.g., a continuous function correlating distance and content resolution, and the display system may obtain the resolution and render the content based on the continuous function. Optionally, the display system may determine a distance from the center of gravity of the content to the fixation point, and may render the content at a resolution based on the distance. Optionally, the display system may render portions of the same content at different resolutions according to the distance of the various portions to the fixation point (e.g., the display system may separate the content into portions and render the more distant portions at a reduced resolution compared to the closer portions).

[0220] In some embodiments, the display system may access information that can be used to separate the user's field of view (e.g., corresponding to a display frustum) into zones, with each zone representing a volume of space in which content may be contained. The accessed information, e.g., the grid illustrated in FIG. 11A1, may indicate a particular resolution to utilize when rendering the content that will be contained within each zone, with the three-dimensional fixation point being set at the center of the grid. In addition, the grid may indicate a rate of descent in resolution to utilize when rendering the content. For content contained within multiple zones (e.g., content located within the three-dimensional space required by two zones), the display system may optionally adjust the resolution of the content to correspond to a single zone, or may optionally adjust the resolution of the content according to the corresponding zone in which the portion is located.

[0221] When setting the content resolution, the display system renders content located at the fixation point (e.g., in the same zone as the fixation point) at full or maximum resolution. The maximum resolution may be based on the maximum value that the display system hardware and / or software is capable of rendering while ensuring that the content is presented to the user at a refresh rate above a threshold (e.g., 60 Hz, 120 Hz) and, optionally, ensuring that the content is updated at a rate that exceeds vergence-divergence rates (e.g., above 60 ms) and accommodation times (e.g., 20 ms-100 ms) to reduce the perception of changes in resolution. The display system may dynamically modify the maximum resolution based on the available resources of the display system, for example, prior to the display system rendering each frame. For example, as more content is to be presented to the user, the maximum resolution of the content may be reduced to ensure that the display system may present frames of rendered content above a desired threshold rate to reduce the perception of changes in resolution. The display system may optionally monitor the frames per second at which content is being presented and may adjust the maximum resolution and / or adjust the rate of resolution drop based on the distance from the fixation point to ensure that the frames per second presented do not fall below a threshold rate. As an example, the display system may render content, such as a first virtual object located in the zone of the fixation point, at full resolution. Instead of reducing the maximum resolution of the first virtual object and ensuring that the frames per second remain above a certain threshold, the display system may dynamically increase the rate of resolution drop based on the distance. In this manner, the display system may adjust the resolution assigned to each zone outside the zone of the fixation point.Optionally, the display system may set a minimum resolution that may be used within each zone outside the zone of the fixation point, and may adjust the maximum resolution if the minimum resolution would be exceeded (e.g., the display system may reduce the maximum resolution if the display system needs to reduce the resolution of content below a minimum to maintain a threshold ratio). Similarly, the display system may reduce the maximum resolution without reducing the resolution of content in zones outside the zone of the fixation point. Optionally, a user of the display system may indicate whether they would prefer that content located proximate to the fixation point be given preference over other content.

[0222] In some embodiments, as described in more detail below with respect to FIGS. 13-14, the display system may optionally adjust the resolution of the content using the angular proximity of the content to the user's line of sight. For example, if the particular content is outside the zone in which the fixation point is located, but within a threshold proximity of the user's line of sight, such that the particular content would be in the fovea of ​​the user's eye, the display system may cause the particular content to be rendered at a higher resolution (e.g., a resolution above the full resolution or that shown in the grid illustrated in FIG. 11A1). Optionally, the display system may reduce the resolution of the particular content and apply a blurring process (e.g., Gaussian blurring) to the particular content. In this way, the particular content may be rendered at a lower resolution while being blurred to represent that the particular content is, for example, farther from the user than the fixation point. In addition, the blurring may reduce the perceptibility of the lower resolution (e.g., the blurring may reduce the perceptibility of the increased pixel size resulting from the lower resolution).

[0223] Exemplary operations associated with presenting virtual content are illustrated in Figs. 12B-12C (e.g., rendering pipeline). In the example of Fig. 12B, a three-dimensional scene is presented to a user without adjustments to resolution made as described herein. In Fig. 12C, adjustments to resolution are made according to fixation information as described herein. For example, one or more of the following adjustments can be made: reducing vertex motion complexity, reducing plane-filling level of detail, reducing geometry generation, reducing pixel motion complexity / aggregation of multiple pixels, etc. The adjustments can be advantageously made at different steps in the pipeline for presenting virtual content as illustrated, and can be optimized according to the particular software and / or hardware utilized to present the virtual content. It should be understood that the fidelity zones described in Fig. 12C are resolution adjustment zones.

[0224] 12A, the display system presents the adjusted content to the user at block 1208. As described above, the display system adjusts the resolution of the content based on its proximity to the three-dimensional fixation point. The display system then presents the rendered content to the user at the associated location. In some embodiments, the display system may perform process 1200 for each frame of content to be rendered, or may adjust the resolution of the content as the user adjusts their fixation point.

[0225] As mentioned above, in some embodiments, virtual objects may be presented at different depths while still being within the user's line of sight. Figure 13 illustrates an example of a representation of a user viewing multiple virtual objects aligned with the user's line of sight. The example representation includes the line of sight 1003A, 1003B of the user's eyes 210, 220, along with the user's field of view (e.g., the display frustum 1004 of a display system), which is fixated on a fixation point on a first virtual object 1008A.

[0226] As illustrated, the second virtual object 1008B is within angular proximity of the user's line of sight (e.g., one or both of the line of sight vectors 1003A, 1003B) such that the second virtual object 1008B would fall into the user's fovea (e.g., would fall into at least one fovea of ​​either of the eyes). For example, in response to rendering frame 1110, the second virtual object 1008B is located behind (e.g., at a deeper perceived depth from) the first virtual object 1008A. It should be appreciated that the fovea is the portion of the retina having the highest visual acuity. If the resolution of the second virtual object 1008B is reduced (e.g., reduced as described above with respect to at least FIG. 11A1) because the second virtual object 1008B would fall into the user's fovea, the user may perceive a reduction in resolution. To avoid a perceptible reduction in resolution, the display system may (1) render the second virtual object 1008B at the same resolution as the first virtual object 1008A or within a threshold resolution of the first virtual object 1008A, and / or (2) render the second virtual object 1008B at a reduced resolution (e.g., as shown in FIG. 11A1) and apply blurring to the second virtual object prior to presentation to the user. Without being limited by theory, the blurring may mask the reduction in resolution while providing depth cues.

[0227] 14 is a flow chart of an example of a process 1400 for adjusting virtual content based on angular distance from a user's line of sight. For convenience, the process 1400 will be described as being implemented by a display system (e.g., a wearable display system 60, which may include processing hardware and software and may optionally provide information to one or more computers or other processing units external systems, e.g., offload processing to the external systems and receive information from the external systems). In the example process 1400, the display system is a variable-focus display system in which each frame is presented on the same depth plane and optionally causes all content to be presented to be folded into a single frame buffer, i.e., the variable-focus display system presents virtual content on one depth plane at a time.

[0228] The display system determines the user's three-dimensional fixation point (block 1402) and obtains location information associated with the content to be presented (block 1404). Blocks 1402 and 1404 may correspond to blocks 1202 and 1204 of FIG. 12A, respectively. As described above with reference to FIG. 12A, the display system monitors the user's eye movement (e.g., eye orientation) and determines the user's fixation point. The display system may obtain the location information (e.g., in the next frame) of the content to be presented and may subsequently adjust the resolution of the content.

[0229] Continuing with reference to FIG. 14, the display system determines content that will have reduced resolution and that is located within a threshold angular distance from the user's line of sight (block 1406). The display system identifies content that will have reduced resolution due to the proximity of the content from the fixation point (e.g., the content is located at a depth greater than the fixation point) but that will fall within the user's fovea (e.g., within a threshold angle from the user's line of sight). Because the content will fall within the user's fovea, the user may be able to perceive a reduction in resolution, such as with 3D fixation-centered rendering described herein. It should be understood that content block 1406 may include steps to implement blocks illustrated in FIG. 12C, particularly blocks identified in section "GPU."

[0230] As a result, in block 1408, the display system may optionally cause the determined content to be rendered at a higher resolution. The display system may adjust the resolution of the determined content to be full resolution (e.g., the same resolution as the fixation point or content located within the same zone or volume of space as the fixation point) or above the reduced resolution that would otherwise be assigned to the content (e.g., as described in block 1406).

[0231] In block 1410, the display system may optionally reduce the resolution of the content and blur the content prior to presentation to the user. As described above, the variable focus display system may present the content to the user using a single display buffer. Because the variable focus display system presents all content at the same depth plane, the variable focus display system may use the same display buffer to output the content, for example from a rendering engine.

[0232] Optionally, the display system may utilize initial depth buffers, where each depth buffer is assigned one or more depth planes, and may combine the initial depth buffers to obtain a display buffer. With reference to the illustration of FIG. 13, the first depth buffer may include a first virtual object 1306, while the second depth buffer may include a second virtual object 1308. The display system may then apply a blurring process to the second depth buffer or to the particular content contained within the second depth buffer (e.g., the display system may apply a blurring process to the second virtual content 1308, but not to other content that is on the same depth plane, but at a greater angular distance from the user's line of sight). After performing the blurring process, the display system may combine the first and second depth buffers (e.g., the display system may add occlusion, e.g., remove portions of the second virtual object 1308 that are not visible due to occlusion by the first virtual object 1306) to obtain a display buffer.

[0233] An exemplary blurring process may include the display system performing a convolution of a kernel associated with the blurring (e.g., a Gaussian kernel, a circular kernel such as to recreate a disruptive effect, a box blur, etc.) on the content. In this manner, the reduction in resolution may be masked while the processing savings resulting from reducing the resolution may be preserved. Optionally, the strength associated with the blurring process (e.g., the degree to which the content is blurred) may be based on the difference in depth between the user's fixation point and the content and / or the angular proximity of the content to the user's line of sight. For example, the degree of blurring may increase with increasing proximity to the user's line of sight.

[0234] In some embodiments, the display system may utilize the features of blocks 1408 or 1410 according to the hardware and / or software of the display system. For example, certain hardware (e.g., a graphics processing unit) may be capable of implementing the blurring process in hardware without reaching a threshold for the performance of the hardware. For this particular hardware, the display system may be configured to reduce the resolution of the content and then blur the content. However, other hardware may be slow to implement the blurring process and rendering the content at a higher resolution may result in higher performance. For this other hardware, the display system may be configured to render the content at a higher resolution. Furthermore, the decision of whether to render the content at a higher resolution or at a lower resolution with blurring may depend on the type of content to be displayed. For example, the display system may be configured to render text at a higher resolution while rendering shapes at a lower resolution and blurring them.

[0235] 14, in block 1412, the display system presents the content to the user. The display system may present the adjusted content to the user from the same display buffer, for example, as described above. II. Resolution Adjustment Based on Ambient Lighting Level

[0236] In addition to, or as an alternative to, the reduction in resolution along the z-axis, various other schemes for presenting virtual content with a reduction in resolution may be implemented in some embodiments. Advantageously, as described herein, some aspects of the virtual content may be presented in a relatively high resolution and certain other aspects may be presented in a relatively low resolution, which may reduce the use of computational and energy resources by the display system, while preferably having little impact on the perceived image quality of the virtual content.

[0237] 15, an example of a representation of the retina of a user's eye is illustrated. The illustrated view shows the retina 1500 as seen when viewed head up along the visual axis of the retina. The retina 1500 includes a fovea 1510 surrounded by a peripheral area 1530. Within the fovea 1510 is a foveola 1520 that intersects the visual axis.

[0238] It should be understood that the retina contains two types of photoreceptors: rods and cones. In addition, the distribution of these photoreceptors across the retina varies, providing different rod and cone densities across the retina.

[0239] Referring now to FIG. 16, an example of the resolution of the retina 1500 of FIG. 15 and the rod and cone densities across it are diagrammatically illustrated. The x-axis indicates eccentricity relative to the point where the visual axis intersects the retina. To the right on the page is toward the nose and to the left on the page is toward the temple. As illustrated, the resolution of the human eye roughly correlates with the density of photoreceptors (rods and cones) in the retina. As a result, in some embodiments, the reduction or taper in resolution (e.g., spatial resolution) of virtual content on the x and y-axes (e.g., on a given depth plane) may substantially track the reduction in cone density, rod density, or collection of rod and cone densities across the retina. For example, the trend of resolution reduction away from the fixation point across the user's visual field may be within ±50%, ±30%, ±20%, or ±10% of the trend in change in photoreceptor density (e.g., cone density, rod density, or a combination of rod and cone densities) across a corresponding portion of the retina. In some embodiments, the reduction in resolution away from the fixation point is gradual and substantially tracks the density change. In some other embodiments, the reduction in resolution may occur in steps (e.g., one step, two steps, etc.). For example, there may be two steps, i.e., a highest resolution region of the visual field correlated with the fovea, a medium resolution region correlated with the fovea, and a lower resolution region correlated with the peripheral area.

[0240] 16 , it should be understood that different photoreceptors have different levels of activity under different lighting conditions, e.g., different ambient lighting levels. As a result, it is possible that a reduction in resolution that follows the density of photoreceptors may not be noticeably perceptible to a user at some lighting levels, but may be perceptible at other lighting levels. As a result, in some embodiments, a reduction in resolution of virtual content along an x-, y-, or z-axis may be set with reference to external lighting conditions.

[0241] For example, the visual behavior of the eye can be divided into three modes based on the light conditions. The three modes are photopic, mesopic, and scotopic. Photopic vision is typically observed in bright conditions, e.g., about 10-10 8 cd / m 2 Including 3cd / m 2 In photopic vision, the cones are primarily active. In scotopic vision, the rods are primarily active. In mesopic vision, both rods and cones may be active. As used herein, ambient light conditions or lighting levels refer to the amount of light to which a user's eye and its retina are exposed.

[0242] Mesopic vision is typically seen in lower light conditions, e.g., around 10 -3 ~10 0.5 cd / m 2 Both cones and rods are active at least at some illumination levels within mesopic vision, and the dominance of rods or cones changes over time depending on whether the ambient illumination level increases or decreases. As the eye adapts to a brighter environment, more cones are activated relative to the rods, whereas as the eye adapts to a darker environment, more rods are activated relative to the cones.

[0243] Scotopic vision typically occurs under lighting conditions where the illumination level is below that for photopic vision. For example, scotopic vision occurs at approximately 10 -3 ~10 -6 cd / m 2 Including about 10 -2 cd / m 2 or less than or about 10 -3 cd / m 2The illumination levels may occur at or below 100 nm. Rods are primarily active in scotopic vision. It should be understood that the illumination levels described herein for photopic, mesopic, and scotopic vision are examples. In some embodiments, the illumination levels associated with each type of vision may be arbitrarily assigned based on user preferences and / or customization for a group to which the user belongs (e.g., based on gender, age, ethnicity, the presence of visual abnormalities, etc.).

[0244] In some embodiments, the type of vision (photopic, mesopic, or scotopic) that is active in the user may be determined based on a measurement of the ambient lighting level. For example, the display system may be configured to measure the ambient lighting level using a light sensor such as an outward-facing camera 112 (FIG. 9D). In some embodiments, the display system may communicate with another sensor or device that provides information regarding the ambient lighting level.

[0245] It should be understood that the head mounted display system may block or attenuate some of the ambient light so that the outward facing camera may not provide a brightness level that accurately reflects the amount of light impinging on the eye. In addition, the display system also projects light onto the eye, becoming a light source that may modify the illumination level to which the eye is exposed when providing virtual content. In some other embodiments, the inward facing camera may be utilized to determine the brightness level. For example, the brightness level is roughly correlated with the size of the pupil. FIG. 17 diagrammatically illustrates an example of a relationship between pupil size and the amount of light incident on the user's eye. The x-axis shows values ​​related to brightness and the y-axis shows values ​​related to pupil area. As a result, the display system may be configured to determine the user's pupil area and then extrapolate the brightness based on this pupil area. For example, the display system may be configured to capture an image of the user's eye 210 using the inward facing camera 500 (FIG. 6) and then analyze the image to determine the pupil area or other metrics indicative of the pupil area (e.g., pupil diameter or width). For example, the area occupied by the pupil of eye 210 in an image captured by the camera may be determined and then corrected for any scaling factor caused by the camera's optics. Advantageously, using the pupil area to determine the luminance level may effectively take into account both the reduction in ambient luminance level caused by the display blocking some ambient light, and also the contribution to the luminance level due to the light output of the display itself.

[0246] 17, the display system may be configured to determine whether the user's eye is in photopic, mesopic, or scotopic mode based on the determined pupil area. For example, the display system may have resident in memory a table or other stored information that defines the expected visual mode for a particular pupil area. As an example, consistent with the graph shown in FIG. 17, the display system may determine whether the user's eye is in photopic, mesopic, or scotopic mode based on the determined pupil area. 2 or less pupil area indicates photopic vision, 3mm 2or larger up to approximately 38mm 2 The pupil area indicates mesopic vision, 38mm 2 A pupil area above 0.05 mm may be categorized as indicative of scotopic vision. It should be understood that these luminance values ​​and associated visual modes are examples and other values ​​may be substituted. For example, different values ​​may be applied to different users in response to input from the users, or different values ​​may be applied based on the particular categories that users may fall into (e.g., gender, age, ethnicity, presence of visual anomalies, etc.). In addition, it should be understood that the display system does not necessarily identify a specific visual mode. Rather, the display system may simply be configured to associate a particular measured pupil area with a particular resolution level or accommodation.

[0247] In some embodiments, inputs from both inwardly facing camera 510 (FIG. 6) and outwardly facing camera 112 (FIG. 9D) may be utilized to determine the brightness level. For example, the display system may be configured to take an average (including a weighted average) of the brightness levels determined using cameras 510 and 112. As noted above, the brightness level determined using camera 510 may be extrapolated from the size of the pupil area of ​​the user's eye based on imaging the user's eye using camera 510.

[0248] It should be understood that rods and cones have different levels of visual acuity and different sensitivity to color and contrast. As a result, there are differences in visual acuity and sensitivity to color and contrast at different ambient luminance levels, since the ambient luminance level affects whether rods and / or cones are active. Advantageously, the light level difference in visual acuity and sensitivity to color and contrast may be applied to provide an additional basis for reducing resolution, which may be utilized in conjunction with the change in resolution based on fixation point as described above (e.g., with respect to Figs. 12A and 14), or may be utilized separately without specifically performing the change in resolution based on fixation point.

[0249] 18, a schematic diagram of an example of a process 1800 for adjusting virtual content based on the amount of light incident on a user's eye is shown. For convenience, the process may be described as being performed by a display system (e.g., wearable display system 60 (FIG. 9D), which may include processing hardware and software and may optionally provide information to one or more computers or other processing units external to the system, e.g., offload processing to the external system and receive information from the external system).

[0250] In block 1810, the display system determines the amount of light reaching the retina. Preferably, this determination is an estimate of the amount of light reaching the retina, rather than a direct measurement of the light impinging on the retina. This estimation may be made using methods disclosed for determining luminance levels, as discussed herein. For example, the luminance level may be assumed to correspond to the amount of light reaching the retina. As a result, determining the amount of light reaching the retina may include determining the size of the user's pupils and / or determining an ambient luminance level using a sensor configured to detect light, such as an outward-facing camera on the display device.

[0251] In block 1820, the display system adjusts the resolution of the virtual content to be presented to the user based on the amount of light found to reach the retina in block 1810. In some embodiments, adjusting the resolution of the virtual content includes adjusting one or more of the spatial resolution, color depth, and light intensity resolution of the virtual content. It should be understood that the human visual system has maximum visual acuity and sensitivity for spatial resolution, color, and light intensity under photopic illumination levels. The ability to perceive differences in spatial resolution, color, and light intensity is reduced under mesopic illumination levels and further reduced under scotopic illumination levels.

[0252] As a result, in some embodiments, if the amount of light present is found to correspond to a level for photopic vision, the virtual object may be rendered at full or high spatial resolution (compared to the spatial resolution that would be utilized for mesopic or scotopic vision). If the amount of light present is found to correspond to a level for mesopic vision levels, the virtual object may be rendered at a reduced spatial resolution compared to the spatial resolution utilized for the virtual object under photopic illumination levels. If the amount of light is found to correspond to a scotopic vision level, the virtual object may be rendered at a lower spatial resolution than that used under mesopic or photopic illumination levels. The spatial resolution may be adjusted, such as by reducing the number of polygons, as described herein.

[0253] The color depth or bit depth may also be adjusted according to the illumination level, with the highest color depth being used under photopic illumination levels, a medium color depth being used under mesopic illumination levels, and the lowest color depth being used under scotopic illumination levels. It should be understood that the color depth may be adjusted by varying the number of bits used per color component of a pixel, with fewer bits being equivalent to a lower color depth.

[0254] Similarly, without being limited by theory, it is believed that the gradations in light intensity are greater as illumination levels progress from photopic to mesopic to scotopic illumination levels. In other words, it is believed that the human visual system is capable of discriminating smaller differences in light intensity as ambient illumination levels decrease. In some embodiments, the display system may be configured to display fewer gradations in light intensity as illumination levels progress from photopic to mesopic to scotopic illumination levels. As a result, the greatest number of gradations in light intensity levels are presented under photopic illumination levels, fewer gradations are presented under mesopic illumination levels, and even fewer gradations are presented under scotopic illumination levels.

[0255] Additionally, in some embodiments, the display system may be capable of providing a greater number of gradations in light intensity than the user is capable of perceiving. An example of this is illustrated in Figures 22a-22c and discussed further below. For example, the display system may be capable of displaying 256 different levels of intensity for a given image pixel, but the user may be capable of perceiving a smaller number of levels, e.g., only 64 levels. In this instance, multiple possible light intensity levels are subsumed within a single one of the perceivable light intensity levels. For example, the display system may be capable of displaying four different light intensity levels, but the user may perceive all four as similar. In such a situation where multiple possible light intensities are perceived by the user as being the same, the display system may be configured to select the lowest intensity value for display from those values ​​that are perceived as similar. As a result, the display system may be able to utilize lower intensities, thereby reducing the amount of power used to illuminate the display and achieve the desired light intensity. This may have particular advantages in display systems where the individual pixels of the spatial light modulator are themselves light emitters, such as organic and inorganic LEDs. In some embodiments, the number of shades decreases with decreasing ambient lighting levels, and the display system is configured to group a larger number of possible light intensity levels together and display the lowest light intensity group.

[0256] It should be understood that one, two, or all three of the spatial resolution, color depth, and light intensity resolution may be varied based on the light conditions (amount of light reaching the user's retina) directed to the user for the virtual content to be displayed. These adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made to the entire virtual content without making adjustments to the resolution based on the distance from the fixation point of the user's eye as disclosed herein. In some other embodiments, adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made in conjunction with adjustments to the resolution based on the distance from the fixation point (see, for example, FIGS. 12A and 14). In some embodiments, if the resolution decreases with distance from the fixation point, the profile of the decrease on a given plane (on the x- and y-axes) preferably matches the profile of the change in cone density across the corresponding portion of the retina.

[0257] In some embodiments, as described herein, adjustments to spatial resolution, color depth, and / or light intensity resolution are preferably tied to the mode of vision (photopic, mesopic, or scotopic) that is active at a given time. These adjustments may change dynamically as the mode of vision changes. For example, as a user progresses from photopic to scotopic vision, the resolution may decrease as discussed herein. Conversely, as a user progresses from scotopic to mesopic vision, the resolution of the virtual content may increase. It should be understood that tying adjustments to resolution to a particular mode of vision does not require a specific determination that the user is in that particular mode. Rather, the display system may simply be configured to associate a particular resolution, regardless of spatial resolution, color depth, or light intensity resolution, with a particular range of ambient illumination levels or pupil sizes. In addition, the resolution adjustment is preferably associated with three levels of light conditions (corresponding to three modes of vision) as discussed herein, although in some embodiments the resolution adjustment may be associated with two levels of light conditions or more than three levels of light conditions.

[0258] It should also be understood that the resolution adjustments may occur in real-time (e.g., as ambient light conditions change) or may be delayed for a set duration to allow the human visual system to adapt to existing light conditions before resolution adjustments are made to the virtual content. Without being limited by theory, it is believed that the human visual system requires a period of time to adapt to different lighting levels, and the period of time increases as lighting levels decrease. As a result, in some embodiments, adjustments in resolution resulting from changing lighting levels do not occur until a user has been exposed (e.g., substantially continuously exposed) to a particular lighting level for a set amount of time. For example, the set amount of time may be 5 minutes, 10 minutes, 15 minutes, or 20 minutes.

[0259] Continuing with reference to Figure 18, at block 1830, the virtual content is presented to the user. This presentation of the virtual content may occur as discussed herein, for example, at block 1208 of Figure 12A or block 1412 of Figure 14.

[0260] Referring now to FIG. 19, an example of the change in resolution detectable by a user's eye as the amount of light incident on the eye changes is diagrammatically illustrated. The figure illustrates an example of the sensitivity of the human visual system to spatial resolution under different vision modes. Scotopic vision occurs in the low light region 1910, mesopic vision occurs in the mid light region 1920, and photopic vision occurs in the bright light region 1930. As shown, the sensitivity to spatial resolution decreases substantially as the ambient illumination level decreases. In some embodiments, the adjustments to spatial resolution discussed above with respect to FIG. 18 correspond to the contours of the illustrated curves. For example, for a given light level in photopic or scotopic mode, the virtual content is rendered with sufficient spatial resolution to meet or exceed the resolution value shown on the y-axis.

[0261] Now referring to FIG. 20, it should be understood that different photoreceptors can be used to perceive different wavelengths or colors of light. FIG. 20 diagrammatically illustrates an example of the difference in the eye's sensitivity to different colors of light at different levels of illumination. The difference in duration on the x-axis is a reflection of the amount of time typically required for the human visual system to adapt to a particular ambient illumination level so that a particular mode of vision is activated. It should be noted that at ambient illumination levels corresponding to some of scotopic and mesopic vision, photoreceptors for red light may no longer be active, while photoreceptors for blue light are active under the lowest light conditions. It should be understood that red, green, and blue light correspond to the colors most typically used as primary colors in display systems to form full-color images (e.g., as discussed herein with respect to FIGs. 8-9B). In some embodiments, the display system may be configured to vary the rendering of different color images depending on the ambient illumination level.

[0262] 21, a schematic diagram of an example process 2100 for adjusting virtual content formed using multiple primary images is shown, where the resolution adjustment is based on the color of the primary images. In block 2110, a display system provides the virtual content to be presented using multiple primary images. These may be different images of different primary colors that are to be directed to different waveguides, as discussed with respect to FIGS. 8-9B. As a result, in some embodiments, each stream of images of different primary colors may be rendered separately. Providing the virtual content to be presented using multiple primary images may include utilizing a display system that outputs image streams of different primary colors to form a full color image.

[0263] In block 2120, the display system may adjust the resolution of the primary color image based on the color. For example, the display system may select one of these primary color images for resolution adjustment. For example, the selection may be made based on a determination of the illumination level, as described above with respect to block 1810 of FIG. 18. As shown in FIG. 19, some primary colors may not be perceived by the user at some illumination levels. The display system may have stored therein information regarding illumination levels and the primary colors that are invisible at those levels. If there is a match between the illumination levels and the primary colors that are invisible at those levels, the image of that primary color may be selected for adjustment. In some environments, one adjustment may simply be to not render or display that primary color image if the ambient illumination levels are such that the user is not expected to perceive that color. For example, under scotopic illumination levels, the display system may be configured to not render or display a primary color red image.

[0264] 21, in block 2130, the virtual content is presented to the user. Presentation of the virtual content may occur as discussed herein, such as in block 1208 of FIG. 12A or block 1412 of FIG.

[0265] 22A-22C, as discussed above, and without being limited by theory, it is believed that the ability of the human visual system to perceive gradations in light intensity changes with ambient illumination levels. FIGS. 22A-22C show examples of contrast sensitivity that changes as the amount of light incident on a user's eye decreases. For example, FIG. 22A may be understood to show contrast sensitivity under photopic light conditions, FIG. 22B may be understood to show contrast sensitivity under mesopic light conditions, and FIG. 22C may be understood to show contrast sensitivity under scotopic light conditions. FIG. 22A shows a gradient 2110 of gradations ... 1 From 2110 i22B shows a progression 2100 from high light intensity at the top to low light intensity at the bottom. 1 From 2110 i 22C shows a progression 2102 from high light intensity to low light intensity. 1 From 2110 i 18 illustrates a progression 2104 from high light intensity to low light intensity. Boxes 2120, 2130, 2140 illustrate groups of intensity gradations that are perceived by a user to be the same. The size of these groups is expected to increase with decreasing ambient lighting levels, as illustrated. As a result, as described above with respect to FIG. 18, in some embodiments the display system may be configured to use the lowest intensity value within each group (e.g., within each of boxes 2120, 2130, 2140).

[0266] 23, an example of a representation of the optic nerve and peripheral blind spot of a user's eye is illustrated. In some embodiments, in addition to or as an alternative to any of the resolution adjustments disclosed herein, the display system may be configured to refrain from rendering content in various locations where the content is not expected to be perceivable by the user. FIG. 23 illustrates a representation of the optic nerve and peripheral blind spot of a user's eye 210 for the left and right eyes, respectively. L and 210 R Each eye has a separate optical axis 1003A and 1003B and an optic nerve 2300. L and 2300 R and Optic nerve 2300L and 2300 R Each of the individual eyes L and 210 R These blind spots are where the viewer is blinded by the light ray 2302. L and 2302 R In addition, there is an area around the periphery of each eye where content cannot be seen by the opposite eye. For example, the left peripheral area P L Contents in Left Eye 210 L, but the right eye 210 R On the other hand, the right peripheral area P R The contents of the right eye are 210 R 2, but the left eye 210 L As a result, in some embodiments, the display system does not include a L and 210 R Content that would be mapped to the blind spot of, for example, ray 2302 L and 2302 R Additionally or alternatively, in some embodiments, the display system may be configured to omit rendering content that falls above the right peripheral region P L If the content enters the left eye, 210 L and / or the display system may be configured to omit rendering the content in the left peripheral region P L If the content enters the right eye, 210 R It should be appreciated that the locations of the blind spots and / or peripheral areas may be preset, for example, based on averages for a population of users, and / or may be adjusted and calibrated for a particular user through testing with content displayed in various locations and input from the user indicating whether a virtual object is visible. III. Multiple Image Streams for Providing Content with Different Resolutions

[0267] In some embodiments, a foveated image having high and low spatial resolution regions may be formed by two or more spatially overlapping image streams, each having a different resolution (e.g., different perceived pixel density). For example, one of the image streams, e.g., a low-resolution image stream, may form an image having a wide field of view, and another of the image streams, e.g., a high-resolution image stream, may form an image having a narrow field of view. The narrow field of view image and the wide field of view image may contain similar content, but are viewed by the user at different resolutions or pixel densities. These images may be overlaid on each other (e.g., occupying the same location in space at the same time or close in time such that the viewer perceives the images to be presented simultaneously). Thus, the viewer may receive an aggregate image with high resolution in a constrained portion of its field of view and a low resolution image over a larger portion of its field of view. Preferably, as discussed herein, the high resolution portion is mapped to the foveal vision region of the user's eye, while the low resolution portion is mapped to the peripheral vision region of the user's eye. Thus, the difference in resolution between the high resolution and low resolution portions of the image is preferably not readily perceptible to a user.

[0268] In some environments, display systems for displaying high and low resolution images utilize the same spatial light modulator to form both images. Thus, the spatial light modulator has pixels of a fixed size and density. In display systems with pixels of a fixed size and density, an increase in the angular field of view (FOV) occurs at the expense of spatial or angular resolution, for example, as governed by the Lagrange invariant. For example, if an SLM with a fixed number of pixels is used to form both high and low resolution images, spreading the pixels across the entire field of view will provide an image with a lower apparent resolution than constraining the pixels to a fraction of the total field of view. That is, the pixel density of the high resolution image is higher than the pixel density of the low resolution image. As a result, there is generally an inverse correlation between the FOV and the angular resolution. Since the FOV and angular resolution affect image visibility and quality, this tradeoff imposes constraints on the user experience and the final achievable FOV and angular resolution in an AR or VR system. As will be apparent from the discussion herein, in some embodiments, the term "resolution" may be used to refer to "angular resolution".

[0269] Head-mounted or wearable display devices can be configured to provide an immersive user experience by projecting virtual content directly into the user's eye. Although it may be beneficial to provide wide FOV images with uniform high resolution across the FOV, physiological limitations of the human visual system may prevent users from appreciating or even perceiving high-resolution images located within the peripheral region of the user's field of view. The inability to perceive high-resolution images within this peripheral region arises from the properties of the retina of the human eye, which contains two types of photoreceptors: rod cells and cone cells. Cones are more responsible for acuity (detailed) vision. Rods and cones are distributed differently within the human eye. The highest concentration of cone cells is found within the fovea (i.e., the center of the retina), while the highest concentration of rod cells is found within the region immediately surrounding the fovea (i.e., the periphery of the retina). Due to this non-uniform distribution of rod cells and cone cells, the fovea is responsible for acuity central vision (also called foveal vision). Visual acuity decreases as distance from the fovea increases.

[0270] For AR or VR applications, a headset is generally worn by one user at a time. The headset can be configured to take advantage of the user's inability to perceive all the details of a wide-field-of-view stream of images at once by limiting the display of high-resolution content to the area in the wide field of view currently focused by the user. In this way, the headset can provide the user with the appearance of a high-resolution wide FOV stream of images without the need for processing power that would otherwise be required to generate high-resolution content across the entire field of view. The stream of images presented to the user can take many forms and will generally be referred to as an image stream. For example, the image stream can show a static image by continually displaying the same image to the user, or can show motion by displaying a stream of different images. In some embodiments, the headset can be configured to display more than one image stream simultaneously, where the different image streams can have different angular resolutions and can extend across different areas of the user's FOV. It should be noted that the image stream associated with the AR system may not show content entirely across the particular area to which it is assigned, as the AR system is designed to mix virtual and real-world content.

[0271] According to some embodiments, the first image stream and the second image stream can be presented to the user simultaneously or in rapid succession such that the two image streams appear to be displayed simultaneously. The first image stream can have a wide FOV and low resolution that can encompass the user's vision and evoke an immersive experience for the user. A portion of the first image stream that corresponds to the momentary portion of the FOV covered by the second image stream can be turned off in some embodiments. The second image stream can have a narrow FOV and high resolution that can be dynamically displayed within the boundaries of the first image stream according to the user's current fixation point as determined in real time using eye-tracking techniques. In other words, the second image stream can be shifted as the user's eye line of sight changes such that the second image stream persistently covers the user's foveal vision. In some embodiments, the first image stream is presented to the user at a fixed position as the second image stream is shifted relative to the first image stream. In some other embodiments, both the first image stream and the second image stream are shifted according to the user's current fixation point.

[0272] The content of the second image stream may include a subset of the content of the first image stream with a higher resolution than the first image stream and may be overlaid on and appropriately aligned with respect to the first image stream. Because the higher resolution second image stream overlays a portion of the first image stream in the user's foveal vision, the modulation transfer function (MTF) in the area including the higher resolution image stream is increased. In some embodiments, the subset of the content of the first image stream overlaid by the second image stream may be turned off or presented at a lower intensity. In this way, the user may perceive the combination of the first image stream and the second image stream as having both a wide FOV and high resolution. Such a display system may provide several advantages. For example, the display system may have a relatively small form factor and may conserve computing resources and computing power while providing a better user experience.

[0273] In an embodiment, the light intensity in the boundary region of the first image stream and the second image stream tapers to a value below the intended image brightness, and the boundary region of the first image stream and the second image stream are overlapped. In the overlap area, the sum of the light intensity due to the two image streams may be relatively constant and equal to the intended image brightness. When crossing the overlap region from the first image stream side to the second image stream side, the MTF changes from a first value equal to or closer to the MTF of the first image stream to a second value equal to or closer to the MTF of the second image stream. In this way, it is possible to avoid creating a sharp boundary between the regions supplied by the two image streams, which may be perceptible to a user in some circumstances.

[0274] According to some embodiments, the first light beam associated with the first image stream and the second light beam associated with the second image stream can be multiplexed into a combined light beam using a multiplexing method. For example, time division multiplexing, polarization division multiplexing, wavelength division multiplexing, and the like can be used according to various embodiments. The combined light beam can be directed to one or more optical elements that serve to demultiplex the combined light beam into two separate optical paths. For example, a beam splitter or optical switching element such as a polarizing beam splitter (PBS) or a dichroic beam splitter can be used to separate the combined light beam depending on the method of multiplexing used. Once separated, the first light beam associated with the first image stream and the second light beam associated with the second image stream can be routed through their respective optical paths and ultimately provided to a user as an output.

[0275] According to some embodiments, a first light beam associated with a first image stream can be angularly expanded by optical elements in a first optical path such that the first image stream is presented with a wider FOV and lower angular resolution (as governed by the invariant), while a second light beam associated with a second image stream is angularly unexpanded, de-expanded, or expanded by an amount less than the amount of expansion applied to the first light beam associated with the first image stream. In this manner, the second image stream can be presented with a narrower FOV and higher angular resolution than the first image stream (as governed by the invariant).

[0276] FIG. 24 shows a field diagram depicting the perimeter of an exemplary monocular field 3002 for a human eye in a two-dimensional angular space. As shown in FIG. 24, the temple-nasal and inferior-superior axes of the field diagram serve to define a two-dimensional angular space into which the perimeter of the monocular field 3002 is mapped. In this manner, the field diagram of FIG. 24 may be considered an equivalent or similar to a "Goldmann" field map or plot for a human eye. As indicated by the depicted arrangement of the temple-nasal and inferior-superior axes, the field diagram shown in FIG. 24 represents a field diagram for a human left eye. Although the field of view may vary slightly from person to person, the depicted field of view approximates what most people are able to see with their left eye. A field diagram depicting the peripheral field of an exemplary monocular field of a right eye would result in the temple-nasal axis and peripheral field of the monocular field 3002 being similar to that of a version of the field diagram of FIG. 24 mirrored about an inferior-superior axis.

[0277] The field of view diagram of FIG. 24 further depicts the perimeter of an exemplary ocular field of view 3004 for a human eye, which represents a portion of the monocular field of view 30022 in angular space at which a person may fixate. In addition, the field of view diagram of FIG. 24 also depicts the perimeter of an exemplary foveal field 3006 for a human eye, which represents a portion of the monocular field of view 3002 in angular space within a direct view of the fovea of ​​the human eye at a given time. As depicted, the person's foveal field 3006 may move anywhere within the ocular field of view 3004. The portion of the monocular field of view 3002 outside the foveal field 3006 in angular space may be referred to herein as the peripheral region of the person's field of view. Because the ability of the human eye to distinguish high levels of detail outside the foveal field 3006 is very limited, displaying reduced resolution images outside the foveal field 3006 is unlikely to be perceived and may allow for substantial savings in terms of power consumption for processing components involved in generating content for the display.

[0278] 25A shows an exemplary wearable display device 4050 configured to provide virtual content to a user, according to some embodiments. The wearable display device 4050 includes a primary display 4052 supported by a frame 4054. The frame 4054 can be attached to a user's head using attachment members in the form of temple arms 4006.

[0279] Referring now to FIG. 25B, an exemplary embodiment of an AR system configured to provide virtual content to a user will now be described. In some embodiments, the AR system of FIG. 25B may represent a system to which the wearable display device 4050 of FIG. 25A belongs. The AR system of FIG. 25B uses a stacked light directing optical element assembly 4000 and generally includes an image generating processor 4010, a light source 4020, a controller 4030, a spatial light modulator ("SLM") 4040, an input optical system 4060, and at least one set of stacked eyepiece layers or light directing optical elements ("LOEs", e.g., planar waveguides) 4000 that function as a multiple planar focus system. The system may also include an eye tracking subsystem 4070. It should be understood that other embodiments may have multiple sets of stacked LOEs 4000, but the following disclosure will focus on the exemplary embodiment of FIG. 25B.

[0280] The image generation processor 4010 is configured to generate the virtual content to be displayed to the user. The image generation processor may convert images or videos associated with the virtual content into a format that can be projected to the user in 3-D. For example, when generating 3-D content, the virtual content may need to be formatted so that parts of certain images are displayed at certain depth planes, while others are displayed at other depth planes. In one embodiment, the images may all be generated at a certain depth plane. In another embodiment, the image generation processor may be programmed to provide slightly different images to the right and left eyes 210 such that when viewed together, the virtual content appears coherent and comfortable to the user's eyes.

[0281] The image generation processor 4010 may further include a memory 4012, a GPU 4014, a CPU 4016, and other circuitry for image generation and processing. The image generation processor 4010 may be programmed with desired virtual content to be presented to a user of the AR system of FIG. 25B. It should be understood that in some embodiments, the image generation processor 4010 may be stored within a wearable AR system. In other embodiments, the image generation processor 4010 and other circuitry may be stored within a belt pack that is coupled to the wearable optics. The image generation processor 4010 is operably coupled to a light source 4020 and one or more spatial light modulators (described below) that project light associated with the desired virtual content.

[0282] The light source 4020 is compact and has high resolution. The light source 4020 includes multiple spatially separated sub-light sources 4022 operably coupled to a controller 4030 (described below). For example, the light source 4020 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 4020 may include similar colored LEDs or lasers, each one coupled to a specific region of the display's field of view. In another embodiment, the light source 4020 may comprise a wide area emitter, such as an incandescent or fluorescent lamp, with a mask overlay for segmentation of the emission area and location. Although the sub-light sources 4022 are directly connected to the AR system of FIG. 2B in FIG. 2B, the sub-light sources 222 may be connected to the system via optical fibers (not shown) as long as the distal ends of the optical fibers (away from the sub-light sources 4022) are spatially separated from each other. The system may also include a concentrator (not shown) configured to collimate the light from the light source 4020.

[0283] The SLM 4040 may be reflective (e.g., DLP DMD, MEMS mirror system, LCOS, or FLCOS), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED) in various exemplary embodiments. The type of spatial light modulator (e.g., speed, size, etc.) can be selected to improve the creation of the 3-D perception. While DLP DMDs, operating at higher refresh rates, can be easily incorporated into stationary AR systems, wearable AR systems typically use DLPs of smaller size and power. The power of the DLP changes the way the 3-D depth plane / focal plane is created. The image generation processor 4010 is operably coupled to the SLM 4040, which encodes the light from the light source 4020 with the desired virtual content. As the light from the light source 4020 reflects from, emits from, or passes through the SLM 4040, it may be encoded with image information.

[0284] Referring back to FIG. 25B, the AR system also includes an input optical system 4060 configured to direct light from the light source 4020 (i.e., the multiple spatially separated sub-light sources 4022) and the SLM 4040 into the LOE assembly 4000. The input optical system 4060 may include one or more lenses configured to direct light into the LOE assembly 4000. The input optical system 4060 is configured to form spatially separated and distinct pupils adjacent the LOE 4000 (at separate focal points of the beams exiting the input optical system 4060) corresponding to the spatially separated and distinct beams from the sub-light sources 4022 of the light source 4020. The input optical system 4060 is configured such that the pupils are spatially displaced from one another. In some embodiments, the input optical system 4060 is configured to spatially displace the beams only in the X and Y directions. In such embodiments, the pupils are formed in one X,Y plane. In other embodiments, the launch optical system 4060 is configured to spatially displace the beam in the X, Y, and Z directions.

[0285] The spatial separation of the light beams forms distinct beams and pupils, which allows for placement of internal coupling gratings in distinct beam paths such that each internal coupling grating is primarily addressed (e.g., intersected or impinged) by only one distinct beam (or group of beams). This in turn facilitates the injection of spatially separated light beams into individual LOEs 4000 of the LOE assembly 4000, while minimizing the injection of other light beams from other sub-light sources 4022 of the plurality (i.e., crosstalk). A light beam from a particular sub-light source 4022 enters an individual LOE 4000 through an internal coupling grating thereon (not shown in FIG. 25B, see FIGS. 24-26). The internal coupling gratings of the individual LOEs 4000 are configured to interact with the spatially separated light beams from the plurality of sub-light sources 4022 such that each spatially separated light beam only intersects with the internal coupling grating of one LOE 4000. Thus, each spatially separated light beam is primarily incident on one LOE 4000. Thus, image data encoded by the SLM 4040 onto the light beams from each of the sub-light sources 4022 can be efficiently propagated along the single LOE 4000 for delivery to the user's eye 210.

[0286] Each LOE 4000 is then configured to project an image or sub-image onto the user's retina that appears to originate from a desired depth plane or FOV angular location. The individual multiple LOEs 4000 and sub-light sources 4022 can thus selectively project images (synchronously encoded by the SLM 4040 under the control of the controller 4030) that appear to originate from various depth planes or locations in space. By using each of the individual multiple LOEs 4000 and sub-light sources 4022 to sequentially project images at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full volumetric frame rate of 60 Hz), the system of FIG. 25B can generate 3-D images of virtual objects at various depth planes that appear to exist simultaneously in the 3-D image.

[0287] The controller 4030 communicates with and is operatively coupled to the image generation processor 4010, the light source 4020 (sub-light source 4022), and the SLM 4040, and coordinates the synchronous display of images by instructing the SLM 4040 to encode the light beam from the sub-light source 4022 with appropriate image information from the image generation processor 4010.

[0288] The AR system also includes an optional eye tracking subsystem 4070 configured to track the user's eye 4002 and determine the user's focus. In one embodiment, only a subset of the sub-light sources 4022 may be activated to illuminate a subset of the LOEs 4000 based on input from the eye tracking subsystem, as discussed below. Based on input from the eye tracking subsystem 4070, one or more sub-light sources 4022 corresponding to a particular LOE 4000 may be activated such that an image is generated at a desired depth plane that matches the user's focus / accommodation. For example, if the user's eyes 210 are parallel to one another, the AR system of FIG. 25B may activate sub-light sources 4022 corresponding to the LOEs 4000 that are configured to deliver collimated light to the user's eyes such that the image appears to originate from optical infinity. In another example, if the eye tracking subsystem 4070 determines that the user's focus is one meter away, then sub-light sources 4022 corresponding to the LOE 4000 configured to focus approximately within that range may be activated instead. It should be appreciated that in this particular embodiment, only one group of sub-light sources 4022 is activated at any given time, while the other sub-light sources 4020 are deactivated to conserve power.

[0289] FIG. 25C diagrammatically illustrates light paths within an exemplary viewing optical assembly (VOA) that may be used to present a digital or virtual image to a viewer, according to some embodiments. In some embodiments, the VOA may be incorporated into a system similar to the wearable display device 4050 as depicted in FIG. 25A. The VOA includes a projector 4001 and an eyepiece 200 that may be worn around the viewer's eye. The eyepiece 4000 may correspond to the LOE 4000, for example, as described above with reference to FIG. 25B. In some embodiments, the projector 4001 may include a group of red LEDs, a group of green LEDs, and a group of blue LEDs. For example, the projector 201 may include two red LEDs, two green LEDs, and two blue LEDs, according to an embodiment. In some examples, projector 4001 and its components (e.g., LED light source, reflective collimator, LCoS SLM, and projector relay) as depicted in FIG. 25C may represent or provide the functionality of one or more of light source 4020, sub-light source 4022, SLM 4040, and input optical system 4060 as described above with reference to FIG. 25B. Eyepiece 4000 may include one or more eyepiece layers, each of which may represent one of LOEs 4000 as described above with reference to FIG. 25B. Each eyepiece layer of eyepiece 4000 may be configured to project an image or sub-image onto the retina of a viewer's eye that appears to result from a respective desired depth plane or FOV angular position.

[0290] In one embodiment, the eyepiece 4000 includes three eyepiece layers, one for each of the three primary colors, i.e., red, green, and blue. For example, in this embodiment, each eyepiece layer of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). In another embodiment, the eyepiece 4000 may include six eyepiece layers, i.e., one set of eyepiece layers for each of the three primary colors configured to form a virtual image at one depth plane, and another set of eyepiece layers for each of the three primary colors configured to form a virtual image at another depth plane. For example, in this embodiment, each ocular lens layer in one set of ocular lens layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters), while each ocular lens layer in another set of ocular lens layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from a distance of 2 meters (0.5 diopters). In other embodiments, the eyepiece 4000 may include three or more ocular lens layers, one for each of three primary colors for three or more different depth planes. For example, in such an embodiment, each of the further sets of ocular lens layers may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter).

[0291] Each eyepiece layer may include a planar waveguide and may include an internal coupling grating 4007, an orthogonal pupil expander (OPE) region 4008, and an exit pupil expander (EPE) region 4009. Further details on internal coupling gratings, orthogonal pupil expansion, and exit pupil expansion are described in U.S. Patent Application Nos. 14 / 555,585 and 14 / 726,424, the contents of which are expressly and fully incorporated herein by reference in their entirety as if set forth in their entirety. Still referring to FIG. 25C, the projector 4001 projects image light onto the internal coupling grating 4007 in the eyepiece layer 4000. The internal coupling grating 4007 couples the image light from the projector 4001 into the waveguide and propagates in a direction toward the OPE region 4008. The waveguide propagates the image light horizontally by total internal reflection (TIR). The OPE region 4008 of the eyepiece lens layer 4000 also includes diffractive elements that couple and redirect a portion of the image light propagating in the waveguide toward the EPE region 4009. More specifically, the collimated light propagates horizontally (i.e., relative to the view of FIG. 25C) along the waveguide by TIR, and in so doing repeatedly intersects with the diffractive elements of the OPE region 4008. In some embodiments, the diffractive elements of the OPE region 4008 have a relatively low diffraction efficiency. This causes a percentage (e.g., 10%) of the light to be diffracted vertically downward toward the EPE region 4009 at each point of intersection with a diffractive element of the OPE region 4008, and a percentage of the light to continue on its original trajectory horizontally along the waveguide via TIR. Thus, at each point of intersection with a diffractive element of the OPE region 4008, additional light is diffracted downward toward the EPE region 4009. By splitting the incoming light into multiple outcoupled sets, the exit pupil of the light is expanded horizontally by the diffractive elements in the OPE region 4008. The expanded light that is outcoupled from the OPE region 4008 enters the EPE region 4009.

[0292] The EPE region 4009 of the eyepiece layer 4000 also includes diffractive elements that couple and redirect a portion of the image light propagating in the waveguide toward the viewer's eye 210. Light entering the EPE region 4009 propagates vertically (i.e., relative to the view of FIG. 25C) along the waveguide by TIR. At each point of intersection between the propagating light and the diffractive elements of the EPE region 4009, a percentage of the light is diffracted toward the adjacent face of the waveguide, allowing the light to escape the TIR, emerge from the face of the waveguide, and propagate toward the viewer's eye 210. In this manner, the image projected by the projector 4001 may be viewed by the viewer's eye 210. In some embodiments, the diffractive elements of the EPE region 4009 may be designed or configured to have a phase profile that is the sum of a linear diffraction grating and a radially symmetric diffractive lens. The radially symmetric lens sides of the diffractive elements of EPE region 4009 additionally impart a level of focus to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefront of the individual beams and steering the beams at an angle that matches the designed level of focus. Each beam of light outcoupled by the diffractive elements of EPE region 4009 may be geometrically extended to an individual focal point located in front of the viewer and may be imparted with a convex wavefront profile with a center of radius at the individual focal point, producing an image or virtual object at a given focal plane.

[0293] Further descriptions of such viewing optical assemblies and other similar setups are provided in U.S. Patent Application Nos. 14 / 331,218, 15 / 146,296, and 14 / 555,585, all of which are incorporated herein by reference in their entirety. In some embodiments, it follows that an exemplary VOA may include and / or take the form of one or more components described above with reference to FIG. 25C and described in any of the patent applications incorporated herein by reference. IV. Large Field of View and High Resolution Foveated Displays Using Multiple Optical Paths

[0294] 26A-26D illustrate example rendering viewpoints to be used and light fields to be produced in an AR system for each of two example eye orientations. In FIG. 26A, the viewer's eye 210 is oriented in a first manner relative to the eyepiece 5000. In some embodiments, the eyepiece 5000 may be similar to a stack of LOEs or the eyepiece 4000, as described above with reference to FIGS. 25B and 25C. More specifically, in this example, the viewer's eye 210 is oriented such that the viewer may be able to view the eyepiece 5000 in a relatively straight-ahead direction. In some examples, the AR system to which the eyepiece 5000 belongs, which may be similar to the AR system described above with reference to FIG. 25B, may perform one or more operations to present virtual content at one or more distances in front of the viewer's eye 210 on one or more depth planes located within the viewer's FOV.

[0295] The AR system may determine a viewpoint in the rendering space based on the position and orientation of the viewer's head from which the viewer views 3-D virtual content of the rendering space, such as virtual objects. As described in more detail below with reference to FIG. 29A, in some embodiments, such an AR system may include one or more sensors and utilize data from these one or more sensors to determine the position and / or orientation of the viewer's head. The AR system may include such one or more sensors in addition to one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to FIG. 25B. Using such data, the AR system may effectively map the position and orientation of the viewer's head in the real world to a specific location and a specific angular position in the 3D virtual environment, create a virtual camera that is positioned at the specific location in the 3D virtual environment and oriented at the specific angular position in the 3D virtual environment relative to the specific location in the 3D virtual environment, and render the virtual content for the viewer as would be captured by the virtual camera. Further details discussing the real-world to virtual-world mapping process are provided in U.S. patent application Ser. No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE," which is expressly incorporated by reference in its entirety for all purposes.

[0296] In some examples, the AR system may create or dynamically reposition and / or reorient one such head-tracked virtual camera for the viewer's left eye or eye socket and another such head-tracked virtual camera for the viewer's right eye or eye socket as the viewer's eyes and / or eye sockets are physically separated from one another and therefore consistently positioned in different locations. It follows that virtual content rendered from the perspective of the head-tracked virtual camera associated with the viewer's left eye or eye socket may be presented to the viewer through an eyepiece on the left side of a wearable display device such as those described above with reference to FIGS. 25A-25C, and virtual content rendered from the perspective of the head-tracked virtual camera associated with the viewer's right eye or eye socket may be presented to the viewer through an eyepiece on the right side of the wearable display device. A head-tracking virtual camera may be created and / or dynamically repositioned for each eye or eye socket based on information about the current position and orientation of the viewer's head, although such head-tracking virtual camera positions and orientations may be independent of either the viewer's individual eye sockets or the position or orientation of each of the viewer's eyes relative to the viewer's head. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process are provided in U.S. Patent Application Serial No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTUAL FEATURES IN 3D RECONSTRUCTION," which is expressly incorporated by reference in its entirety herein for all purposes.

[0297] The AR system of Figure 26A may create or dynamically reposition and / or reorient such a head-tracked virtual camera to render virtual content from the head-tracked virtual camera's viewpoint (point of view 5010) and project light representing the rendering of the virtual content through the eyepiece 5000 and onto the retina of the viewer's eye 210. As shown in Figure 26A, the head-tracked rendering viewpoint 5010 may be aligned diagonally, horizontally, and / or vertically at angles ±θ 310The head-tracking rendering viewpoint 5010 may provide a FOV that spans an area of ​​angular units. As described in more detail below, in some embodiments, the head-tracking rendering viewpoint 5010 may provide a relatively wide FOV. In such embodiments, the AR system may also create or dynamically reposition and / or reorient a separate virtual camera for each eye or eye socket, different from and in addition to the head-tracking virtual camera. In the example of FIG. 26A, the AR system may render and present virtual content from the viewpoint of the head-tracking virtual camera 5010 as well as virtual content from the viewpoint of another virtual camera in the rendering space.

[0298] For example, in such an embodiment, the AR system of FIG. 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera based on the current line of sight of the viewer's eye 210. As described in more detail below with reference to FIG. 29A, in some examples, such an AR system may include one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to FIG. 25B, to determine the viewer's current line of sight, the current position and / or orientation of the viewer's eye 210 relative to the viewer's head, and the like. Using such data, the AR system of FIG. 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera, render virtual content from the viewpoint of the foveal tracking virtual camera (viewpoint 5020A), and project light representing the virtual content as rendered from viewpoint 5020A through the eyepiece 5000 onto the fovea of ​​the viewer's eye 210.

[0299] 26A , the foveated tracking rendering viewpoint 5020A may provide a narrower FOV than that of the head tracking rendering viewpoint 5010. In this manner, the FOV of the foveated tracking rendering viewpoint 5020A may be viewed as occupying a conical subspace of the FOV of the head tracking rendering viewpoint 5010. That is, the FOV of the foveated tracking rendering viewpoint 5020A may be a subfield of the FOV of the head tracking rendering viewpoint 5010. For example, as shown in FIG. 26A , the foveated tracking rendering viewpoint 320A may provide a narrower FOV than that of the head tracking rendering viewpoint 5010 such that the relationship between the FOV of the head tracking rendering viewpoint 5010 and the foveated tracking rendering viewpoint 5020A is −θ 310 ≦-θ 320A ≦θ 320A ≦θ 310 Diagonally, horizontally, and / or vertically, ±θ 320A The head tracking virtual camera 5010 may provide an FOV that spans an area of ​​angular units. In some examples, the FOV of the head tracking rendering viewpoint 5010 may be at least as wide as the viewer's ocular field of view, which in this example would be the total cone-shaped space that the viewer's eye 210 may fixate on when the viewer's head is held in a given position and orientation. Thus, in these examples, the head tracking virtual camera and the foveal tracking virtual camera may be positioned at substantially the same location in the rendering space, or may be positioned at locations in the rendering space that are a fixed distance from each other such that both virtual cameras may be translated linearly and / or angularly in unison within the rendering space as the position and / or orientation of the viewer's head changes. For example, the head tracking virtual camera may be positioned at a location in the rendering space that corresponds to the center of rotation of the viewer's eye 210, while the foveal tracking virtual camera may be positioned at a location in the rendering space that corresponds to the region of the viewer's eye 210 between the center of rotation and the cornea. In fact, the Euclidean distance between two virtual cameras can remain substantially constant when translated in the rendering space, in the same way that the Euclidean distance between two concrete regions of a viewer's eye 210 or another rigid body can remain substantially constant at all times.

[0300] The spatial relationship between each virtual camera in such a pair of virtual cameras may remain substantially fixed in the rendering space throughout use of the AR system; however, in these examples, the orientation of the foveated tracking virtual camera may vary relative to the head-tracking virtual camera as the viewer rotates his or her eye 210. In this manner, the conical subspace of the head-tracking virtual camera's FOV occupied by the foveated tracking virtual camera's FOV may dynamically change as the viewer rotates his or her eye 210.

[0301] Additionally, virtual objects and other content falling within the foveal tracking rendering viewpoint 5020A may be rendered and presented by the AR system at a relatively high resolution. More specifically, the resolution at which virtual content within the FOV of the foveal tracking virtual camera is rendered and presented may be higher than the resolution at which virtual content within the FOV of the head tracking virtual camera is rendered and presented. Thus, the highest resolution subfield of a given light field coupled out by the eyepiece 5000 and projected onto the retina of the viewer's eye 210 may be the one that reaches the fovea of ​​the viewer's eye 210.

[0302] FIG. 3B illustrates an exemplary light field 5030A that is coupled out by the eyepiece 5000 and projected onto the retina of the viewer's eye 210 while the viewer's eye 210 is oriented in a first manner as depicted in and described above with reference thereto. The light field 5030A may include various angular light components that represent the virtual content as would be captured in the rendering space by the pair of virtual cameras described above. As described in more detail below with reference to FIG. 26A et seq., the light representing the virtual content as would be captured in the rendering space by the head-tracking virtual camera and the light representing the virtual content as would be captured in the rendering space by the foveal tracking virtual camera may be multiplexed by the AR system according to any of a variety of different multiplexing schemes. The adoption of such multiplexing schemes may, at least in some instances, enable the AR system to operate with greater efficiency and / or occupy less physical space.

[0303] Still referring to FIG. 26B , the angular light components of the light field 5030A, which represent the virtual content (e.g., virtual objects and other content that fall within the head-tracking rendering viewpoint 5010) as it would be captured in the rendering space by the head-tracking virtual camera, are angled at −θ 310 ~+θ 310 The angular light components of the light field 5030A, which represent the virtual content as it would be captured in the rendering space by the foveated tracking virtual camera (e.g., virtual objects and other content that fall within the foveated tracking rendering viewpoint 5020A), may be projected onto the retina of the viewer's eye 210 at angles spanning angular units. 320A ~+θ 320A Such angular light components associated with the foveal tracking rendering viewpoint 5020A may include those that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. 320A and +θ320A The spacing between angle units is −θ 310 and +θ 310 The spacing between angle units may be more regular than the spacing between angle units. In this manner, the resolution at which virtual content associated with the foveal tracking rendering viewpoint 5020A may be rendered and presented to the viewer may be higher than the resolution at which virtual content associated with the head tracking rendering viewpoint 5010 may be rendered and presented to the viewer.

[0304] In some embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030A are further denoted by −θ 320A ~+θ 320A 5010, which would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. In such an embodiment, such angular light components associated with the head tracking rendering viewpoint 5010 occur within the light field 5030A, such as -θ 320A and +θ 320A The spacing between angle units is −θ 320A and +θ 320A In other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030A are spaced apart by −θ 320A ~+θ 320A 5030A, the angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030A may be excluded from the -θ 310 ~-θ 320A Angle or θ in degrees 320A ~θ 310 2. may be projected onto the retina of the viewer's eye 210 at an angle of

[0305] In Fig. 26C, the viewer's eye 210 is oriented relative to the eyepiece 5000 in a second manner that is different from the first manner in which the viewer's eye 210 is oriented relative to the eyepiece 5000 in Figs. 26A-26B. For purposes of the example, the position and orientation of the viewer's head in Figs. 26C-26D may be treated as being identical to the position and orientation of the viewer's head as described above with reference to Figs. 26A-26B. Thus, Figs. 26A-26B and Figs. 26C-26D may represent the viewer and AR system described above at first and second time series stages, respectively. More specifically, in this example, the viewer's eye 210 is rotated off-center from a relatively straight-ahead orientation as depicted in Figs. 26A-26B.

[0306] During the transition from the first stage to the second stage, the AR system of FIG. 26C may function, for example, to maintain the head-tracking virtual camera in the same position and orientation as described above with reference to FIGS. 26A-26B, since the viewer's head pose (e.g., position and orientation) has not changed. Thus, during the second stage depicted in FIGS. 26C-26D, the AR system may render the virtual content from the head-tracking virtual camera's viewpoint (i.e., head-tracking rendering viewpoint 5010) and project light representing the rendering of the virtual content through the eyepiece 5000 onto the retina of the viewer's eye 210. Although the head-tracking rendering viewpoint 5010 may remain static or relatively static throughout the first and second time-sequential stages of FIGS. 26A-26D, during the transition from the first stage to the second stage, the AR system may function to adjust the orientation of the foveal tracking virtual camera in the rendering space based on the change in the line of sight of the viewer's eye 210 from the first stage to the second stage. That is, the AR system may displace or reorient the foveal tracking virtual camera as employed in the first stage to provide the foveal tracking rendering viewpoint 5020A, such that the foveal tracking virtual camera as employed in the second stage provides a foveal tracking rendering viewpoint 5020C that is different from the foveal tracking rendering viewpoint 5020A. It follows that in the second stage, the AR system may also render the virtual content from the perspective of the foveal tracking virtual camera viewpoint 5020C and project light representing the rendering of the virtual content through the eyepiece 5000 onto the fovea of ​​the viewer's eye 201.

[0307] 26C-26D, the foveated tracking rendering viewpoint 5020C may occupy a different cone-shaped subspace of the head tracking rendering viewpoint 5010 than that of the foveated tracking rendering viewpoint 5020A. For example, as shown in FIG. 26C, the foveated tracking rendering viewpoint 5020C is θ 320C Displaced in degrees, diagonally, horizontally, and / or vertically, ±θ 320AThat is, the foveated tracking rendering viewpoint 5020C may provide an FOV that spans an area in angular units, i.e., diagonally, horizontally, and / or vertically, θ 320C ±θ 320A A FOV may be provided, covering an area in degrees.

[0308] 26D illustrates an example light field 5030C that is coupled out by the eyepiece 5000 and projected onto the retina of the viewer's eye 201 while the viewer's eye 201 is oriented in a second manner as depicted in and described above with reference to FIG. 26C. The light field 5030C may include various angular light components that represent the virtual content as it would be captured in rendering space from the head tracking rendering viewpoint 5010 and the foveal tracking rendering viewpoint 5020C. The angular light components of the light field 5030C that represent the virtual content as it would be captured in rendering space from the head tracking rendering viewpoint 5010 are angled at -θ relative to the viewer's eye 210. 310 ~+θ 310 26A-26B, starting from the first stage, however, the angular light components of the light field 5030C, representing the virtual content as it would be captured in the rendering space by the foveated tracking virtual camera (e.g., virtual objects and other content that fall within the foveated tracking rendering viewpoint 5020C), are angled θ 320C -θ 320A Angle unit ~θ 320C +θ 320A This may include what would be projected onto the retina of the viewer's eye 210 at angles spanning angular units.

[0309] Such angular light components associated with the foveated tracking rendering viewpoint 320C occur within the light field 5030C, θ 320C -θ 320A Angle units and θ 320C +θ 320AThe spacing between the angular units is −θ 310 and +θ 310 In this manner, the resolution at which virtual content associated with the foveal tracking rendering viewpoint 5020C may be rendered and presented to the viewer is notably −θ 320A ~+θ 320A The resolution may be higher than the resolution at which virtual content associated with the head tracking rendering viewpoint 5010, including virtual content represented by angular light components that will be projected onto the retina of the viewer's eye 210 at angles spanning angular units, can be rendered and presented to the viewer.

[0310] In some embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 occurring within the light field 5030C are further denoted by θ 320C -θ 320A Angle unit ~θ 320C +θ 320A In such an embodiment, such angular light components associated with the head tracking rendering viewpoint 310 may occur in the light field 5030C at angles ranging from -θ 320C -θ 320A Angle units and θ 320C +θ 320A The spacing between angle units is the angle at which the angular light components associated with the foveated tracking rendering viewpoint 5020C occur within the light field 5030C, θ 320C -θ 320A Angle units and θ 320C +θ 320A In other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030C are spaced apart by θ 320C -θ 320A Angle unit ~θ 320C +θ 320A5030C, may exclude those that would be projected onto the retina of the viewer's eye 210 at angles spanning angular units. Thus, in these other embodiments, the angular light components associated with the head tracking rendering viewpoint 5010 that occur within the light field 5030C may be expressed as -θ 310 ~θ 320C -θ 320A Angle and θ in degrees 320C +θ 320A Angle~θ 310 It may be what would be projected onto the retina of the viewer's eye 210 in degrees or degrees in degrees.

[0311] 26E-26F diagrammatically illustrate example configurations of images that may be presented to a user, according to some embodiments. Note that the grid squares in FIGS. 26E-26F diagrammatically represent image points where the fields of view 3002, 3004, and 3006 are defined in a two-dimensional angular space as described above with reference to FIG. 24. The low-resolution first image stream 5010E with a wide FOV may be displayed at a static location. The low-resolution first image stream 5010E with a wide FOV may represent one or more images of the virtual content as would be captured by a first virtual camera having a static position and orientation in the rendering space. For example, the low-resolution first image stream 5010E may represent one or more images of the virtual content as would be captured by a head-tracked virtual camera, such as the head-tracked virtual camera described above with reference to FIGS. 26A-26D. The first image stream 5010E may encompass the user's vision and evoke an immersive experience for the user.

[0312] A high-resolution second image stream 5020E having a relatively narrow FOV can be displayed within the bounds of the first image stream 5010E. In some examples, the second image stream 5020E can represent one or more images of the virtual content as would be captured by a second, different virtual camera having an orientation in the rendering space that can be dynamically adjusted in real time to an angular position that coincides with the user's current fixation point based on data acquired using eye-tracking techniques. In these examples, the high-resolution second image stream 5020E can represent one or more images of the virtual content as would be captured by a foveal tracking virtual camera, such as the foveal tracking virtual camera described above with reference to FIGS. 26A-26D. In other words, the viewpoint in the rendering space from which one or more images of the virtual content represented by the second image stream 5020E are captured can be reoriented as the user's eye line of sight changes, such that the viewpoint associated with the second image stream 5020E is persistently aligned with the user's foveal vision.

[0313] For example, the second image stream 5020E can contain virtual content located within a first region of the rendering space when the user's eye line of sight is fixated at a first location, as illustrated in FIG. 26E. As the user's eye line of sight moves to a second location different from the first location, the second image stream 5020E can contain virtual content located within a second region of the rendering space, as illustrated in FIG. 26F, such that a viewpoint associated with the second image stream 5020E can be adjusted. In some embodiments, the first image stream 5010E has a wide FOV but low angular resolution, as indicated by the sparse grid. The second image stream 5020E has a narrow FOV but high angular resolution, as indicated by the dense grid.

[0314] FIG. 26G diagrammatically illustrates an example configuration of images that may be presented to a user, according to some other embodiments. As in FIG. 26E-26F, the grid squares in FIG. 26G diagrammatically represent image points defined in a two-dimensional angular space. Similar to the configuration illustrated in FIG. 26E-26F, a first image stream 5010G of low resolution with a wide FOV contains the virtual content as viewed from a head-tracking rendering viewpoint, while a second image stream 5020G of high resolution with a narrow FOV contains the virtual content as viewed from a foveated tracking rendering viewpoint that may be dynamically reoriented to match the user's current fixation point. Here, the outer boundary of the FOV associated with the first image stream 5010G may form a rectangular boundary with rounded corners, and the outer boundary of the FOV associated with the second image stream 5020G may form a circular boundary.

[0315] FIG. 26H also diagrammatically illustrates an example configuration of images that may be presented to a user according to some other embodiments. As in FIGS. 26E-26G, the grid squares in FIG. 26H diagrammatically represent image points defined in a two-dimensional angular space. Here, both the outer boundary of the FOV associated with the first image stream 5010H and the outer boundary of the FOV associated with the second image stream 5020H may form a circular boundary. In some other embodiments, one or both of the outer boundary of the FOV associated with the first image stream 5010H and the outer boundary of the FOV associated with the second image stream 5020H may form an elliptical boundary or other shape. In some embodiments, the image source of the AR system of FIG. 26H may include a scanning fiber that may be scanned in a predetermined pattern to provide light beams for the first image stream 5010H and the second image stream 5020H with a desired boundary shape.

[0316] Fig. 27 illustrates the field of view 3002 and ocular field of view 3004 as shown in Fig. 24 overlaid on one of the displays 4052 in a wearable display device 4050 as shown in Fig. 25A. According to some embodiments, a first image stream 5010E with a wide FOV and low resolution as illustrated in Figs. 26E-26F may be displayed across the entire area of ​​the display 4052 (the relatively low resolution of the first image stream 5010E is illustrated with a sparse grid), while a second image stream 5020E with a narrow FOV and high resolution may be displayed in the user's current foveated region 3006 (the relatively high resolution of the second image stream 5020E is illustrated with a dense grid). In FIG. 27, the first image stream 5010E and the second image stream 5020E are illustrated as being displayed on the "plane" of the display 4052, but in a see-through augmented reality (AR) display system, the first image stream 5010E and the second image stream 5020E can also be presented to the user as a light field within an angular field of view. Such an AR display system can produce a display plane that appears to be "floating" at a distance (e.g., 2 meters) in front of the user. The display plane can appear to be much larger than the glasses. This floating distance display is used to overlay information about the real world.

[0317] 28A-28B illustrate some of the principles described in FIGS. 26A-26D using example virtual content that may be presented to a user, according to some embodiments. Thus, FIGS. 28A-28B may represent a viewer and an AR system at first and second time-series stages, respectively. Additionally, some or all of the components shown in FIGS. 28A-28B may be the same as or at least similar to components as described above with reference to FIGS. 26A-26D.

[0318] The AR system of FIGS. 28A-28B creates or dynamically relocates and / or reorients a head-tracking virtual camera similar to the head-tracking virtual camera described above with reference to FIGS. 26A-26D, renders virtual content from the perspective of the head-tracking virtual camera, and projects light representing the rendering of the virtual content onto the retina of the viewer's eye 210 through the eyepiece 6000. The AR system of FIGS. 28A-28B also creates or dynamically relocates and / or reorients a fovea-tracking virtual camera similar to the fovea-tracking virtual camera described above with reference to FIGS. 26A-26D, renders virtual content from the perspective of the fovea-tracking virtual camera, and projects light representing the rendering of the virtual content onto the fovea of the viewer's eye 210 through the eyepiece 400. As shown in FIGS. 28A-28B, such virtual content may include 3-D virtual objects 6011, 6012, and 6013. In some embodiments, the AR system of FIGS. 28A-28B may simultaneously perform one or more of the operations described immediately above with respect to the head-tracking rendering perspective and one or more of the operations described immediately above with respect to the fovea-tracking rendering perspective. In other embodiments, the AR system of FIGS. 28A-28B may perform such operations in rapid succession.

[0319] In this example, the FOV of the head tracking rendering viewpoint adopted by the AR system in Figures 28A-28B may be wide enough in angular space to encompass each of the virtual objects 6011, 6012, and 6013, diagonally, horizontally, and / or vertically. For purposes of the example, the position and orientation of the viewer's head may be treated as static throughout the first and second phases as depicted in Figures 28A and 28B, respectively, such that the position and orientation of the head tracking rendering viewpoint remains the same throughout the two phases. For the FOV of the head tracking rendering viewpoint adopted by the AR system to be wide enough to encompass the virtual objects 6011-6013, it must span at least α+ζ angular units diagonally, horizontally, and / or vertically. More specifically, in the example of FIGS. 28A-28B, it can be seen that virtual objects 6011, 6012, and 6013 may span the α-β, γ+δ, and ζ-ε angular unit ranges, respectively.

[0320] In FIG. 28A, the viewer's eye 210 is oriented in a first manner relative to the eyepiece 6000 such that the viewer may be able to view the eyepiece 6000 in a relatively straight-ahead direction. The orientation of the viewer's eye 210 in FIG. 28A may be the same as or similar to the orientation of the viewer's eye 210 as described above with reference to FIGS. 26A-26B, for example, and may be determined by the AR system using a sensing component and / or one or more of the techniques described herein. Thus, at the stage depicted in FIG. 28A, the AR system may adopt a head-tracking and foveated-tracking rendering viewpoint at a relative position and orientation similar to those of the head-tracking and foveated-tracking rendering viewpoints 5010 and 5020A, respectively. In the particular example of FIG. 28A, the FOV of the foveated-tracking rendering viewpoint adopted by the AR system may, for example, encompass the virtual object 6012, but not encompass any of the virtual objects 6011 and 6013. 28A, it follows that the AR system may render virtual object 6012 in high resolution as it would be captured from the perspective of a foveated tracking virtual camera, and may render virtual objects 6011 and 6013 in lower resolution as it would be captured from the perspective of a head tracking virtual camera. In addition, the AR system may project light representing such renderings of virtual objects 6011, 6012, and 6013 through the eyepiece 6000 onto the retina of the viewer's eye 210. In some embodiments, the AR system may also render virtual object 6012 in lower resolution as it would be captured from the perspective of a head tracking virtual camera.

[0321] 28A also illustrates an example light field...

Claims

[Claim 1] The invention described in this specification.

Images