Depth-based foveal rendering for display systems

Abstract

To provide suitable depth-based foveated rendering for a display system.SOLUTION: A display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes by using different wavefront divergence. Some embodiments include monitoring eye orientations of a user of a display system based on detected sensor information. A fixation point is determined based on the eye orientations, and the fixation point represents a three-dimensional location with respect to a field of view. Location information on virtual objects to present is obtained, and the location information indicates three-dimensional positions of the virtual objects. The resolution of at least one virtual object is adjusted based on the degree of proximity of the at least one virtual object to the fixation point. The virtual objects are presented to the user by the display system, and the at least one virtual object is rendered according to the adjusted resolution.SELECTED DRAWING: Figure 10A

Description

[Technical Field]

[0001] (Claiming priority) This application claims priority to U.S. Provisional Application No. 62 / 644,365 (filed March 16, 2018), U.S. Provisional Application No. 62 / 475,012 (filed March 22, 2017), U.S. Provisional Application No. 62 / 486,407 (filed April 17, 2017), and U.S. Provisional Application No. 62 / 539,934 (filed August 1, 2017). The above patent applications are incorporated herein by reference in their entirety for all purposes.

[0002] (Referenced) This application incorporates by reference, in its entirety, the following patent applications and publications: U.S. Application No. 14 / 555,585 (filed November 27, 2014), published U.S. Publication No. 2015 / 0205126 on July 23, 2015; U.S. Application No. 14 / 690,401 (filed April 18, 2015), Published on October 22, 2015, as U.S. Publication No. 2015 / 0302652; U.S. Application No. 14 / 212,961 (filed March 14, 2014), currently U.S. Patent No. 9,417,452 issued August 16, 2016; U.S. Application No. 14 / 331,218 (filed July 14, 2014), published on October 29, 2015, as U.S. Publication No. 2015 / 0309263; U.S. Application No. 15 / 902,927 (filed February 22, 2018); U.S. Provisional Application No. 62 / 475,012 (filed March 22, 2017); and U.S. Provisional Application No. 62 / 539,934 (filed August 1, 2017).

[0003] This disclosure relates to a display system, including an augmented reality imaging and visualization system. [Background technology]

[0004] Modern computing and display technologies are driving the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears, or can be perceived, as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs, while augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality or "MR" scenarios are a type of AR scenario that typically involves virtual objects integrated into and responding to the natural world. For example, an MR scenario may include AR image content that appears blocked by, or is perceived to interact with, objects in the real world in a different way.

[0005] Referring to Figure 1, an augmented reality scene 10 is depicted. The user of the AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. The user also perceives that they are "seeing" "virtual content" such as a robot figure 40 standing on the real-world platform 30 and a flying cartoon-like avatar character 50 that looks like an anthropomorphic bumblebee. These elements 50 and 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex, and producing AR technology that facilitates a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements is difficult.

[0006] The systems and methods disclosed herein address various challenges related to AR or VR technologies. [Overview of the project] [Means for solving the problem]

[0007] According to some embodiments, the system comprises one or more processors and one or more computer storage media that, when executed by the one or more processors, store instructions causing the one or more processors to perform an action. The action includes the step of monitoring the user's eye movements based on information detected via one or more sensors. A fixation point on which the user's eyes are fixated is determined based on the eye movements, and the fixation point is a three-dimensional location within the user's field of view. The action includes the step of obtaining location information associated with one or more virtual objects to be presented to the user, the location information indicating the three-dimensional position of the virtual objects. The action also includes the step of adjusting the resolution of at least one virtual object based at least in part on the proximity of at least one virtual object to the fixation point. The action also includes the step of presenting the virtual objects to the user via a display, the at least one virtual object being rendered according to the adjusted resolution.

[0008] According to some embodiments, the display system comprises a display device configured to present virtual content to a user, one or more processors, and one or more computer storage media that, when executed by the system, store instructions causing the system to perform actions. The action includes monitoring information associated with the user's eye movement. A fixation point within the display frustum of the display device is determined based on the monitored information, and the fixation point indicates a three-dimensional location where the user's eye is fixed. The action also includes presenting virtual content at the three-dimensional location within the display frustum based on the determined fixation point, and the resolution of the virtual content is adjusted based on the proximity of the virtual content to the fixation point.

[0009] According to some other embodiments, the method includes the step of monitoring the eye orientation of a user of a display device based on information detected via one or more sensors. A fixation point on which the user's eyes are fixated is determined based on the eye orientation, and the fixation point is a three-dimensional location within the user's field of view. Location information associated with one or more virtual objects to be presented to the user is acquired, and the location information indicates the three-dimensional position of the virtual objects. The resolution of at least one virtual object is adjusted at least partially based on the proximity of at least one virtual object to the fixation point. The method also includes the step of bringing to the user a presentation of the virtual objects via the display, and at least one virtual object is rendered according to the adjusted resolution.

[0010] According to some embodiments, the display system comprises a frame configured to be mounted on the user's head; an optical modulation system configured to emit light and form an image; and one or more waveguides mounted on the frame and configured to receive light from the optical modulation system and emit light across the surface of one or more waveguides. The system also comprises one or more processors and one or more computer storage media that, when executed by the one or more processors, store instructions causing the one or more processors to perform various operations. The operations include the steps of determining the amount of light that reaches the retina of the user's eye and adjusting the resolution of virtual content to be presented to the user based on the amount of light that reaches the retina.

[0011] According to some other embodiments, the display system comprises one or more processors and one or more computer storage media for storing instructions. When instructions are executed by one or more processors, they cause one or more processors to perform various operations. These operations include determining the amount of light that reaches the retina of the user's eye of the display system, and adjusting the resolution of virtual content to be presented to the user based on the amount of light that reaches the retina.

[0012] According to some embodiments, the method is carried out by a display system comprising one or more processors and a head-mountable display. The method includes the steps of determining the amount of light that reaches the retina of the user's eye of the display system, and adjusting the resolution of virtual content to be presented to the user based on the amount of light that reaches the retina.

[0013] According to some other embodiments, the display system comprises a frame, an optical modulation system, one or more waveguides, one or more processors, and one or more computer storage media for storing instructions, all configured to be mounted on the user's head. The optical modulation system is configured to emit light and form an image. One or more waveguides are mounted on the frame and are configured to receive light from the optical modulation system and emit light across the surface of one or more waveguides. One or more computer storage media store instructions that, when executed by one or more processors, cause one or more processors to perform various operations. The operations include adjusting the resolution of the primary color images forming the virtual content based on the proximity of the virtual content from the user's fixed viewpoint and the colors of the primary color images. At least one of the primary color images has a different resolution from another primary color image.

[0014] In yet another embodiment, the display system comprises one or more processors and one or more computer storage media for storing instructions. When instructions are executed by one or more processors, they cause one or more processors to perform various operations. The operations include adjusting the resolution of the primary color images forming the virtual content based on the proximity of the virtual content from the user's fixed viewpoint and the colors of the primary color images, wherein at least one of the primary color images has a different resolution from a primary color image of another color.

[0015] According to some other embodiments, the method is carried out by a display system comprising one or more processors and a head-mountable display. The method includes adjusting the resolution of primary color images forming virtual content based on the proximity of the virtual content from the user's fixed viewpoint and the colors of the primary color images, wherein at least one of the primary color images has a different resolution from a primary color image of another color.

[0016] In yet another embodiment, the display system comprises an image source equipped with a spatial light modulator for providing a first image stream and a second image stream; a viewing assembly; one or more processors communicating with the image source; and one or more computer storage media storing instructions, when executed by the one or more processors, for causing the one or more processors to perform various operations. The viewing assembly includes an optical guidance optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to the user. Various operations performed by one or more processors include the steps of causing the image source to output a first image stream to the viewing assembly, wherein the image formed by the first image stream has a first pixel density; and causing the image source to output a second image stream to the viewing assembly. The image formed by the second image stream has a second pixel density that is greater than the first pixel density and corresponds to a portion of the image provided by the first image stream. The image formed by the second image stream covers a corresponding portion of the field of view provided by the first image stream.

[0017] According to some embodiments, a wearable display system may include an infinite focus magnifying lens with a circular polarization-dependent magnification. The infinite focus magnifying lens may include a first fixed focal length lens element, a first geometric phase lens exhibiting a positive refractive power for a first polarization of incident circular polarization and a negative refractive power for a second polarization of incident circular polarization, and a second geometric phase lens.

[0018] According to some other embodiments, the optical subsystem for a wearable image projector may include a polarization-selective reflector and a set of four lens elements positioned around the polarization-selective reflector.

[0019] A display system for projecting an image onto a user's eye, according to some other embodiments, may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include a first image source configured to project a first light beam associated with a first image stream. The first image stream may have a first field of view and may be incident on a first surface of the internal coupling grating. A portion of the first light beam may be coupled into the waveguide by the internal coupling grating to position the first image stream in a fixed position relative to the user's eye. The display system may further include a second image source configured to project a second light beam associated with a second image stream. The second image stream may have a second field of view that is narrower than the first field of view. The display system may further include a scanning mirror configured to receive and reflect the second light beam such that the second light beam is incident on a second surface of the internal coupling grating opposite its first surface. A portion of the second light beam may be coupled into the waveguide by the internal coupling grating. The display system may further include an eye-line tracker configured to detect the movement of the user's eyes, and a control circuit that communicates with the eye-line tracker and a scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of a second image stream moves in accordance with the detected movement of the user's eyes.

[0020] According to some other embodiments, a display system for projecting an image onto a user's eye may include an eyepiece. The eyepiece may include a waveguide and an internal coupling grating optically coupled to the waveguide. The display system may further include an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view, and the second image stream may have a second field of view that is narrower than the first field of view. The first and second light beams may be multiplexed. The display system may further include a polarization beam splitter configured to receive the first light beam and reflect it along a first optical path, and to receive the second light beam and transmit it along a second optical path. The display system may further include a first optical reflector positioned along a first optical path and configured to receive and reflect a first light beam such that the first light beam is incident on a first surface of an internally coupled grating. A portion of the first light beam may be coupled into a waveguide by the internally coupled grating to position a first image stream in a fixed position relative to the user's eye. The display system may further include a scanning mirror positioned along a second optical path and configured to receive and reflect a second light beam, and a second optical reflector positioned along a second optical path downstream of the scanning mirror. The second optical reflector may be configured to receive and reflect a second light beam such that the second light beam is incident on a second surface of an internally coupled grating opposite its first surface. A portion of the second light beam may be coupled into a waveguide by the internally coupled grating. The display system may further include an eye-line tracker configured to detect the movement of the user's eye, and a control circuit that communicates with the eye-line tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream moves in accordance with the detected movement of the user's eye.

[0021] According to some other embodiments, a display system for projecting an image onto a user's eye may include a waveguide and an image source configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view, and the second image stream may have a second field of view that is narrower than the first field of view. The first and second light beams may be multiplexed. The display system may further include a polarization beam splitter configured to receive the first light beam and reflect it along a first optical path, and to receive the second light beam and transmit it along a second optical path. The display system may further include a first internal coupling prism positioned adjacent to a first surface of the waveguide along the first optical path. The first internal coupling prism may be configured to couple a portion of the first light beam into the waveguide in order to position the first image stream in a fixed position relative to the user's eye. The display system may further include a scanning mirror positioned along a second optical path and configured to receive and reflect a second light beam. The display system may further include a second internal coupling prism positioned along a second optical path downstream of the scanning mirror, adjacent to the second surface of the waveguide opposite the first surface of the waveguide. The second internal coupling prism may be configured to couple a portion of the second light beam into the waveguide. The display system may further include an eye-line tracker configured to detect the movement of the user's eye, and a control circuit that communicates with the eye-line tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream moves in accordance with the detected movement of the user's eye.

[0022] According to one embodiment, a display system for projecting an image onto a user's eye includes an image source. The image source may be configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view, and the second image stream may have a second field of view that is narrower than the first field of view. The first and second light beams may be multiplexed. The display system may further include a polarization beam splitter. The polarization beam splitter may be configured to receive the first light beam and reflect it along a first optical path toward a viewing assembly, and to receive the second light beam and transmit it along a second optical path, in order to position the first image stream in a fixed position relative to the user's eye. The display system may further include a scanning mirror positioned along a second optical path and configured to receive the second light beam and reflect it toward a viewing assembly. The display system may further include an eye-line tracker configured to detect the movement of the user's eyes, and a control circuit that communicates with the eye-line tracker and a scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of a second image stream moves in accordance with the detected movement of the user's eyes.

[0023] According to another embodiment, a display system for projecting an image onto a user's eye includes an image source. The image source may be configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream. The first image stream may have a first field of view, and the second image stream may have a second field of view that is narrower than the first field of view. The first and second light beams may be multiplexed. The display system may further include a scanning mirror configured to receive the first and second light beams and reflect them toward a viewing assembly in order to project the first and second image streams. The display system may further include an eye-line tracker configured to detect the movement of the user's eye, and a control circuit that communicates with the eye-line tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the positions of the first and second image streams move in accordance with the detected movement of the user's eye. The display system may further include a switchable optical element positioned within the optical paths of the first and second light beams. The switchable optical element can be configured to switch to a first state for a first light beam so that the first light beam is angularly magnified by a first angular magnification factor, and to switch to a second state for a second light beam so that the second light beam is angularly amplified by a second angular magnification factor that is less than the first angular magnification factor. The present invention provides, for example, the following: (Item 1) A system comprising one or more processors and one or more computer storage media, wherein the one or more computer storage media store instructions, and when an instruction is executed by the one or more processors, the one or more processors... Based on information detected via one or more sensors, monitor the user's eye movements, Based on the aforementioned eye movement, the fixation point on which the user's eye is fixated is determined, wherein the fixation point is a three-dimensional location within the user's field of view. To obtain location information associated with one or more virtual objects for presentation to the user, wherein the location information indicates the three-dimensional position of the virtual objects. At least partially, the resolution of at least one virtual object is adjusted based on the proximity of the at least one virtual object to the fixed point, The presenting of one or more virtual objects to the user via a display device, wherein at least one virtual object is rendered according to the individually adjusted resolution. A system that performs actions including those mentioned above. (Item 2) The system according to item 1, wherein adjusting the resolution of a virtual object reduces the presentation quality of the said virtual object. (Item 3) The system described in item 2, which reduces the presentation quality of the virtual object, includes one or more of the following: reducing the polygon count of the virtual object, adjusting the primitives used to generate the virtual object, adjusting the actions performed on the virtual object, adjusting the texture information, adjusting the color resolution or depth, and adjusting the number of rendering cycles or the frame rate. (Item 4) The field of view corresponds to the display frustum of the display device, as described in item 1. (Item 5) The field of view comprises a plurality of zones, each zone representing a volume of space within the field of view in which virtual objects can be presented by the display device. The zone extends from the initial depth within the field of view to the end depth within the field of view, The proximity of the one or more virtual objects from the fixed point corresponds to the distance that separates the first zone where the virtual objects will be presented from the second zone where the fixed point is located. The system described in item 1. (Item 6) The system according to item 5, wherein adjusting the resolution of the at least one virtual object includes reducing the resolution from the resolution initially assigned to the at least one virtual object, and the proximity corresponds to the total number of zones separating the first zone from the second zone. (Item 7) The system according to item 5, wherein the second zone where the fixed point is located is assigned the maximum resolution, and the remaining zones are each assigned a resolution reduced from the maximum resolution according to one or more reduction rates. (Item 8) The aforementioned operation further, To provide a user retinal cone density profile. Includes, The system described in item 5, wherein the rate of descent on each depth plane substantially corresponds to the rate of descent of cone density in the user retinal cone density profile. (Item 9) The system described in item 5, wherein the zone has a three-dimensional polygonal shape. (Item 10) The aforementioned zone is a concentric ellipsoid, as described in item 5. (Item 11) The system according to item 5, wherein the shape of the aforementioned zone varies with respect to the distance to the second zone. (Item 12) The display device further comprises the aforementioned display device, A plurality of stacked waveguides that form a display area and provide a view of the surrounding environment through the display area, wherein at least some of the waveguides are configured to output light with a different wavefront divergence than the other waveguides. The system described in item 1, comprising the features described in item 1. (Item 13) With respect to a first virtual object which is presented at a depth further from the fixed point and is determined to be within the threshold angular distance of the user's line of sight, the operation further: The resolution of the first virtual object is adjusted based on the proximity of the first virtual object to the fixed point, During the presentation of the first virtual object to the user, the first virtual object is blurred, The system described in item 1, including the system described in item 1. (Item 14) Blurring the first virtual object is, The method involves performing a Gaussian blur kernel convolution, wherein the first virtual object is rendered according to the adjusted resolution, The results of the convolution are presented to the user via the aforementioned display device, The system described in item 13, including the system described in item 13. (Item 15) The system according to item 1, wherein the one or more sensors include one or more of the following: an infrared sensor, an ultraviolet sensor, and a visible wavelength optical imaging device, configured to detect the direction of the user's line of sight. (Item 16) A display system, wherein the display system is A display device configured to present virtual content to a user, One or more processors, One or more computer storage media, the one or more computer storage media storing instructions, and when an instruction is executed by the system, the system Monitoring information associated with the user's eye movements, Based on the monitored information, determine a fixation point within the display frustum of the display device, wherein the fixation point indicates a three-dimensional location fixed to by the user's eye. Based on the determined fixation point, the virtual content is presented in a three-dimensional location within the display frustum, wherein the resolution of the virtual content is adjusted based on its proximity to the fixation point. One or more computer storage media that perform operations including A system that includes these features. (Item 17) The aforementioned operation further, The display device provides the determined fixation point to an external system comprising one or more processors, The display device receives rendered virtual content from the external system for presentation, wherein the virtual content is rendered with a resolution based on the individual proximity of the virtual content from the fixed point. The system described in item 16, including the system described in item 16. (Item 18) Adjusting the resolution of a particular virtual content based on its proximity to the fixed point is, Based on the proximity, the reduction at maximum resolution is determined, and the reduction is based on the rate of descent. The system according to item 16, further comprising adjusting the resolution of the specific virtual content based on the reduction determined above. (Item 19) The display device comprises a plurality of waveguides, each waveguide presenting virtual content associated with a separate depth based on the three-dimensional location of the virtual content, as per item 16. (Item 20) The system according to item 16, wherein the display device presents virtual content based on a fovealed image relative to the fixation point, and the fovealed image incorporates (1) depth information associated with the fixation point and (2) the virtual content. (Item 21) It is a method, A system with one or more processors, Monitoring the user's eye orientation of a display device based on information detected via one or more sensors, Based on the aforementioned eye orientation, the fixation point on which the user's eye is fixated is determined, wherein the fixation point represents a three-dimensional location within the user's field of view. To obtain location information associated with one or more virtual objects for presentation to the user, wherein the location information indicates the three-dimensional position of the virtual objects. At least partially, the resolution of at least one virtual object is adjusted based on the individual proximity from the at least one virtual object to the fixed point, The presenting of one or more virtual objects to the user via the display device, wherein at least one virtual object is rendered according to the individually adjusted resolution. Methods that include... (Item 22) The method according to item 21, wherein adjusting the resolution of a virtual object reduces the presentation quality of the virtual object relative to the maximum possible presentation quality. (Item 23) The method according to item 22, wherein reducing the presentation quality of the virtual object includes reducing the polygon count of the virtual object, reducing the primitives associated with the virtual object, or reducing the texture information associated with the virtual object. (Item 24) The fixation point is determined for each frame presented to the user via the display device, according to the method described in item 21. (Item 25) The field of view comprises zones, each zone representing a volume of space within the field of view in which a virtual object may be presented. The zone extends at least from the initial depth within the field of view to the end depth within the field of view, The method according to item 21, wherein the proximity of the virtual object from the fixed point is determined based on the first zone in which the virtual object will be presented relative to the second zone in which the fixed point is located. (Item 26) The method according to item 21, wherein the sensor comprises one or more of the following: an infrared sensor, an ultraviolet sensor, and a visible wavelength optical imaging device. (Item 27) It is a display system, A frame configured to be mounted on the user's head, A light modulation system configured to output light and form an image, One or more waveguides mounted on the frame and configured to receive light from the optical modulation system and output the light across the surface of one or more waveguides, One or more processors, One or more computer storage media, each of which stores instructions, and when an instruction is executed by the one or more processors, the one or more processors, To determine the amount of light reaching the retina of the user's eye, Based on the amount of light reaching the retina, the resolution of the virtual content to be presented to the user is adjusted. One or more computer storage media that perform operations including A display system equipped with these features. (Item 28) The display system according to item 27, further comprising an outward-pointing camera configured to measure ambient light levels, wherein determining the amount of light reaching the retina includes measuring the ambient light levels. (Item 29) The display system according to item 28, wherein determining the amount of light reaching the retina further includes determining the amount of light output to the user's eye by the waveguide. (Item 30) The aforementioned operation further, Determining the user's fixed viewpoint, The method involves obtaining location information for the virtual content, wherein the location information indicates the three-dimensional position of the virtual content. The display system according to item 27, which includes adjusting the resolution of the virtual content, which includes varying the resolution of the virtual content based on the proximity of the virtual content to the fixation point. (Item 31) The display system according to item 30, wherein adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on the proximity of the virtual content to the fixation point on a depth axis that extends away from the user. (Item 32) The display system according to item 27, wherein the one or more waveguides comprise a stack of waveguides, and at least some of the waveguides in the stack of waveguides provide a different wavefront divergence than the other waveguides in the stack of waveguides. (Item 33) The one or more waveguides are Internally coupled diffractive optical elements, External coupled diffractive optical element, A light dispersion element configured to direct light from the internally coupled diffracting optical element to the externally coupled diffracting optical element A display system as described in item 27, comprising: (Item 34) It is a display system, One or more processors, One or more computer storage media, each of which stores instructions, and when an instruction is executed by the one or more processors, the one or more processors, To determine the amount of light reaching the retina of the user's eye in the aforementioned display system, Based on the amount of light reaching the retina, the resolution of the virtual content to be presented to the user is adjusted. One or more computer storage media that perform operations including A display system equipped with these features. (Item 35) The display system according to item 34, wherein determining the amount of light reaching the retina includes determining the ambient light level. (Item 36) The display system according to item 35, wherein determining the amount of light reaching the retina includes determining the amount of light output by the display system to display the virtual content. (Item 37) The display system according to item 34, wherein determining the amount of light reaching the retina includes determining the size of the pupil of the user's eye. (Item 38) The display system according to item 34, wherein adjusting the resolution of the virtual content includes decreasing the resolution of the virtual content as the distance of the virtual content from the user's fixed viewpoint increases. (Item 39) The display system according to item 38, wherein the tendency to reduce the resolution is substantially analogous to the tendency to reduce the cone density in the user's retina. (Item 40) The zone of maximum resolution within the user's field of view is aligned with a point corresponding to the fovea of ​​the user's eye. The resolution of the virtual content within the user's field of view decreases with respect to virtual content located + / - 20° outside the point. The display system described in item 39. (Item 41) The display system according to item 38, wherein reducing the resolution includes reducing the polygon count at all levels of light reaching the retina. (Item 42) The display system according to item 38, wherein adjusting the resolution includes setting the resolution to one of two resolution levels. (Item 43) The display system according to item 42, wherein adjusting the resolution of the virtual content includes reducing one or both of the color depth and / or contrast ratio as the amount of light reaching the retina decreases. (Item 44) The aforementioned operation further, This includes determining whether the amount of light corresponds to photopic, crepuscular, or scotopic illumination levels. The display system according to item 34, wherein adjusting the resolution of the virtual content includes setting the resolution based on whether the illumination level corresponds to a photopic, crepuscular, or scotopic illumination level. (Item 45) The display system according to item 34, wherein adjusting the resolution of the virtual content includes displaying the virtual content using a single color at a scotopic illumination level. (Item 46) The display system according to item 34, wherein adjusting the resolution of the virtual content includes reducing the contrast ratio of the image forming the virtual content as the amount of light reaching the retina of the eye decreases. (Item 47) The aforementioned operation further, Providing virtual content in multiple primary colors Includes, The display system according to item 34, wherein adjusting the resolution of the virtual content provides different resolutions for different primary colors. (Item 48) The display system according to item 34, further comprising the operation of preventing the rendering of virtual content corresponding to the blind spot of the optic nerve of the eye. (Item 49) Adjusting the resolution of the virtual content includes preventing the rendering of virtual content located within the opposite peripheral region of each of the user's eyes, wherein the opposite peripheral region is on the side of the user opposite to the side of the user where the eye is positioned, as in the display system of item 34. (Item 50) The display system according to item 34 further comprises bringing the virtual content to the user via a display device, wherein the virtual content is rendered according to an associated adjusted resolution. (Item 51) A method carried out by a display system comprising one or more processors and a head-mountable display, wherein the method is To determine the amount of light reaching the retina of the user's eye in the aforementioned display system, Based on the amount of light reaching the retina, the resolution of the virtual content to be presented to the user is adjusted. Methods that include... (Item 52) It is a display system, A frame configured to be mounted on the user's head, A light modulation system configured to output light and form an image, One or more waveguides mounted on the frame and configured to receive light from the optical modulation system and output the light across the surface of one or more waveguides, One or more processors, One or more computer storage media, each of which stores instructions, and when an instruction is executed by the one or more processors, the one or more processors, The proximity of the virtual content from the user's fixed viewpoint, The colors of the aforementioned primary color images, wherein at least one of the primary color images has a different resolution from the primary color image of another color. Based on this, one or more computer storage media perform an operation that includes adjusting the resolution of the primary color images forming the virtual content. A display system equipped with these features. (Item 53) The display system according to item 52, wherein adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on the proximity of the virtual content to the fixation point on a depth axis that extends away from the user. (Item 54) The display system according to item 52, wherein the one or more waveguides comprise a stack of waveguides, and at least some of the waveguides in the stack of waveguides provide a different wavefront divergence than the other waveguides in the stack of waveguides. (Item 55) The one or more waveguides are Internally coupled diffractive optical elements, External coupled diffractive optical element, A light-dispersing element configured to direct light from the internally coupled diffracting optical element to the externally coupled diffracting optical element, A display system as described in item 52, comprising: (Item 56) It is a display system, One or more processors, One or more computer storage media, each of which stores instructions, and when an instruction is executed by the one or more processors, the one or more processors, The proximity of the virtual content from the user's fixed viewpoint, The colors of the aforementioned primary color images, wherein at least one of the primary color images has a different resolution from the primary color image of another color. Based on this, one or more computer storage media perform an operation that includes adjusting the resolution of the primary color images forming the virtual content. A display system equipped with these features. (Item 57) The display system according to item 56, wherein the resolution corresponds to at least the color depth. (Item 58) The resolution is at least corresponding to the polygon count, as described in item 56. (Item 59) The aforementioned primary color image comprises a red primary color image, a green primary color image, and a blue primary color image. The aforementioned green primary color image has a higher resolution than the aforementioned red or blue primary color image. The display system described in item 56. (Item 60) The display system according to item 59, wherein the red primary color image has a higher resolution than the blue primary color image. (Item 61) The display system according to item 56, wherein adjusting the resolution of the virtual content includes reducing the contrast ratio of the image forming the virtual content in response to a decrease in either or both of the ambient light and the light output by the display system to display the virtual content. (Item 62) The display system according to item 56, further comprising the operation of preventing the rendering of virtual content corresponding to the blind spot of the optic nerve in the user's eye. (Item 63) Adjusting the resolution of the virtual content includes preventing the rendering of virtual content located within the opposite peripheral region for each of the user's eyes, wherein the opposite peripheral region is on the side of the user opposite to the side of the user where the eye is positioned, as in the display system of item 56. (Item 64) The display system according to item 56 further comprises bringing the virtual content to the user via a display device, wherein the virtual content is rendered according to an associated adjusted resolution. (Item 65) A method carried out by a display system comprising one or more processors and a head-mountable display, wherein the method is The proximity of the virtual content from the user's fixed viewpoint, The colors of the aforementioned primary color images, wherein at least one of the primary color images has a different resolution from the primary color image of another color. A method comprising adjusting the resolution of a primary color image that forms virtual content, based on the above. (Item 66) It is a display system, An image source comprising a spatial light modulator for providing a first image stream and a second image stream, A viewing assembly comprising an optically guided optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to the user, One or more processors that communicate with the aforementioned image source, One or more computer storage media, each of which stores instructions, and when an instruction is executed by the one or more processors, the one or more processors, The image source is to output the first image stream to the viewing assembly, wherein the image formed by the first image stream has a first pixel density. The image source is to output the second image stream to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Perform an action that includes the following: The image formed by the second image stream corresponds to the portion of the image provided by the first image stream. The image formed by the second image stream covers a corresponding portion of the field of view provided by the first image stream, and is stored in one or more computer storage media. A display system equipped with these features. (Item 67) The display system according to item 66, further comprising the operation of modifying the location of the second image relative to the first image. (Item 68) The display system according to item 67, further comprising the operation of modifying the size of the second image over time. (Item 69) The system further comprises an eye-line-of-sight tracker configured to detect changes in the user's eye orientation, The operation includes modifying the location of the second image relative to the first image in accordance with the detected change in the user's eye orientation. The display system described in item 67. (Item 70) Scanning mirror in the path of the second image stream between the image source and the viewing assembly Furthermore, The display system according to item 69, further comprising adjusting the orientation of the scanning mirror and altering the image location of the second image relative to the first image. (Item 71) The viewing assembly comprises an eyepiece, and the eyepiece is The display system according to item 66, comprising a waveguide having an internal coupling grating configured to internally couple light to the waveguide. (Item 72) The display system according to item 71, wherein the internal bonding grid is an internal bonding grid sensitive to the polarization of liquid crystal. (Item 73) The display system according to item 71, wherein the waveguide comprises another internal coupling grating configured to internally couple light to the waveguide, and the other internal coupling grating is arranged along a surface different from the internal coupling grating of the waveguide. (Item 74) Both the aforementioned internally coupled grating and the other internally coupled grating are located within the optical path of the second image stream. The internally coupled grid and the other internally coupled grid are transparent to the second image stream having the first polarization. The system further comprises a mirror spaced apart from and facing the other internally coupled grating, the mirror being configured to reflect the second image stream toward the other internally coupled grating and to change the polarization of the light in the second image stream to a second polarization, and the other internally coupled grating being configured to internally couple the light of the second polarization. The display system described in item 73. (Item 75) The display system according to item 71, wherein the waveguide comprises another internal coupling grating configured to internally couple light to the waveguide, the other internal coupling grating arranged along the same surface as the internal coupling grating and the waveguide, the internal coupling grating positioned in the path of the second image stream, and the other internal coupling grating positioned in the path of the first image stream. (Item 76) The display system described in item 66 further comprises a beam splitting optical system for splitting light into multiple image streams propagating in different directions. (Item 77) The beam splitting optical system is a polarizing beam splitter, as described in item 76 of the display system. The display system according to item 76, wherein the beam splitting optical system is a switchable reflector, the switchable reflector is selectively switchable between a reflective state and a transmittance state. (Item 78) A display system for projecting an image onto a user's eyes, wherein the display system is An image source is configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, wherein the first image stream has a first field of view, and the second image stream has a second field of view narrower than the first field of view, and the first light beam and the second light beam are multiplexed together. A polarizing beam splitter, wherein the polarizing beam splitter is To position the first image stream in a fixed position relative to the user's eye, the first light beam is received and reflected along a first optical path toward the viewing assembly, The second light beam is received and transmitted along the second optical path. A polarizing beam splitter configured to perform the following, A scanning mirror is positioned along the second optical path and configured to receive the second light beam and reflect it toward the viewing assembly, An eye-gaze tracker configured to detect the user's eye movements, A control circuit that communicates with the eye-line tracker and the scanning mirror, wherein the control circuit is configured to position the scanning mirror such that the position of the second image stream moves according to the detected movement of the user's eye. A display system equipped with these features. (Item 79) The display system according to item 78, wherein the first image stream has a first angular resolution, and the second image stream has a second angular resolution higher than the first angular resolution. (Item 80) The display system according to item 78, wherein the content of the second image stream comprises a portion of the content of the first image stream. (Item 81) The display system according to item 80, wherein the content of the second image stream changes as the second image stream moves relative to the first image stream such that the content of the second image stream corresponds to the portion of the first image stream overlaid by the second image stream. (Item 82) The display system according to item 78, wherein the first polarization and the second polarization are each linearly polarized, and the second polarization is orthogonal to the first polarization. (Item 83) The display system according to item 78, further comprising a first relay lens assembly configured to provide a first angular magnification to the first light beam. (Item 84) A second relay lens assembly configured to provide a second angular magnification to the second light beam, wherein the second angular magnification is different from the first angular magnification. The display system described in item 83, further comprising: (Item 85) The display system according to item 84, wherein the first angle magnification is greater than 1 and the second angle magnification is 1 or less. (Item 86) The display system according to item 84, wherein the first relay lens assembly comprises a first optical lens positioned between the image source and the polarizing beam splitter and a second optical lens positioned along the first optical path, and the ratio of the focal length of the first optical lens to the focal length of the second optical lens is greater than 1. (Item 87) The display system according to item 86, wherein the second relay lens assembly comprises the first optical lens and a third optical lens positioned along the second optical path, and the ratio of the focal length of the first optical lens to the focal length of the third optical lens is 1 or less. (Item 88) The first and second light beams are time-division multiplexed, The display system according to item 78, wherein the image source comprises a switching polarization rotor, the switching polarization rotor configured to provide a first light beam in the first polarization and a second light beam in the second polarization. (Item 89) The display system according to item 78, wherein the first light beam and the second light beam are polarization-divided multiplexed. (Item 90) A display system for projecting an image onto a user's eyes, wherein the display system is An image source configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream, wherein the first image stream has a first field of view, the second image stream has a second field of view narrower than the first field of view, and the first and second light beams are multiplexed together. A scanning mirror configured to receive the first light beam and the second light beam and reflect them toward a viewing assembly in order to project the first image stream and the second image stream, An eye-gaze tracker configured to detect the user's eye movements, A control circuit that communicates with the eye-line tracker and the scanning mirror, the control circuit is configured to position the scanning mirror such that the position of the first image stream and the position of the second image stream move according to the detected movement of the user's eye, A switchable optical element positioned within the optical paths of the first and second light beams, wherein the switchable optical element is configured to be switched to a first state for the first light beam such that the first light beam is angularly expanded by a first angular expansion factor, and to be switched to a second state for the second light beam such that the second light beam is angularly amplified by a second angular expansion factor less than the first angular expansion factor. A display system equipped with these features. (Item 91) The display system according to item 90, wherein the first light beam and the second light beam are time-division multiplexed, and the switchable optical element comprises an electrically switchable liquid crystal lens. (Item 92) The display system according to item 90, wherein the first and second light beams are polarization-split multiplexed, and the switchable optical element comprises a multifocal birefringent lens. (Item 93) The multifocal birefringent lens comprises a birefringent polymer, as described in item 92 of the display system. (Item 94) The display system according to item 90, wherein the switchable optical element is located downstream of the scanning mirror. (Item 95) The display system according to item 90, wherein the switchable optical element is arranged on the surface of the scanning mirror. (Item 96) The display system according to item 90, wherein the switchable optical element is positioned between the image source and the scanning mirror. (Item 97) The display system according to item 90, wherein the first angle magnification is greater than 1 and the second angle magnification is approximately 1. (Item 98) A wearable display system, An infinite focus magnifying lens with circular polarization-dependent magnification, wherein the infinite focus magnifying lens is A first fixed focal length lens element, A first geometric phase lens exhibiting a positive refractive power for the first palpability of incident circularly polarized light and a negative refractive power for the second palpability of incident circularly polarized light, The second geometric phase lens and Equipped with an infinite focus magnifying lens A wearable display system equipped with [features / equipment]. (Item 99) When transmitting light, the first geometric phase lens reverses the circular polarization properties. The second geometric phase lens exhibits a negative refractive power for the first palpability of the circularly polarized light and a positive refractive power for the second palpability of the circularly polarized light. Wearable display systems as described in item 98. (Item 100) The wearable display system according to item 98, further comprising a second fixed focal length lens, wherein the second geometric phase lens exhibits a positive refractive power for a first palmarity of the circularly polarized light and a negative refractive power for a second palmarity of the circularly polarized light. (Item 101) The wearable display system according to item 98, wherein the first geometric phase lens reverses the circular polarization when transmitting light. (Item 102) The wearable display system according to item 98, wherein, with respect to light having a first fixed focal length lens element and the first geometric phase lens, the lens group has the positive focal length of the first group, with respect to light having a first palmarity, the lens group has the positive focal length of the second group, and the positive focal length of the first group exceeds the positive focal length of the second group. (Item 103) A polarization rotor optically coupled to the infinite focus magnifying lens, wherein the polarization rotor is configured to receive polarized light and selectively output light of a first linear polarization or a second linear polarization, A first waveplate between the polarizing rotor and the infinite focus magnifying lens, wherein the first waveplate is configured to convert the first linearly polarized light into the first circularly polarized light and the second linearly polarized light into the second circularly polarized light, and A wearable display system as described in item 98, further comprising the features described above. (Item 104) A first optical path between the polarizer rotor and the first waveplate for the first linearly polarized light, The second optical path between the polarizer rotor and the first waveplate for the second linearly polarized light and Wearable display systems as described in item 103, including the following. (Item 105) The wearable display system according to item 104, wherein at least one of the first optical path and the second optical path includes a steerable mirror. (Item 106) The wearable display system according to item 104, wherein the first optical path and the second optical path each include at least a subset of multiple relay lens elements. (Item 107) The wearable display system according to item 103, wherein in each of the first and second optical paths, a series of pairs of the plurality of relay lens elements form an infinite focus composite lens. (Item 108) The wearable display system according to item 104, wherein the first optical path intersects a polarization-selective reflector, and the second optical path intersects the polarization-selective reflector. (Item 109) The wearable display system according to item 108, wherein the first optical path and the second optical path each include at least a subset of four intermediate lens elements, the four intermediate lens elements being positioned around the polarization-selective reflector. (Item 110) The wearable display system according to item 108, wherein the first optical path intersects the polarization-selective reflector in a first direction, and the second optical path intersects the polarization-selective reflector in a second direction. (Item 111) The wearable display system according to item 108, wherein at least one of the first optical path and the second optical path includes a second waveplate and a third waveplate. (Item 112) An optical subsystem for a wearable image projector, Polarization-selective reflectors and, A set of four lens elements positioned around the aforementioned polarization-selective reflector and An optical subsystem comprising: (Item 113) The optical subsystem according to item 112, wherein each continuous pair of the set of four lens elements forms an infinite focus composite lens along at least one optical path through the subsystem. (Item 114) The optical subsystem according to item 112, further comprising a polarization-selective reflector and a polarization-rotating switch optically coupled to the set of four lens elements. (Item 115) The optical subsystem according to item 114, further comprising a plurality of waveplates positioned around the polarization-selective reflector. (Item 116) The optical subsystem according to item 115, wherein each of the plurality of waveplates is positioned between one of the set of four relay lens elements and the polarization-selective reflector. (Item 117) The plurality of waveplates include three waveplates, as described in item 115. (Item 118) The optical subsystem according to item 115, further comprising a foveal tracking mirror positioned on the opposite side of the first of the set of four relay lenses from the polarization-selective reflector. (Item 119) The optical subsystem according to item 118, further comprising an image scanning mirror positioned on the opposite side of the set of four relay lenses from the polarization-selective reflector. (Item 120) The optical subsystem according to item 119, further comprising a light beam source optically coupled to the image scanning mirror. (Item 121) The optical subsystem according to item 120, wherein the light beam source comprises a dichroic reflector, a first laser diode that emits light of a first color reflected by the dichroic reflector, and a second laser diode that emits light of a second color transmitted by the dichroic reflector. (Item 122) A display system for projecting an image onto a user's eyes, wherein the display system is An eyepiece, wherein the eyepiece is Waveguide and An internal coupling grating optically coupled to the waveguide and Includes, eyepiece, A first image source configured to project a first light beam associated with a first image stream, wherein the first image stream has a first field of view incident on a first surface of the internal coupling grating, and a portion of the first light beam is coupled into the waveguide by the internal coupling grating to position the first image stream in a fixed position relative to the user's eye. A second image source configured to project a second light beam associated with a second image stream, wherein the second image stream has a second field of view that is narrower than the first field of view. A scanning mirror is configured to receive and reflect a second light beam such that the second light beam is incident on a second surface of the internal coupling grating opposite to its first surface, and a portion of the second light beam is coupled into the waveguide by the internal coupling grating. An eye-gaze tracker configured to detect the user's eye movements, A control circuit that communicates with the eye-line tracker and the scanning mirror, wherein the control circuit is configured to position the scanning mirror such that the position of the second image stream moves according to the detected movement of the user's eye. A display system equipped with these features. (Item 123) The display system according to item 122, further comprising a first lens assembly, the first lens assembly positioned along a first optical path of the first light beam and configured to provide a first angular magnification to the first light beam. (Item 124) The display system according to item 123, further comprising a second lens assembly, the second lens assembly positioned along a second optical path of the second light beam and configured to provide the second angular magnification to the second light beam, wherein the second angular magnification is different from the first angular magnification. (Item 125) The display system according to item 124, wherein the first angle magnification is greater than 1 and the second angle magnification is 1 or less. (Item 126) The display system according to item 122, wherein the first image stream has a first angular resolution, and the second image stream has a second angular resolution higher than the first angular resolution. (Item 127) A display system for projecting an image onto a user's eyes, wherein the display system is An eyepiece, wherein the eyepiece is Waveguide and An internal coupling grating optically coupled to the waveguide and Includes, eyepiece, An image source is configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, wherein the first image stream has a first field of view, and the second image stream has a second field of view narrower than the first field of view, and the first light beam and the second light beam are multiplexed together. A polarizing beam splitter, wherein the polarizing beam splitter is The first light beam is received and reflected along the first optical path, The second light beam is received and transmitted along the second optical path. A polarizing beam splitter configured to perform the following, A first optical reflector, the first optical reflector is positioned along the first optical path and configured to receive and reflect the first light beam such that the first light beam is incident on the first surface of the internal coupling grating, and a portion of the first light beam is coupled into the waveguide by the internal coupling grating to position the first image stream in a fixed position relative to the user's eye. A scanning mirror is positioned along the second optical path and configured to receive and reflect the second light beam, A second optical reflector, the second optical reflector is positioned along the second optical path downstream of the scanning mirror, and the second optical reflector is configured to receive and reflect the second light beam such that the second light beam is incident on the second surface of the internal coupling grating opposite to its first surface, and a portion of the second light beam is coupled into the waveguide by the internal coupling grating, An eye-gaze tracker configured to detect the user's eye movements, A control circuit that communicates with the eye-line tracker and the scanning mirror, wherein the control circuit is configured to position the scanning mirror such that the position of the second image stream moves according to the detected movement of the user's eye. A display system equipped with these features. (Item 128) A display system for projecting an image onto a user's eyes, wherein the display system is Waveguide and An image source is configured to project a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization, wherein the first image stream has a first field of view, and the second image stream has a second field of view narrower than the first field of view, and the first light beam and the second light beam are multiplexed together. A polarizing beam splitter, wherein the polarizing beam splitter is The first light beam is received and reflected along the first optical path, The second light beam is received and transmitted along the second optical path. A polarizing beam splitter configured to perform the following, A first internal coupling prism positioned adjacent to the first surface of the waveguide along the first optical path, wherein the first internal coupling prism is configured to couple a portion of the first light beam into the waveguide in order to position the first image stream in a fixed position relative to the user's eye, A scanning mirror is positioned along the second optical path and configured to receive and reflect the second light beam, A second internal coupling prism is positioned along the second optical path downstream of the scanning mirror, adjacent to the second surface of the waveguide opposite to the first surface of the waveguide, wherein the second internal coupling prism is configured to couple a portion of the second light beam into the waveguide. An eye-gaze tracker configured to detect the user's eye movements, A control circuit that communicates with the eye-line tracker and the scanning mirror, wherein the control circuit is configured to position the scanning mirror such that the position of the second image stream moves according to the detected movement of the user's eye. A display system equipped with these features. [Brief explanation of the drawing]

[0024] [Figure 1] Figure 1 illustrates the user's view of augmented reality (AR) through an AR device.

[0025] [Figure 2] Figure 2 illustrates a conventional display system for simulating a three-dimensional image for the user.

[0026] [Figure 3] Figures 3A-3C illustrate the relationship between the radius of curvature and the radius of focus.

[0027] [Figure 4A] Figure 4A illustrates the representation of the accommodation-vergence response of the human visual system.

[0028] [Figure 4B] Figure 4B illustrates an example of different near and far accommodative states and convergence / divergence motion states of a pair of user eyes.

[0029] [Figure 4C]Figure 4C illustrates an example of how the upper and lower figures represent a user viewing content through a display system.

[0030] [Figure 4D] Figure 4D illustrates another embodiment of the representation of a user viewing content through a display system.

[0031] [Figure 5] Figure 5 illustrates aspects of an approach to simulating a 3D image by correcting wavefront divergence.

[0032] [Figure 6] Figure 6 illustrates an example of a waveguide stack for outputting image information to the user.

[0033] [Figure 7] Figure 7 illustrates an example of an output beam produced by a waveguide.

[0034] [Figure 8] Figure 8 illustrates an embodiment of a stacked waveguide assembly, where each depth plane includes an image formed using multiple different primary colors.

[0035] [Figure 9A] Figure 9A shows cross-sectional side views of embodiments of stacked waveguide sets, each including an internally coupled optical element.

[0036] [Figure 9B] Figure 9B shows a perspective view of an embodiment of the multiple stacked waveguides shown in Figure 9A.

[0037] [Figure 9C] Figure 9C shows top and bottom plan views of the embodiment of the multiple stacked waveguides shown in Figures 9A and 9B.

[0038] [Figure 9D]Figure 9D illustrates an embodiment of a wearable display system.

[0039] [Figure 10A] Figure 10A illustrates an example of how the upper and lower figures represent a user viewing content through a display system.

[0040] [Figure 10B] Figure 10B illustrates another embodiment of the representation of a user viewing content through a display system.

[0041] [Figure 10C] Figure 10C illustrates yet another embodiment of the representation of a user viewing content through a display system.

[0042] [Figure 10D] Figure 10D is a block diagram of an exemplary display system.

[0043] [Figure 11A1] Figure 11A1 illustrates an example of representing the upper and lower diagrams of the adjustment at different resolution adjustment zones based on three-dimensional fixed-point tracking.

[0044] [Figure 11A2] Figure 11A2 illustrates examples of the representation of the resolution adjustment zones in the upper and lower diagrams at different time points as the size and number of zones change.

[0045] [Figure 11B] Figure 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone in Figure 11A1.

[0046] [Figure 11C] Figure 11C illustrates another embodiment of the configuration for the resolution adjustment zone.

[0047] [Figure 11D]Figure 11D illustrates an example of a three-dimensional representation of the resolution adjustment zone in Figure 11C.

[0048] [Figure 11E] Figure 11E illustrates another embodiment of the three-dimensional representation of the resolution adjustment zone in Figure 11C.

[0049] [Figure 12A] Figures 12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixed point. [Figure 12B] Figures 12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixed point. [Figure 12C] Figures 12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to its proximity to a three-dimensional fixed point.

[0050] [Figure 13] Figure 13 illustrates an example of a user's representation in which multiple virtual objects are viewed in line with the user's line of sight.

[0051] [Figure 14] Figure 14 is a schematic diagram of an example of a process for adjusting virtual content based on the angle of proximity to the user's line of sight.

[0052] [Figure 15] Figure 15 illustrates an example of how the user's retina is represented.

[0053] [Figure 16] Figure 16 schematically illustrates an example of resolution and rod and cone densities across the retina shown in Figure 15.

[0054] [Figure 17] Figure 17 schematically illustrates an example of the relationship between pupil size and the amount of light incident on the user's eye.

[0055] [Figure 18] Figure 18 is a schematic diagram of an example of a process for adjusting virtual content based on the amount of light incident on the user's eyes.

[0056] [Figure 19] Figure 19 schematically illustrates an example of how the resolution detectable by the user's eye changes as the amount of light incident on the eye changes.

[0057] [Figure 20] Figure 20 schematically illustrates an example of the difference in the eye's sensitivity to different colors of light at different levels of illumination.

[0058] [Figure 21] Figure 21 is a schematic diagram of an embodiment of a process for adjusting virtual content formed using multiple primary color images, where the resolution adjustment is performed based on the colors of the primary color images.

[0059] [Figure 22] Figures 22A-22C illustrate an example of contrast sensitivity that changes as the amount of light incident on the user's eye decreases.

[0060] [Figure 23] Figure 23 illustrates an example of how the user's optic nerve and peripheral blind spot are represented.

[0061] [Figure 24] Figure 24 shows an illustrative monocular field of view of the human eye.

[0062] [Figure 25A] Figure 25A shows an exemplary wearable display device configured to provide virtual content to a user.

[0063] [Figure 25B] Figure 25B is a block diagram illustrating an augmented reality system.

[0064] [Figure 25C] FIG. 25C schematically illustrates the optical path within a visual optics assembly (VOA) that can be used to present a digital or virtual image to a viewer.

[0065] [Figure 26A] FIGS. 26A - 26D illustrate the light fields that will be produced within an AR system for an exemplary rendering viewpoint and two exemplary eye orientations that will be used. [Figure 26B] FIGS. 26A - 26D illustrate the light fields that will be produced within an AR system for an exemplary rendering viewpoint and two exemplary eye orientations that will be used. [Figure 26C] FIGS. 26A - 26D illustrate the light fields that will be produced within an AR system for an exemplary rendering viewpoint and two exemplary eye orientations that will be used. [Figure 26D] FIGS. 26A - 26D illustrate the light fields that will be produced within an AR system for an exemplary rendering viewpoint and two exemplary eye orientations that will be used.

[0066] [Figure 26E] FIGS. 26E - 26F schematically illustrate an exemplary configuration of an image that can be presented to a user. [Figure 26F] FIGS. 26E - 26F schematically illustrate an exemplary configuration of an image that can be presented to a user.

[0067] [Figure 26G] FIGS. 26G - 26H schematically illustrate an exemplary configuration of an image that can be presented to a user. [Figure 26H] FIGS. 26G - 26H schematically illustrate an exemplary configuration of an image that can be presented to a user.

[0068] [Figure 27]Figure 27 illustrates the field of view and oculomotor field of view shown in Figure 24, which are overlaid on one of the displays in a wearable display device as shown in Figure 25.

[0069] [Figure 28A] Figures 28A-28B illustrate some of the principles described in Figures 26A-26D. [Figure 28B] Figures 28A-28B illustrate some of the principles described in Figures 26A-26D.

[0070] [Figure 28C] Figures 28C-28D illustrate several exemplary images that may be presented to the user. [Figure 28D] Figures 28C-28D illustrate several exemplary images that may be presented to the user.

[0071] [Figure 28E] Figure 28E illustrates an exemplary high-FOV, low-resolution image frame.

[0072] [Figure 28F] Figure 28F illustrates an exemplary low-FOV high-resolution image frame.

[0073] [Figure 29A] Figure 29A shows a simplified block diagram of the display system.

[0074] [Figure 29B] Figure 29B schematically illustrates a cross-sectional view of an augmented reality (AR) system.

[0075] [Figure 30A] Figures 30A-30B schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 30B] Figures 30A-30B schematically illustrate a display system for projecting an image stream onto the user's eyes.

[0076] [Figure 30C] Figure 30C schematically illustrates a cross-sectional view of an augmented reality (AR) system.

[0077] [Figure 30D] Figure 30D shows a simplified block diagram of a display system.

[0078] [Figure 31A] Figure 31A schematically illustrates the operating principle of a first relay lens assembly within the display system illustrated in FIGS. 30A-30B.

[0079] [Figure 31B] Figure 31B schematically illustrates the operating principle of a second relay lens assembly within the display system illustrated in FIGS. 30A-30B.

[0080] [Figure 31C] Figures 31C-31D schematically illustrate a display system. [Figure 31D] Figures 31C-31D schematically illustrate a display system.

[0081] [Figure 32A] Figures 32A-32C schematically illustrate a display system. [Figure 32B] Figures 32A-32C schematically illustrate a display system. [Figure 32C] Figures 32A-32C schematically illustrate a display system.

[0082] [Figure 33A] Figures 33A-33B schematically illustrate a display system. [Figure 33B] Figures 33A-33B schematically illustrate a display system.

[0083] [Figure 34A] Figures 34A and 34B schematically illustrate the display system. [Figure 34B] Figures 34A and 34B schematically illustrate the display system.

[0084] [Figure 35] Figure 35 schematically illustrates the display system.

[0085] [Figure 36A] Figure 36A schematically illustrates the augmented reality eyepiece display system.

[0086] [Figure 36B] Figure 36B schematically illustrates another augmented reality eyepiece display system.

[0087] [Figure 37A] Figure 37A is a schematic diagram of a 2x infinity magnification lens.

[0088] [Figure 37B] Figure 37B is a schematic diagram of a bifocal magnifying and infinite focal magnifying lens.

[0089] [Figure 38] Figures 38A-38B schematically illustrate exemplary configurations of images that may be presented to the user.

[0090] [Figure 39A] Figures 39A-39B illustrate several exemplary images that may be presented to the user. [Figure 39B] Figures 39A-39B illustrate several exemplary images that may be presented to the user.

[0091] [Figure 40A] Figures 40A-40D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 40B]Figures 40A-40D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 40C] Figures 40A-40D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 40D] Figures 40A-40D schematically illustrate a display system for projecting an image stream onto the user's eyes.

[0092] [Figure 41A] Figures 41A-41D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 41B] Figures 41A-41D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 41C] Figures 41A-41D schematically illustrate a display system for projecting an image stream onto the user's eyes. [Figure 41D] Figures 41A-41D schematically illustrate a display system for projecting an image stream onto the user's eyes.

[0093] [Figure 42] Figure 42 illustrates exemplary frame structures for time-division multiplexed high-FOV low-resolution image streams and low-FOV high-resolution image streams.

[0094] [Figure 43] Figure 43 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0095] [Figure 44] Figure 44 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0096] [Figure 45]Figure 45 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0097] [Figure 46] Figure 46 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0098] [Figure 47] Figure 47 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0099] [Figure 48] Figure 48 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0100] [Figure 49] Figure 49 schematically illustrates a display system for projecting an image stream onto the user's eyes.

[0101] [Figure 50] Figure 50 schematically illustrates a display system for projecting an image stream onto a user's eyes, according to several embodiments.

[0102] [Figure 51] Figure 51 schematically illustrates a display system for projecting an image stream onto a user's eyes, according to several embodiments.

[0103] [Figure 52A] Figures 52A-52B schematically illustrate display systems for projecting image streams onto a user's eyes, according to several embodiments. [Figure 52B] Figures 52A-52B schematically illustrate display systems for projecting image streams onto a user's eyes, according to several embodiments.

[0104] [Figure 53A] Figures 53A-53B schematically illustrate display systems for projecting image streams onto a user's eyes, according to several embodiments. [Figure 53B] Figures 53A-53B schematically illustrate display systems for projecting image streams onto a user's eyes, according to several embodiments. [Modes for carrying out the invention]

[0105] Rendering virtual content for augmented and virtual display systems is computationally intensive. In particular, the computational intensity may require the use of powerful processing units, which can undesirably consume large amounts of memory, result in long latency, and / or be costly and / or energy-intensive.

[0106] In some embodiments, methods and systems save computational resources such as memory and processing time by reducing the resolution of virtual content located away from the user's eye's fixation point. For example, the system may render virtual content at or near the user's eye's fixation point at a relatively high (e.g., highest) resolution, while utilizing one or more lower resolutions for virtual content far from the fixation point. The virtual content is presented by a display system that can display the virtual content on multiple different depths (e.g., multiple different depth planes such as two or more depth planes), and the reduction in resolution preferably occurs along at least the z-axis, where the z-axis is the depth axis (corresponding to the distance from the user). In some embodiments, the resolution reduction occurs along the z-axis and one or both of the x and y axes, where the x-axis is the lateral axis and the y-axis is the vertical axis.

[0107] Determining the appropriate resolution for virtual content may involve the step of determining the fixation point of the user's eye in three-dimensional space. For example, the fixation point may be the x, y, z coordinates within the user's field of view on which the user's eye fixates. The display system may be configured to present virtual objects with differences in resolution, where the resolution decreases as the proximity of the virtual object to the fixation point decreases, or in other words, as the distance from the fixation point increases.

[0108] As discussed herein, a display system may present virtual objects within the display frustum of the display system, and the virtual objects can be presented on different depth planes. In some embodiments, the display frustum is a field of view provided by the display system, over which the display system is configured to present virtual content to the user of the display system. The display system may be a head-mounted display system including one or more waveguides capable of presenting virtual content (e.g., virtual objects, graphics, text, etc.), and one or more waveguides are configured to emit light with different wavefront divergences and / or different binocular differences corresponding to different depth planes (e.g., corresponding to a specific distance from the user). It should be understood that each eye may have one or more associated waveguides. Using different wavefront divergences and / or different binocular differences, the display system may make a first virtual object appear to be located at a first depth in the user's field of view, while a second virtual object appears to be located at a second depth in the user's field of view. In some embodiments, the depth plane of the fixed point or its adjacent depth plane may be determined, and the resolution of content on other depth planes may be reduced based on the distance from those depth planes to the depth plane in which the fixed point is located. It should be understood that the reference to virtual content depth (distance of the virtual content from the user on the z-axis) herein refers to the apparent depth of the virtual content as intended to be seen by the user. In some embodiments, the depth of a virtual object may be understood as the distance from the user to the real object, having wavefront divergence and / or binocular difference similar to that of the virtual object.

[0109] The proximity between a virtual object and a fixed point can be determined by various measurements, and it should be understood that non-limiting embodiments include determining the distance between the fixed point and the virtual object, determining the resolution adjustment zone occupied by the virtual object relative to the resolution adjustment zone occupied by the fixed point (in embodiments where the user's field of view is subdivided into resolution adjustment zones as described below), and determining the angular proximity between the virtual object and the user's fixed point. Proximity may also be determined using a combination of the above techniques. For example, the distance and / or angular proximity between a first zone (where the virtual object is located) and a second zone (where the fixed point is located) may be used to determine proximity. These various measurements will be discussed further below.

[0110] In some embodiments, the step of determining the fixation point may include anticipating the fixation point of the user's eye and using the anticipated fixation point as the fixation point for determining the resolution of the virtual content. For example, a display system may render certain content at a relatively high resolution in anticipation of the user's eye fixing on that content. As an example, it should be understood that the human visual system can be sensitive to sudden changes in the scene (e.g., sudden motion, changes in brightness, etc.). In some embodiments, a display system may determine that the virtual content is of a type that will cause the user's eye to fixate on it (e.g., involving motion in a scene where other virtual and real objects are stationary), and then render the virtual content at a high resolution in anticipation of the user's eye subsequently focusing on that virtual content.

[0111] As described above, in some embodiments, the distance from the determined fixed point to the virtual object may correspond to a distance extending in three dimensions. For example, a first virtual object located at the same depth (e.g., the same depth plane) from the determined fixed point and the user, but positioned horizontally or vertically from the fixed point, may similarly have reduced resolution because the second virtual object is located at a greater depth (e.g., a greater depth plane) from the determined fixed point. As a result, different resolutions may be associated with different distances from the fixed point.

[0112] In some embodiments, the environment surrounding the user may be divided into spatial volumes (also referred herein to as resolution adjustment zones), and the resolution of virtual objects within the same resolution adjustment zone is similar. The resolution adjustment zones may have arbitrary three-dimensional shapes, such as cubes, other three-dimensional polygonal shapes, or curved three-dimensional shapes, as described herein. In some embodiments, all resolution adjustment zones have similar shapes, such as cuboids or spheres. In some other embodiments, different resolution adjustment zones may have different shapes or sizes (for example, the shape and / or size of the volume may change with distance from the fixation point).

[0113] In some embodiments, the resolution adjustment zone is part of the user's field of view. For example, the user's field of view may be separated into volumes of space that form the resolution adjustment zone. In some embodiments, each depth plane may be subdivided into one or more continuous volumes of space, i.e., one or more resolution adjustment zones. In some embodiments, each resolution adjustment zone may encompass a specific range of depth from the user (e.g., depth plane value + / - some dispersion, examples of which include 0.66 dpt, 0.50 dpt, 0.33 dpt, or 0.25 dpt) and specific lateral and specific vertical distances. Virtual objects located within the same resolution adjustment zone as the determined fixation point may be presented (e.g., rendered) at high (e.g., full) resolution, while virtual objects located in volumes of space outside the fixation point's resolution adjustment zone may be rendered at a lower resolution depending on the distance of that volume from the fixation point's volume. In some embodiments, each resolution adjustment zone may be assigned a specific resolution (e.g., a specific reduction in resolution relative to full resolution), and virtual content entering a given zone may be rendered at the associated resolution for that zone. In some embodiments, the distance between a certain volume and the volume occupied by a fixed point may be determined, and the resolution may be set based on this distance.

[0114] Advantageously, the number and size of the resolution adjustment zones used to divide the user's field of view can be modified according to the user's confidence in the fixed point. For example, the size associated with the volume of each space may be increased or decreased based on the user's confidence that their line of sight converges on a precise point in three-dimensional space. If the confidence in the fixed point is high, the display system may present only virtual objects within a certain compact resolution adjustment zone with relatively high resolution (including the fixed point and the compact resolution adjustment zone), while reducing the resolution of other virtual objects, thus saving processing power. However, if the confidence is low, the display system may increase the size of the volume of each space so that the volume of each space contains a larger number of virtual objects within the volume of the fixed point space (e.g., reducing the overall number of volumes). It should be understood that the size and shape of the volumes may be fixed during the production of the display system, for example, based on the tolerances expected in the system for determining the fixed point, and / or may be adjusted or set in the field in response to user characteristics, the user's environment, and / or changes to the software that modifies the tolerances for the system for determining the fixed point.

[0115] It should be understood that a user's sensitivity to resolution can decrease with distance from the fixed point. As a result, by ensuring that full-resolution content is presented at the fixed point, and by allowing tolerance for errors regarding the location of the fixed point, the perceptual ability to perceive reductions in resolution can be reduced or eliminated, thereby providing the perception of a high-resolution display without utilizing the computational resources typically required to present content for such high-resolution displays.

[0116] In some embodiments, the proximity of a virtual object to a fixation point may be determined based on the angular proximity of the virtual object to the user's line of sight, and the resolution of the virtual object may decrease as the angular proximity decreases. In some embodiments, this may result in virtual objects located at different depths from the user being presented with similar resolution. For example, a first virtual object at a location corresponding to the user's determined fixation point may be located in front of a second virtual object (e.g., closer to the user). The second virtual object may optionally be presented with similar (e.g., identical) resolution to the first virtual object, since the second virtual object is along the user's line of sight and therefore similarly will be located on the user's fovea, where the user's eye is most sensitive to changes in resolution. Optionally, the second virtual object may be further modified through a blurring process that reduces its resolution and may indicate that the second virtual object is further from the user (e.g., located on a more distant depth plane) (e.g., a Gaussian blur kernel may be convolved with the second virtual object).

[0117] The reduction in resolution may vary depending on how the virtual content is presented by the display system. In some embodiments, a first exemplary display system, referred herein as a variable focus display system, may present virtual content on different depth planes, and all content (e.g., virtual objects) may be presented on the same depth plane (e.g., via the same waveguide) at once, for example, for each frame presented to the user. That is, the variable focus display system may present content using a single depth plane (e.g., selected from a plurality of depth planes based on the user's fixation point, or selected based on the depth of a particular presented virtual object) at once, or it may change the depth plane within subsequent frames (e.g., select a different depth plane). In some other embodiments, a second exemplary display system, referred herein as a multi-focus display system, may present virtual content on different depth planes, and the content may be displayed on multiple depth planes simultaneously. As will be further described herein, a variable focus display system may optionally utilize a single frame buffer, and in the above embodiment relating to the step of blurring a second virtual object, the second virtual object may be blurred prior to its presentation to the user from the single frame buffer. In contrast, a multi-focus display system may optionally present the second virtual object at a greater depth (e.g., on a greater depth plane) than the first virtual object, at a reduced resolution, and the second virtual object may appear blurred to the user (e.g., the second virtual object will be blurred based on the natural physics of the user's eye without further processing).

[0118] As disclosed herein, a display system may present virtual objects at or near a determined fixation point with relatively high (e.g., full) resolution, and virtual objects further from the fixation point with reduced resolution. Preferably, the relatively high resolution is the highest resolution for presenting virtual objects within the user's field of view. The relatively high resolution may be the maximum resolution of the display system, a user-selectable resolution, a resolution based on the specific computing hardware presenting the virtual objects, and so on.

[0119] It should be understood that the step of adjusting the resolution of a virtual object may include any modifications to the virtual object to alter the quality of the virtual object's presentation. Such modifications may include one or more of the following: adjusting the quality at one or more points in the graphics pipeline of the graphics processing unit (GPU); adjusting the polygon count of the virtual object; adjusting the primitives used to generate the virtual object (e.g., adjusting the shape of the primitives, e.g., adjusting the primitives from a triangular mesh to a quadrilateral mesh); adjusting the actions performed on the virtual object (e.g., shader actions); adjusting texture information; adjusting color resolution or depth; adjusting the number of rendering cycles or frame rate.

[0120] In some embodiments, along the x and y axes, changes in the resolution of virtual content as it moves away from the fixation point can generally track changes in the distribution of photoreceptors in the retina of the user's eye. For example, it should be understood that views of the world and virtual content may be imaged onto the retina so that different parts of the retina can be mapped to different parts of the user's field of vision. Advantageously, the resolution of virtual content across the user's field of vision can generally track the density of the corresponding photoreceptors (rods or cones) across the retina. In some embodiments, the reduction in resolution as it moves away from the fixation point can generally track a reduction in the density of cones across the retina. In some other embodiments, the reduction in resolution as it moves away from the fixation point can generally track a reduction in the density of rods across the retina. In some embodiments, the trend of the reduction in resolution as it moves away from the fixation point may be within ±50%, ±30%, ±20%, or ±10% of the trend of the reduction in the density of rods and / or cones across the retina.

[0121] Rods and cones are active at different levels of incidence. For example, cones are active under relatively bright conditions, while rods are active under relatively low light conditions. As a result, the reduction in resolution generally tracks the density of rods or cones across the retina. In some embodiments, the display system may be configured to determine the amount of light incident on the retina. Based on this amount of light, appropriate adjustments to the resolution can be made. For example, the reduction in resolution generally tracks changes in the density of rods across the retina under low light conditions, while the reduction in resolution generally tracks changes in the density of cones under bright conditions. As a result, in some embodiments, the display system may be configured to change the profile of the reduction in image resolution based on the amount of light incident on the retina.

[0122] It should be understood that the human eye's ability to resolve fine details is not always directly proportional to the density of rods or cones in the retina. In some embodiments, changes in the resolution of virtual content across the user's field of view generally track changes in the eye's ability to resolve fine details. As mentioned above, the rate at which the resolution of virtual content changes may vary with the amount of light reaching the retina.

[0123] In some embodiments, the amount of light reaching the retina may be determined by detecting the amount of ambient light incident on a sensor mounted on the display device. In some embodiments, the step of determining the amount of light reaching the retina may also include the step of determining it by the amount of light output to the user by the display device. In yet another embodiment, the amount of light reaching the retina may be determined by imaging the user's eye and determining the pupil size. Since pupil size is related to the amount of light reaching the retina, determining the pupil size allows the amount of light reaching the retina to be extrapolated.

[0124] It should be understood that full-color virtual content can be formed by multiple primary color images that, as a whole, provide a full-color perception. The human eye may have different sensitivities to different wavelengths or colors of light. In some embodiments, in addition to changing based on proximity to the fixation point, the resolution of the virtual content may vary based on the colors of the primary color images presented by the display system. For example, if the primary color images include red, green, and blue images, the green primary color image may have a higher resolution than the blue primary color image, and the red primary color image may have a higher resolution than the blue primary color image. In some embodiments, the amount of light reaching the retina may be determined to account for the changes in the eye's sensitivity to different colors at different levels of incident light, and the resolution adjustment for a given primary color image may also vary based on the determination of the amount of light reaching the retina.

[0125] It should be understood that the eye's contrast sensitivity can also vary based on the amount of light incident on the retina. In some embodiments, the size or total number of tonal gradations in contrast within virtual content can vary based on the amount of light reaching the retina. In some embodiments, the contrast ratio of the images forming the virtual content can vary based on the amount of light incident on the retina, and the contrast ratio decreases as the amount of light decreases.

[0126] In some embodiments, certain portions of the user's field of view may not be provided with any virtual content. For example, a display system may be configured not to provide virtual content within blind spots caused by the optic nerve and / or peripheral blind spot of a given eye.

[0127] As discussed herein, a display system may be configured to display high-resolution content within a portion of the user's field of view and lower-resolution content within another portion of the user's field of view. It should be understood that high-resolution content may have a higher pixel density than lower-resolution content. In some environments, a display system may be configured to provide such high and low-resolution content by effectively superimposing high and low-resolution images. For example, the system may display a low-resolution image covering the entire field of view, and then display a high-resolution image covering a small portion of the field of view, with the high-resolution image located in the same place as the corresponding portion of the low-resolution image. The high and low-resolution images may be routed through different optical systems that output light at appropriate angles, determining the amount of field of view they occupy.

[0128] In some embodiments, a single-space light modulator (SLM) may be used to encode light with image information, and a beam splitter or optical switch may be used to split the single light stream from the SLM into two streams, one stream propagating through an optical system for a low-resolution image and the second stream propagating through an optical system for a high-resolution image. In some other embodiments, the polarization of the light encoded with image information may be selectively switched and passed through an optical system that effectively provides different angular magnifications for light of different polarizations, thereby providing high and low-resolution images.

[0129] Advantageously, the various embodiments disclosed herein reduce the processing power requirements for delivering content onto a display system. A greater allocation of processing power can be spent on virtual objects closer to the user's three-dimensional fixed point, while processing power for more distant virtual objects can be reduced. Therefore, the overall processing power required for the display system is reduced, and thus one or more of the following can be reduced: the size of the processing components, the heat generated by the processing components, and the energy requirements for the display system (e.g., the display system may optionally be battery-powered, requiring lower capacity batteries, and / or able to operate for a longer duration for a given battery). Thus, the embodiments described herein address the technical problems arising from augmented or virtual reality display systems. In addition, the techniques described manipulate the graphical content so that, depending on the presentation to the user, the graphical content may appear to the user as identical, while being presented in a fundamentally different way (e.g., the resolution is modified). Thus, the display system maintains visual fidelity and saves processing power while transforming the graphical content as the user looks around its surroundings.

[0130] It should be understood that the display system may be part of an augmented reality display system or a virtual reality display system. In one embodiment, the display of the display system may be transparent, allowing the user to view the real world while providing the user with virtual content in the form of images, videos, interactions, etc. In another embodiment, the display system may block the user's view of the real world, and virtual reality images, videos, interactions, etc., may be presented to the user.

[0131] Here, we refer to the drawings, where similar reference numbers refer to the same parts throughout.

[0132] Figure 2 illustrates a conventional display system for simulating a three-dimensional image for a user. It should be understood that when a user's eyes are spaced apart and viewing a real object in space, each eye may have a slightly different view of the object and may form an image of the object in different locations on the retina of each eye. This may be called binocular parallax and can be used by the human visual system to provide a sense of depth. Conventional display systems simulate binocular parallax by presenting two distinctly different images 190, 200, one for each eye 210, 220, with slightly different views of the same virtual object, corresponding to the view of the virtual object that each eye would see as a real object at a desired depth. These images provide binocular cues that the user's visual system can interpret to derive a sense of depth.

[0133] Continuing with Figure 2, images 190 and 200 are spaced 230 units away from eyes 210 and 220 on the z-axis. The z-axis is parallel to the viewer's optical axis when the eye is fixated on an object at optical infinity directly in front of the viewer. Images 190 and 200 are flat and at a fixed distance from eyes 210 and 220. Based on slightly different views of the virtual object in the images presented to eyes 210 and 220, respectively, the eyes may rotate so that the image of the object comes to the corresponding point on the respective retina of the eye, maintaining monobiocular vision. This rotation can converge the lines of sight of eyes 210 and 220 to a point in space where the virtual object is perceived to exist. As a result, the provision of three-dimensional images conventionally involves manipulating the convergence and divergence movements of the user's eyes 210 and 220 and providing binocular cues that the human visual system interprets to provide depth perception.

[0134] However, generating a realistic and comfortable perception of depth is difficult. It should be understood that light from an object at different distances from the eye has wavefronts with different amounts of divergence. Figures 3A-3C illustrate the relationship between distance and ray divergence. The distances between the object and the eye 210 are expressed in the order of decreasing distances R1, R2, and R3. As shown in Figures 3A-3C, the rays diverge more as the distance to the object decreases. Conversely, as the distance increases, the rays become more collimated. In other words, the light field generated by a point (object or part of an object) can be said to have a spherical wavefront curvature, which is a function of the distance the point is from the user's eye. Only a monocular eye 210 is illustrated in Figures 3A-3C and various other figures herein for the sake of clarity in the illustration, but the discussion with respect to the eye 210 can be applied to the binocular eyes 210 and 220 of the viewer.

[0135] Continuing to refer to Figures 3A-3C, light from an object that a viewer's eye is fixated on may have different wavefront divergences. Due to the different wavefront divergences, the light may be focused differently by the eye's lens, which may require the lens to take on different shapes and form focused images on the eye's retina. If a focused image is not formed on the retina, the resulting retinal blur acts as a cue for accommodation, causing a change in the shape of the eye's lens until a focused image is formed on the retina. For example, a cue for accommodation may trigger relaxation or contraction of the ciliary muscle surrounding the eye's lens, thereby modulating the force applied to the low ligament that holds the lens, and thus changing the shape of the eye's lens until the retinal blur of the fixed image is eliminated or minimized, thereby forming a focused image of the fixed object on the eye's retina (e.g., the fovea). The process by which the lens of the eye changes shape can be called accommodation, and the shape of the lens required to form a focused image of the object being fixed on onto the retina of the eye (e.g., the fovea) can be called the accommodative state.

[0136] Referring here to Figure 4A, the representation of the accommodation-convergence-divergence response of the human visual system is illustrated. Eye movement to fixate on an object causes the eye to receive light from the object, and the light forms an image on each retina of the eye. The presence of retinal blur in the image formed on the retina can provide cues for accommodation, and the relative location of the image on the retina can provide cues for convergence-divergence movement. The cues for accommodation produce accommodation, resulting in the lens of the eye taking on a specific accommodative state in which a focused image of the object is formed on the retina of the eye (e.g., the fovea). On the other hand, the cues for convergence-divergence movement produce convergence-divergence movement (rotation of the eye) so that the image formed on each retina of each eye is at the corresponding retinal point that maintains monobiocular vision. At these positions, the eye can be said to be in a specific convergence-divergence state. Continuing to refer to Figure 4A, accommodation can be understood as the process by which the eye achieves a specific state of accommodation, and convergence / divergence can be understood as the process by which the eye achieves a specific state of convergence / divergence. As shown in Figure 4A, the state of accommodation and convergence / divergence of the eye can change when the user fixates on a different object. For example, the accommodated state can change when the user fixates on a new object at a different depth on the z-axis.

[0137] While not limited by theory, it is thought that an object viewer may perceive an object as "three-dimensional" due to a combination of convergence-divergence movements and accommodation. As mentioned earlier, convergence-divergence movements of two eyes relative to each other (e.g., eye rotations such as pupils moving toward or away from each other, converging the lines of sight and fixing on an object) are closely related to the accommodation of the eye's lens. Under normal conditions, changing the shape of the eye's lens and shifting focus from one object to another at a different distance will automatically produce a coherent change in convergence-divergence movements to the same distance, under a relationship known as the "accommodation-convergence-divergence reflex." Similarly, changes in convergence-divergence movements will, under normal conditions, trigger a coherent change in lens shape.

[0138] Referring now to Figure 4B, embodiments of different accommodation and convergence / divergence states of the eyes are illustrated. Eye pair 222a fixates on an object at optical infinity, while eye pair 222b fixates on an object 221 below optical infinity. It is noteworthy that the convergence / divergence states of each pair of eyes are different, with eye pair 222a pointing straight ahead, while eye pair 222 converges on object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of the lenses 210a and 220a.

[0139] Unfortunately, many users of conventional "3-D" display systems find such systems uncomfortable or completely fail to perceive depth due to the mismatch between the accommodation and convergence / divergence states in these displays. As mentioned earlier, many stereoscopic or "3-D" display systems display scenes by providing slightly different images to each eye. Such systems are uncomfortable for many viewers because they, above all, simply provide different presentations of scenes and cause changes in the convergence / divergence states of the eyes, but without corresponding changes in the accommodation states of those eyes. Rather, the images are presented by the display at a fixed distance from the eyes so that the eyes perceive all image information in a single accommodation state. Such arrangements go against the "accommodation-convergence / divergence reflex" by causing changes in the convergence / divergence state without corresponding changes in the accommodation state. This mismatch is thought to cause viewer discomfort. A display system that provides better integration between distance accommodation and convergence / divergence motion can create a more realistic and comfortable simulation of three-dimensional images.

[0140] While not limited by theory, the human eye is typically thought to be capable of interpreting a finite number of depth planes and providing depth perception. Consequently, a highly realistic simulation of perceived depth can be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. In some embodiments, the different presentations may provide both cues for convergence-divergence movements and matching cues for accommodation, thereby providing physiologically correct accommodation-convergence-divergence movement matching.

[0141] Continuing with Figure 4B, two depth planes 240 are illustrated, corresponding to different spatial distances from eyes 210 and 220. With respect to a given depth plane 240, condensation-divergence motion cues may be provided by displaying appropriately different viewpoint images for each eye 210 and 220. In addition, with respect to a given depth plane 240, the light forming the image provided to each eye 210 and 220 may have wavefront divergence corresponding to a light field generated by a point at a distance in that depth plane 240.

[0142] In the illustrated embodiment, the distance along the z-axis of the depth plane 240 containing point 221 is 1 m. As used herein, the distance along the z-axis, or depth, may be measured using a zero point located at the exit pupil of the user's eye. Thus, the depth plane 240 located at a depth of 1 m corresponds to a distance of 1 m from the exit pupil of the user's eye on the optical axis of those eyes. As an approximation, the depth or distance along the z-axis may be supplemented with a value relating to the distance between the user's exit pupil and the display in front of the user's eye (e.g., the surface of a waveguide) when the device and the eye are pointed towards optical infinity. This value is called the pupil distance and may correspond to the distance between the user's exit pupil and the user-worn display in front of the eye. In practice, the value relating to the pupil distance may generally be a normalized value used for all spectators. For example, the pupil distance may be assumed to be 20 mm, and the depth plane at a depth of 1 m may be at a distance of 980 mm from the front of the display.

[0143] Referring here to Figures 4C and 4D, embodiments of aligned accommodation-convergence-divergence distance and misaligned accommodation-convergence-divergence distance are illustrated, respectively. As shown in Figure 4C, the display system may provide images of virtual objects to each eye 210, 220. The images can cause the eyes 210, 220 to assume a convergence-divergence state in which the eyes converge on point 15 on the depth plane 240. In addition, the images may be formed by light having a wavefront curvature corresponding to a real object in its depth plane 240. As a result, the eyes 210, 220 assume an accommodation state in which the image is focused on the retina of their eyes. Thus, the user can perceive the virtual object as being at point 15 on the depth plane 240.

[0144] It should be understood that the accommodation and convergence / divergence states of eyes 210 and 220 are associated with a specific distance on the z-axis. For example, an object at a specific distance from eyes 210 and 220 will cause those eyes to assume a specific accommodation state based on the object's distance. The distance associated with a specific accommodation state is the accommodation distance A. d It can be called a specific convergence-divergence distance V associated with the eyes, in particular the convergence-divergence movement state or relative position to each other. d However, this also exists. When the distance of accommodation and the distance of convergence and divergence are consistent, the relationship between accommodation and convergence / divergence can be said to be physiologically correct. This is considered the most comfortable scenario for the viewer.

[0145] However, in stereoscopic displays, the accommodation distance and the convergence / divergence distance may not always be consistent. For example, as illustrated in Figure 4D, the images displayed on eyes 210 and 220 may be displayed with wavefront divergence corresponding to the depth plane 240, and eyes 210 and 220 may take on a specific accommodation state in which points 15a and 15b on their depth plane are in focus. However, the images displayed on eyes 210 and 220 may provide cues for convergence / divergence that cause eyes 210 and 220 to converge on point 15, which is not located on the depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to the distance from a specific reference point of the user (e.g., the exit pupil of eyes 210 and 220) to the depth plane 240, while the convergence / divergence distance corresponds to a larger distance from that reference point to point 15. Therefore, unlike convergence-divergence motion distance, there is a near-far accommodation-convergence-divergence motion mismatch. Such mismatches are considered undesirable and can cause discomfort to the user. The mismatch is due to the distance (e.g., V d -A d It should be understood that this corresponds to and can be characterized using diopters (units of length, the reciprocal of 1 / m). For example, V of 1.75 diopters d and 1.25 diopters of A d Or V of 1.25 diopters d and 1.75 diopters of A d This would provide a 0.5 diopter accommodation-convergence-divergence motion mismatch.

[0146] It should be understood that in some embodiments, as long as the same reference point is used for the near-to-near-to-far accommodation distance and the convergence-divergence distance, reference points other than the exit pupils of eyes 210, 220 may be used to determine the distance for determining the near-to-near-to-far accommodation-to-convergence-divergence mismatch. For example, the distance may be measured from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., the waveguide of a display device) to the depth plane, etc.

[0147] While not limited by theory, it is conceivable that users may still perceive accommodation-convergence-divergence mismatches of up to approximately 0.25 diopters, up to approximately 0.33 diopters, and up to approximately 0.5 diopters as physiologically correct, even if the mismatch itself does not cause significant discomfort. In some embodiments, the display systems disclosed herein (e.g., display system 250, Figure 6) present to the viewer an image having accommodation-convergence-divergence mismatch of about 0.5 diopters or less. In some other embodiments, the accommodation-convergence-divergence mismatch of the image provided by the display system is about 0.33 diopters or less. In yet more embodiments, the accommodation-convergence-divergence mismatch of the image provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0148] Figure 5 illustrates an aspect of an approach to simulating a three-dimensional image by correcting wavefront divergence. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output that light to the user's eye 210. The waveguide 270 may output light 650 with a defined amount of wavefront divergence corresponding to the wavefront divergence of the light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on that depth plane. In addition, the user's other eye may be shown to be provided with image information from a similar waveguide.

[0149] In some embodiments, a single waveguide may be configured to output light with a set wavefront divergence corresponding to one or a limited number of depth planes, and / or the waveguide may be configured to output light with a limited range of wavelengths. As a result, in some embodiments, multiple or stacked waveguides may be used to provide different wavefront divergences for different depth planes, and / or to output light with different ranges of wavelengths. It should be understood that, as used herein, depth planes can follow the contours of flat or curved surfaces. In some embodiments, for convenience, depth planes can follow the contours of flat surfaces.

[0150] Figure 6 illustrates an embodiment of a waveguide stack for outputting image information to a user. The display system 250 includes a waveguide stack or stacked waveguide assembly 260, which may be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 270, 280, 290, 300, 310. It should be understood that the display system 250 may be considered a light field display in some embodiments. In addition, the waveguide assembly 260 may also be referred to as an eyepiece.

[0151] In some embodiments, the display system 250 may be configured to provide substantially continuous cues for convergence-divergence motion and a plurality of discrete cues for near accommodation. Cues for convergence-divergence motion may be provided by displaying different images to each of the user's eyes, and cues for near accommodation may be provided by outputting light that forms an image with a selectable discrete amount of wavefront divergence. In other words, the display system 250 may be configured to output light with a variable level of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a specific depth plane and may be provided by a particular one of waveguides 270, 280, 290, 300, and 310.

[0152] Continuing with Figure 6, the waveguide assembly 260 may also include several features 320, 330, 340, and 350 between the waveguides. In some embodiments, features 320, 330, 340, and 350 may be one or more lenses. Waveguides 270, 280, 290, 300, and 310 and / or several lenses 320, 330, 340, and 350 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 360, 370, 380, 390, and 400 may function as light sources for the waveguides and may be used to input image information into waveguides 270, 280, 290, 300, and 310, and each may be configured to disperse incident light across each individual waveguide for output toward the eye 210, as described herein. The light exits from the output surfaces 410, 420, 430, 440, and 450 of image input devices 360, 370, 380, 390, and 400 and is input into the corresponding input surfaces 460, 470, 480, 490, and 500 of waveguides 270, 280, 290, 300, and 310. In some embodiments, the input surfaces 460, 470, 480, 490, and 500 may each be the edge of the corresponding waveguide or a portion of the main surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide and output a whole field of cloned collimated beams directed toward the eye 210 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, one of the image input devices 360, 370, 380, 390, and 400 may be associated with a plurality (e.g., three) of waveguides 270, 280, 290, 300, and 310, into which light may be injected.

[0153] In some embodiments, the image input devices 360, 370, 380, 390, and 400 are discrete displays that generate image information for input into the corresponding waveguides 270, 280, 290, 300, and 310, respectively. In some other embodiments, the image input devices 360, 370, 380, 390, and 400 are output terminals of a single multiplexed display that can send image information to each of the image input devices 360, 370, 380, 390, and 400, for example, via one or more optical conduits (such as optical fiber cables). It should be understood that the image information provided by the image input devices 360, 370, 380, 390, and 400 may include light of different wavelengths or colors (e.g., different primary colors, as discussed herein).

[0154] In some embodiments, the light introduced into waveguides 270, 280, 290, 300, and 310 is provided by an optical projector system 520, which comprises an optical module 530, which may include an optical emitter such as a light-emitting diode (LED). The light from the optical module 530 may be directed and modified by an optical modulator 540, such as a spatial light modulator, via a beam splitter 550. The optical modulator 540 may be configured to change the perceived intensity of the light introduced into waveguides 270, 280, 290, 300, and 310, thereby encoding the light with image information. Embodiments of the spatial light modulator include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays. Image input devices 360, 370, 380, 390, and 400 are graphically illustrated, and it should be understood that in some embodiments, these image input devices may represent different optical paths and locations within a common projection system, configured to output light into associated waveguides 270, 280, 290, 300, and 310. In some embodiments, the waveguides of waveguide assembly 260 may function as ideal lenses, relaying the light input into the waveguides to the user's eye. In this concept, the object may be a spatial light modulator 540, and the image may be an image on the depth plane.

[0155] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scanning, helical scanning, Lissajous patterns, etc.) into one or more waveguides 270, 280, 290, 300, 310, and ultimately to the viewer's eye 210. In some embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a single scanning fiber or a bundle of scanning fibers configured to input light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a plurality of scanning fibers or a plurality of bundles of scanning fibers, each configured to input light into one of the associated waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or multiple fibers and one or more waveguides 270, 280, 290, 300, 310, for example, to redirect light emitted from the scanning fiber into one or more waveguides 270, 280, 290, 300, 310.

[0156] The controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image input devices 360, 370, 380, 390, 400, the light source 530, and the optical module 540. In some embodiments, the controller 560 is part of the local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transient medium) to coordinate the timing and provisioning of image information to the waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 560 may be part of the processing module 140 or 150 (Figure 9D).

[0157] Continuing with Figure 6, waveguides 270, 280, 290, 300, and 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 270, 280, 290, 300, and 310 may each be planar or have another shape (e.g., curved), with major upper and lower surfaces and edges extending between their major upper and lower surfaces. In the illustrated configuration, waveguides 270, 280, 290, 300, and 310 may each include external coupling optical elements 570, 580, 590, 600, and 610, respectively, configured to extract light from the waveguides by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguides to the eye 210. The extracted light may also be referred to as externally coupled light, and the external coupling optical elements may also be referred to as light extraction optical elements. The extracted beam of light can be output by the waveguide at the point where light propagating within the waveguide strikes the light extraction optical element. The external coupling optical elements 570, 580, 590, 600, 610 may be gratings, for example, including diffractive optical features as further discussed herein. For the sake of clarity and to facilitate the explanation, they are shown positioned on the bottom main surfaces of the waveguides 270, 280, 290, 300, 310, but in some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be positioned on the top and / or bottom main surfaces, and / or directly within the volume of the waveguides 270, 280, 290, 300, 310, as further discussed herein. In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be mounted on a transparent substrate and formed within a layer of material that forms the waveguides 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be monolithic components of the material, and the external coupling optical elements 570, 580, 590, 600, 610 may be formed on and / or inside the surface of that material component.

[0158] Continuing with reference to Figure 6, as discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to emit light and form an image corresponding to a specific depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (injected into such waveguide 270) to the eye 210. The collimated light may represent the optical infinity focal plane. The next upper waveguide 280 may be configured to emit collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210. Such a first lens 350 may generate some convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 280 as originating from a first focal plane closer inward from optical infinity toward the eye 210. Similarly, the third upper waveguide 290 passes its output light through both the first 350 and second 340 lenses before reaching the eye 210. The combined refractive power of the first 350 and second 340 lenses may be configured to produce another gradually increasing wavefront curvature so that the eye / brain interprets the light originating from the third waveguide 290 as originating from a second focal plane that is even closer inward toward the person from optical infinity than the light from the next upper waveguide 280.

[0159] Other waveguide layers 300, 310 and lenses 330, 320 are configured similarly, with the highest waveguide 310 in the stack emitting its output through all the lenses between it and the eye for a convergent focusing force representing the focal plane closest to the person. When viewing / interpreting light originating from the other side world 510 of the stacked waveguide assembly 260, a compensating lens layer 620 may be positioned on top of the stack to compensate for the convergent forces of the lower lens stacks 320, 330, 340, 350 to compensate for the stack of lenses 320, 330, 340, 350. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the external coupling optical elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0160] In some embodiments, two or more of the waveguides 270, 280, 290, 300, and 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, and 310 may be configured to output images set in the same depth plane, or multiple subsets of waveguides 270, 280, 290, 300, and 310 may be configured to output images set in the same multiple depth planes, with one set for each depth plane. This may offer the advantage of forming tiled images that provide an extended field of view in those depth planes.

[0161] Continuing with Figure 6, the external coupling optical elements 570, 580, 590, 600, and 610 may be configured to redirect light from their respective waveguides for specific depth planes associated with the waveguides and to output the light with an appropriate amount of divergence or collimation. As a result, waveguides with different associated depth planes may have different configurations of the external coupling optical elements 570, 580, 590, 600, and 610, which will output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, and 610 may be volumetric or surface features, which may be configured to output light at specific angles. For example, the light extraction optical elements 570, 580, 590, 600, and 610 may be volumetric holograms, surface holograms, and / or diffraction gratings. In some embodiments, features 320, 330, 340, and 350 may not be lenses. Rather, they may simply be spacers (e.g., structures for forming cladding layers and / or voids).

[0162] In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOEs have sufficiently low diffraction efficiency so that only a portion of the beam light is deflected toward the eye 210 through each intersection of the DOEs, while the remainder continues to travel through the waveguide via TIR. The light carrying the image information is therefore split into several associated emission beams that exit the waveguide at various locations, resulting in a very uniform pattern of emission toward the eye 210 with respect to this particular collimated beam bouncing within the waveguide.

[0163] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal, in which microdroplets have a diffraction pattern in the host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0164] In some embodiments, a camera assembly 630 (e.g., a digital camera including visible light and infrared light cameras) may be provided to capture images of the eye 210 and / or the surrounding tissues, for example, to detect user input and / or monitor the user's physiological state. As used herein, the camera may be any image-capturing device. In some embodiments, the camera assembly 630 may include the image-capturing device and a light source that projects light (e.g., infrared light) onto the eye, which is then reflected by the eye and can be detected by the image-capturing device. In some embodiments, the camera assembly 630 may be mounted on a frame 80 (Figure 9D) and may communicate with processing modules 140 and / or 150 that can process image information from the camera assembly 630. In some embodiments, one camera assembly 630 may be used per eye to monitor each eye separately.

[0165] Referring here to Figure 7, an embodiment of an outgoing beam output by a waveguide is shown. Although one waveguide is illustrated, other waveguides within the waveguide assembly 260 (Figure 6) may function similarly, and it should be understood that the waveguide assembly 260 includes multiple waveguides. Light 640 is introduced into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates through the waveguide 270 by TIR. At the point where the light 640 collides on the DOE 570, a portion of the light exits the waveguide as an outgoing beam 650. The outgoing beam 650 is illustrated as substantially parallel, but may be redirected to propagate to the eye 210 at a certain angle (e.g., divergent outgoing beam formation) depending on the depth plane associated with the waveguide 270, as discussed herein. It should be understood that a nearly parallel emitted beam may represent a waveguide with an external coupling optical element that externally couples the light to form an image that appears to be set in the depth plane at a distance from the eye 210 (e.g., optical infinity). Other waveguides or other sets of external coupling optical elements may output a more divergent emitted beam pattern, which would require the eye 210 to adjust to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0166] In some embodiments, a full-color image may be formed in each depth plane by overlaying an image onto each of the primary colors, for example, three or more primary colors. Figure 8 illustrates an embodiment of a stacked waveguide assembly, where each depth plane includes an image formed using several different primary colors. The illustrated embodiment shows depth planes 240a–240f, but more or fewer depths may also be considered. Each depth plane may have three or more associated primary color images, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. Different depth planes are illustrated by different numbers relating to diopters (dpt) following the letters G, R, and B. As merely an embodiment, the numbers following each of these letters indicate diopters (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the figure represents an individual primary color image. In some embodiments, the exact location of the depth planes relating to different primary colors may vary to account for differences in the focusing of light of different wavelengths on the eye. For example, different primary color images with respect to a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or reduce chromatic aberration.

[0167] In some embodiments, each primary color light may be output by a single dedicated waveguide, and as a result, each depth plane may have multiple waveguides associated with it. In such embodiments, each box in the figure, including the letters G, R, or B, can be understood to represent an individual waveguide, and three waveguides may be provided for each depth plane, and three primary color images are provided for each depth plane. The waveguides associated with each depth plane are shown adjacent to each other in this drawing for ease of explanation, but it should be understood that in a physical device, all waveguides may be arranged in a stack with one waveguide per level. In some other embodiments, multiple primary colors may be output by the same waveguide, for example, so that only a single waveguide may be provided for each depth plane.

[0168] Continuing to refer to Figure 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may be used in addition to or replace one or more of red, green, or blue.

[0169] Throughout this disclosure, any reference to a given color of light should be understood as encompassing one or more wavelengths of light within a range of wavelengths that are perceived by the viewer as that given color. For example, red light may include one or more wavelengths of light in the range of approximately 620–780 nm, green light may include one or more wavelengths of light in the range of approximately 492–577 nm, and blue light may include one or more wavelengths of light in the range of approximately 435–493 nm.

[0170] In some embodiments, the light source 530 (Figure 6) may be configured to emit light of one or more wavelengths outside the viewer's visual perception range, such as infrared and / or ultraviolet wavelengths. In addition, internal coupling, external coupling, and other light redirection structures of the waveguide of the display 250 may be configured to direct and emit this light from the display toward the user's eye 210, for example, for imaging and / or user stimulation applications.

[0171] Referring here to Figure 9A, in some embodiments, light impacting a waveguide may need to be redirected to internally couple that light into the waveguide. Internal coupling optical elements may be used to redirect and internally couple the light into its corresponding waveguide. Figure 9A illustrates cross-sectional side views of embodiments of multiple or set 660 stacked waveguides, each including an internal coupling optical element. Each waveguide may be configured to output light of one or more different wavelengths or one or more different wavelength ranges. Stack 660 may correspond to stack 260 (Figure 6), and the illustrated waveguides of stack 660 may correspond to some of the multiple waveguides 270, 280, 290, 300, 310, but it should be understood that light from one or more of the image input devices 360, 370, 380, 390, 400 is input into the waveguide from a position where the light is required to be redirected for internal coupling.

[0172] The illustrated set of stacked waveguides 660 includes waveguides 670, 680, and 690. Each waveguide includes associated internal coupling optical elements (which may also be referred to as optical input areas on the waveguide), for example, internal coupling optical element 700 is located on the main surface of waveguide 670 (e.g., the upper main surface), internal coupling optical element 710 is located on the main surface of waveguide 680 (e.g., the upper main surface), and internal coupling optical element 720 is located on the main surface of waveguide 690 (e.g., the upper main surface). In some embodiments, one or more of the internal coupling optical elements 700, 710, and 720 may be located on the bottom main surfaces of individual waveguides 670, 680, and 690 (in particular, one or more internal coupling optical elements are reflective deflection optical elements). As illustrated, the internally coupled optical elements 700, 710, and 720 may be located on the upper main surface of their respective waveguides 670, 680, and 690 (or on the upper part of the following lower waveguide), and in particular, these internally coupled optical elements are transmissive deflection optical elements. In some embodiments, the internally coupled optical elements 700, 710, and 720 may be located within the body of the respective waveguides 670, 680, and 690. In some embodiments, as discussed herein, the internally coupled optical elements 700, 710, and 720 are wavelength-selective, selectively redirecting one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of their respective waveguides 670, 680, and 690, it should be understood that in some embodiments, the internally coupled optical elements 700, 710, and 720 may be located within other areas of their respective waveguides 670, 680, and 690.

[0173] As illustrated, the internally coupled optical elements 700, 710, and 720 may be offset laterally from one another. In some embodiments, each internally coupled optical element may be offset so that its light does not pass through another internally coupled optical element to receive light. For example, each internally coupled optical element 700, 710, and 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in Figure 6, and may be separated from other internally coupled optical elements 700, 710, and 720 (e.g., separated laterally) so that it does not substantially receive light from the other internally coupled optical elements 700, 710, and 720.

[0174] Each waveguide also includes associated optical dispersion elements, for example, optical dispersion element 730 is located on the main surface (e.g., upper main surface) of waveguide 670, optical dispersion element 740 is located on the main surface (e.g., upper main surface) of waveguide 680, and optical dispersion element 750 is located on the main surface (e.g., upper main surface) of waveguide 690. In some other embodiments, optical dispersion elements 730, 740, and 750 may be located on the bottom main surfaces of the associated waveguides 670, 680, and 690, respectively. In some other embodiments, optical dispersion elements 730, 740, and 750 may be located on both the top and bottom main surfaces of the associated waveguides 670, 680, and 690, respectively, or optical dispersion elements 730, 740, and 750 may be located on different top and bottom main surfaces within different associated waveguides 670, 680, and 690, respectively.

[0175] Waveguides 670, 680, and 690 may be separated and isolated by, for example, gaseous, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediate vicinity of waveguides 670, 680, and 690). Preferably, the refractive index of the material forming layers 760a and 760b is 0.05 or greater, or 0.10 or less, than the refractive index of the material forming waveguides 670, 680, and 690. Advantageously, lower refractive index layers 760a, 760b may function as cladding layers that facilitate total internal reflection (TIR) ​​of light through waveguides 670, 680, 690 (e.g., TIR between the upper and lower main surfaces of each waveguide). In some embodiments, layers 760a, 760b are formed from air. It should be understood that the upper and lower parts of the illustrated set 660 waveguides may also include immediate cladding layers, although these are not shown.

[0176] Preferably, to facilitate manufacturing and other considerations, the materials forming waveguides 670, 680, and 690 are similar or identical, and the materials forming layers 760a and 760b are similar or identical. In some embodiments, the materials forming waveguides 670, 680, and 690 may differ between one or more waveguides, and / or the materials forming layers 760a and 760b may differ, while still maintaining the various refractive index relationships described above.

[0177] Continuing to refer to Figure 9A, rays 770, 780, and 790 are incident on the waveguide set 660. It should be understood that rays 770, 780, and 790 may also be introduced into waveguides 670, 680, and 690 by one or more image input devices 360, 370, 380, 390, and 400 (Figure 6).

[0178] In some embodiments, the rays 770, 780, and 790 may have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. The internal coupling optical elements 700, 710, and 720 each deflect the incident light so that the light propagates through one of the waveguides 670, 680, and 690 by TIR. In some embodiments, the internal coupling optical elements 700, 710, and 720 each selectively deflect one or more specific wavelengths of light while allowing other wavelengths to pass through the lower waveguide and associated internal coupling optical elements.

[0179] For example, the internally coupled optical element 700 may be configured to transmit rays 780 and 790 having different second and third wavelengths or wavelength ranges, while deflecting a ray 770 having a first wavelength or wavelength range. The transmitted ray 780 collides with an internally coupled optical element 710 configured to deflect light of the second wavelength or wavelength range, and is thereby deflected. The ray 790 is deflected by an internally coupled optical element 720 configured to selectively deflect light of a third wavelength or wavelength range.

[0180] Continuing with Figure 9A, the deflected rays 770, 780, and 790 are deflected so that they propagate through the corresponding waveguides 670, 680, and 690. That is, the internal coupling optical elements 700, 710, and 720 of each waveguide deflect the light into its corresponding waveguide 670, 680, and 690, and internally couple the light into the corresponding waveguide. The rays 770, 780, and 790 are deflected at an angle that causes the light to propagate through the individual waveguides 670, 680, and 690 by TIR. The rays 770, 780, and 790 propagate through the individual waveguides 670, 680, and 690 by TIR until they collide with the corresponding optical dispersion elements 730, 740, and 750 of the waveguide.

[0181] Referring now to Figure 9B, a perspective view of an embodiment of the multiple stacked waveguides shown in Figure 9A is illustrated. As previously mentioned, the internally coupled rays 770, 780, and 790 are deflected by the internally coupled optical elements 700, 710, and 720, respectively, and then propagate by TIR within waveguides 670, 680, and 690, respectively. The rays 770, 780, and 790 then collide with the optical dispersion elements 730, 740, and 750, respectively. The optical dispersion elements 730, 740, and 750 deflect the rays 770, 780, and 790 so that they propagate toward the externally coupled optical elements 800, 810, and 820, respectively.

[0182] In some embodiments, the light dispersion elements 730, 740, and 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse light to the external coupling optical elements 800, 810, and 820, and in some embodiments, they can also increase the beam or spot size of the light as it propagates to the external coupling optical elements. In some embodiments, the light dispersion elements 730, 740, and 750 may be omitted, and the internal coupling optical elements 700, 710, and 720 may be configured to deflect light directly to the external coupling optical elements 800, 810, and 820. For example, referring to Figure 9A, the light dispersion elements 730, 740, and 750 may be replaced by the external coupling optical elements 800, 810, and 820, respectively. In some embodiments, the external coupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light towards the viewer's eye 210 (Figure 7). It should be understood that the OPEs may be configured to increase the dimensions of the eyebox along at least one axis, and the EPEs may increase the eyebox along axes that intersect with the axes of the OPEs, for example, orthogonal axes. For example, each OPE may be configured to redirect a portion of the light impacting the OPE to an EPE in the same waveguide, while allowing the rest of the light to continue propagating along the waveguide. In response to the impact on the OPE, another portion of the remaining light is again redirected to the EPE, and the rest of that portion continues to propagate further along the waveguide, etc. Similarly, in response to the impact on the EPE, a portion of the impacting light is directed out of the waveguide towards the user, and the rest of that light continues to propagate through the waveguide until it impacts the EP again, at which point another portion of the impacting light is directed out of the waveguide, and so on. As a result, the internally coupled single beam of light is "duplicated" each time a portion of its light is redirected by the OPE or EPE, thereby forming a cloned beam field of light, as shown in Figure 6. In some embodiments, the OPE and / or EPE may be configured to modify the size of the beam of light.

[0183] Therefore, referring to Figures 9A and 9B, in some embodiments, the set of waveguides 660 includes, for each primary color, waveguides 670, 680, 690, internally coupled optical elements 700, 710, 720, optical dispersion elements (e.g., OPE) 730, 740, 750, and externally coupled optical elements (e.g., EP) 800, 810, 820. Waveguides 670, 680, 690 may be stacked with air gaps / cladding layers between each one. The internally coupled optical elements 700, 710, 720 redirect or deflect the incident light into their waveguides (using different internally coupled optical elements that receive light of different wavelengths). The light then propagates within the individual waveguides 670, 680, 690 at angles that will result in a TIR. In the embodiment shown, a ray 770 (e.g., blue light) is polarized by the first internal coupling optical element 700 in the manner described above, and then continues to bounce along the waveguide, interacting with the optical dispersion element (e.g., OPE) 730 and then the external coupling optical element (e.g., EP) 800. Rays 780 and 790 (e.g., green and red light, respectively) pass through waveguide 670, with ray 780 colliding with the internal coupling optical element 710, thereby being deflected. Ray 780 will then bounce along waveguide 680 via TIR, proceeding to its optical dispersion element (e.g., OPE) 740 and then the external coupling optical element (e.g., EP) 810. Finally, ray 790 (e.g., red light) passes through waveguide 690 and colliding with the optical internal coupling optical element 720 of waveguide 690. The internal optical coupling element 720 deflects the ray 790 so that it propagates by TIR to the optical dispersion element (e.g., OPE) 750 and then to the external coupling element (e.g., EP) 820. The external coupling element 820 then finally externally couples the ray 790 to the viewer, who also receives externally coupled light from the other waveguides 670, 680.

[0184] Figure 9C illustrates upper and lower plan views of embodiments of the multiple stacked waveguides shown in Figures 9A and 9B. As shown, waveguides 670, 680, and 690 may be vertically aligned with their associated optical dispersion elements 730, 740, and 750 and associated external coupling optical elements 800, 810, and 820. However, as discussed herein, the internal coupling optical elements 700, 710, and 720 are not vertically aligned. Rather, the internal coupling optical elements are preferably non-overlapping (e.g., laterally spaced, as seen in the upper and lower figures). As further discussed herein, this non-overlapping spatial arrangement facilitates the ingress of light from different resources into different waveguides on a one-to-one basis, thereby enabling a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, arrangements including non-overlapping, spatially separated internal coupling optical elements may be referred to as pupil-shifting systems, where the internal coupling optical elements in these arrangements may correspond to subpupils.

[0185] Figure 9D illustrates an embodiment of a wearable display system 60 in which various waveguides and associated systems disclosed herein may be integrated. In some embodiments, the display system 60 is the system 250 of Figure 6, which graphically illustrates some parts of the system 60 in more detail. For example, the waveguide assembly 260 of Figure 6 may be part of the display 70.

[0186] Continuing with reference to Figure 9D, the display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functions of the display 70. The display 70 may be coupled to a frame 80, which is wearable by the display system user or viewer 90 and configured to position the display 70 in front of the user 90's eyes. In some embodiments, the display 70 may be considered an eyepiece. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the user 90's ear canal (in some embodiments, another speaker, not shown, may also be optionally positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display system 60 also includes one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones may be configured to allow the user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.) and / or enable audio communication with other persons (e.g., other users of a similar display system). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system 60 may further include one or more outward-facing environmental sensors 112 configured to detect light, objects, stimuli, people, animals, places, or other aspects of the world around the user. For example, the environmental sensors 112 may include one or more cameras, which may be positioned to face outward to capture images similar to, for example, at least a portion of the user 90's normal field of view. In some embodiments, the display system may also include a peripheral sensor 120a, which is separate from the frame 80 and may be mounted on the user 90's body (e.g., the user 90's head, torso, limbs, etc.). In some embodiments, the peripheral sensor 120a may be configured to obtain data characterizing the user 90's physiological state.For example, the sensor 120a may be an electrode.

[0187] Continuing to refer to Figure 9D, the display 70 is operably coupled to the local data processing module 140 by a communication link 130, such as a wired cable or wireless connectivity, which may be mounted in various configurations, such as being fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise detachably attached to the user 90 (e.g., in a backpack configuration, in a belt-mounted configuration). Similarly, the sensor 120a may be operably coupled to the local data processing module 140 by a communication link 120b, such as a wired cable or wireless connectivity. The local processing and data module 140 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be used to assist in data processing, caching, and storage. Optionally, the local processing and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include a) data captured from sensors (image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein (e.g., operably coupled to frame 80 or otherwise attached to user 90)) and / or b) data acquired and / or processed using the remote processing module 150 and / or remote data repository 160 (including data related to virtual content) for passing to display 70 after processing or reading, as possible. The local processing and data module 140 may be operably coupled to the remote processing module 150 and the remote data repository 160 by communication links 170, 180, such as via wired or wireless communication links, so that these remote modules 150, 160 are operably coupled to each other and available as resources to the local processing and data module 140.In some embodiments, the local processing and data module 140 may include one or more of the following: an image acquisition device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted on the frame 80 or may be a standalone structure communicating with the local processing and data module 140 via a wired or wireless communication path.

[0188] Continuing to refer to Figure 9D, in some embodiments, the remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, the remote data repository 160 may comprise digital data storage facilities that may be available through the internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 160 may comprise one or more remote servers that provide information, for example, augmented reality content, for generating to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored, and all calculations are performed within the local processing and data module, enabling fully autonomous use from the remote module. Optionally, an external system (e.g., one or more processors, one or more computer systems), including a CPU, GPU, etc., may perform at least part of the processing (e.g., generating image information, processing data) and provide information to modules 140, 150, and 160, and receive information from them, for example, via a wireless or wired connection. I. Quality adjustment based on depth information

[0189] As described herein, in various embodiments, a display system (e.g., an augmented reality display system such as display system 60 in Figure 9D) may determine the user's three-dimensional fixation point, for example, by monitoring the user's eyes. The fixation point may indicate the location of a point in space along (1) the x-axis (e.g., the lateral axis), (2) the y-axis (e.g., the vertical axis), and (3) the z-axis (e.g., the depth of the point, e.g., the depth from the user). In some embodiments, the display system may use cameras, sensors, etc., to monitor the user's eyes (e.g., the pupil, cornea, etc. of each eye) and determine the line of sight of each eye. The line of sight of each eye can generally be understood as a vector extending from the center of the retina of that eye through the lens of the eye. For example, the vector can generally extend from the center of the macula (e.g., the fovea) through the lens of the eye. The display system may be configured to determine the location where the vectors associated with the eyes intersect, and this intersection can be understood as the fixation point of the eye. In other words, the fixation point can be a location in three-dimensional space where the user's eyes are converging. In some embodiments, the display system may filter out slight eye movements during, for example, high-speed movements (e.g., saccades, microsaccades) and update the fixation point in response to determining that the eyes are fixated on a location in three-dimensional space. For example, the display system may be configured to ignore eye movements that are fixated on a point for less than a threshold duration.

[0190] The resolution of content presented by a display system, such as virtual objects or content, may be adjusted based on its proximity to a fixed point, as discussed herein. It should be understood that the display system may store in it information about the location of virtual objects in three-dimensional space, or may have access to such information. The proximity of a given virtual object to a fixed point may be determined based on the known location of the virtual object. For example, the proximity of a virtual object to a fixed point may be determined by determining one or more of the following: (1) the three-dimensional distance of the virtual object from the user's fixed point; (2) the resolution adjustment zone in which the virtual object is located relative to the resolution adjustment zone in which the fixed point is located, if the display frustum of the display system is divided into resolution adjustment zones; or (3) the angular separation between the virtual object and the user's line of sight. Virtual content closer to the fixed point may be presented at a higher resolution than content further away from the fixed point. In some embodiments, the resolution of virtual content varies depending on its proximity to the depth plane in which the virtual content is placed relative to the fixed point, or to the depth plane in which the fixed point is placed. In some embodiments, adjustments to the resolution may be performed by a rendering engine, such as a rendering engine included in one or more graphics processing units, for example, one or more of modules 140, 150 (Figure 9D).

[0191] Figure 10A illustrates an embodiment of an up-and-down representation of a user viewing content presented by a display system (e.g., display system 60, Figure 9D) (e.g., content contained within a display frustum 1004). The representation includes the user's eyes 210, 220 and the determination of the fixation points 1006 of eyes 210, 220. As illustrated, the line of sight of each eye is represented as a vector (e.g., vectors 1003A, 1003B), and the display system detects the fixation points 1006 by determining, for example, where those vectors converge in front of eyes 210, 22. In the illustrated embodiment, the fixation points 1006 coincide with the location of a first virtual object 1008A presented by the display system. Embodiments of systems and methods for eye tracking can be found in U.S. Patent Application No. 14 / 690,401, filed April 18, 2015 (incorporated by reference for all purposes) and the attached appendices. For example, eye-tracking systems and methods are illustrated at least in Figure 25-27 of the Appendix and can be used, at least in part, for eye tracking and / or for determining a fixation point, as described herein.

[0192] Continuing with Figure 10A, the second virtual object 1008B is also presented by the display system within the display frustum 1004. Views of these virtual objects 1008A, 1008B as seen by a viewer are shown in the rendered frame 1010. The rendered frame 1010 may include the first virtual object 1008A rendered at a first resolution, while the second virtual object 1008B, located away from the fixed point 1006, is rendered at a second, lower resolution. Specifically, it may be determined that the second virtual object 1008B is located at a greater depth than the first virtual object 1008A and positioned toward its side. For example, the display system may determine the depth of the second virtual object 1008B as discussed herein, or optionally, a content provider associated with virtual content may indicate the depth of the virtual object that the display system can use to render its virtual object. Therefore, as described above, the fixed point 1006 describes a three-dimensional location in the space viewed by the user, and it can be determined that the second virtual object 1008B is located at a greater depth from the user and is displaced laterally from the fixed point 1006.

[0193] While not limited by theory, when the user's eyes 210, 220 are viewing the first virtual object 1008A, the image of the first virtual object 1008A may enter the user's fovea, while the image of the second virtual object 1008B may not. As a result, the resolution of the second virtual object 1008B may be reduced without a significant impact on the perceived image quality of the display system, due to the lower sensitivity of the human visual system to the second virtual object 1008B. In addition, the lower resolution advantageously reduces the computational load required to provide the image. As discussed herein, the resolution at which the second virtual object 1008B is rendered may be based on its proximity to the fixation point 1006, and the reduction in resolution (e.g., relative to the resolution of the first virtual object 1008A) may increase with decreasing proximity (or increasing distance) between the fixation point 1006 and the virtual object 1008A. In some embodiments, the rate of resolution reduction may correspond to the rate of decrease in cone density within the human eye or the decrease in visual acuity as it moves away from the fovea.

[0194] It should be understood that the resolution of various virtual objects presented by the display system can change dynamically as the fixation point changes location. For example, Figure 10B illustrates another embodiment of the representation of a user viewing content presented by the display system. As shown in Figure 10B, the user is now focused on the second virtual object 1008B, compared to Figure 10A where the user was focused on the first virtual object 1008A. By monitoring the user's lines of sight 1003A, 1003B, the display system determines that the eyes 210, 220 are converging on the second virtual object 1008B and sets that location as the new fixation point 1006.

[0195] In response to detecting a real change in the location of the fixed point 1006, the display system renders the second virtual object 1008B at a higher resolution than the first virtual object 1008A, as shown here in the rendered frame 1010. Preferably, the display system monitors the user's line of sight 1003A, 1003B at a sufficiently high frequency and changes the resolution of the virtual objects sufficiently quickly so that the transition in resolution between the first virtual object 1008A and the second virtual object 1008B is substantially imperceptible to the user.

[0196] Figure 10C illustrates another embodiment of the representation of the upper and lower view of a user viewing content through a display system (e.g., display system 60, Figure 9D). In this embodiment, the user's field of view 1004 is shown together with a fixed point 1006. Three virtual objects are shown, with the first virtual object 1012A being closer to the fixed point 1006 than the second virtual object 1012B or the third virtual object 1012C. Similarly, the second virtual object 1012B is shown to be closer to the fixed point 1006 than the third virtual object 1012C. Therefore, when virtual objects 1012A-1012C are presented to the user, the display system may allocate resources such that the step of rendering the first virtual object 1012A is given a greater resource allocation than the second virtual object 1012B (for example, object 1012A is rendered at a higher resolution), and the second virtual object 1012B receives a greater resource allocation than the third virtual object 1012C. The third virtual object 1012C may optionally not be rendered at all because it is outside the field of view 1004.

[0197] A resolution adjustment zone is illustrated in an embodiment of Figure 10C, and the zone is elliptical (e.g., circular) as described along the depth and lateral axes. As illustrated, the fixation point 1006 is inside the central zone 1014A, and the first virtual object 1012A extends between zones 1014B and 1014C and within the cone 1004a of the user's foveal view. The first virtual object 1012A may therefore be presented to the user at the resolution associated with zone 1014B or 1014C, or optionally, a portion of the object 1012A in zone 1014B may be presented according to the resolution of zone 1014B, and the remainder in zone 1014C may be presented according to the resolution of zone 1014C. For example, in one embodiment where zones are assigned a resolution reduced from the maximum (e.g., highest) resolution, the first virtual object 1012A may be presented at the assigned resolution. Optionally, the first virtual object 1012A may be presented in either resolution (for example, the display system may be programmed to display it at the highest resolution associated with any zone across which the first virtual object 1012A extends) or a measured value of the central tendency of the resolution (for example, the measured value may be weighted according to the extent to which the object 1012A is located within zones 1014B, 1014C). Continuing to refer to Figure 10C, it should be understood that the resolution adjustment zones at different distances from the fixation point 1006 may have different shapes. For example, zone 1014C may have a different shape from zones 1014A-1014C and conform to the contour of the field of view 1004. In some other embodiments, one or more of zones 1014A-1014C may have a different shape from one or more others of zones 1014A-1014C.

[0198] Figure 10D is a block diagram of an exemplary display system. The exemplary display system (e.g., display system 60, Figure 9D) may be an augmented reality display system and / or a mixed reality display system which can adjust the use of rendering hardware resources according to the user's fixation point, as described herein. For example, as described above with respect to Figure 10C, the rendering hardware resource 1021 can be adjusted according to the user's fixation point. A resource arbiter 1020 may be implemented to adjust the use of such resource 1021, for example, the arbiter 1020 may allocate the resource 1021 to a particular application process 1022 associated with the step of presenting a virtual object to the user. The resource arbiter 1020 and / or the rendering hardware resource 1021 may optionally be contained within the local processing and data module 140 (e.g., as shown in Figure 9D) and / or teleprocessing module 150 of the display system 60. For example, the rendering hardware resource 1021 may include a graphics processing unit (GPU), which may be contained within module 140 and / or module 150, as described above with respect to Figure 9D.

[0199] As an example of adjusting resource 1021, with respect to Figure 10C, a first virtual object 1012A associated with a first application process can be allocated more resource 1021 than a second virtual object 1012B associated with a second application process. A virtual object associated with application process 1022 can be contained in a frame buffer 1024, which will be rendered based on the allocated resource 1021 and synthesized into a final frame buffer 1028 (e.g., by a synthesizer 1026). The final frame buffer 1028 can then be presented by display hardware 1030, for example, a display 70 illustrated in Figure 9D, and the rendered virtual object will have its resolution adjusted.

[0200] As disclosed herein, the resolution of a virtual object may be determined based on the proximity of the virtual object to a fixed point. In some embodiments, the resolution may be modified as a function of the distance between the virtual object and the fixed point. In some embodiments, the modification may occur in discrete steps; that is, similar modifications may be applied to all virtual objects located within a particular volume or zone. Figure 11A1 illustrates an embodiment of an upper-lower diagram representation of the adjustment in resolution within different resolution adjustment zones based on three-dimensional fixed point tracking. The display system may divide the display frustum into a plurality of volume or resolution adjustment zones and modify the resolution in discrete steps corresponding to these zones. Thus, in some embodiments, to determine the adjustment in resolution of virtual content, the display system may utilize information describing the volume of space (hereinafter referred to as the resolution adjustment zone) and the assignment of resolution adjustment to the volume of each space. As illustrated, the field of view provided by the display system (e.g., the display frustum of the display) is separated into a plurality of different zones, each encompassing a range of depth from the user (e.g., depth range 1102A–1102E). In some embodiments, each depth range 1102A–1102E has a single associated depth plane that can be presented by the display system. Continuing with reference to Figure 11A1, five zones encompass each identified depth range from the user and are continuous along the lateral direction. In the illustrated exemplary upper and lower diagrams, the field of view is divided into a grid 1100 of 25 zones. Each zone represents a volume of real-world space where virtual content can be placed for the user.

[0201] It should also be understood that the zones may extend in this vertical direction (e.g., along the y-axis, not shown) so that the illustrated grid 1100 can be understood to represent one cross-section along the vertical direction. In some embodiments, multiple zones are also provided in the vertical direction. For example, there may be five vertical zones per depth range, i.e., a total of 125 resolution adjustment zones. An embodiment of such zones extending in three dimensions is illustrated in Figure 11B and described below.

[0202] Continuing to refer to Figure 11A1, the user's eyes 210, 220 fixate on a specific fixation point 1006 within the grid 1100. The display system may determine the location of the fixation point 1006 and the zone in which the fixation point 1006 is located. The display system may adjust the resolution of the content based on the proximity of the virtual content to the fixation point 1006, which may include the step of determining the proximity of the virtual content to the zone in which the fixation point 1006 is located. In one embodiment, the resolution for content contained within the zone in which the fixation point 1006 is located may be set to a specific polygon count, in this embodiment, 10,000 polygons. Content contained within the remaining zones may be adjusted as appropriate based on the distance from the fixation point 1006. For example, content contained within zones adjacent to the zone containing the fixation point 1006 may be rendered at a lower resolution (e.g., 1,000 polygons). The embodiment in Figure 11A1 illustrates, as described herein, the step of adjusting the polygon count, but the step of adjusting the resolution may include steps of making other modifications to the resolution of the presented content. For example, adjustments in resolution may include one or more of the following: adjusting the polygon count; adjusting the primitives used to generate the virtual object (e.g., adjusting the shape of the primitives, e.g., adjusting the primitives from a triangular mesh to a quadrilateral mesh); adjusting the actions performed on the virtual object (e.g., shader actions); adjusting texture information; adjusting the color resolution or depth; adjusting the number of rendering cycles or the frame rate; and adjusting the quality at one or more points in the graphics pipeline of the graphics processing unit (GPU).

[0203] In addition, while the embodiment in Figure 11A1 provides a specific example of the difference in polygon count within different resolution adjustment zones, other absolute numbers of polygons and other rates of change in resolution with respect to distance from fixation point 1006 should also be considered. For example, the reduction in resolution from fixation point 1006 may be based on a symmetrical reduction rate with respect to depth and lateral distance from fixation point 1006, but other reduction relationships may also be utilized. For example, lateral distance from fixation point 1006 may be associated with a greater reduction in resolution with respect to depth distance from fixation point 1006. Furthermore, the size of each zone contained within the grid (e.g., the size of the spatial volume of the zone) may be different at will (e.g., zones may vary radially from the foveal axis). In some embodiments, the reduction may be continuous from fixation point 1006 so that resolution relationships with discrete zones or zones containing fixation point 1006 with assigned resolutions are not utilized. For example, a descent from a fixed point 1006 to a specific zone 1108 (e.g., a zone where the content is rendered at a resolution of 100 polygons) may be modified to be a continuous descent from fixed point 1006 to the edge of the grid (e.g., the edge of the specific zone 1108). It should be understood that each of the above considerations also applies to zones that extend vertically.

[0204] In some embodiments, the number and size of zones included in the grid may be based on confidence associated with the user's determination of fixation point 1006. For example, confidence may be based on the amount of time the user's eye has fixed on fixation point 1006, with shorter durations associated with lower confidence. For example, the display system may monitor the user's eye at a specific sampling rate (e.g., 30Hz, 60Hz, 120Hz, 1kHz), and the confidence at fixation point 1006 may be increased as continuous samples indicate that the user has generally maintained fixation point 1006. Optionally, a specific threshold for fixation may be used; for example, fixation on the same or similar fixation point over a specific duration (e.g., 100–300 milliseconds) may be associated with high confidence, while fixation for less than a specific duration may be associated with lower confidence. Similarly, intraocular fluctuations such as pupillary dilation, which may affect the user's determination of fixation point, may reduce the display system's confidence. It should be understood that the display system may monitor the eyes using sensors such as a camera imaging device (e.g., camera assembly 630, Figure 6). Optionally, the display system may use a combination of sensors to determine the user's line of sight (e.g., different line of sight determination processes may be used, such as an infrared sensor used to detect infrared reflection from the eye and identify the pupil, or a visible light imaging device used to detect the iris). The display system may increase its confidence when multiple line of sight determination processes coincide, and decrease its confidence level when they do not coincide. Similarly, with respect to a display system that performs a line of sight determination process for only one eye, each line of sight determination process may be associated with a specific confidence level (e.g., one determination process may be considered more accurate than the other), and the size of the resolution adjustment zone may be selected, at least partially, based on the implemented processes.

[0205] In some embodiments, the display system may increase or decrease the number of zones with each update of the fixed point 1006. For example, more zones may be utilized as the confidence associated with the fixed point 1006 increases, and fewer zones may be utilized as the confidence decreases. Figure 11A2 illustrates an embodiment of the upper and lower representation of the resolution adjustment zones at different times as the size and number of zones change. At time t=1, the user's field of view may be divided into the initial set of zones, as seen in the upper and lower figures. At time t=2, the confidence at the location of the fixed point 1006 increases, and the display system may also decrease the size of the zone occupied by the fixed point 1006 and rendered at high resolution. Optionally, the size of other zones may also decrease, as shown. At time t=3, the confidence at the location of the fixed point 1006 decreases, and the display system may also increase the size of the zone occupied by the fixed point 1006 and rendered at high resolution. Optionally, the size of other zones may also increase, as shown. It should be understood that multiple zones may also extend along the y-axis, and similar increases or decreases in the size and number of zones may also be defined along that axis. For example, the size of a zone extending perpendicularly along the y-axis may decrease with increasing confidence, while the size may increase with decreasing confidence. Optionally, the display system may determine the confidence of the fixation point 1006 for each frame presented to the user by the display system, where t=1, t=2, and t=3 may represent different frames. Allocating more zones may require increased computational power (for example, the display system may need to adjust the resolution of more content, identify zones containing content, etc.), so the display system may balance the increase in computational power required by increasing the number of zones with the savings in computational power resulting from a potential decrease in content resolution.

[0206] Referring again to Figure 11A1, the grid can change dynamically in the sense that the fixation point 1006 can be set to be located at the center of the grid (e.g., the centroid). Thus, the display system can avoid the edge case where the fixation point 1006 is determined to be located on a vertex of the grid. For example, as the user's eyes rotate and then fixate on different three-dimensional locations in space, the grid may also move along with the user's line of sight.

[0207] Figures 11B-11E illustrate embodiments of various resolution adjustment zone configurations. Additional shapes and configurations of resolution adjustment zones not shown may be used, and the embodiments should not be considered exhaustive. In addition, in some drawings, user eyes 210, 220 may be shown separated from the various resolution adjustment zones for ease of illustration and clarity. In all these drawings, it should be understood that eyes 210, 220 may be located at or within the boundaries of the zones (see, for example, Figure 11A1).

[0208] Figure 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone in Figure 11A1. Figure 11A1 can be understood as illustrating a cross-sectional view obtained along the plane 11A1-11A1 of the three-dimensional representation in Figure 11B, and it should be understood that Figure 11B omits some of the resolution adjustment zones in Figure 11A1 for clarity of the illustration. Continuing to refer to Figure 11A1, the field of view provided by the display system is divided into 27 zones. That is, the field of view is divided into three depth ranges 1102B-1102D, and within each depth range, there is a 3x3 grid zone extending laterally and vertically within the depth range.

[0209] The determined fixation point 1006 is illustrated to be located within a zone situated at the center of the field of view. Virtual objects located within zones outside the zone containing fixation point 1006 may have reduced resolution according to their distance from the fixation point 1006 zone, as discussed herein. Since the zones extend laterally and vertically, the reduction in resolution can occur based on the distance along the lateral, vertical, and depth axes (x, y, and z-axis, respectively) from the fixation point's resolution adjustment zone. For example, in some embodiments, virtual objects located in zone 1108 may have reduced resolution according to their lateral distance, as shown in Figure 11A1 (for example, zone 1108 may include the same vertical portion of the user's field of view as the zone containing fixation point 1006 and lie on the same depth plane).

[0210] Similarly to the above, and similar to the zones described in Figures 11C-11E below, the user's fixation point can be optionally maintained at the center of the zone (e.g., the centroid), or the zone can be fixed relative to the user's field of view, with the user's fixation point located anywhere within the zone.

[0211] Figure 11C illustrates another embodiment of the configuration for resolution adjustment zones. In the embodiment, the field of view provided by the display system is illustrated to be separated into elliptical zones, each encompassing a specific three-dimensional volume of space. Similar to Figure 11A1, each zone (e.g., zones 1112A-112D) extends along the lateral and depth dimensions. In some embodiments, each zone also extends to encompass at least a portion of the user's vertical field of view. The fixation point 1006 is illustrated to be at the center of the zone (e.g., within zone 1112A). Virtual objects located in zones outside zone 1112A may have their resolution reduced according to their distance from zone 1112A, for example, according to techniques described herein. For example, each zone outside zone 1112A may be assigned a specific resolution, or a reduction rate may be used to determine the reduction in resolution. Zone 1112D is illustrated as the zone furthest from zone 1110A, and the reduction in resolution may be greatest within zone 1112D.

[0212] Figure 11D illustrates an embodiment of the three-dimensional representation of the resolution adjustment zone in Figure 11C, and Figure 11C shows a cross-sectional view obtained along the plane 11C-11C. In this embodiment, the field of view provided by the display system is illustrated to be separated into ellipsoidal zones, each encompassing the three-dimensional volume of space. The user's fixation point 1006 is illustrated as the centroid of the user's field of view and is located within zone 1112A. Optionally, Figure 11D may represent the state in which each ellipsoid in Figure 11C has been transformed into an ellipsoid. In some embodiments, the size of zone 1112A in Figure 11C along the depth and lateral directions can define the size of the principal axes of zone 1112A in Figure 11D along the X and Z axes. The various zones may form concentric spheres or ellipsoids.

[0213] Figure 11E illustrates another embodiment of the three-dimensional representation of the resolution adjustment zone in Figure 11C, where Figure 11C shows a cross-sectional view obtained along plane 11C-11C. The field of view provided by the display system is illustrated to be separated into stacked levels of similar concentric zones. For example, Figure 11E may represent how the ellipse in Figure 11C extends along the vertical direction to form a cylinder. The cylinder can then be separated vertically such that each cylinder encompasses a portion of the user's vertical field of view. Thus, Figure 11E illustrates nine zones of the cylinder. Each zone, in addition, excludes any internal zones (for example, ellipsoid 1112B would encompass a volume of space, excluding the volume of space encompassed by ellipsoid 1112A). In the embodiment, the fixation point 1006 is illustrated to be within the central zone 1110A, and virtual objects located outside the central zone 1110A can have their resolution reduced according to the techniques described herein.

[0214] Figure 12A illustrates a flowchart of an exemplary process 1200 for adjusting the resolution of content according to its proximity to a three-dimensional fixed point. For convenience, the process 1200 may be described as being carried out by a display system (for example, a wearable display system 60, which may include processing hardware and software and optionally provide information to one or more computers or other external processing systems, for example, by offloading processing to an external system and receiving information from the external system).

[0215] In block 1202, the display system determines the user's three-dimensional fixation point. As described above, the display system may include sensors that monitor information associated with the user's eyes (e.g., eye orientation). A non-exclusive list of sensors includes infrared sensors, ultraviolet sensors, and visible wavelength light sensors. The sensors may optionally emit infrared, ultraviolet, and / or visible light onto the user's eyes and determine the reflection of the emitted light from the user's eyes. In an embodiment, infrared light may be emitted by an infrared light emitter and an infrared light sensor. It should be understood that the sensor, which may include a light emitter, may correspond to the imaging device 630 in Figure 6.

[0216] The display system may use sensors to determine the line of sight associated with each eye (e.g., a vector extending from the user's eye, such as extending from the fovea through the lens of the eye) and the intersection of each eye's line of sight. For example, the display system may output infrared light onto the user's eye, and the reflection from the eye (e.g., corneal reflection) may be monitored. A vector between the pupil center of the eye (e.g., the display system may determine the centroid of the pupil, for example, through infrared imaging) and the reflection from the eye may be used to determine the line of sight of the eye. The intersection of the line of sight may be determined and assigned as a three-dimensional fixation point. The fixation point can therefore indicate the location where the content will be rendered at full or maximum resolution. For example, based on the determined line of sight, the display system may triangulate the three-dimensional location in space where the user is fixating. Optionally, when determining the fixation point, the display system may utilize orientation information associated with the display system (e.g., information describing the orientation of the display system in three-dimensional space).

[0217] In block 1204, the display system acquires location information associated with content that is being or will be presented to the user by the display system. Prior to rendering the content for presentation to the user (for example, via the waveguide output as described above), the display system may acquire location information associated with the content that will be presented to the user. For example, as described above, virtual content may be presented to the user in such a way that it appears as if it is located in the real world (for example, the content may be located at different depths within the user's field of view). It should be understood that the display system may include, or have access to, a three-dimensional map of the surrounding environment that can inform the location of any virtual content in the surrounding environment. Referring to this map, the display system may access and provide information that defines the three-dimensional location of virtual content within the user's field of view (for example, its location within the display frustum as illustrated in Figures 10A-10B).

[0218] In block 1206, the display system adjusts the resolution of the virtual content that will be displayed to the user. The display system adjusts the resolution of the content based on its proximity to a three-dimensional fixed point. For example, a rendering engine, such as a rendering engine implemented by a processing device (e.g., a central processing unit, a graphics processing unit) that renders the content for presentation to the user, may adjust the resources invested when rendering the content (for example, the rendering engine may adjust the resolution of the content).

[0219] The display system may determine the distance in three-dimensional space between the content to be presented to the user and the user's fixation point, and may reduce the resolution of the content based on the determined distance. The reduction may be determined according to a rate of reduction, for example, a continuous function that correlates the distance with the content resolution, and the display system may obtain the resolution based on the continuous function and render the content. Optionally, the display system may determine the distance from the centroid of the content to the fixation point, and may render the content at a certain resolution based on the distance. Optionally, the display system may render parts of the same content at different resolutions according to the distances of various parts to the fixation point (for example, the display system may separate the content into parts and render the farther parts at a reduced resolution compared to the closer parts).

[0220] In some embodiments, the display system may access information that can be used to separate the user's field of view (e.g., corresponding to a display frustum) into zones, each zone representing a volume of space that may contain content. The accessed information, for example, the grid illustrated in Figure 11A1, may indicate a specific resolution to be used when rendering the content that will be contained within each zone, with a three-dimensional fixation point set at the center of the grid. In addition, the grid may indicate a rate of decrease in resolution to be used when rendering the content. With respect to content contained within multiple zones (e.g., content located in three-dimensional space required by two zones), the display system may optionally adjust the resolution of the content to correspond to a single zone, or optionally adjust the resolution of the content according to the corresponding zone in which the portion is located.

[0221] When setting the content resolution, the display system renders content located at a fixed point (e.g., within the same zone as the fixed point) at full or maximum resolution. The maximum resolution may be based on the maximum value that the display system's hardware and / or software can render, ensuring that the content is presented to the user at a refresh rate above a threshold (e.g., 60Hz, 120Hz), and optionally, that the content is updated at a rate exceeding the convergence-divergence rate (e.g., above 60ms) and the distance adjustment time (e.g., 20ms-100ms), thereby reducing the perception of resolution changes. The display system may dynamically modify the maximum resolution based on the resources available to the display system, for example, prior to the display system rendering each frame. For example, as more content is presented to the user, the maximum resolution of the content may be reduced, ensuring that the display system can present frames of the rendered content at a threshold rate above the desired threshold rate to reduce the perception of resolution changes. The display system may optionally monitor the number of frames per second in which content is presented and adjust the maximum resolution and / or the resolution reduction rate based on the distance from the fixation point to ensure that the number of frames per second presented does not fall below a threshold rate. In one embodiment, the display system may render content such as a first virtual object located in the zone of the fixation point at maximum resolution. Instead of reducing the maximum resolution of the first virtual object to ensure that the number of frames per second remains above a certain threshold, the display system may dynamically increase the resolution reduction rate based on the distance. In this way, the display system may adjust the resolution assigned to each zone outside the zone of the fixation point.Optionally, the display system may set a minimum resolution that can be used in each zone outside the fixed point zone, and may adjust the maximum resolution if it would exceed the minimum resolution (for example, if the display system needs to reduce the content resolution below a minimum value to maintain a threshold rate, the display system may reduce the maximum resolution). Similarly, the display system may reduce the maximum resolution without reducing the content resolution in zones outside the fixed point zone. Optionally, the user of the display system may indicate whether they prefer that content located close to the fixed point be given preference over other content.

[0222] In some embodiments, as will be described in more detail below with respect to Figures 13-14, the display system may optionally adjust the resolution of the content by utilizing the angular proximity of the content to the user's line of sight. For example, if a particular piece of content is located outside the zone where the fixation point is situated, but within a threshold proximity of the user's line of sight, such that the content would fall into the fovea of ​​the user's eye, the display system may render the content at a higher resolution (e.g., the maximum resolution or a resolution higher than that shown in the grid illustrated in Figure 11A1). Optionally, the display system may reduce the resolution of a particular piece of content and apply a blurring process (e.g., Gaussian blur) to it. Thus, the content may be rendered at a lower resolution while being blurred, indicating that the content is, for example, further from the user than the fixation point. In addition, blurring can reduce the perception of lower resolution (e.g., blurring can reduce the perception of increased pixel size resulting from lower resolution).

[0223] Exemplary operations associated with the step of presenting virtual content are illustrated in Figures 12B–12C (e.g., rendering pipeline). In the embodiment of Figure 12B, a three-dimensional scene is presented to the user without any adjustment to the resolution, as described herein. In Figure 12C, the adjustment to the resolution is performed according to the fixed point information, as described herein. For example, one or more of the following adjustments may be performed: a step to reduce vertex behavior complexity, a step to reduce the level of detail tiling, a step to reduce geometric shape generation, a step to reduce pixel behavior complexity / aggregation of multiple pixels, etc. The adjustments may be performed advantageously at different steps in the pipeline for presenting virtual content, as illustrated, and may be optimized according to the specific software and / or hardware used to present the virtual content. It should be understood that the fidelity zone shown in Figure 12C is the resolution adjustment zone.

[0224] Referring again to Figure 12A, the display system presents the adjusted content to the user in block 1208. As described above, the display system adjusts the resolution of the content based on its proximity to a three-dimensional fixation point. The display system then presents the rendered content to the user at the associated location. In some embodiments, the display system may perform process 1200 for each frame of content to be rendered, or it may adjust the resolution of the content as the user adjusts its fixation point.

[0225] As described above, in some embodiments, the virtual objects may be within the user's line of sight while also being presented at different depths. Figure 13 illustrates an embodiment of a user representation in which multiple virtual objects are viewed in line with the user's line of sight. The exemplary representation includes the lines of sight 1003A, 1003B of the user's eyes 210, 220, along with the user's field of view (e.g., the display frustum 1004 of the display system), which are fixed to a fixation point on the first virtual object 1008A.

[0226] As illustrated, the second virtual object 1008B is within angular proximity of the user's line of sight (e.g., one or both of the line of sight vectors 1003A and 1003B) such that the second virtual object 1008B will enter the user's fovea (e.g., enter at least one fovea of ​​either eye). For example, depending on the rendering of frame 1110, the second virtual object 1008B is positioned behind the first virtual object 1008A (e.g., at a deeper perceived depth from there). It should be understood that the fovea is the part of the retina with the highest visual acuity. Because the second virtual object 1008B will enter the user's fovea, if the resolution of the second virtual object 1008B is reduced (e.g., reduced at least as described above with respect to Figure 11A1), the user may perceive a reduction in resolution. To avoid perceptible reduction in resolution, the display system may (1) render the second virtual object 1008B at the same resolution as the first virtual object 1008A, or within the threshold resolution of the first virtual object 1008A, and / or (2) render the second virtual object 1008B at a reduced resolution (e.g., as shown in Figure 11A1), and apply a blur to the second virtual object prior to presentation to the user. Although not limited by theory, the blur can mask the reduction in resolution while providing a depth cue.

[0227] Figure 14 is a flowchart of an embodiment of process 1400 for adjusting virtual content based on the angular distance from the user's line of sight. For convenience, process 1400 will be described as being performed by a display system (e.g., a wearable display system 60, which may include processing hardware and software, and optionally provide information to one or more external systems such as a computer or other processing unit, for example, offloading processing to an external system and receiving information from an external system). In the exemplary process 1400, the display system is a variable focus display system in which each frame is presented on the same depth plane and optionally convolves all content to be presented into a single frame buffer, that is, the variable focus display system presents virtual content on one depth plane at a time.

[0228] The display system determines the user's 3D fixation point (block 1402) and acquires location information associated with the presented content (block 1404). Blocks 1402 and 1404 may correspond to blocks 1202 and 1204 in Figure 12A, respectively. As described above with reference to Figure 12A, the display system monitors the user's eye movement (e.g., eye orientation) and determines the user's fixation point. The display system may acquire location information (e.g., within the next frame) of the content to be presented and subsequently adjust the resolution of the content.

[0229] Continuing with Figure 14, the display system determines (block 1406) content that will have reduced resolution and is located within a threshold angular distance from the user's line of sight. The display system identifies content that will have reduced resolution due to its proximity to the fixed viewpoint (e.g., the content is located at a depth greater than the fixed viewpoint) but will be within the user's fovea (e.g., within a threshold angular distance from the user's line of sight). Because the content will be within the user's fovea, the user may be able to perceive a reduction in resolution, as with the 3D fixed-point foveal rendering described herein. It should be understood that content block 1406 may include steps to perform the blocks illustrated in Figure 12C, in particular the blocks identified in section "GPU".

[0230] As a result, in block 1408, the display system may optionally render the determined content at a higher resolution. The display system may adjust the resolution of the determined content to be full resolution (e.g., the same resolution as content located at a fixed point or within the same zone or spatial volume as the fixed point) or to exceed a reduced resolution that would otherwise be assigned to the content (e.g., as described in block 1406).

[0231] In block 1410, the display system may optionally reduce the resolution of the content and blur the content prior to presenting it to the user. As described above, the variable focus display system may present the content to the user using a single display buffer. Since the variable focus display system presents all content on the same depth plane, the variable focus display system may output the content from, for example, a rendering engine using the same display buffer.

[0232] Optionally, the display system may use initial depth buffers to which each depth buffer is assigned one or more depth planes, or it may combine initial depth buffers to obtain a display buffer. Referring to the illustration in Figure 13, the first depth buffer may contain the first virtual object 1306, while the second depth buffer may contain the second virtual object 1308. The display system may then apply a blurring process to the second depth buffer or to specific content contained within the second depth buffer (for example, the display system may apply the blurring process to the second virtual content 1308, but not to other content that is on the same depth plane but located at a greater angular distance from the user's line of sight). After performing the blurring process, the display system may combine the first and second depth buffers (for example, the display system may add occlusion and remove, for example, portions of the second virtual object 1308 that are invisible due to occlusion by the first virtual object 1306) to obtain a display buffer.

[0233] An exemplary blurring process may include a step in which the display system performs a convolution of a blur-associated kernel (e.g., a Gaussian kernel, a circular kernel to reproduce a destructive effect, a box blur, etc.) on the content. In this way, a reduction in resolution can be masked, while the processing savings resulting from the reduction in resolution can be maintained. Optionally, the intensity associated with the blurring process (e.g., the degree to which the content is blurred) may be based on the difference in depth between the user's fixation point and the content and / or the angular proximity of the content to the user's line of sight. For example, the degree of blurring may increase with increasing proximity to the user's line of sight.

[0234] In some embodiments, the display system may utilize the features of block 1408 or 1410 according to the hardware and / or software of the display system. For example, certain hardware (e.g., a graphics processing unit) may be capable of performing the blurring process in-house without reaching a threshold for hardware performance. With respect to this particular hardware, the display system may be configured to reduce the resolution of the content and then blur the content. However, other hardware may be slow to perform the blurring process, and rendering the content at a higher resolution may result in higher performance. With respect to this other hardware, the display system may be configured to render the content at a higher resolution. Furthermore, the decision of whether to render the content at a higher resolution or at a lower resolution with blurring may depend on the type of content to be displayed. For example, the display system may be configured to render text at a higher resolution while rendering shapes at a lower resolution and blurring them.

[0235] Continuing to refer to Figure 14, in block 1412, the display system presents content to the user. The display system may present the adjusted content to the user from the same display buffer, for example, as described above. II. Resolution adjustment based on ambient light level

[0236] In addition to or as an alternative to a reduction in resolution along the z-axis, various other schemes for presenting virtual content with a reduction in resolution may be implemented in some embodiments. Advantageously, as described herein, some aspects of the virtual content may be presented at a relatively high resolution, while others may be presented at a relatively low resolution, which can reduce the computation and energy resource usage by the display system, while preferably having little effect on the perceived image quality of the virtual content.

[0237] Referring now to Figure 15, an embodiment of the representation of the user's eye's retina is illustrated. The illustrated figure shows the retina 1500 as it would appear when viewed with the head up, along the visual axis of the retina. The retina 1500 includes a fovea 1510 surrounded by a peripheral area 1530. Within the fovea 1510 are foveals 1520 that intersect the visual axis.

[0238] It should be understood that the retina contains two types of photoreceptors: rods and cones. In addition, the distribution of these photoreceptors across the retina is variable, providing different rod and cone densities across the retina.

[0239] Referring here to Figure 16, an embodiment of the retinal resolution 1500 and the rod and cone densities across it is schematically illustrated in Figure 15. The x-axis indicates the degree of eccentricity with respect to the point where the visual axis intersects the retina. Rightward on the page is towards the nose, and leftward on the page is towards the temple. As illustrated, the resolution of the human eye roughly correlates with the density of photoreceptors (rods and cones) in the retina. Consequently, in some embodiments, a reduction or taper in the resolution (e.g., spatial resolution) of virtual content on the x and y axes (e.g., on a given depth plane) can substantially follow a reduction across the retina of cone density, rod density, or the set of rod and cone densities. For example, the tendency for resolution to decrease as the user moves away from the fixation point across their field of vision may be within ±50%, ±30%, ±20%, or ±10% of the trend in the change in photoreceptor density (e.g., cone density, rod density, or a combination of rod and cone densities) across the corresponding portion of the retina. In some embodiments, the decrease in resolution as the user moves away from the fixation point is stepwise and substantially follows the change in density. In some other embodiments, the decrease in resolution may occur in steps (e.g., one step, two steps, etc.). For example, there may be two steps, i.e., a highest resolution region of the field of vision correlated with the fovea, a medium resolution region correlated with the fovea, and a lower resolution region correlated with the peripheral area.

[0240] Continuing to refer to Figure 16, it should be understood that different photoreceptors have different levels of activity under different lighting conditions, for example, at different ambient illumination levels. As a result, the reduction in resolution that follows the density of photoreceptors may not be noticeably perceptible to the user at some illumination levels, but may be perceptible at other illumination levels. Consequently, in some embodiments, the reduction in resolution of virtual content along the x, y, or z axis may be set by referring to external lighting conditions.

[0241] For example, the visual behavior of the eye can be divided into three modes based on light conditions. The three modes are photopic vision, mesopic vision, and scotopic vision. Photopic vision typically occurs under bright conditions, such as ambient light or illumination levels of about 10 to 10 8 cd / m 2 including about 3 cd / m 2 or more. In photopic vision, cones are mainly active. In scotopic vision, rods are mainly active. In mesopic vision, both rods and cones can be active. As used herein, ambient light conditions or illumination levels refer to the amount of light to which the user's eye and its retina are exposed.

[0242] Mesopic vision typically occurs under lower light conditions, such as illumination levels of about 10 -3 to 10 0.5 cd / m 2 . Both cones and rods are active at at least some illumination levels within mesopic vision, and the predominance of rods or cones changes over time depending on whether the ambient illumination level increases or decreases. As the eye adapts to a brighter environment, more cones are activated compared to rods, while as the eye adapts to a darker environment, more rods are activated compared to cones.

[0243] Scotopic vision typically occurs under light conditions where the illumination level is less than the illumination level for photopic vision. For example, scotopic vision occurs at about 1 to 10 -3 to 10 -6 cd / m 2 including about 10 -2 cd / m 2 or less or about 10 -3 cd / m 2Or it may occur at illumination levels below that. The rods are primarily active in scotopic vision. It should be understood that the illumination levels described herein with respect to photopic, crepuscular, and scotopic vision are examples. In some embodiments, the illumination levels associated with each type of vision may be arbitrarily assigned based on user preferences and / or customization for the group to which the user belongs (e.g., based on gender, age, ethnicity, presence of visual abnormalities, etc.).

[0244] In some embodiments, the type of vision active in the user (photopic, crepuscular, or scotopic) may be determined based on a measurement of the ambient light level. For example, the display system may be configured to measure the ambient light level using an optical sensor, such as an outward-facing camera 112 (Figure 9D). In some embodiments, the display system may communicate with another sensor or device that provides information regarding the ambient light level.

[0245] It should be understood that a head-mounted display system may block or attenuate some of the ambient light so that an outward-facing camera cannot provide a brightness level that accurately reflects the amount of light striking the eye. In addition, the display system may also be a light source that can modify the level of illumination to which the eye is exposed when projecting light onto the eye and providing virtual content. In some other embodiments, an inward-facing camera may be used to determine the brightness level. For example, the brightness level is roughly correlated with the pupil size. Figure 17 schematically illustrates an example of the relationship between pupil size and the amount of light incident on the user's eye. The x-axis represents a value related to brightness, and the y-axis represents a value related to the pupil area. As a result, the display system may be configured to determine the user's pupil area and then extrapolate brightness based on this pupil area. For example, the display system may be configured to use an inward-facing camera 500 (Figure 6) to capture an image of the user's eye 210, and then analyze the image to determine the pupil area or other metrics that indicate the pupil area (e.g., pupil diameter or width). For example, the area occupied by the pupil of eye 210 in the image captured by the camera may be determined and then corrected for any scaling factor produced by the camera's optical system. Advantageously, the step of determining the brightness level using the pupil area can effectively take into account both the reduction in ambient brightness level caused by the display, which blocks some ambient light, and the contribution to the brightness level by the light output of the display itself.

[0246] Continuing to refer to Figure 17, the display system may be configured to determine, based on the determined pupil area, whether the user's eye is in photopic, twilight, or scotopic mode. For example, the display system may have a table or other stored information in memory that defines the expected visual modes for a particular pupil area. In an embodiment, consistent with the graph shown in Figure 17, the display system is approximately 3 mm 2 A pupil area smaller than 3mm indicates photopic vision. 2Or even larger, up to approximately 38mm 2 The pupil area shows twilight vision, 38mm 2 A pupillary area exceeding a certain value may be categorized as indicating scotopic vision. These luminance values ​​and associated visual modes are examples, and it should be understood that other values ​​may be substituted. For example, different values ​​may be applied to different users in response to user input, or different values ​​may be applied based on specific categories into which the user may belong (e.g., gender, age, ethnicity, presence of visual abnormalities, etc.). Furthermore, it should be understood that the display system does not necessarily identify specific visual modes. Rather, the display system may simply be configured to associate a specific measured pupillary area with a specific resolution level or accommodation.

[0247] In some embodiments, inputs from both an inward-facing camera 510 (Figure 6) and an outward-facing camera 112 (Figure 9D) may be used to determine the luminance level. For example, the display system may be configured to take the average (including a weighted average) of the luminance levels determined using cameras 510 and 112. As described above, the luminance level determined using camera 510 may be extrapolated from the size of the pupil area of ​​the user's eye, based on imaging the user's eye using camera 510.

[0248] It should be understood that rods and cones have different levels of visual acuity and different sensitivities to color and contrast. As a result, differences in visual acuity and sensitivity to color and contrast exist at different ambient brightness levels, as ambient brightness levels affect whether rods and / or cones are active. Advantageously, differences in light levels in visual acuity and sensitivity to color and contrast may be applied to provide an additional basis for reducing resolution, which may be used in conjunction with fixation-based resolution changes, as described above (for example, with respect to Figures 12A and 14), or independently without specifically performing fixation-based resolution changes.

[0249] Referring here to Figure 18, a schematic diagram of an embodiment of process 1800 for adjusting virtual content based on the amount of light incident on the user's eyes is shown. For convenience, the process may be described as being carried out by a display system (for example, a wearable display system 60 (Figure 9D), which may include processing hardware and software and optionally provide information to one or more external systems such as computers or other processing units, for example, by offloading processing to an external system and receiving information from an external system).

[0250] In block 1810, the display system determines the amount of light reaching the retina. Preferably, this determination is an estimation of the amount of light reaching the retina, rather than a direct measurement of the light striking the retina. This estimation may be performed using methods disclosed for determining luminance levels, as discussed herein. For example, luminance levels may be assumed to correspond to the amount of light reaching the retina. Consequently, the step of determining the amount of light reaching the retina may include the step of determining the size of the user's pupil and / or the step of determining the ambient luminance level using a sensor configured to detect light, such as an outward-facing camera on the display device.

[0251] In block 1820, the display system adjusts the resolution of the virtual content to be presented to the user based on the amount of light found to reach the retina in block 1810. In some embodiments, the step of adjusting the resolution of the virtual content includes adjusting one or more of the spatial resolution, color depth, and light intensity resolution of the virtual content. It should be understood that the human visual system has maximum visual acuity and sensitivity to spatial resolution, color, and light intensity under photopic illumination levels. The ability to perceive differences in spatial resolution, color, and light intensity decreases under crepuscular illumination levels and further decreases under scotopic illumination levels.

[0252] As a result, in some embodiments, if the amount of light present is found to correspond to a photopic level, the virtual object may be rendered with full or high spatial resolution (compared to the spatial resolution that would be used for crepuscular or scotopic vision). If the amount of light present is found to correspond to a crepuscular level, the virtual object may be rendered with reduced spatial resolution compared to the spatial resolution used for virtual objects under photopic illumination levels. If the amount of light is found to correspond to a scotopic level, the virtual object may be rendered with a lower spatial resolution than that used under crepuscular or photopic illumination levels. Spatial resolution may be adjusted, for example, by reducing the number of polygons, as described herein.

[0253] Similarly, color depth or bit depth may be adjusted according to the illumination level, with the highest color depth used under photopic illumination levels, the mid-range color depth used under crepuscular illumination levels, and the lowest color depth used under scotopic illumination levels. Color depth may also be adjusted by changing the number of bits used per color component of a pixel, and it should be understood that fewer bits correspond to a lower color depth.

[0254] Similarly, although not limited by theory, it is thought that the number of gradations in light intensity increases as the illumination level progresses from photopic to crepuscular and scotopic illumination levels. In other words, it is thought that the human visual system is able to distinguish smaller differences in light intensity as the ambient illumination level decreases. In some embodiments, the display system may be configured to display fewer gradations in light intensity as the illumination level progresses from photopic to crepuscular and scotopic illumination levels. As a result, the maximum number of gradations in light intensity levels is presented under photopic illumination levels, fewer gradations are presented under crepuscular illumination levels, and even fewer gradations are presented under scotopic illumination levels.

[0255] In addition, in some embodiments, the display system may be able to provide gradations at a greater number of light intensities than those perceptible to the user. Examples of this are illustrated in Figures 22a-22c and further discussed below. For example, the display system may be able to display 256 different levels of intensity with respect to a given image pixel, but the user may be able to perceive only a smaller number of levels, e.g., 64 levels. In this instance, multiple possible light intensity levels are encompassed within a single perceptible light intensity level. For example, the display system may be able to display four different light intensity levels, but the user may perceive all four as similar. In such a situation where multiple possible light intensities are perceived as identical by the user, the display system may be configured to select the lowest intensity value for display from these values ​​that are perceived as similar. As a result, the display system may be able to utilize lower intensities, thereby reducing the amount of power used to illuminate the display and achieving the desired light intensity. This may have particular advantages in display systems where the individual pixels of the spatial light modulator themselves are light emitters such as organic and inorganic LEDs. In some embodiments, the number of gradations decreases with decreasing ambient light levels, and the display system is configured to group together a larger number of possible light intensity levels and display the lowest light intensity group.

[0256] It should be understood that, with respect to the virtual content to be displayed, one, two, or all three of the spatial resolution, color depth, and light intensity resolution may vary based on the light conditions directed towards the user (the amount of light reaching the user's retina). These adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made to the entire virtual content without any adjustments to resolution based on the distance from the fixation point of the user's eye, as disclosed herein. In some other embodiments, adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made in conjunction with adjustments to resolution based on distance from the fixation point (see, for example, Figures 12A and 14). In some embodiments, if the resolution decreases with distance from the fixation point, the profile of the decrease on a given plane (on the x and y axes) preferably matches the profile of the change in cone density across the corresponding portion of the retina.

[0257] In some embodiments, as described herein, adjustments to spatial resolution, color depth, and / or light intensity resolution are preferably tied to the mode of vision (photopic, crepuscular, or scotopic) that is active at a given time. These adjustments may change dynamically as the mode of vision changes. For example, as the user progresses from photopic to scotopic vision, the resolution may decrease, as discussed herein. Conversely, as the user progresses from scotopic to crepuscular vision, the resolution of the virtual content may increase. It should be understood that tying adjustments to resolution to a particular mode of vision does not require a specific determination that the user is in that particular mode. Rather, the display system may simply be configured to associate a particular resolution, regardless of spatial resolution, color depth, or light intensity resolution, with a particular range of ambient illumination levels or pupil size. In addition, the resolution adjustment is preferably linked to three levels of light conditions (corresponding to three modes of vision), as discussed herein, although in some embodiments the resolution adjustment may be linked to two levels of light conditions or more than three levels of light conditions.

[0258] Furthermore, it should be understood that resolution adjustment may occur in real time (e.g., as ambient light conditions change) or may be delayed over a set duration, allowing the human visual system to adapt to existing light conditions before resolution adjustment is performed on the virtual content. While not limited by theory, the human visual system is thought to require a certain time period to adapt to different illumination levels, and this time period is thought to increase as the illumination level decreases. As a result, in some embodiments, adjustment in resolution resulting from a change in illumination level does not occur until the user is exposed to a particular illumination level over a set duration (e.g., substantially continuously). For example, the set duration may be 5 minutes, 10 minutes, 15 minutes, or 20 minutes.

[0259] Continuing to refer to Figure 18, in block 1830, virtual content is presented to the user. This presentation of virtual content may be carried out as discussed herein, for example, in block 1208 of Figure 12A or block 1412 of Figure 14.

[0260] Referring here to Figure 19, an embodiment of the change in resolution detectable by the user's eye as the amount of light incident on the eye changes is schematically illustrated. This figure illustrates embodiments of the sensitivity of the human visual system to spatial resolution under different visual modes. Scotopic vision occurs in the low-light region 1910, twilight vision occurs in the medium-light region 1920, and photopic vision occurs in the bright-light region 1930. As shown, sensitivity to spatial resolution decreases substantially as the ambient illumination level decreases. In some embodiments, the adjustment to spatial resolution discussed above with respect to Figure 18 corresponds to the contour of the illustrated curve. For example, with respect to a given light level in photopic or scotopic vision mode, virtual content is rendered with sufficient spatial resolution to meet or exceed the resolution value shown on the y-axis.

[0261] Referring here to Figure 20, it should be understood that different photoreceptors may be used to perceive light of different wavelengths or colors. Figure 20 schematically illustrates an example of differences in the eye's sensitivity to different colors of light at different levels of illumination. The differences in duration on the x-axis reflect the amount of time typically required for the human visual system to adapt to a particular ambient illumination level so that a particular mode of vision is activated. It should be noted that at ambient illumination levels corresponding to some scotopic and crepuscular vision, the photoreceptor for red light may no longer be active, while the photoreceptor for blue light is active under the lowest light conditions. It should be understood that red, green, and blue light correspond to the colors most typically used as primary colors in a display system to form a full-color image (for example, as discussed herein with respect to Figures 8-9B). In some embodiments, the display system may be configured to vary the rendering of images of different colors depending on the ambient illumination level.

[0262] Referring here to Figure 21, a schematic diagram of an embodiment of process 2100 for adjusting virtual content formed using multiple primary color images is shown, and the resolution adjustment is performed based on the colors of the primary color images. In block 2110, the display system provides virtual content to be presented using multiple primary color images. These may be different images of different primary colors that will be directed to different waveguides, as discussed with respect to Figures 8-9B. As a result, in some embodiments, each stream of different primary color images may be rendered separately. The step of providing virtual content to be presented using multiple primary color images may include the step of utilizing a display system that outputs different primary color image streams to form a full-color image.

[0263] In block 2120, the display system may adjust the resolution of the primary color image based on its color. For example, the display system may select one of these primary color images for resolution adjustment. For example, the selection may be based on a determination of the illumination level, as described above with respect to block 1810 in Figure 18. As shown in Figure 19, some primary colors may not be perceptible to the user at some illumination levels. The display system may store in it information about illumination levels and primary colors that are invisible at those levels. If a match exists between the illumination levels and primary colors that are invisible at those levels, the image of that primary color may be selected for adjustment. In some environments, one adjustment may simply be not rendering or displaying the primary color image if the ambient illumination level is such that the user is not expected to perceive that color. For example, under scotopic illumination levels, the display system may be configured not to render or display the primary color red image.

[0264] Continuing to refer to Figure 21, in block 2130, virtual content is presented to the user. The presentation of virtual content may be carried out as discussed herein, for example, in block 1208 of Figure 12A or block 1412 of Figure 14.

[0265] Referring here to Figures 22A-22C, as discussed above, although not limited by theory, the human visual system's ability to perceive gradations in light intensity is thought to change with ambient illumination levels. Figures 22A-22C illustrate examples of contrast sensitivity changing as the amount of light incident on the user's eye decreases. For example, Figure 22A may be understood as showing contrast sensitivity under photopic conditions, Figure 22B as showing contrast sensitivity under crepuscular conditions, and Figure 22C as showing contrast sensitivity under scotopic conditions. Figure 22A shows gradations from 21101 to 2110 iFigure 22B shows progression from 21101 to 2110. i Figure 22C shows progression to 2102, moving from high light intensity to low light intensity. Similarly, Figure 22C shows the progression from 21101 to 2110 i Figure 18 shows a progression from high light intensity to low light intensity. Boxes 2120, 2130, and 2140 represent groups of intensity gradations that are perceived by the user as identical. The size of these groups is expected to increase with decreasing ambient light levels, as shown in the figure. Consequently, in some embodiments, as described above with respect to Figure 18, the display system may be configured to use the lowest intensity value within each group (e.g., within each of boxes 2120, 2130, and 2140).

[0266] Referring here to Figure 23, an embodiment of the representation of the user's optic nerve and peripheral blind spot is illustrated. In some embodiments, in addition to or as an alternative to any of the resolution adjustments disclosed herein, the display system may be configured to refrain from rendering content in various locations where the content is not expected to be perceptible to the user. Figure 23 shows the left and right eyes 210, respectively. L and 210 R This is illustrated. Each eye has individual optical axes 1003A and 1003B and optic nerve 2300 L and 2300 R It has the optic nerve 2300L and 2300 R Each of its individual eyes 210 L and 210 R There are blind spots at the points of contact. These blind spots allow the viewer to see ray 2302 L and 2302 R This obstructs the view of content in that direction. In addition, there is a region at the periphery of each eye where content cannot be seen by the opposite eye. For example, the left peripheral region P. L The content inside is left eye 210 L This may be seen by the right eye 210 RIt may not be visible depending on the region. On the other hand, in the right peripheral region P R The content inside is for the right eye 210 R This can be seen by, left eye 210 L It may not be visible depending on the case. As a result, in some embodiments, the display system has each eye 210 L and 210 R Content that would be mapped to the blind spots, for example, ray 2302 L and 2302 R The system may be configured to omit rendering the content that is placed above it. In addition, or alternatively, in some embodiments, the display system may omit rendering its content in the right peripheral region P L If you enter, view the content with your left eye 210 L The system may be configured to omit rendering to the left peripheral region P, and / or the display system may omit rendering to the left peripheral region P. L If you enter, view the content with your right eye 210 R It should be understood that the rendering of blind spots and / or peripheral areas may be configured to omit rendering. The locations of blind spots and / or peripheral areas may be pre-set, for example, based on the average of the user population, and / or adjusted and calibrated for a particular user by testing with content displayed in various locations and user input indicating whether virtual objects are visible. III. Multiple image streams for providing content with different resolutions

[0267] In some embodiments, a fovealed image having high and low spatial resolution regions may be formed by two or more spatially overlapping image streams, each having a different resolution (e.g., different perceived pixel densities). For example, one of the image streams, e.g., a low-resolution image stream, may form a wide-field image, while another image stream, e.g., a high-resolution image stream, may form a narrow-field image. The narrow-field and wide-field images may contain similar content but are viewed by the user at different resolutions or pixel densities. These images may be overlaid on each other (e.g., occupying the same location in space simultaneously or in temporal proximity, so that the viewer perceives the images as being presented at the same time). Thus, the viewer may receive the high-resolution aggregated image in a constrained portion of their field of view, and the low-resolution image over a larger portion of their field of view. Preferably, as discussed herein, the high-resolution portion is mapped to the foveal visual region of the user's eye, while the low-resolution portion is mapped to the peripheral visual region of the user's eye. Therefore, the difference in resolution between the high-resolution and low-resolution parts of the image is preferably not easily perceptible to the user.

[0268] In some environments, display systems for displaying high and low-resolution images utilize the same spatial light modulator to form both images. Therefore, the spatial light modulator has pixels of fixed size and density. In display systems with fixed-size and density pixels, an increase in angular field of view (FOV) comes at the expense of spatial or angular resolution, for example, depending on the Lagrange invariant. For instance, if an SLM with a fixed number of pixels is used to form both high and low-resolution images, diffusing those pixels across the entire field of view will provide an image with lower apparent resolution than confining those pixels to a small portion of the total field of view. That is, the pixel density of a high-resolution image is higher than the pixel density of a low-resolution image. Consequently, generally, there is an inverse correlation between FOV and angular resolution. Because FOV and angular resolution affect image visibility and quality, this trade-off imposes constraints on the user experience and ultimately achievable FOV and angular resolution in AR or VR systems. As will become apparent from the discussion herein, in some embodiments, the term “resolution” may be used to mean “angular resolution.”

[0269] Head-mounted or wearable display devices can be configured to provide an immersive user experience by projecting virtual content directly into the user's eyes. While providing wide-field-of-view (FOV) images uniformly with high resolution across the FOV can be beneficial, the physiological limitations of the human visual system may prevent users from viewing or even recognizing high-resolution images located within the peripheral regions of their field of vision. This inability to perceive high-resolution images within the peripheral region is due to the characteristics of the human retina, which contains two types of photoreceptors: rods and cones. Cones are more involved in sharp (detailed) vision. Rods and cones are distributed differently within the human eye. The highest concentration of cone cells is found in the fovea (i.e., the center of the retina), while the highest concentration of rod cells is found in the region directly surrounding the fovea (i.e., the periphery of the retina). Due to this non-uniform distribution of rods and cones, the fovea is involved in sharp central vision (also known as foveal vision). Visual acuity decreases as the distance from the fovea increases.

[0270] For AR or VR applications, headsets are generally worn by one user at a time. Headsets can be configured to exploit the user's inability to perceive all the details of a wide-field-of-view stream of images at once by limiting the display of high-resolution content to an area within the wide field of view currently in focus by the user. Thus, the headset can provide the user with the appearance of a high-resolution wide-FOV stream of images without the processing power that would otherwise be required to generate high-resolution content across the entire field of view. The stream of images presented to the user can take many forms and would generally be referred to as an image stream. For example, an image stream can show a static image by continuously displaying the same image to the user, or it can show motion by displaying a stream of different images. In some embodiments, the headset can be configured to display more than one image stream simultaneously, with different image streams having different angular resolutions and extending across different areas of the user's FOV. It should be noted that image streams associated with AR systems cannot display content holistically across the specific area to which they are assigned, as AR systems are designed to blend virtual and real-world content.

[0271] According to some embodiments, a first image stream and a second image stream can be presented to the user simultaneously or rapidly in succession so that the two image streams appear to be displayed at the same time. The first image stream may have a wide FOV and low resolution, encompassing the user's vision and evoking an immersive experience. In some embodiments, a portion of the first image stream corresponding to a momentary portion of the FOV covered by the second image stream may be turned off. The second image stream may have a narrow FOV and high resolution, and can be dynamically displayed within the boundaries of the first image stream according to the user's current fixation point, such as being determined in real time using eye-line tracking techniques. In other words, the second image stream may be shifted as the user's eye line of sight changes so that the second image stream continuously covers the user's foveal vision. In some embodiments, the first image stream is presented to the user at a fixed position as the second image stream is shifted relative to the first image stream. In some other embodiments, both the first and second image streams are shifted according to the user's current fixation point.

[0272] The content of the second image stream may include a subset of the content of the first image stream with a higher resolution than the first image stream, and may be overlaid on the first image stream and appropriately aligned with it. Because the higher-resolution second image stream overlays a portion of the first image stream within the user's foveal view, the modulation transfer function (MTF) in the area containing the higher-resolution image stream is increased. In some embodiments, the subset of the first image stream content overlaid by the second image stream may be turned off or presented at a lower intensity. Thus, the user can perceive a combination of the first and second image streams that has both a wide field of view and high resolution. Such a display system can offer several advantages. For example, the display system may have a relatively small shape factor and conserve computing resources and power while providing a better user experience.

[0273] In one embodiment, the light intensity within the boundary region between the first and second image streams tapers to a value below the intended image brightness, and the boundary regions of the first and second image streams overlap. In the overlapping area, the sum of the light intensities from the two image streams is relatively constant and can be equal to the intended image brightness. Crossing the overlapping region from the first image stream side to the second image stream side, the MTF changes from a first value equal to or closer to the MTF of the first image stream to a second value equal to or closer to the MTF of the second image stream. In this way, it is possible to avoid creating a sharp boundary between the regions supplied by the two image streams, which may be perceptible to the user in some situations.

[0274] According to some embodiments, a first light beam associated with a first image stream and a second light beam associated with a second image stream can be multiplexed into a combined light beam using a multiplexing method. For example, time-division multiplexing, polarization-division multiplexing, wavelength-division multiplexing, and equivalents can be used according to various embodiments. The combined light beam can be directed to one or more optical elements that serve to demultiplex the combined light beam into two separate optical paths. For example, beam splitters or optical switching elements such as polarization beam splitters (PBSs) or dichroic beam splitters can be used to separate the combined light beam, depending on the multiplexing method used. Once separated, the first light beam associated with the first image stream and the second light beam associated with the second image stream can be routed through their separate optical paths and ultimately provided to the user as an output.

[0275] According to some embodiments, a first light beam associated with a first image stream can be angularly expanded by optical elements in a first optical path so that the first image stream presents a wider FOV and lower angular resolution (as depended on invariants), while a second light beam associated with a second image stream is not angularly expanded, is de-expanded, or is expanded by an amount less than the amount of expansion applied to the first light beam associated with the first image stream. Thus, the second image stream can present a narrower FOV and higher angular resolution than the first image stream (as depended on invariants).

[0276] Figure 24 shows a field diagram illustrating the peripheral boundaries of an exemplary monocular field of view 3002 for a human eye in two-dimensional angular space. As shown in Figure 24, the temple-nose and lower-up axes of the field diagram serve to define the two-dimensional angular space into which the peripheral boundaries of the monocular field of view 3002 are mapped. Thus, the field diagram of Figure 24 can be considered an equivalent or similar to a "Goldmann" field map or plot for a human eye. As indicated by the depicted arrangement of the temple-nose and lower-up axes, the field diagram shown in Figure 24 represents a field diagram for a human left eye. While the field of view may vary slightly from person to person, the depicted field of view approximates what many people are able to see with their left eye. The schematic diagram of the visual field, which depicts the peripheral boundary of the exemplary monocular visual field of the right eye, shows that the temple-nasal axis and peripheral boundary of monocular visual field 3002 may be similar to that of the schematic diagram of the visual field in Figure 24, which is mirrored around the lower-upper axis.

[0277] The field diagram in Figure 24 further depicts the periphery of an exemplary oculomotor field of view 3004 for a human eye, which represents a portion of the monocular field of view 30022 in angular space where a person can fixate. In addition, the field diagram in Figure 24 also depicts the periphery of an exemplary foveal field of view 3006 for a human eye, which represents a portion of the monocular field of view 3002 in angular space within the direct view of the fovea of ​​the human eye at a given time. As depicted, the person's foveal field of view 3006 can move at any location within the oculomotor field of view 3004. The portion of the monocular field of view 3002 outside the foveal field of view 3006 in angular space may be referred to herein as the peripheral region of the person's field of view. Because the human eye's ability to distinguish high levels of detail outside the foveal field of view 3006 is very limited, displaying reduced-resolution images outside the foveal field of view 3006 is less likely to be perceived and can allow for substantial savings in terms of power consumption for processing components involved in generating content for the display.

[0278] Figure 25A shows an exemplary wearable display device 4050 configured to provide virtual content to a user, according to several embodiments. The wearable display device 4050 includes a main display 4052 supported by a frame 4054. The frame 4054 can be attached to the user's head using mounting members in the form of vine arms 4006.

[0279] Referring here to Figure 25B, an exemplary embodiment of an AR system configured to provide virtual content to a user will be described. In some embodiments, the AR system in Figure 25B may represent a system to which the wearable display device 4050 in Figure 25A belongs. The AR system in Figure 25B uses a stacked optical optics element assembly 4000 and generally includes an image generation processor 4010, a light source 4020, a controller 4030, a spatial light modulator ("SLM") 4040, an input optics system 4060, and at least one set of stacked eyepiece layers or optical optics elements ("LOE", e.g., plane waveguides) 4000 that function as a plurality of planar focus systems. The system may also include an eye-tracking subsystem 4070. Other embodiments may have multiple sets of stacked LOEs 4000, but it should be understood that the following disclosure will focus on the exemplary embodiment in Figure 25B.

[0280] The image generation processor 4010 is configured to generate virtual content that will be displayed to the user. The image generation processor may convert the image or video associated with the virtual content into a format that can be projected to the user in 3D. For example, when generating 3D content, the virtual content may need to be formatted so that certain parts of the image are displayed in a specific depth plane, while others are displayed in other depth planes. In one embodiment, all images may be generated in a specific depth plane. In another embodiment, the image generation processor may be programmed to provide slightly different images to the right and left eyes 210 so that, when viewed together, the virtual content appears coherently and comfortably to the user's eyes.

[0281] The image generation processor 4010 may further include memory 4012, a GPU 4014, a CPU 4016, and other circuitry for image generation and processing. The image generation processor 4010 may be programmed with desired virtual content to be presented to the user of the AR system in Figure 25B. It should be understood that in some embodiments, the image generation processor 4010 may be housed within a wearable AR system. In other embodiments, the image generation processor 4010 and other circuitry may be housed within a beltpack coupled to a wearable optical system. The image generation processor 4010 is operably coupled to a light source 4020 that projects light associated with the desired virtual content, and to one or more spatial light modulators (described below).

[0282] Light source 4020 is compact and has high resolution. Light source 4020 includes a plurality of spatially separated sub-light sources 4022 operably coupled to controller 4030 (described below). For example, light source 4020 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, light source 4020 may include LEDs or lasers of similar colors, one of each coupled to a specific area of ​​the display's field of view. In another embodiment, light source 4020 may include a wide-area emitter such as an incandescent or fluorescent lamp, with a mask overlaid for segmentation of the emission area and position. The sub-light sources 4022 are directly connected to the AR system in Figure 2B, but the sub-light sources 222 may be connected to the system via optical fibers (not shown), as long as the distal ends of the optical fibers (away from the sub-light sources 4022) are spatially separated from each other. The system may also include a concentrator (not shown) configured to collimate light from light source 4020.

[0283] In various exemplary embodiments, the SLM4040 may be reflective (e.g., DLP DMD, MEMS mirror system, LCOS, or FLCOS), transmissive (e.g., LCD), or luminescent (e.g., FSD or OLED). The type of spatial light modulator (e.g., speed, size, etc.) can be selected to improve the creation of 3-D perception. DLP DMDs, operating at higher refresh rates, can be easily incorporated into stationary AR systems, while wearable AR systems typically use DLPs of smaller size and power. The power of the DLP alters how the 3-D depth plane / focal plane is created. An image generation processor 4010 is operably coupled to the SLM4040, which encodes light from the light source 4020 with desired virtual content. Light from the light source 4020 may be encoded with image information as it is reflected from, emitted from, or passed through the SLM4040.

[0284] Referring back to Figure 25B, the AR system also includes an input optics system 4060 configured to direct light from the light source 4020 (i.e., a plurality of spatially separated sub-light sources 4022) and the SLM 4040 into the LOE assembly 4000. The input optics system 4060 may include one or more lenses configured to direct light into the LOE assembly 4000. The input optics system 4060 is spatially separated from the sub-light sources 4022 of the light source 4020 and configured to form a spatially separated and distinctly distinct pupil adjacent to the LOE 4000 (to the individual focal points of the beams emanating from the input optics system 4060), corresponding to distinctly different beams. The input optics system 4060 is configured so that the pupils are spatially displaced from each other. In some embodiments, the input optics system 4060 is configured to spatially displace the beam only in the X and Y directions. In such embodiments, the pupils are formed in a single X,Y plane. In other embodiments, the input optics system 4060 is configured to spatially displace the beam in the X, Y, and Z directions.

[0285] The spatial separation of light beams forms distinctly different beams and pupils, which allows for the placement of internally coupled gratings in distinctly different beam paths, such that each internally coupled grating is addressed (e.g., intersects or collides) primarily by only one distinctly different beam (or group of beams). This, in turn, facilitates the incidence of spatially separated light beams into individual LOE4000s of the LOE assembly 4000, while minimizing the incidence (i.e., crosstalk) of other light beams from other sub-sources 4022 of the multiple. Light beams from a particular sub-source 4022 are incident on the individual LOE4000 through an internally coupled grating on it (not shown in Figure 25B, see Figures 24-26). The internally coupled gratings of the individual LOE4000 are configured to interact with spatially separated light beams from the multiple sub-sources 4022 such that each spatially separated light beam intersects with only one internally coupled grating of the LOE4000. Therefore, each spatially separated light beam primarily incident on a single LOE4000. Thus, the image data encoded on the light beams from each of the sub-light sources 4022 by the SLM4040 can be effectively propagated along the single LOE4000 for delivery to the user's eye 210.

[0286] Each LOE4000 is then configured to project an image or sub-image onto the user's retina, appearing as if arising from a desired depth plane or FOV angle position. Multiple individual LOE4000s and sub-light sources 4022 can therefore selectively project images (synchronously encoded by the SLM4040 under the control of the controller 4030) that appear as if arising from various depth planes or positions in space. By sequentially projecting images at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full volume frame rate of 60 Hz) using each of the multiple individual LOE4000s and sub-light sources 4022, the system in Figure 25B can generate 3D images of virtual objects in various depth planes that appear to exist simultaneously within a 3D image.

[0287] The controller 4030 communicates with and is operablely coupled to the image generation processor 4010, the light source 4020 (sub-light source 4022), and the SLM 4040, coordinating the synchronized display of images by instructing the SLM 4040 to encode the light beam from the sub-light source 4022 with appropriate image information from the image generation processor 4010.

[0288] The AR system also includes a voluntary eye-tracking subsystem 4070 configured to track the user's eye 4002 and determine the user's focus. In one embodiment, only a subset of sub-light sources 4022 may be activated based on input from the eye-tracking subsystem to illuminate a subset of LOE 4000, as discussed below. Based on input from the eye-tracking subsystem 4070, one or more sub-light sources 4022 corresponding to a particular LOE 4000 may be activated so that an image is produced in a desired depth plane that matches the user's focus / accommodation. For example, if the user's eyes 210 are parallel to each other, the AR system in Figure 25B may activate sub-light sources 4022 corresponding to LOE 4000, configured to deliver collimated light to the user's eye so that the image appears to originate from optical infinity. In another embodiment, if the eye-tracking subsystem 4070 determines that the user's focus is 1 meter away, a sub-light source 4022 corresponding to the LOE 4000 may be activated instead, configured to focus within approximately that range. In this particular embodiment, it should be understood that only one group of sub-light sources 4022 is activated at any given time, while the other sub-light sources 4020 are deactivated to conserve power.

[0289] Figure 25C schematically illustrates the optical paths within an exemplary VOA (Visual Optics Assembly) that may be used to present digital or virtual images to a viewer, according to several embodiments. In some embodiments, the VOA may be incorporated into a system similar to a wearable display device 4050, as depicted in Figure 25A. The VOA includes a projector 4001 and an eyepiece 200, which may be fitted around the viewer's eyes. The eyepiece 4000 may correspond to an LOE 4000, as described above with reference to Figure 25B, for example. In some embodiments, the projector 4001 may include a group of red LEDs, a group of green LEDs, and a group of blue LEDs. For example, according to one embodiment, the projector 201 may include two red LEDs, two green LEDs, and two blue LEDs. In some embodiments, a projector 4001 and its components (e.g., an LED light source, a reflective collimator, an LCoS SLM, and a projector relay) as depicted in Figure 25C may represent or provide functionality for one or more of the light sources 4020, sub-light sources 4022, SLM 4040, and input optics systems 4060 as described above with reference to Figure 25B. The eyepiece 4000 may include one or more eyepiece layers, each representing one of the LOE 4000 as described above with reference to Figure 25B. Each eyepiece layer of the eyepiece 4000 may be configured to project an image or sub-image onto the retina of the viewer's eye, appearing to result from a separate desired depth plane or FOV angle position.

[0290] In one embodiment, the eyepiece 4000 includes three eyepiece layers, one for each of the three primary colors: red, green, and blue. For example, in this embodiment, each eyepiece layer of the eyepiece 4000 may be configured to deliver collimated light to the eye, appearing to originate from the optical depth plane (0 diopters). In another embodiment, the eyepiece 4000 may include six eyepiece layers, one set of eyepiece layers for each of the three primary colors configured to form a virtual image in one depth plane, and another set of eyepiece layers for each of the three primary colors configured to form a virtual image in another depth plane. For example, in this embodiment, each eyepiece layer in one set of eyepiece layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from the optical infinite depth plane (0 diopters), while each eyepiece layer in another set of eyepiece layers of the eyepiece 4000 may be configured to deliver collimated light to the eye that appears to originate from a distance of 2 meters (0.5 diopters). In other embodiments, the eyepiece 4000 may include three or more eyepiece layers, for each of three primary colors for three or more different depth planes. For example, in such an embodiment, each of yet another set of eyepiece layers may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter).

[0291] Each eyepiece layer comprises a planar waveguide and may include an internal coupling grating 4007, an orthogonal pupil expander (OPE) region 4008, and an exit pupil expander (EPE) region 4009. Further details of the internal coupling grating, orthogonal pupil expansion, and exit pupil expansion are described in U.S. Patent Application No. 14 / 555,585 and U.S. Patent Application No. 14 / 726,424, the contents of which are explicitly and fully incorporated herein by reference, as if described in whole. Still referring to Figure 25C, the projector 4001 projects image light onto the internal coupling grating 4007 in the eyepiece layer 4000. The internal coupling grating 4007 couples the image light from the projector 4001 into the waveguide and propagates it toward the OPE region 4008. The waveguide propagates the image light horizontally by total internal reflection (TIR). The OPE region 4008 of the eyepiece layer 4000 also includes a diffraction element that combines a portion of the image light propagating through the waveguide and redirects it toward the EPE region 4009. More specifically, the collimated light propagates horizontally along the waveguide (i.e., relative to the diagram in Figure 25C) by TIR, thereby repeatedly intersecting with the diffraction element of the OPE region 4008. In some embodiments, the diffraction element of the OPE region 4008 has a relatively low diffraction efficiency. This is because, at each point of intersection with the diffraction element of the OPE region 4008, a certain percentage (e.g., 10%) of the light is diffracted vertically downward toward the EPE region 4009, while a certain percentage of the light continues horizontally along the waveguide along its original trajectory via TIR. Thus, at each point of intersection with the diffraction element of the OPE region 4008, additional light is diffracted downward toward the EPE region 4009. By splitting the incident light into multiple externally coupled sets, the light's exit pupil is horizontally expanded by the diffracting element of OPE region 4008. The expanded light, externally coupled from OPE region 4008, enters EPE region 4009.

[0292] The EPE region 4009 of the eyepiece layer 4000 also includes a diffraction element that combines a portion of the image light propagating through the waveguide and redirects it toward the viewer's eye 210. Light incident on the EPE region 4009 propagates perpendicularly along the waveguide (i.e., relative to the diagram in Figure 25C) by the TIR. At each intersection point between the propagating light and the diffraction element of the EPE region 4009, a certain proportion of the light is diffracted toward the adjacent surface of the waveguide, allowing the light to escape from the TIR, emerge from the surface of the waveguide, and propagate toward the viewer's eye 210. In this scheme, the image projected by the projector 4001 can be viewed by the viewer's eye 210. In some embodiments, the diffraction element of the EPE region 4009 may be designed or configured to have a phase profile that is the sum of a linear diffraction grating and a radially symmetric diffraction lens. The radially symmetric lens sides of the EPE region 4009 diffraction element also perform both to impart a certain focal level to the diffracted light, shaping the wavefront of individual beams (e.g., imparting curvature), and to steer the beams at a certain angle to match the designed focal level. Each beam of light externally coupled by the EPE region 4009 diffraction element may geometrically extend to individual focal points positioned in front of the viewer, and is given a convex wavefront profile with the center of the radius at each focal point, thereby producing an image or virtual object at a given focal plane.

[0293] Descriptions of such visible optical assemblies and other similar settings are further provided in U.S. Patent Application No. 14 / 331,218, U.S. Patent Application No. 15 / 146,296, and U.S. Patent Application No. 14 / 555,585 (all of which are incorporated herein by reference as a whole). In some embodiments, the exemplary VOA may include and / or take the form of one or more components described in any of the patent applications, which are described above with reference to Figure 25C and incorporated herein by reference. IV. Foveal-oriented displays with high field of view and high resolution using multiple optical paths

[0294] Figures 26A–26D illustrate exemplary rendering viewpoints to be used and the light fields to be produced within the AR system for each of the two exemplary eye orientations. In Figure 26A, the viewer's eye 210 is oriented in a first manner toward the eyepiece 5000. In some embodiments, the eyepiece 5000 may be analogous to a stack of LOE or eyepiece 4000, as described above with reference to Figures 25B and 25C. More specifically, in this embodiment, the viewer's eye 210 is oriented such that the viewer may be able to view the eyepiece 5000 in a relatively straight line. In some embodiments, the AR system to which the eyepiece 5000 belongs, which may be analogous to the AR system described above with reference to Figure 25B, may perform one or more actions for presenting virtual content at one or more distances in front of the viewer's eye 210 on one or more depth planes located within the viewer's FOV.

[0295] An AR system may determine a viewpoint in the rendering space based on the position and orientation of the viewer's head, from which the viewer views 3-D virtual content in the rendering space, such as virtual objects. In some embodiments, such an AR system may include one or more sensors, and utilize data from these one or more sensors to determine the position and / or orientation of the viewer's head, as will be described in more detail below with reference to Figure 29A. The AR system may include one or more such sensors in addition to one or more eye-tracking components, such as one or more components of the eye-tracking subsystem 4070, as described above with reference to Figure 25B. Using such data, the AR system may effectively map the position and orientation of the viewer's head in the real world to specific locations and specific angular positions in the 3D virtual environment, create a virtual camera positioned at a specific location in the 3D virtual environment and oriented to a specific angular position in the 3D virtual environment relative to that specific location in the 3D virtual environment, and render virtual content for the viewer that would be captured by the virtual camera. Further details discussing the real-world-virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE" (expressly incorporated herein in whole by reference for any purpose).

[0296] In some embodiments, the AR system may create or dynamically reposition and / or reorient one such head-tracking virtual camera for the viewer's left eye or orbit and another such head-tracking virtual camera for the viewer's right eye or orbit, as the viewer's eyes and / or orbits are physically separated from each other and therefore consistently positioned in different locations. Virtual content rendered from the viewpoint of the head-tracking virtual camera associated with the viewer's left eye or orbit may be presented to the viewer through an eyepiece on the left side of a wearable display device, such as those described above with reference to Figures 25A-25C, and virtual content rendered from the viewpoint of the head-tracking virtual camera associated with the viewer's right eye or orbit may be presented to the viewer through an eyepiece on the right side of the wearable display device. A head-tracking virtual camera may be created and / or dynamically repositioned for each eye or orbit based on information regarding the current position and orientation of the viewer's head, but the position and orientation of such a head-tracking virtual camera may not depend on either the individual orbits of the viewer or the position or orientation of each of the viewer's eyes relative to the viewer's head. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTUAL FEATURES IN 3D RECONSTRUCTION" (expressly incorporated herein in whole by reference for any purpose).

[0297] The AR system in Figure 26A may create or dynamically reposition and / or reorient such a head-tracking virtual camera, render virtual content from the viewpoint of the head-tracking virtual camera (viewpoint 5010), and project light representing the rendering of the virtual content onto the retina of the viewer's eye 210 through the eyepiece 5000. As shown in Figure 26A, the head-tracking rendering viewpoint 5010 may be oriented diagonally, horizontally, and / or vertically, ±θ 310An FOV extending to an angular region may be provided. In some embodiments, as will be described in more detail below, the head-tracking rendering viewpoint 5010 may provide a relatively wide FOV. In such embodiments, the AR system may also create or dynamically reposition and / or reorient for each eye or orbit, a separate virtual camera different from and in addition to the head-tracking virtual camera. In the embodiment of Figure 26A, the AR system may render and present virtual content from the viewpoint of another virtual camera in rendering space, along with virtual content from the viewpoint of the head-tracking virtual camera 5010.

[0298] For example, in such embodiments, the AR system in Figure 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera based on the current line of sight of the viewer's eye 210. In some embodiments, as will be described in more detail below with reference to Figure 29A, such an AR system may include one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to Figure 25B, which determine the viewer's current line of sight, the current position and / or orientation of the viewer's eye 210 relative to the viewer's head, and equivalents. Using such data, the AR system in Figure 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera, render virtual content from the viewpoint of the foveal tracking virtual camera (viewpoint 5020A), and project light representing the virtual content as rendered from viewpoint 5020A onto the fovea of ​​the viewer's eye 210 through the eyepiece 5000.

[0299] As shown in Figure 26A, the foveal tracking rendering viewpoint 5020A may provide a narrower FOV than that of the head tracking rendering viewpoint 5010. Thus, the FOV of the foveal tracking rendering viewpoint 5020A may appear to occupy a conical subspace of the FOV of the head tracking rendering viewpoint 5010. That is, the FOV of the foveal tracking rendering viewpoint 5020A may be a subfield of the FOV of the head tracking rendering viewpoint 5010. For example, as shown in Figure 26A, the relationship between the FOV of the head tracking rendering viewpoint 5010 and the foveal tracking rendering viewpoint 5020A is -θ 310 ≤ -θ 320A ≤θ 320A ≤θ 310 As given by, diagonally, horizontally, and / or vertically, ±θ 320A An FOV extending to an angular region may be provided. In some embodiments, the FOV of the head-tracking rendering viewpoint 5010 may be at least as wide as the viewer's eye-moving field of view, which in this embodiment would be a total conical space to which the viewer's eye 210 can fix when the viewer's head is held in a given position and orientation. Thus, in these embodiments, the head-tracking virtual camera and the foveal-tracking virtual camera may be positioned in substantially the same location in the rendering space, or both virtual cameras may be positioned at a fixed distance from each other in the rendering space so that they can be linearly and / or angularly translated in conjunction in the rendering space as the position and / or orientation of the viewer's head changes. For example, the head-tracking virtual camera may be positioned at a location in the rendering space corresponding to the rotation center of the viewer's eye 210, while the foveal-tracking virtual camera may be positioned at a location in the rendering space corresponding to the region of the viewer's eye 210 between the rotation center and the cornea. In fact, the Euclidean distance between two virtual cameras can remain substantially constant when translated in rendering space, just as the Euclidean distance between two specific regions of a viewer's eye 210 or another rigid body can always remain substantially constant.

[0300] While the spatial relationships between each virtual camera in such a pair of virtual cameras may remain substantially fixed in rendering space throughout the entire use of the AR system, in these embodiments, the orientation of the foveal-tracking virtual camera may change relative to the head-tracking virtual camera as the viewer rotates their eye 210. Thus, the conical subspace of the head-tracking virtual camera's FOV, occupied by the FOV of the foveal-tracking virtual camera, can change dynamically as the viewer rotates their eye 210.

[0301] Furthermore, virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020A can be rendered and presented by the AR system at a relatively high resolution. More specifically, the resolution at which virtual content within the FOV of the foveal tracking virtual camera is rendered and presented may be higher than the resolution at which virtual content within the FOV of the head tracking virtual camera is rendered and presented. Thus, the highest resolution subfield of a given light field, externally coupled by the eyepiece 5000 and projected onto the retina of the viewer's eye 210, may reach the fovea of ​​the viewer's eye 210.

[0302] Figure 3B illustrates an exemplary light field 5030A, externally coupled by the eyepiece 5000 and projected onto the retina of the viewer's eye 210, while the viewer's eye 210 is oriented in the first manner as depicted in Figure 26A and described above with reference thereto. The light field 5030A may include various angular light components representing virtual content that will be captured in rendering space by the pair of virtual cameras described above. As will be described in more detail below with reference to Figure 26A and subsequent figures, the light representing virtual content that will be captured in rendering space by the head-tracking virtual camera and the light representing virtual content that will be captured in rendering space by the foveal-tracking virtual camera may be multiplexed by the AR system according to one of various different multiplexing schemes. The adoption of such multiplexing schemes may, in at least some instances, allow the AR system to operate with greater efficiency and / or occupy less physical space.

[0303] Still referring to Figure 26B, the angular light component of the light field 5030A, which represents virtual content (e.g., virtual objects and other content that enter the head-tracking rendering viewpoint 5010) that would be captured in rendering space by the head-tracking virtual camera, is -θ with respect to the viewer's eye 210. 310 ~+θ 310 This may include what will be projected onto the retina of the viewer's eye 210 at angles in units of degrees. Similarly, the angular light component of the light field 5030A representing virtual content (e.g., virtual objects and other content that enter within the foveal-tracking rendering viewpoint 5020A) that will be captured in rendering space by the foveal-tracking virtual camera is -θ relative to the viewer's eye 210. 320A ~+θ 320A This may include what will be projected onto the retina of the viewer's eye 210 at angles in units of degrees. Such angular light components associated with the foveal tracking rendering viewpoint 5020A occur within the light field 5030A, -θ 320A and +θ320A The interval between angular units is -θ, where the angular light component associated with the head-tracking rendering viewpoint 5010 occurs within the light field 5030A. 310 and +θ 310 The regularity may be higher than the interval between angular units. Thus, the resolution at which virtual content associated with the foveal tracking rendering viewpoint 5020A can be rendered and presented to the viewer may be higher than the resolution at which virtual content associated with the head tracking rendering viewpoint 5010 can be rendered and presented to the viewer.

[0304] In some embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010, which occurs within the light field 5030A, is further -θ relative to the viewer's eye 210. 320A ~+θ 320A This may include an angle that will be projected onto the retina of the viewer's eye 210 at an angle in units of degrees. In such embodiments, such angular light components associated with the head-tracking rendering viewpoint 5010 will occur in the light field 5030A, -θ 320A and +θ 320A The interval between angular units is -θ, where the angular light component associated with the foveal tracking rendering viewpoint 5020A is generated within the light field 5030A. 320A and +θ 320A The regularity may be less than the interval between angular units. In other embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010, which occurs within the light field 5030A, is -θ relative to the viewer's eye 210. 320A ~+θ 320A It is possible to exclude what would be projected onto the retina of the viewer's eye 210 at angles in units of an angle. Thus, in these other embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs within the light field 5030A is -θ 310 ~-θ 320A Angle or θ in units of angle 320A ~θ 310 This could be projected onto the retina of the viewer's eye 210 at that angle.

[0305] In Figure 26C, the viewer's eye 210 is oriented to the eyepiece 5000 in a second manner, different from the first manner in which the viewer's eye 210 is oriented to the eyepiece 5000 in Figures 26A-26B. For the purposes of this embodiment, the position and orientation of the viewer's head in Figures 26C-26D may be treated as identical to the position and orientation of the viewer's head as described above with reference to Figures 26A-26B. Thus, Figures 26A-26B and 26C-26D may represent the viewer and AR system described above in the first and second time-series stages, respectively. More specifically, in this embodiment, the viewer's eye 210 is rotated with its center off from a relatively straight orientation, as depicted in Figures 26A-26B.

[0306] During the transition from the first to the second stage, the AR system in Figure 26C may function to maintain the head-tracking virtual camera in the same position and orientation, as described above with reference to Figures 26A-26B, since the viewer's head posture (e.g., position and orientation) has not changed. Thus, in the second stage depicted in Figures 26C-26D, the AR system may render the virtual content from the viewpoint of the head-tracking virtual camera (i.e., the head-tracking rendering viewpoint 5010) and project the light representing the rendering of the virtual content onto the retina of the viewer's eye 210 through the eyepiece 5000. The head-tracking rendering viewpoint 5010 may remain static or relatively static throughout the entire first and second time-series stages in Figures 26A-26D, but during the transition from the first to the second stage, the AR system may function to adjust the orientation of the foveal-tracking virtual camera in the rendering space based on the change in the viewer's line of sight of the viewer's eye 210 from the first to the second stage. In other words, the AR system may replace or reorient the foveal tracking virtual camera, such as the one employed in the first stage to provide the foveal tracking rendering viewpoint 5020A, so that the foveal tracking virtual camera employed in the second stage provides a foveal tracking rendering viewpoint 5020C that is different from the foveal tracking rendering viewpoint 5020A. In the second stage, the AR system may also render virtual content from the viewpoint of the foveal tracking virtual camera viewpoint 5020C and project light representing the rendering of the virtual content onto the fovea of ​​the viewer's eye 201 through the eyepiece 5000.

[0307] In the embodiment shown in Figures 26C-26D, the foveal tracking rendering viewpoint 5020C may occupy a conical subspace of the head tracking rendering viewpoint 5010 that is different from that of the foveal tracking rendering viewpoint 5020A. For example, as shown in Figure 26C, the foveal tracking rendering viewpoint 5020C is θ from the FOV of the foveal tracking rendering viewpoint 5020A. 320C Displaced by an angular unit, diagonally, horizontally, and / or vertically, ±θ 320AIt can provide a FOV that extends over an angular unit region. That is, the foveal tracking rendering viewpoint 5020C is at an angle of θ diagonally, horizontally, and / or vertically. 320C ±θ 320A It can provide a FOV that extends over an angular unit region.

[0308] FIG. 26D illustrates an exemplary light field 5030C that is externally coupled by the eyepiece 5000 and projected onto the retina of the viewer's eye 201 while the viewer's eye 201 is oriented in a second manner as depicted in FIG. 26C and described above in reference thereto. The light field 5030C can include various angular light components that represent virtual content that would be captured within the rendering space from the head tracking rendering viewpoint 5010 and the foveal tracking rendering viewpoint 5020C. The angular light components of the light field 5030C that represent virtual content that would be captured within the rendering space from the head tracking rendering viewpoint 5010 are at an angle of -θ 310 ~+θ 310 and can include those that would be projected onto the retina of the viewer's eye 210 at an angle that extends over an angular unit. However, as described above in reference to FIGS. 26A-26B, at the start from the first stage, the angular light components of the light field 5030C that represent virtual content (e.g., virtual objects and other content that fall within the foveal tracking rendering viewpoint 5020C) that would be captured within the rendering space by the foveal tracking virtual camera are at an angle of θ 320C -θ 320A angular unit ~θ 320C +θ 320A and can include those that would be projected onto the retina of the viewer's eye 210 at an angle that extends over an angular unit.

[0309] Such angular light components associated with the foveal tracking rendering viewpoint 320C occur within the light field 5030C at an angle of θ 320C -θ 320A angular unit and θ 320C +θ 320AThe spacing between angular units is -θ at which the angular light component associated with the head-tracking rendering viewpoint 5010 occurs within the light field 5030C 310 and +θ 310 and can be higher than the spacing between angular units. Thus, the resolution at which virtual content associated with the foveal-tracking rendering viewpoint 5020C can be rendered and presented to the viewer should be noted as -θ with respect to the viewer's eye 210 320A ~+θ 320A and can be higher than the resolution at which virtual content associated with the head-tracking rendering viewpoint 5010, including the virtual content represented by the angular light component, is projected onto the retina of the viewer's eye 210 at an angle spanning the angular units

[0310] In some embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs within the light field 5030C further spans an angle of θ 320C -θ 320A angular units to θ 320C +θ 320A and can be projected onto the retina of the viewer's eye at an angle spanning the angular units. In such embodiments, such an angular light component associated with the head-tracking rendering viewpoint 310 occurs within the light field 5030C at -θ 320C -θ 320A angular units to θ 320C +θ 320A The spacing between angular units can be less regular and lower than the spacing between angular units at which the angular light component associated with the foveal-tracking rendering viewpoint 5 O20C occurs within the light field 5030C at θ 320C -θ 320A angular units to θ 320C +θ 320A In other embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs within the light field 5030C spans an angle of θ 320C -θ 320A angular units to θ 320C +θ 320AIt is possible to exclude what would be projected onto the retina of the viewer's eye 210 at angles in units of an angle. Thus, in these other embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs within the light field 5030C is -θ 310 ~θ 320C -θ 320A Angle and θ in angular units 320C +θ 320A Angle~θ 310 This may be projected onto the retina of the viewer's eye 210 in units of angles or in degrees.

[0311] Figures 26E-26F schematically illustrate exemplary configurations of images that may be presented to the user according to several embodiments. Note that the grid squares in Figures 26E-26F schematically represent image points where fields 3002, 3004, and 3006, as described above with reference to Figure 24, are defined in two-dimensional angular space. A low-resolution first image stream 5010E with a wide FOV can be displayed in a static location. The low-resolution first image stream 5010E with a wide FOV can represent one or more images of virtual content that would be captured by a first virtual camera having a static position and orientation in rendering space. For example, the low-resolution first image stream 5010E can represent one or more images of virtual content that would be captured by a head-tracking virtual camera, such as the head-tracking virtual camera described above with reference to Figures 26A-26D. The first image stream 5010E can encompass the user's vision and evoke an immersive experience in the user.

[0312] A high-resolution second image stream 5020E, having a relatively narrow FOV, can be displayed within the boundaries of the first image stream 5010E. In some embodiments, the second image stream 5020E can represent one or more images of virtual content that would be captured by a second different virtual camera in rendering space, which has an orientation that can be dynamically adjusted in real time to an angular position matching the user's current fixation point, based on data acquired using eye-line-tracking techniques. In these embodiments, the high-resolution second image stream 5020E can represent one or more images of virtual content that would be captured by a foveal-tracking virtual camera, such as the foveal-tracking virtual camera described above with reference to Figures 26A-26D. In other words, the viewpoint in rendering space from which one or more images of the virtual content represented by the second image stream 5020E are captured can be reoriented as the user's eye-line of sight changes, so that the viewpoint associated with the second image stream 5020E is continuously aligned with the user's foveal vision.

[0313] For example, the second image stream 5020E can contain virtual content located in a first region of rendering space when the user's line of sight is fixed to a first position, as shown in Figure 26E. As the user's line of sight moves to a second position different from the first position, the viewpoint associated with the second image stream 5020E can be adjusted so that the second image stream 5020E can contain virtual content located in a second region of rendering space, as shown in Figure 26F. In some embodiments, the first image stream 5010E has a wide FOV but low angular resolution, as indicated by a sparse grid. The second image stream 5020E has a narrow FOV but high angular resolution, as indicated by a dense grid.

[0314] Figure 26G schematically illustrates exemplary configurations of images that may be presented to the user according to several other embodiments. As in Figures 26E-26F, the grid squares in Figure 26G schematically represent image points defined in two-dimensional angular space. Similar to the configurations illustrated in Figures 26E-26F, a first low-resolution image stream 5010G with a wide FOV contains virtual content that is visible from a head-tracking rendering viewpoint, while a second high-resolution image stream 5020G with a narrow FOV contains virtual content that is visible from a foveal-tracking rendering viewpoint, which can be dynamically reoriented to coincide with the user's current fixation point. Here, the peripheral boundary of the FOV associated with the first image stream 5010G can form a rectangular boundary with rounded corners, and the peripheral boundary of the FOV associated with the second image stream 5020G can form a circular boundary.

[0315] Figure 26H schematically illustrates exemplary configurations of images that may be presented to the user according to several other embodiments. As shown in Figures 26E-26G, the grid squares in Figure 26H schematically represent image points defined in two-dimensional angular space. Here, both the peripheral boundaries of the FOV associated with the first image stream 5010H and the peripheral boundaries of the FOV associated with the second image stream 5020H can form circular boundaries. In some other embodiments, one or both of the peripheral boundaries of the FOV associated with the first image stream 5010H and the peripheral boundaries of the FOV associated with the second image stream 5020H can form elliptical boundaries or other shapes. In some embodiments, the image source of the AR system in Figure 26H may include a scanning fiber that can be scanned in a predetermined pattern and provide light beams for the first image stream 5010H and the second image stream 5020H with a desired boundary shape.

[0316] Figure 27 illustrates a field of view 3002 and an oculomo...

Claims

1. A display system, wherein the display system is An image source comprising a spatial light modulator for providing a first image stream and a second image stream, wherein the second image stream has a polarization different from that of the first image stream, A viewing assembly comprising an optically guided optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to the user, One or more processors that communicate with the aforementioned image source, One or more computer storage media and Equipped with, The one or more computer storage media store instructions, and when an instruction is executed by the one or more processors, the one or more processors... The image source is made to output the first image stream to the viewing assembly, wherein the image formed by the first image stream has a first pixel density. The image source is made to output the second image stream to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Perform an action that includes the following: The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream. The display system further comprises a polarization rotor, the polarization rotor configured to rotate the polarization of one of the first and second image streams such that the first and second image streams have the same polarization before they are incident on the viewing assembly.

2. The display system according to claim 1, further comprising modifying the location of the image formed by the second image stream relative to the image formed by the first image stream.

3. The display system according to claim 1 or 2, further comprising modifying the size of the image formed by the second image stream over time.

4. The system further comprises an eye-line-of-sight tracker configured to detect changes in the user's eye orientation, The display system according to any one of claims 1 to 3, wherein the operation includes modifying the location of the image formed by the second image stream relative to the image formed by the first image stream in accordance with the detected change in the orientation of the user's eyes.

5. The second image stream path between the image source and the viewing assembly further comprises a scanning mirror in the path of the second image stream, The display system according to any one of claims 1 to 4, further comprising adjusting the orientation of the scanning mirror and modifying the location of the image formed by the second image stream relative to the image formed by the first image stream.

6. The display system according to any one of claims 1 to 5, wherein the viewing assembly comprises an eyepiece, the eyepiece comprises a waveguide having an internal coupling grating, and the internal coupling grating is configured to internally couple light to the waveguide.

7. A display system, wherein the display system is An image source comprising a spatial light modulator for providing a first image stream and a second image stream, A viewing assembly comprising an optical guidance optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to a user, and an eyepiece, wherein the eyepiece comprises a waveguide, and the waveguide comprises an internal coupling grating configured to internally couple light to the waveguide, and another internal coupling grating configured to internally couple light to the waveguide, One or more processors that communicate with the aforementioned image source, One or more computer storage media and Equipped with, The one or more computer storage media store instructions, and when an instruction is executed by the one or more processors, the one or more processors... The image source is made to output the first image stream to the viewing assembly, wherein the image formed by the first image stream has a first pixel density. The image source is made to output the second image stream to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Perform an action that includes the following: The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream. Both the aforementioned internally coupled grating and the other internally coupled grating are located within the optical path of the second image stream. The internally coupled grid and the other internally coupled grid are transparent to the second image stream having the first polarization. The eyepiece further comprises a mirror spaced apart from the other internal bonding grid and facing the other internal bonding grid, A display system in which the mirror is configured to reflect the second image stream toward the other internally coupled grating and to change the polarization of the light in the second image stream to a second polarization, and the other internally coupled grating is configured to internally couple the light of the second polarization.

8. A display system, wherein the display system is An image source comprising a spatial light modulator for providing a first image stream and a second image stream, A viewing assembly comprising an optical guidance optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to a user, and an eyepiece, wherein the eyepiece comprises a waveguide, and the waveguide comprises an internal coupling grating configured to internally couple light to the waveguide, and another internal coupling grating configured to internally couple light to the waveguide, One or more processors that communicate with the aforementioned image source, One or more computer storage media and Equipped with, The one or more computer storage media store instructions, and when an instruction is executed by the one or more processors, the one or more processors... The image source is made to output the first image stream to the viewing assembly, wherein the image formed by the first image stream has a first pixel density. The image source is made to output the second image stream to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Perform an action that includes the following: The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream. A display system in which the other internal coupling grid is arranged along the same surface of the waveguide as the internal coupling grid, the internal coupling grid is located in the path of the second image stream, and the other internal coupling grid is located in the path of the first image stream.

9. The display system according to claim 6, wherein the waveguide comprises another internal coupling grating configured to internally couple light to the waveguide.

10. The display system according to claim 7 or 9, wherein the other internal coupling grid is arranged along a surface of the waveguide different from the internal coupling grid.

11. The display system according to any one of claims 6 to 10, wherein the internal bonding grid is made at least partially from a liquid crystal material, and the internal bonding grid is configured such that incident light having a predetermined polarization is diffracted by the internal bonding grid, and incident light having at least one other polarization passes through the internal bonding grid.

12. The display system according to any one of claims 1 to 11, further comprising a beam splitting optical system for splitting light into multiple image streams propagating in different directions.

13. The display system according to claim 12, wherein the beam splitting optical system is a polarizing beam splitter.

14. The display system according to claim 12, wherein the beam splitting optical system is a switchable reflector, and the switchable reflector is selectively switchable between a reflective state and a transmittance state.

15. A method for displaying image content through a display system, wherein the method is performed by one or more processors of the display system, The image source of the display system is instructed to output a first image stream of the image content to the viewing assembly of the display system, wherein the image formed by the first image stream has a first pixel density, and the viewing assembly includes an optical guidance optical system for receiving the first and second image streams from the image source and outputting the first and second image streams to the user. The image source is to output a second image stream of the image content to the viewing assembly, wherein the second image stream has a polarization different from that of the first image stream, and the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Includes, The polarization rotor rotates the polarization of one of the first and second image streams so that the first and second image streams have the same polarization before it enters the viewing assembly. The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, A method wherein the image formed by the second image stream covers a corresponding portion of the field of view provided by the first image stream.

16. The method of claim 15, further comprising modifying the location of the image formed by the second image stream relative to the image formed by the first image stream.

17. The method according to either claim 15 or 16, further comprising modifying the size of the image formed by the second image stream over time.

18. The display system further comprises an eye-line-of-sight tracker configured to detect changes in the orientation of the user's eyes, The method according to any one of claims 15 to 17, further comprising modifying the location of the image formed by the second image stream relative to the image formed by the first image stream in accordance with the detected change in the orientation of the user's eyes.

19. The display system further comprises a scanning mirror in the path of the second image stream between the image source and the viewing assembly, The method according to any one of claims 15 to 18, further comprising adjusting the orientation of the scanning mirror and modifying the location of the image formed by the second image stream relative to the image formed by the first image stream.

20. The method according to any one of claims 15 to 19, wherein the viewing assembly comprises an eyepiece, the eyepiece comprises a waveguide having an internal coupling grating, and the internal coupling grating is configured to internally couple light to the waveguide.

21. A method for displaying image content through a display system, wherein the method is performed by one or more processors of the display system, The display system's image source is instructed to output a first image stream of the image content to the display system's viewing assembly, wherein the image formed by the first image stream has a first pixel density, and the viewing assembly comprises an optical guidance optical system and an eyepiece for receiving the first and second image streams from the image source and outputting the first and second image streams to the user, the eyepiece comprising a waveguide, the waveguide comprising an internal coupling grating configured to internally couple light to the waveguide, and another internal coupling grating configured to internally couple light to the waveguide, The image source is made to output a second image stream of the image content to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Includes, The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream. Both the aforementioned internally coupled grating and the other internally coupled grating are located within the optical path of the second image stream. The internally coupled grid and the other internally coupled grid are transparent to the second image stream having the first polarization. The eyepiece further comprises a mirror spaced apart from the other internal bonding grid and facing the other internal bonding grid, A method wherein the mirror is configured to reflect the second image stream toward the other internally coupled grating and to change the polarization of the light in the second image stream to a second polarization, and the other internally coupled grating is configured to internally couple the light of the second polarization.

22. A method for displaying image content through a display system, wherein the method is performed by one or more processors of the display system, The display system's image source is instructed to output a first image stream of the image content to the display system's viewing assembly, wherein the image formed by the first image stream has a first pixel density, and the viewing assembly comprises an optical guidance optical system and an eyepiece for receiving the first and second image streams from the image source and outputting the first and second image streams to the user, the eyepiece comprising a waveguide, the waveguide comprising an internal coupling grating configured to internally couple light to the waveguide, and another internal coupling grating configured to internally couple light to the waveguide, The image source is made to output a second image stream of the image content to the viewing assembly, wherein the image formed by the second image stream has a second pixel density that exceeds the first pixel density. Includes, The image formed by the second image stream corresponds to the portion of the image formed by the first image stream, The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream. A method wherein the other internal coupling grid is arranged along the same surface of the waveguide as the internal coupling grid, the internal coupling grid is located in the path of the second image stream, and the other internal coupling grid is located in the path of the first image stream.

23. The method according to claim 20, wherein the waveguide comprises another internal coupling grating configured to internally couple light to the waveguide.

24. The method according to claim 21 or 23, wherein the other internal coupling grid is arranged along a surface of the waveguide that is different from the internal coupling grid.

25. The method according to any one of claims 20 to 24, wherein the internal bonding grid is made from a liquid crystal material, and the internal bonding grid is configured such that incident light having a predetermined polarization is diffracted by the internal bonding grid, and incident light having at least one other polarization passes through the internal bonding grid.

26. The method according to any one of claims 15 to 25, further comprising a beam splitting optical system for splitting light into a plurality of image streams propagating in different directions.

27. ​​The method according to claim 26, wherein the beam splitting optical system is a polarizing beam splitter.

28. The method according to claim 26, wherein the beam splitting optical system is a switchable reflector, and the switchable reflector is selectively switchable between a reflective state and a transmittance state.

Images