Virtual and augmented reality systems and methods

Abstract

To provide favorable virtual and augmented reality systems and methods.SOLUTION: Methods and systems are disclosed for presenting virtual objects on a limited number of depth planes using, e.g., an augmented reality display system. The farthest one of the depth planes is within a mismatch tolerance of optical infinity. The display system may switch the depth plane on which content is actively displayed, so that the content is displayed on the depth plane which a user is fixating. The impact of errors in fixation tracking is addressed using partially overlapping depth planes. A fixation depth of the fixation by the user is determined, and the display system determines whether to adjust selection of a selected depth plane at which a virtual object is presented. The determination may be based on whether the fixation depth falls within a depth overlap region of adjacent depth planes. The display system may switch the active depth plane depending upon whether the fixation depth falls outside the overlap region.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] (Priority Claim) This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62 / 313,698, filed Mar. 25, 2016, and U.S. Patent Application No. 62 / 378,109, filed Aug. 22, 2016. The disclosure of each of these priority documents is hereby incorporated by reference herein.

[0002] (Cross - References to Related Applications) This application incorporates by reference in their entireties U.S. Application No. 14 / 555,585, filed Nov. 27, 2014; U.S. Application No. 14 / 690,401, filed Apr. 18, 2015; U.S. Application No. 14 / 212,961, filed Mar. 14, 2014; and U.S. Application No. 14 / 331,218, filed Jul. 14, 2014.

[0003] This disclosure relates to display systems, including augmented reality imaging and visualization systems.

Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as being real. A virtual reality or "VR" scenario typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, and an augmented reality or "AR" scenario typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world surrounding the user. A mixed reality or "MR" scenario is a type of AR scenario that typically involves virtual objects that are integrated into and responsive to the natural world. For example, an MR scenario may include AR image content that appears to be blocked by or otherwise interact with objects within the real world.

[0005] Referring to FIG. 1, an augmented reality scene 10 is depicted. To a user of AR technology, a real-world park-like setting 20 featuring people, trees, and buildings in the background, and a concrete platform 30 are visible. The user also "sees" "virtual content" such as a robotic figure 40 standing on the real-world platform 30 and an avatar character 50 in the form of a flying cartoon that appears anthropomorphic like a bumblebee. These elements 50, 40 are "virtual" in that they do not exist in the real world. The generation of AR technology, which is complex and promotes a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements, is difficult for the human visual perception system.

[0006] The systems and methods disclosed herein address various issues related to AR or VR technology. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0007] In some embodiments, the display system comprises a head-mounted display having a hyperfocal distance. The head-mounted displays each comprise one or more waveguide(s) having a refractive power and an associated depth plane. The one or more waveguides are configured to project light to a viewer and display image content on the associated depth plane. Each depth plane is less than optical infinity.

[0008] In some other embodiments, the display system comprises a head-mounted display. The head-mounted displays each comprise one or more waveguide(s) having a refractive power and an associated depth plane. The one or more waveguides are configured to project light to a viewer and display image content on the associated depth plane. The farthest of the depth planes is within about 0.33 diopters of optical infinity.

[0009] In yet other embodiments, the display system comprises a head-mounted display. The head-mounted display comprises a plurality of waveguides forming a stack of waveguides. Each waveguide has a refractive power and an associated depth plane, and the waveguides are configured to project light to a viewer and display image content on the associated depth plane. The farthest of the depth planes is within the mismatch tolerance of optical infinity. The mismatch tolerance is about 0.5 diopters.

[0010] In some other embodiments, a method for displaying image content on a head-mounted display is provided. The method includes determining whether an accommodation-vergence / divergence motion mismatch for the image content exceeds a threshold, and modifying the image content if the accommodation-vergence / divergence motion mismatch exceeds the threshold.

[0011] In yet other embodiments, a method for displaying image content on a head-mounted display worn by a user is provided. The method includes determining the presence of eye strain in the user and, if eye strain is determined to be present, modifying the image content.

[0012] In some embodiments, a wearable head-mounted display system includes a frame configured to be worn by a user. The display is attached to the frame. The display system also includes a support structure configured to extend from one side of the user's head to the other side of the head. The support structure is mechanically coupled to the frame.

[0013] In some embodiments, the display system includes a display device, one or more processors, and a computer storage medium. The display system performs an operation that includes determining a fixation depth, where the fixation depth is the depth at which the user's eye is fixated. The operation also includes determining whether to adjust the selection of a selected depth plane at which a virtual object is presented to the user's eye, based on whether the fixation depth is within a depth plane range that is (1) encompassed only by the selected depth plane or (2) encompassed by both the selected depth plane and an adjacent depth plane. The presentation of the virtual object occurs at the selected particular depth plane based on a determination made in the step of determining whether to adjust the selection of the selected depth plane.

[0014] In some embodiments, the display system comprises a display device, one or more processors, and a computer storage medium. The display system performs operations including determining a fixation depth, where the fixation depth is the depth at which the user's eye is fixated. A particular depth plane of a plurality of depth planes at which a virtual object is to be presented to the user is determined, and the determination is based on the fixation depth and the depth plane range encompassed by each of the depth planes, with adjacent depth planes both encompassing a depth overlap region. Presentation of the virtual object occurs at the particular depth plane.

[0015] In some embodiments, the method includes determining a fixation depth, where the fixation depth is the depth at which the user's eye is fixated. Whether to adjust the selection of a selected depth plane at which a virtual object is presented to the user's eye is determined based on whether the fixation depth is within a depth plane range that is (1) encompassed only by the selected depth plane, or (2) encompassed by both the selected depth plane and an adjacent depth plane. Presentation of the virtual object occurs at the selected particular depth plane based on the determination of whether to adjust the selection of the selected depth plane.

[0016] In some embodiments, the display system comprises a display device, one or more processors, and a computer storage medium. The display system performs operations including determining a user's fixation point indicative of a three-dimensional location at which the user is fixated. The operations also include determining whether to switch the depth plane at which a virtual object is to be presented, and the determination is at least partially based on the depth of the determined fixation point. The depth plane at which the virtual object is to be presented is switched, and the step of switching the depth plane is triggered by a user perception-limiting event.

[0017] Additional examples of embodiments are provided below.

[0018] Example 1: A display system, A head-mounted display, One or more waveguides, each waveguide having a refractive power and configured to provide content on an associated depth plane, the one or more waveguides projecting light to a viewer and configured to display image content on the associated depth plane of the one or more waveguides, the one or more waveguides Comprising, each depth plane being less than optical infinity, a head-mounted display Comprising, a display system.

[0019] Example 2: The display system according to claim 1, wherein the farthest of the depth planes is within about 0.50 diopters of optical infinity.

[0020] Example 3: The display system according to claim 2, wherein the farthest of the depth planes is within about 0.33 diopters of optical infinity.

[0021] Example 4: The display system according to claim 3, wherein the farthest of the depth planes is within about 0.25 diopters of optical infinity.

[0022] Example 5: The display system according to any one of claims 1-4, wherein the depth planes are separated by no more than about 0.7 diopters.

[0023] Example 6: The display system according to any one of claims 1-6, wherein the depth planes are separated by no more than about 0.5 diopters.

[0024] Example 7: The display system according to any one of claims 1-7, wherein the display is configured to display image information on only two depth planes, each of the two depth planes being less than optical infinity.

[0025] Example 8: The display is configured to display image information on only one depth plane, and the one depth plane is less than optically infinite. The display system according to claim 1.

[0026] Example 9: A display system, A head-mounted display, One or more waveguide(s), each waveguide having a refractive power and an associated depth plane, the one or more waveguide(s) being configured to project light to a viewer and display image content on the associated depth plane, comprising one or more waveguide(s). The farthest of the depth planes is within about 0.33 diopters of optically infinite. Head-mounted display Comprising the display system.

[0027] Example 10: The next farthest of the depth planes is within about 0.66 diopters of the farthest of the depth planes. The display system according to claim 10.

[0028] Example 11: The total number of depth planes is two. The display system according to any one of claims 10-11.

[0029] Example 12: The total number of depth planes is more than two, and the separation between the nearest depth planes is less than about 0.66 diopters. The display system according to any one of claims 10-12.

[0030] Example 13: The total number of depth planes is less than four. The display system according to any one of claims 10-13.

[0031] Example 14: One or more waveguide(s) form a stack of waveguides, each waveguide being configured to redirect incident light and propagate it by total internal reflection inside the waveguide, comprising an internally coupled optical element. The display system according to any one of claims 10-13.

[0032] Example 15: The display system according to claim 15, wherein the internal coupling optical element of each waveguide is configured to selectively redirect light having a wavelength corresponding to a single primary color.

[0033] Example 16: The display system according to claim 15, wherein, as seen in a top plan view, the internal coupling optical element of each waveguide is laterally spaced from the internal coupling optical elements of other waveguides.

[0034] Example 17: The display system according to any one of claims 15 - 17, wherein each waveguide further comprises an external coupling optical element configured to redirect light propagating within the waveguide out of the waveguide.

[0035] Example 18: The display system according to any one of claims 15 - 18, further comprising an optical projector system configured to direct image content to the internal coupling optical elements of the waveguides, the optical projector system comprising: a light emitter, a spatial light modulator, and.

[0036] Example 19: The display system according to any one of claims 10 - 19, wherein each waveguide has a refractive power for creating only a single depth plane.

[0037] Example 20: A display system, a head - mounted display, comprising a plurality of waveguides forming a stack of waveguides, each waveguide having a refractive power and being configured to provide content on an associated depth plane, the waveguides being configured to project light to a viewer and display image content on an associated depth plane, wherein the farthest of the depth planes is within an aberration tolerance of optical infinity, the aberration tolerance being about 0.5 diopter. A head - mounted display comprising.

[0038] Example 21: The display system according to claim 21, wherein the mismatch tolerance is about 0.33 diopter.

[0039] Example 22: The display system according to any one of claims 21-22, wherein the separation between the associated depth plane of the stack and the nearest associated depth plane is about 2 times or less than the mismatch tolerance.

[0040] Example 23: The display system according to any one of claims 21-23, wherein the total number of depth planes is 4 or less.

[0041] Example 24: The display system according to claim 24, wherein the total number of depth planes is 2.

[0042] Example 25: A method for displaying image content on a head-mounted display, the method comprising: determining whether the perspective adjustment-convergence / divergence motion mismatch for the image content exceeds a threshold; when the perspective adjustment-convergence / divergence motion mismatch exceeds the threshold, modifying the image content. The method includes these steps.

[0043] Example 26: The method according to claim 26, wherein the perspective adjustment-convergence / divergence motion mismatch threshold is 0.5 diopter or less.

[0044] Example 27: The method according to claim 27, wherein the perspective adjustment-convergence / divergence motion mismatch threshold is 0.33 diopter or less.

[0045] Example 28: The method according to any one of claims 26-28, wherein the step of modifying the image content includes fading the image content.

[0046] Example 29: The method according to claim 29, wherein the step of fading the image content includes reducing the resolution of the image content.

[0047] Example 30: The method according to claim 30, wherein reducing the resolution of the image content increases with an increase in the far - near adjustment - vergence - divergence motion inconsistency.

[0048] Example 31: The method according to any one of claims 26 - 31, wherein the step of modifying the image content includes the step of not displaying the image content.

[0049] Example 32: A display system, a processor, and a computer storage medium storing instructions that, when executed by the display system, cause the display system to perform operations including the method according to any one of claims 26 - 32. A display system comprising the same.

[0050] Example 33: The display system according to claim 33, wherein the display system is configured to display image information on only one depth plane, and the one depth plane is less than optical infinity.

[0051] Example 34: The display system according to claim 33, wherein the display system is configured to display image information on only two depth planes, and each of the two depth planes is less than optical infinity.

[0052] Example 35: A method for displaying image content on a head - mounted display worn by a user, the method comprising: determining the presence of eye fatigue in the user; and when it is determined that eye fatigue is present, modifying the image content. A method comprising the same.

[0053] Example 36: The method according to claim 36, wherein the step of determining the presence of eye fatigue includes imaging one or both eyes of the user.

[0054] Example 37: The method according to any one of claims 36 - 37, wherein the step of determining the presence of eye fatigue includes the step of detecting one or more of mydriasis, convergence nystagmus, and pupil nystagmus.

[0055] Example 38: The method according to any one of claims 36 - 38, wherein the step of determining the presence of eye fatigue includes the step of measuring the galvanic skin response.

[0056] Example 39: The method according to any one of claims 36 - 39, wherein the step of determining the presence of eye fatigue includes the step of detecting the duration of exposure to image content having a vergence - accommodation - divergence movement inconsistency greater than 0.25 diopters.

[0057] Example 40: The vergence - accommodation - divergence movement inconsistency is greater than 0.33 diopters, according to the method of claim 40.

[0058] Example 41: The vergence - accommodation - divergence movement inconsistency is greater than 0.50 diopters, according to the method of claim 41.

[0059] Example 42: The step of modifying the image content includes increasing the size of the features of the image content, reducing the resolution of the image content, displaying the image content on a depth plane farther from the viewer than originally defined for the image content, and includes one or more of the above, according to the method of any one of claims 36 - 42.

[0060] Example 43: The step of modifying the image content is carried out until the presence of eye fatigue is no longer detected in the user, according to the method of any one of claims 36 - 43.

[0061] Example 44: The method according to any one of claims 36 - 44, wherein the step of modifying the image content is performed over a set duration.

[0062] Example 45: A display system comprising a processor and a computer storage medium storing instructions that, when executed by the display system, cause the display system to perform the method according to any one of claims 36 - 0.

[0063] Example 46: The display system according to claim 45, wherein the display system is configured to display image information on only one depth plane, and the one depth plane is less than optical infinity.

[0064] Example 47: The display system according to claim 45, wherein the display system is configured to display image information on only two depth planes, and each of the two depth planes is less than optical infinity.

[0065] Example 48: A wearable head - mounted display system, a frame configured to be worn by a user, a display attached to the frame, and a support structure configured to extend from one side of the user's head to the other side of the head, wherein the support structure is mechanically coupled to the frame.

[0066] Example 49: The wearable head - mounted display system according to claim 48, further comprising a sound transducer attached to the support structure and configured to direct sound into the user's ear.

[0067] Example 50: The wearable head - mounted display system according to claim 49, wherein the sound transducer is a speaker.

[0068] Example 51: The wearable head-mounted display system according to any one of claims 48-50, wherein the support structure is a band configured to extend from one side of the head to the other side of the head.

[0069] Example 52: The wearable head-mounted display system according to claim 51, wherein the band crosses the user's head at an angle of 35° to 55° with respect to a plane intersecting the user's eyes and ears.

[0070] Example 53: The wearable head-mounted display system according to claim 51, wherein the band crosses the user's head at an angle of 80° to 100° with respect to a plane intersecting the user's eyes and ears.

[0071] Example 54: The wearable head-mounted display system according to any one of claims 52-53, wherein the angle of the band with respect to a plane intersecting the user's eyes and the first and second ears is adjustable.

[0072] Example 55: An optical waveguide, an internal coupling optical element configured to selectively internally couple incident light into the optical waveguide based on the properties of the incident light, and an external coupling optical element configured to project light onto the viewer's eyes by externally coupling the light internally coupled into the optical waveguide comprising the optical waveguide The wearable head-mounted display system according to any one of claims 48-54, further comprising the optical waveguide.

[0073] Example 56: The wearable head-mounted display system according to claim 55, wherein the external coupling optical element has a refractive power and is configured to project light onto the eyes and display image content on an associated depth plane, and the associated depth plane is less than optical infinity.

[0074] Example 57: The waveguide is part of a stack of waveguides, and at least some of the waveguides in the stack of waveguides each have different associated depth planes and external coupling optical elements with different refractive powers so as to provide different divergences of the emitted light for at least some of the waveguides, the wearable head-mounted display system according to claim 56.

[0075] Example 58: A display system, A display device configured to present virtual objects to a user at a plurality of depth planes, One or more processors, A computer storage medium storing instructions that, when executed by the display system, cause the display system to Determine a fixation depth, where the fixation depth is the depth at which the user's eye is fixated, Based on whether the fixation depth is (1) Contained only by the selected depth plane, or (2) Contained by both the selected depth plane and an adjacent depth plane, Determine whether to adjust the selection of the selected depth plane on which the virtual object is presented to the user's eye based on whether it is within the depth plane range, Cause the presentation of the virtual object to occur on the selected specific depth plane based on the determination of whether to adjust the selection of the selected depth plane, And a computer storage medium for performing operations including A display system comprising.

[0076] Example 59: The fixation depth is within a depth plane range contained only by the selected depth plane, and the determination of whether to adjust the selection of the depth plane is negative, the display system according to claim 58.

[0077] Example 60: The fixation depth is within the depth plane range encompassed by both the selected depth plane and the adjacent depth plane, and the determination of whether to adjust the selection of the depth plane is negative. The display system according to claim 58.

[0078] Example 61: The fixation depth is (1) encompassed only by the selected depth plane, (2) within a specific depth plane outside the depth plane range encompassed by both the selected depth plane and the adjacent depth plane, and the determination of whether to adjust the selection of the depth plane is positive. The display system according to claim 58.

[0079] Example 62: The virtual object is presented on the depth plane that encompasses a specific depth plane range. The display system according to claim 61.

[0080] Example 63: The fixation depth is within the depth plane range encompassed only by the adjacent depth plane, and the determination of whether to adjust the selection of the depth plane is positive. The display system according to claim 58.

[0081] Example 64: The virtual object is presented on the adjacent depth plane. The display system according to claim 63.

[0082] Example 65: In response to a positive determination to adjust the selection of the depth plane, and in response to detecting the user's implementation of a blink or saccadic eye movement, trigger the presentation on the adjusted depth plane. The display system according to claim 58.

[0083] Example 66: The operation further comprises a step of determining the user's fixation point indicating the three-dimensional location at which the user is fixating, wherein the three-dimensional location indicates the fixation depth. The display system according to claim 58.

[0084] Example 67: A display system, A display device configured to present virtual objects to a user in a plurality of depth planes, One or more processors, A computer storage medium storing instructions that, when executed by the display system, cause the display system to, Determine a fixation depth, where the fixation depth is the depth at which the user's eye is fixated, Determine a particular depth plane of the plurality of depth planes in which the virtual object is to be presented to the user, the determination being based on the fixation depth and the depth plane ranges encompassed by each of the depth planes, and adjacent depth planes both encompassing a depth overlap region, Cause the presentation of the virtual object to occur in the particular depth plane And a computer storage medium that causes the performance of operations including A display system comprising.

[0085] Example 68: The depth plane range encompassed by the depth plane indicates, when fixated, the range of depth from the user at which the presentation of the virtual object occurs in the depth plane, the display system according to claim 66.

[0086] Example 69: The presentation in the particular depth plane includes presenting the virtual object with a perspective adjustment cue associated with the nominal focal depth of the particular depth plane and a convergence / divergence motion cue based on the location information associated with the virtual object, the display system according to claim 66.

[0087] Example 70: The size of the depth plane range encompassed by the particular depth plane is based on a perspective adjustment - convergence / divergence motion mismatch tolerance, where the perspective adjustment - convergence / divergence motion mismatch tolerance indicates the maximum difference between the perceived depth associated with the convergence / divergence motion cue of the presented virtual object and the perceived depth associated with the perspective adjustment cue of the virtual object, the display system according to claim 69.

[0088] Example 71: The display system according to claim 66, wherein the size of the depth overlap region is based on an error associated with the step of determining the fixation depth.

[0089] Example 72: The display system according to claim 66, wherein the fixation depth is included only by a specific depth plane such that the depth is within the depth plane range.

[0090] Example 73: The display system according to claim 66, wherein the fixation depth is within a depth overlap region included by a specific depth plane and an adjacent depth plane, and the step of determining the specific depth plane is based on the user's previous fixation depth.

[0091] Example 74: The operation further includes the step of identifying that the user has fixated on a fixation depth included only by a specific depth plane prior to fixating on one or more fixation depths within the depth overlap region, based on the previous fixation depth. The display system according to claim 73.

[0092] Example 75: The operation further includes the step of determining a subsequent fixation depth of the user within the depth overlap region, and the step of causing the presentation of a virtual object to occur at the specific depth plane. The display system according to claim 73. Example 76: The operation further

[0093] includes the step of determining a subsequent fixation depth of the user that is included by an adjacent depth plane and outside the depth overlap region, and the step of causing the presentation of a virtual object to occur at the adjacent depth plane. The display system according to claim 73.

[0094] ​​Example 77: The display system according to claim 76, configured such that when the user performs (1) a blink or (2) a saccadic eye movement, a presentation is caused in an adjacent depth plane in response thereto.

[0095] Example 78: A method implemented by a display device configured to present a virtual object to a user in a plurality of depth planes, at least in part, the method comprising: determining a fixation depth, the fixation depth being the depth at which the user's eye is fixated; if the fixation depth is (1) included only by a selected depth plane, or (2) included by both the selected depth plane and an adjacent depth plane, determining whether to adjust the selection of the selected depth plane at which the virtual object is presented to the user's eye based on whether it is within the depth plane range; causing the presentation of the virtual object to occur at the selected specific depth plane based on the determination of whether to adjust the selection of the selected depth plane; and including a method.

[0096] Example 79: The method according to claim 78, wherein the fixation depth is (1) included only by a selected depth plane, (2) included by both the selected depth plane and an adjacent depth plane, within a specific depth plane outside the depth plane range, and the determination of whether to adjust the selection of the depth plane is positive.

[0097] Example 80: The method according to claim 78, wherein in response to a positive determination to adjust the selection of the depth plane, and in response to detection of the user performing a blink or a saccadic eye movement, a presentation at the adjusted depth plane is triggered.

[0098] Example 81: A display system, A display device configured to present virtual objects to a user in a plurality of depth planes, a processor, a computer storage medium storing instructions that, when executed by the display system, cause the display system to, determine a user's fixation point indicating a three-dimensional location on which the user is fixating, determine whether to switch a depth plane on which a virtual object is to be presented, the determination being based at least in part on the depth of the determined fixation point, switch a depth plane on which a virtual object is to be presented, the step of switching the depth plane being triggered by a user perception-limited event, and a computer storage medium that causes the performance of operations including, A display system comprising:

[0099] Example 82: The operations further include detecting a user perception-limited event including monitoring the user's eyes and including the performance of one or more of a blink or saccadic eye movement, the display system of claim 81.

[0100] Example 83: The step of monitoring the user's eyes includes monitoring the user's pupil, and the step of detecting a saccadic eye movement is based on the rotational speed of the pupil exceeding a threshold speed, the display system of claim 82.

[0101] Example 84: The step of detecting a saccadic eye movement is further based on movement information associated with the user's head, the display system of claim 83.

[0102] Example 85: The step of switching the depth plane includes detecting a blink being performed by the user, and in response thereto, switching the depth plane. The display system according to claim 81, comprising

[0103] Example 86: The step of switching the depth plane includes detecting impulsive eye movements performed by the user, and in response thereto, switching the depth plane The display system according to claim 81, comprising

[0104] Example 87: The step of switching the depth plane includes switching the depth plane in response to detecting no blinking or impulsive eye movement after a threshold amount of time. The display system according to claim 81, comprising

[0105] Example 88: The step of determining whether to switch the depth plane includes determining that the depth of the determined fixation point is encompassed by the switched depth plane. The display system according to claim 81, comprising

[0106] Example 89: The operation further includes storing information indicating that the depth plane is to be switched, monitoring the user's eyes, and determining a user-perceived limited event. The display system according to claim 81, comprising

[0107] Example 90: The display device includes a plurality of stacked waveguides that form a display area and provide a view of the surrounding environment through the display area. At least some of the plurality of waveguides are configured to output light with a different wavefront divergence from other waveguides, and each waveguide is associated with a depth plane. The step of presenting the virtual object on the switched depth plane includes the waveguide associated with the switched depth plane outputting light for forming the virtual object. The display system according to claim 81, comprising

[0108] Example 91: A method implemented by a display device configured to present virtual objects to a user at least partially in a plurality of depth planes, the method comprising: determining a user's fixation point indicating a three-dimensional location at which the user is fixating; determining whether to switch a depth plane at which a virtual object is to be presented, the determining being based at least in part on the depth of the determined fixation point; and switching a depth plane at which a virtual object is to be presented, the switching of the depth plane being triggered by a user perception-limiting event A method comprising the steps of:

[0109] Example 92: The method according to claim 91, further comprising detecting a user perception-limiting event including one or more of a blink or an impulsive eye movement of the user's eye being performed.

[0110] Example 93: The method according to claim 92, wherein the step of monitoring the user's eye includes monitoring the user's pupil, and the step of detecting an impulsive eye movement is based on the rotational speed of the pupil exceeding a threshold speed.

[0111] Example 94: The method according to claim 93, wherein the step of detecting an impulsive eye movement is further based on movement information associated with the user's head.

[0112] Example 95: The step of switching a depth plane includes: detecting a blink being performed by the user; and in response thereto, switching a depth plane. The method according to claim 91, comprising the steps of:

[0113] Example 96: The step of switching a depth plane includes: detecting an impulsive eye movement being performed by the user; and In response thereto, a step of switching the depth plane; The method according to claim 91, comprising:

[0114] Example 97: A display system, the display system comprising a display device, a processor, and a computer storage medium storing instructions, the instructions, when executed by the processor, cause the display system to Presenting, by a display device, frames including virtual content to a user in a plurality of depth planes, wherein for each frame presented to the user, the virtual content is presented in the same depth plane selected based on user gaze information; In response to identifying that the selection of the selected depth plane should be adjusted, storing information indicating that one or more frames of virtual content should be presented in an adjusted depth plane by the display device in response to detecting a blink or saccadic eye movement performed by the user; A display system that causes an operation including:

[0115] Example 98: The operation further includes Detecting the performance of a blink or saccadic eye movement; In response to the determination, presenting the virtual content in an adjusted depth plane; The display system according to claim 97, comprising:

[0116] Example 99: The operation further includes Determining that the user has not performed a blink or saccadic eye movement for a time exceeding a threshold time; In response to the determination, presenting the virtual content in an adjusted depth plane; The display system according to claim 97, comprising:

[0117] Example 100: The operation further includes A display system according to claim 97, comprising the steps of monitoring a user's eyes, detecting a blink or a saccadic eye movement, and presenting one or more frames of virtual content on a selected depth plane while monitoring.

[0118] Example 101: While waiting for the detection of a blink or a saccadic eye movement, determine that the user is fixating on a depth associated with a specific depth plane different from the adjusted depth plane, and in response to the detection of a blink or a saccadic eye movement, store information indicating that virtual content should be presented on the specific depth plane. A display device according to claim 97. The present invention provides, for example, the following. (Item 1) A display system, A head-mounted display, One or more waveguide tubes, each waveguide tube having a refractive power and configured to provide content on an associated depth plane, the one or more waveguide tubes projecting light to a viewer and configured to display image content on the associated depth plane of the one or more waveguide tubes Comprising a head-mounted display, each of the depth planes being less than optical infinity. A display system comprising the same. (Item 2) The display system according to item 1, wherein the farthest of the depth planes is within about 0.50 diopters of optical infinity. (Item 3) The display system according to item 2, wherein the farthest of the depth planes is within about 0.33 diopters of optical infinity. (Item 4) The display system according to item 3, wherein the farthest of the depth planes is within about 0.25 diopters of optical infinity. (Item 5) The display system according to item 1, wherein the farthest of the depth planes is at optical infinity. (Item 6) The display system according to item 1, wherein the depth planes are separated by about 0.7 diopters or less. (Item 7) The display system according to item 6, wherein the depth planes are separated by about 0.5 diopters or less. (Item 8) The display system according to item 1, wherein the display is configured to display image information on only two depth planes, and each of the two depth planes is less than optical infinity. (Item 9) The display system according to item 1, wherein the display is configured to display image information on only one depth plane, and the one depth plane is less than optical infinity. (Item 10) A display system, A head-mounted display, One or more waveguide tubes, each waveguide tube having a refractive power and an associated depth plane, the one or more waveguide tubes being configured to project light to a viewer and display image content on the associated depth plane. Comprising a head-mounted display, wherein the farthest of the depth planes is within about 0.33 diopters of optical infinity. A display system comprising the same. (Item 11) The display system according to item 10, wherein the next farthest of the depth planes is within about 0.66 diopters of the farthest of the depth planes. (Item 12) The display system according to item 10, wherein the total number of depth planes is two. (Item 13) The display system according to item 10, wherein the total number of depth planes is more than two, and the separation between the nearest depth planes is less than about 0.66 diopters. (Item 14) The display system according to item 10, wherein the total number of depth planes is less than 4. (Item 15) The display system according to item 10, wherein the one or more waveguides form a stack of waveguides, and each waveguide comprises an internal coupling optical element configured to redirect incident light and propagate it by total internal reflection inside the waveguide. (Item 16) The display system according to item 15, wherein the internal coupling optical element of each waveguide is configured to selectively redirect light having a wavelength corresponding to a single primary color. (Item 17) The display system according to item 15, wherein as seen in a top and bottom plan view, the internal coupling optical element of each waveguide is laterally spaced from the internal coupling optical elements of other waveguides. (Item 18) The display system according to item 15, wherein each waveguide further comprises an external coupling optical element configured to redirect light propagating within each waveguide out of the waveguide. (Item 19) The display system according to item 15, further comprising an optical projector system configured to direct image content to the internal coupling optical elements of the waveguides, the optical projector system comprising: a light emitter, a spatial light modulator and. (Item 20) The display system according to item 10, wherein each waveguide has a refractive power for creating only a single depth plane. (Item 21) A display system, a head-mounted display, a plurality of waveguides forming a stack of waveguides, each waveguide having a refractive power and being configured to provide content on an associated depth plane, the waveguides being configured to project light to a viewer and display image content on the associated depth plane. A head-mounted display comprising, wherein the farthest of the depth planes is within the accommodation infinity mismatch tolerance, and the mismatch tolerance is about 0.5 diopters. A display system comprising. (Item 22) The display system according to Item 21, wherein the mismatch tolerance is about 0.33 diopters. (Item 23) The display system according to Item 21, wherein the separation between the associated depth plane of the stack and the nearest associated depth plane is about twice or less than the mismatch tolerance. (Item 24) The display system according to Item 21, wherein the total number of depth planes is 4 or less. (Item 25) The display system according to Item 24, wherein the total number of the depth planes is 2. (Item 26) A method for displaying image content on a head-mounted display, the method comprising: Determining whether the accommodation-convergence / divergence movement mismatch for the image content exceeds a threshold; When the accommodation-convergence / divergence movement mismatch exceeds the threshold, modifying the image content; A method including. (Item 27) The method according to Item 26, wherein the accommodation-convergence / divergence movement mismatch threshold is 0.5 diopters or less. (Item 28) The method according to Item 27, wherein the accommodation-convergence / divergence movement mismatch threshold is 0.33 diopters or less. (Item 29) The method according to Item 26, wherein the step of modifying the image content includes fading the image content. (Item 30) The method according to Item 29, wherein the step of fading the image content includes reducing the resolution of the image content. (Item 31) The method according to item 30, wherein the reduction in the resolution of the image content increases as the increase in the far - near adjustment - vergence / divergence movement inconsistency increases. (Item 32) The method according to item 26, wherein the step of modifying the image content includes the step of not displaying the image content. (Item 33) A display system, a processor, a computer storage medium storing instructions, which, when executed by the display system, cause the display system to perform operations including the method according to any one of items 26 - 32 a computer storage medium comprising the display system. (Item 34) The display system according to item 33, wherein the display system is configured to display image information on only one depth plane, and the one depth plane is less than optical infinity. (Item 35) The display system according to item 33, wherein the display system is configured to display image information on only two depth planes, and each of the two depth planes is less than optical infinity. (Item 36) A method for displaying image content on a head - mounted display worn by a user, the method comprising: determining the presence of eye strain in the user; when it is determined that eye strain is present, modifying the image content and. (Item 37) The method according to item 36, wherein the step of determining the presence of eye strain includes imaging one or both eyes of the user. (Item 38) The method according to item 36, wherein the step of determining the presence of eye fatigue includes the step of detecting one or more of mydriasis, convergence fluctuation, and pupil fluctuation. (Item 39) The method according to item 36, wherein the step of determining the presence of eye fatigue includes the step of measuring the galvanic skin response. (Item 40) The method according to item 36, wherein the step of determining the presence of eye fatigue includes the step of detecting the duration of exposure to image content having a vergence - accommodation - divergence movement inconsistency exceeding 0.25 diopters. (Item 41) The vergence - accommodation - divergence movement inconsistency exceeds 0.33 diopters, according to the method of item 40. (Item 42) The vergence - accommodation - divergence movement inconsistency exceeds 0.50 diopters, according to the method of item 41. (Item 43) The step of modifying the image content includes the step of increasing the size of the features of the image content, the step of reducing the resolution of the image content, the step of displaying the image content on a depth plane farther from the viewer than originally defined for the image content, and includes one or more of the above, according to the method of item 36. (Item 44) The step of modifying the image content is performed until the presence of the eye fatigue is no longer detected in the user, according to the method of item 43. (Item 45) A display system comprising a processor and a computer storage medium storing instructions that, when executed by the display system, cause the display system to perform the method according to any one of items 36 - 45. (Item 46) The display system is configured to display image information on only one depth plane, and the one depth plane is less than optically infinite. The display system according to item 45. (Item 47) The display system is configured to display image information on only two depth planes, and the two depth planes are each less than optically infinite. The display system according to item 45. (Item 48) A wearable head-mounted display system, A frame configured to be worn by a user, A display attached to the frame, A support structure configured to extend from one side of the user's head to the other side of the head, and comprising, The support structure is mechanically coupled to the frame. The wearable head-mounted display system. (Item 49) The wearable head-mounted display system according to item 48, further comprising a sound transducer attached to the support structure and configured to direct sound into the user's ear. (Item 50) The sound transducer is a speaker. The wearable head-mounted display system according to item 49. (Item 51) The support structure is a band configured to extend from one side of the head to the other side of the head. The wearable head-mounted display system according to item 48. (Item 52) The band crosses the user's head at an angle of 35° to 55° with respect to a plane intersecting the user's eyes and ears. The wearable head-mounted display system according to item 51. (Item 53) The band crosses the user's head at an angle of 80° to 100° with respect to a plane intersecting the user's eyes and ears. The wearable head-mounted display system according to item 51. (Item 54) The wearable head-mounted display system according to item 51, wherein an angle of the band with respect to a plane intersecting the user's eyes and the first and second ears is adjustable. (Item 55) A waveguide, an internal coupling optical element configured to selectively internally couple incident light into the waveguide based on properties of the incident light; an external coupling optical element configured to project the light onto the viewer's eyes by externally coupling the light internally coupled into the waveguide; and a waveguide comprising the same. The wearable head-mounted display system according to item 48, further comprising the same. (Item 56) The wearable head-mounted display system according to item 55, wherein the external coupling optical element has a refractive power and is configured to project light onto the eyes and display image content on an associated depth plane, and the associated depth plane is less than optical infinity. (Item 57) The waveguide is part of a stack of waveguides, and at least some of the waveguides in the stack of waveguides each have a different associated depth plane and an external coupling optical element having a different refractive power so as to provide different divergences of emitted light. The wearable head-mounted display system according to item 56. (Item 58) A display system, a display device configured to present virtual objects to a user at a plurality of depth planes; one or more processors; a computer storage medium storing instructions that, when executed by the display system, cause the display system to determine a fixation depth, the fixation depth being the depth at which the user's eyes are fixated; the fixation depth is (1) included only by a selected depth plane, or (2) included by both the selected depth plane and an adjacent depth plane, determining whether to adjust the selection of the selected depth plane based on whether the virtual object is within the depth plane range included by the selected depth plane; causing the presentation of the virtual object to occur on the selected specific depth plane based on the step of determining whether to adjust the selection of the selected depth plane; A computer storage medium for implementing operations, including A display system comprising (Item 59) The display system according to item 58, wherein the fixation depth is within a depth plane range included only by the selected depth plane, and the determination of whether to adjust the selection of the depth plane is negative. (Item 60) The display system according to item 58, wherein the fixation depth is within a depth plane range included by both the selected depth plane and an adjacent depth plane, and the determination of whether to adjust the selection of the depth plane is negative. (Item 61) The fixation depth is (1) included only by the selected depth plane, (2) included by both the selected depth plane and an adjacent depth plane, within a specific depth plane range outside the depth plane range, and the determination of whether to adjust the selection of the depth plane is positive. The display system according to item 58. (Item 62) The display system according to item 61, wherein the virtual object is presented on a depth plane including the specific depth plane range. (Item 63) The fixation depth is within a depth plane range that is encompassed only by the adjacent depth planes, and the determination of whether to adjust the selection of the depth plane is positive. The display system according to item 58. (Item 64) The virtual object is presented on the adjacent depth planes. The display system according to item 63. (Item 65) In response to a positive determination to adjust the selection of the depth plane, and in response to detecting the user's blink or involuntary eye movement, trigger the presentation on the adjusted depth plane. The display system according to item 58. (Item 66) The operation further Determining a user's fixation point indicating a three-dimensional location at which the user is fixating, wherein the three-dimensional location indicates the fixation depth. Step Including the display system according to item 58. (Item 67) A display system, A display device configured to present a virtual object to a user at a plurality of depth planes, One or more processors, A computer storage medium storing instructions that, when executed by the display system, cause the display system to Determine a fixation depth, wherein the fixation depth is the depth at which the user's eye is fixating. Step Determine a specific depth plane of the plurality of depth planes at which the virtual object is to be presented to the user, wherein the determination is based on the fixation depth and a depth plane range encompassed by each of the depth planes, and adjacent depth planes both include a depth overlap region. Step Cause the presentation of the virtual object to occur at the specific depth plane And a computer storage medium for performing operations including Comprising a display system. (Item 68) The depth plane range included by the depth plane, when gazed at, indicates the depth range from the user that causes the presentation of the virtual object in the depth plane, for the display system according to item 66. (Item 69) The presentation in the specific depth plane includes the step of presenting the virtual object together with a perspective adjustment cue associated with the nominal focal depth of the specific depth plane and a convergence / divergence movement cue based on the location information associated with the virtual object, for the display system according to item 66. (Item 70) The size of the depth plane range included by the specific depth plane is based on a perspective adjustment - convergence / divergence movement inconsistency tolerance, and the perspective adjustment - convergence / divergence movement inconsistency tolerance indicates the maximum difference between the perceived depth associated with the convergence / divergence movement cue of the presented virtual object and the perceived depth associated with the perspective adjustment cue of the virtual object, for the display system according to item 69. (Item 71) The size of the depth overlap region is based on an error associated with the step of determining the fixation depth, for the display system according to item 66. (Item 72) The fixation depth is only included by the specific depth plane such that the depth is within the depth plane range, for the display system according to item 66. (Item 73) The fixation depth is within the depth overlap region included by the specific depth plane and adjacent depth planes, and the step of determining the specific depth plane is based on the user's previous fixation depth, for the display system according to item 66. (Item 74) The operation further includes the step of identifying that the user has gazed at a fixation depth that is only included by the specific depth plane prior to gazing at one or more fixation depths within the depth overlap region, based on the previous fixation depth, for the display system according to item 73. (Item 75) The operation further includes determining a subsequent fixation depth of the user within the depth overlap region; and causing the presentation of the virtual object to occur at the specific depth plane The display system according to item 73, including the above. (Item 76) The operation further includes determining a subsequent fixation depth of the user, which is included by the adjacent depth planes and outside the depth overlap region; and causing the presentation of the virtual object to occur at the adjacent depth plane The display system according to item 73, including the above. (Item 77) The display system according to item 76, wherein the display system is configured to determine that the user has performed (1) a blink or (2) a saccade, and in response, cause the presentation to occur at the adjacent depth plane. (Item 78) A method implemented by a display device configured to at least partially present a virtual object to a user at a plurality of depth planes, the method including determining a fixation depth, which is the depth at which the user's eye is fixated; determining whether to adjust the selection of a selected depth plane at which the virtual object is presented to the user's eye based on whether the fixation depth is (1) included only by the selected depth plane, or (2) within a depth plane range included by both the selected depth plane and an adjacent depth plane; causing the presentation of the virtual object to occur at the selected specific depth plane based on the determination of whether to adjust the selection of the selected depth plane The method including the above. (Item 79) The fixation depth is (1) included only by the selected depth plane, (2) included by both the selected depth plane and an adjacent depth plane, within a specific depth plane outside the depth plane range, and the determination of whether to adjust the selection of the depth plane is positive, the method according to item 78. (Item 80) In response to a positive determination to adjust the selection of the depth plane, in response to detecting the performance of a blink or saccadic eye movement by the user, triggering the presentation in the adjusted depth plane, the method according to item 78. (Item 81) A display system, A display device configured to present virtual objects to a user in a plurality of depth planes, A processor, A computer storage medium storing instructions, which, when executed by the display system, cause the display system to Determine a user's fixation point indicating a three-dimensional location at which the user is fixating, Determine whether to switch the depth plane in which the virtual object is to be presented, the determination being at least partially based on the depth of the determined fixation point, the step, Switch the depth plane in which the virtual object is to be presented, the step of switching the depth plane being triggered by a user-perception-limited event, the step And perform operations including, a computer storage medium Comprising a display system. (Item 82) The operations further include detecting the user-perception-limited event including monitoring the user's eyes and detecting that one or more than one of a blink or saccadic eye movement is performed, the display system according to item 81. (Item 83) The step of monitoring the user's eyes includes the step of monitoring the user's pupils, and the step of detecting saccadic eye movements is based on the rotational speed of the pupils exceeding a threshold speed, according to the display system of item 82. (Item 84) The step of detecting the saccadic eye movements further includes, based on movement information associated with the user's head, according to the display system of item 83. (Item 85) The step of switching the depth plane includes the step of detecting a blink being performed by the user and, in response thereto, the step of switching the depth plane according to the display system of item 81. (Item 86) The step of switching the depth plane includes the step of detecting saccadic eye movements being performed by the user and, in response thereto, the step of switching the depth plane according to the display system of item 81. (Item 87) The step of switching the depth plane includes the step of switching the depth plane in response to not detecting a blink or saccadic eye movements after a threshold amount of time according to the display system of item 81. (Item 88) The step of determining whether to switch the depth plane includes the step of determining that the depth of the determined fixation point is encompassed by the switched depth plane according to the display system of item 81. (Item 89) The operation further includes the step of storing information indicating that the depth plane is to be switched, monitoring the user's eyes, and determining the user perception-limiting event according to the display system of item 81. (Item 90) The display device includes a plurality of stacked waveguides that form a display area and provide a view of the surrounding environment through the display area. At least some of the plurality of waveguides are configured to output light with a different wavefront divergence from other waveguides. Each waveguide is associated with a depth plane, The step of presenting the virtual object on the switched depth plane includes the step of the waveguide associated with the switched depth plane outputting light for forming the virtual object. The display system according to Item 81. (Item 91) A method implemented by a display device configured to at least partially present a virtual object to a user at a plurality of depth planes, the method comprising: determining a user's fixation point indicating a three-dimensional location at which the user is fixating; determining whether to switch a depth plane at which the virtual object is to be presented, the determination being based at least in part on the depth of the determined fixation point; switching a depth plane at which the virtual object is to be presented, the step of switching the depth plane being triggered by a user perception-limiting event; and including. (Item 92) The method according to Item 91, further comprising detecting the user perception-limiting event including one or more of a blink or a saccadic eye movement by monitoring the user's eyes. (Item 93) The step of monitoring the user's eyes includes monitoring the user's pupil, and the step of detecting a saccadic eye movement is based on the rotational speed of the pupil exceeding a threshold speed. The method according to Item 92. (Item 94) The step of detecting the saccadic eye movement further includes the method according to item 93 based on movement information associated with the user's head. (Item 95) The step of switching the depth plane includes detecting a blink being performed by the user, and in response thereto, switching the depth plane, and includes the method according to item 91. (Item 96) The step of switching the depth plane includes detecting a saccadic eye movement being performed by the user, and in response thereto, switching the depth plane, and includes the method according to item 91. (Item 97) A display system, the display system comprising a display device, a processor, and a computer storage medium storing instructions, the instructions, when executed by the processor, cause the display system to present, by the display device, frames including virtual content to a user in a plurality of depth planes, wherein for each frame presented to the user, the virtual content is presented in the same depth plane selected based on user fixation information; store information indicating that one or more frames of the virtual content should be presented by the display device in the adjusted depth plane in response to detecting a blink or saccadic eye movement being performed by the user in response to an identification that the selection of the selected depth plane should be adjusted; and perform operations including. (Item 98) The operations further include detecting the performance of a blink or saccadic eye movement, and in response to the determination, presenting the virtual content in the adjusted depth plane. The display system according to item 97, including (Item 99) The operation further includes Determining that the user has not performed a blink or a saccadic eye movement for a time exceeding a threshold time; In response to the determination, presenting the virtual content on the adjusted depth plane; The display system according to item 97, including (Item 100) The operation further includes Monitoring the user's eyes to detect a blink or a saccadic eye movement, and while monitoring, presenting one or more frames of virtual content on the selected depth plane; The display system according to item 97, including (Item 101) Determining that the user is gazing at a depth associated with a specific depth plane different from the adjusted depth plane while waiting for detection of a blink or a saccadic eye movement, and storing information indicating that the virtual content should be presented on the specific depth plane in response to detection of a blink or a saccadic eye movement. The display device according to item 97.

Brief Description of Drawings

[0119]

Figure 1

[0120]

Figure 2

[0121]

Figure 3

[0122]

Figure 4A

[0123]

Figure 4B

[0124]

Figure 5

[0125]

Figure 6

[0126]

Figure 7

[0127]

Figure 8

[0128]

Figure 9A

[0129]

Figure 9B

[0130]

Figure 9C

[0131]

Figure 9D

[0132]

Figure 10

[0133]

Figure 11

[0134]

Figure 12

[0135]

Figure 13

[0136]

Figure 14

[0137]

Figure 15

[0138]

Figure 16

[0139]

Figure 17

[0140]

Figure 18A

Figure 18B

[0141]

Figure 19

[0142]

Figure 20

[0143]

Figure 21A

[0144]

Figure 21B

[0145]

Figure 22A

[0146]

Figure 22B

[0147] The drawings are provided to illustrate exemplary embodiments and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0148] Virtual and augmented display systems can provide various image contents, the richness of which can increase with the capabilities of a user wearing the system over a long period of time. For example, an augmented display system can replace a conventional display (e.g., a computer monitor, a smartphone display, etc.) and a single device that can also expand the perception of users around the world by providing content that would otherwise not be available. However, these display systems can be bulky and / or heavy, and some image contents displayed on the system can impair long-term user comfort. For example, some display systems as discussed herein utilize a stack of waveguides to project image information across multiple depth planes to a user, thereby providing a three-dimensional viewing experience. Such a stack of waveguides can be heavy, which is undesirable for the long-term use of a display system incorporating such a stack.

[0149] Advantageously, in some embodiments, systems and methods are provided that can promote long-term wearer comfort. In some embodiments, a realistic and comfortable three-dimensional viewing experience is provided using a reduced stack of waveguides configured to project image information to the user in only one or only two depth planes. In some embodiments, the number of depth planes can be more, including three or four depth planes.

[0150] As described herein, it should be understood that a display system can present the displayed content with a sense of depth by utilizing both a convergence / divergence motion cue and a focus adjustment cue. The convergence / divergence motion cue may be generated by presenting slightly different views of a virtual object to each eye of a user. The focus adjustment cue may be derived from the wavefront divergence of light that forms those slightly different views. The convergence / divergence motion cue causes the eyes to rotate and assume a particular convergence / divergence motion state, e.g., where the eyes converge on a virtual object. The focus adjustment cue can cause the eye's lens to assume a particular shape that provides a focused image of the virtual object on the retina of the eye. Thus, a particular convergence / divergence motion cue can cause the eyes to assume a particular convergence / divergence motion state, and a particular focus adjustment cue can cause the eyes to assume a particular focus adjustment state. A real object in space varies in terms of the convergence / divergence motion and focus adjustment cues along the optical axis or z-axis from the viewer such that a particular convergence / divergence motion cue can be correlated with a particular convergence / divergence motion distance and a particular focus adjustment cue can similarly be correlated with a particular focus adjustment distance away from the viewer. Conventionally, it has been thought that the convergence / divergence motion and focus adjustment cues should be closely aligned with each other to prevent viewer discomfort. That is, it has been thought that the convergence / divergence motion and focus adjustment distances with respect to a virtual object should be such to avoid a focus-convergence / divergence motion mismatch. A focus-convergence / divergence motion mismatch when displaying a virtual object can be defined as the dioptric difference between the convergence / divergence motion and the focus adjustment distance with respect to the virtual object.

[0151] However, the human visual system has been found to be tolerant of a certain level of accommodation-vergence / divergence motion mismatch. As a result, within the mismatch tolerance, the accommodation cues can remain the same while the vergence / divergence motion cues can vary, thereby varying the perceived depth of the virtual object. Thus, in some embodiments, the vergence / divergence motion cues can vary continuously while the accommodation cues vary in discrete steps, and the mismatch between accommodation and vergence / divergence motion is maintained below the mismatch tolerance level. Examples of accommodation-vergence / divergence motion mismatch tolerances include 0.5 diopters or less, 0.33 diopters or less, or 0.25 diopters or less. In some embodiments, the farthest depth plane can be within the accommodation-vergence / divergence motion mismatch tolerance of optical infinity, and the next farthest depth plane can be within the accommodation-vergence / divergence motion mismatch tolerance of the volume created by the accommodation-vergence / divergence motion mismatch tolerance of the farthest depth plane, and so on.

[0152] It is to be understood that a particular amount of wavefront divergence is associated with a particular depth plane. That is, the wavefront divergence of the light output by the display system corresponds to the wavefront divergence of the light originating from the real object at a particular depth along the z-axis. As a result, it can be understood that changing the wavefront divergence and the depth cue involves switching the depth plane at which the display system presents the virtual object. In some embodiments, the depth plane may be switched to maintain the accommodation-convergence / divergence motion mismatch below an acceptable tolerance level. Each depth plane may have a nominal focal depth with a corresponding wavefront divergence for the light that appears to originate from that depth plane. However, due to the accommodation-convergence / divergence motion mismatch tolerance, the content may be displayed "above" that depth plane (i.e., with the wavefront divergence corresponding to that depth plane) even when the convergence / divergence cue is utilized to provide the perception that the virtual object is closer to or farther from the viewer than the depth plane. The outer boundaries of the distance at which a particular depth plane can be utilized are determined by the accommodation-convergence / divergence motion mismatch, which can be measured in diopters as disclosed herein.

[0153] Some display systems, referred to herein as variable focus display systems, can present virtual content in discrete depth planes, and all virtual content is presented in the same depth plane at a given time (e.g., only one depth plane is active or outputs image information at a time). Displaying the content on one depth plane at a time can have the advantage of saving computational resources within the display system. To determine the depth plane on which to present the virtual content, a variable focus display system can determine the depth (also referred to herein as the fixation depth) at which the user's eye is fixated, for example, by determining the distance to a target on which the user's eye is fixated. Once the fixation depth is determined, the display system can present the content on a depth plane that aligns with or corresponds to the depth of fixation. What is considered to be aligned can be the fixation depth that aligns with the depth plane and / or is within the depth-of-field accommodation-vergence-divergence motion misalignment tolerance of that depth plane. As used herein, the depth of an object is its distance from the user as measured along the optical axis or z-axis.

[0154] As an example of determining the fixation depth, the display system may determine the fixation point of the user's eye. For example, the display system may monitor the orientation of the user's eye, estimate the line-of-sight vector associated with the user's eye, and determine the three-dimensional location where the individual determined lines of sight of the eyes intersect. The display system may determine that the user's eye is fixating on a particular three-dimensional location, and the display system may present the virtual content on a depth plane corresponding to that three-dimensional location. In this way, the display system can ensure that the content presented to the viewer is appropriate for that depth plane.

[0155] As a result, in some embodiments, the display system may be configured to track the user's eyes and provide the content on a depth plane corresponding to the depth at which the user's eyes are fixated. As the fixation point of the user's eyes changes, the display system may be configured to switch between different depth planes, which may cause an instantaneous jump in the retinal blur caused by the image of the virtual object. For a typical user, this may appear as flicker within the display as the eyes accommodate to the wavefront divergence provided by the new depth plane, and a short (e.g., 100 - 300 ms) period of blur will follow.

[0156] If the depth plane providing the image content can be associated with the fixation depth of the user's eyes, the error in determining the fixation depth can cause an error during the switching between depth planes. Possible sources of error include, for example, errors associated with monitoring the user's eyes (e.g., orientation), gaze tracking, electrical, computational, and / or optical limitations of the monitoring hardware, etc. Due to these sources of error, successive determinations of the location of the fixation point may provide different values with respect to that location. If the fixation depth is near the vicinity between two depth planes, any jitter in the determined location of the fixation point can cause jitter during the switching between depth planes. Undesirably, the display system may then alternate between presenting the virtual content on the first depth plane and the second depth plane, and each alternation is perceived as flicker by the user. Without being limited by theory, this flicker may cause discomfort to the user and reduce the user's immersion in the viewing experience.

[0157] In some embodiments, techniques are provided for limiting the range in which undesirable switching between depth planes occurs. As will be described below, depth overlap may be utilized such that a portion of the depth plane range covered or encompassed by a first depth plane may overlap with a portion of the depth plane range covered or encompassed by a second depth plane. The depth range encompassed by a depth plane represents, for example, the distance from the user such that when fixated, as will be described below with respect to FIGS. 16 - 18, the display system is caused to select that depth plane for presenting virtual content. Thus, if the user's point of fixation varies in depth but is located within the depth overlap, the display system may not need to change the depth plane on which the virtual content is presented. Accordingly, the display system may be prevented from unnecessary switching between different depth planes.

[0158] In some embodiments, a depth plane may have an associated depth plane range that includes depths extending back and forth on the z - axis from the depth plane by a particular value. For example, each depth plane range may extend from a particular distance closer to the nominal focal depth to a particular distance farther from the nominal focal depth associated with the depth plane. As an example, the particular distance may be 0.2, 0.33, or 0.5 diopters. For an example of a depth plane associated with a nominal focal depth of 1 diopter and a 0.33 diopter example, the display system may present virtual content on that exemplary depth plane with a depth from the user's eye (e.g., the user's eye's exit pupil) of 0.66 - 1.33 diopters when the user is fixating on a certain 3 - D location. In some embodiments, the particular distance farther from the nominal focal depth may represent a near - accommodation - vergence / divergence motion mismatch tolerance level (e.g., maximum mismatch).

[0159] As described above, portions of adjacent depth plane ranges may overlap such that both encompass a set of depths along the z-axis. In some embodiments, the range of depth overlap may be based on an error associated with determining the point of fixation. For embodiments in which each depth plane encompasses a fixed depth plane range (e.g., based on a focus adjustment - vergence / accommodation mismatch tolerance), the depth overlap may result in a shift in the nominal focal depth associated with one or more depth planes compared to a depth plane layout without an overlap region. For example, the distal boundary of a particular depth plane may be adjusted to extend beyond the proximal boundary of the more distant adjacent depth plane. Since the distal boundary of a particular depth plane is adjusted to be further in depth from the user, the proximal boundary of the particular depth plane may likewise be adjusted to be further in depth in order to maintain the fixed depth plane range. The nominal focal depth is also adjusted, and thus likewise, to ensure that all depths encompassed by the depth plane range of a particular depth plane extend less than the focus adjustment - vergence / accommodation mismatch tolerance from the nominal focal depth of the particular depth plane. The adjustment of the nominal focal depth of the depth plane is described in more detail below with respect to FIGS. 18A - 18B.

[0160] The aforementioned adjustment of the nominal focal depth associated with the depth plane may, in some exemplary display systems, reduce the overall depth plane range within which virtual content can be presented to the user by the display system. For example, since adjacent depth planes overlap or share a portion of their depth plane ranges, the total range of depths encompassed by all depth planes may be less than if there were no overlap between the depth planes. However, the depth overlap may reduce the occurrence of flicker caused by rapid switching due to errors in determining the point of fixation of the eye, such that the user's viewing experience may be improved despite the fact that the available range of depths available from the display system is less.

[0161] In addition, if the user is fixating on a particular depth outside the range of depths encompassed by any depth plane, the display system may optionally present a virtual object along with a convergence-divergence motion cue corresponding to the particular depth. Since the depth-of-field adjustment cue (e.g., the nominal focus depth) associated with the presentation would be based on the depth plane closest to the particular depth, the depth-of-field-convergence / divergence motion mismatch would exceed the mismatch tolerance level described above. As explained above, exceeding the depth-of-field-convergence / divergence motion mismatch tolerance can cause discomfort to the user, so the display system may limit the amount of time that virtual content can be presented while the mismatch exceeds a threshold (e.g., 10 seconds, 30 seconds, 3 minutes, and / or an amount of time selectable by the user).

[0162] In some embodiments, the user's viewing comfort can be improved by performing a depth plane switch during an event that masks or otherwise reduces the perceptibility of the switch. For example, the display system may determine that virtual content should be presented on a particular depth plane that is different from the depth plane on which the virtual content is currently selected (e.g., due to the user fixating on a new fixation point). For example, depth plane switching for a variable focus display system causes perceivable flicker as described above. As a result, in some embodiments, the display system may delay the depth plane switch until an event occurs in which the visual system is, for example, temporarily inactive, and perform the switch during such an event to mask the switch. Such events may include (1) a blink or (2) the performance of a saccadic eye movement. In response to the detection of such an event, the depth plane switch may be performed. As an example, the display system may determine, based on monitoring the user's 3D fixation point, that different depth planes should be selected to present the virtual content. The display system may then wait until (1) the user blinks or (2) the user performs a saccadic eye movement before switching the presentation of the virtual content to a different depth plane. Preferably, the switch occurs during a blink or saccadic eye movement, such that the user may not notice the switch because the eye may be closed or in motion at the moment of the switch.

[0163] Using the techniques described herein, the perceived presentation quality of virtual content can be improved. For example, perceivable visual artifacts such as flicker that result from switching content between different depth planes can be reduced, particularly when the display system is operating in variable focus mode.

[0164] The various embodiments disclosed herein also provide additional systems and methods for providing a comfortable viewing experience to a user. For example, as discussed herein, the accommodation-vergence / divergence motion mismatch can exceed a range that a user may find tolerable. In some embodiments, to reduce user discomfort caused by a large mismatch in accommodation-vergence / divergence motion, the display system may be configured to actively monitor the image content and determine whether the content will cause a large accommodation-vergence / divergence motion mismatch. In response to the detection of such a large mismatch, the image content may be modified to reduce or eliminate the mismatch before being displayed. For example, the display system may be configured to fade or not display the image content that results in a large mismatch.

[0165] In some other embodiments, the system may be configured to actively monitor user eye strain. In response to the detection of eye strain, the system may be configured to modify the image content to reduce the eye strain.

[0166] In some embodiments, physical discomfort related to the weight and / or balance of the display system may be addressed using a support structure, which may extend across and / or behind the user's head. The support structure may be, for example, a band that provides an upward pulling force on the display (e.g., reducing the pressure on the nose pads of the device) and / or counteracts the weight of the display on both the front and back of the user's head.

[0167] It should be understood that the display system may be part of an augmented reality display system or a virtual reality display system. As an example, the display system may be transparent and may enable a user to view the real world while providing virtual content to the user in the form of images, videos, interactions, etc. As another example, the display system may block the user's view of the real world, and virtual reality images, videos, interactions, etc. may be presented to the user.

[0168] Here, referring to the figures, like reference numerals refer to like parts throughout. Exemplary display system

[0169] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. It should be understood that when the user's eyes are separated and looking at a real object in space, each eye has a slightly different view of the object, and an image of the object can be formed at different locations on the retina of each eye. This can be referred to as binocular disparity and can be utilized by the human visual system to provide depth perception. The conventional display system simulates binocular disparity by presenting two distinct different images 190, 200 with slightly different views of one identical virtual object for each of the eyes 210, 220, corresponding to the views of the virtual object that would be seen by each eye as if the virtual object were a real object at the desired depth. These images provide binocular cues that the user's visual system can interpret to derive depth perception.

[0170] Continuing to refer to FIG. 2, images 190, 200 are separated from eyes 210, 220 by a distance 230 on the z-axis. The z-axis is parallel to the optical axis of the viewer in a state where the eyes are gazing at an object at optical infinity directly in front of the viewer. Images 190, 200 are flat and at a fixed distance from eyes 210, 220. Based on slightly different views of the virtual object in the images presented to eyes 210, 220 respectively, the eyes can necessarily rotate so that the image of the object comes to corresponding points on the respective retinas of the eyes and a single binocular vision is maintained. This rotation can converge the respective lines of sight of eyes 210, 220 onto a point in the space where the virtual object is perceived to exist. As a result, providing a three-dimensional image has conventionally involved providing binocular cues that can manipulate the convergence / divergence movement of the user's eyes 210, 220 and are interpreted by the human visual system to provide depth perception.

[0171] However, generating a realistic and comfortable perception of depth is difficult. It should be understood that light from objects at different distances from the eyes has wavefronts with different amounts of divergence. FIGS. 3A - 3C illustrate the relationship between distance and divergence of light rays. The distances between the object and eye 210 are represented in the order of decreasing distances R1, R2, and R3. As shown in FIGS. 3A - 3C, the light rays diverge more as the distance to the object decreases. Conversely, as the distance increases, the light rays become more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of the distance the point is away from the user's eyes. The curvature increases with the decreasing distance between the object and eye 210. Only a single eye 210 is illustrated in FIGS. 3A - 3C and various other figures in this specification for clarity of illustration, but the discussion regarding eye 210 can be applied to both eyes 210 and 220 of the viewer.

[0172] Continuing to refer to FIGS. 3A - 3C, light from an object on which a viewer's eye is fixated can have different wavefront divergences. Due to the different amounts of wavefront divergence, the light can be focused differently by the eye's lens, which in turn can require the lens to take on different shapes to form a focused image on the eye's retina. If the focused image is not formed on the retina, the resulting retinal blur acts as a cue for accommodation by causing a change in the shape of the eye's lens until the focused image is formed on the retina. For example, the cue for accommodation triggers relaxation or contraction of the ciliary muscle surrounding the eye's lens, thereby modulating the force applied to the zonular fibers that hold the lens, and thus changing the shape of the eye's lens until the retinal blur is eliminated or minimized, thereby forming a focused image of the fixated object on the eye's retina / fovea. The process by which the eye's lens changes shape can be referred to as accommodation, and the shape of the eye's lens required to form a focused image of the fixated object on the eye's retina / fovea can be referred to as the accommodative state.

[0173] Referring now to FIG. 4A, the representation of the accommodation-convergence / divergence motion response of the human visual system is illustrated. The movement of the eyes to fixate on an object causes the eyes to receive light from the object, which forms an image on each of the retinas of the eyes. The presence of retinal blur in the image formed on the retina can provide a cue for accommodation, and the relative location of the image on the retina can provide a cue for convergence / divergence motion. The cue for accommodation results in accommodation, causing the eye's lens to assume a particular accommodation state that forms a focused image of the object on the eye's retina / fovea. On the other hand, the cue for convergence / divergence motion causes movement (rotation of the eyes) due to convergence / divergence motion such that the images formed on each retina of each eye are at corresponding retinal points that maintain single binocular vision. At these positions, it can be said that the eyes are in a particular convergence / divergence motion state. Continuing to refer to FIG. 4A, accommodation can be understood as the process by which the eyes achieve a particular accommodation state, and convergence / divergence motion can be understood as the process by which the eyes achieve a particular convergence / divergence motion state. As shown in FIG. 4A, the accommodation and convergence / divergence motion states of the eyes can change when the user fixates on another object. For example, the accommodated state can change when the user fixates on a new object at a different depth along the z-axis.

[0174] Without being limited by theory, it is believed that the viewer of an object may perceive the object as "three-dimensional" due to the combination of convergence / divergence motion and accommodation. As described above, the movement due to convergence / divergence motion of the two eyes relative to each other (e.g., the rotation of the eyes such that the pupils move towards each other or away from each other, converging the lines of sight of the eyes to fixate on an object) is closely associated with the accommodation of the eye's lens. Under normal conditions, changing the shape of the eye's lens to change the focus from one object to another at a different distance will automatically cause a corresponding change in convergence / divergence motion to the same distance under a relationship known as the "accommodation-convergence / divergence reflex". Similarly, a change in convergence / divergence motion will, under normal conditions, trigger a corresponding change in the shape of the lens.

[0175] Referring now to FIG. 4B, examples of different focusing and vergence / divergence motion states of the eyes are illustrated. The pair of eyes 222a is gazing at an object at optical infinity, while the pair of eyes 222b is gazing at an object 221 at less than optical infinity. It should be noted that the vergence / divergence motion states of each pair of eyes are different, with the pair of eyes 222a being directed straight ahead, while the pair of eyes 222 converges onto the object 221. The focusing states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of the lenses 210a, 220a.

[0176] Unfortunately, many users of conventional "3-D" display systems find such conventional systems uncomfortable or perceive no sense of depth due to the inconsistency between the focusing and vergence / divergence motion states in these displays. As described above, many stereoscopic or "3-D" display systems display a scene by providing slightly different images to each eye. Such systems are uncomfortable for many viewers because they merely provide different presentations of the scene, causing a change in the vergence / divergence motion state of the eyes, but without a corresponding change in the focusing state of those eyes. Rather, the images are presented at a fixed distance from the eyes by the display such that the eyes view all the image information in a single focusing state. Such an arrangement goes against the "accommodation-vergence reflex" by causing a change in the vergence / divergence motion state without a corresponding change in the focusing state. This inconsistency is thought to cause viewer discomfort. A display system that provides better alignment between accommodation and vergence / divergence motion can create a more realistic and comfortable simulation of three-dimensional images.

[0177] Although not limited by theory, the human eye is typically thought to be able to interpret a finite number of depth planes and provide depth perception. As a result, a highly realistic simulation of the perceived depth can be achieved by providing different presentations of images corresponding to each of these limited number of depth planes to the eye. In some embodiments, the different presentations may provide both cues for convergence / divergence motion and matching cues for accommodation, thereby providing physiologically correct accommodation-convergence / divergence motion matching.

[0178] Continuing to refer to FIG. 4B, two depth planes 240 corresponding to different distances in space from eyes 210, 220 are illustrated. For a given depth plane 240, convergence / divergence motion cues may be provided by appropriately displaying images of different viewpoints for each of eyes 210, 220. Additionally, for a given depth plane 240, the light forming the images provided to each eye 210, 220 may have wavefront divergence corresponding to a light field generated by a point at the distance of that depth plane 240.

[0179] In the illustrated embodiment, the distance along the z-axis of depth plane 240 containing point 221 is 1 m. As used herein, the distance or depth along the z-axis may be measured using a zero point located at the exit pupil of the user's eye. Thus, depth plane 240 located at a depth of 1 m corresponds to a distance 1 m away from the exit pupil of the user's eye on the optical axis of those eyes. As an approximation, the depth or distance along the z-axis may be measured from the front of the display (e.g., the surface of the waveguide) of the user's eye, and a value related to the distance between the device and the exit pupil of the user's eye may be added. That value is referred to as the pupil distance and may correspond to the distance between the exit pupil of the user's eye and the display worn by the user in front of the eye. In practice, the value for the pupil distance may generally be a normalized value used for all viewers. For example, the pupil distance may be assumed to be 20 mm, and the depth plane at a depth of 1 m may be at a distance of 980 mm in front of the display.

[0180] Figure 5 illustrates aspects of an approach for simulating a three-dimensional image by correcting wavefront divergence. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to the user's eye 210. The waveguide 270 may output light 650 with a defined amount of wavefront divergence corresponding to the wavefront divergence of the light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on that depth plane. Additionally, the user's other eye would be illustrated as being provided with image information from a similar waveguide.

[0181] In some embodiments, a single waveguide may be configured to output light with a set amount of wavefront divergence corresponding to a single or limited number of depth planes, and / or the waveguide may be configured to output light within a limited range of wavelengths. As a result, in some embodiments, a plurality or stack of waveguides may be utilized to provide different amounts of wavefront divergence for different depth planes and / or output light within different ranges of wavelengths.

[0182] Figure 6 illustrates an example of a stack of waveguides for outputting image information to a user. The display system 250 includes a stack or stacked waveguide assembly 260 of waveguides 270, 280, 290, 300, 310 that can be utilized to provide three-dimensional perception to the eye / brain. It should be understood that the display system 250 may be considered a light field display in some embodiments. Additionally, the waveguide assembly 260 may also be referred to as an eyepiece.

[0183] In some embodiments, the display system 250 may be configured to provide a substantially continuous queue for convergence / divergence movement and a plurality of discrete queues for focus adjustment. The queue for convergence / divergence movement may be provided by displaying different images to each of the user's eyes, and the queue for focus adjustment may be provided by outputting light that forms an image with a discrete amount of wavefront divergence. In some embodiments, each discrete level of wavefront divergence may correspond to a particular depth plane and may be provided by a particular one of the waveguides 270, 280, 290, 300, 310.

[0184] Continuing to refer to FIG. 6, waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between waveguides. In some embodiments, features 320, 330, 340, 350 may be one or more lenses. Waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 360, 370, 380, 390, 400 may function as light sources for the waveguides and may be utilized to input image information into waveguides 270, 280, 290, 300, 310, and each may be configured to disperse incident light across each individual waveguide for output toward eye 210 as described herein. Light exits from output surfaces 410, 420, 430, 440, 450 of image input devices 360, 370, 380, 390, 400 and is input into corresponding input surfaces 460, 470, 480, 490, 500 of waveguides 270, 280, 290, 300, 310. In some embodiments, input surfaces 460, 470, 480, 490, 500 may each be an edge of the corresponding waveguide or a portion of the major surface of the corresponding waveguide (i.e., one of the waveguide surfaces facing directly toward world 510 or viewer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams, which are directed toward eye 210 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of image input devices 360, 370, 380, 390, 400 may be associated with and input light into a plurality (e.g., three) of waveguides 270, 280, 290, 300, 310.

[0185] In some embodiments, the image input devices 360, 370, 380, 390, 400 are each discrete displays that generate image information for input into their respective waveguides 270, 280, 290, 300, 310. In some other embodiments, the image input devices 360, 370, 380, 390, 400 are the output ends of a single multiplexed display that can send image information, for example, via one or more optical waveguides (such as optical fiber cables) to each of the image input devices 360, 370, 380, 390, 400. It should be understood that the image information provided by the image input devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different primary colors as discussed herein).

[0186] In some embodiments, the light input into the waveguides 270, 280, 290, 300, 310 is provided by an optical projector system 520, which includes an optical module 530 that may include a light emitter such as a light emitting diode (LED). The light from the optical module 530 may be directed and modified by a light modulator 540, such as a spatial light modulator, via a beam splitter 550. The light modulator 540 may be configured to vary the perceived intensity of the light input into the waveguides 270, 280, 290, 300, 310 and to encode the light with image information. Examples of spatial light modulators include liquid crystal displays (LCDs) and liquid crystal on silicon (LCOS) displays. The image input devices 360, 370, 380, 390, 400 are shown schematically and, in some embodiments, these image input devices may represent different optical paths and locations within a common projection system that are configured to output light into the associated ones of the waveguides 270, 280, 290, 300, 310. In some embodiments, the waveguides of the waveguide assembly 260 may function as ideal lenses while relaying the light input into the waveguides to the user's eyes. In this concept, the object may be the spatial light modulator 540 and the image may be an image on a depth plane.

[0187] In some embodiments, the display system 250 may be a scanning fiber display comprising one or more scanning fibers configured to project light in various patterns (e.g., raster scan, helical scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately into the viewer's eye 210. In some embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a single scanning fiber or a bundle of scanning fibers configured to input light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image input devices 360, 370, 380, 390, 400 may schematically represent a plurality of scanning fibers or a plurality of bundles of scanning fibers, each configured to input light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning fiber or fibers and one or more waveguides 270, 280, 290, 300, 310, e.g., to redirect light exiting the scanning fiber into one or more of the waveguides 270, 280, 290, 300, 310.

[0188] Controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image input devices 360, 370, 380, 390, 400, the light source 530, and the optical module 540. In some embodiments, controller 560 is part of the local data processing module 140. Controller 560 includes programming (e.g., instructions in a non-transitory medium) that adjusts the timing and provisioning of image information to waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single monolithic device or a distributed system connected by wired or wireless communication channels. Controller 560 may, in some embodiments, also be part of processing module 140 or 150 (FIG. 9D).

[0189] Continuing to refer to FIG. 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 270, 280, 290, 300, 310 may each be planar or have another shape (e.g., curved), with a major top and bottom surface and an edge extending between those major top and bottom surfaces. In the illustrated configuration, the waveguides 270, 280, 290, 300, 310 each include external coupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguide by redirecting the light propagating within each individual waveguide out of the waveguide and outputting the image information to the eye 210. The extracted light may also be referred to as external coupled light, and the external coupling optical elements may also be referred to as light extraction optical elements. The beam of extracted light may be output by the waveguide at the location where the light propagating within the waveguide impinges on the light extraction optical element. The external coupling optical elements 570, 580, 590, 600, 610 may be gratings, for example, including diffractive optical features as further discussed herein. For ease of explanation and clarity of the drawings, the external coupling optical elements 570, 580, 590, 600, 610 are shown disposed on the bottom major surface of the waveguides 270, 280, 290, 300, 310, but in some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be disposed on the top and / or bottom major surfaces and / or directly disposed within the volume of the waveguides 270, 280, 290, 300, 310, as further discussed herein. In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 may be formed within a layer of material that forms the waveguides 270, 280, 290, 300, 310 and is attached to a transparent substrate. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be monolithic parts of material, and the external coupling optical elements 570, 580, 590, 600, 610 may be formed on and / or within the surface of that part of the material.

[0190] Continuing to refer to FIG. 6, as discussed herein, each of the waveguides 270, 280, 290, 300, 310 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light (input into such waveguide 270) to the eye 210. The collimated light may represent an optically infinite focal plane. The next upper waveguide 280 may be configured to output collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210. Such a first lens 350 may be configured to generate a somewhat convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 280 as originating from a first focal plane that is closer inwardly toward the eye 210 from the optically infinite. Similarly, the third upper waveguide 290 passes its output light through both the first lens 350 and a second lens 340 before reaching the eye 210. The combined refractive power of the first lens 350 and the second lens 340 may be configured to generate another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 290 as originating from a second focal plane that is closer inwardly toward the person from the optically infinite than the light from the next upper waveguide 280 was.

[0191] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, and the top waveguide 310 in the stack sends its output through all of the lenses between it and the eye for the converging focusing power that represents the focal plane closest to the person. When viewing / interpreting light originating from the world 510 on the other side of the stacked waveguide assembly 260, a compensation lens layer 620 may be disposed on top of the stack to compensate for the stack of lenses 320, 330, 340, 350. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the external coupling optical elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0192] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, a plurality of waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same depth plane, or a plurality of subsets of waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same plurality of depth planes, with one set per depth plane. This can provide the advantage of forming tiled images that provide an expanded field of view in those depth planes.

[0193] Continuing to refer to FIG. 6, the external coupling optical elements 570, 580, 590, 600, 610 may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have different configurations of the external coupling optical elements 570, 580, 590, 600, 610, which output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 may be volume or surface features, which may be configured to output light at a specific angle. For example, the light extraction optical elements 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (e.g., structures for forming cladding layers and / or voids).

[0194] In some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 are diffraction features that form a diffraction pattern or "diffractive optical element" (also referred to herein as a "DOE"). Preferably, the DOE has a diffraction efficiency low enough such that only a portion of the light of the beam is deflected towards the eye 210 using each intersection of the DOE while the remainder continues to travel through the waveguide via TIR. The light carrying the image information is thus split into several associated output beams that exit the waveguide at various locations, resulting in a very uniform pattern of output emission towards the eye 210 with respect to this particular collimated beam that bounces within the waveguide.

[0195] In some embodiments, one or more DOEs may be switchable between an “on” state that actively diffracts and an “off” state that does not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal, in which microdroplets have a diffraction pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0196] In some embodiments, a camera assembly 630 (e.g., a digital camera including visible and infrared light cameras) may be provided to capture an image of the eye 210 and / or the tissue surrounding the eye 210, and, for example, detect user input and / or monitor the user's physiological state. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source that projects light (e.g., infrared light) into the eye, the light then being reflected by the eye and detectable by the image capture device. In some embodiments, the camera assembly 630 may be attached to the frame 80 (FIG. 9D) and may communicate electrically with a processing module 140 and / or 150 that may process the image information from the camera assembly 630. In some embodiments, one camera assembly 630 may be utilized per eye to monitor each eye separately.

[0197] Referring now to FIG. 7, an embodiment of an output beam output by a waveguide is shown. Although one waveguide is illustrated, it should be understood that other waveguides within the waveguide assembly 260 (FIG. 6) may function similarly, and the waveguide assembly 260 includes a plurality of waveguides. Light 640 is input into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates through the waveguide 270 by TIR. At the point where the light 640 impinges on the DOE 570, a portion of the light exits the waveguide as the output beam 650. The output beam 650 is illustrated as being substantially parallel, but as discussed herein, it may also be redirected to propagate to the eye 210 at an angle (e.g., forming a diverging output beam) depending on the depth plane associated with the waveguide 270. It should be understood that a waveguide with an external coupling optical element that externally couples light to form an image that appears to be set in a depth plane at a long distance (e.g., optically infinite) from the eye 210 may be shown for the substantially parallel output beam. Other waveguides or other sets of external coupling optical elements may output a more divergent output beam pattern, which would require the eye 210 to focus at a closer distance and would be interpreted by the brain as light from a distance closer to the eye 210 than optically infinite.

[0198] In some embodiments, a full-color image may be formed in each depth plane by overlaying images in each of the primary colors, e.g., three or more primary colors. FIG. 8 illustrates an example of a stacked waveguide assembly, and each depth plane includes an image formed using a plurality of different primary colors. The illustrated embodiment shows depth planes 240a-240f, although more or fewer depths may also be considered. Each depth plane may have three or more primary color images associated therewith, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. Different depth planes are shown in the figure by different numbers associated with the diopters (dpt) following the letters G, R, and B. As a mere example, the numbers following each of these letters indicate the diopter (1 / m), i.e., the inverse distance of the depth plane from the viewer, and each box in the figure represents an individual primary color image. In some embodiments, the exact location of the depth planes for different primary colors may vary to account for differences in the focusing of light of different wavelengths by the eye. For example, the different primary color images for a given depth plane may be placed on a depth plane corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or reduce chromatic aberration.

[0199] In some embodiments, the light of each primary color may be output by a single dedicated waveguide, and as a result, each depth plane may have a plurality of waveguides associated therewith. In such embodiments, each box in the figure, including those containing the letters G, R, or B, may be understood to represent an individual waveguide, and three waveguides may be provided for each depth plane, with three primary color images being provided for each depth plane. The waveguides associated with each depth plane are shown adjacent to each other in this figure for ease of explanation, but it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per level. In some other embodiments, a plurality of primary colors may be output by the same waveguide, e.g., such that only a single waveguide is provided for each depth plane.

[0200] Continuing to refer to FIG. 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, may also be used in addition to or in place of one or more of red, green, or blue.

[0201] It should be understood that references throughout this disclosure to the color of a given light include light of one or more wavelengths within the range of wavelengths of the light that is perceived by a viewer as that given color. For example, red light may include light of one or more wavelengths within the range of about 620 - 780 nm, green light may include light of one or more wavelengths within the range of about 492 - 577 nm, and blue light may include light of one or more wavelengths within the range of about 435 - 493 nm.

[0202] In some embodiments, the light source 530 (FIG. 6) may be configured to emit light of one or more wavelengths outside the viewer's visual perception range, such as infrared and / or ultraviolet wavelengths. Additionally, the internal coupling, external coupling, and other light redirection structures of the waveguide of the display 250 may be configured to direct and emit this light from the display towards the user's eye 210, for example, for imaging and / or user stimulation applications.

[0203] Referring now to FIG. 9A, in some embodiments, light impinging on the waveguide may need to be redirected to internally couple the light into the waveguide. An internal coupling optical element may be used to redirect and internally couple the light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an example of a stack 660 of a plurality or set of waveguides, each including an internal coupling optical element. Each waveguide may be configured to output light of one or more different wavelengths or one or more different wavelength ranges. Stack 660 may correspond to stack 260 (FIG. 6), and the illustrated waveguides of stack 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310, but it should be understood that light from one or more of the image input devices 360, 370, 380, 390, 400 is input into the waveguide from a position that requires the light to be redirected for internal coupling.

[0204] The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as an optical input area on the waveguide). For example, internal coupling optical element 700 is disposed on a major surface (e.g., the upper major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., the upper major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., the upper major surface) of waveguide 690. In some embodiments, one or more of internal coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the respective waveguides 670, 680, 690 (in particular, one or more of the internal coupling optical elements are reflective deflecting optical elements). As illustrated, internal coupling optical elements 700, 710, 720 may be disposed on the upper major surface (or the upper portion of the next lower waveguide) of their respective waveguides 670, 680, 690, and in particular, those internal coupling optical elements are transmissive deflecting optical elements. In some embodiments, internal coupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguides 670, 680, 690. In some embodiments, as discussed herein, internal coupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of their respective waveguides 670, 680, 690, it should be understood that in some embodiments, internal coupling optical elements 700, 710, 720 may be disposed within other areas of their respective waveguides 670, 680, 690.

[0205] As shown, the internally coupled optical elements 700, 710, 720 may be laterally offset from each other. In some embodiments, each internally coupled optical element may be offset such that its light is received without passing through another internally coupled optical element. For example, each internally coupled optical element 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6, and may be separated (e.g., laterally spaced) from other internally coupled optical elements 700, 710, 720 such that light is not substantially received from other internally coupled optical elements 700, 710, 720.

[0206] Each waveguide also includes an associated light dispersing element. For example, the light dispersing element 730 is disposed on a major surface (e.g., the upper major surface) of the waveguide 670, the light dispersing element 740 is disposed on a major surface (e.g., the upper major surface) of the waveguide 680, and the light dispersing element 750 is disposed on a major surface (e.g., the upper major surface) of the waveguide 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on the bottom major surface of the associated waveguides 670, 680, 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on both the upper and bottom major surfaces of the associated waveguides 670, 680, 690, or the light dispersing elements 730, 740, 750 may each be disposed on different ones of the upper and bottom major surfaces within different associated waveguides 670, 680, 690.

[0207] Waveguides 670, 680, 690 may be separated and isolated, for example, by a gas, liquid, and / or solid layer of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediate waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more or 0.10 or less of the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote total internal reflection (TIR) of light (e.g., TIR between the upper and bottom major surfaces of each waveguide) through waveguides 670, 680, 690. In some embodiments, layers 760a, 760b are formed from air. It should be understood that although not shown, the upper and bottom of the illustrated set 660 of waveguides may include an immediate cladding layer.

[0208] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or identical, and the materials forming layers 760a, 760b are similar or identical. In some embodiments, the materials forming waveguides 670, 680, 690 may be different between one or more waveguides, and / or the materials forming layers 760a, 760b may still be different while maintaining the various refractive index relationships described above.

[0209] Continuing to refer to FIG. 9A, light rays 770, 780, 790 are incident on the set 660 of waveguides. It should be understood that light rays 770, 780, 790 may be input into waveguides 670, 680, 690 by one or more image input devices 360, 370, 380, 390, 400 (FIG. 6).

[0210] In some embodiments, the light rays 770, 780, 790 may have different properties, such as different wavelengths or different wavelength ranges, corresponding to different colors. The internal coupling optical elements 700, 710, 720 each deflect the incident light so that the light propagates through an individual one of the waveguides 670, 680, 690 by TIR. In some embodiments, the internal coupling optical elements 700, 710, 720 each selectively deflect one or more specific wavelengths of light while transmitting other wavelengths to the underlying waveguide and the associated internal coupling optical element.

[0211] For example, the internal coupling optical element 700 may be configured to deflect the light ray 770 having the first wavelength or wavelength range while transmitting the light rays 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light ray 780 impinges on the internal coupling optical element 710 configured to deflect the light of the second wavelength or wavelength range, and is thereby deflected. The light ray 790 is deflected by the internal coupling optical element 720 configured to selectively deflect the light of the third wavelength or wavelength range.

[0212] Continuing to refer to FIG. 9A, the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the internal coupling optical elements 700, 710, 720 of each waveguide deflect the light into its corresponding waveguide 670, 680, 690 and internally couple the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the individual waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the individual waveguides 670, 680, 690 by TIR until they impinge on the corresponding light dispersion elements 730, 740, 750 of the waveguide.

[0213] Referring now to FIG. 9B, a perspective view of an embodiment of the plurality of stacked waveguides of FIG. 9A is illustrated. As described above, the internally coupled light rays 770, 780, 790 are each deflected by the internal coupling optical elements 700, 710, 720 and then each propagate by TIR within the waveguides 670, 680, 690. The light rays 770, 780, 790 then each impinge on the light dispersion elements 730, 740, 750. The light dispersion elements 730, 740, 750 each deflect the light rays 770, 780, 790 so as to propagate towards the external coupling optical elements 800, 810, 820.

[0214] In some embodiments, the light dispersing elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse light to the external coupling optical elements 800, 810, 820, and in some embodiments, also increase the beam or spot size of the present light as it propagates to the external coupling optical elements. In some embodiments, the light dispersing elements 730, 740, 750 may be omitted, and the internal coupling optical elements 700, 710, 720 may be configured to deflect light directly to the external coupling optical elements 800, 810, 820. For example, referring to FIG. 9A, the light dispersing elements 730, 740, 750 may each be replaced with the external coupling optical elements 800, 810, 820. In some embodiments, the external coupling optical elements 800, 810, 820 are an exit pupil (EP) or an exit pupil expander (EPE) that directs light to the viewer's eye 210 (FIG. 7). It should be understood that the OPE may be configured to increase the size of the eyebox in at least one axis, and the EPE may increase the eyebox in an axis that intersects, for example, is orthogonal to, the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light impinging on the OPE to the EPE of the same waveguide while allowing the remaining portion of the light to continue to propagate along the waveguide. In response to a collision with the OPE, again, another portion of the remaining light is redirected to the EPE, and the remaining portion of that portion continues to propagate further along the waveguide and the like. Similarly, in response to a collision with the EPE, a portion of the colliding light is directed from the waveguide towards the user, and the remaining portion of that light continues to propagate through the waveguide until it impinges on the EP again, at which point another portion of the colliding light is directed from the waveguide, and so on. As a result, a single beam of internally coupled light is "replicated" each time a portion of that light is redirected by the OPE or EPE, thereby forming a beam field of cloned light as shown in FIG. 6. In some embodiments, the OPE and / or EPE may be configured to modify the size of the beam of light.

[0215] Thus, referring to FIGS. 9A and 9B, in some embodiments, the waveguide set 660 includes, for each primary color, waveguides 670, 680, 690, internal coupling optical elements 700, 710, 720, light dispersion elements (e.g., OPE) 730, 740, 750, and external coupling optical elements (e.g., EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with a gap / cladding layer between each one. The internal coupling optical elements 700, 710, 720 redirect or deflect the incident light into their respective waveguides (using different internal coupling optical elements that receive light of different wavelengths). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the example shown, the light ray 770 (e.g., blue light) is polarized by the first internal coupling optical element 700 in the manner described above, then bounces along the waveguide and interacts with the light dispersion element (e.g., OPE) 730, then the external coupling optical element (e.g., EP) 800. The light rays 780 and 790 (e.g., green and red light, respectively) pass through the waveguide 670, and the light ray 780 impinges on the internal coupling optical element 710 and is thereby deflected. The light ray 780 then bounces along the waveguide 680 via TIR and will proceed to its light dispersion element (e.g., OPE) 740, then the external coupling optical element (e.g., EP) 810. Finally, the light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the internal coupling optical element 720 of the waveguide 690. The internal coupling optical element 720 deflects the light ray 790 such that the light ray propagates by TIR to the light dispersion element (e.g., OPE) 750, then by TIR to the external coupling optical element (e.g., EP) 820. The external coupling optical element 820 then finally externally couples the light ray 790 to the viewer, who also receives the externally coupled light from the other waveguides 670, 680.

[0216] FIG. 9C illustrates a top and bottom plan view of an example of a plurality of stacked waveguides of FIGS. 9A and 9B. As illustrated, waveguides 670, 680, 690 may be vertically aligned with associated optical dispersion elements 730, 740, 750 and associated external coupling optical elements 800, 810, 820 of each waveguide. However, as discussed herein, internal coupling optical elements 700, 710, 720 are not vertically aligned. Rather, the internal coupling optical elements are preferably non-overlapping (e.g., laterally spaced as seen in the top and bottom views). As further discussed herein, this non-overlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a one-to-one basis, thereby enabling a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, an array including non-overlapping spatially separated internal coupling optical elements may be referred to as an offset pupil system, and the internal coupling optical elements within these arrays may correspond to sub-pupils.

[0217] FIG. 9D illustrates an example of a wearable display system 60 into which various waveguides and related systems disclosed herein may be integrated. In some embodiments, display system 60 is the system 250 of FIG. 6, which schematically shows some portions of that system 60 in more detail. For example, the waveguide assembly 260 of FIG. 6 may be part of display 70.

[0218] Continuing to refer to FIG. 9D, the display system 60 includes a display 70 and various mechanical and electronic modules and systems for supporting the functions of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and is configured to position the display 70 in front of the user 90's eyes. The display 70 may be considered eyewear in some embodiments. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the user 90's outer ear canal (in some embodiments, another speaker, not shown, may also be optionally positioned adjacent to the other outer ear canal of the user to provide stereo / formable sound control). The display system may also include one or more microphones 110 or other devices that may detect sound. In some embodiments, the microphone is configured to enable the user to provide an input or command to the system 60 (e.g., selection of a voice menu command, natural language question, etc.) and / or enable audio communication with other persons (e.g., other users of a similar display system). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sound from the user and / or the environment). In some embodiments, the display system may also include a peripheral sensor 120a, which is separate from the frame 80 and may be attached on the user 90's body (e.g., the user 90's head, torso, limbs, etc.). The peripheral sensor 120a may be configured to obtain data characterizing the physiological state of the user 90 in some embodiments. For example, the sensor 120a may be an electrode.

[0219] Continuing to refer to FIG. 9D, the display 70 is operatively coupled to a local processor or data processing module 140 by a communication link 130 such as a wired conductor or wireless connectivity, which may be mounted in various configurations such as fixed to a helmet or hat worn by a user to which the frame 80 is fixedly attached, embedded within headphones, or otherwise removably attached to the user 90 (e.g., in a backpack-style configuration, a belt-coupled configuration). Similarly, the sensor 120a may be operatively coupled to the local data processing module 140 by a communication link 120b, such as a wired conductor or wireless connectivity. The local processor and data module 140 may comprise a digital memory such as a hardware processor and non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to assist in the processing, caching, and storage of data. Optionally, the local processor and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include a) data captured from sensors (image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein (e.g., operatively coupled to the frame 80 or otherwise attachable to the user 90)) and / or b) data obtained and / or processed using a remote processing module 150 and / or remote data repository 160 (including data related to virtual content) for possible passage to the display 70 after processing or retrieval. The local processing and data module 140 may be operatively coupled to the remote processing module 150 and remote data repository 160 by communication links 170, 180, such as via a wired or wireless communication link, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to the local processing and data module 140.In some embodiments, the local processing and data module 140 may include one or more than one of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more than one of these sensors may be attached to the frame 80 or may be an independent structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0220] Continuing to refer to FIG. 9D, in some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information, for example, including one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, the remote data repository 160 may include a digital data storage facility that may be available through the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information, for example, information for generating augmented reality content, to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, enabling fully autonomous use from the remote module. Optionally, an external system (e.g., one or more processors, one or more computer systems) including a CPU, GPU, etc. may perform at least a portion of the processing (e.g., generating image information, processing data), for example, providing information to and receiving information from modules 140, 150, 160 via a wireless or wired connection. Depth plane configuration

[0221] Referring now to FIGS. 10A and 10B, examples of a matched vergence-accommodation distance and a mismatched vergence-accommodation distance are illustrated, respectively. As shown in FIG. 10A, the display system may provide an image of a virtual object to each of the eyes 210, 220. The image may cause the eyes 210, 220 to assume a vergence state in which the eyes converge on a point 15 on the depth plane 240. Additionally, the image may be formed by light having a wavefront curvature corresponding to the real object in the depth plane 240. As a result, the eyes 210, 220 assume an accommodation state in which the image is in focus on their retinas. Thus, the user may perceive the virtual object as being at the point 15 on the depth plane 240.

[0222] It should be understood that each of the accommodation and vergence states of the eyes 210, 220 is associated with a specific distance on the z-axis. For example, an object at a specific distance from the eyes 210, 220 causes those eyes to assume a specific accommodation state based on the distance of the object. The distance associated with a specific accommodation state may be referred to as the accommodation distance A. d Similarly, there also exists a specific vergence distance V associated with the eyes, particularly with the vergence state or the position relative to each other. When the accommodation distance and the vergence distance are matched, the relationship between accommodation and vergence can be said to be physiologically correct. This is regarded as the most comfortable scenario for the viewer. d

[0223] However, in a stereoscopic display, the accommodation distance and the vergence / accommodation movement distance may not always match. For example, as shown in FIG. 10B, the images displayed on eyes 210, 220 may be displayed with wavefront divergence corresponding to depth plane 240, and eyes 210, 220 may take a specific accommodation state in which points 15a, 15b on that depth plane are in focus. However, the images displayed on eyes 210, 220 may provide a cue for vergence / accommodation movement that converges eyes 210, 220 on point 15 that is not located on depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to the distance from the exit pupils of eyes 210, 220 to depth plane 240, while the vergence / accommodation movement distance corresponds to a greater distance from the exit pupils of eyes 210, 220 to point 15. The accommodation distance is different from the vergence / accommodation movement distance. As a result, an accommodation-vergence / accommodation movement mismatch exists. Such a mismatch is considered undesirable and can cause discomfort to the user. The mismatch corresponds to a distance (e.g., V d -A d ) and can be characterized using diopters.

[0224] It should be understood that in some embodiments, a reference point other than the exit pupils of eyes 210, 220 may be used to determine the distance as long as the same reference point is used for the accommodation distance and the vergence / accommodation movement distance. For example, the distance can be measured from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., the waveguide of the display device) to the depth plane, etc.

[0225] Due to the potential for accommodation-vergence / divergence motion incongruence, the perception that such incongruence is generally undesirable, and the ability of a display system to provide a near-infinite number of possible vergence / divergence motion cues, it is considered desirable to provide a display system capable of providing a virtual content on a number of depth planes, in order to match as closely as possible the potentially numerous vergence / divergence motion cues. However, as described herein, a number of depth planes may require a number of associated waveguides. This can, undesirably, result in a large and heavy device, which can be uncomfortable, difficult to manufacture, and, additionally, can cause optical aberrations (both in the image content projected by the display and the light received from the external world) due to the numerous optically active features within the stack through which light must propagate to reach the viewer.

[0226] Advantageously, it has been found that a realistic three-dimensional experience can be achieved using a display system that provides virtual content on a relatively small number of depth planes. For example, in some embodiments, the total number of depth planes provided by the display system may be two. In other embodiments, the total number of depth planes provided by the display system may be one. Additionally, display systems having more than two depth planes are also contemplated. For example, the total number of depth planes may be four or less or three or less in some embodiments. As discussed herein, the total number of depth planes may be adjusted for a particular application based on the distance to the user at which the display system is expected to display virtual objects. For example, the number of depth planes may increase as the distance to the user of the virtual object decreases.

[0227] Although not limited by theory, it is believed that a user can still perceive vergence-accommodation mismatches of 0.25 diopters, 0.33 diopters, and up to about 0.5 diopters that are considered physiologically correct without the mismatch itself causing significant discomfort. As a result, it is possible to provide physiologically correct vergence-accommodation mismatches using a limited number of depth planes, in light of the viewer's tolerance for mismatches within a particular range. Depending on the content being displayed, the number of depth planes may be one or two. In some other embodiments, more than two depth planes may also be implemented as discussed herein.

[0228] In some embodiments, the placement of the depth planes on the z-axis is selected with reference to optical infinity. FIG. 11 illustrates an example of depth plane placement considering acceptable vergence-accommodation mismatches. The farthest depth plane on the z-axis may be selected to be at a distance within the physiologically acceptable vergence-accommodation mismatch of optical infinity. The mismatch is preferably about 0.5 diopters or less, more preferably about 0.33 diopters or less, or about 0.25 diopters or less. In some embodiments, a mismatch of about 0.55 diopters is considered. It should be understood that this depth plane can be considered the farthest depth plane provided by the display and is less than optical infinity.

[0229] Additional depth planes may also be provided closer to the viewer 60. Preferably, this additional depth plane is separated from the farther depth plane by no more than twice the acceptable vergence-accommodation mismatch. For example, the spacing between two depth planes is preferably about 1.0 diopters or less, more preferably about 0.66 diopters or less (e.g., 0.667 diopters or less), or about 0.5 diopters or less.

[0230] Continuing to refer to FIG. 11, in the illustrated embodiment, two depth plane configurations are shown. A relatively close depth plane 3000 (proximity is determined relative to viewer 60) is provided at 1.0 diopters, and the farthest depth plane 3002 is provided at 0.3 diopters (including approximately 0.33 diopters). Depth planes 3000 and 3002 are separated by less than 1.0 diopter. As shown, depth plane 3000 has an acceptable vergence-accommodation motion mismatch region defined by 3000a and 3000b, assuming an acceptable mismatch range of ±0.3 diopters. Advantageously, although not limited by theory, image content depicting a virtual object at a distance from the user within the acceptable mismatch range may be displayed on depth plane 3000 without causing an uncomfortable vergence-accommodation motion mismatch detectable by the user.

[0231] It should be understood that the lower limit 3000a still leaves a gap between it and viewer 60. Since the mismatch was selected to be 0.3 diopters, there is still a space 3008 that is within 0.5 diopters. As a result, some content can still be displayed at this distance. Preferably, since this is outside the acceptable mismatch, the duration and / or spatial frequency of the content can be limited as discussed herein.

[0232] FIG. 12 illustrates another embodiment of depth plane placement considering acceptable focus-accommodation-vergence-divergence motion mismatches. In this embodiment, instead of placing the farthest depth plane 3002 within the acceptable mismatch of optical infinity, the farthest depth plane 3002 may be placed at a depth along the z-axis that exceeds the acceptable focus-accommodation-vergence-divergence motion mismatch with optical infinity. In such a configuration, placing the content at optical infinity based on the vergence-divergence motion cue may provide an uncomfortable mismatch with the focus cue provided by a waveguide that outputs light with wavefront divergence corresponding to the depth plane 3002 at 0.6 diopters. Nevertheless, such a configuration allows the content to be displayed closer to the user 60 than the configuration of FIG. 11. For example, the closest depth at which the arrangement of FIG. 12 can provide the content while maintaining appropriate focus-accommodation-vergence-divergence motion alignment is 63 cm. On the other hand, the closest depth for the arrangement of FIG. 11 is 76 cm.

[0233] FIG. 13 illustrates an example of a plot of focus-accommodation-vergence-divergence motion mismatches for a single depth plane display system. The dotted lines indicate the focus-accommodation-vergence-divergence motion mismatches at different (virtual) distances from the viewer. The horizontal axis corresponds to the distance from the viewer, and the vertical axis corresponds to the focus-accommodation-vergence-divergence motion mismatch, which is also referred to as AVM. The single depth plane is positioned at 2 m, which corresponds to zero mismatch. Advantageously, at distances greater than 2 m (e.g., z = 2 m to z = optical infinity), the focus-accommodation-vergence-divergence motion mismatch always remains below 0.5 diopters. At closer distances, the mismatch increases, and at distances less than 1 m from the viewer, the mismatch may exceed what is considered physiologically correct. At distances less than 1 m, viewer discomfort may also be expected simply from viewing the image content at that distance.

[0234] FIG. 14 illustrates an example of a plot of accommodation-convergence / divergence motion mismatch for two depth plane display systems and one depth plane display system. The solid line represents the two depth plane system and the dotted line represents the single depth plane system. It should be understood that the accommodation-convergence / divergence motion mismatch at a large distance from the viewer is less for the two depth plane system, and the two depth plane system still allows the content to be displayed at a closer distance to the viewer while maintaining an acceptable mismatch value. In some embodiments, the farthest depth plane may be within an acceptable mismatch value of optical infinity. As discussed herein, the acceptable mismatch may be about 0.5 diopters or less, about 0.33 diopters or less, or about 0.25 diopters or less. As shown, the acceptable mismatch may be 0.33 diopters. In some embodiments, the farthest depth plane may be set to 0.33 diopters (corresponding to 3 m from the user), and the closer second depth plane may be set inward from the farthest depth plane by a value equal to twice the acceptable mismatch, e.g., 0.33 diopters × 2, i.e., 0.66 diopters. As a result, the closer, second depth plane may be set to 1 diopter in some embodiments.

[0235] Continuing to refer to FIG. 14, as one proceeds along the solid line plot from right to left (as the distance from the user's eye decreases), the accommodation-convergence / divergence motion mismatch increases until a mismatch value of 0.33 diopters is observed. As a result, assuming the illustrated placement of the farthest depth plane at 0.33 diopters, image content for virtual objects from a distance of 1.2 m to infinity can be displayed on the same depth plane (set to 0.33 diopters or 3 m from the user's eye) without significant discomfort since all image content within this range is within the acceptable mismatch. For virtual objects at a distance closer than 1.2 m, as shown, a second depth plane may be provided. As previously described, the second depth plane may be at 1 diopter.

[0236] It should be understood that the distance represented by an acceptable mismatch of 0.33 diopters becomes smaller as the depth plane is placed closer to the user. In some other embodiments, it may be desirable to display a virtual object closer than the minimum distance provided by a separation of 0.33 diopters from the 1 diopter depth plane. As shown, at distances of 0.75 m or less, the acceptable mismatch value increases above 0.33 diopters. As a result, one or more additional depth planes may be provided to display the image content at closer distances. For example, a third depth plane may be formed to display the image content at a distance closer than 0.75 m. It should be understood that each depth plane is separated from the nearest depth plane by no more than twice the acceptable mismatch.

[0237] FIG. 15 illustrates another example of a plot of focus-convergence / divergence motion mismatch for a two-depth-plane display system and a single-depth-plane display system. In this example, the farthest depth plane of the two-plane system is positioned at the same distance (2 m or 0.5 diopters) as the single-depth-plane system. Of note, the mismatches at distances farther than the single depth plane are equivalent. In this example, the main advantage of the two-plane system is the ability to provide the content closer to the viewer than the single-plane system while maintaining the acceptable mismatch value.

[0238] As can be seen in FIGS. 11 - 15, for some types of image content, a display system with a single depth plane can be sufficient to achieve physiologically correct accommodation - vergence / divergence motion matching while enabling a realistic 3 - D experience over a relatively large distance. Preferably, the farthest depth plane is less than optical infinity and within an acceptable mismatch range of optical infinity, whether it is a 1 - depth - plane display system or a multi - depth - plane display system. In some other embodiments, the farthest depth plane may be set within an acceptable mismatch of optical infinity. It should be understood that in some embodiments, a single - depth - plane system may have a depth - plane position that is set according to the type of image content to be displayed by the system. For example, the depth plane may be set to be closer to the viewer than that shown, especially in applications where the image content is expected to be relatively close to the viewer. Thus, referring to FIGS. 11 - 15, the display system may be configured to provide accommodation cues in discrete steps, while an almost infinite number of different vergence / divergence motion cues may be provided. Depth - plane switching

[0239] As described herein, display systems according to various embodiments (e.g., an extended reality display system such as display system 60 of FIG. 9D) may utilize overlapping depth planes. In variable focus mode, the display system may determine the depth at which the user is gazing and, based on the gaze depth, select a depth plane for presenting virtual content. For example, the display system may determine a three-dimensional gaze point at which the user is gazing and use the determined depth of the gaze point to select a depth plane. Errors associated with determining the gaze depth can introduce uncertainty regarding the location of the gaze depth. Given these errors, successive measurements of the gaze point can provide different results. If these different results occur at the boundary between two depth planes, the changing results can cause the display system to provide results having a gaze point that rapidly switches back and forth between the two depth planes as the successive measurements of the gaze point move back and forth between the depth planes. As a result, flicker or other visual artifacts can be introduced as the display system switches back and forth between depth planes to present virtual content to the user.

[0240] Depth repetition may be utilized to reduce the aforementioned visual artifacts. As described herein, adjacent depth planes may have associated depth plane ranges that partially overlap along the z-axis (e.g., adjacent depth planes may overlap at depths within a particular range along the z-axis). An exemplary representation of such an overlap is illustrated in FIG. 17 and further described below. In some embodiments, the size of the depth repetition may be based on an estimated uncertainty associated with determining a fixation point (e.g., determining the depth at which a user is fixating). By utilizing depth repetition, the display system may select a particular depth plane for presenting virtual content based on an identification that the user is fixating (1) only within a depth plane range associated with a particular depth plane and / or (2) within a depth repetition associated with a particular depth plane. If the user changes their fixation point such that it is within a depth plane range associated only with a different depth plane, the display system may switch to the different depth plane. For example, the display system may continue to present virtual content on a particular depth plane while the user's determined fixation point is at any one of the depths encompassed by a particular depth plane (e.g., a depth encompassed only by a particular depth plane, or a depth within a depth repetition encompassed by a particular depth plane and an adjacent depth plane). If the user then fixates on a depth not encompassed by the particular depth plane, the display system may switch to a different depth plane.

[0241] In some embodiments, the fixation point may be located in space along (1) the x-axis (e.g., the lateral axis), (2) the y-axis (e.g., the vertical axis), and (3) the z-axis (e.g., the depth of the point, e.g., the depth from the exit pupil of the user's eye to the fixation point). In some embodiments, the display system may utilize a sensor such as a camera (e.g., sensor 630 of FIG. 6) to monitor the user's eyes (e.g., the pupils and / or corneas of each eye, etc.) and determine the line-of-sight direction of each eye. It can be understood that the line-of-sight direction of each eye is parallel to the vector extending from the fovea through the center of the eye's lens. The display system may be configured to extrapolate the location where the vectors associated with the eyes intersect, and this intersection point can be understood to be the fixation point of the eyes. In other words, the fixation point may be a location in the three-dimensional space where the user's both eyes are performing convergence / divergence movements. In some embodiments, the display system may filter out slight movements of the user's eyes during, for example, fast movements (e.g., saccadic eye movements, microsaccades), and update the fixation point in response to the determination that the eyes are fixating on a location in the three-dimensional space. For example, the display system may be configured to ignore eye movements that are fixating on a point for less than a threshold duration and / or ignore involuntary eye movements (e.g., blinking).

[0242] FIG. 16 illustrates an example of a user fixating on a fixation point 1604. For example, the user may utilize a display system (e.g., a variable focus display system as described above) that can include two depth planes 240b, 240a. Each depth plane 240b, 240a can encompass a specific depth range (e.g., depth plane region 1809 and depth plane region 1808, respectively). As illustrated, depth plane region 1809 is clearly different from and directly adjacent to depth plane 1808. Thus, when the fixation point 1604 is, for example, within depth plane region 1809, the display system can select depth plane 240b to present virtual content. The virtual content can then be presented on depth plane 240b such that the virtual content would be output with cues for perspective adjustment associated with depth plane 240b. As an example, the display system may include two waveguides configured to output light with individual cues (e.g., wavefront divergence) for perspective adjustment, each waveguide corresponding to a depth plane.

[0243] The example of FIG. 16 shows two depth planes, but it should be understood that any number of depth planes (and their associated depth plane ranges) may be included within the display system using the techniques described herein. As illustrated, the display system has determined that the user's eyes 210, 220 are fixating on a particular fixation point 1604 (e.g., converging / diverging). In some scenarios, the determined fixation point 1604 is located at a depth close to the boundary between depth plane region 1809 and depth plane region 1808.

[0244] An estimated determination error 1605 associated with the fixation point 1604 is illustrated. As explained above, the error can be introduced when the display system determines the fixation point. For example, the display system may not be able to precisely determine the line of sight of each of the user's eyes 210, 220. For example, the optical axis of the eye determined based on the geometry of the eye may be different from the visual axis of the eye at the fovea of the eye. Since the display system monitors the user's eyes 210, 220 and thus the optical axis, the determination of the fixation point by the display system may deviate from the correct location that would be given by the analysis of the visual axis. The display system may have access to training information regarding the user. For example, during the initial use of the display system, the user may calibrate the system so that the visual axis can be better determined, but the error may still exist. As another example, the user's eyes may have unique medical problems or may be uniquely focused such that the estimation of the fixation point may deviate from the actual fixation point. In addition, the sensors utilized to image or track the eyes may also have limitations in error or resolution that introduce an error into the determined fixation point. As a result, the fixation point determined by the display system may have a range of uncertainty. The determination error 1605 thus represents the uncertainty with respect to the exact three-dimensional fixation point. For example, the determination error 1605 may indicate the uncertainty with respect to the depth of the fixation point 1604, such as 0.1 diopter, 0.2 diopter, 0.3 diopter, 0.58 diopter, etc. Since the actual depth at which the user is fixating may be either in front of or behind the determined fixation point 1604, the actual three-dimensional location at which the user can fixate is included within a range of depth that is twice the determination error 1605.

[0245] Since the determination error 1605 extends into the depth plane region 1809 and the depth plane region 1808, the display system may determine that the fixation point 1604 is at a depth encompassed by either depth plane 240B or 240A. For example, the display system may present the virtual content on depth plane 240A for one or more consecutive frames and then switch the presentation to be on depth plane 240B, etc. Such switching between presentations of images on different depth planes can occur rapidly and introduce undesirable visual artifacts to the user. As an example, flicker may become apparent to the user. As another example, when switching between different depth planes, the depth cueing may be adjusted such that the user would be required to adjust their focus (e.g., the wavefront divergence of the output light would be different for each depth plane).

[0246] To minimize the occurrence of undesirable switching between depth planes due to errors in determining the fixation point, depth overlap encompassing a portion of depth plane region 1808 and a portion of depth plane region 1809 may be utilized. As will be explained, the display system may continue to present the content on a particular depth plane if the determined fixation point is (1) only within the depth plane range of a particular depth plane or (2) within the depth overlap between depth plane region 1808 and depth plane region 1809. On the other hand, if the user's fixation point is located at a depth encompassed only by different depth planes, the display system may then switch to that depth plane and present the virtual content on a different depth plane.

[0247] FIG. 17 illustrates a depth overlap 1812 between adjacent depth plane regions 1808, 1809 of depth planes 240A, 240B. As described above, determining a three-dimensional fixation point can include sources of error such that uncertainty exists with respect to the precise three-dimensional location at which the user is fixating. For example, the determination error 1605 illustrated in FIG. 16 can introduce uncertainty with respect to the depth at which the user is fixating. In some embodiments, the depth overlap 1812 may thus be utilized by the display system to represent this determination error 1605. In some other embodiments, the depth overlap 1812 may have a size that is arbitrarily set.

[0248] As illustrated, the depth overlap 1812 is within both the depth plane region 1808 and the depth plane region 1809. Specifically, in the example of FIG. 17, the depth plane region 1808 is adjusted such that the distal end is offset further away from the user's eyes 210, 220. Thus, the depth plane extent that was previously only within the depth plane region 1809 is now also encompassed by the adjusted depth plane 240B1. In the example of FIG. 17, the depth overlap 1812 encompasses a depth range that can be twice the size of the determination error 1605 illustrated in FIG. 16. In some embodiments, if the display system can reliably determine the user's fixation point within a particular range of depths (e.g., 0.1 diopter, 0.2 diopter, etc.), the depth overlap may extend into only the adjacent depth planes by a particular range of depths.

[0249] When presenting virtual content, the display system may present it on either the adjusted depth plane 240B1 or the depth plane 240A. To select a particular depth plane for presenting virtual content, depth overlap may be considered as an extension of either depth plane. For example, if the user is gazing at a fixation point within the depth plane region 1808 that includes a fixation point within the depth overlap 1812, the display system may maintain the presentation of the virtual content on the depth plane 240A. However, if the user is gazing at a fixation point only within the depth plane region 1809, i.e., the fixation point is not included within the depth overlap 1812, the system selects the adjusted depth plane 240B1 and presents the virtual content. Similarly, the presentation of the virtual content on the adjusted depth plane 240B1 may be maintained if the user is gazing at a fixation point within the depth plane region 1809 that includes a fixation point within the depth overlap 1812. However, as soon as the fixation point moves outside the depth plane range 1809 or the depth overlap 1812, the depth plane 240A is presented to present the virtual content.

[0250] Figures 18A-18B illustrate the representation of the user's field of view 1800 of the display system and further illustrate the use of one or more depth plane range overlaps. The field of view 1800 includes representations of a first depth plane 240A and a second depth plane 240B. As illustrated, the depth plane range may define a volume of the real-world space associated with each depth plane (e.g., volumes 1808, 1809). For example, the depth plane range of depth plane 2 240B extends from the proximal depth 240B to the distal depth 240B. Virtual objects to be presented at depths within the proximal range 240B to the distal 240B may be presented with a wavefront divergence corresponding to depth plane 2 (also identified as reference numeral 240B). As an example, light containing image information regarding the virtual object may be output via a waveguide associated with depth plane 2. Additionally, the wavefront divergence of any virtual object to be presented at depths within the proximal range 240B to the distal 240B may be the same and, thus, may be associated with depth plane 2. It should be understood that the size and shape of the depth plane may be different from that illustrated in FIG. 18A. For example, the volume defining the depth plane may have a curved or other arbitrary shape in some embodiments.

[0251] As described above, the display system may determine a fixation point at which the user's eye is fixated. If the fixation point is within the proximal range 240B to the distal 240B, the display system may present virtual content with a wavefront divergence associated with depth plane 2 240B. If the user then fixates on a location within the depth plane range encompassed by depth plane 1 240A, the display system may present the content with a wavefront divergence associated with depth plane 1 240A. As described above, the display system may be a variable focus display system such that a single depth plane is utilized for any frame presented to the user. For example, one waveguide may be utilized and output all virtual content for each frame.

[0252] As shown, depth plane 1240A and depth plane 2240B are each shown to be positioned at a particular nominal focus depth from the user's eye. For example, depth plane 2 240B is shown to be set to the nominal depth perceived by the user such that, if depth plane 2 240B is selected to present virtual content, the virtual content will provide a cue for depth adjustment associated with the nominal depth. Thus, the perceived depth of the virtual content considering only depth adjustment will be the nominal depth. In some embodiments, each depth plane range may be of the same size (e.g., in diopter units) and may include, for example, the same range of depths. As an example, depth plane 2 240B may be set to a nominal depth of 1 diopter and may include a depth plane range from 0.66 diopters to 1.33 diopters. Similarly, as an example, depth plane 1 240A may be set to a nominal depth of 0.33 diopters and may include a depth plane range from 0 diopters to 0.66 diopters. Thus, the display system may include an overall depth plane range from 0 diopters to 1.33 diopters. The embodiment of FIG. 18A illustrates two depth planes, but additional depth planes may be utilized to further divide the overall depth plane range and / or increase the distance at which the user can fixate on the virtual content (e.g., proximity to the user) without exceeding acceptable depth adjustment - vergence - divergence motion inconsistencies (e.g., the proximal depth of the overall depth plane range may be set to 1.66 diopters, 2 diopters, etc.).

[0253] The depth plane ranges encompassed by each depth plane may optionally be based on a vergence-accommodation mismatch tolerance such that the vergence cues associated with the presentation of virtual content in the depth plane do not overly mismatch the vergence-divergence motion cues to cause viewer discomfort. For an example of the depth plane range encompassed by depth plane 1 240A that is 0 diopters to 0.66 diopters, the vergence cues for the virtual content presented in depth plane 1 240A may correspond to 0.33 diopters. In this example, the threshold vergence-accommodation mismatch may be 0.33 diopters, and in other examples, the mismatch may be 0.2 diopters, 0.5 diopters, or any other suitable value for avoiding viewer discomfort. As explained above, the vergence-accommodation mismatch tolerance indicates the maximum difference in the perceived depth of the virtual content associated with the vergence-divergence motion cues and the vergence cues. As the difference between the vergence-divergence motion cues and the vergence cues increases, for example, if the depth plane ranges of each depth plane extend too far apart, the user may experience a negative physiological response. Thus, the vergence-accommodation mismatch tolerance may be utilized to define the depth plane ranges encompassed by each depth plane.

[0254] In the embodiment of FIG. 18A, the proximal range of depth plane 1 240A corresponds to the distal range of depth plane 2 240B. As explained above with respect to FIG. 16, the fixation point located near this boundary may be determined to be encompassed by either depth plane 1 240A or depth plane 2 240B due to the uncertainty at the precise location.

[0255] FIG. 18B illustrates a representation of the field of view 1800 and includes a depth overlap 1812. As shown, the distal boundary of the depth plane 2 240B extends further in depth such that the adjusted depth plane 2 240B1 encompasses a range of depths that was previously only covered by the depth plane 1 240A. To ensure that the adjusted depth plane 2 240B1 covers the same range of depths as in FIG. 18A, the proximal boundary of the depth plane 2 240B1 also extends further in depth. For example, as described above, the range of depths encompassed by a depth plane may be based on a focus-convergence / divergence motion mismatch tolerance. In some embodiments, the focus-convergence / divergence motion mismatch tolerance may depend on the included depth. For example, the focus-convergence / divergence motion mismatch tolerance may be greater for a range of depths that is further from the user than for a range of depths that is closer to the user. For example, the depth plane 1 240A may be configured to encompass a larger range of depths than the depth plane 2 240B1. Similarly, the nominal focal depth of a depth plane may be set at a location that is not at the center between the distal and proximal boundaries of the depth range encompassed by the depth plane. For example, the range of depths from the distal boundary of the depth plane to the nominal focal depth may be larger or smaller than the range of depths from the nominal focal depth to the proximal boundary of the depth plane.

[0256] Since the proximal and distal boundaries 240B1 of the depth plane 2 are further offset in depth, the nominal focal depth of the depth plane 2 240B1 is similarly adjusted. For example, the nominal focal depth of the depth plane 2 240B1 may be positioned at the center between the proximal boundary 240B1 and the distal boundary 240B1. Thus, when the depth plane 2 240B1 is selected to present virtual content, the light output from the waveguide associated with the depth plane 2 240B1 will present light with a wavefront divergence corresponding to the adjusted nominal focal depth. Further, due to the offset in depth of the depth plane 2 240B1, a reduction in the renderable volume 1814 may occur. For example, the range of depths that was previously encompassed by the depth plane 2 240B1 may not be included here.

[0257] Examples for determining the depth repetition 1812 follow. In the following examples, the exemplary vergence-accommodation disparity tolerance is 0.33 diopters, the exemplary display system includes two depth planes, the first depth plane is set to a nominal focal depth of 0.33 diopters, and the second depth plane is set to a nominal focal depth of 1 diopter.

[0258] To determine the nominal focal depth of the second depth plane, an indication of the fixation point determination error may be obtained. The nominal focal depth may then be determined based on the fixation point determination error.

[0259] For example, in some embodiments, the nominal focal depth may be equivalent to the following.

[0260] 3 * (vergence-accommodation disparity tolerance)-2 * (fixation point determination error)

[0261] For an exemplary fixation point determination error of 0.1 diopters, the nominal focal depth of depth plane 2 in the foregoing example would be 0.79 diopters. Since the vergence-accommodation disparity is 0.33 diopters, the depth plane range of depth plane 2 would be 0.46 to 1.12 diopters.

[0262] Accordingly, the depth repetition would be 0.46 diopters to 0.66 diopters. For example, the distal end of depth plane 2 would be determined to be 0.46 diopters, and the proximal end of depth plane 1 would be 0.66 diopters.

[0263] As another example, for a fixation point determination error of 0.25 diopters, the nominal focal depth of depth plane 2 would be 0.49, and the depth plane range of depth plane 2 would be 0.11 to 0.82 diopters.

[0264] For example, the adjustment for a nominal depth of focus of 1 diopter to 0.49 diopter may be a modification of the hardware included within the display system. For example, the adjustment of the waveguide representing depth plane 2 may be implemented such that the wavefront divergence of the light output through the waveguide corresponds to a perceived depth of 0.49 diopter. Optionally, the waveguide may be adjustable via instructions executed by the display system. As an example, in some embodiments, the external coupling optical elements 570, 580, 590, 600, 610 described above with respect to FIG. 6 may be diffraction gratings adjustable via an applied electric field. Thus, as improvements are made against the fixation point determination error, the resulting depth overlap can correspondingly be reduced. As will be described below with respect to FIG. 19, the depth plane range encompassed by each depth plane, including depth overlap, may be maintained as information accessible by the display system. When rendering content for presentation to the user, the display system may utilize the maintained information to select the depth plane for presenting the virtual content.

[0265] FIG. 19 is a flowchart of an exemplary process 1900 for presenting virtual content. For convenience, process 1900 may be described as being performed by a display system (e.g., may include processing hardware and software, and optionally provides information to one or more computers or other external systems for processing, e.g., offloads processing to an external system and receives information from an external system, wearable display system 60 (FIG. 9D)).

[0266] In block 1902, the display system determines the fixation depth at which the user's eyes are fixated. For example, the display system may determine the three-dimensional fixation points of the user's eyes for each frame that is rendered and presented to the user, or the display system may determine a threshold number of fixation points for each frame that is being rendered. As an example, the display rate of the frames presented to the user may be a specific rate (e.g., 30 Hz, 60 Hz, 120 Hz, etc.), and the display system may determine the three-dimensional fixation points at a higher rate (e.g., 60 Hz, 120 Hz, 240 Hz, etc.). In this way, the display system may utilize the determined three-dimensional fixation points to determine the exact location at which the user is fixated. For example, saccadic eye movements, temporary eye movements such as when the user briefly looks at something else, etc. may be removed. As described above, the display system may include sensors for monitoring information associated with the user's eyes (e.g., eye orientation). A non-exhaustive list of sensors includes infrared sensors, ultraviolet sensors, and visible wavelength light sensors. The sensors may optionally output infrared, ultraviolet, visible light, and / or polarization onto the user's eyes and determine the reflection of the light output from the user's eyes. As an example, infrared light may be output by an infrared light emitter, and an infrared light sensor may be used to image the eyes. It should be understood that the sensors, which may include a light emitter, may correspond to the imaging device 630 of FIG. 6.

[0267] The display system may track the user's gaze by using sensors to determine a line of sight associated with each eye (e.g., a vector extending from the user's eye such as extending from the fovea through the lens of the eye) and an intersection of the lines of sight of each eye. For example, the display system may output infrared light onto the user's eyes, and reflections from the eyes (e.g., corneal reflections) may be monitored. A vector between the pupil centers of the eyes (e.g., the display system may determine the center of gravity of the pupil, for example, through infrared imaging) and the reflections from the eyes may be used to determine the line of sight of the eyes. The intersection of the lines of sight may thus be assigned as a three-dimensional fixation point. Optionally, the display system may utilize orientation information associated with the display system (e.g., information describing the orientation of the display system in three-dimensional space) when determining the fixation point.

[0268] As another example, the display system may utilize one or more imaging devices (e.g., a camera) along with a threshold number of lights per eye, e.g., LEDs (e.g., four LEDs). The threshold number of LEDs may emit light that illuminates each eye, and one or more imaging devices may capture one or more images of each eye. The center of the pupil of each eye (e.g., the centroid) may be determined based on the location of the light from each LED such that it can be identified from the image of the eye (e.g., four flashes from the LEDs may be visible on the pupil of each eye within each image). The optical axis of each eye may then be determined based on the center of the pupil. As described above, prior to use of the display system, the display system may be calibrated for an individual user, and optionally, the display system may maintain calibration (e.g., training) information for one or more users. For example, the user may have a user account associated with the display system, and optionally, the display system may access calibration information stored by an external system that communicates with the display system via a network (e.g., the Internet). As an example of calibration, the user may be required to correlate the actual location of an object in space with the eye line of sight such that a determination of the difference between the optical axis of the eye and the visual axis of the eye can be made. For example, a target object may be moved to a threshold number of real-world positions (e.g., five positions, nine positions, twelve positions), and a polynomial map may be determined to define coefficients for use in determining the line-of-sight vector. By utilizing the polynomial map, the user's visual axis can be determined more accurately. Optionally, instead of determining the line-of-sight vector, the interpupillary distance of the user between the pupils of the user's eyes may be utilized (e.g., the distance between the centers of the pupils of the two eyes). As an example, objects closer to the user (e.g., proximal) may have a smaller interpupillary distance, and these interpupillary distances may correlate to different depths along the z-axis.

[0269] In some embodiments, the display system may be configured to monitor a fixation point determined to track an object being viewed by a user. For example, the display system may determine that the user is viewing a first virtual object based on a determined three-dimensional fixation point corresponding to a three-dimensional location where the first virtual object is presented. Additionally, the display system may determine that the user is fixating on a location that does not correspond to a virtual object and may determine that a real-world object is likely to be located at the fixation point.

[0270] Continuing to refer to FIG. 19, at block 1904, the display system obtains location information associated with a virtual object for presentation to the user. Prior to rendering the virtual object for presentation to the user (e.g., via the output of a waveguide as described above), the display system may obtain three-dimensional location information associated with the virtual object. For example, as described above, the virtual object may be presented to the user such that the content appears to be located within the real world (e.g., the content may be located in different depth planes within the user's field of view). It should be understood that the display system may include or have access to a three-dimensional map of the surrounding environment that includes the intended location of any virtual content within the surrounding environment. Referring to this map, the display system may access and provide information that defines the three-dimensional location of virtual content within the user's field of view (e.g., a location within the display frustum as illustrated in FIGS. 18A-18B).

[0271] As described above, the location information regarding the virtual object may include a three-dimensional location. Based on the three-dimensional location, when the user gazes at the virtual object, a specific depth plane may be selected and associated with a specific perceived depth so that all virtual content can be presented. For example, the perspective cue associated with the gazed virtual object would correspond to a specific perceived depth as determined from the convergence / divergence motion cue.

[0272] In block 1906, the depth plane for presenting the virtual object is selected. As described above with respect to FIGS. 17-18, the display frustum may include one or more depth overlaps that can include both adjacent depth plane ranges. To select a depth plane, the display system may identify whether the determined gaze depth (e.g., as described above with respect to block 1902) is within the depth plane range encompassed only by the depth plane or within the depth plane range encompassed by the depth overlap. In other words, when the display system presents the virtual object on a specific depth plane, the display system may maintain the presentation of the virtual object on that specific depth plane if the gaze depth is encompassed by the specific depth plane (e.g., within the depth plane range encompassed only by the specific depth plane or within the depth plane range included in the depth overlap encompassed by the specific depth plane and the adjacent depth plane).

[0273] Regarding the fixation depth within the depth plane range included only by a certain depth plane, the display system may select that depth plane and present virtual objects. For example, regarding the fixation depth within the depth plane range included by depth overlap, including the depth range included by the first depth plane and the second depth plane, in some embodiments, the display system may identify the nearest depth plane with only the determined fixation depth. For example, if one or more fixation depths prior to the current fixation depth are determined to be within the depth overlap, the display system may identify the nearest fixation depth that is only within either the first depth plane or the second depth plane. The depth plane with only the identified fixation depth may then present virtual objects. As described above, the depth overlap may represent the extent of the depth plane range included by the first depth plane and the second depth plane. Thus, as an example, if the fixation depth is within the second depth plane and the fixation depth is then within the depth overlap, the display system may withhold the selection of the second depth plane for presenting virtual content to the user.

[0274] Optionally, if the fixation depth is within a specific depth overlap and the nearest previous fixation depth is within a depth plane range that does not include the specific depth overlap, the display system may select the depth plane having the nominal focus depth closest to that fixation depth. For example, the user may fixate on a virtual object positioned distally from the user and then rapidly fixate on a virtual object positioned proximally to the user. In this example, the first depth plane may be selected while the user is fixating on the distal object, and when the user fixates on the proximal object, the user's fixation may be within a specific depth overlap between the second depth plane and the third depth plane. Since the fixation depth is within the specific depth overlap, the display system may select either the second depth plane or the third depth plane based on whether the nominal focus depth of either depth plane is closer to the determined fixation depth. Optionally, the display system may randomly select from among the depth planes.

[0275] In some embodiments, the confidence level may optionally be determined with respect to the fixation depth. For example, the display system may determine a confidence level that the determined fixation depth accurately represents the user's actual fixation. For example, poor lighting conditions may increase the difficulty associated with determining the user's fixation depth, and the confidence level may be reduced. Additionally, rapid eye movements may also increase the difficulty of determining the fixation depth, and the confidence level may be reduced. Optionally, the display system may utilize the determined confidence level to inform the selection of the depth plane for presenting the virtual object. For example, if the fixation depth is within a depth overlap, the display system may select a depth plane that includes the depth overlap having a nominal focus depth closer to the fixation depth. The display system may utilize the determined confidence level along with the proximity of the fixation depth to the edge of the depth overlap. For example, if the fixation depth is within the depth overlap but within a threshold depth from the edge of the depth overlap, the display system may select a depth plane having the nominal focus depth closest to the edge. This threshold depth may be based on a confidence level such that the threshold depth may be decreased as the determined confidence level of the display system increases. Additionally, the size of the depth overlap may be adjusted based on the confidence level. For example, as the confidence level increases, the uncertainty with respect to the fixation depth decreases, and the depth overlap may be reduced such that the overlap between adjacent depth planes is reduced.

[0276] In block 1908, the display system presents a virtual object to the user. For example, the display system may cause the presentation to occur on the selected depth plane such that the perspective adjustment cue of the presented virtual object corresponds to the selected depth plane. As described above, a perceivable flicker may become apparent to the user in response to a switch from a first depth plane to a second depth plane. Similarly, the user will be required to adjust the perspective of the light output provided via the display system (e.g., change the shape of the eye's lens based on the perspective adjustment cue).

[0277] In some embodiments, as will be described below with respect to FIG. 20, the display system may delay switching of depth planes until an event occurs that reduces the user's perception of the switch (e.g., a perception-limiting event). In some embodiments, such an event may be (1) a blink or (2) the occurrence of a saccadic eye movement. For example, in response to identifying that a depth plane switch should occur, the display system may store information (e.g., a flag) indicating that the display system should perform the switch to the selected depth plane (e.g., as described in block 1906) in response to detecting a blink or saccadic eye movement by the user. Prior to performing the switch, the display system may render the virtual object and present it on the previous depth plane, and after the blink or saccadic eye movement, render the virtual object and present it on the selected depth plane. Thus, the display system may use blinks and / or saccadic eye movements to mask the switching of depth planes.

[0278] In addition, in some embodiments, the display system may update the presentation in different (e.g., switched) depth planes without determining that a blink or saccadic eye movement has been made by the user. For example, if the user does not perform a blink or saccadic eye movement within a threshold amount of time (e.g., 10 seconds, 30 seconds, 60 seconds), the display system may switch to presenting the virtual content in a different depth plane. Further, if the different depth plane is at a nominal focus depth that exceeds a threshold depth from the nominal focus depth of the current depth plane, the display system may update the presentation without waiting for a blink or saccadic eye movement. As an example, if the currently selected depth plane is at a nominal focus depth of 0.2 diopters and the depth plane to be switched to is at a nominal focus depth of 1 diopter, the display system may update the presentation without waiting for a blink or saccadic eye movement, for example, due to the potential for a large accommodation - vergence / divergence movement mismatch if the switch were to occur without a blink. In addition, the threshold amount of time to wait for the user to perform a blink or saccadic eye movement may be based on the difference in the accommodation cues to be made to the presented virtual objects. For example, as the difference in the nominal focus depth between two depth planes increases, the threshold amount of time may decrease.

[0279] FIG. 20 is a flowchart of an exemplary process 2000 for switching depth planes and adjusting the presentation of content to the user while the user's perception is limited. For convenience, process 2000 may be described as being performed by a display system (e.g., including processing hardware and software, optionally providing information to one or more computers or other external systems of the process, e.g., offloading processing to an external system and receiving information from an external system, wearable display system 60).

[0280] In block 2002, the display system obtains information indicating that a switching of the depth plane for presenting virtual objects should occur. As described above with respect to FIG. 19, the display system may determine the fixation depth at which the user is fixating (e.g., monitoring a three-dimensional fixation point), and further determine that the depth plane at which the content is provided should be switched based on the determined fixation depth. For example, the user may fixate on a depth encompassed by a first depth plane and subsequently fixate on a depth encompassed by a second depth plane. In response to determining that a virtual object should be presented on the second depth plane, the display system may store information indicating that the switching should be performed.

[0281] Next, the display system may determine whether an event that reduces the user's perception of the switching occurs. Such an event may be a blink of the user's eyelids and / or a saccadic eye movement. For example, in block 2004, the display system determines whether the user has blinked. As an example, the display system may monitor the user's eyes, such as by using a camera 630 (FIG. 6) to obtain an image of the user's eyes, and if the pupil is no longer detected in the obtained image (e.g., as described above with respect to FIG. 19), the display system may determine that the user is blinking. As another example, an exemplary eye-tracking algorithm may be utilized (e.g., a starburst algorithm), and if the eye-tracking algorithm fails to detect the reflection of light from the user's pupil or eye, the display system may determine that the user is blinking.

[0282] Simultaneously with or as an alternative to implementing block 2004, the display system may implement block 2006. In block 2006, the display system determines whether the user has performed saccadic eye movements. Saccadic eye movements represent rapid eye movements during which the user's perception is limited. The display system may monitor saccadic eye movements, for example, using an acquired image of the user's eye above a threshold frequency (e.g., 500 Hz, 750 Hz, 1200 Hz, etc.). Since the duration of saccadic eye movements can be substantially shorter than the duration of a blink, a higher frequency imaging device may be utilized to detect saccadic eye movements, or the same sensor operating at a higher frequency may be used.

[0283] As an example of determining saccadic eye movements, the display system may determine the rotational speed of the pupil of the eye and utilize the rotational speed to at least partially distinguish saccadic eye movements from smooth pursuit eye movements performed by the eye. The display system may, for example, utilize a gyroscope to obtain information indicating the user's head pose, and if the measured rotational speed of the pupil exceeds a threshold speed associated with smooth pursuit eye movements and the user's head is not moving above the threshold speed, the display system may determine that saccadic eye movements are being performed.

[0284] In block 2010, the display system updates the selection of the depth plane and presents the virtual object to the user. In response to the detection of a blink or saccadic eye movement, the display system may perform an adjustment of the depth plane. Alternatively, in block 2008, if a blink or saccadic eye movement is not determined to exceed a threshold amount of time, the display system may perform an adjustment of the depth plane. Exemplary threshold amounts of time may be 20 seconds, 30 seconds, 120 seconds, a user-selectable amount of time, etc.

[0285] In addition, the display system waits for the user to perform a blink or a saccadic eye movement, so that the user can fixate on a depth included by a depth plane different from the adjusted depth plane. For example, with respect to block 2002, the user can fixate on a depth at which the adjustment of the depth plane should occur. While waiting for the user to perform a blink or a saccadic eye movement and updating the selection of the depth plane to the adjusted depth plane, the user can fixate on a new fixation depth. The display system may then optionally update the selection of the depth plane to a depth plane that includes the new fixation depth. Thus, if the user then performs a saccadic eye movement or a blink, the display system may select a depth plane that includes the new fixation depth. Accommodation for viewer eye fatigue

[0286] As is apparent from FIGS. 11 - 15, typically there exists a range of distances that are very close to the viewer and where the accommodation - vergence / divergence movement mismatch is large, but the content can still be displayed. As discussed herein, such content can cause viewer discomfort and as a result may be undesirable. In some embodiments, display content determined to cause an unacceptable accommodation - vergence / divergence movement mismatch is modified to protect against viewer discomfort.

[0287] FIG. 21A illustrates an example of a method 4000 for maintaining viewer comfort when image content provides an accommodation - vergence / divergence movement mismatch that exceeds a threshold. At block 4002, the image content is analyzed to determine whether the image content will result in an accommodation - vergence / divergence movement mismatch that exceeds a threshold. At block 4004, if the mismatch is determined to exceed the threshold, the image content is modified. In some embodiments, the accommodation - vergence / divergence movement mismatch threshold is 0.5 diopters or less or 0.33 diopters or less.

[0288] Modifying the image content may include one or more of reducing the duration for which the content is displayed, fading the image content (e.g., by reducing the resolution or spatial frequency of the image content), or simply not displaying content that exceeds a threshold. In some embodiments, when the resolution of the image content is decreased, the degree of decrease in the resolution of the image content increases with an increase in the accommodation - vergence / divergence motion inconsistency (e.g., as the content gets closer to the viewer).

[0289] It should be understood that even when the accommodation - vergence / divergence motion inconsistency is acceptable, long - term use of the head - mounted display device may still potentially cause some eye strain. Referring now to FIG. 21B, an example of a method 5000 for reducing viewer eye strain is illustrated. At block 5002, the presence of eye strain in the user is determined. At block 5004, if it is determined that eye strain is present, the image content is modified.

[0290] It should be understood that the step of determining the presence of eye fatigue may include, for example, the step of imaging one or both eyes of the user using the camera assembly 500 (FIG. 6). The step of determining the presence of eye fatigue may include the step of detecting one or more of mydriasis, convergence fluctuation, and pupil fluctuation. In some embodiments, the step of determining the presence of eye fatigue includes the step of measuring the galvanic skin response. In some other embodiments, the step of determining the presence of eye fatigue includes the step of detecting the duration of exposure to image content having a phoria-convergence / divergence movement mismatch greater than 0.25 diopters, greater than 0.33 diopters, or greater than 0.5 diopters. Although the aforementioned detected risk factors may also be caused by other problems, one or more of these methods for determining the presence of eye fatigue may be implemented together such that multiple variables are evaluated to increase the accuracy of the determination of eye fatigue. For example, one or more of the aforementioned methods may be implemented and evaluated to determine whether the risk factor is at least partially associated with the use of a head-mounted display device. Additionally, the occurrence of the risk factor is measured over time and correlated with the content being displayed by the display system and / or the duration of use of the display system, which may further increase the confidence that the risk factor is a result of the display. Additionally, the determination of eye fatigue may involve the step of evaluating a change in one or more of these variables, or the step of determining whether the variable exceeds a predetermined threshold.

[0291] When eye strain is determined to be present, the image content is modified to reduce eye strain. In some embodiments, the step of modifying the image content may include one or more of increasing the size of features of the image content, reducing the resolution of the image content, and displaying the image content on a depth plane farther from the viewer than originally defined for the image content. For example, when displaying content for a video game as an example, content that prompts the viewer to focus may be selected to be on a farther depth plane. In one embodiment, rather than interacting with the viewer's closest virtual object, the game may be instructed to provide an interaction where the object is at a distance from the viewer.

[0292] In some embodiments, the determination of eye strain and the modification of the image content may be performed continuously. In response to a determination that eye strain is no longer present, the image modification may stop. In some other embodiments, the image modification may be set to occur over a set duration or until an event occurs (e.g., when a viewer playing a video game reaches a new level). Structure for supporting and / or balancing a head-mounted display

[0293] Referring now to FIGS. 22A-22B, as discussed herein, a head-mounted display system can be bulky or heavy, which can detract from the comfort of the system, especially for long-term use. Additionally, the distribution of the weight of the system on the user's head can be uneven, which can also contribute to discomfort during long-term use. Advantageously, the head-mounted display system may be equipped with one or more support structures to increase user comfort.

[0294] FIG. 22A illustrates an exemplary embodiment of a head-mounted display with a support structure. As shown in FIG. 22A, user 90 is depicted wearing a head-mounted display system, which includes a frame structure coupled to a display 70 positioned in front of the eyes of user 90.

[0295] Support structure 900 is included as part of the head-mounted display system and may, for example, distribute the weight of the display to different parts of the head of user 90 for weight balance and pressure point reduction (such as the user's nose due to the weight distributed to the nose pads of the display system). In some embodiments, support structure 900 is configured to extend from one side of the head of the user to the other side of the head. In some embodiments, support structure 900 may optionally include an audio transducer (e.g., a speaker) 100. Support structure 900 may be a band (e.g., a metal band and / or a plastic band) configured to extend from one side of the head to the other side of the head. In some embodiments, support structure 900 traverses the head laterally from ear to ear. In some embodiments, support structure 900 may traverse the head longitudinally from the eyes to the back of the head. In some embodiments, the support structure may include a plurality of such support structures 900 that traverse the head at an angular interval in either the lateral or longitudinal direction.

[0296] The support structure 900 may cross the user's head at different angles. FIG. 22A illustrates an embodiment where the support structure 900 crosses the user's head horizontally, i.e., approximately from ear to ear. The angle 904 may be defined between the plane 902 that intersects the user's eyes and ears and the centerline of the support structure extending from one side of the user 90's head to the other side of the head. In certain embodiments, the angle 904 is about 35 - 55 degrees. In some other embodiments, the angle 904 is about 80 - 100 degrees. In yet other embodiments, the angle 904 may be about zero degrees, for example, when the support structure 900 is approximately in the plane 902 of the user's eyes and ears. As discussed herein, the head-mounted display system may include a plurality of such support structures 900 that cross the user's head at various angles 904. Further, in some embodiments, the position of the support structure 900 relative to the user's head may be moved such that the angle 904 relative to the plane 902 is adjustable.

[0297] The frame 80 may intersect the support structure 900 at various positions. In some embodiments, for example, as shown in FIG. 22A, the frame 80 may intersect the support structure 900 above the user's ear. In some other embodiments, the frame 80 may intersect at the sound converter 100. In yet other embodiments, the frame 80 and the support structure 900 may be integrated into a single structure that crosses the user's head, as described herein. In some embodiments, the sound converter 100 is attached to the frame 80. In some embodiments, the sound converter may be attached to the support structure 900. In other embodiments (not shown), the sound converter may be attached entirely by other means or separate structures. As described herein, in certain embodiments, the head-mounted display may include the frame 80 and the support structure 900, but may not contain the sound converter 100.

[0298] FIG. 22B illustrates an exemplary embodiment where the sound converter is a speaker that covers the user's ear. The speaker may optionally be coupled to the frame 80 in the depicted configuration and positioned across the user's ear and / or adjacent to the user's ear canal (in one embodiment, another speaker, not shown, is positioned across the ear and / or adjacent to the user's other ear canal to provide stereo / formable sound control).

[0299] It should be understood that the processes, methods, and algorithms described herein and / or depicted in the figures are each embodied in one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in a dynamic link library that can be compiled and linked into an executable program, or written in an interpreted programming language. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0300] Furthermore, because the functional implementations of the present disclosure are sufficiently mathematical, computer, or technically complex, a special-purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results in substantially real time. For example, a video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0301] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. In some embodiments, the non-transitory computer-readable medium may be part of one or more than one of local processing and data module (140), remote processing module (150), and remote data repository (160). The methods and modules (or data) may also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal) and may take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0302] Any process, block, state, step, or functionality in a flowchart described in and / or depicted in the accompanying figures herein is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in a suitable other sequence, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the implementations described herein is for purposes of illustration and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products.

[0303] In the foregoing specification, the invention has been described with reference to its specific embodiments. However, it will be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a limiting sense.

[0304] In fact, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure.

[0305] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Further, features may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination. No single feature or group of features is necessary or essential to every embodiment.

[0306] In particular, conditional clauses used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes certain features, elements, and / or steps, other embodiments do not include them, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional clauses generally do not imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more than one" or "at least one" unless otherwise defined. Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not necessary for achieving the desired result, i.e., that such operations are performed in the particular order shown, or in a sequential order, or that all of the illustrated operations are performed. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted can also be incorporated into the exemplary methods and processes schematically illustrated.For example, one or more additional operations can be performed before, after, simultaneously with, or during any of the illustrated operations. Additionally, the operations can be rearranged or reordered in other implementations. In certain situations, multitasking and parallel processing can be advantageous. Further, the separation of the various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0307] Accordingly, the claims are not intended to be limited to the implementations shown herein but should be accorded the widest scope consistent with the disclosure, principles, and novel features disclosed herein.

Claims

[Claim 1] The invention described in this specification.

Images