Display system and method for determining alignment between a display and a user's eyes

Abstract

To provide favorable display systems and methods for determining registration between display and a user's eyes.SOLUTION: A wearable device may include an HMD for rendering a 3D virtual object which appears to be located in an ambient environment of a user of a display. The relative positions of the HMD and one or more eyes of the user may not be in desired positions to receive or register image information outputted by the HMD. For example, the HMD-eye alignment may vary for different users and may change over time (e.g., as a given user moves around or as the HMD slips or otherwise becomes displaced). The wearable device may determine a relative position or alignment between the HMD and the user's eyes. Based on the relative positions, the wearable device may determine if it is properly fitted to the user, may provide feedback on the quality of the fit to the user, and may take actions to reduce or minimize effects of any misalignment.SELECTED DRAWING: Figure 13A

Description

Technical Field

[0001] (Claim of Priority) This application claims priority to U.S. Provisional Patent Application No. 62 / 644,321, filed on Mar. 16, 2018, entitled “DISPLAY SYSTEMS AND METHODS FOR DETERMINING REGISTRATION BETWEEN A DISPLAY AND A USER’S EYES”; U.S. Provisional Patent Application No. 62 / 618,559, filed on Jan. 17, 2018, entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS”; and U.S. Provisional Patent Application No. 62 / 702,849, filed on Jul. 24, 2018, entitled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS”. The above applications are hereby incorporated by reference in their entireties. (Incorporation by Reference)

[0002] This application incorporates by reference in its entirety each of the following patent applications and publications: U.S. Patent Application No. 14 / 555,585, filed November 27, 2014, and published as U.S. Patent Publication No. 2015 / 0205126 on July 23, 2015; U.S. Patent Application No. 14 / 690,401, filed April 18, 2015, and published as U.S. Patent Publication No. 2015 / 0302652 on October 22, 2015; U.S. Patent Application No. 14 / 212,961, filed March 14, 2014, and issued as current U.S. Patent No. 9,417,452 on August 16, 2016; U.S. Patent Application No. 14 / 331,218, filed July 14, 2014, and published as U.S. Patent Publication No. 2015 / 0309263 on October 29, 2015; U.S. Patent Publication No. 2016 / 0270656, published on September 27, 2017; U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015; U.S. Patent Publication No. 2015 / 0016777; U.S. Patent Application No. 15 / 274,823; U.S. Patent Application No. 15 / 296,869; U.S. Patent Application No. 15 / 717,747, filed September 27, 2017; U.S. Patent Application No. 15 / 497,726, filed April 26, 2017; U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017; U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017; U.S. Patent Application No. 15 / 341,760, filed November 2, 2016, and published as U.S. Patent Publication No. 2017 / 0122725 on May 4, 2017; U.S. Patent Application No. 15 / 341,822, filed November 2, 2016, and published as U.S. Patent Publication No. 2017 / 0124928 on May 4, 2017; U.S. Provisional Patent Application No. 62 / 618,559, filed January 17, 2018; and U.S. Provisional Patent Application No. 62 / 642,761, filed March 14, 2018. (Technical Field)

[0003] The present disclosure relates to display systems, including virtual reality and augmented reality display systems, and more particularly, to systems and methods for evaluating the fit of a display on a user. [Background Art]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as being real. Virtual reality, i.e., the "VR" scenario, typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality or "MR" is related to the fusion of the real and virtual worlds to create a new environment where physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is very complex, and the production of VR, AR, or MR technologies that facilitate a comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements is difficult. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Summary of the Invention

Means for Solving the Problems

[0005] Various embodiments of alignment observation and response within a mixed reality system are disclosed.

[0006] In some embodiments, a display system is provided for projecting light into a user's eyes to display virtual image content. The display system includes a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time to display the virtual image content, one or more eye-tracking cameras configured to image the user's eyes, and processing electronics communicatively coupled to the display and the one or more eye-tracking cameras. The processing electronics determine the position of the eyes based on an image of the eyes acquired using the one or more eye-tracking cameras, determine whether the position of the eyes is within a display alignment volume of the head-mounted display system, and provide a notification based on determining whether the position of the eyes is within the display alignment volume, the notification being configured to indicate at least that the display and the eyes are not properly aligned.

[0007] In some other embodiments, a display system is provided for projecting light into a user's eyes to display virtual image content. The display system includes a frame configured to be supported on the user's head, a head-mounted display disposed on the frame and configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time to display the virtual image content, one or more eye-tracking cameras configured to image the user's eyes, and processing electronics communicatively coupled to the display and the one or more eye-tracking cameras. The processing electronics are configured to determine an eye position based on an image of the eye acquired using the one or more eye-tracking cameras, determine whether the eye position exceeds a first threshold distance outside a viewing volume of the head-mounted display system, and in response to determining that the eye position exceeds the first threshold distance outside the viewing volume of the head-mounted display system, provide feedback to the user indicating that the display and the eyes are not properly aligned for output.

[0008] In some other embodiments, a display system is provided for projecting light into a user's eyes to display virtual image content. The display system includes a frame configured to be supported on the user's head, a head-mounted display disposed on the frame and configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time to display the virtual image content, one or more eye-tracking cameras configured to image the user's eyes, and processing electronics communicatively coupled to the display and the one or more eye-tracking cameras. The processing electronics are configured to determine whether light projected by the head-mounted display is properly aligned with the user's eyes, and to provide feedback to the user if the head-mounted display is not properly adjusted to fit the user for aligning the light projected by the display system.

[0009] In yet other embodiments, a method is provided for evaluating the alignment of virtual image content from a head-mounted display system by a user's eyes. The method includes determining a first position of the eye and determining whether the first position of the eye is within a display alignment volume of the head-mounted display system, where the display alignment volume is an imaginary volume associated with a proper fit of the head-mounted display system to the user's eyes, and providing a notification based on whether the position of the eye is within the display alignment volume, the notification indicating at least that the display and the eye are not properly aligned.

[0010] Additional examples are listed below.

[0011] Example 1. A display system configured to project light into a user's eyes and display virtual image content, a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, the display configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time and display the virtual image content, one or more eye-tracking cameras configured to image the user's eyes, processing electronics in communication with the display and the one or more eye-tracking cameras, determining the position of the eye based on an image of the eye obtained using the one or more eye-tracking cameras, determining whether the position of the eye is within a display alignment volume of the head-mounted display system, Based on determining whether the position of the eye is within the display alignment volume, a notification is provided, and the notification indicates at least that the display and the eye are not properly aligned. A processing electronic device configured as follows, A display system comprising.

[0012] Example 2. The processing electronic device is further configured to provide feedback to the user that the head-mounted display is not properly adjusted to fit the user in response to determining that the position of the eye is outside the display alignment volume, and the feedback is a notification provided based on determining whether the position of the eye is within the display alignment volume, the display system according to Example 1.

[0013] Example 3. The display system according to Example 1, further comprising at least one replaceable fitting component removably mounted on the frame and configured to adjust the fit of the frame.

[0014] Example 4. The display system according to Example 3, wherein the replaceable fitting component comprises a replaceable nasal bridge part configured to adjust the fit of the frame between the frame and the user's nasal bridge.

[0015] Example 5. The display system according to Example 3 or 4, wherein the replaceable fitting component comprises a replaceable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

[0016] Example 6. The display system according to any one of Examples 3-5, wherein the replaceable fitting component comprises a replaceable back pad configured to adjust the fit of the frame between the frame and the back of the user's head.

[0017] Example 7. The processing electronic device is further configured to provide a notification including providing feedback to the user that the head-mounted display is not properly adjusted to fit the user, and to provide a proposal to the user to replace the currently installed replaceable fitting part with another replaceable fitting part, a display system according to any of Examples 2-6.

[0018] Example 8. The display system according to any of Examples 1-7, further comprising one or more light sources disposed on the frame and illuminating the user's eyes, and one or more eye tracking cameras configured to form an image of the eyes using light from the one or more light sources.

[0019] Example 9. The display system according to Example 8, wherein the one or more light sources comprise at least two light sources disposed on the frame and illuminating the user's eyes.

[0020] Example 10. The display system according to any of Examples 8-9, wherein the one or more light sources comprise an infrared light emitter.

[0021] Example 11. The display system according to any of Examples 8-10, wherein the one or more light sources form one or more flashes on the eyes, and the processing electronic device is configured to determine the location of the cornea based on the one or more flashes.

[0022] Example 12. The display system according to any of Examples 1-11, wherein the eye position is the location of the center of rotation of the eye.

[0023] Example 13. The display system according to any of Examples 1-11, wherein the cornea has a corneal sphere with a center of curvature associated therewith, and the processing electronic device is configured to determine the location of the center of curvature of the corneal sphere.

[0024] Example 14. The processing electronic device is configured to provide a notification to the display by providing an instruction to increase the brightness of a plurality of pixels of the display with respect to other pixels of the display, and the plurality of pixels with increased brightness include pixels that are expected to receive a perception of darkening under inappropriate alignment. The display system according to Example 1.

[0025] Example 15. A display system configured to project light into a user's eye and display virtual image content, A frame configured to be supported on the user's head, A head-mounted display disposed on the frame, configured to project light into the user's eye and display virtual image content with different amounts of wavefront divergence for presenting virtual image content that appears to be located at different depths for different periods of time, One or more eye-tracking cameras configured to image the user's eye, A processing electronic device communicating with the display and the one or more eye-tracking cameras, Determining the position of the eye based on an image of the eye acquired using the one or more eye-tracking cameras, Determining whether the position of the eye exceeds a first threshold distance outside the viewing volume of the head-mounted display system, In response to the determination that the position of the eye exceeds the first threshold distance outside the viewing volume of the head-mounted display system, providing feedback to the user indicating that the display and the eye are not properly aligned for output, A processing electronic device configured as such, A display system comprising.

[0026] Example 16. The processing electronic device at least Determining whether the position of the eye is less than a second threshold distance from the eyepiece, In response to determining that the eye position is less than a second threshold distance from the head-mounted display system, providing feedback to the user indicating that the display and the eye are not properly aligned for output; A display system according to Example 15, configured to determine whether the eye position exceeds a first threshold distance outside the viewing volume by .

[0027] Example 17. The processing electronic device at least determining whether the eye position exceeds a second threshold distance from the eyepiece; In response to determining that the eye position exceeds a second threshold distance from the head-mounted display system, providing feedback to the user indicating that the display and the eye are not properly aligned for output; A display system according to Example 15, configured to determine whether the eye position exceeds a first threshold distance outside the viewing volume by .

[0028] Example 18. The processing electronic device at least determining whether the eye position exceeds a second threshold distance outside a sub-space of the field of view of the eye-tracking camera; In response to determining that the eye position exceeds a second threshold distance outside a sub-space of the viewing volume of the eye-tracking camera, providing feedback to the user indicating that the display and the eye are not properly aligned for output; A display system according to Example 15, configured to determine whether the eye position exceeds a first threshold distance outside the viewing volume by .

[0029] Example 19. The viewing volume of the head-mounted display is a volume through which light representing all pixels of the virtual image content presented by the head-mounted display is expected to pass, of the display system according to Example 15.

[0030] Example 20. A display system configured to project light into a user's eyes and display virtual image content, a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, configured to project light into the user's eyes and display virtual image content with different amounts of wavefront divergence for presenting virtual image content that appears to be located at different depths for different periods of time, one or more eye-tracking cameras configured to image the user's eyes, a processing electronic device in communication with the display and the one or more eye-tracking cameras, determining whether the light projected by the head-mounted display is properly aligned with the user's eyes, and providing feedback to the user if the head-mounted display is not properly adjusted to fit the user for aligning the light projected by the display system. A display system configured as such, comprising.

[0031] Example 21. The display system according to Example 20, further comprising at least one replaceable fitting component removably mounted on the frame and configured to adjust the fit of the frame.

[0032] Example 22. The display system according to Example 21, wherein the replaceable fitting component comprises a replaceable nose bridge portion configured to adjust the fit of the frame between the frame and the user's nose bridge.

[0033] Example 23. The display system according to Example 20 or 22, wherein the replaceable fitting component comprises a replaceable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

[0034] Example 24. The display system according to any one of Examples 20-23, comprising a replaceable back pad configured to adjust the fit of the frame between the frame and the back of the user's head.

[0035] Example 25. The display system according to any one of Examples 20-24, wherein the processing electronic device is further configured to include providing the user with a proposal to replace the currently installed replaceable fit component with another replaceable fit component to provide the user with feedback that the head-mounted display is not properly adjusted to fit the user.

[0036] Example 26. A method for evaluating the alignment of virtual image content from a head-mounted display system by a user's eye, determining a first position of the eye; determining whether the first position of the eye is within a display alignment volume of the head-mounted display system, the display alignment volume being an imaginary volume associated with a proper fit of the head-mounted display system to the user's eye; providing a notification based on determining whether the position of the eye is within the display alignment volume, the notification indicating at least that the display and the eye are not properly aligned; A method comprising the steps of:

[0037] Example 27. The method according to Example 26, wherein the head-mounted display system includes an eye-tracking camera, and the step of determining the first position of the eye includes imaging the user's eye using the eye-tracking camera.

[0038] Example 28. The method according to Example 27, wherein the first position of the eye is the position of the center of rotation of the eye, and further includes calculating the center of rotation of the eye based on imaging the eye by the eye-tracking camera.

[0039] Example 29. The head-mounted display system further includes the step of projecting light into the eyes and displaying virtual image content within the user's field of view, and displaying an indication that the wearable system is properly fitted, the method according to Example 26.

[0040] Example 30. The method according to any one of Examples 26-29, further including the step of automatically tracking the center of rotation of the eye over time and notifying the user when the center of rotation of the eye moves outside the alignment display volume.

[0041] Example 31. determining a second position of the eye; determining that the second position of the eye is within the display alignment volume; in response to determining that the second position of the eye is within the display alignment volume, providing additional feedback to the user indicating that the wearable system is properly fitted to the user; The method according to Example 26 or 29, further including.

[0042] Example 32. When the user's eyes are not within the display alignment volume, at least some of the pixels of the head-mounted display system are darkened or made invisible to the user, the method according to any one of Examples 26-31.

[0043] Example 33. When the position of the eye is outside the display alignment volume, further including the step of changing the field of view of the head-mounted display system, The head-mounted display system includes at least one display having a first field of view when the position of the eye is within the display alignment volume, and the display has a second field of view when the position of the eye is outside the display alignment volume, and the second field of view is smaller than the first field of view, the method according to any one of Examples 26-32.

[0044] Example 34. The method according to Example 33, wherein the step of providing a notification includes the step of providing feedback to the user within a second field of view.

[0045] Example 35. The wearable system comprises at least one replaceable fitting part, and further includes the step of providing a notification to the user indicating that the wearable system is not properly fitted to the user, wherein the notification includes a proposal or instruction to the user to replace the currently installed replaceable fitting part with an alternative replaceable fitting part, according to any of the methods described in Examples 26 - 34.

[0046] Example 36. The method according to Example 35, wherein the replaceable fitting part comprises at least one fitting part selected from the group consisting of a nose bridge pad, a forehead pad, and a back pad that fits between the wearable system and the back of the user's head.

[0047] Example 37. The wearable system comprises at least one replaceable nose bridge pad, and further includes the step of determining that the display of the head-mounted system is too low with respect to the eyes, and the step of providing a notification to the user includes the step of prompting the user to install a larger nose bridge pad, according to the method described in Example 36.

[0048] Example 38. identifying a plurality of pixels of a display of a head-mounted display system, where the first position of the eye is outside the display alignment volume, such that the user is expected to perceive being darkened; increasing the brightness of the plurality of pixels of the display relative to other pixels within the display to reduce the expected darkening; and further includes the method according to any of Examples 26 - 37.

[0049] Example 39. A display system configured to project light into a user's eyes and display virtual image content, a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time, and configured to display virtual image content rendered by a virtual rendering camera, one or more eye tracking cameras configured to image the user's eyes, a processing electronic device in communication with the display and the one or more eye tracking cameras, determining a distance from the display at which the eyes are located based on an image of the eyes obtained using the one or more eye tracking cameras, adjusting a focal length of the virtual rendering camera based on the determined distance, a processing electronic device configured as such, a display system comprising.

[0050] Example 40. A display system configured to project light into a user's eyes and display virtual image content, a frame configured to be supported on the user's head, a head-mounted display disposed on the frame, configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time, and configured to display virtual image content, one or more eye tracking cameras configured to image the user's eyes, a processing electronic device in communication with the display and the one or more eye tracking cameras, determining the position of the eyes relative to the display based on an image of the eyes obtained using the one or more eye tracking cameras, Based on the position of the eyes relative to the display, determine the amount of pixels of virtual image content that the user is expected to perceive as being darkened, Based on the determined amount of pixels, control the operation of the display, A processing electronic device configured as follows, A display system comprising.

[0051] Example 41. The processing electronic device is the display system according to Example 40, configured to determine the amount of pixels of virtual image content that the user is not expected to perceive as being darkened based on the position of the eyes relative to the display.

[0052] Example 42. The processing electronic device is Based on the position of the eyes relative to the display, by increasing the brightness of the pixels of virtual image content that the user is expected to perceive as being darkened, The display system according to Example 40 or 41, configured to control the operation of the display.

[0053] Example 43. The amount of pixels of virtual image content is the display system according to any of Examples 40-42, including the percentage of pixels.

[0054] Example 44. The processing electronic device compares the amount of pixels of virtual image content with one or more thresholds, and in response to determining that the amount of pixels of virtual image content exceeds one or more thresholds, displays and provides feedback to the user indicating that the display and the eyes are not properly aligned for output, the display system according to any of Examples 40-43.

[0055] Example 45. A display system configured to project light onto the user's eyes and display virtual image content, A frame configured to be supported on the user's head, A head-mounted display disposed on a frame, which projects light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time, and is configured to display the virtual image content, a display; One or more eye-tracking cameras configured to image the user's eyes; A processing electronic device that communicates with the display and the one or more eye-tracking cameras, defines an alignment volume for the display based on one or more parameters, determines the position of the eyes based on an eye image acquired using the one or more eye-tracking cameras, determines whether the position of the eyes is within the alignment volume of the head-mounted display system, and controls the operation of the display based on determining whether the position of the eyes is within the display alignment volume. A processing electronic device configured as such; A display system comprising the above.

[0056] Example 46. The display system according to Example 45, wherein the one or more parameters include the type of application running on the display system.

[0057] Example 47. The display system according to any one of Examples 45-46, wherein the one or more parameters include one or more physical parameters of the head-mounted display.

[0058] Example 48. The display system according to any one of Examples 45-47, wherein the one or more physical parameters of the head-mounted display include one or more of the display field of view, display surface size, display shape, outer housing of the display, and amount of refractive power imparted to the light representing the virtual image content by the display.

[0059] Example 49. The processing electronic device is configured to control the operation of the display by presenting virtual image content that indicates at least that the display and the eyes are not properly aligned to the user, according to any of Examples 45-48 of the display system described.

[0060] Example 50. The processing electronic device is at least determining whether the position of the eye exceeds a second threshold distance outside the subspace of the viewing volume of the outer housing of the head-mounted display, responding to the determination that the position of the eye exceeds the second threshold distance outside the subspace of the viewing volume of the outer housing of the head-mounted display by providing feedback to the user indicating that the display and the eyes are not properly aligned for output, and is configured to determine whether the position of the eye exceeds a first threshold distance outside the viewing volume, according to the display system described in Example 15.

[0061] Example 52. The processing electronic device further identifies an application running on the display system, and determines a first threshold distance based on the identified application, and is configured as described in Example 15 of the display system. This specification also provides, for example, the following items. (Item 1) A display system configured to project light onto a user's eyes and display virtual image content, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time and display virtual image content; One or more eye-tracking cameras configured to image the eyes of the user A processing electronic device that communicates with the display and the one or more eye-tracking cameras, wherein the processing electronic device Determines the position of the eye based on an image of the eye obtained using the one or more eye-tracking cameras Determines whether the position of the eye is within a display alignment volume of the head-mounted display system Providing a notification based on determining whether the position of the eye is within the display alignment volume, the notification indicating at least that the display and the eye are not properly aligned A processing electronic device configured to perform the above A display system comprising the above (Item 2) The processing electronic device is further configured to provide feedback to the user that the head-mounted display is not properly adjusted to fit the user in response to determining that the position of the eye is outside the display alignment volume, the feedback being a notification provided based on determining whether the position of the eye is within the display alignment volume. The display system according to Item 1 (Item 3) The display system according to Item 1, further comprising at least one replaceable fitting component removably mounted on the frame and configured to adjust the fit of the frame (Item 4) The replaceable fitting component according to Item 3, comprising a replaceable nose bridge portion configured to adjust the fit of the frame between the frame and the user's nose bridge (Item 5) The replaceable fitting component according to Item 3, comprising a replaceable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead (Item 6) The display system according to item 3, wherein the replaceable fitting part includes a replaceable back pad configured to adjust the fitting feeling of the frame between the frame and the back of the user's head. (Item 7) The display system according to item 3, wherein the processing electronic device is configured to provide the notification including providing feedback to the user that the head-mounted display is not properly adjusted to fit the user, and further including providing a proposal to the user to replace the currently installed replaceable fitting part with another replaceable fitting part. (Item 8) The display system according to item 1, further comprising one or more light sources disposed on the frame for illuminating the user's eyes, wherein the one or more eye-tracking cameras use light from the one or more light sources to form an image of the eyes. (Item 9) The display system according to item 8, wherein the one or more light sources include at least two light sources disposed on the frame for illuminating the user's eyes. (Item 10) The display system according to item 8, wherein the one or more light sources include an infrared light emitter. (Item 11) The display system according to item 8, wherein the one or more light sources form one or more flashes on the eyes, and the processing electronic device is configured to determine the location of the cornea based on the one or more flashes. (Item 12) The display system according to item 8, wherein the position of the eyes is the location of the center of rotation of the eyes. (Item 13) The display system according to item 1, wherein the cornea has a corneal sphere with a center of curvature associated therewith, and the processing electronic device is configured to determine the location of the center of curvature of the corneal sphere. (Item 14) The processing electronic device is configured to provide the notification by providing an instruction to the display to increase the brightness of a plurality of pixels of the display relative to other pixels of the display, and the plurality of pixels with increased brightness include pixels that are expected to receive a perception of darkening under improper alignment, the display system according to item 1. (Item 15) A display system configured to project light into a user's eyes and display virtual image content, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes with different amounts of wavefront divergence for presenting virtual image content that appears to be located at different depths for different periods of time and to display virtual image content; one or more eye-tracking cameras configured to image the user's eyes; a processing electronic device in communication with the display and the one or more eye-tracking cameras, the processing electronic device being configured to: determine the position of the eye based on an image of the eye obtained using the one or more eye-tracking cameras; determine whether the position of the eye exceeds a first threshold distance outside the viewing volume of the head-mounted display system; in response to determining that the position of the eye exceeds the first threshold distance outside the viewing volume of the head-mounted display system, provide feedback to the user indicating that the display and the eye are not properly aligned for output; and a processing electronic device configured to perform the above; A display system comprising the above. (Item 16) The processing electronic device is at least configured to: determine whether the position of the eye is less than a second threshold distance from the eyepiece; In response to determining that the position of the eye is less than the second threshold distance from the head-mounted display system, providing feedback to the user indicating that the display and the eye are not properly aligned for output The display system according to item 15, which is configured to determine whether the position of the eye exceeds a first threshold distance outside the viewing volume. (Item 17) The processing electronic device at least determining whether the position of the eye exceeds a second threshold distance from the eyepiece; In response to determining that the position of the eye exceeds the second threshold distance from the head-mounted display system, providing feedback to the user indicating that the display and the eye are not properly aligned for output The display system according to item 15, which is configured to determine whether the position of the eye exceeds a first threshold distance outside the viewing volume. (Item 18) The processing electronic device at least determining whether the position of the eye exceeds a second threshold distance outside a sub-space of the field of view of the eye tracking camera; In response to determining that the position of the eye exceeds the second threshold distance outside a sub-space of the viewing volume of the eye tracking camera, providing feedback to the user indicating that the display and the eye are not properly aligned for output The display system according to item 15, which is configured to determine whether the position of the eye exceeds a first threshold distance outside the viewing volume. (Item 19) The viewing volume of the head-mounted display is the volume through which light representing all the pixels of the virtual image content presented by the head-mounted display is expected to pass, the display system according to item 15. (Item 20) The processing electronic device at least determining whether the position of the eye exceeds a second threshold distance outside a sub-space of a viewing volume of an outer housing of the head-mounted display; in response to determining that the position of the eye exceeds the second threshold distance outside a sub-space of a viewing volume of an outer housing of the head-mounted display, providing feedback to the user indicating that the display and the eye are not properly aligned for output; The display system according to item 15, wherein the display system is configured to determine whether the position of the eye exceeds the first threshold distance outside the viewing volume. (Item 21) The processing electronic device further identifying an application running on the display system; determining the first threshold distance based on the identified application; The display system according to item 15, wherein the display system is configured to perform the above operations. (Item 22) A display system configured to project light onto a user's eyes and display virtual image content, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display projecting light into the user's eyes with different amounts of wavefront divergence to present virtual image content that appears to be located at different depths for different periods of time, and configured to display virtual image content; one or more eye tracking cameras configured to image the user's eyes; a processing electronic device communicating with the display and the one or more eye tracking cameras, the processing electronic device determining whether light projected by the head-mounted display is properly aligned with the user's eyes; If the head-mounted display is not properly adjusted to fit the user for aligning the light projected by the display system, providing feedback to the user and configured to perform, a processing electronic device and A display system comprising. (Item 23) The display system according to item 22, further comprising at least one replaceable fitting part removably mounted on the frame and configured to adjust the fit feeling of the frame. (Item 24) The display system according to item 23, wherein the replaceable fitting part includes a replaceable nose bridge part configured to adjust the fit feeling of the frame between the frame and the user's nose bridge. (Item 25) The display system according to item 23, wherein the replaceable fitting part includes a replaceable forehead pad configured to adjust the fit feeling of the frame between the frame and the user's forehead. (Item 26) The display system according to item 23, wherein the replaceable fitting part includes a replaceable back pad configured to adjust the fit feeling of the frame between the frame and the back of the user's head. (Item 27) The processing electronic device is further configured to provide a proposal to the user to replace the currently installed replaceable fitting part with another replaceable fitting part, including providing feedback to the user that the head-mounted display is not properly adjusted to fit the user. The display system according to item 23. (Item 28) A method for evaluating the alignment of virtual image content from a head-mounted display system by a user's eye, the method comprising: Determining a first position of the eye; Determining whether the first position of the eye is within a display alignment volume of the head-mounted display system, the display alignment volume being an imaginary volume associated with an appropriate fit of the head-mounted display system to the user's eyes, Providing a notification based on determining whether the position of the eye is within the display alignment volume, the notification indicating at least that the display and the eye are not properly aligned, A method comprising: (Item 29) The method according to item 28, wherein the head-mounted display system comprises an eye-tracking camera, and determining the first position of the eye comprises imaging the user's eye using the eye-tracking camera. (Item 30) The method according to item 29, wherein the first position of the eye is the position of the center of rotation of the eye, and further comprises calculating the center of rotation of the eye based on imaging of the eye by the eye-tracking camera. (Item 31) The method according to item 28, wherein the head-mounted display system is configured to project light into the eye and display virtual image content within the user's field of view, and further comprises displaying an indication that the wearable system is properly fitted. (Item 32) The method according to item 28, further comprising automatically tracking over time the center of rotation of the eye using the head-mounted display system and notifying the user when the center of rotation of the eye moves outside the alignment display volume. (Item 33) Determining a second position of the eye; Determining that the second position of the eye is within the display alignment volume; In response to determining that the second position of the eye is within the display alignment volume, the wearable system provides additional feedback to the user indicating that the wearable system is properly fitted to the user The method according to item 28, further comprising. (Item 34) The method according to item 28, wherein when the user's eye is not within the display alignment volume, at least some pixels of the head-mounted display system are darkened or made invisible to the user. (Item 35) When the position of the eye is outside the display alignment volume, further comprising changing the field of view of the head-mounted display system, The head-mounted display system comprises at least one display having a first field of view when the position of the eye is within the display alignment volume, the display having a second field of view when the position of the eye is outside the display alignment volume, the second field of view being smaller than the first field of view, the method according to item 28. (Item 36) The method according to item 35, wherein providing the notification includes providing feedback to the user within the second field of view. (Item 37) The wearable system comprises at least one replaceable fitting part, Further comprising providing a notification to the user indicating that the wearable system is not properly fitted to the user, The method according to item 28, wherein the notification includes a proposal or instruction to the user to replace the currently installed replaceable fitting part with an alternative replaceable fitting part. (Item 38) The method according to item 37, wherein the replaceable fitting part comprises at least one fitting part selected from the group consisting of a nasal bridge pad, a forehead pad, and a back pad that fits between the wearable system and the back of the user's head. (Item 39) The wearable system includes at least one replaceable nose bridge pad and further includes determining that the display of the head-mounted system is too low relative to the eye, and providing the notification to the user includes prompting the user to install a larger nose bridge pad, the method according to item 38. (Item 40) Identifying a plurality of pixels of the display of the head-mounted display system where, as a result of the first position of the eye being outside the display alignment volume, the user is expected to perceive being darkened, and increasing the brightness of the plurality of pixels of the display relative to other pixels within the display to reduce the expected darkening, further comprising the method according to item 28.

[0062] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will be apparent from the description, the drawings, and the claims. Neither this summary nor any of the following forms for carrying out the invention purports to define or limit the scope of the subject matter of the invention.

Brief Description of the Drawings

[0063]

Figure 1

[0064]

Figure 2

[0065]

Figure 3

[0066]

Figure 4

[0067]

Figure 5

[0068]

Figure 6

[0069]

Figure 7A

[0070]

Figure 7B

[0071]

Figure 7C

[0072]

Figure 8A

[0073]

Figure 8B

[0074]

Figure 8C

Figure 8D

Figure 8E

[0075]

Figure 9A

Figure 9B

Figure 9C

[0076]

Figure 9D

Figure 9E

Figure 9F

Figure 9G

[0077]

Figure 10

[0078]

Figure 11

[0079]

Figure 12A

Figure 12B

[0080]

Figure 13A

Figure 13B

[0081]

Figure 13C

Figure 13D

[0082]

Figure 14

[0083]

Figure 15A

Figure 15B

[0084]

Figure 16

[0085]

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 17E

Figure 17F

Figure 17G

Figure 17H

[0086]

Figure 18

[0087]

Figure 19A

[0088]

Figure 19B

[0089]

Figure 19C

[0090] Throughout the drawings, reference numerals may be reused to indicate correspondences between referenced elements. The drawings are provided to illustrate the exemplary embodiments described in this specification and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0091] The display portion of the display system may include a head-mounted display (HMD) that can display three-dimensional (3D) virtual objects such that the objects appear to be located within the user's surrounding environment. As a result, the 3D virtual objects can be perceived by the user in a manner similar to real-world objects.

[0092] The HMD can display an image by outputting spatially modulated light to the user, and the light corresponds to the virtual object. The spatially modulated light can contain image information, which can be referred to as image light. For the user to perceive it, the image light travels from the HMD to the user's eyes, propagates through the pupils, and impinges on the retina of the eyes. It should be understood that if all or part of the image light for the image does not enter the pupils of the eyes and / or does not impinge on the retina of the eyes, the viewer will not be able to see the image, or the quality of the image can be degraded. As used herein, alignment relates to the relative positioning of the display and the user's eyes. For example, the display can be said to be properly aligned when the user's eyes and the display are positioned relative to each other such that a desired amount of image light enters the eyes. An alignment observer (e.g., a computer program) within the display device may be programmed to monitor whether the display is properly aligned or whether the eyes are positioned to receive the image light from the display.

[0093] For example, in order to appropriately display content to a user by positioning the user's eyes to receive image light, the user's eyes may need to be located within a specific region or volume of space relative to the HMD. This volume may be referred to as the display alignment volume. If the user's eyes are outside the display alignment volume, the display quality may be degraded (e.g., there may be dimming and / or display content that does not reach the user's eyes). Various factors may be combined to determine the position of the user's eyes relative to the HMD, and thus whether the user's eyes are located within the desired display alignment volume. As an example, anatomical variations between users may mean that a head-mounted display fits some users such that their eyes are placed outside the display alignment volume. As another example, the HMD may not be tightly attached to the user's head and may shift over time on the user's head, particularly when the user moves around. In a specific example, the HMD may slip off the user's nose or may be tilted relative to the line between the user's eyes (interpupillary axis), such that the HMD may be unable to provide the desired virtual content due to misalignment of the display relative to the user's eyes (e.g., without some undesirable degradation).

[0094] The various systems and techniques described herein are at least in part directed to solving problems related to the proper alignment of a display to enable a viewer to view image content as desired. In some embodiments, a head-mounted display system may be configured to determine the position of a user's eyes. The display system may then determine whether the position of those eyes is within a display alignment volume of the head-mounted display system. The step of determining the position of the eyes may include the step of determining the position of a representative pointer volume associated with the eyes, such as the center of rotation of the eyes. The step of determining whether the position of the eyes is within the display alignment volume may include the step of determining whether the center of rotation of the eyes is within the display alignment volume. As discussed herein, the center of rotation of the eyes may be determined using an inward-facing imaging system configured to image the eyes. Additionally, in some embodiments, the display alignment volume is an imaginary volume associated with a proper fit of the head-mounted display system to the user's eyes. For example, the display alignment volume may be a volume defined by a projection from the surface of the head-mounted display system that outputs image light. More specifically, the display alignment volume may be a three-dimensional geometric shape that tapers from a base to a vertex. The shape of the base of the display alignment volume may be at least partially defined by the geometry of the display, and the depth of the display alignment volume (i.e., the distance from the base to the vertex along the z-axis) may be at least partially defined by the field of view (FOV) of the display. For example, a rounded or circular display (e.g., the shape of the area on the surface from which image light is output to the viewer) may result in a conical display alignment volume, and a polygonal display may result in a pyramidal display alignment volume. As an additional example, a display with a larger FOV may result in a display alignment volume having a smaller depth than a display with a smaller FOV.In some embodiments, the display alignment volume may have a general shape of a frustum of a cone or a pyramid. For example, the display alignment volume may have a general shape of a frustum of a pyramid, such as a frustum of a rectangular pyramid.

[0095] In some embodiments, the imaging system facing inward of the head-mounted display system may obtain an image of the user's face, including the eyes. The imaging system facing inward may be an eye-tracking system that can be mounted on the frame of the head-mounted display. The head-mounted display system may analyze the image and determine the relative position of the user's eyes and the HMD, and whether the respective positions of the user's eyes are within the display alignment volume with respect to the eyes. Based on this information, the head-mounted display system may notify the user to adjust the fit of the HMD. For example, the notification may inform the user that the device is slipping and needs adjustment, or may provide a proposal for adjusting the HMD. In at least some embodiments, the head-mounted display system may take steps to reduce any display degradation caused by the misalignment of the HMD with the user, such as increasing the brightness or light output to the user within an area that would otherwise be darkened by misaligned or moving virtual content. Thus, such embodiments of the HMD can assist the user in properly fitting the HMD and reducing problems caused by an improper fit of the HMD, such as when the HMD slips, moves, or tilts relative to the user's head. It should be understood that in some embodiments, the display system may be configured to notify the user of the misalignment and take steps to reduce the display degradation caused by the misalignment. In some other embodiments, the display system may not provide the notification to the user. Rather, the notification may simply be an instruction or flag within the display system that triggers the display system to take an action to reduce the image degradation caused by the misalignment.

[0096] Advantageously, the alignment analysis may be performed automatically using an image obtained from an inward-facing imaging system and information regarding a display alignment volume stored or accessible by the display system. As a result, the fit of the HMD may be corrected initially, optionally, in response to using the HMD, and during the course of continued use of the HMD, to ensure a high level of image quality in the use of the head-mounted display system.

[0097] Accordingly, various implementations of systems and methods for observing the alignment of a head-mounted display system and taking action in response to the observed alignment are provided herein. (Example of a 3D display of a wearable system)

[0098] Reference is now made to the drawings, where like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale.

[0099] A wearable system (also referred to herein as a head-mounted display system or an augmented reality (AR) system) may be configured to present 2D or 3D virtual images to a user. The images may be still images, video frames, or video in a combination or equivalent. At least a portion of the wearable system may be implemented on a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device may be used synonymously with an AR device (ARD). Further, for the purposes of the present disclosure, the term "AR" is used synonymously with the term "MR".

[0100] Figure 1 depicts an illustration of a mixed reality scenario with a virtual object and a physical object as viewed by a person. In Figure 1, an MR scene 100 is depicted, and to a user of MR technology, a real-world park-like setting 110 is visible, featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of MR technology also "sees" a robotic image 130 standing on the real-world platform 120 and a flying cartoon-like avatar character 140 that appears as an anthropomorphic bumblebee, but these elements do not exist in the real world.

[0101] It may be desirable for a 3D display to generate an accommodation response corresponding to its virtual depth for each point within the display's field of view in order to produce a true sense of depth, more specifically, a simulated sense of surface depth. If the accommodation response for a display point does not correspond to the virtual depth of that point such that it is determined by both binocular depth cues of convergence and stereopsis, the human eye experiences an accommodation conflict, resulting in unstable imaging, harmful eye strain, headaches, and in the absence of accommodation information, a nearly complete lack of surface depth.

[0102] VR, AR, and MR experiences may be provided by a display system having a display that provides an image viewer with images corresponding to a plurality of depth planes. The images may vary for each depth plane (e.g., providing a somewhat different presentation of a scene or object), are separately focused by the viewer's eyes, and thereby serve to provide depth cues to the user based on the eye accommodation required to focus on different image features of scenes located on different depth planes or based on observing different image features on different depth planes that are out of focus. As discussed anywhere in this specification, such depth cues provide a reliable perception of depth.

[0103] Figure 2 illustrates an embodiment of a wearable system 200, which may be configured to provide an AR / VR / MR scenario. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functions of the display 220. The display 220 may be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. Since the display 220 may be configured to be worn on the head of the user 210, it may also be referred to as a head-mounted display (HMD), and the wearable system 200 including the display 220 may also be referred to as a head-mounted display system.

[0104] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's external auditory canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other external auditory canal to provide stereo / formable acoustic control). The display 220 may include an audio sensor (e.g., a microphone) 232 to detect an audio stream from the environment and capture ambient sound. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. Stereo sound reception may be used to determine the location of the sound source. The wearable system 200 may perform voice or speech recognition on the audio stream.

[0105] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world within the environment around the user. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system can track either the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may communicate electrically with a processing module 260 or 270 that processes the image information obtained by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user 210's eyes, eye movement, or eye pose. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. The images obtained by the cameras may be used to determine the pupil size or eye pose separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye.

[0106] As an example, the wearable system 200 may use the outward-facing imaging system 464 or the inward-facing imaging system 462 to obtain an image of the user's pose. The image may be a still image, a video frame, or a video.

[0107] The display 220 may be operably coupled to the local data processing module 260 (250) and may be mounted in various configurations, such as fixed to the frame 230 by a wired conductor or wireless connection, fixed to a helmet or hat worn by the user, incorporated within headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt attachment configuration).

[0108] The local processing and data module 260 may include a digital memory such as a hardware processor and a non-volatile memory (e.g., flash memory), both of which can be used to assist in data processing, caching, and storage. The data may be a) data captured from sensors such as an image capture device (e.g., a camera within an inward-facing imaging system and / or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope (e.g., operatively coupled to the frame 230 or otherwise attachable to the user 210), or b) data obtained or processed using the remote processing module 270 or the remote data repository 280, optionally for passage to the display 220 after processing or reading. The local processing and data module 260 may be operatively coupled to the remote processing module 270 or the remote data repository 280 via a communication link 262 or 264, such as a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.

[0109] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, enabling fully autonomous use from the remote modules. Exemplary components of a wearable system

[0110] FIG. 3 schematically illustrates exemplary components of a wearable system. FIG. 3 shows a wearable system 200, which can include a display 220 and a frame 230. The exploded view 202 schematically illustrates various components of the wearable system 200. In one implementation, one or more of the components illustrated in FIG. 3 may be part of the display 220. The various components may collect various data (e.g., auditory or visual data, etc.) associated with the user or the user's environment of the wearable system 200, either alone or in combination. It should be understood that other embodiments may have additional or fewer components depending on the application for which the wearable system is used. Note that FIG. 3 provides an overview of some of the various components and the basic concept of the types of data that can be collected, analyzed, and stored through the wearable system.

[0111] Figure 3 shows an exemplary wearable system 200, which may include a display 220. The display 220 may include a display lens 226 corresponding to a frame 230 and may be mounted on the user's head or housing or frame 230. The display lens 226 may include one or more transparent mirrors positioned in front of the user's eyes 302, 304 by the housing 230, which bounce the projected light 338 into the eyes 302, 304 and are configured to allow transmission of at least some light from the local environment while promoting beam shaping. The wavefront of the projected light beam 338 may be bent or focused to match the desired focal length of the projected light. As shown, two wide field of view machine vision cameras 316 (also referred to as world cameras) may be coupled to the housing 230 to image the environment around the user. These cameras 316 may be dual capture visible light / non-visible (e.g., infrared) light cameras. The cameras 316 may be part of an outward-facing imaging system 464 shown in FIG. 4. Images obtained by the world cameras 316 may be processed by a pose processor 336. For example, the pose processor 336 may implement one or more object recognition devices 708 (e.g., shown in FIG. 7) to identify the pose of the user or another person within the user's environment or to identify physical objects within the user's environment.

[0112] Continuing to refer to FIG. 3, a pair of light projector modules (e.g., a scanning laser shaped wavefront (e.g., for depth) light projector module) with a display mirror and an optical system configured to project light 338 into eyes 302, 304 is shown. The depicted figure also shows two small infrared cameras 324 paired with an infrared light source 326 (such as a light emitting diode "LED", etc.) configured to track the user's eyes 302, 304 and support rendering and user input. The cameras 324 may be part of an inward-facing imaging system 462 shown in FIG. 4. The wearable system 200 may further feature a sensor assembly 339, which has X, Y, and Z axis accelerometer capabilities and a magnetic compass and X, Y, and Z axis gyroscope capabilities and may preferably provide data at a relatively high frequency such as 200 Hz. The sensor assembly 339 may be part of an IMU described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336 such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), or ARM processor (advanced reduced instruction set machine), which may be configured to calculate from the wide field of view image information output from the capture device 316 the real-time or near real-time user head pose. The head pose processor 336 may be a hardware processor and may be implemented as part of the local processing and data module 260 shown in FIG. 2A.

[0113] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure the distance between objects in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., LIDAR), an ultrasonic depth sensor, or a depth sensing camera. In some implementations where the camera 316 has depth sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0114] Also shown is a processor 332 configured to perform digital or analog processing and derive from gyroscope, compass, or accelerometer data from sensor assembly 339. The processor 332 may be part of the local processing and data module 260 shown in FIG. 2. The wearable system 200 may also include a positioning system such as, for example, GPS 337 (Global Positioning System) as shown in FIG. 3 to assist with attitude and positioning analysis. Additionally, the GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0115] The wearable system may combine data obtained by the GPS 337 and a remote computing system (e.g., remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As one example, the wearable system may determine the user's location based on GPS data and read out a world map that includes virtual objects associated with the user's location (e.g., by communicating with the remote processing module 270). As another example, the wearable system 200 may use a world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4) to monitor the environment. Based on the images obtained by the world camera 316, the wearable system 200 may detect objects within the environment (e.g., by using one or more object recognition devices 708 shown in FIG. 7). The wearable system may further interpret characters using data obtained by the GPS 337.

[0116] The wearable system 200 may also include a rendering engine 334, which may be configured to provide rendering information local to the user for the view of a user worldwide and to facilitate the operation of the scanner and the imaging into the user's eyes. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled to other components of the wearable system 200 (e.g., via a wired or wireless link). For example, the rendering engine 334 may be coupled to the eye camera 324 via a communication link 274 and to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via a communication link 272. The rendering engine 334 may also communicate with other processing units, such as the sensor attitude processor 332 and the image attitude processor 336, via links 276 and 294, respectively.

[0117] The camera 324 (e.g., a small infrared camera) may be used to track eye pose and support rendering and user input. Some exemplary eye poses may include where the user is looking or the depth at which the user is focused (which may be estimated using the vergence of the eyes). The GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide gross or high-speed pose estimation. One or more of the cameras 316 may obtain images and poses, which may be used to map the local environment and share the user view with others in conjunction with data from associated cloud computing resources.

[0118] The exemplary components depicted in FIG. 3 are for illustrative purposes only. A plurality of sensors and other functional modules are shown together for ease of illustration and explanation. Some embodiments may include only one or a subset of these sensors or modules. Further, the locations of these components are not limited to the positions depicted in FIG. 3. Some components may be mounted or stored within other components, such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be positioned within a belt pack and configured to communicate with other components of the wearable system via wireless communication such as ultra-wideband, Wi-Fi, Bluetooth®, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0119] Regarding the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may generally be utilized to measure the location where the center of the user's eye geometrically converges and diverges, which generally coincides with the position of the focus of the eye or the "depth of focus". The three-dimensional surface of all points where the eye converges and diverges may be referred to as the "cyclopean locus". The focal distance may take a finite number of depths or may vary infinitely. Light projected from the convergence / divergence movement distance appears to be focused on the target eyes 302, 304, while light in front of or behind the convergence / divergence movement distance is blurred. Examples of the wearable system and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0120] The human visual system is complex and it is difficult to provide a realistic perception of depth. An object viewer can perceive an object in three dimensions due to the combination of convergence / divergence movement and accommodation. The convergence / divergence movement of two eyes relative to each other (for example, the rotation of the pupils so that they move towards each other or away from each other, converging the lines of sight of the eyes to fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, a change in the focus of the eye's lens or the eye's accommodation to change the focus from one object to another object at a different distance will automatically cause a coordinated change in convergence / divergence at the same distance under a relationship known as the "accommodation-convergence / divergence reflex". Similarly, a change in convergence / divergence will, under normal conditions, induce a coordinated change in accommodation. A display system that provides better coordination between accommodation and convergence / divergence can form a more realistic and comfortable simulation of a three-dimensional image.

[0121] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Thus, to create an illusion of appropriate depth of focus, the eye's convergence / divergence movement may be tracked using camera 324, and rendering engine 334 and projection subsystem 318 may be utilized to focus and render all objects on or near the single viewing trajectory and render all other objects with a variable degree of defocus (for example, using intentionally created blur). Preferably, system 220 renders to the user at a frame rate of about 60 frames per second or more. As described above, preferably, camera 324 may be utilized for eye tracking and the software may be configured to take into account not only the convergence / divergence geometry but also a focus location queue for use as user input. Preferably, such a display system is configured with brightness and contrast suitable for daytime or nighttime use.

[0122] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute for visual object alignment, which, although not limited by theory, is considered to be approximately the limit of the human eye. The display system 220 may be integrated with a positioning system, which may involve a GPS element, optical tracking, a compass, an accelerometer, or other data sources and may assist in position and orientation determination. The positioning information may be utilized to facilitate accurate rendering within the view of a user of the relevant world (for example, such information would facilitate glasses in ascertaining their location with respect to the real world).

[0123] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the focusing adjustment of the user's eyes. Different from the conventional 3D display approach that forces the user to focus on the location where the image is projected, in some embodiments, the wearable system automatically varies the focus of the projected virtual content and is configured to enable more comfortable viewing of the one or more images presented to the user. For example, if the user's eyes have a current focus of 1 m, the image may be projected to coincide with the user's focus. If the user shifts the focus to 3 m, the image is projected to coincide with the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments enables the user's eyes to function in a more natural manner.

[0124] Such a wearable system 200 can eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project a virtual image at a variable focal distance through one or more variable focus elements (VFE). In one or more embodiments, 3D perception may be achieved through a multi-plane focus system that projects the image onto a fixed focal plane from the user. Other embodiments employ variable plane focus, where the focal plane is reciprocally moved in the z-direction to match the current state of the user's focus.

[0125] In both the multi-plane focus system and the variable plane focus system, the wearable system 200 may employ eye tracking to determine the convergence / divergence movement of the user's eyes, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 comprises a light modulator that projects a variable focus light beam in a raster pattern across the retina through a fiber scanner or other light generating source. Thus, the display capabilities of the wearable system 200 that project an image at a variable focal distance not only facilitate depth adjustment for the user to visually recognize an object in 3D, but may also be used to compensate for the user's eye abnormalities, as further described in U.S. Patent Publication No. 2016 / 0270656, which is hereby incorporated by reference in its entirety. In some other embodiments, the spatial light modulator may project the image to the user through various optical components. For example, as further described below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user. Waveguide stack assembly

[0126] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4A schematically shows some portions of that wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated within display 220 of FIG. 2.

[0127] Continuing to refer to FIG. 4, waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, features 458, 456, 454, 452 may be lenses. In other embodiments, features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., a cladding layer or structure for forming an air gap).

[0128] Waveguides 432b, 434b, 436b, 438b, 440b or a plurality of lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may each be utilized to input image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, 428 and is input into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams directed toward the eye 410 at a particular angle (and amount of divergence) corresponding to the depth plane associated with a particular waveguide.

[0129] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that each generate image information for input into their respective corresponding waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, image input devices 420, 422, 424, 426, 428 are the output ends of a single multiplexed display that can send image information, for example, via one or more optical conduits (such as optical fiber cables), to each of image input devices 420, 422, 424, 426, 428.

[0130] Controller 460 controls the operation of the stacked waveguide assemblies 480 and the image input devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that adjusts the timing and provides the image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 460 may, in some embodiments, be part of processing module 260 or 270 (illustrated in FIG. 2).

[0131] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved), with a major top surface and a major bottom surface and an edge extending between their major top and major bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as external coupled light, and the light extraction optical elements may also be referred to as external coupling optical elements. The beam of the extracted light is output by the waveguide at the location where the light propagating within the waveguide impinges on the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. For ease of explanation and clarity of the drawings, they are shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top major surface or the bottom major surface, or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of material forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic material components, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the surface of the material component.

[0132] Continuing to refer to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 432b. The collimated light may represent an optically infinite focal plane. The next upper waveguide 434b may be configured to output collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane that is closer inwardly toward the eye 410 from optically infinite. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 436b as originating from a second focal plane that is closer inwardly toward the person from optically infinite, which was the light from the next upper waveguide 434b.

[0133] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, and using the highest waveguide 440b in the stack, its output is sent through all of the lenses between it and the eye for the aggregated focusing power representing the focal plane closest to the person. When viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensation lens layer 430 may be disposed at the top of the stack to compensate for the stack of lenses 458, 456, 454, 452. (The compensation lens layer 430 and the stacked waveguide assembly 480 may be configured such that light originating from the world 470 is transmitted to the eye 410 with substantially the same level of divergence (or collimation) as the light had when it was first received by the stacked waveguide assembly 480.) Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical element of the waveguide and the focusing side of the lens may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0134] Continuing to refer to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have light extraction optical elements of different configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be three-dimensional features or surface features configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published Jun. 25, 2015, which is incorporated herein by reference in its entirety.

[0135] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 410 at each intersection of the DOE while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of output emission towards the eye 304 with respect to this particular collimated beam that bounces within the waveguide.

[0136] In some embodiments, one or more DOEs may be switchable between an “on” state in which they actively diffract and an “off” state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal, in which the microdroplets have a diffraction pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0137] In some embodiments, the number and distribution of depth planes or depth of field may be varied dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer. As a result, as the pupil size of the viewer's eye decreases, one plane that is indistinguishable because its location in the plane is beyond the depth of focus of the eye becomes distinguishable, and increases such that it may appear more in focus with the reduction in pupil size and the corresponding increase in depth of field. Similarly, the number of separated depth planes used to present different images to the viewer may be decreased with a decreased pupil size. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focus from one depth plane to the other. However, these two depth planes may be sufficient for the user to focus on at another pupil size without changing the focus adjustment at the same time.

[0138] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is indistinguishable between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0139] In some embodiments, it may be desirable to satisfy the condition that the outgoing beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition can be difficult in light of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the outgoing beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the outgoing beam may also decrease. In some embodiments, the outgoing beam size may be varied using a variable aperture.

[0140] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210 and encompasses a portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the FOV of the world camera may be larger than the field of view of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other contexts, the movement of the wearer may be more restricted, and accordingly, the wearer's FOR may touch a smaller solid angle. Images obtained from the outward-facing imaging system 464 may be used to track gestures made by the user (e.g., hand or finger gestures) and detect objects within the world 470 in front of the user, etc.

[0141] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors may be positioned to provide stereo sound reception useful for determining the location of the source of speech. As another example, the audio sensor 232 may comprise a directional microphone, which may also provide such useful directional information regarding the location where the audio source is located. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 when locating the source of speech or determining the active speaker at a particular instant. For example, the wearable system 400 may use speech recognition, alone or in combination with a reflected image of the speaker (as seen in a mirror, for example), to determine the identity of the speaker. As another example, the wearable system 400 may be able to determine the location of a speaker within the environment based on the sound obtained from a directional microphone. The wearable system 400 may use a speech recognition algorithm to analyze the sound resulting from the location of the speaker, determine the content of the speech, and use speech recognition techniques to determine the identity of the speaker (e.g., name or other demographic information).

[0142] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements such as eye movement and face movement. The inward-facing imaging system 466 may capture an image of the eye 410 and may be used to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 may be used to determine the direction the user is looking (e.g., eye pose), or to obtain an image for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be used to independently determine the pupil size or eye pose of each eye separately, thereby enabling the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. The image obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye pose or mood, which may be used by the wearable system 400 to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head pose (e.g., head position or head orientation).

[0143] The wearable system 400 may include a user input device 466 through which a user can input commands to the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitance sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in translational (e.g., left / right, forward / backward, or up / down) or rotational (e.g., yaw, pitch, or roll) movements that are possible for some or all of the controller. A multi-DOF controller that supports translational movement may be referred to as 3DOF, while a multi-DOF controller that supports both translational and rotational movement may be referred to as 6DOF. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 either wired or wirelessly. Other components of the wearable system

[0144] In many implementations, a wearable system may include, in addition to or instead of the components of the wearable system described above, other components. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a haptic sensation of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The haptic sensation may reproduce the sensation of a physical object represented by the virtual object, or may reproduce the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the haptic device or component may be held by the user.

[0145] A wearable system may include, for example, one or more physical objects that are operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In certain implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and a trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and a virtual trackpad so as to appear to be on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard or virtual trackpad. The user input device 466 (shown in FIG. 4) may be an embodiment of a totem that may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system or other users.

[0146] Examples of haptic devices and totems that can be used with the wearable devices, HMDS, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety. Example of an eye image

[0147] FIG. 5 illustrates an image of an eye 500 with an eyelid 504, a sclera 508 (“white of the eye”), an iris 512, and a pupil 516. Curve 516a indicates the pupil boundary between the pupil 516 and the iris 512, and curve 512a indicates the edge boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is illustrated in a natural rest position (e.g., oriented such that both the user's face and line of sight will be directed towards a distant object directly in front of the user). The natural rest position of the eye 500 is in a natural rest position (e.g., out of the page with respect to the eye 500 shown in FIG. 5), and in the present example, can be indicated by a natural rest direction 520 that is the direction orthogonal to the surface of the eye 500 when centered within the pupil 516.

[0148] As the eye 500 moves to look towards different objects, the eye pose will change with respect to the natural rest direction 520. The current eye pose is a direction orthogonal to the surface of the eye (and centered within the pupil 516), but may be determined with reference to an eye pose direction 524 that is oriented towards the object that the eye is currently directed towards. Referring to the exemplary coordinate system shown in FIG. 5, the pose of the eye 500 can be represented as two angular parameters that both indicate the azimuthal deviation and the zenith deviation of the eye pose direction 524 of the eye with respect to the natural rest direction 520 of the eye. For illustrative purposes, these angular parameters can be represented as θ (azimuthal deviation, determined from a reference azimuth) and φ (zenith deviation, sometimes also referred to as polar deviation). In some implementations, the angular roll of the eye around the eye pose direction 524 may be included in the determination of the eye pose, and the angular roll may be included in the following analysis. In other implementations, other techniques for determining the eye pose may be used, such as, for example, pitch, yaw, and optionally, a roll system.

[0149] The eye image may be obtained from a video using any suitable process, such as a video processing algorithm that can extract the image from one or more sequential frames. The eye pose may be determined from the eye image using various eye tracking techniques. For example, the eye pose may be determined by considering the lens effect of the cornea on the provided light source. Any suitable eye tracking technique may be used to determine the eye pose in the eyelid shape estimation technique described herein. (Example of an eye tracking system)

[0150] FIG. 6 illustrates a schematic diagram of a wearable or head-mounted display system 600 that includes an eye-tracking system. The head-mounted display system 600 may include, in at least some embodiments, components located within a head-mounted unit 602 and components located within a non-head-mounted unit 604. The non-head-mounted unit 604 may be, by way of example, a belt-mounted component, a handheld component, a component within a backpack, a remote component, and the like. Incorporating some of the components of the head-mounted display system 600 within the non-head-mounted unit 604 can help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some implementations, some or all of the functionality described as being implemented by one or more components of the head-mounted unit 602 and / or the non-head-mounted 604 may be provided using one or more components included anywhere within the head-mounted display system 600. For example, some or all of the functionality described below in connection with the CPU 612 of the head-mounted unit 602 may be provided using the CPU 616 of the non-head-mounted unit 604, and vice versa. In some embodiments, some or all of such functionality may be provided using a peripheral device of the head-mounted display system 600. Further, in some implementations, some or all of such functionality may be provided using one or more cloud computing devices or other remotely located computing devices in a manner similar to that described above with reference to FIG. 2.

[0151] As shown in FIG. 6, the head-mounted display system 600 may include an eye-tracking system that includes a camera 324 that captures an image of the user's eye 610. Optionally, the eye-tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs"). The light sources 326a and 326b may generate a flash (i.e., a reflection from the user's eye that appears in the image of the eye captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, such that the position of the flash in the image captured by the camera 324 may be used in tracking the user's eye (as will be discussed in more detail below in connection with FIGS. 7-11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In yet another embodiment, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources such as light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610.

[0152] The eye tracking module 614 may receive an image from the eye tracking camera 324, analyze the image, and extract various information. As an example, the eye tracking module 614 may detect the user's eye posture, the three-dimensional position of the user's eyes with respect to the eye tracking camera 324 (and the head-mounted unit 602), the direction of one or both of the user's focused eyes 610, the user's convergence / divergence motion depth (i.e., the depth from the user on which the user is focused), the position of the user's pupils, the position of the user's corneas and corneal spheres, the respective centers of rotation of the user's eyes, and the respective centers of the user's viewpoints. The eye tracking module 614 may extract such information using the techniques described below in connection with FIGS. 7-11. As shown in FIG. 6, the eye tracking module 614 may be a software module implemented using the CPU 612 within the head-mounted unit 602.

[0153] Data from the eye tracking module 614 may be provided to other components within the wearable system. As an example, such data may be transmitted to components within a non-head-mounted unit 604, such as the CPU 616, including software modules for the light field rendering controller 618 and the alignment observer 620, which may be configured to evaluate whether the display of the head-mounted display system 600 is properly aligned with the user's eyes.

[0154] The rendering controller 618 may adjust the image displayed to the user using the information from the eye tracking module 614 by a rendering engine 622 (which may be a software module within the GPU 621 and may provide images to the display 220, the rendering engine). As an example, the rendering controller 618 may adjust the image displayed to the user based on the center of rotation or the center of the user's viewpoint. In particular, the rendering controller 618 may use information regarding the center of the user's viewpoint to simulate a rendering camera (i.e., simulate the collection of an image from the user's viewpoint), and may adjust the image displayed to the user based on the simulated rendering camera.

[0155] Sometimes referred to as a "pinhole perspective projection camera" (or simply, a "perspective projection camera") or a "virtual pinhole camera" (or simply, a "virtual camera"), a "rendering camera" is a simulated camera potentially used to render virtual image content from a database of objects within a virtual world. The objects may have locations and orientations with respect to a user or wearer and potentially real objects within an environment surrounding the user or wearer. In other words, a rendering camera may represent a viewpoint within a rendering space from which a user or wearer is to view 3D virtual content (e.g., virtual objects) within the rendering space. The rendering camera is managed by a rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eye. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components with respect to the virtual world). The virtual image is captured from the viewpoint of such a camera having the position and orientation of the rendering camera (e.g., the extrinsic parameters of the rendering camera). The system is to be able to define and / or adjust the intrinsic and extrinsic rendering camera parameters. For example, the system may define a specific set of extrinsic rendering camera parameters such that the virtual image is rendered as if it were captured from the viewpoint of a camera having a specific location with respect to the user or wearer's eye so that the image appears as if it were from the viewpoint of the user or wearer. The system may later dynamically adjust the extrinsic rendering camera parameters on-the-fly to maintain alignment with the specific location. Similarly, the intrinsic rendering camera parameters may also be defined and dynamically adjusted over time.In some implementations, the image is rendered as if it were captured from the perspective of a camera having an aperture (e.g., a pinhole) at a specific location (such as the center of the viewpoint or the center of rotation or other locations) with respect to the user's or wearer's eye.

[0156] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and a different rendering camera for the user's right eye as the user's eyes are physically separated from each other and thus consistently located at different places. In at least some implementations, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye may be presented to the user through the left eyepiece on the left side of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the perspective of the rendering camera associated with the user's right eye may be presented to the user through the right eyepiece on the right side of such a head-mounted display. Further details regarding the creation, adjustment, and use of the rendering cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION" (which is hereby expressly incorporated by reference in its entirety for all purposes).

[0157] In some embodiments, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 622, etc.) may determine the position and orientation of a rendering camera within the rendering space based on the position and orientation of the user's head and eyes (e.g., as determined based on head pose and eye tracking data, respectively). That is, system 600 effectively maps the position and orientation of the user's head and eyes to a particular location and angular position within the 3D virtual environment, positions and orients the rendering camera at the particular location and angular position within the 3D virtual environment, and may render virtual content for the user as it can be captured by the rendering camera. Further details discussing the real-world / virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE", which is hereby expressly incorporated by reference in its entirety for all purposes. As an example, rendering controller 618 may adjust the depth at which an image is displayed by selecting the depth plane (or depth planes) to be utilized at any given time for displaying the image. In some implementations, such depth plane switching may be done through adjustment of one or more of the rendering camera's intrinsic parameters.

[0158] The alignment observer 620 may identify whether the head-mounted unit 602 is properly positioned on the user's head using information from the eye-tracking module 614. As an example, the eye-tracking module 614 may provide eye location information such as the position of the center of rotation of the user's eyes indicating the three-dimensional position of the user's eyes relative to the camera 324, and the head-mounted unit 602 and the eye-tracking module 614 may use the location information to determine whether the display 220 is properly aligned within the user's field of view, or whether the head-mounted unit 602 (or headset) has slipped or is otherwise misaligned with the user's eyes. As an example, the alignment observer 620 may determine whether the head-mounted unit 602 has slipped from the user's nasal bridge and thus moved the display 220 away from and downward from the user's eyes (which may not be desirable), whether the head-mounted unit 602 has moved above the user's nasal bridge and thus moved the display 220 closer to and upward from the user's eyes, whether the head-mounted unit 602 has shifted left or right relative to the user's nasal bridge, whether the head-mounted unit 602 has been lifted above the user's nasal bridge, or whether the head-mounted unit 602 has moved away from the desired position or range of positions in these or other ways. Generally, the alignment observer 620 may generally be able to determine whether the head-mounted unit 602, and in particular the display 220, is properly positioned in front of the user's eyes. In other words, the alignment observer 620 may determine whether the left display within the display system 220 is properly aligned with the user's left eye and whether the right display within the display system 220 is properly aligned with the user's right eye. The alignment observer 620 may determine whether the head-mounted unit 602 is properly positioned by determining whether the head-mounted unit 602 is positioned and oriented within the desired range of positions and / or orientations relative to the user's eyes.

[0159] In at least some embodiments, the alignment observer 620 may generate user feedback in the form of an alert, message, or other content. Such feedback may be provided to the user, informing the user of any misalignment of the head-mounted unit 602, along with optional feedback regarding how to correct the misalignment (such as a suggestion to adjust the head-mounted unit 602 in a particular manner).

[0160] Exemplary alignment observation and feedback techniques that may be utilized by the alignment observer 620 are described in U.S. Patent Application No. 15 / 717,747, filed on September 27, 2017 (Attorney Docket No. MLEAP.052A2), which is incorporated herein by reference in its entirety. (Example of an eye-tracking module)

[0161] A detailed block diagram of an exemplary eye-tracking module 614 is shown in FIG. 7A. As shown in FIG. 7A, the eye-tracking module 614 may include various different sub-modules, may provide various different outputs, and may utilize various available data when tracking the user's eyes. By way of example, the eye-tracking module 614 may utilize available data including the geometric arrangement of the light source 326 and the eye-tracking camera 324 with respect to the head-mounted unit 602, an assumed eye dimension 704 such as a typical distance of about 4.7 mm between the center of the user's corneal curvature and the average center of rotation of the user's eye, or a typical distance between the user's center of rotation and the center of the viewing point, and user-specific calibration data 706 such as the interpupillary distance of a particular user. Additional examples of incidental, intrinsic, and other information that may be employed by the eye-tracking module 614 are described in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0162] The image preprocessing module 710 may receive an image from an eye camera such as the eye camera 324, and may perform one or more preprocessing (i.e., adjustment) operations on the received image. As an example, the image preprocessing module 710 may apply Gaussian blur to the image, may downsample the image to a lower resolution, may apply an unsharp mask, may apply an edge sharpening algorithm, or may apply other suitable filters that assist in subsequent detection, localization, and labeling of flashes, pupils, or other features within the image from the eye camera 324. The image preprocessing module 710 may apply a low-pass filter such as an open filter or a morphological filter that can remove noise, such as high-frequency noise from the pupil boundary 516a (see FIG. 5), which may otherwise interfere with pupil and flash determination. The image preprocessing module 710 may output the preprocessed image to the pupil identification module 712 and the flash detection and labeling module 714.

[0163] The pupil recognition module 712 may receive the pre - processed image from the image pre - processing module 710 and may identify the regions of those images that contain the user's pupils. In some embodiments, the pupil recognition module 712 may determine the coordinates of the position of the user's pupil in the eye - tracking image from the camera 324, i.e., the coordinates of the center or centroid. In at least some embodiments, the pupil recognition module 712 may identify the contour (e.g., the contour of the pupil - iris boundary) in the eye - tracking image, identify the contour moment (i.e., the center of mass), apply the starburst pupil detection and / or Canny edge detection algorithm, exclude outliers based on intensity values, identify sub - pixel boundary points, correct for eye camera distortion (i.e., the distortion in the images captured by the eye camera 324), apply the random sample consensus (RANSAC) iterative algorithm, fit an ellipse to the boundary in the eye - tracking image, apply a tracking filter to the image, and identify the sub - pixel image coordinates of the user's pupil centroid. The pupil recognition module 712 may output pupil recognition data, which may indicate the region of the pre - processed image module 712 identified as showing the user's pupil, to the flash detection and labeling module 714. The pupil recognition module 712 may provide the 2D coordinates of the user's pupil (i.e., the 2D coordinates of the user's pupil centroid) in each eye - tracking image to the flash detection module 714. In at least some embodiments, the pupil recognition module 712 may also provide the same type of pupil recognition data to the coordinate system normalization module 718.

[0164] Pupil detection techniques that may be utilized by the pupil recognition module 712 are described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, each of which is incorporated herein by reference in its entirety.

[0165] The flash detection and labeling module 714 may receive the pre-processed image from module 710 and the pupil identification data from module 712. The flash detection module 714 may use this data to detect and / or identify a flash (i.e., the reflection of light from the light source 326 from the user's eye) within the region of the pre-processed image indicating the user's pupil. As an example, the flash detection module 714 may search for bright regions in the eye tracking image that are in the vicinity of the user's pupil and are sometimes also referred to herein as "blobs" or local intensity maxima. In at least some embodiments, the flash detection module 714 may re-scale (e.g., enlarge) the pupil ellipse to include additional flashes. The flash detection module 714 may filter the flashes by size and / or intensity. The flash detection module 714 may also determine the 2D position of each flash in the eye tracking image. In at least some examples, the flash detection module 714 may determine the 2D position of the flash relative to the user's pupil, which may also be referred to as a pupil-flash vector. The flash detection and labeling module 714 may label the flashes and output the pre-processed image with the labeled flashes to the 3D corneal center estimation module 716. The flash detection and labeling module 714 may also communicate data such as the pre-processed image from module 710 and the pupil identification data from module 712.

[0166] Pupil and flash detection, such as that implemented by modules such as modules 712 and 714, may use any suitable technique. As an example, edge detection may be applied to the eye image to identify the flash and the pupil. Edge detection may be applied by various edge detectors, edge detection algorithms, or filters. For example, the Canny edge detector may be applied to the image to detect edges such as lines in the image. The edges may include points located along a line corresponding to a local maximum derivative. For example, the pupil boundary 516a (see FIG. 5) may be located using the Canny edge detector. Once the location of the pupil is determined, various image processing techniques may be used to detect the "pose" of the pupil 116. The determination of the eye pose of the eye image may also be referred to as the detection of the eye pose of the eye image. The pose may also be referred to as the line of sight, the direction being faced, or the orientation of the eye. For example, the pupil may be looking left towards an object, and the pose of the pupil may be classified as a left-facing pose. Other methods may also be used to detect the location of the pupil or the flash. For example, concentric rings may be located within the eye image using the Canny edge detector. As another example, an integral differential operator may be used to find the corneal limbus boundary of the pupil or the iris. For example, the Daugman integral differential operator, the Hough transform, or other iris segmentation techniques may be used to return a curve that estimates the boundary of the pupil or the iris.

[0167] The 3D corneal center estimation module 716 may receive a pre-processed image, including the detected flash data and pupil identification data, from modules 710, 712, 714. The 3D corneal center estimation module 716 may use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 may estimate the 3D position of the center of the corneal curvature of the eye or the center of the spherical surface of the user's cornea, i.e., generally, the center of an imaginary spherical surface having a surface portion coextensive with the user's cornea. The 3D corneal center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal spherical surface and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further details of the operation of the 3D corneal center estimation module 716 are provided herein in connection with FIGS. 8A-8E. Techniques for estimating the position of eye features such as the cornea or corneal spherical surface that may be utilized by the 3D corneal center estimation module 716 and other modules within the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726, filed Apr. 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0168] The coordinate system normalization module 718 may optionally be included within the eye tracking module 614 (as indicated by its dashed outline). The coordinate system normalization module 718 may receive data indicating the estimated 3D coordinates of the center of the user's cornea (and / or the center of the user's corneal sphere) from the 3D corneal center estimation module 716, and may also receive data from other modules. The coordinate system normalization module 718 may normalize the eye camera coordinate system, which may help compensate for slippage of the wearable device (e.g., slippage of a head-mounted component from its normal resting position on the user's head, which may be identified by the alignment observer 620). The coordinate system normalization module 718 may rotate the coordinate system and align the z-axis of the coordinate system (i.e., the convergence / divergence motion depth axis) with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716), and may translate the camera center (i.e., the origin of the coordinate system) to a predetermined distance away from the corneal center, such as 30 mm (i.e., the module 718 may enlarge or shrink the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). By using this normalization process, the eye tracking module 614 may be able to establish consistent orientation and distance within the eye tracking data, relatively independently of variations in the headset positioned on the user's head. The coordinate system normalization module 718 may provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and the pre-processed eye tracking image to the 3D pupil center locator module 720. Further details of the operation of the coordinate system normalization module 718 are provided herein in connection with FIGS. 9A-9C.

[0169] The 3D pupil center locator module 720 may receive data including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil location data, and pre - processed eye - tracking images, within a normalized or non - normalized coordinate system. The 3D pupil center locator module 720 may analyze such data to determine the 3D coordinates of the center of the user's pupil within a normalized or non - normalized eye camera coordinate system. The 3D pupil center locator module 720 may determine the location of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), assumed eye dimensions 704 such as the size of the typical user's corneal sphere and the typical distance from the corneal center to the pupil center, and optical properties of the eye such as the refractive index of the cornea (relative to the refractive index of air), or any combination of these. Further details of the operation of the 3D pupil center locator module 720 are provided herein in connection with FIGS. 9D - 9G. Techniques for estimating the position of eye features such as the pupil, which may be utilized by the 3D pupil center locator module 720 and other modules within the wearable system of the present disclosure, are discussed in U.S. Patent Application No. 15 / 497,726 (Attorney Docket No. MLEAP.023A7), filed on Apr. 26, 2017, which is incorporated herein by reference in its entirety.

[0170] The optical axis determination module 722 may receive data indicating the 3D coordinates of the center of the user's cornea and the center of the user's pupil from modules 716 and 720. Based on such data, the optical axis determination module 722 may identify a vector from the position of the corneal center (i.e., from the center of the corneal sphere) to the center of the user's pupil that may define the optical axis of the user's eye. As an example, the optical axis determination module 722 may provide an output defining the user's optical axis to modules 724, 728, 730, and 732.

[0171] The center of rotation (CoR) estimation module 724 may receive data from module 722 that includes parameters of the user's eye optical axis (i.e., data indicating the direction of the optical axis within a coordinate system with a known relationship to the head-mounted unit 602). The CoR estimation module 724 may estimate the center of rotation of the user's eye (i.e., the point around which the user's eye rotates when the user's eye rotates left, right, up, and / or down). Assume that a single point may be sufficient even if the eye cannot rotate perfectly around a single point. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eye by moving the center of the pupil (identified by module 720) or the center of curvature of the cornea (as identified by module 716) a specific distance along the optical axis (identified by module 722) towards the retina. This specific distance may be the assumed eye dimension 704. As an example, the specific distance between the center of curvature of the cornea and the CoR may be about 4.7 mm. This distance may be varied for a particular user based on any relevant data including the user's age, gender, vision prescription, other relevant characteristics, etc.

[0172] In at least some embodiments, the CoR estimation module 724 may refine over time the estimated value of the respective center of rotation of the user's eyes. As an example, over time, the user may eventually rotate the eye (to look at something else, closer, farther, or sometimes left, right, up, or down), which will cause an offset along the respective optical axis of the eye. The CoR estimation module 724 may then analyze the two (or more) optical axes identified by module 722 and locate the 3D point at the intersection of those optical axes. The CoR estimation module 724 may then determine the center of rotation at the 3D point of that intersection. Such techniques may provide an estimated value of the center of rotation with accuracy that improves over time. Various techniques may be employed to increase the accuracy of the CoR estimation module 724 and the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 may estimate the CoR by finding the average point of the intersections of the optical axes determined over time for various different eye postures. As an additional example, module 724 may filter or average the estimated CoR positions over time, may calculate a moving average of the estimated CoR positions over time, and / or may apply a Kalman filter and the known dynamics of the eye and eye tracking system to estimate the CoR position over time. As a specific example, module 724 may slowly move over time from a hypothesized CoR position (i.e., 4.7 mm behind the center of the corneal curvature of the eye) to a slightly different location within the user's eye as eye tracking data regarding the user is acquired, such that the determined point of the optical axis intersection and the hypothesized CoR position (such as 4.7 mm from the center of the corneal curvature of the eye) are weighted and averaged to enable per-user refinement of the CoR position.

[0173] The interpupillary distance (IPD) estimation module 726 may receive data indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes from the CoR estimation module 724. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils when the user is looking at optical infinity (i.e., the optical axes of the user's eyes are substantially parallel to each other), which is the typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules within the wearable system. As an example, the user's IPD may be provided to the alignment observer 620 and used when assessing the degree to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are appropriately spaced according to the user's IPD). As another example, the user's IPD may be provided to the convergence / divergence motion depth estimation module 728 and used when determining the user's convergence / divergence motion depth. Module 726 may employ various techniques such as those discussed in relation to the CoR estimation module 724 to increase the accuracy of the estimated IPD. As an example, the IPD estimation module 724 may apply filtering, averaging over time, weighted averaging, Kalman filtering, etc., including an assumed IPD distance, as part of the estimation of the user's IPD in an accurate manner.

[0174] The convergence / divergence motion depth estimation module 728 may receive data from various modules and sub-modules within the eye tracking module 614 (as shown in connection with FIG. 7A). In particular, the convergence / divergence motion depth estimation module 728 may adopt data indicating the estimated 3D position of the pupil center (e.g., as provided by module 720 described above), one or more determined parameters of the optical axis (e.g., as provided by module 722 described above), the estimated 3D position of the center of rotation (e.g., as provided by module 724 described above), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as provided by module 726 described above), and / or one or more determined parameters of the optical axis and / or the visual axis (e.g., as provided by module 722 and / or module 730 described below). The convergence / divergence motion depth estimation module 728 may detect or otherwise obtain a measurement of the user's convergence / divergence motion depth, which may be the distance from the user at which the user's eyes are focused. As an example, when the user is looking at an object 3 feet from their front, the user's left and right eyes have a convergence / divergence motion depth of 3 feet, while when the user is looking at a distant landscape (i.e., the optical axes of the user's eyes are substantially parallel to each other such that the distance between the centers of the user's pupils can be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have an infinite convergence / divergence motion depth. In some implementations, the convergence / divergence motion depth estimation module 728 may utilize data indicating the estimated center of the user's pupil (e.g., as provided by module 720) and determine the 3D distance between the estimated centers of the user's pupils. The convergence / divergence motion depth estimation module 728 may obtain a measurement of the convergence / divergence motion depth by comparing such a determined 3D distance between the pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (as shown by module 726 described above).In addition to the 3D distance between pupil centers and the estimated IPD, the convergence / divergence motion depth estimation module 728 may calculate the convergence / divergence motion depth using known, assumed, estimated, and / or determined geometries. As an example, module 728 may combine the 3D distance between pupil centers, the estimated IPD, and the 3DCoR position in trigonometric calculations to estimate (i.e., determine) the user's convergence / divergence motion depth. In fact, the evaluation of such a determined 3D distance between pupil centers relative to the estimated IPD may serve to indicate a measurement of the user's current convergence / divergence motion depth relative to optical infinity. In some embodiments, the convergence / divergence motion depth estimation module 728 may simply receive or access data indicating the estimated 3D distance between the estimated centers of the user's pupils for the purpose of obtaining such a measurement of the convergence / divergence motion depth. In some embodiments, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by comparing the user's left and right optical axes. In particular, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by locating the distance from the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes on a plane such as a horizontal plane intersect). Module 728 may utilize the user's IPD in this calculation by setting zero depth to be the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the convergence / divergence motion depth estimation module 728 may determine the convergence / divergence motion depth by triangulating the eye tracking data with known or derived spatial relationships.

[0175] In some embodiments, the convergence / divergence motion depth estimation module 728 may estimate the user's convergence / divergence motion depth based on the intersection of the user's visual axes (instead of its optical axis), which may provide a more accurate indication of the distance at which the user is focused. In at least some embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. As will be discussed in more detail in connection with FIG. 10, the user's optical axis and visual axis generally do not coincide. The visual axis is the axis along which a person is looking, while the optical axis is defined by the center of the person's lens and pupil and may pass through the center of the person's retina. In particular, the user's visual axis is generally offset from the center of the user's retina, thereby resulting in different optical and visual axes, which are defined by the location of the user's fovea. In at least some of these embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. The optical axis / visual axis mapping module 730 may correct for the difference between the user's optical axis and visual axis and provide information regarding the user's visual axis to other components within the wearable system, such as the convergence / divergence motion depth estimation module 728 and the light field rendering controller 618. In some examples, the module 730 may use an assumed eye dimension 704 that includes a typical offset of approximately 5.2° inward (towards the nose, towards the user's nose) between the optical axis and the visual axis. In other words, the module 730 may shift the user's left optical axis 5.2° to the right towards the nose (nasally) and the user's right optical axis 5.2° to the left towards the nose (nasally) to estimate the directions of the user's left and right optical axes. In other examples, the module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as indicated by the module 722 described above) to the visual axis. As an additional example, the module 730 may shift the user's optical axis nasally by any range formed by 4.0° to 6.5°, 4.5° to 6.0°, 5.0° to 5.4°, etc., or any of these values.In some arrays, module 730 may apply an offset, at least in part, based on characteristics of a particular user, such as their age, gender, visual prescription, or other relevant characteristics, and / or at least in part based on a calibration process for a particular user (i.e., to determine the optical axis - visual axis offset for a particular user). In at least some embodiments, module 730 may also offset the origins of the left and right optical axes and correspond to the user's CoP (as determined by module 732) instead of the user's CoR.

[0176] When an optional center of perspective (CoP) estimation module 732 is provided, it may estimate the locations of the user's left and right centers of perspective (CoP). The CoP is a useful location for the wearable system and, in at least some embodiments, can be the position directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the locations of the user's left and right centers of perspective based on the 3D location of the user's pupil center, the 3D location of the center of the user's corneal curvature, or such suitable data, or any combination thereof. As an example, the user's CoP can be approximately 5.01 mm in front of the center of the corneal curvature (i.e., 5.01 mm in the direction along the optical axis from the center of the corneal sphere towards the eye's cornea) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of perspective can be directly in front of the center of their pupil. As an example, the user's CoP can be less than about 2.0 mm from the user's pupil, less than about 1.0 mm from the user's pupil, or less than about 0.5 mm from the user's pupil, or any range between these values. As another example, the center of perspective can correspond to a location within the anterior chamber of the eye. In other examples, the CoP can be at 1.0 mm - 2.0 mm, about 1.0 mm, 0.25 mm - 1.0 mm, 0.5 mm - 1.0 mm, or 0.25 mm - 0.5 mm.

[0177] (As the potentially desirable position of the pinhole of the rendering camera and the anatomical position within the user's eye) The center of perspective described herein can be a position that serves to reduce and / or eliminate undesirable parallax shifts. In particular, the optical system of the user's eye approximately matches the theoretical system formed by the front pinhole of the lens projecting onto the screen, and the pinhole, lens, and screen approximately correspond to the user's pupil / iris, lens, and retina, respectively. Further, when two point light sources (or objects) at different distances from the user's eye rotate precisely about the opening of the pinhole (e.g., rotated along a radius of curvature equal to its individual distance from the opening of the pinhole), it may be desirable for there to be little or no parallax shift. Thus, the CoP would be considered to be located at the center of the pupil of the eye (and such a CoP may be used in some embodiments). However, the human eye includes the cornea, which in addition to the pinholes of the lens and pupil, imparts additional refractive power to the light propagating towards the retina. Thus, the anatomical equivalent of the pinhole within the theoretical system described in this paragraph can be the region of the user's eye located between the outer surface of the user's eye's cornea and the center of the user's eye's pupil or iris. For example, the anatomical equivalent of the pinhole can correspond to the region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP to such a position within the anterior chamber of the user's eye.

[0178] As discussed above, the eye tracking module 614 may provide data such as the estimated 3D positions of the left and right eye centers of rotation (CoR), the vergence / accommodation depth, the left and right optical axes of the eyes, the 3D positions of the user's eyes, the 3D positions of the left and right centers of the user's corneal curvature, the 3D positions of the left and right pupil centers of the user, the 3D positions of the left and right viewpoint centers of the user, the user's IPD, etc. to other components such as the light field rendering controller 618 and the alignment observer 620 within the wearable system. The eye tracking module 614 may also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 may include a blink detection module that provides a flag or other alert each time the user blinks, and a saccade detection module that provides a flag or other alert each time the user's eyes saccade (i.e., rapidly shift focus to another point). Example of a rendering controller

[0179] A detailed block diagram of an exemplary light field rendering controller 618 is shown in FIG. 7B. As shown in FIGS. 6 and 7B, the rendering controller 618 may receive eye tracking information from the eye tracking module 614 and may provide an output to the rendering engine 622, which may generate an image to be displayed for viewing by a user of the wearable system. As an example, the rendering controller 618 may receive information regarding the vergence / accommodation depth, the left and right eye centers of rotation (and / or viewpoint centers), and other eye data such as blink data, saccade data, etc.

[0180] The depth plane selection module 750 may receive the convergence / divergence motion depth information, and based on such data, cause the rendering engine 622 to provide the user with the content in a state where the content appears to be located on a specific depth plane (i.e., a specific perspective adjustment or focal length). As discussed in connection with FIG. 4, the wearable system may include a plurality of discrete depth planes formed by a plurality of waveguides that each transmit image information with a variable level of wavefront curvature. In some embodiments, the wearable system may include one or more variable depth planes, such as optical elements, that transmit image information with a level of wavefront curvature that varies over time. In these and other embodiments, the depth plane selection module 750 may cause the rendering engine 622 to transmit the content to the user at a selected depth, at least in part based on the user's convergence / divergence motion depth (i.e., instruct the rendering engine 622 to switch the depth plane on the display 220). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 may render the content at different depths, and also generate and / or provide depth plane selection data to a display hardware such as the display 220. The display hardware such as the display 220 may perform an electrical depth plane switching in response to depth plane selection data (which may be a control signal) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0181] Generally, it may be desirable for the depth plane selection module 750 to select a depth plane that matches the user's current convergence / divergence motion depth so that the user is provided with an accurate perspective adjustment cue. However, it may also be desirable to switch the depth plane in a careful and unobtrusive manner. As an example, it may be desirable to avoid excessive switching between depth planes and / or to switch the depth plane at times when the user is less likely to notice the switch, such as during a blink or an eye saccade.

[0182] The hysteresis band crossing detection module 752 can be particularly useful for avoiding excessive switching between depth planes when the depth of the user's convergence / divergence movement varies at the midpoint or transition point between two depth planes. In particular, the module 752 may cause the depth plane selection module 750 to exhibit hysteresis in its selection of depth planes. As an example, the module 752 may cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the depth of the user's convergence / divergence movement has passed a first threshold. Similarly, the module 752 may cause the depth plane selection module 750 (and thus, can instruct a display such as the display 220) to switch to the first, more distant depth plane only after the depth of the user's convergence / divergence movement has passed a second threshold that is further from the user than the first threshold. In the overlapping region between the first threshold and the second threshold, the module 750 may cause the depth plane selection module 750 to be maintained as currently selected with either depth plane as the selected depth plane, and thus, may avoid excessive switching between depth planes.

[0183] The eye event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A, and may delay several depth plane switches to the depth plane selection module 750 until an eye event occurs. As an example, the eye event detection module 750 may delay the planned depth plane switch to the depth plane selection module 750 until a user blink is detected, and may receive data from the blink detection component in the eye tracking module 614 indicating when the user is currently blinking, and in response, may cause the depth plane selection module 750 to perform the planned depth plane switch during the blink event (such as by instructing the module 750 to perform the depth plane switch on the display 220 during the blink event). In at least some embodiments, the wearable system may be able to shift the content onto a new depth plane during the blink event such that the user is less likely to perceive the shift. As another example, the eye event detection module 750 may delay the planned depth plane switch until an eye saccade is detected. As discussed in connection with eye blinks, such an arrangement may facilitate discrete shifts in the depth plane.

[0184] Optionally, the depth plane selection module 750 may delay the planned depth plane switch for only a limited period of time before performing the depth plane switch, even in the absence of an eye event. Similarly, the depth plane selection module 750 may perform the depth plane switch even in the absence of an eye event when the user's convergence / divergence motion depth is substantially outside the currently selected depth plane (i.e., when the user's convergence / divergence motion depth exceeds a predetermined threshold that exceeds the normal threshold for depth plane switching). These arrangements may help ensure that the eye event detection module 754 does not indefinitely delay the depth plane switch and does not delay the delayed depth plane switch when there is a large depth of field adjustment error.

[0185] The rendering camera controller 758 may provide information indicating the locations of the user's left and right eyes to the rendering engine 622. The rendering engine 622 may then generate content by simulating the camera at the locations of the user's left and right eyes and generating the content based on the viewpoints of the simulated cameras. As discussed above, the rendering camera may potentially be a simulated camera for use in rendering virtual image content from a database of objects within a virtual world. The objects may have locations and orientations relative to the user or wearer, potentially relative to real objects within an environment surrounding the user or wearer. The rendering camera may be included within the rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eyes. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a camera (corresponding to the "rendering camera") having an aperture, lens, and detector that visually perceives the objects within the virtual world. The virtual image is captured from the viewpoint of such a camera having the position of the "rendering camera". For example, the virtual image may be rendered as if it were captured from a camera viewpoint having a specific location relative to the user's or wearer's eyes so as to provide an image that appears to be from the viewpoint of the user or wearer. In some implementations, the image may be rendered as if it were captured from a camera viewpoint having an aperture at a specific location relative to the user's or wearer's eyes (such as a viewpoint center or rotation center or other location as discussed herein).

[0186] The rendering camera controller 758 may determine the positions of the left and right cameras based on the left and right centers of rotation (CoR) determined by the CoR estimation module 724 and / or based on the left and right centers of perspective (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR location and the CoP location based on various factors. As an example, the rendering camera controller 758, in various modes, may constantly align the rendering camera to the CoR location, constantly align the rendering camera to the CoP location, toggle between aligning the rendering camera to the CoR location and aligning the rendering camera to the CoP location over time based on various factors, or discretely switch, or dynamically align the rendering camera to any of a range of different positions along the optical (or visual) axis between the CoR location and the CoP location over time based on various factors. The CoR and CoP positions may optionally pass through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning), which may average the CoR and CoP locations over time, reduce noise at these positions, and prevent jitter when rendering the simulated rendering camera.

[0187] In at least some embodiments, the rendering camera may be simulated as a pinhole camera with a pinhole located at the estimated CoR or CoP position identified by the eye tracking module 614. Since the CoP is offset from the CoR, whenever the position of the rendering camera is based on the user's CoP, the locations of both the rendering camera and its pinhole are offset as the user's eyes rotate. In contrast, whenever the position of the rendering camera is based on the user's CoR, the location of the pinhole of the rendering camera does not move with eye rotation, but the rendering camera (behind the pinhole) may move with eye rotation in some embodiments. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (i.e., rotate) with the user's eyes. (Example of Alignment Observer)

[0188] A block diagram of an exemplary alignment observer 620 is shown in FIG. 7C. As shown in FIGS. 6, 7A, and 7C, alignment observer 620 may receive eye tracking information from eye tracking module 614 (FIGS. 6 and 7A). As an example, alignment observer 620 may receive information regarding the user's left and right eye rotation centers (e.g., the 3D positions of the user's left and right eye rotation centers, which may be on a common coordinate system or have a reference common frame with the head-mounted display system 600). As another example, alignment observer 620 may receive display-related properties, fit tolerance, and an eye tracking validity indicator. The display-related properties may include information regarding the display (e.g., display 200 of FIG. 2), such as the field of view of the display, the size of one or more display surfaces, and the position of the display surface relative to the head-mounted display system 600. The fit tolerance may include information regarding the display alignment volume, which may indicate the distance by which the user's left and right eyes can move from their nominal positions before the display performance is affected. Additionally, the fit tolerance may indicate the amount of display performance impact expected as a function of the position of the user's eyes.

[0189] As shown in FIG. 7C, alignment observer 620 may include a 3D position fit module 770. The position fit module 770 may, as an example, acquire and analyze various data, including the left eye rotation center 3D position (e.g., CoR left), the right eye rotation center 3D position (e.g., CoR right), display-related properties, and fit tolerance. The 3D position fit module 770 may determine the distances of the user's left and right eyes from their respective individual left and right eye nominal positions (e.g., may calculate 3D left error and 3D right error) and provide the error distances (e.g., 3D left error and 3D right error) to the device 3D fit module 772.

[0190] The 3D position fit sense module 770 may also compare the error distance to the display-attached properties and the fit sense tolerance to determine whether the user's eye is within the nominal volume, a partially degraded volume (e.g., a volume in which the performance of the display 220 is partially degraded), or a completely degraded or almost completely degraded volume (e.g., a volume in which it is substantially impossible for the display 220 to provide content to the user's eye). In at least some embodiments, the 3D position fit sense module 770 or the 3D fit sense module 772 may provide an output that qualitatively describes the fit sense of the HMD on the user, such as the fit sense quality output shown in FIG. 7C. As an example, the module 770 may provide an output indicating whether the current fit sense of the HMD on the user is good, within tolerance, or a failure. A good fit sense may correspond to a fit sense that allows the user to view at least a certain percentage (such as 90%) of the image, a within-tolerance fit sense may allow the user to view at least a lower percentage (such as 80%) of the image, while a failed fit sense may be a fit sense in which only an even lower percentage of the image is visible to the user.

[0191] As another example, the 3D position fit sense module 770 and / or the device 3D fit sense module 772 may calculate a visible area metric that may be a percentage of the overall area (or pixels) of an image displayed by the user-visible display 220. The modules 770 and 772 may use one or more models (e.g., mathematical or geometric models), one or more look-up tables, or other techniques for determining the percentage of the image visible to the user as a function of the position of the user's eyes, or a combination of these and other techniques, to calculate the visible area metric by evaluating the positions of the user's left and right eyes relative to the display 220 (e.g., which may be based on the center of rotation of the user's eyes). Additionally, the modules 770 and 772 may determine the area or portion of the image displayed by the display 220 that is expected to be visible to the user as a function of the position of the user's eyes.

[0192] The alignment observer 620 may also include a device 3D fit sense module 772. The module 772 may receive data from the 3D position fit sense module 770 and may also receive an eye tracking valid indicator, which may be provided by the eye tracking module 614 and may indicate whether the eye tracking system is currently tracking the position of the user's eyes, or whether the eye tracking data is unavailable or under error conditions (e.g., determined to be unreliable). The device 3D fit sense module 772 may, if desired, modify the fit sense quality data received from the 3D position fit sense module 770 according to the state of the eye tracking valid data. For example, if data from the eye tracking system is shown to be unavailable or in error, the device 3D fit sense module 772 may provide a notification that an error exists and / or may not provide an output to the user regarding the fit sense quality or fit sense error.

[0193] In at least some embodiments, the alignment observer 620 may provide feedback to the user regarding the quality of the fit and the nature and magnitude of the error. As an example, a head-mounted display system may provide feedback to the user during calibration or fitting processes (e.g., as part of a setup procedure), and may also provide feedback during operation (e.g., if the fit degrades due to slippage, the alignment observer 620 may prompt the user to readjust the head-mounted display system). In some embodiments, the alignment analysis may be performed automatically (e.g., during use of the head-mounted display system), and the feedback may be provided without user input. These are merely illustrative examples. Example of locating the user's cornea using an eye tracking system

[0194] FIG. 8A is a schematic view of an eye showing the corneal sphere of the eye. As shown in FIG. 8A, the user's eye 810 may have a cornea 812, a pupil 822, and a lens 820. The cornea 812 may have a generally spherical shape as indicated by the corneal sphere 814. The corneal sphere 814 may have a center point 816, also referred to as the corneal center, and a radius 818. The hemispherical cornea of the user's eye may curve around the corneal center 816.

[0195] FIGS. 8B-8E illustrate examples of locating the user's corneal center 816 using the 3D corneal center estimation module 716 and the eye tracking module 614.

[0196] As shown in FIG. 8B, the 3D corneal center estimation module 716 may receive an eye tracking image 852 that includes a corneal flash 854. The 3D corneal center estimation module 716 may then simulate the known 3D positions of the eye camera 324 and the light source 326 (obtainable based on data in the eye tracking incidental and intrinsic property database 702, the assumed eye dimension database 704, and / or the per-user calibration data 706) within the eye camera coordinate system 850 in order to project a light ray 856 within the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 may have its origin at the 3D position of the eye tracking camera 324.

[0197] In FIG. 8C, the 3D corneal center estimation module 716 simulates a corneal spherical surface 814a (obtainable based on the assumed eye dimensions from the database 704) and a corneal curvature center 816a at a first position. The 3D corneal center estimation module 716 may then check whether the corneal spherical surface 814a would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8C, the first position does not match because the light ray 860a does not intersect the light source 326.

[0198] Similar to FIG. 8D, the 3D corneal center estimation module 716 simulates a corneal spherical surface 814b and a corneal curvature center 816b at a second position. The 3D corneal center estimation module 716 then checks whether the corneal spherical surface 814b would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8D, the second position also does not match.

[0199] As shown in FIG. 8E, the 3D corneal center estimation module 716 can ultimately determine that the correct positions of the corneal sphere are the corneal sphere 814c and the corneal curvature center 816c. The 3D corneal center estimation module 716 checks that the light from the source 326 will be appropriately reflected from the corneal sphere and imaged at the correct location of the flash 854 on the image 852 by the camera 324, thereby confirming that the illustrated position is correct. By using this arrangement, and the known 3D positions of the light source 326, the camera 324, and the optical properties (such as focal length) of the camera, the 3D corneal center estimation module 716 can determine the 3D location (with respect to the wearable system) of the center 816 of the corneal curvature.

[0200] The processes described herein, at least in relation to FIGS. 8C - 8E, can in fact be an iterative, repetitive, or optimization process for identifying the 3D position of the user's corneal center. Thus, any of a plurality of techniques (e.g., iterative, optimization techniques, etc.) may be used to efficiently and quickly sort through, or reduce the search space of possible positions. Further, in some embodiments, the system may include two, three, four, or more light sources such as the light source 326, and some of all of these light sources may be arranged at different positions, resulting in a plurality of flashes such as the flash 854 located at different positions on the image 852 and a plurality of light rays such as the light ray 856 having different origins and directions. Such embodiments can improve the accuracy of the 3D corneal center estimation module 716 because the module 716 can search for corneal positions that result in some or all of the flashes and light rays being appropriately reflected between their individual light sources and their individual positions on the image 852. In other words, in these embodiments, the positions of some or all of the light sources can rely on the 3D corneal positioning (e.g., iterative, optimization techniques, etc.) process of FIGS. 8B - 8E. (Example of normalizing the coordinate system of the eye tracking image)

[0201] Figures 9A-9C illustrate an exemplary normalization of the coordinate system of an eye-tracking image by components within a wearable system, such as the coordinate system normalization module 718 of FIG. 7A. Normalization of the coordinate system of the eye-tracking image with respect to the location of the user's pupil may compensate for slippage of the wearable system with respect to the user's face (i.e., headset slippage), and such normalization may establish a consistent orientation and distance between the eye-tracking image and the user's eye.

[0202] As shown in FIG. 9A, the coordinate system normalization module 718 may receive the estimated 3D coordinates 900 of the center of rotation of the user's cornea and may receive an unnormalized eye-tracking image, such as image 852. The eye-tracking image 852 and the coordinates 900 may be in an unnormalized coordinate system 850, based, for example, on the location of the eye-tracking camera 324.

[0203] In a first normalization step, the coordinate system normalization module 718 may rotate the coordinate system 850 to a rotated coordinate system 902 such that the z-axis of the coordinate system (i.e., the vergence / accommodation depth axis) may be aligned with the vector between the origin of the coordinate system and the corneal curvature center coordinates 900, as shown in FIG. 9B. In particular, the coordinate system normalization module 718 may rotate the eye-tracking image 850 to a rotated eye-tracking image 904 until the coordinates 900 of the user's corneal curvature center are normal to the plane of the rotated image 904.

[0204] As a second normalization step, the coordinate system normalization module 718 may translate the rotated coordinate system 902 to a normalized coordinate system 910 such that the corneal curvature center coordinates 900 are at a standard normalized distance 906 from the origin of the normalized coordinate system 910, as shown in FIG. 9C. In particular, the coordinate system normalization module 718 may translate the rotated eye-tracking image 904 to a normalized eye-tracking image 912. In at least some embodiments, the standard normalized distance 906 may be about 30 millimeters. Optionally, the second normalization step may be performed prior to the first normalization step. (Example of locating the user's pupil centroid using an eye-tracking system)

[0205] Figures 9D-9G illustrate an example of locating the user's pupil center (i.e., the center of the user's pupil 822 as shown in Figure 8A) using the 3D pupil center locator module 720 and the eye tracking module 614.

[0206] As shown in Figure 9D, the 3D pupil center locator module 720 may receive a normalized eye tracking image 912 that includes the pupil centroid 913 (i.e., the center of the user's pupil as identified by the pupil identification module 712). The 3D pupil center locator module 720 may then simulate the normalized 3D position 910 of the eye camera 324 and project a light ray 914 through the pupil centroid 913 within the normalized coordinate system 910.

[0207] In Figure 9E, the 3D pupil center locator module 720 may simulate a corneal spherical surface such as the corneal spherical surface 901 having a center of curvature 900 based on data from the 3D corneal center estimation module 716 (and as discussed in more detail in relation to Figures 8B-8E). As an example, the corneal spherical surface 901 may be positioned within the normalized coordinate system 910 based on the location of the center of curvature 816c identified in relation to Figure 8E and based on the normalization process of Figures 9A-9C. Additionally, the 3D pupil center locator module 720 may identify a first intersection 916 between the light ray 914 (i.e., the light ray between the origin of the normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea as shown in Figure 9E.

[0208] As shown in FIG. 9F, the 3D pupil center locator module 720 may determine the pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 shares the common center of curvature with the corneal sphere 901, but may have a small radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (i.e., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the assumed eye dimensions 704 of FIG. 7A, from the eye tracking incidental and intrinsic property database 702, and / or from the per-user calibration data 706. In other embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the per-user calibration data 706 of FIG. 7A.

[0209] As shown in FIG. 9G, the 3D pupil center locator module 720 may locate the 3D coordinates of the user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 may use the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection 916 between the simulated corneal sphere 901 and the ray 914 associated with the pupil centroid 913 in the normalized eye tracking image 912, information regarding the refractive index of the cornea, and other relevant information such as the refractive index of air (which may be stored in the eye tracking incidental and intrinsic property database 702) to determine the 3D coordinates of the center of the user's pupil. In particular, in the simulation, the 3D pupil center locator module 720 may bend the ray 916 into the refracted ray 922 based on the refractive difference between air (at a first refractive index of approximately 1.00) and the corneal material (at a second refractive index of approximately 1.38). After considering the refraction caused by the cornea, the 3D pupil center locator module 720 may determine the 3D coordinates of the first intersection 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 may determine that the user's pupil center 920 is located approximately at the first intersection 920 between the refracted ray 922 and the pupil sphere 918. By using this arrangement, the 3D pupil center locator module 720 may determine the 3D location of the pupil center 920 (with respect to the wearable system) within the normalized coordinate system 910. Optionally, the wearable system may denormalize the coordinates of the pupil center 920 to the original eye camera coordinate system 850. The pupil center 920 may be used together with the corneal curvature center 900, and in particular, the optical axis determination module 722 may be used to determine the user's optical axis, and the convergence / divergence motion depth estimation module 728 may be used to determine the user's convergence / divergence motion depth. (Example of the difference between the optical axis and the visual axis)

[0210] As discussed in connection with the optical axis / view axis mapping module 730 of FIG. 7A, the user's optical axis and view axis are generally not aligned, in part because the user's view axis is defined by the fovea, which is generally not at the center of a person's retina. Thus, when a person desires to focus on a particular object, the person aligns their view axis with the object, ensuring that light from the object strikes the fovea while their optical axis (defined by the center of their pupil and the center of curvature of their cornea) is actually slightly offset from the object. FIG. 10 is an example of an eye 1000 that illustrates the optical axis 1002 of the eye, the view axis 1004 of the eye, and the offset between these axes. Additionally, FIG. 10 illustrates the pupil center 1006 of the eye, the center of curvature 1008 of the eye's cornea, and the average center of rotation (CoR) 1010 of the eye. In at least some populations, the center of curvature 1008 of the eye's cornea can be approximately 4.7 mm in front of the average center of rotation (CoR) 1010 of the eye, as indicated by dimension 1012. Additionally, the center of the eye's point of view 1014 can be approximately 5.01 mm in front of the center of curvature 1008 of the eye's cornea, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or directly in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As an additional example, dimension 1012 can be in the range of 3.0 mm to 7.0 mm, 4.0 to 6.0 mm, 4.5 to 5.0 mm, or 4.6 to 4.8 mm, or any range between any value within these ranges and any other value. The center of the eye's point of view (CoP) 1014 can be a useful location for a wearable system in at least some embodiments because aligning the rendering camera to the CoP can help reduce or eliminate parallax artifacts.

[0211] FIG. 10 also illustrates such a location within the human eye 1000 that can be aligned with the pinhole of the rendering camera. As shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 or visual axis 1004 of the human eye 1000 that is closer to the outer surface of the cornea than both (a) the center of the pupil or iris 1006 and (b) the center 1008 of the corneal curvature of the human eye 1000. For example, as shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 of the human eye 1000 that is about 2.97 millimeters rearward from the outer surface of the cornea 1016 and about 5.01 millimeters forward from the center 1008 of the corneal curvature. The location 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to the location 1014 may be regarded as representing the center of the viewing point of the human eye 1000. The optical axis 1002 of the human eye 1000 as shown in FIG. 10 represents the shortest line passing through the center 1008 of the corneal curvature and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 is different from the optical axis 1002 because it represents a line extending from the fovea of the human eye 1000 to the center of the pupil or iris 1006. (Exemplary process of rendering content and checking alignment based on eye tracking)

[0212] FIG. 11 is a process flow diagram of an exemplary method 1100 for providing feedback regarding alignment within a wearable device using eye tracking when rendering content. The method 1100 may be implemented by the wearable system described herein. Embodiments of the method 1100 may be used by the wearable system to render content and provide feedback regarding alignment (i.e., the fit of the wearable device with the user) based on data from an eye tracking system.

[0213] In block 1110, the wearable system may capture an image of one or both eyes of the user. The wearable system may capture an eye image using one or more eye cameras 324, at least as shown in the embodiment of FIG. 3. Optionally, the wearable system may also include one or more light sources 326 configured to shine IR light onto the user's eyes and produce a corresponding flash within the eye image captured by the eye camera 324. As discussed herein, the flash may be used by the eye tracking module 614 to derive various information about the user's eyes, including where the eyes are looking.

[0214] In block 1120, the wearable system may detect a flash and a pupil within the eye image captured in block 1110. As an example, block 1120 may include processing the eye image by a flash detection and labeling module 714 to identify the two-dimensional position of the flash within the eye image, and processing the eye image by a pupil identification module 712 to identify the two-dimensional position of the pupil within the eye image.

[0215] In block 1130, the wearable system may estimate the three-dimensional positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the positions of the centers of curvature of the user's left and right corneas and the distances between those centers of curvature and the user's left and right corneas. Block 1130 may be accompanied by a 3D corneal center estimation module 716 that identifies the positions of the centers of curvature, at least as described herein in connection with FIGS. 7A and 8A-8E.

[0216] In block 1140, the wearable system may estimate the three-dimensional positions of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and a 3D pupil center locator module 720 may estimate the positions of the user's left and right pupil centers as part of block 1140, particularly as described in connection with FIGS. 7A and 9D-9G.

[0217] In block 1150, the wearable system may estimate the three-dimensional position of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and CoR estimation module 724 may estimate the position of the CoR with respect to the user's left and right eyes, particularly as described in relation to at least FIGS. 7A and 10. As a particular example, the wearable system may find the eye's CoR by tracing back along the optical axis from the center of curvature of the cornea towards the retina.

[0218] In block 1160, the wearable system may estimate the user's IPD, vergence / accommodation depth, center of perspective (CoP), optical axis, visual axis, and other desired attributes from the eye-tracking data. As an example, as part of block 1160, the IPD estimation module 726 may estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the vergence / accommodation depth estimation module 728 may estimate the user's depth by finding the intersection (or near the intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 may identify the left and right optical axes over time, the optical axis / visual axis mapping module 730 may identify the left and right visual axes over time, and the CoP estimation module 732 may identify the left and right centers of perspective.

[0219] In block 1170, the wearable system may render content and optionally provide feedback regarding alignment (i.e., the fit of the wearable system with respect to the user's head), based in part on the eye-tracking data identified in blocks 1120-1160. As an example, the wearable system may identify a suitable location for the rendering camera, as discussed in relation to the light field rendering controller 618 (FIG. 7B) and the rendering engine 622, and then generate content for the user based on the location of the rendering camera. As another example, the wearable system may determine whether it is properly fitted to the user or has slipped from its proper location with respect to the user, as discussed in relation to the alignment observer 620 and as discussed in relation to block 1608 of FIG. 16, and may optionally provide feedback to the user indicating whether adjustment of the fit of the device is necessary. In some embodiments, the wearable system may adjust the rendered content based on improper or quasi-ideal alignment in an attempt to reduce, minimize, or compensate for the effects of improper or misaligned alignment, as discussed in relation to block 1610 of FIG. 16. (Overview of Device Alignment)

[0220] The wearable system 200 described in this specification, in order to output high-perception high-quality images, the display 220 of the wearable system 200 (Figure 2) is preferably appropriately fitted to the user (e.g., the input and output of the system 200 interface appropriately with the corresponding parts of the user's head, and the device is positioned and oriented with respect to the user's head so as to be stable and comfortable for wearing and use). As an example, for the display 220 to provide visual content to the user's eyes, the display 220 is preferably positioned in front of the user's eyes, and depending on the relevant properties of the display 220, the user's eyes are preferably positioned within a specific volume (see further discussion associated with Figures 13A and 13B). As an additional example, the speaker 240 is preferably positioned near, on, or in the user's ear to provide high-quality audio content to the user, the audio sensor (e.g., microphone) 232 is preferably positioned within a specific area to receive sound from the user, and the imaging system 462 facing inward (which may include one or more cameras 324 and one or more infrared light sources 326) is preferably positioned appropriately in position and orientation to obtain a clear and unobstructed image of the user's eyes (which may be part of an eye tracking system). These are simply examples of various reasons why the wearable system 200 is preferably appropriately fitted to the user.

[0221] To ensure that the wearable system 200 is properly positioned with respect to the user, the wearable system 200 may include an alignment observer, such as the alignment observer 620 of FIG. 6. In some embodiments, a properly positioned wearable system 200 includes a display positioned such that one or both of the user's eyes receive sufficient image light to be able to view substantially the entire field of view provided by the display 220 of the wearable display system 200. For example, a properly positioned display may enable an image to be visible across about 80% or more, about 85% or more, about 90% or more, or about 95% or more of the field of view of the display, with a brightness uniformity of 80% or more, about 85% or more, about 90% or more, or about 95% or more. Brightness uniformity is equal to 100% times the minimum luminance divided by the maximum luminance across the entire field of view of the display when the display is showing the same content across the entire field of view (100%×L min / L max ). It should be understood that it can be equal to.)

[0222] The alignment observer 620 may use various sensors to determine the extent to which the wearable system 200 fits on the user (e.g., whether the display 220 of the wearable system 200 is properly positioned on the user). As an example, the alignment observer 620 may use an inward-facing imaging system 462, which may include an eye-tracking system, to determine the extent to which relevant portions of the wearable system 200 are spatially oriented with respect to the user, particularly the user's eyes, ears, mouth, or other portions that interface with the wearable system 200.)

[0223] The alignment observer 620 may assist in the calibration process of an initial or subsequent configuration or setting etc. of the wearable system 200 for a particular user. As an example, the alignment observer 620 may provide feedback to the user during the configuration or setting of the wearable system 200 for that particular user. Additionally, or alternatively, the alignment observer 620 may continuously or intermittently monitor the alignment of the wearable system 200 on the user, check for continuous proper alignment during use, and may provide on-the-fly user feedback. The alignment observer 620 may provide user feedback indicating when the wearable system 200 is properly aligned and when it is not, either as part of the configuration process or as part of the alignment monitoring during use. The alignment observer 620 may also provide specific recommendations for ways in which the user can correct any alignment misalignment and achieve proper alignment. As an example, after detecting slippage of the wearable device (such as downward from the user's nasal bridge), the alignment observer 620 may recommend that the user push the wearable device back upward, and may recommend that the user adjust some adjustable components of the wearable device (for example, as described herein in connection with FIGS. 15A and 15B), etc. (Example of an alignment coordinate system)

[0224] FIGS. 12A-12B illustrate an exemplary eye position coordinate system that can be used to define the three-dimensional positions of a user's left and right eyes relative to the display of the wearable system described herein. As an example, the coordinate system may include axes x, y, and z. The z-axis of the coordinate system may correspond to a depth such as the distance between the plane in which the user's eyes are located and the plane in which the display 220 is located (for example, the direction normal to the plane of the front of the user's face). The x-axis of the coordinate system may correspond to a left-right direction such as the distance between the user's left and right eyes. The y-axis of the coordinate system may correspond to an up-down direction that may be the vertical direction when the user is standing upright.

[0225] FIG. 12A illustrates a side view of a user's eye 1200 and a display surface 1202 (which may be part of the display 220 of FIG. 2), while FIG. 12B illustrates a top - down view of the user's eye 1200 and the display surface 1202. The display surface 1202 may be positioned in front of the user's eyes and may output image light to the user's eyes. As an example, the display surface 1202 may comprise one or more externally coupled light elements, active or pixel display elements, and may be part of a stack of waveguides such as the stacked waveguide assembly 480 of FIG. 4. In some embodiments, the display surface 1202 may be planar. In some other embodiments, the display surface 1202 may have other topologies (e.g., be curved). It should be understood that the display surface 1202 may be the physical surface of the display, or simply a plane or other imaginary surface from which it is understood that image light propagates from the display 220 to the user's eyes.

[0226] As shown in FIG. 12A, the user's eye 1200 may have an actual position 1204 offset from a nominal position 1206, and the display surface 1202 may be at position 1214. FIG. 12A also illustrates the corneal apex 1212 of the user's eye 1200. The user's line of sight (e.g., its optical axis and / or visual axis) may be substantially along the line between the actual position 1204 and the corneal apex 1212. As shown in FIGS. 12A and 12B, the actual position 1204 may be offset from the nominal position 1206 by a z - offset 1210, a y - offset 1208, and an x - offset 1209. The nominal position 1206 may represent a preferred position for the user's eye 1200 with respect to the display surface 1202 (sometimes also referred to as a design position that can generally be centered within a desired volume). As the user's eye 1200 moves away from the nominal position 1206, the performance of the display surface 1202 may degrade, as discussed herein in relation to FIG. 14, for example.

[0227] In addition, it should be understood that the default position for the rendering camera can be the nominal position 1206. As discussed herein, the display system may be configured to render content from the perspective of an imaginary rendering camera. As a result, various parameters of the rendering camera, such as the focal length, can affect the appearance of the content presented to the user. For example, the focal length can determine the magnification and size of the virtual content presented to the user. Thus, different focal lengths can be associated with different depth planes.

[0228] In some embodiments, the lens of the rendering camera may be positioned at the nominal position 1206 by default, and the nominal position 1206 is assumed to correspond to the center of rotation, which can be understood to be point 1204 in this example. However, the offset of the center of rotation 1204 from the nominal position 1206 can cause undesirable viewer discomfort. For example, it should be understood that magnification errors can occur from eye to eye, and virtual content can appear larger or smaller than intended. In one scenario, if the focal length of the rendering camera is shorter than expected (e.g., as a result of the user's center of rotation of the eye being positioned behind the nominal position, but without compensating for this displacement or offset within the rendering space), the virtual content can appear smaller than intended. Similarly, if the focal length of the rendering camera is longer than expected (e.g., as a result of the user's center of rotation of the eye being positioned in front of the nominal position, but without compensating for this displacement or offset within the rendering space), the virtual content can appear larger than intended. If such magnification errors vary from eye to eye (e.g., as a result of the center of rotation of one eye being positioned behind the nominal position and the center of rotation of the user's other eye being positioned in front of the nominal position, but without appropriate compensation for these offsets within the rendering space), the perceived size of the same virtual object can vary from eye to eye. This actual difference in size can cause a certain level of discomfort to the user (e.g., potential eye strain and / or headache from trying to resolve the binocular size disparity).

[0229] In some embodiments, the focal length of the rendering camera may be determined based on the z - axis offset between the default position 1206 (the position assumed with respect to the center of rotation) and the actual position of the center of rotation 1204. For example, if the center of rotation is positioned behind the nominal position, then the focal length of the rendering camera may be reduced (e.g., reduced by the amount of the offset). On the other hand, if the center of rotation is positioned in front of the nominal position, the focal length may be increased (e.g., increased by the amount of the offset).

[0230] In addition, in some embodiments, the focal length of the rendering camera may also be calculated based on the depth plane being used by the system. For example, in some embodiments, the optics of the rendering camera may be assumed to follow the thin lens equation (1 / o + 1 / i = 1 / f), where o is the object distance (e.g., the depth plane on which the content is presented), i is a constant (e.g., the distance from the center of rotation to the user's retina), and f is the focal length. As discussed herein, the depth plane on which the content is presented has a certain set distance from the user. As a result, since the quantities o and i are known, the focal length may be determined by solving for f. In some implementations, the rendering camera focal length adjustment may be performed in connection with one or more operations described herein such as step 1170 as described above with reference to FIG. 11 and step 1610 as described in more detail below with reference to FIG. 16. Examples of additional rendering camera adjustment schemes that may be employed by one or more of the systems described herein are "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, U.S. Provisional Patent Application No. 62 / 618,559, filed on January 17, 2018, entitled "AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS", and U.S. Provisional Patent Application No. 62 / 702,849, filed on July 24, 2018, entitled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS" (both of which are incorporated herein by reference in their entirety).

[0231] Continuing to refer to FIG. 12A, it should be understood that a point or volume associated with the user's eye 1200 can be used to represent the position of the user's eye in the alignment analysis herein. The representative point or volume can preferably be any point or volume associated with the eye 1200 that is consistently used. For example, the point or volume may be on or within the eye 1200, or may be arranged away from the eye 1200. In some embodiments, the point or volume is the center of rotation of the eye 1200. The center of rotation can be determined as described herein, is generally symmetrically arranged on various axes within the eye 1200, and has the advantage of simplifying the alignment analysis to enable a single display alignment volume aligned with the optical axis to be utilized for the analysis.

[0232] FIG. 12A also illustrates that the display surface 1202 can be aligned below the user's horizontal line (such that it is seen along the y-axis when the user is looking straight ahead with their optical axis parallel to the ground) and can be tilted (with respect to the y-axis). In particular, the display surface 1202 can be disposed slightly below the user's horizontal line such that the user would need to look down at approximately angle 1216 to view the center of the display surface 1202 when the eye 1200 is at position 1206. This can promote a more natural and comfortable interaction with the display surface 1202, particularly when viewing content rendered at a shorter depth (or distance from the user), as it may be more comfortable for the user to view content below their horizontal line than above it. Additionally, the display surface 1202 can be tilted at an angle 1218 etc. (with respect to the y-axis) such that the display surface 1202 is substantially perpendicular to the user's line of sight when the user is viewing the center of the display surface 1202 (e.g., looking slightly below the user's horizontal line). In at least some embodiments, the display surface 1202 may also be offset left or right (e.g., along the x-axis) with respect to the nominal position of the user's eyes. As an example, when the user's line of sight is focused at a distance less than infinity, the left-eye display surface may be offset to the right and the right-eye display surface may be offset to the left (e.g., the display surfaces 1202 may be offset towards each other) such that they strike the center of the display surface, which can increase user comfort during typical use on a wearable device. (Example of the display alignment volume)

[0233] Figs. 13A - 13B illustrate an exemplary display alignment volume 1302a. The display alignment volume 1302a can represent a spatial volume within which the eye 1200 is positioned to receive image light from the display device. In some embodiments, the center of rotation of the user's eye is preferably positioned such that the eye aligns or receives image information from the display device. In some embodiments, when the center of rotation of the user's eye is within the display alignment volume 1302a, the user can view the entire image output by the display device with high brightness uniformity. For example, as described herein, a properly aligned display can enable an image to be seen across about 80% or more, about 85% or more, about 90% or more, or about 95% or more of the display field of view with a brightness uniformity of 80% or more, about 85% or more, about 90% or more, or about 95% or more. In other words, a display with "good" alignment (as determined, for example, by module 772 in Fig. 7C) can have a brightness uniformity of 90% or more, a display with "acceptable" alignment can have a brightness uniformity of 80% or more, and a display with "failed" alignment can have a brightness uniformity of less than 80%.

[0234] Also, as described herein, the center of rotation 1204 can serve as a convenient reference point for referencing and determining the three-dimensional position of the user's eye. The respective centers of rotation of the user's eyes may be determined using the techniques described herein, such as by tracing back along the user's optical axis from the center of curvature of the cornea to the center of rotation (CoR). However, in general, any desired reference point associated with the user's eye may be utilized in the processes and systems described herein. The display alignment volume 1203 may represent a volume of space within which the display surface 1202 can operate with substantially full potential (e.g., without significant degradation of the type described in relation to FIGS. 15A and 15B of the performance of the display surface 1202). If the user's eye (e.g., the center of rotation 1204 of the user's eye) is not within the alignment volume 1302a, the user may experience degraded performance, and some or all of the content provided by the display surface 1202 may be partially darkened or completely invisible to the user.

[0235] As shown in FIG. 13A, the alignment volume 1302a may have a frustum shape, where its upper portion is typically the portion of a pyramid remaining after being cut by a plane parallel to its base. In other words, the alignment volume 1302a can become larger along the x and y axes when the user's eye is closer to the display surface 1202 (see, e.g., FIGS. 12A and 12B), and can become smaller along the x and y axes when the user's eye is farther from the display surface 1202. The frustum is an example of a truncated shape where the shear plane (e.g., the line where a part of the original shape is cut) within it is parallel to the base of the volume. In general, an alignment volume such as volume 1302a may take the shape of a volume truncated in any manner, such as by one or more non-parallel shear planes (such as those shown in FIG. 13B) or by one or more non-planar shears.

[0236] The dimensions of the alignment volume may depend on the specific implementation of the display surface 1202 and other elements of the wearable system. As an example, FIG. 13B illustrates that the alignment volume 1302b can be angled with respect to the display surface 1202. In the example of FIG. 13B, the portion of the alignment volume 1302b closest to the display surface 1202 is angled away from the display surface 1202 such that as the user's eye moves vertically (in the y direction) at the front of the volume (the z position closest to the display surface 1202), the user's eye must move away from the display surface (along the z axis) and remain inside the alignment volume 1302b. In some embodiments, the shape of the alignment volume 1302b may be based on the capabilities of the eye tracking system such that it may not be possible to track the user's eye outside of the angled volume 1302b of FIG. 13B.

[0237] The dimensions and shape of the alignment volume can also depend on the nature of the various parts of the display 220, which may include the display surface 1202. As an example, the display 220 may include one or more waveguides (which may be stacked and may provide multiple converging / diverging motion cues to the user), an internal coupling element that receives light from an image input device and couples the light into the waveguides, a light dispersing element (sometimes also referred to as an orthogonal pupil expander (OPE)) that is disposed on the waveguide and disperses the light to an external coupling element, and an external coupling element (sometimes also referred to as an exit pupil expander (EPE)) that directs the light towards the viewer's eye, a light field display. In some embodiments, as described herein, the display surface 1202 is the surface or a portion of the surface from which light with image information is output from the display system and forms an image in the user's eye. For example, the display surface 1202 may be an area on the waveguide surface defined by the external coupling element or EPE, and the perimeter of the display surface 1202 is the perimeter of the area defined by the external coupling element or EPE. Further examples and details of light field displays and components of such displays are also described in connection with at least FIGS. 9A-9C of U.S. Provisional Patent Application No. 62 / 642,761, filed Mar. 14, 2018, which is incorporated herein by reference in its entirety.

[0238] In some embodiments, the x-dimension of the alignment volume 1302a may range from about 3.0 mm, 3.0 mm, 3.5 mm, 4.0 mm, 4.5 mm, 4.7 mm, 5.0 mm, 5.5 mm, or 6.0 mm along the back of the volume (e.g., the maximum distance along the z-axis from the display surface), or may be less than 3.0 mm or greater than 6.0 mm. Similarly, the y-dimension of the alignment volume 1302a may range from about 2.5 mm, 3.0 mm, 3.5 mm, 3.9 mm, 4.0 mm, 4.5 mm, 5.0 mm, 5.5 mm, or 6.0 mm along the back of the volume, or may be less than 2.5 mm or greater than 6.0 mm. At the nominal x and y positions, the z-dimension of the alignment volume 1302a may range from about 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.5 mm, or 11.0 mm, or may be less than 7.0 mm or greater than 11.0 mm. The x and y dimensions may be larger at the front of the volume. As an example, the x and y dimensions of the alignment volume at the front of the volume may be about 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 8.9 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.0 mm, 10.5 mm, 11.0 mm, 11.4 mm, 11.5 mm, 12.0 mm, or 12.5 mm, or may be less than 7.0 mm or greater than 12.5 mm. As a specific example, the dimensions of the alignment volume may include a z-dimension of about 9 mm, an x-dimension of about 4.7 mm at the back of the volume and about 11.4 mm at the surface of the volume, and a y-dimension of about 3.9 mm at the back of the volume and about 8.9 mm at the front of the volume.

[0239] In at least some embodiments, there can be a plurality of alignment volumes, such as volumes 1302b and 1304, each associated with a different minimum level of display performance. By way of example, volume 1304 in FIG. 13B can be smaller than volume 1302 and can represent a volume in which a user perceives all of the content provided by display surface 1202 with 100% brightness uniformity, while the larger volume 1302b can represent a volume in which a user perceives at least 90% of the content provided by display surface 1202 with 100% brightness uniformity. Thus, in some embodiments, the display system may be configured to determine whether the user's eye is within the alignment, i.e., viewing volume, of the display and / or whether the user's eye is within a threshold distance from the alignment volume. For example, in some embodiments, the smaller volume 1304 can be considered the baseline alignment volume or viewing volume, and the boundary of the larger volume 1302b can be considered to define the limit of an acceptable threshold distance from the alignment volume. In some embodiments, if the display system determines that the position of the eye exceeds a threshold distance outside the viewing volume of the display system, the display system may provide feedback to the user indicating that the display and the eye are not properly aligned for output, as discussed herein, and / or may be configured to take an action to reduce the display degradation caused by the misalignment.

[0240] In some embodiments, the display alignment volume may be determined, at least in part, based on an application that is currently running or will be launched on the system. For example, a larger display alignment volume (e.g., volume 1302b) may be employed by the display system when launching an application in which the read material may occupy only a portion (e.g., the central portion) of the display's field of view, such as a read-based application, or in which only a portion of the field of view is utilized to display virtual content. As a result, the loss of the user's ability to perceive image content at the periphery of the field of view due to misalignment may be imperceptible when such a read-based application is launched, as the application may not be able to present content at the periphery. As a result, a larger display alignment volume may be utilized, for example, to reduce unnecessary notifications to the user regarding misalignment when such image alignment may not affect the content presented by the application running on the display system. Similarly, in another example, a larger display alignment volume (e.g., volume 1302b) may be employed by the display system when launching an application such as a motion-oriented application or another application that requires a relatively high level of user engagement and / or physical activity, in which the head-mounted display is expected to shift with respect to the user's eyes. In such applications, notifications regarding misalignment may be considered, for example, distracting or otherwise detrimental to the user experience. That is, providing a larger display alignment volume reduces the likelihood that the display system will generate such corrections for the user. In some other embodiments, a smaller display alignment volume (e.g., volume 1304) may be utilized when launching an application in which it is desired to provide content across the entire field of view of the display or to the peripheral portions of the field of view.Such applications may include immersive gaming applications in which content is displayed across the entire field of view of the display. In some implementations, the display alignment volume may be determined based on other factors including user preferences, user visual prescriptions, system operating conditions, and the like.

[0241] Figures 13C and 13D illustrate an exemplary display alignment volume configured to use the center of rotation of the eye as a reference point for the position of the user's eye and the eye relative to the display surface. In particular, FIG. 13C illustrates an exemplary positioning of a display alignment volume, such as alignment volume 1302b, within user eye 1200. In the embodiment of FIG. 13C, the center of rotation 1204 of eye 1200 is generally centered within alignment volume 1302b. Additionally, alignment volume 1302b is illustrated with an exemplary dimension of approximately 9 mm in depth, and approximately 3.5 mm in width and 3 mm in height at the midpoint of the depth axis. As discussed herein, the dimensions of the alignment volume can vary and can be related to the nature of the various components of the wearable system. FIG. 13C also illustrates an eye structure 1340, which can be the lens or pupil of eye 1200.

[0242] FIG. 13D shows a larger context in which the user's eye 1200 is generally positioned within alignment volume 1302b and viewing virtual content 1350 through display surface 1202. As discussed herein, virtual content, such as virtual content 1350, can be provided to the user with vergence / accommodation cues associated with a depth that exceeds the depth of display surface 1202. In other words, virtual content 1350 can appear to the user with eye 1200 to be at a distance away from the user greater than display 1202. Such an arrangement is illustrated in the embodiment of FIG. 13D.

[0243] Continuing to refer to FIG. 13D, it should be understood that the display alignment volume 1302b can be an imaginary volume having a boundary defined by a projection from the perimeter of the display surface 1202 to a point inside the eye 1200. For example, the projection can define a pyramid, and the display alignment volume 1302b can be a frustum of that pyramid. Thus, the cross-sectional shape of the display alignment volume 1302b along a plane facing the display surface 1202 on the optical axis is similar to the shape created by the perimeter of the display surface 1202. For example, as shown, if the display surface 1202 is square, the cross-sectional shape of the display alignment volume 1302b is also square. Additionally, as also shown, if the center of the display surface 1202 is below the user's horizontal line, the frustum can also be tilted so that the center of the front of the display alignment volume 1302b is also below the user's horizontal line. In some embodiments, it should be understood that the relevant perimeter of the display surface 1202 is the perimeter of the area of the display over which image light or display content is output. The boundaries of the display alignment volume 1302b are defined within the same coordinate system in which various features such as the center of rotation 1204 of the eye 1200 are mapped therein, thereby allowing a comparison between the display alignment volume 1302b and these various features.

[0244] In some embodiments, the center of rotation 1204 of the eye is centered within the frustum that defines the display alignment volume 1302b. However, it should be understood that the nominal location of the center of rotation 1204 of the eye and / or the overall shape of the frustum can be determined experimentally or selected using criteria other than the projection from the display surface 1202 such that the display system can provide accurate feedback regarding properly aligning the display and the quality and level of alignment that, even if not ideal, can be acceptable. (Examples of Display Performance at Various Alignment Positions)

[0245] FIG. 14 illustrates how the performance of display surface 1202 can vary with the position of user's eye 1200. As shown, the light rays from display surface 1202 can be directed at the eye at an angle such that the light rays from the edge of display surface 1202 propagate inwards towards eye 1200. Thus, cone 1202’ represents the cone of light that is output by display surface 1202 and forms an image on eye 1200.

[0246] As a result, as display surface 1202 shifts with respect to eye 1200, the exit pupils of the pixels corresponding to the individual portions of the visual field do not reach the retina of eye 1200, and the image appears dark in those portions of the visual field. The positions 1204a, 1204b, 1204c, and 1204d of the center of rotation of the eye are effectively offset with respect to the idealized position 1204’ of the center of rotation. That is, movement of display surface 1202 with respect to eye 1200 can potentially move the center of rotation of the eye outside of the display alignment volumes 1302a, 1304, 1302b (FIGS. 13A and 13B) for display surface 1202. As discussed herein, the display alignment volumes can be associated with display surface 1202; for example, the display alignment volumes can be defined by projection from display surface 1202. As a result, as display surface 1202 moves with respect to eye 1200, the display alignment volumes 1302a, 1302b (FIGS. 13A and 13B) also move. FIG. 14 illustrates the various positions (e.g., positions 1204a, 1204b, 1204c, and 1204d) of the center of rotation of the user's eye, the relative position of display surface 1202, and the representation (e.g., representations 1400a, 1400b, 1400c, and 1400d) of how the content provided by display surface 1202 will be perceived by the user at each of the various positions.

[0247] In Example 1400a, the center of rotation of the user's eye can be centered at position 1204a within an alignment volume such as alignment volume 1300b (e.g., a volume with high image quality therein because the eye 1200 receives substantially all of the image light output by the display surface 1202 on its retina). Representation 1400a can represent the user's perception (or view) of the content provided by the display surface 1202 when the user's eye is at position 1204a. As shown by representation 1400a, the luminance for substantially all of the content across the display surface 1202 can be uniform and can be at a full or nearly full brightness level.

[0248] In Example 1400b, the center of rotation of the user's eye is outside a preferred display alignment volume such as volume 1304 (FIG. 13B), but can be at position 1204b within a secondary alignment volume such as volume 1302b (e.g., a volume in which the display performance is only slightly degraded therein). Representation 1400b can represent the user's perception (or view) of the content provided by the display surface 1202 when the center of rotation of the user's eye is at position 1204b. As shown by representation 1400b, portion 1402 of the image along the right side of the display surface 1202 can have a perceived reduced luminance (e.g., 50% luminance) due to a misalignment of the user's eye with respect to the display surface 1202.

[0249] In Example 1400c, the center of rotation of the user's eye can be at position 1204c, which can be outside (or on the outer edge of) a second alignment volume such as volume 1302b (FIG. 13B). Representation 1400c can represent the user's perception (or view) of the content provided by display surface 1202 when the center of rotation of the user's eye is at position 1204c. As shown by representation 1400c, portion 1406 along the edge of the displayed image that the user perceives appears completely (or almost completely) darkened due to misalignment and thus cannot be seen by the user. In an array where some of the pixels of the display therein are below the luminance level at which they are perceived, the display can provide a reduced field of view (e.g., the user may not be able to perceive the full field of view that the display would otherwise present). Additionally, there can be a band or portion 1404 of the image having a gradually reduced lightness between the dark portion 1406 and the rest of the representation.

[0250] In Example 1400d, the center of rotation of the user's eye can be at position 1204d, which can be significantly off from the desired alignment volume. Representation 1400d can represent the user's perception (or view) of the content provided by display surface 1202 when the center of rotation of the user's eye is at position 1204d. As shown by representation 1400d, due to a significant misalignment, most of the image 1410 can appear completely (or almost completely) dark to the user, and a substantial portion 1408 of the image can appear darkened.

[0251] As discussed herein, it should be understood that a display system can be configured to increase the amount of light output or the perceived brightness of portions of the display's field of view that are darkened due to misalignment. For example, in response to determining that there is misalignment, as discussed herein, the display system may provide a notification in the form of a flag or instruction to the display and increase the amount of light output to the user for pixels that are expected to be darkened due to misalignment. For example, in Example 1400b, the pixels representing the image information within portion 1402 may increase their perceived brightness so as to reduce the expected reduction in perceived brightness due to misalignment.

[0252] It should be understood that the ability to increase brightness and compensate for darkening can decrease with a high level of misalignment. For example, in Example 1400d, area 1410 can become dark because the eye 1200 does not receive any light from the pixels within those areas. As a result, increasing brightness can reduce the darkening due to misalignment in portion 1402 (Example 1400b), but increasing brightness can make it impossible to reduce the darkening within portion 1410 (Example 1400d) of the display where the misalignment prevents the eye from receiving any light at all. Portion 1410 is larger than portion 14, and as a rough indicator, the size of the portion expected to be darkened can indicate whether increasing brightness will be effective. That is, if the size of the portion expected to be darkened (e.g., the number of pixels) is large enough, in some embodiments, it can be assumed that the misalignment is large enough that increasing brightness will not be effective for most of those pixels. As a result, in some embodiments, the display system may compare the number of pixels within the portion expected to be darkened and, if that number exceeds a threshold, be configured to provide feedback to the user indicating that the display and the eye are not properly aligned. (Example of a replaceable fitting component for a wearable system)

[0253] Figures 15A and 15B show exploded perspective views of a wearable system 220 that may include replaceable fitting components. In particular, FIG. 15A illustrates a situation where the wearable system 200 may include replaceable back padding such as pads 1500a, 1500b, and 1500c, while FIG. 15B illustrates a situation where the system 200 may include replaceable forehead pads such as pad 1502 and replaceable bridge-of-nose pads such as pad 1504. These replaceable pads may be used to adjust the fit of the wearable system 200 for individual users, which may have varying anatomical attributes (e.g., the degree to which the display 220 and frame 230 fit different users). As an example, a user with a relatively small head may benefit from attaching relatively large back pads 1500a, 1500b, and 1500c to the frame 230, while a user with a relatively large head may achieve better results (e.g., better optical performance and stability of the frame 300 on their head) by attaching relatively small back pads or even omitting the back pads altogether. Similarly, a user with a prominent nose and / or forehead may benefit from smaller forehead pads 1502 and / or bridge-of-nose pads 1504, while a user with a less prominent nose and / or forehead may benefit from larger forehead pads 1502 and / or bridge-of-nose pads 1504. These are merely illustrative examples, and generally, determining the set of replaceable pads that will provide the best fit for any particular user can be complex. As described herein, the display system may show a notification to the user indicating that different replaceable fitting components may desirably provide proper alignment of the display with the user.

[0254] Referring to FIG. 15A, the wearable system may include one or more housing openings 1510. The housing openings 1510 may be openings within the frame 230 and may optionally include a lens or other structure, such as an optically transmissive structure for mechanical protection of the waveguide of the display, as desired. The lens within the housing openings 1510 may be clear (e.g., completely or substantially transparent) or may be partially opaque (e.g., to reduce the level of ambient light passing through the openings 1510). The openings within the frame 230 are illustrated as being generally circular in FIG. 15A, but may have any desired shape. Exemplary Process for Observing Device Alignment

[0255] FIG. 16 is a process flow diagram of an exemplary method 1600 for observing device alignment and providing feedback regarding the alignment or compensating for misalignment within a wearable device. The method 1600 may be implemented by the wearable system described herein. Embodiments of the method 1600 may be used by the wearable system to provide feedback regarding alignment (i.e., the fit of the wearable device with the user) based on data from an eye tracking system, adjust the display, and attempt to compensate for fit errors (e.g., misalignment).

[0256] In block 1602, the wearable system may obtain a fit tolerance. The fit tolerance may include information associated with display alignment volumes such as volumes 1302a, 1302b, or 1304. In particular, the fit tolerance may include information associated with the nominal (e.g., normal) position of the user's eyes relative to the wearable device, and may include information associated with the degree to which the variance from the nominal position affects device performance. As an example, the fit tolerance may include information regarding the range of nominal positions within which the wearable device can interface with the user, with at least a desired amount of performance (e.g., with less than 50% dimming on any pixel within the display).

[0257] In block 1604, the wearable system may obtain alignment data. The alignment data may include the spatial relationship between various components of the wearable system and the associated parts of the user. As an example, the alignment data may include one or more of the three-dimensional position of the user's left eye relative to the left-eye display of the wearable system, the 3D position of the user's right eye relative to the right-eye display, and the 3D position of the user's ear relative to the audio output (e.g., speaker, headphones, headset, etc.) of the wearable system. The wearable system may obtain the alignment data using any suitable mechanism. As an example, the wearable system may use an eye-tracking camera 324 of the type shown in FIG. 3 (or other cameras, which may or may not be facing inward) to capture an image of one or both of the user's eyes and determine the relative position of the user's eyes and the wearable system. As another example, the wearable system may include a depth sensor, pressure sensor, temperature sensor, light sensor, audio sensor, or other sensor and may measure or obtain alignment data such as the position of the wearable device relative to the user.

[0258] In block 1606, the wearable system may determine the fit characteristics. As an example, the wearable system may determine whether the user's left eye is within the left eye alignment volume (such as one of volumes 1302a, 1302b, or 1304 for the left eye) and whether the user's right eye is within the right eye alignment volume (such as one of volumes 1302a, 1302b, or 1304 for the right eye). Block 1606 may also involve the step of determining the distance of the user's eye (or other body part) from its nominal position. As an example, in block 1606, the wearable system may determine that at least one of the user's eyes is outside its individual display alignment volume and the extent and direction to which the user's eye is outside its display alignment volume. Information regarding the direction and magnitude of the misalignment (e.g., the distance between the alignment volume or nominal position and the actual position of the user's eye or other body part) may beneficially be utilized in blocks 1608 and 1610.

[0259] In block 1608, the wearable system may provide feedback to the user (or some other entity) regarding the fit characteristic determined in block 1608. As an example, if in block 1606 the wearable system determines that the wearable device is too low relative to the user's eyes, the wearable system may provide the user with a notice suggesting that in block 1608 the user utilize an appropriate nose bridge pad 1504 (e.g., add a nose bridge pad if neither were previously attached, or replace an existing nose bridge pad with a larger or taller one). Conversely, if the wearable device determines that it is too high relative to the user's eyes, the system may provide the user with a proposal to use a smaller nose bridge pad or to remove the pad entirely (if designed to be worn without a pad). As another example, the wearable system may provide feedback to the user proposing changes to forehead pads such as pad 1502, changes to rear pads such as pads 1500a - 1500c, changes to other adjustable components of the wearable system, or changes to the way the user is wearing the wearable system (e.g., instructions to move or rotate the system in a particular orientation relative to the user). Generally, the user feedback may be generated based on other metrics such as the position of the user's eyes relative to the display or the portion of the visible image identified by the system. As an example, if the system determines that the user's eyes are above the alignment volume, the system may recommend to the user that the user push the wearable device up along their nose bridge to correct the alignment misalignment.

[0260] User feedback may be provided using any suitable device. As an example, user feedback may be provided via video presented by a display within a wearable device or an external display, or via audio presented by a wearable device or an external device. In various embodiments, the wearable device may provide a two-way guide to assist the user in achieving proper alignment in a relatively intuitive manner. As an example, the wearable device may display two virtual targets, one representing the position of the user's eyes and the other representing the nominal alignment position. As the user moves the wearable device and adjusts its fit, the user can perceive the extent to which the adjustment affects the alignment, and the user can quickly and intuitively achieve proper alignment.

[0261] In an arrangement where user feedback is provided by an output device such as a display that is part of a wearable device, the wearable device may provide user feedback in a manner that ensures the user can perceive the feedback. As an example, consider rendition 1400d of FIG. 14. In such an example, the wearable system may move user feedback into a portion of the displayed image that is perceived by the user, e.g., the left half of the display, as opposed to the non-visible right half of the display of example 1400d of FIG. 14.

[0262] In some embodiments, feedback of the type described herein may be provided to a salesperson in a retail environment, and the feedback may be communicated via a network to the salesperson's computer or mobile device.

[0263] In block 1608, the wearable system may adjust its output and input and compensate for uncorrected fit error. In some embodiments, block 1608 may be implemented only after the user has failed to correct the fit error in response to feedback. In other embodiments, block 1608 may be implemented until the user corrects the fit error. In some embodiments, block 1608 may be implemented each time the user decides to continue using the wearable system with a fit error. In some embodiments, block 1608 may be omitted.

[0264] As an example, the wearable system may adjust its output and input in block 1608 by adjusting a portion of the displayed image (e.g., to compensate for misregistration-induced darkening of the type shown in FIG. 14), by adjusting the microphone input (e.g., increasing the microphone gain when the user is too far from the microphone or decreasing the microphone gain when the user is too close to the microphone), and by adjusting the speaker output (e.g., increasing or decreasing the speaker volume when the user is too close to or too far from the speaker, respectively, within the wearable device). As one particular example, the wearable system may selectively increase the luminance of a portion of an image such as portions 1402, 1404, or 1408 of FIG. 14 in an attempt to reduce misregistration-induced darkening. In some other embodiments, the wearable system may recognize that certain portions of the image such as portions 1406 or 1410 of FIG. 14 are invisible to the user and may reduce the light output within those regions to reduce the energy consumption by the wearable system. For example, in a configuration where different portions of the image may have dedicated selectively activatable light sources or portions of light sources, one or more light sources or portions of light sources associated with the invisible portions of the image may have their light output reduced or turned off. (Example of identifying display alignment volume)

[0265] Figures 17A - 17H illustrate a diagram of a light field projected by a display and how the intersections of the light fields can partially define a display alignment volume. FIG. 18 illustrates a top - down view of overlapping light fields projected by a display and how the intersections of the light fields can partially define a display alignment volume. As FIGS. 17A - 17H and 18 illustrate, the size and shape of the display alignment volume can depend, in part, on the geometry of the display (which can be the display 220 of FIG. 2) and the angle at which externally coupled light propagates out of the display (e.g., out of a waveguide display). It should be understood that the angle at which light is output can define the FOV of the display. That is, the larger the angle with respect to the normal, the larger the FOV provided. In some embodiments, the display surface can output an angle large enough to provide the desired FOV.

[0266] FIGS. 17A - 17H and 18 illustrate the display 220, which can be a light field display (forming the display surface 1202, also illustrated in various other figures of this specification, including FIGS. 12A - 14) that includes elements such as a waveguide 1701, an internal coupling element 1702, an orthogonal pupil expander (OPE) 1704, and an exit pupil expander (EPE) 1706. By way of example, the internal coupling element 1702 can receive light from an image source and couple the light into the waveguide 1701. The waveguide 1701 can transmit the light to the OPE 1704, which can provide pupil expansion and direct the light to the EPE 1706, and the EPE 1706 (which can be provided on the display surface 1202) can provide further pupil expansion and transmit the light to the user's eye. Further examples and details of light field displays and components of such displays are also described in relation to at least FIGS. 9A - 9C of U.S. Provisional Patent Application No. 62 / 642,761, filed on Mar. 14, 2018, which is incorporated herein by reference in its entirety.

[0267] FIG. 17A illustrates an example where display 220 projects light 1710 associated with virtual image content at optical infinity and in the rightmost region (e.g., the rightmost pixel) of the FOV of the display. In contrast, FIG. 17B illustrates an example where display 220 projects light 1712 associated with an object at optical infinity and in the leftmost region (e.g., the leftmost pixel) of the FOV of the display. FIG. 17C illustrates the overlapping region 1714 of the light 1710 of FIG. 17A and the light 1712 of FIG. 17B. Region 1714 may be a horizontal alignment volume. In particular, when the user's eye is positioned within region 1714 of FIG. 17C, the user is able to perceive the object in both the rightmost region of the FOV (as in FIG. 17A) and the leftmost region of the FOV (as in FIG. 17B) (e.g., display 220 is able to provide light from there to the user).

[0268] FIGS. 17D - F illustrate examples similar to those of FIGS. 17A - 17E but in the vertical direction. In particular, FIG. 17D illustrates an example where display 220 projects light 1716 associated with an object at optical infinity and in the bottommost region (e.g., the bottommost pixel) of the FOV of the display, while FIG. 17E illustrates an example where display 220 projects light 1718 associated with an object at optical infinity and in the uppermost region (e.g., the uppermost pixel) of the FOV of the display. Similarly, FIG. 17F illustrates the overlapping region 1720 of the light 1716 of FIG. 17D and the light 1718 of FIG. 17E. Region 1720 may be a vertical alignment volume. In particular, when the user's eye is positioned within region 1720 of FIG. 17F, the user is able to perceive the object in both the bottommost region of the FOV (as in FIG. 17D) and the uppermost region of the FOV (as in FIG. 17E) (e.g., display 220 is able to provide light from there to the user).

[0269] Figures 17G and 17H illustrate the intersection of region 1714 of FIG. 17C and region 1720 of FIG. 17F (as region 1722). In particular, FIG. 17G illustrates region 1722 where light from objects at the four corners of the FOV of display 220 overlaps. FIG. 17H simply illustrates the contour of region 1722. As should be clear, when the user's eye is placed within region 1722, the user can perceive an object at any location within the FOV of the display (e.g., display 220 can provide light therefrom to the user). In some embodiments, the alignment volume of display 220 can be understood to be the viewing volume of the head-mounted display through which light representing all the pixels of the virtual image content presented by the head-mounted display is expected to pass.

[0270] In some embodiments, increasing the FOV of display 220 (horizontally, vertically, or in a combination thereof) while keeping other attributes (such as display size) constant can have the effect of shrinking the associated alignment volume (e.g., horizontal volume 1714, vertical volume 1720, or combined alignment volume 1722). As an example, consider FIGS. 17A-C and the horizontal FOV and alignment volume 1714. An increase in the horizontal FOV of display 220 means that light 1710 from an object on the right horizontal edge is projected at a more acute angle (e.g., a larger angle from the normal to display surface 1202) by display surface 1202 (e.g., EPE 1706). Similarly, light 1712 from an object on the left horizontal edge is also projected at a more acute angle. Thus, from the perspective of FIG. 17C, the apex of the horizontal alignment volume 1714 moves towards display surface 1202 with an increase in the horizontal FOV, thereby shrinking volume 1714. Similar considerations can apply in some embodiments to the vertical FOV and vertical alignment volume and the overall FOV and overall alignment volume.

[0271] FIG. 18 shows a top view of display 220, including display surface 1202, which may have a rectangular shape and a particular FOV, and the light rays produced by the display. Generally, the alignment volume of display 220 in FIG. 18 may be volume 1802, which appears triangular in the top and bottom viewpoints of FIG. 18. Volume 1802 may represent a volume where the various light fields formed by the light shown in FIGS. 17A - 17G overlap. It should be understood that when the user's eye is located outside volume 1802 (e.g., within volume 1804), the light of the light field from at least a portion of display 220 will not be able to reach the user's eye, resulting in a partial or complete darkening of a portion of the FOV.

[0272] The side view of the display and the alignment volume will have approximately the same appearance as shown in FIG. 18 (at least with respect to a rectangular display), but note that the illustrated dimension of display 220 will be the height of display 220, not its width, and the illustrated FOV will be the vertical FOV, not the horizontal FOV shown in FIG. 18. Thus, volume 1802 may actually have a generally pyramidal shape. In other embodiments, the display may have a non - rectangular shape, such as a circular shape, an oval shape, a free - form shape, or any other desired shape. In such embodiments, the corresponding alignment volume may be determined by projecting the light fields onto the associated FOV and identifying where those light fields intersect (which may correspond to volume 1802) and where they do not intersect (which may correspond to volume 1804).

[0273] As discussed herein, the "base" of the pyramid may be truncated (which may help move the user's eye away from the display so that the user's eyelashes do not affect the display when properly aligned), and the "top" of the pyramid may also be truncated (otherwise, it may be useful in reducing the effect of noise in determining the location of the user's eye, where in the "top" of the pyramidal alignment volume, it may quickly align or deviate therefrom). It should be understood that the "top" is close to the apex of volume 1802 and the "base" is close to waveguide 1701. When the user's eye is located within region 1804 outside alignment volume 1802, the user may perceive dimming of some or all of the pixels of display 220 as discussed herein (see, e.g., FIG. 14).

[0274] Generally, the alignment volume may be adjusted in any number of ways (e.g., truncated or otherwise reduced) for various reasons. As an example, the alignment volume may be truncated such that the volume has a minimum distance from the display 220 and prevents the user's eyelashes or eyelids from affecting the display 220. As a result, in some embodiments, the display system (e.g., the processing electronics of the display system) may be configured to determine whether the user's eye is within the alignment volume 1802 at least in part by determining whether one or both eyes are less than a minimum threshold distance (e.g., a minimum allowable distance) from the display 220. If the eye is determined to be less than the minimum threshold distance from the display, the display system may interpret this result as meaning that the eye is outside the alignment volume 1802 and, thus, the display and the eye are not properly aligned. As a result, the display system may provide feedback to the user indicating that the alignment is inappropriate, as discussed herein, and / or may be configured to take action to reduce the display degradation caused by the misalignment. In some implementations, such a minimum threshold distance may vary in one or more dimensions. For example, the minimum threshold distance may vary linearly along the z-axis as a function of the nominal position of the display and / or the distance from the surface.

[0275] In addition to, or alternatively to, determining whether the eyes are within a minimum distance from the display 220, in some embodiments, the display system (e.g., the processing electronics of the display system) may be configured to determine whether the user's eyes are within the alignment volume 1802 by determining at least in part whether one or both eyes exceed a maximum threshold distance from the display 220. It should be understood that the maximum threshold distance may correspond to the distance at which the "top" of the pyramid 1802 described above is truncated. If the eyes are determined to exceed the maximum threshold distance from the display, the display system may interpret this result as meaning that the eyes are outside the alignment volume 1802 and, thus, that the display and the eyes are not properly aligned. As a result, the display system may provide the user with feedback indicating that the alignment is inappropriate, as discussed herein, and / or may be configured to take actions to reduce the display degradation caused by the misalignment.

[0276] In addition to, or alternatively to, determining whether the eye is within a minimum distance from and / or beyond a maximum distance from the display 220, the wearable system may have an eye-tracking system that includes elements such as the camera 324 and the light source 326 of FIG. 6 that can track only the user's eye when the user's eye is within the eye-tracking volume and cannot exactly overlap with the display alignment volume. In some embodiments, the camera of the eye-tracking system may have a field of view that encompasses the display alignment volume. Thus, the display alignment volume can be considered a sub-space or a part of the field of view of the camera. The display system may be configured to determine whether the user's eye is within that sub-space or within a threshold distance from that sub-space when imaged by the camera. If the eye is within the sub-space or within a threshold distance from the sub-space, the display system may interpret this result as meaning that the eye is within the display alignment volume. If the eye is outside the sub-space or outside the threshold distance from the target, the display system may interpret this result as meaning that the eye is outside the display alignment volume. If the display system determines that the eye is outside the display alignment volume, the display system may be configured to display and provide feedback to the user indicating that the display and the eye are not properly aligned for output, as discussed herein, and / or may be configured to take actions to reduce display degradation caused by misalignment. Examples of the housing alignment volume and the system alignment volume

[0277] FIG. 19A illustrates an exemplary housing alignment volume 1900. Light from the surrounding environment passes through the frame of the display or an opening in the housing and reaches the user. It should be understood that the frame or housing for the display may block some of the ambient light from reaching the user's eyes from some angles. As a result, similar to the display alignment volume, the housing alignment volume 1900 represents a volumetric space in which the user's eyes are positioned so as to receive light from the external environment and create a full field of view available through the housing or frame of the display. In some embodiments, the center of rotation of the user's eyes is preferably positioned within the housing alignment volume 1900 such that the eyes receive light at an angle corresponding to a full available field of view from the external environment. In some embodiments, when the center of rotation of the user's eyes is positioned within the housing alignment volume 1900, the user is able to see an acceptable portion of the world around them. Proper alignment of the user's eyes with the wearable system (e.g., by providing the center of rotation within the housing alignment volume 1900) can help reduce the likelihood that the user will be able to see obstacles in their path, can provide the user with left and right fields of view that overlap and promote binocular vision, and / or can provide the user with a more comfortable visual experience.

[0278] As shown in FIG. 19A, the housing alignment volume 1900 may be determined partially or entirely with reference to the housing opening 1510, which may be an opening (or lens) within the frame 230 of the wearable system as previously described in connection with FIG. 15A. The housing alignment volume 1900 may, by way of example, have a conical shape with a base defined by the shape and size of the housing opening 1510. The portion of the conical shape nearest the housing opening 1510 and the portion farthest from the housing opening 1510 may be truncated (e.g., excluded from the housing alignment volume 1900), which may serve to move the user's eye away from the display and the housing opening 1510 so that the user's eyelashes do not affect the display when properly aligned, and may also serve to reduce the effect of noise in determining the location of the user's eye, which could otherwise quickly align or misalign within a small volume in the "upper portion" of the conical alignment volume. In some embodiments, the display system may be configured to determine whether the user's eye (e.g., the center of rotation of the eye) is within the housing alignment volume 1900 and provide a misalignment notification if the user's eye is outside the alignment volume 1900.

[0279] FIG. 19B illustrates the display alignment volume 1302a (associated with the display surface 1202) of FIG. 13A superimposed on the housing alignment volume 1900 (associated with the housing opening 1510) of FIG. 1900. In at least some embodiments, it may be desirable for the center of rotation of the user's eye to be located within both of the display alignment volumes such as the housing alignment volume 1900 of FIG. 19A and the display alignment volume 1302a of FIG. 13A (or any of the other display alignment volumes discussed herein). When the user's eye is located within both the housing and the display alignment volumes, the user may be able to receive image information from the display device (with a full field of view provided by the display device), while also potentially being able to view a full field of view of the external environment provided by the housing.

[0280] FIG. 19C illustrates an example of the combined alignment volume 1902 where all points within the combined alignment volume 1902 are within both the housing and the display alignment volume (e.g., within the display alignment volume 1302a and the housing alignment volume 1900). As shown in FIGS. 19B and 19C, in at least some embodiments, the display alignment volume 1302a may generally be smaller than the housing alignment volume 1900 (e.g., the housing alignment volume 1900 may be smaller only at the corners of the display alignment volume 1302a). In such embodiments, the combined alignment volume 1902 may have a shape similar to a frustum with rounded corners, as illustrated in FIG. 19C. In some embodiments, the display system may be configured to determine whether the user's eye (e.g., the center of rotation of the eye) is within the housing alignment volume 1900, the display alignment volume 1204, or both the housing alignment volume 1900 and the display alignment volume 1204. If the user's eye is outside the particular volume being analyzed, the display system may be configured to provide a notification of the alignment misalignment when the user's eye is outside the alignment volume being analyzed (e.g., the housing alignment volume 1900, the display alignment volume 1204, or both the housing alignment volume 1900 and the display alignment volume 1204).

[0281] For example, in some embodiments, the combined overlapping alignment volume defined by the housing alignment volume 1900 and the display alignment volume 1204 may be analyzed to determine whether the user's eye is within this combined alignment volume. In some embodiments, this can be understood as part or a sub - space of the alignment or viewing volume of the housing, which may also be referred to as the outer housing. The display system may be configured to determine whether the user's eye is within this sub - space (or an acceptable threshold distance outside the sub - space). If the display system determines that the position of the eye exceeds the threshold distance outside the sub - space of the viewing volume of the outer housing of the display, feedback indicating that the display and the eye are not properly aligned may be provided to the user. (Computer vision for detecting objects in the surrounding environment)

[0282] As discussed above, the display system may be configured to detect objects or their properties in the environment surrounding the user. The detection may be performed using various techniques, including various environmental sensors (e.g., cameras, audio sensors, temperature sensors, etc.) as discussed herein.

[0283] In some embodiments, objects present in the environment may be detected using computer vision techniques. For example, as disclosed herein, a camera facing forward of the display system may be configured to image the surrounding environment, and the display system may be configured to perform image analysis on the image to determine the presence of objects in the surrounding environment. The display system may analyze the image obtained by the outward-facing imaging system to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. As another example, the display system may be configured to perform face and / or eye recognition to determine the presence and location of faces and / or human eyes within the user's field of view. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), Bundle adjustment, Adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), Histogram of Feature Points, various machine learning algorithms (e.g., Support Vector Machine, k-Nearest Neighbor algorithm, Naive Bayes, Neural Network (including convolutional or deep neural network), or other supervised / unsupervised models, etc.).

[0284] One or more of these computer vision techniques may also be used in combination with data obtained from other environmental sensors (e.g., microphones, etc.) to detect and determine various properties of the objects detected by the sensors.

[0285] As discussed herein, objects in the surrounding environment may be detected based on one or more criteria. When a display system uses a computer vision algorithm or data received from one or more sensor assemblies (which may or may not be part of the display system) to detect the presence or absence of criteria in the surrounding environment, the display system may then signal the presence of an object. (Machine learning)

[0286] Various machine learning algorithms may be used to learn to identify the presence of objects in the surrounding environment. Once trained, the machine learning algorithm may be stored by the display system. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., Apriori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models may be customized for individual datasets. For example, a wearable device may generate or store a base model. The base model is used as a starting point and may generate additional models specific to the data type (e.g., a particular user), the dataset (e.g., a set of additional images to be acquired), the conditional situation, or other variations. In some embodiments, the display system can be configured to generate a model for the analysis of aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0287] The criteria for detecting an object may include one or more threshold conditions. If the analysis of the data obtained by the environmental sensor indicates that the threshold conditions have been met, the display system may provide a signal indicating the detection of the presence of an object in the surrounding environment. The threshold conditions may involve quantitative and / or qualitative measurement values. For example, the threshold conditions may include a score or percentage associated with the likelihood of reflection and / or the presence of an object in the environment. The display system may compare the score calculated from the data of the environmental sensor with a threshold score. If the score is higher than the threshold level, the display system may detect the presence of reflection and / or an object. In some other embodiments, the display system may signal the presence of an object in the environment if the score is lower than the threshold. In some embodiments, the threshold conditions may be determined based on the emotional state of the user and / or the interaction of the user with the surrounding environment.

[0288] In some embodiments, the threshold conditions, machine learning algorithms, or computer vision algorithms may be specialized for a specific context. For example, in a diagnostic context, the computer vision algorithm may be specialized to detect a certain response to a stimulus. As another example, the display system may execute a face recognition algorithm and / or an event tracing algorithm as discussed herein to sense the user's reaction to a stimulus.

[0289] The processes, methods, and algorithms described in this specification and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, whereby they can be fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc. programmed with specific computer instructions. The code modules can be installed in dynamic link libraries that can be compiled and linked into an executable program, or written in a programming language that is interpreted. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0290] Furthermore, because the functional embodiments of the present disclosure are sufficiently mathematically, computationally, or technically complex, application-specific hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video can include many frames, each of which can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0291] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same, and / or equivalents. In some embodiments, the non-volatile computer-readable medium can be part of one or more of the local processing and data module (140), remote processing module (150), and remote data repository (160). The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wire / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0292] Any process, block, state, step, or functionality in a flowchart described herein and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the embodiments described herein is for purposes of illustration and should not be understood as requiring such separation in all implementations. It should be understood that the program components, methods, and systems described can generally be integrated together in a single computer product or packaged in multiple computer products. (Other Considerations)

[0293] The processes, methods, and algorithms described in this specification and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, the computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0294] Furthermore, because the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex, special-purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video or a video may include many frames, each frame may have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications within a commercially reasonable amount of time.

[0295] A code module or any type of data can be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagation signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0296] Any process, block, state, step, or functionality in a flowchart described herein and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. The various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed illustrative embodiments. Further, the separation of the various system components in the implementations described herein is for purposes of illustration and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0297] The present process, method, and system may be implemented in a network (or distributed) computing environment. The network environment may include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0298] The systems and methods of the present disclosure each have several innovative aspects, none of which alone participate in or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0299] Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, the various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable sub-combination. Furthermore, a feature may be described above as acting in a certain combination and may further be initially claimed as such, but one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination. No single feature or group of features is necessary or essential to every embodiment.

[0300] In particular, conditional statements used herein such as "can", "could", "might", "may", "e.g.", and equivalents, unless specifically stated otherwise or understood otherwise within the context in which they are used, generally are intended to convey that while one embodiment includes a certain feature, element, or step, another embodiment does not include them. Thus, such conditional statements generally are not intended to imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are to be included in or implemented in any particular embodiment, whether or not there is author input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more" or "at least one" unless otherwise defined.

[0301] As used herein, the phrase that refers to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as the phrase "at least one of X, Y, and Z" are generally understood in a context such that they are used to convey that an item, term, etc. can be at least one of X, Y, or Z, unless specifically described otherwise. Thus, such connective phrases are generally not intended to imply that an embodiment requires that at least one of each of X, at least one of Y, and at least one of Z be present respectively.

[0302] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not required for achieving the desired results, and that such operations may be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations need not be performed. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated into the exemplary methods and processes schematically illustrated. For example, one or more additional operations may be performed before, after, concurrently with, or during any of the illustrated operations. Additionally, operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described are generally integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results.

Claims

1. A method for evaluating the alignment of virtual image content from a head-mounted display system by a user's eyes, wherein the head-mounted display system comprises a display disposed on a frame supported on the user's head, the method comprising: determining the position of the eyes based on an image of the eyes obtained using one or more eye-tracking cameras attached to the frame; determining whether the position of the eyes is within a threshold volume, wherein the threshold volume is an imaginary volume surrounding the viewing volume of the head-mounted display system, the viewing volume being an imaginary volume in which the eyes should be positioned to receive 100% image light from the display, the threshold volume being larger than the viewing volume, and the walls of the threshold volume being a predetermined threshold distance away from the walls of the viewing volume; providing feedback to the user indicating that the display and the eyes are not properly aligned in response to determining that the position of the eyes is not within the threshold volume; A method comprising the above.

2. Further comprising obtaining a fit tolerance, the fit tolerance including information associated with a change in the alignment of the virtual image content caused by the dispersion of the user's eyes from a nominal position relative to the display system, the method according to claim 1.

3. Further comprising obtaining alignment data, obtaining the alignment data including determining a spatial relationship between the display system and the user's eyes, the method according to claim 1.

4. identifying an application running on the display system; determining the threshold volume based on the identified application; Further comprising the above, wherein the predetermined threshold distance is determined in advance based on the application, the method according to claim 1.

Images