Display system and method for determining registration between a display and a user's eyes

Abstract

A wearable device may include a head-mounted display (HMD) for rendering three-dimensional (3D) virtual objects that appear to be in the user's surrounding environment of the display. The relative position of the HMD and one or more eyes of the user may not be in a desired position to receive or register the image information output by the HMD. For example, the alignment of the HMD to the eyes varies for different users and may change over time (e.g., when a given user moves around or as the HMD slides or otherwise shifts). The wearable device may determine the relative position or alignment between the HMD and the user's eyes. Based on this relative position, the wearable device may determine whether it is properly fitted to the user, may provide feedback to the user regarding the quality of the fit, and may take measures to reduce or minimize the effects of any misalignment.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 644,321, filed on March 16, 2018, entitled "DISPLAY SYSTEMS AND METHODS FOR DETERMINING REGISTRATION BETWEEN A DISPLAY AND A USER'S EYES"; U.S. Provisional Patent Application No. 62 / 618,559, filed on January 17, 2018, entitled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS"; and U.S. Provisional Patent Application No. 62 / 702,849, filed on July 24, 2018, entitled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS". Each of the above - mentioned applications is hereby incorporated by reference in its entirety.

[0003] This application incorporates by reference in its entirety each of the following patent applications and publications: U.S. Application No. 14 / 555,585, filed November 27, 2014, published as U.S. Publication No. 2015 / 0205126 on July 23, 2015; U.S. Application No. 14 / 690,401, filed April 18, 2015, published as U.S. Publication No. 2015 / 0302652 on October 22, 2015; U.S. Application No. 14 / 212,961, filed March 14, 2014, now U.S. Patent No. 9,417,452, issued August 16, 2016; U.S. Application No. 14 / 331,218, filed July 14, 2014, published as U.S. Publication No. 2015 / 0309263 on October 29, 2015; U.S. Patent Publication No. 2016 / 0270656; U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015; U.S. Patent Publication No. 2015 / 0016777; U.S. Patent Application No. 15 / 274,823; U.S. Patent Application No. 15 / 296,869; U.S. Patent Application No. 15 / 717,747, filed September 27, 2017; U.S. Patent Application No. 15 / 497,726, filed April 26, 2017; U.S. Patent Publication No. 2017 / 0053165, published February 23, 2017; U.S. Patent Publication No. 2017 / 0053166, published February 23, 2017; U.S. Application No. 15 / 341,760, filed November 2, 2016, published as U.S. Publication No. 2017 / 0122725 on May 4, 2017; U.S. Application No. 15 / 341,822, filed November 2, 2016, published as U.S. Publication No. 2017 / 0124928 on May 4, 2017; U.S. Provisional Patent Application No. 62 / 618,559, filed January 17, 2018; and U.S. Provisional Patent Application No. 62 / 642,761, filed March 14, 2018. TECHNICAL FIELD

[0004] This disclosure relates to display systems including virtual reality and augmented reality display systems, and more particularly, to systems and methods for evaluating the fit of a display on a user. BACKGROUND ART

[0005] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, in which digital reproduced images or portions thereof are presented to a user in a manner that they appear to be real or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information that is opaque to other actual real-world visual inputs; augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as a visual enhancement to the real-world environment surrounding the user; mixed reality or "MR" is related to the fusion of the real world and the virtual world to create a new environment in which physical and virtual objects coexist and interact in real time. The human visual perception system has proven to be very complex, and it is challenging to produce VR, AR, or MR technologies that facilitate a comfortable, natural-feeling, and rich presentation of virtual image elements, especially other virtual or real image elements. The systems and methods disclosed herein address various challenges related to VR, AR, and MR technologies. SUMMARY OF THE INVENTION

[0006] Various examples of registering observations and responses in a mixed reality system are disclosed.

[0007] In some embodiments, a display system for projecting light into a user's eyes to display virtual image content is provided. The display system includes: a frame configured to be supported on a user's head; a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times; one or more eye-tracking cameras configured to image the user's eyes; and processing electronics in communication with the display and the one or more eye-tracking cameras. The processing electronics are configured to: determine the position of the eyes based on images of the eyes obtained by the one or more eye-tracking cameras; determine whether the position of the eyes is within a display registration volume of the head-mounted display system; and provide a notification based on determining whether the position of the eyes is within the display registration volume, wherein the notification at least indicates that the display and the eyes are not properly registered.

[0008] In some other embodiments, a display system is configured to project light into a user's eyes to display virtual image content. The display system includes: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times; one or more eye tracking cameras configured to image the user's eyes; and processing electronics communicatively coupled to the display and the one or more eye tracking cameras. The processing electronics are configured to: determine the position of the eyes based on an image of the eyes obtained by the one or more eye tracking cameras; determine whether the position of the eyes is outside a viewing volume of the head-mounted display system by more than a first threshold distance; and in response to determining that the position of the eyes is outside the viewing volume of the head-mounted display system by more than the first threshold distance, provide feedback to the user indicating that the display and the eyes are not properly registered for output.

[0009] In some other embodiments, a display system for projecting light into a user's eyes to display virtual image content is provided. The display system includes: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times; one or more eye tracking cameras configured to image the user's eyes; and processing electronics communicatively coupled to the display and the one or more eye tracking cameras. The processing electronics are configured to: determine whether light projected by the head-mounted display is properly registered by the user's eyes and, if the head-mounted display is not properly adjusted to fit the user to register light projected by the display system, provide feedback to the user.

[0010] In other embodiments, a method for evaluating registration of virtual image content from a head-mounted display system through a user's eyes is provided. The method includes: determining a first position of the eyes; determining whether the first position of the eyes is within a display registration volume of the head-mounted display system, where the display registration volume is a hypothetical volume associated with properly fitting the head-mounted display system relative to the user's eyes; and providing a notification based on determining whether the position of the eyes is within the display registration volume, where the notification at least indicates that the display and the eyes are not properly registered.

[0011] Additional examples of embodiments are listed below.

[0012] Example 1. A display system configured to project light into a user's eyes to display virtual image content, the display system including:

[0013] A frame configured to be supported on the user's head;

[0014] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0015] One or more eye tracking cameras configured to image the user's eyes; and

[0016] Processing electronics in communication with the display and the one or more eye tracking cameras, the processing electronics configured to:

[0017] Determine the position of the eyes based on images of the eyes obtained by the one or more eye tracking cameras;

[0018] Determine whether the position of the eyes is within a display registration volume of the head-mounted display system; and

[0019] Provide a notification based on determining whether the position of the eyes is within the display registration volume, wherein the notification at least indicates that the display and the eyes are not properly registered.

[0020] Example 2. The display system according to Example 1, wherein the processing electronics are further configured to: when determining that the position of the eyes is outside the display registration volume, provide feedback to the user that the head-mounted display is not properly adjusted to fit the user, wherein the feedback is the notification provided based on determining whether the position of the eyes is within the display registration volume.

[0021] Example 3. The display system according to Example 1, further comprising at least one interchangeable adapter removably mounted to the frame and configured to adjust the fit of the frame.

[0022] Example 4. The display system according to Example 3, wherein the interchangeable adapter includes an interchangeable nose bridge configured to adjust the fit of the frame between the frame and the user's nose.

[0023] Example 5. The display system according to Example 3 or 4, wherein the interchangeable adapter includes an interchangeable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

[0024] Example 6. The display system according to any one of Examples 3 to 5, wherein the interchangeable adapter includes an interchangeable rear pad configured to adjust the fit of the frame between the frame and the rear of the user's head.

[0025] Example 7. The display system according to any one of Examples 2 to 6, wherein the processing electronic device is further configured such that providing the notification includes: providing feedback to the user that the head-mounted display is not properly adjusted to fit the user, including providing suggestions to the user to replace a currently installed interchangeable adapter with another interchangeable adapter.

[0026] Example 8. The display system according to any one of Examples 1 to 7, further comprising: one or more light sources disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras using light from the one or more light sources to form an image of the eyes.

[0027] Example 9. The display system according to Example 8, wherein the one or more light sources include: at least two light sources disposed on the frame relative to the user's eyes to illuminate the user's eyes.

[0028] Example 10. The display system according to any one of Examples 8 to 9, wherein the one or more light sources include infrared light emitters.

[0029] Example 11. The display system according to any one of Examples 8 to 10, wherein the one or more light sources form one or more flashes on the eyes, and the processing electronic device is configured to determine the position of the cornea based on the one or more flashes.

[0030] Example 12. The display system according to any one of Examples 1 to 11, wherein the position of the eyes is the position of the center of rotation of the eyes.

[0031] Example 13. The display system according to any one of Examples 1 to 11, wherein the cornea has a corneal sphere associated therewith having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere.

[0032] Example 14. The display system according to Example 1, wherein the processing electronic device is configured to provide the notification by providing instructions to increase the brightness of a plurality of pixels of the display relative to other pixels of the display, wherein the plurality of pixels having increased brightness include pixels expected to experience perceived dimming under incorrect registration.

[0033] Example 15. A display system configured to project light onto a user's eyes to display virtual image content, the display system comprising:

[0034] A frame configured to be supported on the user's head;

[0035] A head-mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0036] One or more eye-tracking cameras configured to image the user's eyes; and

[0037] Processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics being configured to:

[0038] Determine the position of the eyes based on images of the eyes obtained by the one or more eye-tracking cameras;

[0039] Determine whether the position of the eyes is outside the viewing volume of the head-mounted display system by more than a first threshold distance; and

[0040] In response to determining that the position of the eyes is outside the viewing volume of the head-mounted display system by more than the first threshold distance, provide feedback to the user indicating that the display and the eyes are not properly registered for output.

[0041] Example 16. The display system according to Example 15, wherein the processing electronics is configured to determine whether the position of the eyes is outside the viewing volume by more than the first threshold distance by at least the following operations:

[0042] Determine whether the position of the eyes is less than a second threshold distance from the eyepiece; and

[0043] In response to determining that the position of the eyes is less than the second threshold distance from the head-mounted display system, provide feedback to the user indicating that the display and the eyes are not properly registered for output.

[0044] Example 17. The display system according to Example 15, wherein the processing electronics is configured to determine whether the position of the eyes is outside the viewing volume by more than the first threshold distance by at least the following operations:

[0045] Determine whether the position of the eyes is greater than a second threshold distance from the eyepiece; and

[0046] In response to determining that the position of the eyes is greater than the second threshold distance from the head-mounted display system, provide feedback to the user indicating that the display and the eyes are not properly registered for output.

[0047] Example 18. The display system according to Example 15, wherein the processing electronics is configured to determine whether the position of the eye is outside the viewing volume by at least the following operations: greater than the first threshold distance:

[0048] Determine whether the position of the eye is outside a subspace of the field of view of the eye tracking camera by more than a second threshold distance; and

[0049] In response to determining that the position of the eye is outside the subspace of the viewing volume of the eye tracking camera by more than the second threshold distance, provide feedback indicating that the display and the eye are not properly registered to the user for output.

[0050] Example 19. The display system according to Example 15, wherein the viewing volume of the head-mounted display is the volume through which light from each pixel of the virtual image content expected to be presented by the head-mounted display passes.

[0051] Example 20. A display system configured to project light onto a user's eyes to display virtual image content, the display system comprising:

[0052] A frame configured to be supported on the user's head;

[0053] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0054] One or more eye tracking cameras configured to image the user's eyes; and

[0055] Processing electronics in communication with the display and the one or more eye tracking cameras, the processing electronics configured to:

[0056] Determine whether the light projected by the head-mounted display is properly registered by the user's eyes; and

[0057] If the head-mounted display is not properly adjusted to fit the user to register the light projected by the display system, provide feedback to the user.

[0058] Example 21. The display system according to Example 20, further comprising at least one interchangeable adapter removably mounted to the frame and configured to adjust the fit of the frame.

[0059] Example 22. The display system according to Example 21, wherein the interchangeable adapter includes an interchangeable nose bridge configured to adjust the fit of the frame between the frame and the user's nose bridge.

[0060] Example 23. The display system according to Example 20 or 22, wherein the interchangeable adapter includes an interchangeable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

[0061] Example 24. The display system according to any one of Examples 20 to 23, wherein the interchangeable adapter includes an interchangeable rear pad configured to adjust the fit of the frame between the frame and the rear of the user's head.

[0062] Example 25. The display system according to any one of Examples 20 to 24, wherein the processing electronics are further configured such that providing feedback to the user that the head-mounted display is not properly adjusted to fit the user includes providing the user with a suggestion to replace the currently installed interchangeable adapter with another interchangeable adapter.

[0063] Example 26. A method for evaluating the registration of virtual image content from a head-mounted display system through a user's eyes, the method comprising:

[0064] Determining a first position of the eye;

[0065] Determining whether the first position of the eye is within a display registration volume of the head-mounted display system, wherein the display registration volume is a hypothetical volume associated with the proper alignment of the head-mounted display system relative to the user's eyes; and

[0066] Providing a notification based on determining whether the position of the eye is within the display registration volume, wherein the notification at least indicates that the display and the eye are not properly registered.

[0067] Example 27. The method according to Example 26, wherein the head-mounted display system includes an eye tracking camera, and wherein determining the first position of the eye includes imaging the user's eye using the eye tracking camera.

[0068] Example 28. The method according to Example 27, wherein the first position of the eye is the position of the center of rotation of the eye, and further comprising: calculating the center of rotation of the eye based on the imaging of the eye by the eye tracking camera.

[0069] Example 29. The method according to Example 26, wherein the head-mounted display system is configured to project light into the eye to display virtual image content in the user's field of view, and further includes: displaying an indication that the wearable system has been properly adapted.

[0070] Example 30. The method according to any one of Examples 26 to 29, further includes: automatically tracking, over time, the center of rotation of the eye by the head-mounted display system and notifying the user when the center of rotation of the eye moves outside the registration display volume.

[0071] Example 31. The method according to Example 26 or 29, further includes:

[0072] determining a second position of the eye;

[0073] determining that the second position of the eye is within the display registration volume; and

[0074] in response to determining that the second position of the eye is within the display registration volume, providing additional feedback to the user indicating that the wearable system is properly adapted to the user.

[0075] Example 32. The method according to any one of Examples 26 to 31, wherein when the user's eye is not within the display registration volume, at least some pixels of the head-mounted display system are dimmed or invisible to the user.

[0076] Example 33. The method according to any one of Examples 26 to 32, further includes: changing the field of view of the head-mounted display system when the position of the eye is outside the display registration volume,

[0077] wherein the head-mounted display system includes at least one display having a first field of view when the position of the eye is within the display registration volume; wherein when the position of the eye is outside the display registration volume, the display has a second field of view, and wherein the second field of view is less than the first field of view.

[0078] Example 34. The method according to Example 33, wherein providing the notification includes providing feedback to the user within the second field of view.

[0079] Example 35. The method according to any one of Examples 26 to 34, wherein the wearable system includes at least one interchangeable adapter, and the method further includes:

[0080] providing a notification to the user indicating that the wearable system is not properly adapted to the user,

[0081] Wherein, the notification includes a recommendation or instruction to the user to replace a currently installed interchangeable adapter with an alternative interchangeable adapter.

[0082] Example 36. The method according to Example 35, wherein the interchangeable adapter includes at least one adapter selected from the group consisting of a nose bridge pad, a forehead pad, and a rear pad, and the rear pad is located between the wearable system and the rear of the user's head.

[0083] Example 37. The method according to Example 36, wherein the wearable system includes at least one interchangeable nose bridge pad, and the method further includes: determining that a display of the head-mounted system is too low relative to the eyes, and wherein providing the notification to the user includes prompting the user to install a larger nose bridge pad.

[0084] Example 38. The method according to any one of Examples 26 to 37, further comprising:

[0085] identifying a plurality of pixels of a display of the head-mounted display system that the user is expected to perceive as being dimmed due to the first position of the eyes being outside the display registration volume; and

[0086] increasing the brightness of the plurality of pixels of the display relative to other pixels in the display to mitigate the expected dimming.

[0087] Example 39. A display system configured to project light into a user's eyes to display virtual image content, the display system comprising:

[0088] a frame configured to be supported on the user's head;

[0089] a head-mounted display disposed on the frame, the display being configured to project light into the user's eyes to display virtual image content rendered by a virtual rendering camera with different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0090] one or more eye-tracking cameras configured to image the user's eyes; and

[0091] processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics being configured to:

[0092] determine a distance from the eyes to the display based on an image of the eyes obtained by the one or more eye-tracking cameras, and

[0093] Adjust the focal length of the virtual rendering camera based on the determined distance.

[0094] Example 40. A display system configured to project light onto a user's eyes to display virtual image content, the display system comprising:

[0095] A frame configured to be supported on the user's head;

[0096] A head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0097] One or more eye-tracking cameras configured to image the user's eyes; and

[0098] Processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to:

[0099] Determine the position of the eyes relative to the display based on an image of the eyes obtained by the one or more eye-tracking cameras;

[0100] Determine the amount of pixels of virtual image content that is expected to be perceived as dimmed by the user based on the position of the eyes relative to the display; and

[0101] Control the operation of the display based on the determined amount of pixels.

[0102] Example 41. The display system according to Example 40, wherein the processing electronics is configured to determine the amount of pixels of virtual image content that the user is not expected to perceive as dimmed based on the position of the eyes relative to the display.

[0103] Example 42. The display system according to Example 40 or 41, wherein the processing electronics is configured to control the operation of the display by:

[0104] Increasing the brightness of the pixels of the virtual image content that the user is expected to perceive as dimmed based on the position of the eyes relative to the display.

[0105] Example 43. The display system according to any one of Examples 40-42, wherein the amount of pixels of the virtual image content comprises a percentage of pixels.

[0106] Example 44. The display system according to any one of Examples 40 - 43, wherein the processing electronics is configured to compare the amount of pixels of the virtual image content with one or more thresholds, and in response to determining that the amount of pixels of the virtual image content exceeds one or more thresholds, provide feedback indicating that the display and the eye are not properly registered to the user for output.

[0107] Example 45. A display system configured to project light into a user's eye to display virtual image content, the display system comprising:

[0108] A frame configured to be supported on the user's head;

[0109] A head-mounted display disposed on the frame, the display being configured to project light into the user's eye to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times;

[0110] One or more eye-tracking cameras configured to image the user's eye; and

[0111] Processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics being configured to:

[0112] Define a registration volume relative to the display based on one or more parameters;

[0113] Determine the position of the eye based on an image of the eye obtained by the one or more eye-tracking cameras;

[0114] Determine whether the position of the eye is within the registration volume of the head-mounted display system; and

[0115] Control the operation of the display based on determining whether the position of the eye is within the display registration volume.

[0116] Example 46. The display system according to Example 45, wherein the one or more parameters include the type of application running on the display system.

[0117] Example 47. The display system according to any one of Examples 45 - 46, wherein the one or more parameters include one or more physical parameters of the head-mounted display.

[0118] Example 48. The display system according to any one of Examples 45-47, wherein the one or more physical parameters of the head-mounted display include one or more of the following: the display field of view, the display surface size, the shape of the display, the housing of the display, the amount of optical power of the light imparted by the display to represent virtual image content.

[0119] Example 49. The display system according to any one of Examples 45-48, wherein the processing electronics are configured to control the operation of the display by presenting to the user virtual image content that at least indicates that the display and the eyes are not correctly registered.

[0120] Example 50. The display system according to Example 15, wherein the processing electronics are configured to determine whether the position of the eyes is outside the viewing volume by at least the following operations by more than the first threshold distance:

[0121] Determine whether the position of the eyes is outside a subspace of the viewing volume of the housing of the head-mounted display by more than a second threshold distance; and

[0122] In response to determining that the position of the eyes is outside the subspace of the viewing volume of the housing of the head-mounted display by more than the second threshold distance, provide feedback to the user indicating that the display and the eyes are not correctly registered for output.

[0123] Example 52. The display system according to Example 15, wherein the processing electronics are further configured to:

[0124] Identify an application running on the display system; and

[0125] Determine the first threshold distance based on the identified application.

[0126] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, the drawings, and the claims. Neither this summary nor the following detailed description is intended to limit or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 An illustration depicting a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person.

[0128] Figure 2 Schematically shows an example of a wearable system.

[0129] Figure 3 Schematically shows example components of a wearable system.

[0130] Figure 4 Schematically shows an example of a waveguide stack of a wearable device for outputting image information to a user.

[0131] Figure 5 Schematically shows an example of an eye.

[0132] Figure 6 Is a schematic diagram of a wearable system including an eye tracking system.

[0133] Figure 7A Is a block diagram of a wearable system that may include an eye tracking system.

[0134] Figure 7B Is a block diagram of a rendering controller in a wearable system.

[0135] Figure 7C Is a block diagram of a registration viewer in a head-mounted display system.

[0136] Figure 8A Is a schematic diagram of an eye showing the corneal globe of the eye.

[0137] Figure 8B Shows an example corneal flash detected by an eye tracking camera.

[0138] Figure 8C - 8E Shows an example phase of locating the corneal center of a user by an eye tracking module in a wearable system.

[0139] Figure 9A - 9C Shows an example normalization of the coordinate system of an eye tracking image.

[0140] Figure 9D - 9G Shows an example phase of locating the pupil center of a user by an eye tracking module in a wearable system.

[0141] Figure 10 Shows an example of an eye including the optical axis and visual axis of the eye and the center of rotation of the eye.

[0142] Figure 11 Is a process flow diagram of an example of a method for using eye tracking in rendering content and providing feedback on registration in a wearable device.

[0143] Figure 12A and Figure 12B Shows the nominal position of a display element relative to a user's eye and shows a coordinate system for describing the position of the display element and the user's eye relative to each other.

[0144] Figure 13A and 13BShows the nominal positioning and positioning tolerances of a display element relative to a user's eyes in a head-mounted display system.

[0145] Figure 13C And Figure 13D Shows a display registration volume and a user's eyes viewing content from the display.

[0146] Figure 14 Shows an example of the perceived dimming of a display for various positions of a user's eyes relative to the display.

[0147] Figure 15A And 15B Is an exploded perspective view of a head-mounted display system having interchangeable parts such as a rear pad, a forehead pad, and a nose bridge pad to adjust the fit of the head-mounted display of the display system for different users.

[0148] Figure 16 Is a process flow diagram of an example of a method for observing registration and providing feedback on registration with a head-mounted display system.

[0149] Figure 17A - 17H Shows a view of a light field projected by a display and how the intersection of the light fields can partially define a display registration volume.

[0150] Figure 18 Shows a top view of a light field projected by a display and how the intersection of the light fields can partially define a display registration volume.

[0151] Figure 19A Shows a registration volume derived from a display housing of a head-mounted display system.

[0152] Figure 19B Shows a superimposed registration volume of a display housing and a display of a head-mounted display system.

[0153] Figure 19C Shows the total registration volume derived from the Figure 19B superimposed registration volume.

[0154] In all the figures, reference numerals may be reused to indicate the correspondence between reference elements. The figures are provided to illustrate the example embodiments described herein and are not intended to limit the scope of the disclosure. Detailed Description

[0155] The display portion of the display system may include a head-mounted display (HMD) that may display three-dimensional (3D) virtual objects such that the objects appear to be located within the user's surrounding environment. As a result, the user may perceive the 3D virtual objects in a manner similar to real-world objects.

[0156] The HMD can display an image by outputting spatially modulated light to the user, where the light corresponds to a virtual object. The spatially modulated light can contain image information and can be referred to as image light. In order to be perceived by the user, the image light travels from the HMD to the user's eyes, propagates through the pupil, and falls on the retina of the eyes. It should be understood that if all or part of the image light for an image does not enter the pupil of the eye and / or does not fall on the retina of the eye, the viewer will not see the image or the quality of the image may deteriorate. As used herein, registration involves the relative positioning of the display and the user's eyes. For example, when the user's eyes and the display are positioned relative to each other such that the desired amount of image light enters the eyes, it can be said that the display has been correctly registered. A registration viewer (e.g., a computer program) in the display device can be programmed to monitor whether the display is correctly registered or positioned such that the eyes receive image light from the display.

[0157] In order to correctly display content to the user, for example, by positioning the user's eyes to receive image light, the user's eyes may need to be located within a specific region or spatial volume relative to the HMD. This volume can be referred to as the display registration volume. If the user's eyes are outside the display registration volume, the display quality may deteriorate (e.g., there may be dimming and / or the content being displayed may not reach the user's eyes). Various factors can combine to determine the position of the user's eyes relative to the HMD and thus determine whether the user's eyes are located within the desired display registration volume. As an example, anatomical variations between users can mean that a head-mounted display is adapted for some users in such a way that their eyes are placed outside the display registration volume. As another example, the HMD may not be securely fixed to the user's head and can move on the user's head over time, especially when the user is moving around. As a specific example, the HMD may slip down the user's nose or tilt relative to the line between the user's eyes (interpupillary axis), and as a result, due to the movement of the display relative to the user's eyes, the HMD may not provide the desired virtual content (e.g., without some undesired deterioration).

[0158] The various systems and techniques described herein are at least partially directed to solving problems related to the correct registration of a display to allow a viewer to view image content on demand. In some embodiments, a head-mounted display system may be configured to determine the position of a user's eyes. The display system may then determine whether the position of the eyes is within a display registration volume of the head-mounted display system. Determining the position of the eyes may include determining the position of a representative pointer volume associated with the eyes, e.g., the center of rotation of the eyes. Determining whether the position of the eyes is within the display registration volume may include determining whether the center of rotation of the eyes is within the display registration volume. As discussed herein, an inward-facing imaging system configured to image the eyes may be used to determine the center of rotation of the eyes. Additionally, in some embodiments, the display registration volume is a hypothetical volume associated with the correct alignment of the head-mounted display system relative to the user's eyes. For example, the display registration volume may be a volume defined by the projection of the surface of the head-mounted display system that outputs image light. More specifically, the display registration volume may be a three-dimensional geometry that tapers from a base to a top. The shape of the base of the display registration volume may be at least partially defined by the geometry of the display, and the depth of the display registration volume (i.e., the distance along the z-axis from the base to the top) may be at least partially defined by the field of view (FOV) of the display. For example, an arcuate or circular display (e.g., the shape of the region on the surface from which image light is output to the viewer) may produce a conical display registration volume, while a polygonal display may produce a pyramidal display registration volume. As an additional example, a display with a larger FOV may produce a display registration volume with a smaller depth than a display with a smaller FOV. In some embodiments, the display registration volume may have an overall shape of a frustum of a cone or a frustum of a pyramid. For example, the display registration volume may have an overall shape of a frustum (e.g., a frustum of a pyramid such as a rectangular pyramid).

[0159] In some embodiments, an inward-facing imaging system of a head-mounted display system may acquire an image of a user's face, including their eyes. The inward-facing imaging system may be an eye-tracking system, which may be mounted on a frame of the head-mounted display. The head-mounted display system may analyze the image to determine the relative position of the user's eyes and the HMD, and whether the position of each of the user's eyes is within the display registration volume of that eye. Based on this information, the head-mounted display system may notify the user to adjust the fit of the HMD. For example, the notification may inform the user that the device has slipped and needs adjustment or suggest an adjustment for the HMD. In at least some embodiments, the head-mounted display system may take steps to mitigate any display degradation due to misalignment of the HMD with respect to the user, such as by increasing the brightness or light output to the user in areas that would otherwise be dimmed due to misalignment or by moving virtual content. Thus, such embodiments of the HMD may assist the user in properly fitting the HMD and mitigate problems caused by improper fitting of the HMD, such as when the HMD slides, moves, or tilts relative to the user's head. In some embodiments, it will be appreciated that the display system may be configured to notify the user of misalignment and also take steps to mitigate the display degradation caused by the misalignment. In some other embodiments, the display system may not provide a notification to the user; instead, the notification may be merely instructions or flags within the display system that trigger the display system to take action to mitigate the image degradation caused by the misalignment.

[0160] Advantageously, the registration analysis may be automatically performed using the image acquired from the inward-facing imaging system and information about the display registration volume stored or accessible by the display system. As a result, the fit of the HMD may be corrected when the HMD is first used and optionally also during the continuous use of the HMD to ensure a high level of image quality when using the head-mounted display system.

[0161] Accordingly, various embodiments of systems and methods for observing the registration of a head-mounted display system and taking action in response to the observed registration are provided herein.

[0162] Example of 3D display of a wearable system

[0163] Reference will now be made to the accompanying drawings, in which like reference numerals always refer to like parts. Unless otherwise noted, the drawings are schematic and not necessarily drawn to scale.

[0164] A wearable system (also referred to herein as a head-mounted display system or an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images can be still images, frames of a video, or a video, or a combination, etc. At least a part of the wearable system can be implemented on a wearable device, which can present a VR, AR, or MR environment for user interaction alone or in combination. The wearable device can be used interchangeably as an AR device (ARD). Additionally, for the purposes of this disclosure, the term “AR” can be used interchangeably with the term “MR”.

[0165] Figure 1 An illustration depicting a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1 it, an MR scene 100 is depicted, in which a user of MR technology sees a real-world park-like setting 110, characterized by people, trees, buildings, and a concrete platform 120 in the background. In addition to these items, the user of MR technology also perceives that he “sees” a robotic statue 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140, seemingly an avatar of a bumblebee, even though these elements do not exist in the real world.

[0166] To make a 3D display produce a realistic sense of depth, more specifically, a simulated sense of surface depth, it may be desirable for each point in the field of view of the display to generate an adaptive response corresponding to its virtual depth. If the adaptive response of a display point does not match the virtual depth of that point (as determined by binocular depth convergence cues and stereopsis), the human eye may encounter an adaptation conflict, resulting in imaging instability, harmful eye fatigue, headaches, and an almost complete lack of surface depth in the absence of adaptation information.

[0167] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different renditions of a scene or an object), and may be separately focused by the viewer's eyes, thereby helping to provide depth cues to the user based on the adaptation required by the eyes, focusing different image features for scenes located on different depth planes or based on defocusing different image features observed on different depth planes. As discussed elsewhere herein, such depth cues provide a credible perception of depth.

[0168] Figure 2FIG. 0 shows an example of a wearable system 200 that can be configured to provide AR / VR / MR scenarios. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems that support the functions of the display 220. The display 220 may be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. The display 220 may be located in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. Since the display 220 can be configured to be worn on the head of the user 210, it may also be referred to as a head-mounted display (HMD), and the wearable system 200 that includes the display 220 may also be referred to as a head-mounted display system.

[0169] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, may be positioned adjacent to the user's other ear canal to provide stereo / spatial sound control). The display 220 may include an audio sensor (e.g., a microphone) 232 for detecting an audio stream from the environment and capturing ambient sound. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo reception. Stereo reception may be used to determine the location of a sound source. The wearable system 200 may perform sound or speech recognition on the audio stream.

[0170] The wearable system 200 may include an outward-facing imaging system 464 ( Figure 4 as shown) that observes the world in the environment around the user. The wearable system 200 may also include an inward-facing imaging system 462 ( Figure 4 as shown) that can be used to track the eye movements of the user. The inward-facing imaging system may track the movement of one eye or both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may be in electrical communication with a processing module 260 or 270 that can process the image information acquired by the inward-facing imaging system to determine, for example, the pupil diameter or orientation, eye movement, or eye pose of the eyes of the user 210. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. The images acquired by the cameras may be used to determine the pupil size or eye pose of each eye separately, thereby allowing the image information to be presented to each eye to dynamically adapt to that eye.

[0171] As an example, the wearable system 200 may use the outward-facing imaging system 464 or the inward-facing imaging system 462 to acquire an image of the user's pose. The image may be a still image, a frame of a video, or a video.

[0172] The display 220 can be operatively coupled to the local data processing module 260, for example, via a wired lead or a wireless connection 250, and the local data processing module 260 can be installed in various configurations, for example, fixedly attached to the frame 230, fixedly attached to a helmet or a hat worn by the user, embedded in headphones, or otherwise detachably attached to the user 210 (for example, in a backpack configuration or a belt-coupled configuration).

[0173] The local processing and data module 260 can include a hardware processor and a digital memory (e.g., a non-volatile memory such as a flash memory), both of which can be used to assist in processing, caching, and storing data. The data can include: a) data captured from sensors that can be operatively coupled to the frame 230 or otherwise attached to the user 210, such as an image capture device (e.g., a camera in an inward-facing imaging system or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a radio device, or a gyroscope; or b) data obtained or processed using the remote processing module 270 or the remote data repository 280, which may be passed to the display 220 after such processing or retrieval. The local processing and data module 260 can be operatively coupled to the remote processing module 270 or the remote data repository 280 via a communication link 262 or 264 (e.g., via a wired or wireless communication link), such that these remote modules can be used as resources for the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 can be operatively coupled to each other.

[0174] In some embodiments, the remote processing module 270 can include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 can be a digital data storage facility that can be used via the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use of the remote modules.

[0175] Example components of a wearable system

[0176] Figure 3 An example component of the wearable system is schematically shown. Figure 3 A wearable system 200 is shown, which may include a display 220 and a frame 230. An exploded view 202 schematically shows the various components of the wearable system 200. In certain embodiments, Figure 3One or more of the components shown can be part of the display 220. The various components, alone or in combination, can collect various data (e.g., audio or visual data) associated with the user of the wearable system 200 or the user's environment. It should be understood that depending on the application for which the wearable system is used, other embodiments can have more or fewer components. Nevertheless, Figure 3 provides the basic idea of some of the various components and the types of data that can be collected, analyzed, and stored by the wearable system.

[0177] Figure 3 An example wearable system 200 that can include a display 220 is shown. The display 220 can include a display lens 226 that can be mounted to the user's head or a housing or frame 230, and the display lens 226 corresponds to the frame 230. The display lens 226 can include one or more transparent lenses that are positioned by the housing 230 in front of the user's eyes 302, 304 and can be configured to bounce the projected light 338 into the eyes 302, 304 and facilitate beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 can be bent or focused to be consistent with the desired focal length of the projected light. As shown, two wide-field-of-view machine vision cameras 316 (also referred to as world cameras) can be coupled to the housing 230 to image the environment around the user. These cameras 316 can be dual-capture visible / non-visible (e.g., infrared) light cameras. The cameras 316 can be Figure 4 part of the outward-facing imaging system 464 shown. The images acquired by the world cameras 316 can be processed by the pose processor 336. For example, the pose processor 336 can implement one or more object recognizers 708 (e.g., as shown in FIG. 7) to identify the pose of the user or another person in the user's environment or to identify physical objects in the user's environment.

[0178] Continuing to refer to Figure 3 , a pair of light projector modules (e.g., a scanned laser shaped wavefront (e.g., for depth) light projector module) with display mirrors and optics is shown, which is configured to project light 338 into the eyes 302, 304. The depicted view also shows two miniature infrared cameras 324 paired with an infrared light source 326 (e.g., a light-emitting diode "LED"), which is configured to be able to track the user's eyes 302, 304 to support rendering and user input. The cameras 324 can be Figure 4Part of the internal-facing imaging system 462 shown. The wearable system 200 is further characterized by a sensor assembly 339, which may include X, Y, and Z axis accelerometer capabilities, a magnetic compass, and X, Y, and Z axis gyroscope capabilities, preferably providing data at a relatively high frequency (e.g., 200 Hz). The sensor assembly 339 may be part of the IMU described in reference Figure 2 A. The depicted system 200 may also include a head pose processor 336, such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or ARM processor (Advanced RISC Machine), which may be configured to calculate the real-time or near real-time user head pose based on the wide field of view image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as Figure 2 Part of the local processing and data module 260 shown in A.

[0179] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure the distance between an object in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., lidar), an ultrasonic depth sensor, or a depth sensing camera. In certain embodiments, where the camera 316 has depth sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0180] A processor 332 is also shown, which is configured to perform digital or analog processing to derive a pose based on gyroscope, compass, or accelerometer data from the sensor assembly 339. The processor 332 may be Figure 2 Part of the local processing and data module 260 shown in. As Figure 3 Shown, the wearable system 200 may also include a positioning system such as GPS 337 (Global Positioning System) to assist in pose and location analysis. Additionally, the GPS may further provide remote (e.g., cloud-based) information about the user's environment. This information may be used to identify objects or information in the user's environment.

[0181] The wearable system may combine data obtained by the GPS 337 and a remote computing system (e.g., the remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As an example, the wearable system may determine the user's location based on GPS data and retrieve a world map including virtual objects associated with the user's location (e.g., by communicating with the remote processing module 270). As another example, the wearable system 200 may use the world camera 316, which may be Figure 4(a portion of the externally - facing imaging system 464 shown) to monitor the environment. Based on the images acquired by the world camera 316, the wearable system 200 can detect objects in the environment (e.g., by using one or more object recognizers 708 shown in FIG. 7). The wearable system can further use the data acquired by the GPS 337 to interpret the role.

[0182] The wearable system 200 may also include a rendering engine 334, which may be configured to provide rendering information local to the user to facilitate the operation of the scanner and imaging into the user's eyes for the user to view the world. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled (e.g., via a wired or wireless link) to other components of the wearable system 200. For example, the rendering engine 334 may be coupled to the eye camera 324 via the communication link 274, and may be coupled to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanned laser arrangement in a manner similar to a retinal scanning display) via the communication link 272. The rendering engine 334 may also communicate with other processing units via the links 276 and 294, respectively, such as the sensor pose processor 332 and the image pose processor 336.

[0183] The camera 324 (e.g., a micro - infrared camera) can be used to track the eye pose to support rendering and user input. Some example eye poses may include where the user is looking, or the depth at which he or she is focusing (which can be estimated by eye vergence). The GPS 337, gyroscopes, compass, and accelerometers 339 can be used to provide rough or quick pose estimates. One or more of the cameras 316 can acquire images and poses, which, together with data from associated cloud computing resources, can be used to map the local environment and share the user's view with other users.

[0184] Figure 3 The example components shown are for illustrative purposes only. For ease of illustration and description, multiple sensors and other functional modules are shown together. Some embodiments may include only one or a subset of these sensors or modules. Additionally, the locations of these components are not limited to Figure 3The positions shown. Some components may be mounted to or housed within other components, e.g., a belt-mounted component, a hand-held component, or a helmet component. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be placed in a belt pack and configured to communicate with other components of the wearable system via wireless communication (e.g., ultra-wideband, Wi-Fi, Bluetooth, etc.) or via wired communication. The depicted housing 230 is preferably user-head-wearable and wearable. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0185] Regarding the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may be used to measure the position to which the centers of the user's eyes are geometrically converged, which generally coincides with the focal position or "depth of focus" of the eyes. The three-dimensional surface of all points to which the eyes converge may be referred to as the "horopter". The focal length may have a finite amount of depth or may vary infinitely. Light projected from the convergence distance appears to be focused onto the subject's eyes 302, 304, while light before or after the convergence distance becomes blurred. Examples of the wearable device and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, the entire content of which is incorporated herein by reference.

[0186] The human visual system is complex, and providing a realistic sense of depth is challenging. Due to the combination of vergence and accommodation, a viewer of an object can perceive the object as three-dimensional. The vergence movement of the two eyes relative to each other (i.e., the rolling movement of the pupils towards or away from each other to converge the lines of sight of the eyes to fix on an object) is closely associated with the focusing (or "accommodation") of the eye lenses. Under normal circumstances, in a relationship called the "accommodation-vergence reflex", changing the focus of the eye lenses or accommodating the eyes to change the focus from one object to another object at a different distance will automatically result in a matching change in vergence at the same distance. Similarly, under normal circumstances, a change in vergence will trigger a matching change in accommodation. A display system that provides a better match between accommodation and vergence can form a more realistic and comfortable three-dimensional image simulation.

[0187] The human eye can correctly distinguish spatially coherent light with a beam diameter of less than about 0.7 millimeters, regardless of where the eye is focused. Thus, to create the illusion of an appropriate depth of focus, the camera 324 can track eye convergence, and the rendering engine 334 and projection subsystem 318 can be utilized to render all objects in focus on or near the horopter, and all other objects out of focus to varying degrees (e.g., using deliberately created blur). Preferably, the system 220 renders to the user at a frame rate of about 60 frames per second or higher. As described above, preferably, the camera 324 can be used for eye tracking, and the software can be configured to pick up not only the convergence geometry, but also focus position cues for use as user input. Preferably, such a display system is configured with brightness and contrast suitable for use during the day or at night.

[0188] In some embodiments, the display system preferably has a visual object alignment latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute, which is believed to be approximately the limit of the human eye without being limited by theory. The display system 220 can be integrated with a positioning system that can involve GPS elements, optical tracking, compasses, accelerometers, or other data sources to assist in determining position and orientation; the positioning information can be used to facilitate accurate rendering when the user views the relevant world (e.g., such information will help the glasses understand their position relative to the real world).

[0189] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the accommodation of the user's eyes. In some embodiments, unlike existing 3D display methods that force the user to focus on where the image is projected, the wearable system is configured to automatically vary the focus of the projected virtual content to allow for more comfortable viewing of the one or more images presented to the user. For example, if the user's current focus is 1m, the image can be projected to be consistent with the user's focus. If the user moves the focus to 3m, the projected image is made consistent with the new focus. Thus, the wearable system 200 of some embodiments does not force the user to achieve a predetermined focus, but rather allows the user's eyes to function in a more natural manner.

[0190] Such a wearable system 200 can eliminate or reduce the occurrence of eye strain, headaches, and other physiological symptoms commonly observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images at varying focal lengths through one or more variable focus elements (VFE). In one or more embodiments, 3D perception can be achieved through a multi-plane focusing system that projects images on a fixed focal plane away from the user. Other embodiments employ variable plane focus, where the focal plane moves back and forth in the z direction to be consistent with the user's current focus state.

[0191] In multi-plane focusing systems and variable plane focusing systems, the wearable system 200 can use eye tracking to determine the vergence of the user's eyes, determine the user's current focus, and project a virtual image at the determined focus. In other embodiments, the wearable system 200 includes a light modulator that projects a light beam across the retina in a raster pattern at different foci in a variable manner through a fiber optic scanner or other light generating source. Thus, as further described in U.S. Patent Publication No. 2016 / 0270656, the entire content of which is incorporated herein by reference, the ability of the display of the wearable system 200 to project images at varying focal lengths not only allows the user to easily adapt to viewing 3D objects, but can also be used to compensate for user eye abnormalities. In some other embodiments, a spatial light modulator can project an image to the user through various optical components. For example, as further described below, the spatial light modulator can project an image onto one or more waveguides, which then send the image to the user.

[0192] Waveguide stack component

[0193] Figure 4 An example of a waveguide stack for outputting image information to a user is shown. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480, which can be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 can correspond to Figure 2 the wearable system 200, Figure 4 Some parts of the wearable system 200 are schematically shown in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into Figure 2 the display 220.

[0194] Continuing to refer to Figure 4 , the waveguide assembly 480 can also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 can be lenses. In other embodiments, the features 458, 456, 454, 452 can not be lenses. Instead, they can simply be spacers (e.g., cladding or structures for forming an air gap).

[0195] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to send image information to the eye with various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be used to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, and each waveguide may be configured to distribute incident light across each corresponding waveguide for output to the eye 410. Light leaves the output surface of image injection devices 420, 422, 424, 426, 428 and is injected into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single light beam (e.g., a collimated beam) may be injected into each waveguide to output the entire field of view of a cloned collimated beam that is directed at the eye 410 at a specific angle (and amount of divergence) corresponding to the depth plane associated with a particular waveguide.

[0196] In some embodiments, image injection devices 420, 422, 424, 426, 428 are discrete displays that each generate image information for injection into a corresponding waveguide 440b, 438b, 436b, 434b, 432b. In some other embodiments, image injection devices 420, 422, 424, 426, 428 are the output ports of a single multiplexed display that can convey image information to each of image injection devices 420, 422, 424, 426, 428, for example, via one or more light pipes (e.g., fiber optic cables).

[0197] Controller 460 controls the operation of the stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that schedules and provides image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 460 may be part of processing module 260 or 270 (shown in Figure 2 ).

[0198] Waveguides 440b, 438b, 436b, 434b, 432b can be configured to propagate light within each respective waveguide by total internal reflection (TIR). Each of waveguides 440b, 438b, 436b, 434b, 432b can be planar or have another shape (e.g., curved), and can have major top and bottom surfaces and edges extending between those major top and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b can each include light extraction optical elements 440a, 438a, 436a, 434a, 432a, which are configured to extract light from the waveguides by redirecting light that propagates within the respective waveguides so as to output image information to the eye 410. The extracted light can also be referred to as out-coupled light, and the light extraction optical elements can also be referred to as out-coupling optical elements. The extracted light beam is output by the waveguide at the location where the light propagating in the waveguide impinges on the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although shown for ease of description and clarity of drawing as being disposed on the bottom major surface of waveguides 440b, 438b, 436b, 434b, 432b, in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be disposed on the top major surface or the bottom major surface, or can be disposed directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed in a material layer that is attached to a transparent substrate to form waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, waveguides 440b, 438b, 436b, 434b, 432b can be a single-piece material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be formed on or within the surface of the piece of material.

[0199] Continue to refer to Figure 4, as described herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to transmit collimated light injected into such waveguide 432b to the eye 410. The collimated light may represent an optically infinite focal plane. The next up waveguide 434b may be configured to emit collimated light that passes through a first lens 452 (e.g., a negative lens) before the collimated light reaches the eye 410. The first lens 452 may be configured to produce a slightly convex wavefront curvature such that the eye / brain interprets the light from the next up waveguide 434b as coming from a first focal plane that is closer in towards the eye 410 from optically infinite distance. Similarly, the third up waveguide 436b passes its output light through the first lens 452 and a second lens 454 before reaching the eye 410. The combined optical power of the first lens 452 and the second lens 454 may be configured to produce another increment of wavefront curvature such that the eye / brain interprets the light from the third waveguide 436b as coming from a second focal plane that is closer in towards the person from optically infinite distance than the light from the next up waveguide 434b.

[0200] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all the lenses between it and the eye for representing the aggregate diopter of the focal plane closest to the person. When viewing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be provided at the top of the stack to compensate for the aggregate diopter of the lens stack 458, 456, 454, 452 below. (The compensating lens layer 430 and the stacked waveguide assembly 480 as a whole may be configured such that light from the world 470 is transmitted to the eye 410 at substantially the same level of divergence (or collimation) as the light had when initially received by the stacked waveguide assembly 480). Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. The light extraction optics of the waveguides and the focusing aspects of the lenses can both be static (e.g., not dynamic or electrically actuated). In some alternative embodiments, one or both of them may be dynamic when using electrically actuated features.

[0201] Continuing to refer to Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to both redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides having different associated depth planes can have different configurations of light extraction optical elements that output light with different amounts of divergence according to the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume or surface features that can be configured to output light at a particular angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015, which is hereby incorporated by reference in its entirety.

[0202] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features that form a diffraction pattern or a diffractive optical element (also referred to herein as a “DOE”). Preferably, the DOE has a relatively low diffraction efficiency such that with each interaction with the DOE, only a portion of the light of the light beam is deflected towards the eye 410 while the remainder continues to travel through the waveguide by total internal reflection. The light carrying the image information can thus be split into multiple related outgoing light beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of outgoing emission towards the eye 304 for that particular collimated light beam bouncing within the waveguide.

[0203] In some embodiments, one or more DOEs can be switched between an “on” state in which they actively diffract and an “off” state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which microdrops contain a diffraction pattern in a host medium, and the refractive index of the microdrops can be switched to be substantially matched to the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdrops can be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0204] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eyes. The depth of field may be inversely proportional to the pupil size of the viewer. As a result, as the pupil size of the viewer's eyes decreases, the depth of field increases, such that planes that were not distinguishable due to their position being outside the depth of focus of the eyes may become distinguishable and appear more in focus as the pupil size decreases and the depth of field correspondingly increases. Similarly, as the pupil size decreases, the number of spaced-apart depth planes used to present different images to the viewer may be reduced. For example, at one pupil size, without adjusting the eye's accommodation from one depth plane to another, the viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane. However, these two depth planes may be simultaneously in sufficient focus for a user at another pupil size without changing the accommodation.

[0205] In some embodiments, the display system may change the number of waveguides receiving image information based on a determination of the pupil size or orientation, or based on receiving an electrical signal indicating a particular pupil size or orientation. For example, if the user's eyes cannot distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE of the waveguide can be switched between an on and an off state, the DOE may be switched to the off state when the waveguide does receive image information.

[0206] In some embodiments, it may be desirable for the outgoing light beam to satisfy the condition of having a diameter smaller than the diameter of the viewer's eyes. However, given the variability of the viewer's pupil size, satisfying this condition may be challenging. In some embodiments, this condition is satisfied over a wide range of pupil sizes by changing the size of the outgoing light beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the outgoing light beam may also decrease. In some embodiments, a variable aperture may be used to change the size of the outgoing light beam.

[0207] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes referred to as the FOV camera. The FOV of the world camera may be the same as or different from the FOV of the viewer 210, which encompasses the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some cases, the FOV of the world camera may be larger than the FOV of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of regard (FOR). The FOR may include 4π steradians of solid angle around the wearable system 400, since the wearer can move his body, head, or eyes to perceive substantially any direction in space. In other contexts, the movement of the wearer may be more restricted, and thus the wearer's FOR may subtend a smaller solid angle. Images obtained from the outward-facing imaging system 464 may be used to track gestures made by the user (e.g., hand or finger gestures), detect objects in the world 470 in front of the user, and so on.

[0208] The wearable system 400 may include an audio sensor 232 (e.g., a microphone) to capture ambient sound. As described above, in some embodiments, one or more other audio sensors may be positioned to provide stereo reception useful for determining the location of the voice source. As another example, the audio sensor 232 may include a directional microphone, which may also provide useful directional information about where the audio source is located. The wearable system 400 may use information from the outward-facing imaging system 464 and the audio sensor 230 when localizing the voice source, or determining the active speaker, etc. at a particular time. For example, the wearable system 400 may use voice recognition alone or in combination with a reflected image of the speaker (e.g., seen in a mirror) to determine the identity of the speaker. As another example, the wearable system 400 may determine the location of the speaker in the environment based on the sound obtained from the directional microphone. The wearable system 400 may use a speech recognition algorithm to parse the sound from the location of the speaker to determine the content of the speech and use speech recognition techniques to determine the identity of the speaker (e.g., name or other demographic information).

[0209] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements, such as eye movements and facial movements. The inward-facing imaging system 466 can be used to capture images of the eyes 410 to determine the size and / or orientation of the pupils of the eyes 304. The inward-facing imaging system 466 can be used to acquire images for determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, each eye can utilize at least one camera to independently determine the pupil size or eye pose of each eye, thereby allowing the image information presented for each eye to be dynamically tailored to that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 is determined (e.g., only a single camera is used per pair of eyes), and it is assumed that the pupil diameter or orientation is similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 can be analyzed to determine the user's eye pose or mood, and the wearable system 400 can use this eye pose or mood to determine which audio or visual content should be presented to the user. The wearable system 400 can also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head pose (e.g., head position or head orientation).

[0210] The wearable system 400 may include a user input device 466 through which the user can input commands to the controller 460 to interact with the wearable system 400. For example, the user input device 466 can include a touchpad, touch screen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, D-pad, stick, haptic device, totem (e.g., acting as a virtual user input device), and so on. A multi-DOF controller can sense user input in some or all of the possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translational movement can be referred to as 3DOF, while a multi-DOF controller that supports both translational and rotational movement can be referred to as 6DOF. In some cases, the user can use a finger (e.g., a thumb) to press or slide on a touch-sensitive input device to provide input to the wearable system 400 (e.g., provide user input to the user interface provided by the wearable system 400). The user input device 466 can be held by the user's hand during the use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 either wired or wirelessly.

[0211] Other components of a wearable system

[0212] In many embodiments, in addition to or as an alternative to the components of the wearable system described above, the wearable system may include other components. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component is operable to provide a haptic sensation to the user. For example, when touching virtual content (e.g., virtual objects, virtual tools, other virtual constructs), the haptic device or component may provide a haptic sensation of pressure or texture. The haptic sensation may replicate the feeling of a physical object represented by the virtual object, or may replicate the feeling of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the user may wear the haptic device or component (e.g., a glove wearable by the user). In some embodiments, the haptic device or component may be held by the user.

[0213] For example, the wearable system may include one or more physical objects manipulable by the user to allow input or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, a table surface. In certain embodiments, the totem may not actually have any physical input structures (e.g., buttons, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface so that it appears to the user to be on one or more surfaces of the totem. For example, the wearable system may cause an image of a computer keyboard and touchpad to be rendered as if residing on one or more surfaces of the totem. For example, the wearable system may cause a virtual computer keyboard and virtual touchpad to be rendered as if on the surface of a thin aluminum rectangular plate used as a totem. The rectangular plate itself may not have any physical buttons, touchpads, or sensors. However, the wearable system may detect user operations or interactions or touches on the rectangular plate as selections or inputs made via the virtual keyboard or virtual touchpad. The user input device 466 ( Figure 4 shown in) may be an example of a totem, and it may include a touchpad, a touch-sensitive pad, a trigger, a joystick, a trackball, a rocker, or a virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or other physical input devices. The user may use the totem alone or in combination with a gesture to interact with the wearable system or other users.

[0214] Examples of haptic devices and totems for use with wearable devices, HMDS, and display systems in accordance with the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference.

[0215] Example of an eye image

[0216] Figure 5An image of an eye 500 is shown, which has an eyelid 504, a sclera 508 (the "white" of the eye), an iris 512, and a pupil 516. A curve 516a shows the pupil boundary between the pupil 516 and the iris 512, and a curve 512a shows the edge boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is shown in a natural resting position (e.g., where the user's face and gaze are both oriented towards a distant object straight ahead of the user). The natural resting position of the eye 500 can be indicated by a natural resting direction 520, which is the direction orthogonal to the surface of the eye 500 when in the natural resting position (e.g., directly outside the plane of the eye 500 shown in Figure 5 and, in this example, located at the center of the pupil 516).

[0217] When the eye 500 moves to look at a different object, the eye position will change relative to the natural resting direction 520. The current eye position can be determined with reference to the eye position direction 520, which is the direction orthogonal to the surface of the eye (and located at the center of the pupil 516) but oriented towards the object that the eye is currently pointing at. Referring to Figure 5 the example coordinate system shown, the position of the eye 500 can be expressed as two angular parameters, which indicate the azimuthal deflection and the zenithal deflection of the eye position direction 524 of the eye, both relative to the natural resting direction 520 of the eye. For illustrative purposes, these angular parameters can be represented as θ (the azimuthal deflection determined from a fiducial azimuth) and φ (the zenithal deflection, sometimes also referred to as the polar deflection). In some embodiments, an angular roll of the eye about the eye position direction 524 can be included in the determination of the eye position, and the angular roll can be included in the following analysis. In other embodiments, other techniques for determining the eye position can use, for example, pitch, yaw, and optionally roll systems.

[0218] Any suitable process can be used to obtain an eye image from a video, e.g., using a video processing algorithm that can extract an image from one or more consecutive frames. A variety of eye tracking techniques can be used to determine the position of the eye based on the eye image. For example, the eye position can be determined by considering the lensing effect of the cornea on the provided light source. Any suitable eye tracking technique can be used to determine the eye position in the eyelid shape estimation techniques described herein.

[0219] Example of an eye tracking system

[0220] Figure 6 FIG. shows a schematic diagram of a wearable or head-mounted display system 600 including an eye tracking system. In at least some embodiments, the head-mounted display system 600 may include components located in the head-mounted unit 602 and components located in the non-head-mounted unit 604. The non-head-mounted unit 604 may be, for example, a belt-mounted component, a hand-held component, a component in a backpack, a remote component, etc. Incorporating some components of the head-mounted display system 600 into the non-head-mounted unit 604 can help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some embodiments, some or all of the functions described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted 604 may be provided by one or more components included elsewhere in the head-mounted display system 600. For example, some or all of the functions described below associated with the CPU 612 of the head-mounted unit 602 may be provided by the CPU 616 of the non-head-mounted unit 604, and vice versa. In some examples, some or all such functions may be provided by a peripheral device of the head-mounted display system 600. Additionally, in some embodiments, some or all such functions may be provided by one or more cloud computing devices or other remote computing devices in a manner similar to that referenced above Figure 2 and described.

[0221] As Figure 6 shown, the head-mounted display system 600 may include an eye tracking system that includes a camera 324 that captures an image of the user's eye 610. If desired, the eye tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs"). The light sources 326a and 326b may produce flashes (i.e., that are reflected by the user's eye and appear in the image of the eye captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, and thus, the positions of the flashes within the image captured by the camera 324 can be used to track the user's eye (as will be discussed in more detail below in connection with FIGS. 7-11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In other embodiments, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources (e.g., light sources 326a and 326b) and one or more cameras 324 associated with each of the user's eyes 610.

[0222] The eye tracking module 614 can receive images from the eye tracking camera 324 and analyze the images to extract various pieces of information. As an example, the eye tracking module 614 can detect the user's eye pose, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and relative to the head-mounted unit 602), the direction in which one or both of the user's eyes 610 are focused, the user's vergence depth (i.e., the depth from the user at which the user is focusing), the position of the user's pupils, the position of the user's corneas and cornea spheres, the center of rotation of each of the user's eyes, and the center of perspective of each of the user's eyes. The eye tracking module 614 can use the techniques described below in connection with FIGS. 7-11 to extract such information. As Figure 6 shown, the eye tracking module 614 can be a software module implemented using the CPU 612 in the head-mounted unit 602.

[0223] Data from the eye tracking module 614 can be provided to other components in the wearable system. For example, such data can be sent to components in the non-head-mounted unit 604, such as the CPU 616 that includes software modules for the light field rendering controller 618 and the registration viewer 620, and the registration viewer 620 can be configured to evaluate whether the display of the head-mounted display system 600 is correctly registered with the user's eyes.

[0224] The rendering controller 618 can use the information from the eye tracking module 614 to adjust the images displayed to the user by the rendering engine 622 (e.g., a rendering engine that can be a software module in the GPU 621 and that provides images to the display 220). As an example, the rendering controller 618 can adjust the images displayed to the user based on the center of rotation or the center of perspective of the user. In particular, the rendering controller 618 can use the information about the center of perspective of the user to simulate a rendering camera (i.e., simulate collecting images from the user's perspective), and can adjust the images displayed to the user based on the simulated rendering camera.

[0225] A "rendering camera", sometimes also referred to as a "pinhole perspective camera" (or simply "perspective camera") or a "virtual pinhole camera" (or simply "virtual camera"), is an analog camera used in rendering virtual image content that may be from an object database in a virtual world. The objects may have positions and orientations relative to the user or wearer and possibly relative to real objects in the environment surrounding the user or wearer. In other words, the rendering camera may represent a perspective view within the rendering space from which the user or wearer will view the 3D virtual content (e.g., virtual objects) of the rendering space. The rendering camera may be managed by a rendering engine to render a virtual image based on a database of virtual objects to be presented to the eyes. The virtual image may be rendered as if it were captured from the perspective of the user or wearer. For example, the virtual image may be rendered as if it were captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, skew / distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation components and rotation components relative to the virtual world). The virtual image is captured from the perspective of such a camera having the position and orientation of the rendering camera (e.g., the extrinsic parameters of the rendering camera). Thus, the system may define and / or adjust the intrinsic and extrinsic rendering camera parameters. For example, the system may define a specific set of extrinsic rendering camera parameters such that the virtual image can be rendered as if it were captured from the perspective of a camera having a specific position relative to the eyes of the user or wearer, to provide an image that appears to be from the perspective of the user or wearer. The system may then dynamically adjust the extrinsic rendering camera parameters on-the-fly in order to maintain registration with a specific position. Similarly, the intrinsic rendering camera parameters may be defined and dynamically adjusted over time. In some embodiments, the image is rendered as if it were captured from the perspective of a camera having an aperture (e.g., a pinhole) at a specific position (e.g., the center of perspective or the center of rotation or elsewhere) relative to the eyes of the user or wearer.

[0226] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye, and another rendering camera for the user's right eye, as the user's eyes are physically separated from each other and thus are always located at different positions. Accordingly, in at least some implementations, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye may be presented to the user through the left eyepiece of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the perspective of the rendering camera associated with the user's right eye may be presented to the user through the right eyepiece of such a head-mounted display. More details regarding creating, adjusting, and using rendering cameras during the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION", which is hereby expressly incorporated by reference in its entirety for all purposes.

[0227] In some examples, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 622, etc.) may determine the position and orientation of the rendering camera within the rendering space based on the position and orientation of the user's head and eyes (e.g., determined based on head pose and eye tracking data, respectively). That is, system 600 may effectively map the position and orientation of the user's head and eyes to a specific position and angular position within the 3D virtual environment, place and orient the rendering camera at the specific position and angular position within the 3D virtual environment, and render virtual content for the user as if captured by the rendering camera. More details regarding the real-world to virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE", which is hereby expressly incorporated by reference in its entirety for all purposes. As an example, the rendering controller 618 may adjust the depth of the displayed image by selecting which depth plane (or planes) to utilize to display the image at any given time. In some implementations, such depth plane switching may be performed by adjusting one or more intrinsic rendering camera parameters.

[0228] The registration viewer 620 can use information from the eye tracking module 614 to identify whether the head-mounted unit 602 is correctly positioned on the user's head. As an example, the eye tracking module 614 can provide eye position information, such as the position of the center of rotation of the user's eyes, which indicates the three-dimensional position of the user's eyes relative to the camera 324 and the head-mounted unit 602, and the eye tracking module 614 can use this position information to determine whether the display 220 is correctly aligned in the user's field of view, or whether the head-mounted unit 602 (or headset) has slipped or become misaligned with the user's eyes. As an example, the registration viewer 620 can determine whether the head-mounted unit 602 has slipped down the user's nose bridge, thereby moving the display 220 away from and below the user's eyes (which may be undesirable); whether the head-mounted unit 602 has moved up the user's nose bridge, thereby moving the display 220 closer to and above the user's eyes; whether the head-mounted unit 602 has moved left or right relative to the user's nose bridge, whether the head-mounted unit 602 has been raised above the user's nose bridge, or whether the head-mounted unit 602 has moved away from the desired position or range of positions in these or other ways. Generally, the registration viewer 620 can determine whether the head-mounted unit 602, and in particular the display 220, is correctly positioned in front of the user's eyes. In other words, the registration viewer 620 can determine whether the left display in the display system 220 is correctly aligned with the user's left eye, and whether the right display in the display system 220 is correctly aligned with the user's right eye. The registration viewer 620 can determine whether the head-mounted unit 602 is correctly positioned by determining whether the head-mounted unit 602 is positioned and oriented within a desired range of positions and / or orientations relative to the user's eyes.

[0229] In at least some embodiments, the registration viewer 620 can generate user feedback in the form of an alert, message, or other content. Such feedback can be provided to the user to inform the user of any misalignment of the head-mounted unit 602, as well as optional feedback on how to correct the misalignment (such as a suggestion to adjust the head-mounted unit 602 in a particular way).

[0230] Example registration viewing and feedback techniques that can be used by the registration viewer 620 are described in U.S. Patent Application No. 15 / 717,747 (Attorney Docket No. MLEAP.052A2), filed on September 27, 2017, the entire disclosure of which is incorporated herein by reference.

[0231] Example of an eye tracking module

[0232] In Figure 7A a detailed block diagram of an example eye tracking module 614 is shown. As Figure 7AAs shown, the eye tracking module 614 can include various different sub-modules, can provide various different outputs, and can utilize various available data to track a user's eyes. As an example, the eye tracking module 614 can utilize available data including eye tracking extrinsics and intrinsics, such as the geometric arrangement of the eye tracking camera 324 relative to the light source 326 and the head-mounted unit 602; a hypothesized eye size 704, such as the typical distance between the corneal curvature center of the user and the average rotation center of the user's eye being approximately 4.7 mm, or the typical distance between the rotation center of the user and the perspective center; and per-user calibration data 706, such as the inter-pupillary distance for a particular user. Additional examples of extrinsic parameters, intrinsic parameters, and other information that the eye tracking module 614 can employ are described in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), the entire text of which is incorporated herein by reference.

[0233] The image preprocessing module 710 can receive images from an eye camera (such as the eye camera 324) and can perform one or more preprocessing (i.e., conditioning) operations on the received images. As an example, the image preprocessing module 710 can apply Gaussian blur to the images, can downsample the images to a lower resolution, can apply an unsharp mask, can apply an edge sharpening algorithm, or can apply other suitable filters to assist in the later detection, localization, and marking of glints, pupils, or other features in the images from the eye camera 324. The image preprocessing module 710 can apply a low-pass filter or a morphological filter (such as an opening filter), which can remove high-frequency noise such as from the pupil boundary 516a (see Figure 5 ), thereby removing noise that may impede pupil and glint determination. The image preprocessing module 710 can output the preprocessed images to the pupil identification module 712 and the glint detection and marking module 714.

[0234] The pupil identification module 712 may receive the pre - processed images from the image pre - processing module 710 and may identify the regions in those images that include the user's pupils. In some embodiments, the pupil identification module 712 may determine the coordinates of the position of the user's pupil, or the center or centroid coordinates in the eye - tracking images from the camera 324. In at least some embodiments, the pupil identification module 712 may identify the contours (e.g., the contour of the pupil - iris boundary) in the eye - tracking images, identify the contour moments (i.e., the centroid), apply starburst pupil detection and / or Canny edge detection algorithms, reject outliers based on intensity values, identify sub - pixel boundary points, correct eye - camera distortion (i.e., the distortion in the images captured by the eye - camera 324), apply the Random Sample Consensus (RANSAC) iterative algorithm to fit an ellipse to the boundary in the eye - tracking image, apply a tracking filter to the image, and identify the sub - pixel image coordinates of the user's pupil centroid. The pupil identification module 712 may output pupil identification data to the flash detection and marking module 714, and the pupil identification data may indicate which regions of the pre - processed image module 712 are identified as showing the user's pupils. The pupil identification module 712 may provide the 2D coordinates of the user's pupil (i.e., the 2D coordinates of the centroid of the user's pupil) within each eye - tracking image to the flash detection module 714. In at least some embodiments, the pupil identification module 712 may also provide the same kind of pupil identification data to the coordinate system normalization module 718.

[0235] Pupil detection techniques that the pupil identification module 712 may utilize are described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017, the entire contents of each of which are incorporated herein by reference in their entirety.

[0236] The flash detection and marking module 714 can receive the pre - processed image from module 710 and the pupil identification data from module 712. The flash detection module 714 can use this data to detect and / or identify flashes (i.e., light reflected from the user's eye from the light source 326) within the region of the pre - processed image showing the user's pupil. As an example, the flash detection module 714 can search for bright regions (sometimes referred to herein as "blobs" or local intensity maxima) within the eye - tracking image that are near the user's pupil. In at least some embodiments, the flash detection module 714 can re - scale (e.g., magnify) the pupil ellipse to include additional flashes. The flash detection module 714 can filter flashes by size and / or intensity. The flash detection module 714 can also determine the 2D position of each flash within the eye - tracking image. In at least some examples, the flash detection module 714 can determine the 2D position of the flash relative to the user's pupil, which can also be referred to as the pupil - flash vector. The flash detection and marking module 714 can mark the flashes and output the pre - processed image with the marked flashes to the 3D corneal center estimation module 716. The flash detection and marking module 714 can also transfer data, such as the pre - processed image from module 710 and the pupil identification data from module 712.

[0237] Pupil and flash detection performed by modules such as modules 712 and 714 can use any suitable technique. As an example, edge detection can be applied to the eye image to identify flashes and pupils. Edge detection can be applied through various edge detectors, edge - detection algorithms, or filters. For example, the Canny edge detector can be applied to the image to detect edges such as those in the lines of the image. Edges can include points corresponding to local maximum derivatives located along the line. For example, the Canny edge detector can be used to locate the pupil boundary 516a (see Figure 5 ). Given the position of the pupil, various image - processing techniques can be used to detect the "pose" of the pupil 116. Determining the eye pose of an eye image can also be referred to as detecting the eye pose of the eye image. The pose can also be referred to as the gaze, pointing direction, or orientation of the eye. For example, the pupil may be looking left at an object, and the pose of the pupil can be classified as a left - pose. Other methods can be used to detect the position of the pupil or flash. For example, concentric rings can be placed in the eye image using the Canny edge detector. As another example, integral - differential operators can be used to find the pupil or limbus boundary of the iris. For example, the Daugman integral - differential operator, Hough transform, or other iris - segmentation techniques can be used to return a curve estimating the boundary of the pupil or iris.

[0238] The 3D corneal center estimation module 716 can receive a pre - processed image including detected flash data and pupil identification data from modules 710, 712, and 714. The 3D corneal center estimation module 716 can use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 can estimate the corneal curvature center of the eye or the 3D position of the user's corneal sphere, i.e., the center of a virtual sphere having a surface portion that generally co - extends with the user's cornea. The 3D corneal center estimation module 716 can provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. More details of the operation of the 3D corneal center estimation module 716 are provided herein in connection with Figure 8A - 8E Provided. Techniques for estimating the position of eye features such as the cornea or corneal sphere are discussed in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), the entire content of which is incorporated herein by reference, and can be used by the 3D corneal center estimation module 716 and other modules of the wearable system of the present disclosure.

[0239] The coordinate system normalization module 718 can optionally (as indicated by its dashed outline) be included in the eye tracking module 614. The coordinate system normalization module 718 can receive the estimated 3D coordinates indicating the center of the user's cornea (and / or the center of the user's corneal sphere) from the 3D corneal center estimation module 716 and can also receive data from other modules. The coordinate system normalization module 718 can normalize the eye camera coordinate system, which can help compensate for slippage of the wearable device (e.g., slippage of the head - mounted component from its normal rest position on the user's head, which can be identified by the registration viewer 620). The coordinate system normalization module 718 can rotate the coordinate system to align the z - axis of the coordinate system (i.e., the vergence depth axis) with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716), and can translate the camera center (i.e., the origin of the coordinate system) to a predetermined distance (such as 30 millimeters) from the corneal center (i.e., the module 718 can magnify or reduce the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). Through this normalization process, the eye tracking module 614 can establish a consistent orientation and distance in the eye tracking data, relatively independently of changes in the positioning of the head - mounted earpiece on the user's head. The coordinate system normalization module 718 can provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and the pre - processed eye tracking image to the 3D pupil center locator module 720. More details of the operation of the coordinate system normalization module 718 are provided in connection with Figure 9A - 9C Provided herein.

[0240] The 3D pupil center locator module 720 can receive data in a normalized or non-normalized coordinate system, including the 3D coordinates of the center of the user's cornea (and / or corneal globe), pupil position data, and pre-processed eye tracking images. The 3D pupil center locator module 720 can analyze such data to determine the 3D coordinates of the center of the user's pupil in a normalized or non-normalized eye camera coordinate system. The 3D pupil center locator module 720 can determine the position of the user's pupil in three dimensions based on: the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), the assumed eye size 704 (e.g., the size of the corneal globe of a typical user and the typical distance from the corneal center to the pupil center), and the optical properties of the eye (e.g., the refractive index of the cornea (relative to the refractive index of air) or any combination of these. More details of the operation of the 3D pupil center locator module 720 are provided herein in connection with Figure 9D - 9G The techniques for estimating the position of eye features such as the pupil that can be utilized by the 3D pupil center locator module 720 and other modules in the wearable system of the present disclosure are discussed in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), the entire content of which is incorporated herein by reference.

[0241] The optical axis determination module 722 can receive data from modules 716 and 720 that indicates the 3D coordinates of the center of the user's cornea and the user's pupil. Based on such data, the optical axis determination module 722 can identify the vector from the position of the corneal center (i.e., from the center of the corneal globe) to the center of the user's pupil, which can define the optical axis of the user's eye. As an example, the optical axis determination module 722 can provide an output specifying the user's optical axis to modules 724, 728, 730, and 732.

[0242] The center of rotation (CoR) estimation module 724 may receive data from module 722, the data including parameters of the optical axis of the user's eyes (i.e., data indicating the direction of the optical axis in a coordinate system having a known relationship with the head-mounted unit 602). The CoR estimation module 724 may estimate the center of rotation of the user's eyes (i.e., the point about which the user's eyes rotate when the user's eyes rotate left, right, up, and / or down). Although the eyes may not rotate perfectly about a singular point, it may be sufficient to assume a singular point. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eyes by moving a specific distance along the optical axis (identified by module 722) from the center of the pupil (identified by module 720) or the center of curvature of the cornea (identified by module 716) towards the retina. The specific distance may be a hypothesized eye size 704. As an example, the specific distance between the center of curvature of the cornea and the CoR may be approximately 4.7 mm. This distance may be changed for a particular user based on any relevant data, including the user's age, gender, vision prescription, other relevant characteristics, etc.).

[0243] In at least some embodiments, the CoR estimation module 724 may refine its estimate of the center of rotation of each of the user's eyes over time. For example, over time, the user will eventually rotate the eyes (look elsewhere, look at something closer, look at something farther away, or look left, right, up, or down at some point), causing the optical axis of each of the user's eyes to shift. The CoR estimation module 724 may then analyze the two (or more) optical axes identified by module 722 and locate the 3D intersection point of those optical axes. The CoR estimation module 724 may then determine that the center of rotation is located at that 3D intersection point. Such techniques may provide an estimate of the center of rotation whose accuracy improves over time. Various techniques may be employed to improve the accuracy of the CoR estimation module 724 and the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 may estimate the CoR by finding the average intersection point of the optical axes determined over time for various different eye poses. As an additional example, module 724 may filter or average the CoR positions estimated over time, may calculate a moving average of the CoR positions estimated over time, and / or may apply a Kalman filter and the known dynamics of the eyes and the eye tracking system to estimate the CoR position over time. As a specific example, module 724 may calculate a weighted average of the determined optical axis intersection point and a hypothesized CoR position (e.g., 4.7 mm from the center of curvature of the eye's cornea) such that the determined CoR may slowly drift over time from the hypothesized CoR position (i.e., 4.7 mm behind the center of curvature of the eye's cornea) to a slightly different position within the user's eye as user eye tracking data is obtained, thereby enabling per-user refinement of the CoR position.

[0244] The interpupillary distance (IPD) estimation module 726 can receive data from the CoR estimation module 724, which indicates the estimated 3D positions of the centers of rotation of the user's left and right eyes. The IPD estimation module 726 can then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, when the user is fixating on optical infinity (i.e., the optical axes of the user's eyes are substantially parallel to each other), the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye can be approximately equal to the distance between the centers of the user's pupils, which is the typical definition of the interpupillary distance (IPD). The user's IPD can be used by various components and modules in the wearable system. For example, the user's IPD can be provided to the registration viewer 620 and used to evaluate the alignment of the wearable device with the user's eyes (e.g., whether the left and right display lenses are correctly spaced according to the user's IPD). As another example, the user's IPD can be provided to the vergence depth estimation module 728 and used to determine the user's vergence depth. The module 726 can employ various techniques, such as those discussed in conjunction with the CoR estimation module 724, to improve the accuracy of the estimated IPD. As an example, the IPD estimation module 724 can apply filtering, averaging over time, weighted averaging including hypothesized IPD distances, Kalman filters, etc., as part of estimating the user's IPD in an accurate manner.

[0245] The vergence depth estimation module 728 can receive data from various modules and sub-modules in the eye tracking module 614 (as described in conjunction with Figure 7AAs shown). Specifically, the vergence depth estimation module 728 may employ the estimated 3D positions of the data pupil centers (e.g., provided by module 720 as described above), one or more determined optical axis parameters (e.g., provided by module 722 as described above), the estimated 3D position of the center of rotation (e.g., provided by module 724 as described above), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., provided by module 726 as described above), and / or one or more determined optical axis and / or visual axis parameters (e.g., provided by module 722 and / or module 730 described below). The vergence depth estimation module 728 may detect or otherwise obtain a measure of the user's vergence depth, which may be the distance from the user at which the user's eyes are in focus. For example, when the user is looking at an object three feet in front of them, the vergence depth of the user's left and right eyes is three feet; and when the user is looking at a distant landscape (i.e., the optical axes of the user's eyes are substantially parallel to each other such that the distance between the centers of the user's pupils may be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the vergence depth of the user's left and right eyes is infinite. In some embodiments, the vergence depth estimation module 728 may utilize data indicating the estimated centers of the user's pupils (e.g., as provided by module 720) to determine the 3D distance between the estimated centers of the user's pupils. The vergence depth estimation module 728 may obtain a measure of the vergence depth (e.g., as indicated by module 726 as described above) by comparing this determined 3D distance between the pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation). In addition to the 3D distance between the pupil centers and the estimated IPD, the vergence depth estimation module 728 may utilize known, assumed, estimated, and / or determined geometries to calculate the vergence depth. As an example, module 728 may combine the 3D distance between the pupil centers, the estimated IPD, and the 3D CoR position in a triangulation calculation to estimate (i.e., determine) the user's vergence depth. In fact, evaluating such a determined 3D distance between the pupil centers based on the estimated IPD may be used to indicate the measure of the user's current focus depth relative to optical infinity. In some examples, the vergence depth estimation module 728 may simply receive or access data indicating the estimated 3D distance between the estimated centers of the user's pupils in order to obtain such a measure of the vergence depth. In some embodiments, the vergence depth estimation module 728 may estimate the vergence depth by comparing the user's left and right optical axes. Specifically, the vergence depth estimation module 728 may estimate the vergence depth by locating the distance from the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes intersect on a plane such as a horizontal plane).By setting the zero depth to the depth at which the user's left and right optical axes are separated by the user's IPD, the vergence depth estimation module 728 can utilize the user's IPD in this calculation. In at least some embodiments, the vergence depth estimation module 728 can determine the vergence depth by triangulating the eye tracking data together with known or derived spatial relationships.

[0246] In some embodiments, the vergence depth estimation module 728 can estimate the user's vergence depth based on the intersection of the user's visual axes (rather than their optical axes), which can provide a more accurate indication of the distance at which the user is focused. In at least some embodiments, the eye tracking module 614 can include an optical axis to visual axis mapping module 730. As discussed in further detail in Figure 10 the user's optical axis and visual axis typically do not align. The visual axis is the axis along which a person looks, while the optical axis is defined by the center of the person's lens and pupil and can pass through the center of the person's retina. In particular, the user's visual axis is typically defined by the position of the user's fovea, which may be offset from the center of the user's retina, resulting in the optical axis and visual axis being different. In at least some of these embodiments, the eye tracking module 614 can include an optical axis to visual axis mapping module 730. The optical axis to visual axis mapping module 730 can correct the difference between the user's optical axis and visual axis and provide information about the user's visual axis to other components in the wearable system (e.g., the vergence depth estimation module 728 and the light field rendering controller 618). In some examples, the module 730 can use a hypothesized eye size 704, including a typical offset of approximately 5.2° inward (nasally, towards the user's nose) between the optical axis and the visual axis. In other words, the module 730 can shift the user's left optical axis (nasally) 5.2° to the right towards the nose and shift the user's right optical axis (nasally) 5.2° to the left towards the nose in order to estimate the directions of the user's left and right optical axes. In other examples, the module 730 can utilize per-user calibration data 706 when mapping the optical axis (e.g., as indicated by the above module 722) to the visual axis. As an additional example, the module 730 can shift the user's optical axis nasally between 4.0° and 6.5°, between 4.5° and 6.0°, between 5.0° and 5.4°, etc., or any range formed by any of these values. In some arrangements, the module 730 can apply the shift at least in part based on characteristics of a particular user (e.g., their age, gender, vision prescription, or other relevant characteristics), and / or can apply the shift at least in part based on a calibration process for a particular user (i.e., to determine the optical-visual axis offset for a particular user). In at least some embodiments, the module 730 can also shift the origin of the left and right optical axes to correspond to the user's CoP (determined by module 732) rather than corresponding to the user's CoR.

[0247] (When provided) An optional center of perspective (CoP) estimation module 732 may estimate the positions of the user's left and right centers of perspective (CoP). The CoP can be a useful location for a wearable system and, in at least some embodiments, is the position directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the positions of the user's left and right centers of perspective based on the 3D position of the user's pupil center, the 3D position of the user's corneal curvature center, or such suitable data or any combination thereof. As an example, the user's CoP may be approximately 5.01 mm in front of the corneal curvature center (i.e., 5.01 mm from the corneal apex of the eye and along the optical axis in the direction towards the cornea) and may be approximately 2.97 mm behind the outer surface of the user's cornea along the optical axis or visual axis. The user's center of perspective may be directly in front of the center of their pupil. For example, the user's CoP may be less than approximately 2.0 mm from the user's pupil, less than approximately 1.0 mm from the user's pupil, or less than approximately 0.5 mm from the user's pupil, or any range between these values. As another example, the center of perspective may correspond to a location within the anterior chamber of the eye. As other examples, the CoP may be between 1.0 mm and 2.0 mm, approximately 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

[0248] The center of perspective described herein (as a potentially desired location for the pinhole of a rendering camera and an anatomical location within the user's eye) can be a location for reducing and / or eliminating unwanted parallax shifts. In particular, the optical system of the user's eye is very similar to a theoretical system formed by a pinhole in front of a lens, projected onto a screen, and the pinhole, lens, and screen roughly correspond to the user's pupil / iris, lens, and retina, respectively. Additionally, it is desirable that when two point light sources (or objects) at different distances from the user's eye rotate rigidly around the opening of the pinhole (e.g., rotate along a radius of curvature equal to their distance from the opening of the pinhole), there is little or no parallax shift. Thus, it might seem that the CoP should be located at the center of the pupil of the eye (and in some embodiments, such a CoP may be used). However, the human eye includes a cornea in addition to the pinhole of the lens and pupil, which imparts additional optical power to the light propagating to the retina. Therefore, in the theoretical system described in this section, the anatomical equivalent of the pinhole can be the region of the user's eye located between the outer surface of the user's cornea and the center of the user's pupil or iris. For example, the anatomical equivalent of the pinhole may correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP to such a location within the anterior chamber of the user's eye.

[0249] As described above, the eye tracking module 614 can provide data to other components in the wearable system, such as the light field rendering controller 618 and the registration viewer 620, such as the estimated 3D positions of the left and right eye centers of rotation (CoR), the vergence depth, the left and right eye optical axes, the 3D positions of the user's eyes, the 3D positions of the user's left and right corneal curvature centers, the 3D positions of the user's left and right pupil centers, the 3D positions of the user's left and right viewing angle centers, the user's IPD, etc. The eye tracking module 614 can also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 can include a blink detection module that provides a flag or other alert whenever the user blinks, and a saccade detection module that provides a flag or other alert whenever the user's eyes saccade (i.e., quickly shift focus to another point).

[0250] Example of a rendering controller

[0251] In Figure 7B is shown a detailed block diagram of an example light field rendering controller 618. As Figure 6 and 7B shown, the rendering controller 618 can receive eye tracking information from the eye tracking module 614, and can provide an output to the rendering engine 622, which can generate an image to be displayed for viewing by the user of the wearable system. As an example, the rendering controller 618 can receive information about the vergence depth, the left and right eye centers of rotation (and / or viewing angle centers), and other eye data (such as blink data, saccade data, etc.).

[0252] The depth plane selection module 750 can receive vergence depth information, and based on such data can cause the rendering engine 622 to provide content to the user, where the content appears to be located on a particular depth plane (i.e., at a particular distance accommodation or focal length). As combined with Figure 4As discussed, the wearable system can include a plurality of discrete depth planes formed by a plurality of waveguides, each depth plane transmitting image information at a varying level of wavefront curvature. In some embodiments, the wearable system can include one or more variable depth planes, such as optical elements that transmit image information at a wavefront curvature level that varies over time. In these and other embodiments, the depth plane selection module 750 can cause the rendering engine 622 to transmit content to the user at a selected depth (i.e., cause the rendering engine 622 to direct the display 220 to switch depth planes), at least in part based on the user's vergence depth. In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 can render content at different depths and also generate and / or provide depth plane selection data to display hardware such as the display 220. Display hardware such as the display 220 can perform electrical depth plane switching in response to depth plane selection data (which can be control signals) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0253] Generally, it can be desirable for the depth plane selection module 750 to select a depth plane that matches the user's current vergence depth, thereby providing the user with accurate accommodation cues. However, it can also be desirable to switch depth planes in a cautious and unobtrusive manner. As an example, it can be desirable to avoid excessive switching between depth planes and / or it can be desirable to switch depth planes when the user is less likely to notice the switch (e.g., during a blink or an eye saccade).

[0254] The hysteresis band crossing detection module 752 can help avoid excessive switching between depth planes, particularly when the user's vergence depth fluctuates at the midpoint or transition point between two depth planes. In particular, the module 752 can cause the depth plane selection module 750 to exhibit hysteresis in its selection of depth planes. As an example, the module 752 can cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the user's vergence depth exceeds a first threshold. Similarly, the module 752 can cause the depth plane selection module 750 (which can in turn direct a display such as the display 220) to switch to the first, more distant depth plane only after the user's vergence depth exceeds a second threshold that is further from the user than the first threshold. In the overlapping region between the first threshold and the second threshold, the module 750 can cause the depth plane selection module 750 to maintain whichever depth plane is currently selected as the selected depth plane, thereby avoiding excessive switching between depth planes.

[0255] The eye event detection module 750 can obtain from Figure 7AThe eye tracking module 614 receives other eye data and may cause the depth plane selection module 750 to delay some depth plane switches until an eye event occurs. As an example, the eye event detection module 750 may cause the depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; may receive data from a blink detection component in the eye tracking module 614 that indicates when the user is currently blinking; and in response, may cause the depth plane selection module 750 to perform the planned depth plane switch during the blink event (e.g., by causing module 750 to direct the display 220 to perform a depth plane switch during the blink event). In at least some embodiments, the wearable system is capable of shifting content to a new depth plane during a blink event such that the user is less likely to perceive the shift. As another example, the eye event detection module 750 may delay a planned depth plane switch until an eye saccade is detected. Such an arrangement may facilitate discrete shifting of the depth plane, as discussed in connection with blinks.

[0256] If desired, even in the absence of an eye event, the depth plane selection module 750 may delay a planned depth plane switch for a limited period of time before performing the depth plane switch. Similarly, when the user's vergence depth is substantially outside the currently selected depth plane (i.e., when the user's vergence depth has exceeded a predetermined threshold that is beyond the normal threshold for depth plane switching), the depth plane selection module 750 may perform a depth plane switch even in the absence of an eye event. These settings may help ensure that the eye event detection module 754 does not indefinitely delay depth plane switches and does not delay depth plane switches when there is a large adaptation error.

[0257] The rendering camera controller 758 may provide information indicating where the user's left and right eyes are to the rendering engine 622. The rendering engine 622 may then generate content by simulating cameras at the positions of the user's left and right eyes and generating content based on the perspective of the simulated cameras. As described above, the rendering camera is a simulated camera used in rendering virtual image content that may come from a database of objects in a virtual world. Objects may have positions and orientations relative to the user or wearer and may be relative to real objects in the environment surrounding the user or wearer. The rendering camera may be included in the rendering engine to render virtual images based on a database of virtual objects to be presented to the eyes. The virtual image may be rendered as if it were taken from the perspective of the user or wearer. For example, the virtual image may be rendered as if it were captured by a camera (corresponding to a "rendering camera") having an aperture, a lens, and a detector that views an object in the virtual world. The virtual image is taken from the perspective of such a camera having the position of the "rendering camera". For example, the virtual image may be rendered as if it were captured from the perspective of a camera having a specific position relative to the eyes of the user or wearer, thereby providing an image that appears to be seen from the perspective of the user or wearer. In some embodiments, the image is rendered as if captured from the perspective of a camera having an aperture at a particular position relative to the user or wearer's eyes (e.g., the center of view or center of rotation discussed herein or elsewhere).

[0258] The rendering camera controller 758 may determine the positions of the left and right cameras based on the center of rotation (CoR) of the left and right eyes determined by the CoR estimation module 724 and / or based on the center of view (CoP) of the left and right eyes determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR and CoP positions based on various factors. As an example, the rendering camera controller 758 may, in various modes, always register the rendering camera to the CoR position, always register the rendering camera to the CoP position, toggle or discretely switch between registering the rendering camera to the CoR position and registering the rendering camera to the CoP position over time based on various factors, or dynamically register the rendering camera along the optical (or visual) axis between the CoR and CoP positions to any position in a range of different positions based on various factors over time. The CoR and CoP positions may optionally be passed through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning) which may average the CoR and CoP positions over time to reduce noise in these positions and prevent jitter in the rendering simulation rendering camera.

[0259] In at least some embodiments, the rendering camera may be simulated as a pinhole camera, where the pinhole is set at the location of the estimated center of rotation (CoR) or center of projection (CoP) identified by the eye tracking module 614. When the CoP is offset from the CoR, whenever the position of the rendering camera is based on the user's CoP, the position of the rendering camera and its pinhole shift as the user's eyes rotate. In contrast, whenever the position of the rendering camera is based on the user's CoR, the position of the pinhole of the rendering camera does not move with eye rotation, although in some embodiments, the rendering camera (which is behind the pinhole) may move with eye rotation. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (i.e., rotate) with the user's eyes.

[0260] Example of a registration viewer

[0261] In Figure 7C a block diagram of an example registration viewer 620 is shown. As Figure 6 , Figure 7A and Figure 7C shown, the registration viewer 620 may receive eye tracking information from the eye tracking module 614 ( Figure 6 and 7A ). As an example, the registration viewer 620 may receive information about the centers of rotation of the user's left and right eyes (e.g., the three-dimensional positions of the centers of rotation of the user's left and right eyes, which may be on a common coordinate system or have a common reference frame with the head-mounted display system 600). As other examples, the registration viewer 620 may receive display extrinsic parameters, adaptation tolerances, and an eye tracking validity indicator. The display extrinsic parameters may include information about the display (e.g., Figure 2 the display 200), such as the field of view of the display, the dimensions of one or more display surfaces, and the position of the display surface relative to the head-mounted display system 600. The adaptation tolerances may include information about the display registration volume, which may indicate how far the user's left and right eyes can move from their nominal positions before affecting display performance. Additionally, the adaptation tolerances may indicate the amount of display performance impact expected based on the position of the user's eyes.

[0262] As Figure 7C shown, the registration viewer 620 may include a 3D position adaptation module 770. The position adaptation module 770 may acquire and analyze various pieces of data, including, for example, the 3D position of the left eye center of rotation (e.g., left CoR), the 3D position of the right eye center of rotation (e.g., right CoR), the display extrinsic parameters, and the adaptation tolerances. The 3D position adaptation module 770 may determine how far the user's left and right eyes are from their respective nominal positions of the left and right eyes (e.g., may calculate a left 3D error and a right 3D error), and may provide the error distances (e.g., the left 3D error and the right 3D error) to the device 3D adaptation module 772.

[0263] The 3D position adaptation module 770 can also compare the error distance with the external parameters of the display and the adaptation tolerance to determine whether the user's eyes are within the nominal volume, a partially degraded volume (e.g., a volume in which the performance of the display 220 is partially degraded), or in a fully degraded or almost fully degraded volume (e.g., a volume in which the display 220 can basically not provide content to the user's eyes). In at least some embodiments, the 3D position adaptation module 770 or the 3D adaptation module 772 can provide an output that qualitatively describes the adaptation of the HMD to the user, such as Figure 7C the adaptation quality output shown in. As an example, the module 770 can provide an output indicating whether the current adaptation of the HMD on the user is good, marginal, or failed. A good adaptation can correspond to an adaptation that enables the user to view at least a certain percentage of the images (e.g., 90%), a marginal adaptation can enable the user to view at least a lower percentage of the images (e.g., 80%), and a failed adaptation can be an adaptation where the user can only see an even lower percentage of the images.

[0264] As another example, the 3D position adaptation module 770 and / or the device 3D adaptation module 772 can calculate a visible area metric, which can be the percentage of the total area (or pixels) of the images displayed by the display 220 that are visible to the user. The modules 770 and 772 can calculate the visible area metric by: evaluating the positions of the user's left and right eyes relative to the display 220 (e.g., which can be based on the center of rotation of the user's eyes) and using one or more models (e.g., mathematical or geometric models), one or more look-up tables, or other techniques or a combination of these techniques with other techniques to determine the percentage of the images visible to the user based on the positions of the user's eyes. Additionally, the modules 770 and 772 can determine which regions or portions of the images expected to be displayed by the display 220 are visible to the user based on the positions of the user's eyes.

[0265] The registration viewer 620 can also include the device 3D adaptation module 772. The module 772 can receive data from the 3D position adaptation module 770 and can also receive an eye tracking validity indicator, which can be provided by the eye tracking module 614 and can indicate whether the eye tracking system is currently tracking the position of the user's eyes or whether the eye tracking data is unavailable or in an error condition (e.g., determined to be unreliable). If needed, the device 3D adaptation module 772 can modify the quality of the adaptation data received from the 3D position adaptation module 770 based on the status of the eye tracking valid data. For example, if the data from the eye tracking system is indicated as unavailable or in error, the device 3D adaptation module 772 can provide a notification about the existence of the error and / or not provide an output to the user regarding the adaptation quality or adaptation error.

[0266] In at least some embodiments, the registration viewer 620 may provide the user with feedback regarding the quality of the fit and details of the nature and magnitude of the error. As an example, the head-mounted display system may provide feedback to the user during the calibration or fitting process (e.g., as part of the setup process), and may provide feedback during operation (e.g., if the fit degrades due to slippage, the registration viewer 620 may prompt the user to readjust the head-mounted display system). In some embodiments, the registration analysis may be performed automatically (e.g., during use of the head-mounted display system), and feedback may be provided without user input. These are merely illustrative examples.

[0267] Example of using an eye tracking system to locate the user's cornea

[0268] Figure 8A is a schematic view of an eye showing the corneal sphere of the eye. As Figure 8A shown, the user's eye 810 may have a cornea 812, a pupil 822, and a lens 820. The cornea 812 may have an approximately spherical shape, as shown by the corneal sphere 814. The corneal sphere 814 may have a center point 816 (also referred to as the corneal center) and a radius 818. The hemispherical cornea of the user's eye may curve around the corneal center 816.

[0269] Figure 8B - 8E illustrates an example of using the 3D corneal center estimation module 716 and the eye tracking module 614 to locate the corneal center 816 of the user.

[0270] As Figure 8B shown, the 3D corneal center estimation module 716 may receive an eye tracking image 852 including a corneal flash 854. The 3D corneal center estimation module 716 may then simulate the known 3D positions of the eye camera 324 and the light source 326 in the eye camera coordinate system 850 (which may be based on data in the eye tracking extrinsic and intrinsic parameter database 702, the assumed eye size database 704, and / or per-user calibration data 706) in order to project light rays 856 in the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 may have its origin at the 3D position of the eye tracking camera 324.

[0271] In Figure 8C it, the 3D corneal center estimation module 716 simulates a corneal sphere 814a (which may be based on the assumed eye size from the database 704) and a corneal curvature center 816a at a first position. The 3D corneal center estimation module 716 may then check to see if the corneal sphere 814a will correctly reflect light from the light source 326 to the flash position 854. As Figure 8C shown in it, the first position does not match because the light ray 860a does not intersect the light source 326.

[0272] Similarly, inFigure 8D In this case, the 3D corneal center estimation module 716 simulates the corneal sphere 814b and the corneal curvature center 816b at the second position. Then, the 3D corneal center estimation module 716 checks to see if the corneal sphere 814b correctly reflects the light from the light source 326 to the flash position 854. As Figure 8D shown, the second position also does not match.

[0273] As Figure 8E shown, the 3D corneal center estimation module 716 is finally able to determine that the correct position of the corneal sphere is the corneal sphere 814c and the corneal curvature center 816c. The 3D corneal center estimation module 716 confirms that the shown position is correct by checking that the light from the light source 326 will be correctly reflected off the corneal sphere and imaged by the camera 324 at the correct position of the flash 854 on the image 852. With this arrangement and given the 3D positions of the known light source 326, the camera 324, and the optical characteristics (focal length, etc.) of the camera, the 3D corneal center estimation module 716 can determine the 3D position of the corneal curvature center 816 (relative to the wearable system).

[0274] At least in combination with Figure 8C - 8E The processes described herein can effectively be iterative, repetitive, or optimization processes to identify the 3D position of the user's corneal center. In this way, any one of a variety of techniques (e.g., iterative techniques, optimization techniques, etc.) can be used to effectively and quickly prune or reduce the search space of possible positions. Additionally, in some embodiments, the system can include two, three, four, or more light sources, such as the light source 326, and some of all of these light sources can be set at different positions, resulting in multiple flashes, such as flashes 854 located at different positions on the image 852 and multiple light rays (e.g., light rays 856) with different origins and directions. Such embodiments can enhance the accuracy of the 3D corneal center estimation module 716 because the module 716 can attempt to identify the corneal position that causes some or all of the flashes and light rays to be correctly reflected between their respective light sources and their respective positions on the image 852. In other words, and in these embodiments, the position of some or all of the light sources can be relied upon during the Figure 8B - 8E 3D corneal position determination (e.g., iterative, optimization techniques, etc.) process.

[0275] Example of normalizing the coordinate system of an eye tracking image

[0276] Figure 9A - 9C shown by, for example, Figure 7AExample normalization of the coordinate system of an eye tracking image by components in a wearable system, such as coordinate system normalization module 718. Normalizing the coordinate system of the eye tracking image relative to the pupil position of the user can compensate for slippage of the wearable system relative to the user's face (i.e., headset slippage), and such normalization can establish a consistent orientation and distance between the eye tracking image and the user's eyes.

[0277] As Figure 9A shown, coordinate system normalization module 718 can receive the estimated 3D coordinates 900 of the user's corneal center of rotation, and can receive a non-normalized eye tracking image such as image 852. As an example, eye tracking image 852 and coordinates 900 can be in a non-normalized coordinate system 850 based on the position of eye tracking camera 324.

[0278] As a first normalization step, as Figure 9B shown, coordinate system normalization module 718 can rotate coordinate system 850 to a rotated coordinate system 902 such that the z-axis of the coordinate system (i.e., the vergence depth axis) can be aligned with the vector between the origin of the coordinate system and the corneal curvature center coordinates 900. In particular, coordinate system normalization module 718 can rotate eye tracking image 850 to a rotated eye tracking image 904 until the coordinates 900 of the user's corneal curvature center are perpendicular to the plane of the rotated image 904.

[0279] As a second normalization step, as Figure 9C shown, coordinate system normalization module 718 can convert the rotated coordinate system 902 into a normalized coordinate system 910 such that the corneal curvature center coordinates 900 are a standard normalized distance 906 from the origin of the normalized coordinate system 910. In particular, coordinate system normalization module 718 can convert the rotated eye tracking image 904 into a normalized eye tracking image 912. In at least some embodiments, the standard normalized distance 906 can be approximately 30 millimeters. The second normalization step can be performed before the first normalization step if desired.

[0280] Example of using an eye tracking system to locate the centroid of the user's pupil

[0281] Figure 9D - 9G Shows an example of using 3D pupil center locator module 720 and eye tracking module 614 to locate the pupil center of the user (i.e., the center of pupil 822 of the user as Figure 8A shown).

[0282] As Figure 9DAs shown, the 3D pupil center locator module 720 may receive a normalized eye tracking image 912, which includes a pupil centroid 913 (i.e., the center of the user's pupil identified by the pupil identification module 712). Then, the 3D pupil center locator module 720 may simulate the normalized 3D position 910 of the eye camera 324 to project a ray 914 through the pupil centroid 913 in the normalized coordinate system 910.

[0283] In Figure 9E , the 3D pupil center locator module 720 may simulate a corneal sphere, such as corneal sphere 901 having a center of curvature 900, based on data from the 3D corneal center estimation module 716 (and as discussed in more detail in connection with Figure 8B - 8E ). As an example, the corneal sphere 901 may be positioned in the normalized coordinate system 910 based on the position of the identified center of curvature 816c in connection with Figure 8E and based on the normalization process of Figure 9A - 9C . Additionally, as shown in Figure 9E , the 3D pupil center locator module 720 may identify a first intersection 916 between the ray 914 (i.e., the ray between the origin of the normalized coordinate system 910 and the normalized position of the user's pupil) and the simulated cornea.

[0284] As shown in Figure 9F , the 3D pupil center locator module 720 may determine a pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 may share a common center of curvature with the corneal sphere 901, but have a smaller radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (i.e., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal curvature center may be determined from the Figure 7A assumed eye size 704, from the eye tracking extrinsic and intrinsic parameter database 702, and / or from per-user calibration data 706. In other embodiments, the distance between the pupil center and the corneal curvature center may be determined from the Figure 7A per-user calibration data 706.

[0285] As shown in Figure 9GAs shown, the 3D pupil center locator module 720 can locate the 3D coordinates of the user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 can utilize the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection point 916 between the simulated corneal sphere 901 and the ray 914 associated with the pupil centroid 913 in the normalized eye tracking image 912, information about the refractive index of the cornea, and other relevant information (such as the refractive index of air (which can be stored in the eye tracking extrinsic and intrinsic parameter database 702)) to determine the 3D coordinates of the center of the user's pupil. In particular, the 3D pupil center locator module 720 can bend the ray 916 into a refracted ray 922 based on the refractive difference between air (a first refractive index of approximately 1.00) and the corneal material (a second refractive index of approximately 1.38) in the simulation. After taking into account the refraction caused by the cornea, the 3D pupil center locator module 720 can determine the 3D coordinates of the first intersection point 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 can determine the user's pupil center 920 to be located at approximately the first intersection point 920 between the refracted ray 922 and the pupil sphere 918. With this arrangement, the 3D pupil center locator module 720 can determine the 3D position of the pupil center 920 (relative to the wearable system) in the normalized coordinate system 910. If needed, the wearable system can denormalize the coordinates of the pupil center 920 to the original eye camera coordinate system 850. The pupil center 920 can be used together with the corneal curvature center 900 to determine, in particular, the user's optical axis using the optical axis determination module 722 and the user's vergence depth using the vergence depth estimation module 728.

[0286] Example of the difference between the optical axis and the visual axis

[0287] As discussed in connection with Figure 7A the optical to visual mapping module 730, the user's optical axis and visual axis are generally not aligned, partly because the user's visual axis is defined by their fovea, and the fovea is generally not at the center of a person's retina. Thus, when a person wishes to focus their attention on a particular object, the person aligns their visual axis with the object to ensure that light from the object falls on their fovea, while their optical axis (defined by the center of their pupil and the curvature center of their cornea) is actually slightly offset from the object. Figure 10 is an example of an eye 1000 that shows the optical axis 1002 of the eye, the visual axis 1004 of the eye, and the offset between these axes. Additionally, Figure 10Shows the pupil center 1006 of the eye, the corneal curvature center 1008 of the eye, and the average center of rotation (CoR) 1010 of the eye. In at least some populations, the corneal curvature center 1008 of the eye may be located approximately 4.7 mm anteriorly, as indicated by dimension 1012 of the average center of rotation (CoR) 1010 of the eye. Additionally, the center of perspective 1014 of the eye may be located approximately 5.01 millimeters in front of the corneal curvature center 1008 of the eye, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or directly in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As another example, dimension 1012 may be between 3.0 mm and 7.0 mm, between 4.0 mm and 6.0 mm, between 4.5 mm and 5.0 mm, or between 4.6 mm and 4.8 mm, or any range between any values and any value within any of these ranges. The center of perspective (CoP) 1014 of the eye can be a useful location for a wearable system because in at least some embodiments, registering the rendering camera at the CoP can help reduce or eliminate parallax artifacts.

[0288] Figure 10 Also shown are such locations within the human eye 1000 with which the pinhole of the rendering camera can be aligned. As Figure 10 shown, the pinhole of the rendering camera can be aligned with location 1014 along the optical axis 1002 or the visual axis 1004 of the human eye 1000, which is closer to the outer surface of the cornea than (a) the center of the pupil or iris 1006 and (b) the corneal curvature center 1008 of the human eye 1000. For example, as Figure 10 shown, the pinhole of the rendering camera can be registered with location 1014 along the optical axis 1002 of the human eye 1000, which is approximately 2.97 millimeters behind the outer surface of the cornea 1016 and approximately 5.01 millimeters in front of the corneal curvature center 1008. The location 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to location 1014 can be considered to represent the center of perspective of the human eye 1000. As Figure 10 shown in the optical axis 1002 of the human eye 1000 represents the most direct line passing through the corneal curvature center 1008 and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 is different from the optical axis 1002 because it represents the line extending from the fovea of the human eye 1000 to the center of the pupil or iris 1006.

[0289] Example process of rendering content and checking registration based on eye tracking

[0290] Figure 11is a process flow diagram of an example method 1100 for using eye tracking in rendering content and providing feedback on registration in a wearable device. Method 1100 may be performed by a wearable system described herein. Embodiments of method 1100 may be used by a wearable system to render content based on data from an eye tracking system and provide feedback on registration (i.e., the fit of the wearable device to the user).

[0291] At block 1110, the wearable system may capture an image of one or both of the user's eyes. The wearable system may use one or more eye cameras 324 to capture the eye image, as shown at least in the example of Figure 3 . If desired, the wearable system may also include one or more light sources 326 configured to shine IR light on the user's eyes and create corresponding flashes in the eye image captured by the eye camera 324. As discussed herein, the flashes may be used by the eye tracking module 614 to derive various information about the user's eyes, including where the eyes are looking.

[0292] At block 1120, the wearable system may detect flashes and pupils in the eye image captured at block 1110. As an example, block 1120 may include processing the eye image by a flash detection and marking module 714 to identify the two-dimensional location of the flash in the eye image and processing the eye image by a pupil identification module 712 to identify the two-dimensional location of the pupil in the eye image.

[0293] At block 1130, the wearable system may estimate the three-dimensional positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the positions of the centers of curvature of the user's left and right corneas and the distances between these centers of curvature and the user's left and right corneas. Block 1130 may involve the 3D corneal center estimation module 716 for identifying the positions of the centers of curvature described at least in conjunction with Figure 7A and 8A -8E.

[0294] At block 1140, the wearable system may estimate the three-dimensional positions of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and the 3D pupil center locator module 720 may in particular estimate the positions of the user's left and right pupil centers, as part of block 1140 described at least in conjunction with Figure 7A and 9D -9G.

[0295] At block 1150, the wearable system may estimate the three-dimensional positions of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and the CoR estimation module 724 may in particular estimate the positions of the CoRs of the user's left and right eyes, as described at least in conjunction with Figure 7A and 10As described. As a specific example, the wearable system can find the CoR of the eye by returning along the optical axis from the center of curvature of the cornea to the retina.

[0296] At block 1160, the wearable system can estimate the user's IPD, vergence depth, center of perspective (CoP), optical axis, visual axis, and other desired attributes from the eye tracking data. As an example, the IPD estimation module 726 can estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the vergence depth estimation module 728 can estimate the user's depth by finding the intersection (or near intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 can identify the left and right optical axes over time, the optical axis to visual axis mapping module 730 can identify the left and right visual axes over time, and the CoP estimation module 732 can identify the left and right centers of perspective, as part of block 1160.

[0297] At block 1170, the wearable system can render content based in part on the eye tracking data identified in blocks 1120 - 1160 and can optionally provide feedback regarding registration (i.e., the fit of the wearable system to the user's head). As an example, the wearable system can identify the appropriate position of the rendering camera and then generate content for the user based on the position of the rendering camera, as discussed in connection with Figure 7B the light field rendering controller 618 and the rendering engine 622. As another example, the wearable system can determine whether it is properly fitted to the user or has slipped from its correct position relative to the user and can provide optional feedback to the user indicating whether the fit of the device needs adjustment, as discussed in connection with the registration viewer 620 and in connection with Figure 16 block 1608. In some embodiments, as discussed in connection with Figure 16 block 1610, the wearable system can adjust the rendered content based on incorrect or less than ideal registration to attempt to reduce, minimize, or compensate for the effects of incorrect or unregistered conditions.

[0298] Overview of device registration

[0299] To enable the wearable system 200 described herein to output images of high perceived image quality, the display 220 of the wearable system 200 ( Figure 2 ) is preferably properly fitted to the user (e.g., positioned and oriented relative to the user's head such that the input and output of the system 200 are properly docked with the corresponding parts of the user's head and such that the device is stable and comfortable to wear and use). As an example, to enable the display 220 to provide visual content to the user's eyes, the display 220 is preferably located in front of the user's eyes and, depending on the relevant characteristics of the display 220, the user's eyes are preferably located within a specific volume (see, e.g., in connection with Figure 13A and13B (discussed further in connection therewith). As another example, the speaker 240 is preferably located near, on, or within the user's ear to provide high-quality audio content to the user, the audio sensor (e.g., microphone) 232 is preferably located in a particular area to receive sound from the user, and the inward-facing imaging system 462 (which may include one or more cameras 324 and one or more infrared light sources 326) is preferably positioned and oriented correctly to obtain a clear, unobstructed image of the user's eyes (which may be part of an eye tracking system). These are merely examples of the various reasons why the wearable system 200 is preferably correctly adapted to the user.

[0300] To ensure that the wearable system 200 is correctly registered with the user, the wearable system 200 may include a registration viewer 620 such as Figure 6 . In some embodiments, the correctly registered wearable system 200 includes a display that is positioned such that one or both of the user's eyes can receive sufficient image light to substantially view the entire field of view provided by the display 220 of the wearable display system 200. For example, a correctly registered display may allow viewing of an image within a range of about 80% or higher, about 85% or higher, about 90% or higher, or about 95% or higher of the field of view of the display, with a brightness uniformity of 80% or higher, about 85% or higher, about 90% or higher, or about 95% or higher. It will be understood that when the display is displaying the same content throughout the entire field of view, the brightness uniformity may be equal to 100% times the minimum brightness divided by the maximum brightness over the entire field of view of the display (100% × L min / L max ).

[0301] The registration viewer 620 may use various sensors to determine how the wearable system 200 fits on the user (e.g., if the display 220 of the wearable system 200 is correctly positioned on the user). As an example, the registration viewer 620 may use the inward-facing imaging system 462 (which may include an eye tracking system) to determine how to spatially orient the relevant parts of the wearable system 200 relative to the user, particularly relative to the user's eyes, ears, mouth, or other parts with which the wearable system 200 interfaces.

[0302] The registration viewer 620 can assist in the calibration process for this initial or subsequent configuration or setting of the wearable system 200 for a particular user. As an example, the registration viewer 620 can provide feedback to the user during the configuration or setting of the wearable system 200 for that particular user. Additionally or alternatively, the registration viewer 620 can continuously or intermittently monitor the registration of the wearable system 200 on the user to check for continued proper registration during use and can provide user feedback on the fly. The registration viewer 620 can provide user feedback, either as part of the configuration process or as part of the registration monitoring during use, that indicates when the wearable system 200 is properly registered and when the wearable system 200 is not properly registered. The registration viewer 620 can also provide specific suggestions on how the user can correct any misregistration and achieve proper registration. As an example, the registration viewer 620 can recommend that the user push the wearable device backward after detecting a slide of the wearable device (e.g., down the user's nose bridge), can recommend that the user adjust some adjustable components of the wearable device (e.g., as described herein in connection with Figure 15A and 15B described), and so on.

[0303] Example of a registration coordinate system

[0304] Figure 12A - 12B An example eye position coordinate system is shown that can be used to define the three-dimensional positions of a user's left and right eyes relative to the display of the wearable system described herein. As an example, the coordinate system can include axes X, Y, and Z. The axis z of the coordinate system can correspond to depth, e.g., the distance between the plane in which the user's eyes are located and the plane in which the display 220 is located (e.g., the direction perpendicular to the plane of the front of the user's face). The axis x of the coordinate system can correspond to the left-right direction, such as the distance between the user's left and right eyes. The axis y of the coordinate system can correspond to the up-down direction, which can be the vertical direction when the user is standing upright.

[0305] Figure 12A A side view of the user's eyes 1200 and the display surface 1202 (which can be part of the Figure 2 display 220) is shown, while Figure 12B a top view of the user's eyes 1200 and the display surface 1202 is shown. The display surface 1202 can be located in front of the user's eyes and can output image light to the user's eyes. As an example, the display surface 1202 can include one or more out-coupling light elements, active or pixel display elements, and can be part of a waveguide stack, such as Figure 4Stacked waveguide assembly 480. In some embodiments, the display surface 1202 may be planar. In some other embodiments, the display surface 1202 may have other topologies (e.g., curved). It should be understood that the display surface 1202 may be the physical surface of the display, or merely a planar or other imaginary surface from which image light is understood to propagate from the display 220 to the user's eyes.

[0306] As Figure 12A shown, the user's eye 1200 may have an actual position 1204 that is offset from the nominal position 1206, and the display surface 1202 may be in position 1214. Figure 12A Also shown is the corneal apex 1212 of the user's eye 1200. The user's line of sight (e.g., their optical axis and / or visual axis) may be substantially along the line between the actual position 1204 and the corneal apex 1212. As Figure 12A and Figure 12B shown, the actual position 1204 may be offset from the nominal position 1206 by a z-offset 1210, a y-offset 1208, and an x-offset 1209. The nominal position 1206 may represent the preferred position for the user's eye 1200 relative to the display surface 1202 (sometimes referred to as the design position, which may typically be centered within the desired volume). As the user's eye 1200 moves away from the nominal position 1206, the performance of the display surface 1202 may degrade, as discussed herein in connection with, for example, Figure 14 discussed.

[0307] Additionally, it will be understood that the default position of the rendering camera may be the nominal position 1206. As discussed herein, the display system may be configured to render content from the perspective of a virtual rendering camera. As a result, various parameters of the rendering camera (e.g., focal length) may affect the appearance of the content provided to the user. For example, the focal length may determine the magnification and size of the virtual content presented to the user. Thus, different focal lengths may be associated with different depth planes.

[0308] In some embodiments, by default, the lens of the rendering camera may be positioned at the nominal position 1206, which is assumed to correspond to the center of rotation, which may be understood as point 1204 in this example. However, an offset of the center of rotation 1204 from the nominal position 1206 may result in undesirable viewer discomfort. For example, it will be recognized that magnification errors may occur on a per-eye basis, and virtual content may appear larger or smaller than expected. In one scenario, if the focal length of the rendering camera is shorter than expected (e.g., because the center of rotation of the user's eye is located behind the nominal position without compensating for this displacement or offset in the rendering space), the virtual content may appear smaller than expected. Similarly, if the focal length of the rendering camera is longer than expected (e.g., because the center of rotation of the user's eye is located in front of the nominal position but without compensating for this displacement or offset in the rendering space), the virtual content may appear larger than expected. If the magnification error is different for each eye (e.g., because the center of rotation of one eye is located behind the nominal position while the center of rotation of the user's other eye is located in front of the nominal position without properly compensating for these offsets in the rendering space), the perceived size of the same virtual object may be different per eye. Such a difference in size may cause the user to experience a certain degree of discomfort (e.g., attempting to reconcile the difference in binocular sizes can cause potential eye fatigue and / or headaches).

[0309] In some embodiments, the focal length of the rendering camera may be determined based on the z-axis offset between the default position 1206 (the position assumed to be the center of rotation) and the actual position of the center of rotation 1204. For example, if the center of rotation is located behind the nominal position, the focal length of the rendering camera may be decreased (e.g., the offset is decreased). On the other hand, if the center of rotation is located in front of the nominal position, the focal length may be increased (e.g., the offset is increased).

[0310] Additionally, in some embodiments, the focal length of the rendering camera may also be calculated based on the depth plane being used by the system. For example, in some embodiments, it may be assumed that the optics of the rendering camera follow the thin lens equation (1 / o + 1 / i = 1 / f), where o is the object distance (e.g., the depth plane on which the content is presented), i is a constant (e.g., the distance from the center of rotation to the user's retina), and f is the focal length. As discussed herein, the depth plane on which the content is presented is at a set distance from the user. As a result, since the quantities o and i are known, the focal length can be determined by solving for f. In some implementations, the rendering camera focal length adjustment may be performed in combination with one or more operations described herein, such as the steps 1170 described above and hereinafter referenced Figure 11 as described and hereinafter referenced Figure 16Step 1610, described in more detail. Examples of additional render camera adjustment schemes that may be employed by one or more of the systems described herein are provided in U.S. Provisional Patent Application No. 62 / 618,559, filed on January 17, 2018, entitled "EYECENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS," and U.S. Provisional Patent Application No. 62 / 702,849, filed on July 24, 2018, entitled "EYECENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS," the entire disclosures of which are incorporated herein by reference.

[0311] Continuing to refer Figure 12A , it will be appreciated that a point or volume associated with the user's eye 1200 can be used to represent the position of the user's eye in the registration analysis herein. The representative point or volume can be any point or volume associated with the eye 1200 and is preferably used consistently. For example, the point or volume can be on or within the eye 1200 or can be placed remote from the eye 1200. In some embodiments, the point or volume is the center of rotation of the eye 1200. The center of rotation can be determined as described herein and can have the advantage of simplifying the registration analysis since it is generally symmetrically disposed about various axes within the eye 1200 and allows a single display registration volume aligned with the optical axis to be used for the analysis.

[0312] Figure 12AIt is also shown that the display surface 1202 can be located at the center below the user's field of view (looking along the y-axis when the user looks straight ahead, with their optical axis parallel to the ground) and can be tilted (relative to the y-axis). In particular, the display surface 1202 can be set at a position below the user's field of view such that when the eyes 1200 are at position 1206, the user will have to look down at an angle of approximately 1216 to look at the center of the display surface 1202. This can facilitate a more natural and comfortable interaction with the display surface 1202, especially when viewing content rendered at a shorter depth (or distance from the user), because the user can more comfortably view content below the field of view rather than above the field of view. Additionally, the display surface 1202 can be tilted, for example, at an angle 1218 (relative to the y-axis) such that when the user gazes at the center of the display surface 1202 (e.g., gazing slightly below the user's field of view), the display surface 1202 is generally perpendicular to the user's line of sight. In at least some embodiments, the display surface 1202 can also be offset left or right (e.g., along the x-axis) relative to the nominal position of the user's eyes. As an example, the left-eye display surface can be offset to the right and the right-eye display surface can be offset to the left (e.g., the display surfaces 1202 can be offset towards each other) such that the user's line of sight hits the center of the display surface when focused at a distance less than infinity, which may increase user comfort during typical use of the wearable device.

[0313] Example of displaying a registration volume

[0314] Figure 13A - 13B An example display registration volume 1302a is shown. The display registration volume 1302a can represent the volume of space in which the eyes 1200 are located in order to receive image light from the display device. In some embodiments, the center of rotation of the user's eyes is preferably positioned such that the eyes are registered or receive image information from the display device. In some embodiments, when the center of rotation of the user's eyes is within the display registration volume 1302a, the user can see the entire image output by the display device with high luminance uniformity. For example, as described herein, a properly registered display can allow the image to be seen over approximately 80% or more, approximately 85% or more, approximately 90% or more, or approximately 95% or more of the display's field of view, with a luminance uniformity of 80% or higher, approximately 85% or higher, approximately 90% or higher, or approximately 95% or higher. In other words, a display with "good" registration (e.g., determined by Figure 7C module 772) can have a luminance uniformity of 90% or higher, a display with "average" registration can have a luminance uniformity of 80% or higher, and a display with "failed" registration can have a luminance uniformity of less than 80%.

[0315] As also described herein, the center of rotation 1204 can be used as a convenient reference point for referencing and determining the three-dimensional position of the user's eyes. The techniques described herein can be used to determine the center of rotation of each of the user's eyes, for example, by returning along the user's optical axis from the center of curvature of the cornea to the center of rotation (CoR). However, in general, any desired reference point associated with the user's eyes can be utilized in the processes and systems described herein. The display registration volume 1203 can represent the volume within which the display surface 1202 can operate in near full potential space (e.g., without significant degradation of the performance of the display surface 1202 Figure 15A and 15B of the type described). If the user's eyes (e.g., the center of rotation 1204 of the user's eyes) are not within the registration volume 1302a, the user may experience performance degradation, and some or all of the content provided by the display surface 1202 may be partially dimmed or completely invisible to the user.

[0316] As Figure 13A shown, the registration volume 1302a can have the shape of a frustum of a cone, which is the part remaining after the upper part of the pyramid has been cut off by a plane generally parallel to its base. In other words, the registration volume 1302a can be larger along the x-axis and y-axis when the user's eyes are closer to the display surface 1202 (see, for example, Figure 12A and 12B ), and smaller along the x-axis and y-axis when the user's eyes are farther from the display surface 1202. A frustum of a cone is an example of a truncated shape where the shear plane (e.g., the line where a part of the original shape is cut off) is parallel to the base of the volume. In general, the registration volume, such as volume 1302a, can take the shape of a volume truncated in any way, such as by one or more non-parallel shear planes (e.g., as Figure 13B shown) or by one or more non-planar shears.

[0317] The dimensions of the registration volume can depend on the specific implementation of the display surface 1202 and other elements of the wearable system. For example, Figure 13B shows that the registration volume 1302b can be angled relative to the display surface 1202. In the example of Figure 13B , the part of the registration volume 1302b closest to the display surface 1202 can be angled with respect to the display surface 1202 such that when the user's eyes move vertically (along the y-direction) in the front part of the volume (the z-position closest to the display surface 1202), the user's eyes will need to move away from the display surface (along the z-axis) to remain within the registration volume 1302b. In some embodiments, the shape of the registration volume 1302b can be based on the capabilities of the eye tracking system, which may not be able to track the user's eyes outside of the angled volume 1302b of Figure 13B .

[0318] The size and shape of the registered volume may also depend on the characteristics of the various parts of the display 220, which may include a display surface 1202. As an example, the display 220 may be a light field display having: one or more waveguides (which may be stacked and may provide multiple vergence cues to the user), an in-coupling element that receives light from an image injection device and couples the light into the waveguide, a light distribution element (sometimes referred to as an orthogonal pupil expander (OPE) disposed on the waveguide to distribute light to an out-coupling element), and an out-coupling element (sometimes referred to as an exit pupil expander (EPE)) that directs the light to the viewer's eye. In some embodiments, as described herein, the display surface 1202 is the surface or a portion of the surface from which the display system outputs light having image information to form an image in the user's eye. For example, the display surface 1202 may be the region defined by the out-coupling element or EPE on the waveguide surface, and the perimeter of the display surface 1202 is the perimeter of the region defined by the out-coupling element or EPE. At least in connection with the Figure 9A - 9C Further examples and details of light field displays and components of such displays are also described, the entire content of which is incorporated herein by reference.

[0319] In some embodiments, the x-dimension of the registration volume 1302a may span approximately 3.0 mm, 3.0 mm, 3.5 mm, 4.0 mm, 4.5 mm, 4.7 mm, 5.0 mm, 5.5 mm, or 6.0 mm; or may be less than 3.0 mm; or may be greater than 6.0 mm along the back of the volume (e.g., the maximum distance along the z-axis from the display surface). Similarly, the y-dimension of the registration volume 1302a may span approximately 2.5 mm, 3.0 mm, 3.5 mm, 3.9 mm, 4.0 mm, 4.5 mm, 5.0 mm, 5.5 mm, or 6.0 mm; or may be less than 2.5 mm; or may be greater than 6.0 mm along the back of the volume. At the nominal x and y positions, the z-dimension of the registration volume 1302a may span approximately 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.5 mm, or 11.0 mm; or may be less than 7.0 mm; or may be greater than 11.0 mm. The x and y dimensions may be larger at the front of the volume. As an example, the x and y dimensions of the registration volume at the front of the volume may be approximately 7.0 mm, 7.5 mm, 8.0 mm, 8.5 mm, 8.9 mm, 9.0 mm, 9.5 mm, 10.0 mm, 10.0 mm, 10.5 mm, 11.0 mm, 11.4 mm, 11.5 mm, 12.0 mm, or 12.5 mm; or may be less than 7.0 mm; or may be greater than 12.5 mm. As a specific example, the dimensions of the registration volume may include a z-dimension of approximately 9 mm; an x-dimension of approximately 4.7 mm at the back of the volume and an x-dimension of approximately 11.4 mm at the front of the volume; a y-dimension of approximately 3.9 mm at the back of the volume and a y-dimension of approximately 8.9 mm at the front of the volume.

[0320] In at least some embodiments, there may be multiple registration volumes, such as volumes 1302b and 1304, each registration volume associated with a different minimum display performance level. By way of example, Figure 13BThe volume 1304 can be less than the volume 1302 and can represent the volume within which the user perceives all the content provided by the display surface 1202 with 100% brightness uniformity, while the larger volume 1302b can represent the volume within which the user perceives at least 90% of the content provided by the display surface 1202 with 100% brightness uniformity. Thus, in some embodiments, the display system can be configured to determine whether the user's eyes are within the registration or viewing volume of the display, and / or can be configured to determine whether the user's eyes are within a threshold distance from the registration volume. For example, in some embodiments, the smaller volume 1304 can be considered the baseline registration volume or viewing volume, and the boundaries of the larger volume 1302b can be considered to define an acceptable threshold distance from the registration volume. In some embodiments, if the display system determines that the position of the eyes is outside the viewing volume of the display system by more than the threshold distance, the display system can be configured to provide the user with feedback indicating that the display and the eyes are not properly registered, for output and / or to take measures to mitigate display degradation caused by the misregistration, as discussed herein.

[0321] In some embodiments, the display registration volume can be determined at least in part based on the applications currently running or to be run on the system. For example, when running an application in which only a portion of the field of view is used to display virtual content (e.g., a reading-based application in which reading material may only occupy a portion (e.g., the central portion) of the display's field of view), the display system can use a larger display registration volume (e.g., volume 1302b). As a result, due to misregistration, the loss of the user's ability to perceive image content in the periphery of the field of view may not be perceivable when running such a reading-based application, as the application may not present content in the periphery. As a result, a larger display registration volume can be used, for example, to reduce unnecessary notifications to the user about misregistration when such image registration may not affect the content presented by the application running on the display system. Similarly, in another example, when running an application in which the head-mounted display is expected to be displaced relative to the user's eyes (e.g., a motion-oriented application or other applications that require a relatively high level of user engagement and / or physical activity), the display system can use a larger display registration volume (e.g., volume 1302b). In such applications, notifications about misregistration can be considered, for example, distracting or otherwise impairing the user experience; providing a larger display registration volume reduces the likelihood that the display system will generate such notifications for the user. In some other embodiments, when running an application in which it is desired to provide content across the entire field of view of the display or in the peripheral portions of the field of view, a smaller display registration volume (e.g., volume 1304) can be utilized. Such applications can include immersive gaming applications in which the content is displayed across the entire display field of view. In some implementations, the display registration volume can be determined based on other factors, including user preferences, user visual prescription, operating conditions of the system, and the like.

[0322] Figure 13C and 13D illustrates an example display registration volume configured to use the center of rotation of the eye as a reference point indicating the position of the eye relative to the user's eye and the display surface. In particular, Figure 13C illustrates an exemplary positioning of a display registration volume (e.g., registration volume 1302b) within the user's eye 1200. In Figure 13C the example, the center of rotation 1204 of the eye 1200 is approximately centered within the registration volume 1302b. Additionally, the registration volume 1302b is shown as having an exemplary size with a depth of approximately 9 mm and a width of approximately 3.5 mm and a height of 3 mm at the midpoint of the depth axis. As discussed herein, the size of the registration volume can vary and can be related to the characteristics of the various components of the wearable system. Figure 13C Also illustrated is an eye structure 1340, which can be the lens or pupil of the eye 1200.

[0323] Figure 13D illustrates a larger environment in which the user's eye 1200 is generally located within the registration volume 1302b and is viewing virtual content 1350 through the display surface 1202. As discussed herein, virtual content such as virtual content 1350 can be provided to the user with vergence and accommodation cues associated with a depth greater than the depth of the display surface 1202. In other words, the virtual content 1350 appears to the user to be at a greater distance from the user than the display 1202 when viewed with the eye 1200. In Figure 13D the example, such an arrangement is illustrated.

[0324] Continuing to refer to Figure 13D, it will be recognized that the display registration volume 1302b can be a virtual volume having a boundary defined by projections from the perimeter of the display surface 1202 to points within the eye 1200. For example, the projection can define a pyramid, and the display registration volume 1302b can be a frustum of that pyramid. Thus, along a plane facing the display surface 1202 on the optical axis, the cross-sectional shape of the display registration volume 1302b is similar to the shape formed by the perimeter of the display surface 1202. For example, as shown, where the display surface 1202 is square, the cross-sectional shape of the display registration volume 1302b is also square. Additionally, as also shown, where the center of the display surface 1202 is below the user's field of view, the frustum can also be tilted such that the center of the front portion of the display registration volume 1302b is also below the user's field of view. It will be understood that in some embodiments, the relevant perimeter of the display surface 1202 is the perimeter of the area of the display on which image light or display content is output. The boundary of the display registration volume 1302b can be defined in the same coordinate system in which various features such as the center of rotation 1204 of the eye 1200 are mapped, allowing comparison between the display registration volume 1302b and these various features.

[0325] In some embodiments, the center of rotation 1204 of the eye is centered within the frustum defining the display registration volume 1302b. However, it should be understood that the nominal position of the center of rotation 1204 of the eye and / or the overall shape of the frustum can be determined empirically or selected using criteria other than projection from the display surface 1202 such that the display system can correctly register the display and provide accurate feedback regarding registration quality and an acceptable level of registration even if not ideal.

[0326] Example of display performance at various registration positions

[0327] Figure 14 Illustrates how the performance of the display surface 1202 can vary with the position of the user's eye 1200. As shown, light rays from the display surface 1202 can be directed at an angle to the eye such that light rays from the edges of the display surface 1202 propagate inwardly towards the eye 1200. Thus, the cone 1202' represents the light cone output by the display surface 1202 to the eye 1200 to form an image.

[0328] Thus, when the display surface 1202 is displaced relative to the eye 1200, the exit pupils of the pixels corresponding to the respective parts of the field of view do not reach the retina of the eye 1200, and the image appears dimmed in those parts of the field of view. The positions 1204a, 1204b, 1204c, and 1204d of the center of rotation of the eye are effectively displaced relative to the ideal position 1204' of the center of rotation; the movement of the display surface 1202 relative to the eye 1200 can cause the center of rotation of the eye to potentially move outside of the display registration volumes 1302a, 1304, 1302b ( Figure 13A and 13B ) of the display surface 1202. As discussed, the display registration volumes can be bound to the display surface 1202, e.g., the display registration volumes can be defined by projections from the display surface 1202. Thus, as the display surface 1202 moves relative to the eye 1200, so do the display registration volumes 1302a, 1302b ( Figure 13A and 13B ). Figure 14 Shows various positions of the center of rotation of the user's eye (e.g., positions 1204a, 1204b, 1204c, and 1204d), the relative position of the display surface 1202, and how the user perceives (or views) the representation of the content provided by the display surface 1202 at the respective positions (e.g., representations 1400a, 1400b, 1400c, and 1400d).

[0329] In example 1400a, the center of rotation of the user's eye can be at position 1204a, which can be centered within a registration volume such as registration volume 1300b (e.g., a volume where the image quality is high because the eye 1200 receives almost all of the image light output by the display surface 1202 on its retina). Representation 1400a can represent the user's perception (or view) of the content provided by the display surface 1202 when the user's eye is at position 1204a. As shown in representation 1400a, the brightness of substantially all of the content on the display surface 1202 is uniform and can be at or near the full brightness level.

[0330] In example 1400b, the center of rotation of the user's eye can be at position 1204b, which can be in a volume such as volume 1304 ( Figure 13B) outside of the preferred display registration volume such as, but within a secondary registration volume such as volume 1302b (e.g., a volume where display performance is only slightly degraded). Representation 1400b may represent the user's perception (or viewing) of the content provided by display surface 1202 when the center of rotation of the user's eyes is at position 1204b. As shown in representation 1400b, due to the misregistration of the user's eyes relative to display surface 1202, part 1402 of the image along the right side of display surface 1202 may have a perceived reduced brightness (e.g., 50% brightness).

[0331] In example 1400c, the center of rotation of the user's eyes may be at position 1204c, which may be outside (or on the outer edge) of a secondary registration volume such as volume 1302b( Figure 13B ). Representation 1400c may represent the user's perception (or viewing) of the content provided by display surface 1202 when the center of rotation of the user's eyes is at position 1204c. As shown in representation 1400c, part 1406 along the edge of the displayed image may be completely (or almost completely) dimmed due to misregistration and thus not visible to the user. In an arrangement where some pixels of the display are below the perceived brightness level, the display may provide a reduced field of view (e.g., the user may not be able to perceive the entire field of view that the display could otherwise present). Additionally, there may be a band or portion 1404 of the image with a gradually decreasing brightness between the dark portion 1406 and the rest of the representation.

[0332] In example 1400d, the center of rotation of the user's eyes may be at position 1204d, which may be completely outside of the desired registration volume. Representation 1400d may represent the user's perception (or viewing) of the content provided by display surface 1202 when the center of rotation of the user's eyes is at position 1204d. As shown in representation 1400d, due to significant misregistration, most of image 1410 may appear completely (or almost completely) dark to the user, and a substantial portion 1408 of the image may appear dimmed.

[0333] As discussed herein, it will be appreciated that a display system may be configured to increase the light output or perceived brightness of portions of the display's field of view that are dimmed due to misalignment. For example, upon determining that misregistration exists, as discussed herein, the display system may provide a notification for the display in the form of a flag or instruction to increase the amount of light output to the user for pixels expected to be dimmed due to misregistration. For example, in example 1400b, the pixels representing the image information in part 1402 may have their perceived brightness increased to mitigate the expected reduction in perceived brightness due to misregistration.

[0334] It will be appreciated that as the level of misregistration increases, the ability to increase brightness to compensate for dimming decreases. For example, in Example 1400d, since the eye 1200 does not receive any light from the pixels in region 1410, region 1410 can be dark. Thus, while the increased brightness can mitigate the dimming caused by misregistration in portion 1402 (Example 1400b), the increased brightness may not mitigate the dimming in portion 1410 of the display (Example 1400d), where misregistration causes the eye to receive no light at all. Portion 1410 is larger than portion 14, and as an approximation, the size of the portion expected to be dimmed can indicate whether increasing brightness is effective; that is, if the size (e.g., number of pixels) of the portion expected to be dimmed is large enough, then in some embodiments, it can be assumed that the misregistration is large enough such that increasing brightness is not effective for most of these pixels. As a result, in some embodiments, the display system can be configured to compare the number of pixels in the portion expected to be dimmed, and if that number exceeds a threshold, provide feedback to the user indicating that the display and the eye are not properly registered.

[0335] Example of an interchangeable adapter for a wearable system

[0336] Figure 15A and 15B shows an exploded perspective view of a wearable system 220 that can include interchangeable adapters. In particular, Figure 15A shows how the wearable system 200 can include interchangeable rear pads such as pads 1500a, 1500b, and 1500c; while Figure 15B shows how the system 200 can include an interchangeable forehead pad such as pad 1502 and an interchangeable nose bridge pad such as pad 1504. These interchangeable pads can be used to adjust the fit of the wearable system 200 for individual users, who may have varying anatomical attributes (e.g., how the display 220 and the frame 230 fit various different users). As an example, a user with a relatively small head may benefit from attaching relatively large rear pads 1500a, 1500b, and 1500c to the frame 230, while a user with a relatively large head may obtain better results (e.g., better optical performance and stability of the frame 300 on their head) by attaching relatively small rear pads or even omitting the rear pads. Similarly, users with a prominent nose and / or forehead may benefit from smaller forehead pad 1502 and / or nose bridge pad 1504; while users with a less prominent nose and / or forehead may benefit from larger forehead pad 1502 and / or nose bridge pad 1504. These are only exemplary examples, and generally determining the set of interchangeable pads to best fit any particular user can be complex. As described herein, the display system can display a notification to the user that indicates that different interchangeable adapters may be needed to provide proper registration of the display to the user.

[0337] Reference Figure 15A , the wearable system may include one or more housing openings 1510. The housing opening 1510 may be an opening in the frame 230 and may optionally include a lens or other structure, such as a light-transmissive structure for a waveguide that mechanically protects the display, if desired. The lens in the housing opening 1510 may be transparent (e.g., completely or nearly completely transparent), or may be partially opaque (e.g., to reduce the level of ambient light passing through the opening 1510). The opening in the frame 230, although shown as generally circular in Figure 15A , may have any desired shape.

[0338] Example process of observing device registration

[0339] Figure 16 is a flowchart of an exemplary method 1600 for observing device registration and providing feedback regarding registration or compensation for misregistration in a wearable device. Method 1600 may be performed by the wearable system described herein. Embodiments of method 1600 may be used by the wearable system to provide feedback regarding registration (i.e., the fit of the wearable device to the user) based on data from an eye tracking system and to adjust the display to attempt to compensate for fitting errors (e.g., misregistration).

[0340] At block 1602, the wearable system may obtain a fitting tolerance. The fitting tolerance may include information associated with display registration volumes such as volumes 1302a, 1302b, or 1304. In particular, the fitting tolerance may include information associated with the nominal (e.g., normal) position of the user's eyes relative to the wearable device and may include information regarding how differences from the nominal position affect device performance. As an example, the fitting tolerance may include information regarding the nominal position range within which the wearable device can impede the user's performance by at least a certain desired amount (e.g., no more than 50% dimming on any pixel in the display).

[0341] At block 1604, the wearable system may obtain registration data. The registration data may include the spatial relationships between various components of the wearable system and the associated parts of the user. As examples, the registration data may include one or more of the following: the three-dimensional position of the user's left eye relative to the left-eye display of the wearable system; the 3D position of the user's right eye relative to the right-eye display; and the 3D position of the user's ears relative to the audio output (e.g., speakers, headphones, headsets, etc.) of the wearable system. The wearable system may use any suitable mechanism to obtain the registration data. As an example, the wearable system may use Figure 3An eye tracking camera 324 of the type shown (or other cameras, which may or may not be inward-facing cameras) captures images of one or both of the user's eyes to determine the relative position of the user's eyes and the wearable system. As other examples, the wearable system may include a depth sensor, a pressure sensor, a temperature sensor, a light sensor, an audio sensor, or other sensors to measure or obtain registration data such as the position of the wearable device relative to the user.

[0342] At block 1606, the wearable system may determine fitting characteristics. For example, the wearable system may determine whether the user's left eye is within a left-eye registration volume (e.g., one of volumes 1302a, 1302b, or 1304 of the left eye) and whether the user's right eye is within a right-eye registration volume (e.g., one of volumes 1302a, 1302b, or 1304 of the right eye). Block 1606 may also involve determining how far the user's eyes (or other body parts) are from their nominal positions. As an example, in block 1606, the wearable system may determine that at least one of the user's eyes is outside its respective display registration volume, how much outside the display registration volume the user's eyes are, and in which direction. In blocks 1608 and 1610, information about the direction and magnitude of the misregistration (e.g., the distance between the registration volume or nominal position and the actual position of the user's eyes or other body parts) may be beneficially utilized.

[0343] At block 1608, the wearable system may provide feedback to the user (or some other entity) about the fitting characteristics determined in block 1606. As an example, if the wearable system determines in block 1606 that the wearable device is too low relative to the user's eyes, the wearable system may provide a notification to the user in block 1608 that recommends the user use an appropriate nose bridge pad 1504 (e.g., add a nose bridge pad if none was previously attached, or replace an existing nose bridge pad with a larger or higher one). Conversely, if the wearable device determines that it is too high relative to the user's eyes, the system may provide a recommendation to the user to use a smaller nose bridge pad or remove the pad entirely (if the wearable device is designed to be worn without a pad). As other examples, the wearable system may provide feedback to the user recommending changes to the forehead pad such as pad 1502, changes to the rear pads such as pads 1500a - 1500c, changes to other adjustable components of the wearable system, changes to how the user wears the wearable system (e.g., instructions to move or rotate the system in a particular direction relative to the user). Generally, user feedback may be generated based on the position of the user's eyes relative to the display or other metrics (e.g., the visible image portion identified by the system). As an example, when the system determines that the user's eyes are above the registration volume, the system may recommend to the user that they push the wearable device up along their nose to correct the misregistration.

[0344] Any suitable device can be used to provide user feedback. As an example, user feedback can be provided via video presented on a display in a wearable device or an external display, or via audio presented by a wearable device or an external device. In various embodiments, the wearable device can provide an interactive guide to help the user obtain correct registration in a relatively intuitive manner. As an example, the wearable device can display two virtual targets, one representing the position of the user's eyes and the other representing the nominal registration position. Then, as the user moves the wearable device around and adjusts its fit, the user can perceive how their adjustments affect the registration, and the user can quickly and intuitively achieve correct registration.

[0345] In an arrangement where user feedback is provided by an output device such as a display that is part of a wearable device, the wearable device can provide feedback to the user in a manner that ensures the user can perceive the feedback. As an example, consider Figure 14 representation 1400d. In such an example, the wearable system can move the user feedback to a portion of the displayed image that is perceived by the user, in Figure 14 example 1400d of, for example, the left half of the display rather than the invisible right half of the display.

[0346] In some embodiments, feedback of the type described herein can be provided to a salesperson in a retail environment, and the feedback can be communicated over a network to the salesperson's computer or mobile device.

[0347] At block 1608, the wearable system can adjust its output and input to compensate for uncorrected fit errors. In some embodiments, block 1608 is only executed after the user fails to correct the fit error in response to the feedback. In other embodiments, block 1608 can be executed until the user corrects the fit error. In some embodiments, block 1608 can be executed whenever the user decides to continue using the wearable system with a fit error. In some embodiments, block 1608 can be omitted.

[0348] As an example, in block 1608, the wearable system can adjust its output and input by: adjusting a portion of the displayed image (e.g., to compensate for dimming caused by Figure 14 misregistration of the type shown), by adjusting the microphone input (e.g., increasing the microphone gain when the user is too far from the microphone, or decreasing the microphone gain when the user is too close to the microphone), by adjusting the speaker output (e.g., increasing or decreasing the speaker volume when the user is too close to or too far from the speaker in the wearable device), etc. As a specific example, the wearable system can selectively increase portions of the image (e.g., Figure 14the brightness of portions 1402, 1404, or 1408) in an attempt to reduce dimming due to misregistration. In some other embodiments, the wearable system may identify certain portions of the image (such as Figure 14 portions 1406 or 1410) that are not visible to the user and may reduce the light output in those areas to reduce the energy consumption of the wearable system. For example, in a configuration where different portions of the image may have dedicated, selectively activatable light sources or portions of light sources, the light output of one or more light sources or portions of light sources associated with the invisible portions of the image may be reduced or turned off.

[0349] Example of identifying a display registration volume

[0350] Figure 17A - 17H shows a view of the light field projected by the display and how the intersections of the light fields can partially define the display registration volume. Figure 18 shows a top view of the overlapping light fields projected by the display and how the intersections of the light fields can partially define the display registration volume. As Figure 17A - 17H shown in FIGS. 13 and 18, the size and shape of the display registration volume may depend in part on the geometry of the display (which may be the Figure 2 display 220) and the angle at which the out-coupled light propagates out of the display (e.g., the waveguide through which the light propagates out of the display). It will be understood that the angle at which the light is output may define the FOV of the display; a larger angle relative to the normal provides a larger FOV. In some embodiments, the display surface may output at an angle large enough to provide the desired FOV.

[0351] Figure 17A - 17H and Figure 18 shows the display 220, which may be a light field display, and includes elements such as a waveguide 1701, an in-coupling element 1702, an orthographic pupil expander (OPE) 1704, and an exit pupil expander (EPE) 1706 (which may form the display surface 1202, which is also shown in various other figures including Figure 12A - 14 ). As an example, the in-coupling element 1702 may receive light from an image source and couple the light into the waveguide 1701. The waveguide 1701 may transmit the light to the OPE 1704, which may provide pupil expansion and direct the light to the EPE 1706, which (which may be disposed on the display surface 1202) provides further pupil expansion and transmits the light to the user's eye. Further examples and details of light field displays and components of such displays are also described in U.S. Provisional Patent Application No. 62 / 642,761, filed on Mar. 14, 2018, the entire content of which is incorporated herein by reference. Figure 9A - 9C The entire content of which is incorporated herein by reference.

[0352] Figure 17AAn example is shown in which the display 220 projects light 1710 associated with virtual image content at optical infinity and in the rightmost region (e.g., the rightmost pixel) of the FOV of the display. In contrast, Figure 17B An example is shown in which the display 220 projects light 1712 associated with an object at optical infinity and in the leftmost region (e.g., the leftmost pixel) of the FOV of the display. Figure 17C An example is shown Figure 17A of light 1710 and Figure 17B of light 1712 that overlap in an overlapping region 1714. The region 1714 can be a horizontal registration volume. In particular, when the user's eyes are located within Figure 17C the region 1714, the user can perceive objects in the rightmost region of the FOV (as shown in Figure 17A ) and in the leftmost region of the FOV (as shown in Figure 17B ). (For example, the display 220 can provide light from the object to the user.)

[0353] Figure 17D -F shows examples similar to Figure 17A - 17E these examples, except in the vertical direction. In particular, Figure 17D an example is shown in which the display 220 projects light 1716 associated with an object at optical infinity and in the bottommost region (e.g., the bottommost pixel) of the FOV of the display, while Figure 17E an example is shown in which the display 220 projects light 1718 associated with an object at optical infinity and in the topmost region (e.g., the bottommost pixel) of the FOV of the display. Similarly, Figure 17F an example is shown Figure 17D of light 1716 and Figure 17E of light 1718 that overlap in an overlapping region 1720. The region 1720 can be a vertical registration volume. In particular, when the user's eyes are located within Figure 17F the region 1720, the user can perceive objects in the bottommost region of the FOV (as shown in Figure 17D ) and in the topmost region of the FOV (as shown in Figure 17E ). (For example, the display 220 can provide light from the object to the user.)

[0354] Figure 17G and 17H show an intersection (as region 1722) of Figure 17C the region 1714 and Figure 17F the region 1720. In particular, Figure 17G an example is shown of a region 1722 where light from objects at the four corners of the FOV of the display 220 overlaps. Figure 17HOnly the outline of region 1722 is shown. Obviously, when the user's eyes are within region 1722, the user can perceive objects anywhere within the FOV of the display (e.g., display 220 can provide light from the object to the user). In some embodiments, the registration volume of display 220 can be understood as the viewing volume of the head-mounted display, representing the light of each pixel of the virtual image content presented by the head-mounted display that is expected to pass through this viewing volume.

[0355] In some embodiments, increasing the FOV (horizontal, vertical, or a combination thereof) of display 220 while keeping other properties (such as display size) constant can have the effect of shrinking the associated registration volume (e.g., horizontal volume 1714, vertical volume 1720, or combined registration volume 1722). As an example, consider Figure 17A -C and the horizontal FOV and registration volume 1714. An increase in the horizontal FOV of display 220 means that light 1710 from an object on the right horizontal edge is projected by display surface 1202 (e.g., EPE 1706) at a sharper angle (e.g., a greater angle from the normal to display surface 1202). Similarly, light 1712 from an object on the left horizontal edge is projected at a sharper angle. Thus, Figure 17C from the perspective of, the vertex of the horizontal registration volume 1714 moves towards display surface 1202 as the horizontal FOV increases, thereby shrinking volume 1714. In some embodiments, similar considerations can be applied to the vertical FOV and vertical registration volume, as well as the overall FOV and overall registration volume.

[0356] Figure 18 A top view of display 220 including display surface 1202 is shown. Display surface 1202 can have a rectangular shape and a specific FOV, as well as the light rays generated by the display. Generally, Figure 18 the registration volume of display 220 can be volume 1802, which is shown as a triangle in the Figure 18 top view. Volume 1802 can represent the volume where various light fields formed by the Figures 17A - 17G shown light overlap. If the user's eyes are outside volume 1802 (e.g., in volume 1804), then the light of the light field from at least some parts of display 220 that can be seen will not reach the user's eyes, resulting in part or all of the FOV being dimmed.

[0357] It should be noted that the side view of the display and the registration volume will have an appearance that is almost the same as the Figure 18 shown appearance (at least for a rectangular display), although the size of display 220 shown will be the height of display 220 rather than its width, and the FOV shown will be Figure 18The vertical FOV as shown instead of the horizontal FOV. Thus, volume 1802 may actually have a somewhat pyramidal shape. In other embodiments, the display may have a non-rectangular shape, such as circular, oval, freeform, or any other desired shape. In such embodiments, the corresponding registration volume may be determined by projecting light fields at the relevant FOVs and identifying the locations where those light fields intersect (which may correspond to volume 1802) and the locations where the light fields do not intersect (which may correspond to volume 1804).

[0358] As discussed herein, the "bottom" of the pyramid may be truncated (which may help to move the user's eyes away from the display so that the user's eyelashes do not affect the display when properly registered), and the "top" of the pyramid may also be truncated (which may help to reduce the effect of noise in the determination of the user's eye position, which might otherwise quickly move in and out of registration at the "top" of the pyramidal registration volume). It should be understood that the "top" is near the vertex of volume 1802, and the bottom is near waveguide 1701. When the user's eyes are in region 1804 outside of registration volume 1802, the user may perceive dimming of some or all of the pixels of display 220, as discussed herein (e.g., see Figure 14 ).

[0359] Generally, for various reasons, the registration volume may be adjusted (e.g., truncated or otherwise reduced) in any number of ways. As an example, the registration volume may be truncated such that the volume has a minimum distance from display 220 to prevent the user's eyelashes or eyelids from affecting display 220. Thus, in some embodiments, the display system (e.g., the processing electronics of the display system) may be configured to determine whether the user's eyes are within registration volume 1802 by at least partially determining whether one or both eyes are less than a minimum threshold distance (e.g., a minimum allowable distance) from display 220. If it is determined that the eyes are at a distance less than the minimum threshold from the display, the display system may interpret this result as meaning that the eyes are outside of registration volume 1802, and thus, the display and the eyes are not properly registered. As a result, as discussed herein, the display system may provide feedback to the user indicating incorrect registration, and / or may be configured to take measures to mitigate display degradation caused by the misregistration. In some implementations, such a minimum threshold distance may vary in one or more dimensions. For example, the minimum threshold distance may vary linearly along the z-axis, which is a function of the distance from a nominal position and / or the surface of the display.

[0360] As a supplement to or alternative for determining whether the eyes are within a minimum distance from the display 220, in some embodiments, the display system (e.g., the processing electronics of the display system) may be configured to determine whether a user's eyes are within the registration volume 1802 by at least partially determining whether one or both eyes are greater than a maximum threshold distance from the display 220. It will be understood that the maximum threshold distance may correspond to the distance at which the "top" of the pyramid 1802 mentioned above is truncated. If it is determined that the eyes are at a distance from the display that is greater than the maximum threshold distance, the display system may interpret this result as meaning that the eyes are outside the registration volume 1802, and thus, the display and the eyes are not properly registered. As a result, as discussed herein, the display system may provide feedback to the user indicating improper registration, and / or may be configured to take measures to mitigate display degradation caused by the misregistration.

[0361] In addition to or as an alternative to determining whether the eyes are within a minimum distance from the display 220 and / or beyond a maximum distance, the wearable system may also have an eye tracking system including elements such as Figure 6 a camera 324 and a light source 326, etc., which can track the user's eyes only when the user's eyes are within the eye tracking volume, and the eye tracking volume may not completely overlap with the display registration volume. In some embodiments, the camera of the eye tracking system may have a field of view that includes the display registration volume. Thus, the display registration volume may be regarded as a subspace or a part of the field of view of the camera. The display system may be configured to determine whether the user's eyes are within this subspace or within a threshold distance from this subspace (when imaged by the camera). If the eyes are within the subspace or within the threshold distance from the subspace, the display system may interpret this result as meaning that the eyes are within the display registration volume. If the eyes are outside the subspace or outside the threshold distance from the subject, the display system may interpret this result as meaning that the eyes are outside the display registration volume. If the display system determines that the eyes are outside the display registration volume, the display system may be configured to provide feedback to the user indicating that the display and the eyes are not properly registered for output, and / or may be configured to take measures to mitigate display degradation due to misregistration, as discussed herein.

[0362] Example of a housing registration volume and a system registration volume

[0363] Figure 19AAn example housing registration volume 1900 is shown. Light from the surrounding environment passes through an opening in the bezel or housing of the display to reach the user. It should be understood that the bezel or housing of the display may block certain ambient light from reaching the user's eyes from certain angles. As a result, similar to the display registration volume, and the housing registration volume 1900 may represent the volume of space within which the user's eyes are positioned to receive light from the external environment to obtain the entire available field of view through the housing or bezel of the display. In some embodiments, the center of rotation of the user's eyes is preferably located within the housing registration volume 1900 such that the eyes receive light from the external environment at an angle corresponding to the entire available field of view. In some embodiments, when the center of rotation of the user's eyes is located within the housing registration volume 1900, the user is able to see an acceptable portion of the world around them. Properly registering the user's eyes to the wearable system (e.g., by providing a center of rotation within the housing registration volume 1900) can help reduce the likelihood that the user will not see an obstacle in their path, can provide the user with overlapping left and right fields of view to facilitate binocular vision, and / or can provide the user with a more comfortable visual experience.

[0364] As Figure 19A shown, the housing registration volume 1900 may be determined in part or in whole with reference to the housing opening 1510, which may be an opening (or lens) in the bezel 230 of the wearable system, as previously discussed in connection with Figure 15A . The housing registration volume 1900 may, for example, have a conical shape, the base of which is defined by the shape and dimensions of the housing opening 1510. The portion of the cone closest to the housing opening 1510 and the portion furthest from the housing opening 1510 may be truncated (e.g., excluded from the housing registration volume 1900), which can help move the user's eyes away from the display and the housing opening 1510 such that the user's eyelashes do not affect the display during proper registration and can help reduce the effect of noise on the determination of the position of the user's eyes, which might otherwise quickly move into or out of the small volume at the "top" of the conical registration volume. In some embodiments, the display system may be configured to determine whether the user's eyes (e.g., the center of rotation of the eyes) are within the housing registration volume 1900 and, if the user's eyes are outside the registration volume 1900, provide an unregistered notification.

[0365] Figure 19B Shown superimposed on the housing registration volume 1900 (associated with the housing opening 1510) of FIG. 1900 is the Figure 13A display registration volume 1302a (associated with the display surface 1202). In at least some embodiments, it may be desirable for the center of rotation of the user's eyes to be located within the Figure 19A housing registration volume 1900 and such as Figure 13Awithin the display registration volume of the display registration volume 1302a (or any other display registration volume discussed herein). When the user's eyes are within the housing and the display registration volume, the user may be able to receive image information from the display device (the display device provides a full field of view), while also being able to view the entire field of view of the external environment provided by the housing.

[0366] Figure 19C An example of the combined registration volume 1902 is shown, where each point in the combined registration volume 1902 is within the housing and the display registration volume (e.g., within the displacement registration volume 1302a and the housing registration volume 1900). As Figure 19B and 19C shown, and in at least some embodiments, the display registration volume 1302a may generally be smaller than the housing registration volume 1900 (e.g., the housing registration volume 1900 may be smaller only at the corners of the display registration volume 1302a). In such embodiments, the combined registration volume 1902 may have a shape similar to a truncated pyramid with rounded corners, as Figure 19C shown. In some embodiments, the display system may be configured to determine whether the user's eyes (e.g., the center of rotation of the eyes) are within the housing registration volume 1900, within the display registration volume 1204, or within both the housing registration volume 1900 and the display registration volume 1204. If the user's eyes are outside the particular volume being analyzed, the display system may be configured to provide an unregistered notification if the user's eyes are outside the analyzed registration volume (e.g., the housing registration volume 1900, the display registration volume 1204, or both the housing registration volume 1900 and the display registration volume 1204).

[0367] For example, in some embodiments, the combined overlapping registration volume defined by the housing registration volume 1900 and the display registration volume 1204 may be analyzed to determine whether the user's eyes are within the combined registration volume. In some embodiments, this may be understood as a part or subspace of the registration or viewing volume of the housing, which may also be referred to as the outer shell. The display system may be configured to determine whether the user's eyes are within this subspace (or an acceptable threshold distance outside the subspace). If the display system determines that the position of the eyes is outside the subspace of the viewing volume of the display housing by more than the threshold distance, it may provide feedback to the user indicating that the display and the eyes are not properly registered.

[0368] Computer vision for detecting objects in the surrounding environment

[0369] As discussed above, the display system may be configured to detect objects or characteristics in the environment around the user. As discussed herein, various techniques using various environmental sensors (e.g., cameras, audio sensors, temperature sensors, etc.) may be used to accomplish the detection.

[0370] In some embodiments, computer vision techniques can be used to detect objects present in the environment. For example, as disclosed herein, the forward-facing camera of the display system can be configured to image the surrounding environment, and the display system can be configured to perform image analysis on the image to determine the presence of objects in the surrounding environment. The display system can analyze the image acquired by the outward-facing imaging system to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. As other examples, the display system can be configured to perform face and / or eye recognition to determine the presence and location of faces and / or human eyes in the user's field of view. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include: Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technology, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), histogram of feature points, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithm, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and so on.

[0371] One or more of these computer vision techniques can also be used in conjunction with data obtained from other environmental sensors (e.g., microphones) to detect and determine various characteristics of the objects detected by the sensors.

[0372] As discussed herein, objects in the surrounding environment can be detected based on one or more criteria. When the display system detects the presence or absence of criteria in the surrounding environment using computer vision algorithms or data received from one or more sensor components (which may or may not be part of the display system), the display system can then signal the presence of the object.

[0373] Machine learning

[0374] A variety of machine learning algorithms can be used to learn the presence of objects in the surrounding environment. Once trained, the machine learning algorithms can be stored by the display system. Some examples of machine learning algorithms can include: supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., Apriori algorithm), artificial neural network algorithms (e.g., perceptron), deep learning algorithms (e.g., deep Boltzmann machines, or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., stacking generalization), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual data sets. For example, a wearable device can generate or store a base model. The base model can be used as a starting point to generate additional models specific to a data type (e.g., a specific user), a data set (e.g., a collection of additional images obtained), a conditional situation, or other variations. In some embodiments, the display system can be configured to utilize multiple techniques to generate models for analyzing aggregated data. Other techniques can include using predefined thresholds or data values.

[0375] The criteria for detecting an object can include one or more threshold conditions. If the analysis of data obtained by the environmental sensors indicates that the threshold conditions are met, the display system can provide a signal indicating the detection of the presence of an object in the surrounding environment. The threshold conditions can involve quantitative and / or qualitative measurements. For example, the threshold conditions can include a score or percentage associated with the reflection and / or the likelihood of an object being present in the environment. The display system can compare the score calculated based on the data from the environmental sensors with the threshold score. If the score is higher than the threshold level, the display system can detect the presence of the reflection and / or the object. In some other embodiments, if the score is lower than the threshold, the display system can signal the presence of an object in the environment. In some embodiments, the threshold conditions can be determined based on the user's emotional state and / or the user's interaction with the surrounding environment.

[0376] In some embodiments, the threshold conditions, machine learning algorithms, or computer vision algorithms can be dedicated to a specific environment. For example, in a diagnostic environment, the computer vision algorithm can be dedicated to detecting certain responses to stimuli. As another example, as discussed herein, the display system can perform a face recognition algorithm and / or an event tracking algorithm to sense the user's reaction to a stimulus.

[0377] It will be appreciated that each process, method, and algorithm described herein and / or depicted in the figures can be embodied in and / or automated in whole or in part by one or more physical computing systems, hardware computer processors, specialized circuits, and / or electronic hardware configured to execute specific and particular computer instructions. For example, a computing system can include a general-purpose computer (such as a server) programmed with specific computer instructions, a special-purpose computer, a specialized circuit, etc. Code modules can be compiled and linked into an executable program, installed in a dynamic link library, or can be written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.

[0378] In addition, certain embodiments of the functions of the present disclosure are complex enough mathematically, computationally, or technically that dedicated hardware or one or more physical computing devices (utilizing appropriate dedicated executable instructions) may be required to perform the functions, e.g., due to the amount or complexity of the calculations involved, or to provide results substantially in real time. For example, a video can include many frames, each with millions of pixels, and requires specially programmed computer hardware to process the video data to provide the desired image processing tasks or applications within a commercially reasonable amount of time.

[0379] Code modules or any type of data can be stored on any type of non-transitory computer-readable medium, such as physical computer memory, including hard disk drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical discs, volatile or non-volatile storage devices, combinations thereof and / or similar memories, etc. In some embodiments, the non-transitory computer-readable medium can be part of one or more of a local processing and data module (140), a remote processing module (150), and a remote data repository (160). Methods and modules (or data) can also be sent on various computer-readable transmission media (including wireless-based and wired / cable-based media) as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) and can take many forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps can be permanently or otherwise stored in any type of non-transitory tangible computer memory or can be transmitted via a computer-readable transmission medium.

[0380] Any process, block, state, step, or function in the flowcharts described herein and / or depicted in the figures should be understood as potentially representing code modules, code segments, or portions of code that include one or more executable instructions for implementing a specific function (such as logic or arithmetic) or step in the process. The various processes, blocks, states, steps, or functions may be combined, rearranged, added to the illustrative examples provided herein, deleted from the illustrative examples provided herein, modified, or otherwise changed. In some embodiments, additional or different computing systems or code modules may perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the associated blocks, steps, or states may be performed in a suitable other order (such as serially, in parallel, or in some other manner). Tasks or events may be added to or deleted from the disclosed example embodiments. Additionally, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as required in all embodiments. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products.

[0381] Other considerations

[0382] Each process, method, and algorithm described herein and / or depicted in the figures can be embodied in and / or automated in whole or in part by one or more physical computing systems, hardware computer processors, specialized circuits, and / or electronic hardware configured to execute specialized and specific computer instructions. For example, a computing system can include a general-purpose computer (such as a server) programmed with specific computer instructions or a special-purpose computer, specialized circuit, etc. Code modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, specific operations and methods can be performed by circuits specific to a given function.

[0383] Furthermore, certain implementations of the functions of the present disclosure are sufficiently complex mathematically, computationally, or technically that dedicated hardware or one or more physical computing devices (utilizing appropriate dedicated executable instructions) may be required to perform the functions, such as due to the amount or complexity of the calculations involved or to provide results substantially in real time. For example, an animation or video can include many frames, each with millions of pixels, and requires specially programmed computer hardware to process the video data to provide the desired image processing tasks or applications within a commercially reasonable amount of time.

[0384] Code modules or any type of data can be stored on any type of non - transitory computer - readable medium, such as physical computer memory, including hard disk drives, solid - state memory, random access memory (RAM), read - only memory (ROM), optical discs, volatile or non - volatile storage devices, combinations thereof and / or similar memories, etc. Methods and modules (or data) can also be transmitted on various computer - readable transmission media, including both wireless - based and wire / cable - based media, as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal), and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps or actions can be permanently or otherwise stored in any type of non - transitory tangible computer memory or can be transmitted via a computer - readable transmission medium.

[0385] Any process, block, state, step, or function in the flowcharts described herein and / or depicted in the figures should be understood as potentially representing a code module, code segment, or code portion that includes one or more executable instructions for implementing a particular function (e.g., logical or arithmetic) or step in the process. The various processes, blocks, states, steps, or functions can be combined, rearranged, added to the illustrative examples provided herein, deleted from the illustrative examples provided herein, modified, or otherwise changed. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the associated blocks, steps, or states can be performed in a suitable other order (e.g., serially, in parallel, or in some other way). Tasks or events can be added to or deleted from the disclosed example embodiments. Additionally, the separation of the various system components in the implementations described herein is for illustrative purposes and should not be understood as required in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0386] Processes, methods, and systems can be implemented in a network (or distributed) computing environment. The network environment includes enterprise - wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourcing computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0387] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desired attributes disclosed herein. The various features and processes described above can be used independently of each other or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the implementations described in the present disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein but are to be accorded the broadest scope consistent with the invention, principles, and novel features disclosed herein.

[0388] Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, the various features that are described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable sub-combination. Moreover, although the features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination can be excised from the combination, and the claimed combination can be directed to a sub-combination or a variant of the sub-combination. For each embodiment, no single feature or group of features is necessary or essential.

[0389] The conditional language used herein, such as, without limitation, "can," "could," "might," "may," "for example," etc., unless specifically stated otherwise, is generally understood in the context in which it is used to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that the features, elements, and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements, and / or steps are included or will be performed in any particular embodiment. The terms "comprising," "including," "having," etc. are synonyms and are used inclusively in an open-ended manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), so for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, as used in this application and the appended claims, the articles "a," "an," and "the" are to be construed to mean "one or more" or "at least one" unless otherwise specified.

[0390] As used herein, the phrase referring to "at least one" in a list of items means any combination of those items, including a single member. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless otherwise specifically stated, words such as the phrase "at least one of X, Y, and Z" should be understood in the context which is generally used to convey that the items, terms, etc. can be at least one of X, Y, or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0391] Similarly, although operations may be depicted in the figures in a particular order, it should be recognized that such operations need not be performed in the particular order or sequential order shown, or that all of the shown operations need to be performed to achieve the desired result. Additionally, the figures may schematically depict one or more example processes in the form of a flowchart. However, other operations not shown may be incorporated in the example methods and processes schematically shown. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the shown operations. Additionally, in other implementations, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the above-described implementations should not be understood to be required in all implementations, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

Claims

1. A display system configured to project light into a user's eyes to display virtual image content, the display system comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eyes to display virtual image content having different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times; one or more eye-tracking cameras configured to image the user's eyes; and processing electronics in communication with the display and the one or more eye-tracking cameras, the processing electronics configured to: determine a display registration volume relative to the display based at least in part on a portion of a field of view for displaying virtual content of a currently running application, wherein the display registration volume is a three-dimensional spatial volume defined with reference to the display such that when the user's eyes are within the display registration volume, the virtual image content is received by the user's eyes and such that when the user's eyes are not within the display registration volume, the virtual image content is not received by the user's eyes or the virtual image content is dimmed; determine the position of the eyes based on images of the eyes obtained by the one or more eye-tracking cameras; determine whether the position of the eyes is within the display registration volume; and provide a notification based on determining whether the position of the eyes is within the display registration volume, wherein the notification at least indicates that the display and the eyes are not properly registered.

2. The display system according to claim 1, wherein the processing electronics is further configured to: provide feedback to the user that the head-mounted display is not properly adjusted for the user when it is determined that the position of the eyes is outside the display registration volume, wherein the feedback is the notification provided based on determining whether the position of the eyes is within the display registration volume.

3. The display system according to claim 1, further comprising at least one interchangeable adapter removably mounted to the frame and configured to adjust the fit of the frame.

4. The display system according to claim 3, wherein the interchangeable adapter includes an interchangeable nose bridge configured to adjust the fit of the frame between the frame and the user's nose.

5. The display system according to claim 3, wherein the interchangeable adapter includes an interchangeable forehead pad configured to adjust the fit of the frame between the frame and the user's forehead.

6. The display system according to claim 3, wherein the interchangeable adapter includes an interchangeable rear pad configured to adjust the fit of the frame between the frame and the rear of the user's head.

7. The display system according to claim 3, wherein The processing electronic device is further configured such that providing the notification includes: providing feedback to the user that the head-mounted display is not properly adjusted to fit the user, which includes providing a recommendation to the user to replace a currently installed interchangeable adapter with another interchangeable adapter.

8. The display system according to claim 1, further comprising one or more light sources, the one or more light sources being disposed on the frame relative to the user's eyes to illuminate the user's eyes, and the one or more eye tracking cameras using light from the one or more light sources to form an image of the eyes.

9. The display system according to claim 8, wherein, the one or more light sources include: at least two light sources disposed on the frame relative to the user's eyes to illuminate the user's eyes.

10. The display system according to claim 8, wherein, the one or more light sources include infrared light emitters.

11. The display system according to claim 8, wherein, one or more light sources form one or more flashes on the eyes, and the processing electronic device is configured to determine the position of the cornea based on the one or more flashes.

12. The display system according to claim 8, wherein, the position of the eyes is the position of the center of rotation of the eyes.

13. The display system according to claim 1, wherein, the cornea is associated with a corneal sphere having a center of curvature, and the processing electronic device is configured to determine the position of the center of curvature of the corneal sphere.

14. The display system according to claim 1, wherein, the processing electronic device is configured to provide the notification by providing an instruction to increase the brightness of a plurality of pixels of the display relative to other pixels of the display, wherein the plurality of pixels having increased brightness include pixels that are expected to experience perceivable dimming under incorrect registration.

15. A method for a user's eyes to evaluate the registration of virtual image content from a display of a head-mounted display system, the method comprises: determining a display registration volume relative to the display based at least in part on a portion of a field of view for displaying virtual content of a currently running application, wherein the display registration volume is a three-dimensional spatial volume defined with reference to the display such that when the user's eyes are within the display registration volume, the virtual image content is received by the user's eyes, and such that when the user's eyes are not within the display registration volume, the virtual image content is not received by the user's eyes, or the virtual image content is dimmed; determining a first position of the eyes; Determine whether the first position of the eye is within the display registration volume of the head-mounted display system, where the display of the head-mounted display system is configured to project light into the user's eyes to display virtual image content with different amounts of wavefront divergence to present virtual image content that appears to be at different depths at different times, and where the display registration volume is a hypothetical volume associated with the correct alignment of the head-mounted display system relative to the user's eyes; and Provide a notification based on determining whether the position of the eye is within the display registration volume, where the notification at least indicates that the display and the eye are not correctly aligned.

16. The method according to claim 15, wherein, The head-mounted display system includes an eye tracking camera, and determining the first position of the eye includes imaging the user's eye using the eye tracking camera.

17. The method according to claim 16, wherein, The first position of the eye is the position of the center of rotation of the eye, and the method further includes: calculating the center of rotation of the eye based on the imaging of the eye by the eye tracking camera.

18. The method according to claim 15, wherein, The head-mounted display system is configured to project light into the eye to display virtual image content in the user's field of view, and the method further includes: displaying an indication that the head-mounted display system has been correctly aligned.

19. The method according to claim 15, further includes: Automatically tracking the center of rotation of the eye over time by the head-mounted display system and notifying the user when the center of rotation of the eye moves outside the display registration volume.

20. The method according to claim 15, further includes: Determine a second position of the eye; Determine that the second position of the eye is within the display registration volume; and In response to determining that the second position of the eye is within the display registration volume, provide additional feedback to the user indicating that the head-mounted display system is correctly adapted to the user.

21. The method according to claim 15, wherein, When the user's eye is not within the display registration volume, at least some pixels of the head-mounted display system are dimmed or completely dark to the user.

22. The method according to claim 15, further includes: When the position of the eye is outside the display registration volume, change the field of view of the head-mounted display system, where the head-mounted display system includes at least one display that has a first field of view when the position of the eye is within the display registration volume, where the display has a second field of view when the position of the eye is outside the display registration volume, and where the second field of view is less than the first field of view.

23. The method according to claim 22, wherein, Providing the notification includes: providing feedback to the user within the second field of view.

24. The method according to claim 15, wherein, the head-mounted display system includes at least one interchangeable adapter, and the method further includes: providing the user with a notification indicating that the head-mounted display system is not properly adapted to the user, wherein the notification includes a suggestion or indication for the user to replace the currently installed interchangeable adapter with an alternative interchangeable adapter.

25. The method according to claim 24, wherein, the interchangeable adapter includes at least one adapter selected from the following: a nose pad, a forehead pad, and a rear pad, and the rear pad is located between the head-mounted display system and the rear of the user's head.

26. The method according to claim 25, wherein, the head-mounted display system includes at least one interchangeable nose pad, and the method further includes: determining that the display of the head-mounted display system is too low relative to the eyes, and wherein providing the user with the notification includes prompting the user to install a larger nose pad.

27. The method according to claim 15, further includes: identifying a plurality of pixels of the display of the head-mounted display system that the user is expected to perceive as dimmed due to the first position of the eyes being outside the display registration volume; and increasing the brightness of the plurality of pixels of the display relative to other pixels in the display to mitigate the expected dimming.

Images