Display system and method for determining vertical alignment between left and right displays and a user's eyes

Abstract

To provide a display system and method for determining appropriate vertical alignment between left and right displays and user's eyes.SOLUTION: A wearable device may include a head-mounted display (HMD) for rendering a three-dimensional (3D) virtual object which appears to be located in an ambient environment of a user of the display. The relative positions of the HMD and one or more eyes of the user may not be in desired positions to receive image information output by the HMD. For example, the HMD-to-eye vertical alignment may be different between the left and right eyes. The wearable device may determine if the HMD is level on the user's head and may then provide the user with a left-eye alignment marker and a right-eye alignment marker. Based on user feedback, the wearable device may determine if there is any left-right vertical misalignment and may take actions to reduce or minimize the effects of any misalignment.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] (Incorporation by Reference) This application incorporates by reference in its entirety each of the following patent applications: U.S. Provisional Application No. 62 / 714,649, filed Aug. 3, 2018; U.S. Provisional Application No. 62 / 875,474, filed Jul. 17, 2019; U.S. Patent Application No. 16 / 251,017, filed Jul. 17, 2019; and PCT Application No. PCT / US2019 / 043096, filed Jul. 23, 2019.

[0002] The present disclosure relates to display systems, including virtual reality and augmented reality display systems, and more particularly, to systems and methods for aligning left and right displays within a wearable display system.

Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. Virtual reality, or “VR” scenarios, typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or “AR” scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality or “MR” is related to the fusion of the real and virtual worlds to create a new environment in which physical and virtual objects coexist and interact in real time. In summary, the human visual perception system is very complex, and the production of VR, AR, or MR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements is difficult. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Summary of the Invention

Means for Solving the Problems

[0004] Various embodiments of display alignment within a mixed reality system are disclosed.

[0005] In some embodiments, an augmented reality system is provided. The augmented reality system includes a head-mounted display configured to present virtual content by outputting light to a user, an imaging device configured to capture an image of the user's eyes, and at least one processor communicatively coupled to the head-mounted display and the imaging device. The at least one processor is configured to provide a left-eye alignment marker using the left-eye display, provide a right-eye alignment marker using the right-eye display, receive user input, adjust at least one of the left-eye and right-eye alignment markers, and vertically adjust image content within at least one of the left-eye and right-eye displays based on the received user input. In some embodiments, the at least one processor is configured to determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes and provide feedback to the user regarding whether the head-mounted display system is horizontal with respect to the user's left and right eyes before providing the left or right-eye alignment marker.

[0006] In some other embodiments, a method is provided for vertically aligning the content displayed on the left-eye display and the right-eye display of a head-mounted display system using a user's left and right eyes. The method includes providing a left-eye alignment marker using the left-eye display, providing a right-eye alignment marker using the right-eye display, receiving user input, adjusting at least one of the left-eye and right-eye alignment markers, and vertically adjusting the image content within at least one of the left-eye and right-eye displays based on the received user input. In some embodiments, the method further includes determining the levelness of the head-mounted display system with respect to the user's left and right eyes, and providing feedback to the user regarding whether the head-mounted display system is level with respect to the user's left and right eyes prior to providing the left or right-eye alignment marker.

[0007] In some embodiments, an augmented reality system is provided. The augmented reality system includes a head-mounted display (HMD) configured to present virtual content by outputting light to a user, and at least one processor communicatively coupled to the HMD. The HMD includes a left-eye display configured to present virtual content to the user's left eye and a right-eye display configured to present virtual content to the user's right eye. The at least one processor is configured to provide a left-eye alignment marker using the left-eye display, provide a right-eye alignment marker using the right-eye display, receive user input, adjust at least one of the left-eye and right-eye alignment markers, and vertically adjust the image content within at least one of the left-eye and right-eye displays based on the received user input.

[0008] In some embodiments, an augmented reality system is provided. The augmented reality system includes a head-mounted display (HMD) configured to present virtual content by outputting light to a user, an imaging system configured to capture an image of the user's eyes, and at least one processor communicatively coupled to the HMD and the imaging device. The at least one processor is configured to determine, at least in part, an interpupillary axis of the user that extends between the user's left and right eyes based on one or more images captured by the imaging system, determine an orientation of the HMD relative to the user's interpupillary axis, and provide feedback to the user based on the determined orientation of the HMD relative to the user's interpupillary axis.

[0009] Additional embodiments are listed below.

[0010] (Example 1) An augmented reality system, a head-mounted display configured to present virtual content by outputting light to a user, an imaging device configured to capture an image of the user's eyes, at least one processor communicatively coupled to the head-mounted display and the imaging device, providing a left-eye alignment marker using a left-eye display, providing a right-eye alignment marker using a right-eye display, receiving user input and adjusting at least one of the left-eye and right-eye alignment markers, vertically adjusting image content within at least one of the left-eye and right-eye displays based on the received user input, at least one processor configured as such, a system comprising.

[0011] (Example 2) The processor is The head-mounted display system determines whether it is horizontal with respect to the user's left and right eyes, and provides feedback to the user regarding whether the head-mounted display system is horizontal with respect to the user's left and right eyes before providing a left-eye alignment marker using the left-eye display. The augmented reality system according to Example 1, configured as described above.

[0012] (Example 3) The processor is configured to provide feedback to the user regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a horizontal marker having an orientation that changes in relation to the orientation of the head-mounted display with respect to the user's left and right eyes, as described in Example 1.

[0013] (Example 4) The augmented reality system according to Example 1, further comprising an eye-tracking system, wherein the processor is configured to determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes based on eye-tracking data from the eye-tracking system.

[0014] (Example 5) The augmented reality system according to Example 1, further comprising an eye-tracking system, wherein the processor is further configured to determine the inter-pupillary axis of the user, which extends between the user's left and right eyes, based on eye-tracking data from the eye-tracking system.

[0015] (Example 6) The augmented reality system according to Example 5, wherein the processor is further configured to determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes by determining the orientation of the eye-tracking system with respect to the user's inter-pupillary axis.

[0016] (Example 7) The processor is configured to provide feedback to the user regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a horizontal marker to the user that has an orientation that varies in relation to the orientation of the eye-tracking system with respect to the interpupillary axis, as described in Example 5.

[0017] (Example 8) The processor is configured to provide feedback to the user regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a static horizontal marker associated with the orientation of the head-mounted display and a dynamic horizontal marker associated with the orientation of the interpupillary axis, and the dynamic horizontal marker moves relative to the static horizontal marker as the orientation of the interpupillary axis changes with respect to the head-mounted display, as described in Example 5.

[0018] (Example 9) The dynamic horizontal marker merges with the static horizontal marker when the head-mounted display is horizontal with respect to the interpupillary axis, as described in Example 8.

[0019] (Example 10) The left-eye alignment marker comprises a first horizontal line, and the right-eye alignment marker comprises a second horizontal line, as described in Example 1.

[0020] (Example 11) The processor is configured to receive user input in the form of user input for raising or lowering at least one of the first and second horizontal lines and to adjust at least one of the left-eye and right-eye alignment markers, as described in Example 10.

[0021] (Example 12) The processor receives user input in the form of user input to raise or lower at least one of the first and second horizontal lines until the first and second horizontal lines are horizontal from the user's perspective, and is configured to adjust at least one of the left and right eye alignment markers, the augmented reality system described in Example 10.

[0022] (Example 13) The head-mounted display includes a first waveguide stack configured to pass light from the world into the user's left eye and a second waveguide stack configured to pass light from the world into the user's right eye, and each waveguide stack includes a plurality of waveguides, the augmented reality system described in Example 1.

[0023] (Example 14) The head-mounted display includes a first waveguide stack configured to pass light from the world into the user's left eye and a second waveguide stack configured to pass light from the world into the user's right eye, each waveguide stack includes a plurality of waveguides, and one or more waveguides of the waveguide stack are configured to output light to the user with a different amount of wavefront divergence than one or more other waveguides of the waveguide stack, the different amounts of wavefront divergence are associated with different accommodation by the eye, the output light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the user, the augmented reality system described in Example 1.

[0024] (Example 15) A method for vertically aligning the displayed content of the left-eye display and the right-eye display of a head-mounted display system using the user's left and right eyes, providing a left-eye alignment marker using the left-eye display, providing a right-eye alignment marker using the right-eye display, Receiving user input and adjusting at least one of the left-eye and right-eye alignment markers; Vertically adjusting image content within at least one of the left-eye and right-eye displays based on the received user input; A method comprising the above.

[0025] (Example 16) Determining the horizontal level of a head-mounted display system with respect to a user's left and right eyes; Providing feedback to the user regarding whether the head-mounted display system is horizontal with respect to the user's left and right eyes before providing a left or right-eye alignment marker; The method according to Example 15, further comprising the above.

[0026] (Example 17) The head-mounted display system includes an eye-tracking system, and the step of determining the horizontal level of the head-mounted display system with respect to the user's left and right eyes includes using the eye-tracking system to determine the interocular axis of the user that extends between the user's left eye and right eye, and determining the horizontal level of the head-mounted display system with respect to the interocular axis. The method according to Example 15.

[0027] (Example 18) The step of using the eye-tracking system to determine the interocular axis of the user includes: Using the eye-tracking system to determine the center of rotation of the user's left eye; Using the eye-tracking system to determine the center of rotation of the user's right eye; Determining the position of a line extending between the centers of rotation of the user's left and right eyes, the line constituting the interocular axis; The method according to Example 17, further comprising the above.

[0028] (Example 19) The step of providing the left-eye alignment marker includes the step of providing a first vertical alignment marker and a first horizontal alignment marker using a left-eye display, and the step of providing the right-eye alignment marker includes the step of providing a second vertical alignment marker and a second horizontal alignment marker using a right-eye display, the method according to Example 15.

[0029] (Example 20) When viewed by the user, the first and second vertical alignment markers fuse together within the user's vision, and the first and second horizontal alignment markers remain unfused within the user's vision, the method according to Example 19.

[0030] (Example 21) The step of receiving user input for adjusting at least one of the left-eye and right-eye alignment markers includes the step of receiving user input for vertically moving at least one of the first and second horizontal alignment markers, the method according to Example 19.

[0031] (Example 22) The step of receiving user input for adjusting at least one of the left-eye and right-eye alignment markers includes the step of receiving user input for vertically moving at least one of the first and second horizontal alignment markers until the first and second horizontal alignment markers are vertically aligned with each other within the user's vision, the method according to Example 19.

[0032] (Example 23) The step of determining that the user is wearing the head-mounted display system for a given threshold time, and

[0033] The step of performing the step of determining the levelness of the head-mounted display system in response to the determination that the user is wearing the head-mounted display system for a given threshold time, and further comprising the method according to Example 15.

[0034] (Example 24) An augmented reality system, A head-mounted display configured to present virtual content by outputting light to a user, the head-mounted display comprising a left-eye display configured to present virtual content to the user's left eye and a right-eye display configured to present virtual content to the user's right eye, At least one processor communicatively coupled to the head-mounted display, Providing a left-eye alignment marker using the left-eye display, Providing a right-eye alignment marker using the right-eye display, Receiving user input and adjusting at least one of the left-eye and right-eye alignment markers, Vertically adjusting image content within at least one of the left-eye and right-eye displays based on the received user input, At least one processor configured as such, An augmented reality system comprising.

[0035] (Example 25) The augmented reality system according to Example 24, wherein the left-eye alignment marker comprises a first horizontal line and the right-eye alignment marker comprises a second horizontal line.

[0036] (Example 26) The processor is configured to receive user input and adjust at least one of the left-eye and right-eye alignment markers in a form of user input for raising or lowering at least one of the first and second horizontal lines, the augmented reality system according to Example 25.

[0037] (Example 27) The processor receives user input in the form of user input to raise or lower at least one of the first and second horizontal lines until the first and second horizontal lines are horizontal from the user's perspective, and is configured to adjust at least one of the left-eye and right-eye alignment markers, the augmented reality system described in Example 25.

[0038] (Example 28) The left-eye display includes a first waveguide stack configured to pass light from the world into the user's left eye, and the right-eye display includes a second waveguide stack configured to pass light from the world into the user's right eye, and each waveguide stack includes a plurality of waveguides, the augmented reality system described in Example 24.

[0039] (Example 29) The left-eye display includes a first waveguide stack configured to pass light from the world into the user's left eye, and the right-eye display includes a second waveguide stack configured to pass light from the world into the user's right eye, each waveguide stack includes a plurality of waveguides, and one or more waveguides of the waveguide stack are configured to output light to the user with a different amount of wavefront divergence than one or more other waveguides of the waveguide stack, the different amounts of wavefront divergence are associated with different accommodation by the eye, the output light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the user, the augmented reality system described in Example 24.

[0040] (Example 30) To provide a left-eye alignment marker and a right-eye alignment marker using the left-eye display and the right-eye display respectively, at least one processor uses the left-eye display to provide a first vertical alignment marker and a first horizontal alignment marker, Using the right-eye display, providing a second vertical alignment marker and a second horizontal alignment marker, An augmented reality system according to Example 24, configured as such.

[0041] (Example 31) When viewed by the user, the first and second vertical alignment markers fuse together within the user's vision, and the first and second horizontal alignment markers remain unfused within the user's vision. An augmented reality system according to Example 30.

[0042] (Example 32) To receive user input and adjust at least one of the left-eye and right-eye alignment markers, at least one processor is configured to receive user input for vertically moving at least one of the first and second horizontal alignment markers. An augmented reality system according to Example 30.

[0043] (Example 33) To receive user input and adjust at least one of the left-eye and right-eye alignment markers, at least one processor is configured to receive user input for vertically moving at least one of the first and second horizontal alignment markers until the first and second horizontal alignment markers are vertically aligned with each other within the user's vision. An augmented reality system according to Example 30.

[0044] (Example 34) At least one processor further selects a first vertical position for presenting the left-eye alignment marker, selects a second vertical position for presenting the right-eye alignment marker, configured as such. To provide the left-eye alignment marker and the right-eye alignment marker using the left-eye display and the right-eye display respectively, at least one processor uses the left-eye display to provide the left-eye alignment marker at the first vertical position, Using the right-eye display, providing a right-eye alignment marker at a second vertical position, An augmented reality system according to Example 24, configured as such.

[0045] (Example 35) To select a first vertical position for presenting a left-eye alignment marker and a second vertical position for presenting a right-eye alignment marker, at least one processor Selects, pseudo-randomly or quasi-randomly, a first vertical position for presenting a left-eye alignment marker, An augmented reality system according to Example 24, configured to select, pseudo-randomly or quasi-randomly, a second vertical position for presenting a right-eye alignment marker.

[0046] (Example 36) To vertically adjust the image content in at least one of the left-eye and right-eye displays based on the received user input, at least one processor An augmented reality system according to Example 24, configured to vertically adjust the image content in at least one of the left-eye and right-eye displays based on the received user input, the selected first vertical position, and the selected second vertical position.

[0047] (Example 37) To vertically adjust the image content in at least one of the left-eye and right-eye displays based on the received user input, at least one processor An augmented reality system according to Example 24, configured to adjust one or more accessory parameters of at least one of a first rendering camera associated with the left-eye display and a second rendering camera associated with the right-eye display.

[0048] (Example 38) An augmented reality system according to Example 37, wherein the one or more accessory parameters include at least one of position and orientation.

[0049] (Example 39) Further comprising an imaging system configured to capture an image of the user's eye, At least one processor is communicatively coupled to the imaging system, and the at least one processor is further configured to, Based at least in part on one or more images captured by the imaging system, determine whether the head-mounted display is horizontal with respect to the user's left and right eyes, an augmented reality system according to Example 24.

[0050] (Example 40) In response to determining that the head-mounted display is horizontal with respect to the user's left and right eyes, at least one processor is configured to provide a left-eye alignment marker and a right-eye alignment marker, respectively, using a left-eye display and a right-eye display, an augmented reality system according to Example 39.

[0051] (Example 41) The left-eye display includes a first waveguide stack, and the right-eye display includes a second waveguide stack, each of which includes a plurality of waveguides configured to output light to the user, To provide a left-eye alignment marker and a right-eye alignment marker, respectively, using the left-eye display and the right-eye display, at least one processor, Provide a left-eye alignment marker using a single one of the plurality of waveguides in the first waveguide stack, Provide a right-eye alignment marker using a single one of the plurality of waveguides in the second waveguide stack, An augmented reality system according to Example 24.

[0052] (Example 42) An augmented reality system, A head-mounted display configured to present virtual content by outputting light to the user, and An imaging system configured to capture an image of the user's eye, At least one processor communicatively coupled to a head-mounted display and an imaging system, configured to: Determine an interpupillary axis of a user extending between the user's left and right eyes, at least in part based on one or more images captured by the imaging system; Determine an orientation of the head-mounted display relative to the user's interpupillary axis; Provide feedback to the user based on the determined orientation of the head-mounted display relative to the user's interpupillary axis; At least one processor configured as such; and An augmented reality system comprising the same.

[0053] (Example 43) To provide feedback to the user based on the determined orientation of the head-mounted display relative to the user's interpupillary axis, at least one processor is configured to: Present a horizontal marker having an orientation that varies in relation to the determined orientation of the head-mounted display with respect to the user's left and right eyes, as described in Example 42.

[0054] (Example 44) To provide feedback to the user based on the determined orientation of the head-mounted display relative to the user's interpupillary axis, at least one processor is configured to: Present a static horizontal marker associated with the orientation of the head-mounted display; Present a dynamic horizontal marker associated with the orientation of the interpupillary axis, the dynamic horizontal marker moving relative to the static horizontal marker as the orientation of the interpupillary axis changes relative to the head-mounted display; As described in Example 42.

[0055] (Example 45) The dynamic horizontal marker merges with the static horizontal marker when the head-mounted display is horizontal relative to the interpupillary axis, as described in Example 44.

[0056] (Example 46) At least one processor is configured to determine an inter-ocular axis of a user, extending between the user's left and right eyes, based at least in part on one or more images captured by an imaging system, determine a center of rotation of the user's left eye based at least in part on one or more images of the user's left eye captured by the imaging system, determine a center of rotation of the user's right eye based at least in part on one or more images of the user's right eye captured by the imaging system, determine a position of a line extending between the centers of rotation of the user's left and right eyes, the line constituting the user's inter-ocular axis, an augmented reality system according to Example 42, configured as such.

[0057] (Example 47) At least one processor is further configured to determine whether the user is wearing a head-mounted display for at least a threshold amount of time, and in response to determining that the user is wearing a head-mounted display for at least a threshold amount of time, at least one processor is configured to determine an inter-ocular axis of the user, extending between the user's left and right eyes, based at least in part on one or more images captured by the imaging system, an augmented reality system according to Example 42.

[0058] (Example 48) At least one processor is further configured to determine whether a head-mounted display is horizontal with respect to the user's left and right eyes based on a determined orientation of the head-mounted display relative to the user's inter-ocular axis, an augmented reality system according to Example 42.

[0059] (Example 47) At least one processor is further configured to, in response to a determination that the head-mounted display is horizontal with respect to the user's left and right eyes, provide a left-eye alignment marker using the left-eye display of the head-mounted display, provide a right-eye alignment marker using the right-eye display of the head-mounted display, receive user input and adjust at least one of the left-eye and right-eye alignment markers, vertically adjust image content within at least one of the left-eye and right-eye displays based on the received user input, an augmented reality system according to Example 46, configured as such.

[0060] (Example 48) To provide feedback to the user based on the determined orientation of the head-mounted display with respect to the user's interpupillary axis, at least one processor is

[0061] configured to provide feedback to the user indicating whether the head-mounted display is horizontal with respect to the user's left and right eyes, an augmented reality system according to Example 46.

[0062] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, the drawings, and the claims. Neither this summary nor any of the forms for carrying out the invention below purports to define or limit the scope of the subject matter of the invention. The present invention provides, for example, the following. (Item 1) An augmented reality system, a head-mounted display configured to present virtual content by outputting light to a user, an imaging device configured to capture an image of the user's eye, At least one processor communicably coupled to the head-mounted display and the imaging device, the at least one processor being configured to: Provide a left-eye alignment marker using the left-eye display; Provide a right-eye alignment marker using the right-eye display; Receive user input and adjust at least one of the left-eye alignment marker and the right-eye alignment marker; Vertically adjust image content within at least one of the left-eye display and the right-eye display based on the received user input; At least one processor configured to perform the above; An augmented reality system comprising the above. (Item 2) The processor is configured to: Determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes; Before providing the left-eye alignment marker or the right-eye alignment marker, provide feedback to the user regarding whether the head-mounted display system is horizontal with respect to the user's left and right eyes; The augmented reality system according to Item 1, configured to perform the above. (Item 3) The processor is configured to provide feedback regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a horizontal marker having an orientation that changes in relation to the orientation of the head-mounted display with respect to the user's left and right eyes to the user. The augmented reality system according to Item 1, configured to perform the above. (Item 4) The augmented reality system according to Item 1, further comprising an eye tracking system, wherein the processor is configured to determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes based on eye tracking data from the eye tracking system. (Item 5) The augmented reality system according to item 1, further comprising an eye tracking system, wherein the processor is further configured to determine an interocular axis of the user extending between the user's left and right eyes based on eye tracking data from the eye tracking system. (Item 6) The augmented reality system according to item 5, wherein the processor is further configured to determine whether the head-mounted display system is horizontal with respect to the user's left and right eyes by determining an orientation of the eye tracking system with respect to the interocular axis of the user. (Item 7) The augmented reality system according to item 5, wherein the processor is configured to provide feedback to the user regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a horizontal marker having an orientation that changes in relation to the orientation of the eye tracking system with respect to the interocular axis of the user. (Item 8) The augmented reality system according to item 5, wherein the processor is configured to provide feedback to the user regarding whether the head-mounted display is horizontal with respect to the user's left and right eyes by presenting a static horizontal marker associated with the orientation of the head-mounted display and a dynamic horizontal marker associated with the orientation of the interocular axis, and the dynamic horizontal marker moves relative to the static horizontal marker as the orientation of the interocular axis changes with respect to the head-mounted display. (Item 9) The augmented reality system according to item 8, wherein the dynamic horizontal marker merges with the static horizontal marker when the head-mounted display is horizontal with respect to the interocular axis. (Item 10) The augmented reality system according to item 1, wherein the left-eye alignment marker comprises a first horizontal line and the right-eye alignment marker comprises a second horizontal line. (Item 11) The processor of the augmented reality system according to item 10 is configured to receive a user input in a form of user input for raising or lowering at least one of the first and second horizontal lines, and to adjust at least one of the left-eye alignment marker and the right-eye alignment marker. (Item 12) The processor of the augmented reality system according to item 10 is configured to receive a user input in a form of user input for raising or lowering at least one of the first and second horizontal lines until the first and second horizontal lines are horizontal from the user's perspective, and to adjust at least one of the left-eye alignment marker and the right-eye alignment marker. (Item 13) The head-mounted display of the augmented reality system according to item 1 includes a first waveguide stack configured to pass light from the world into the user's left eye, and a second waveguide stack configured to pass light from the world into the user's right eye, and each waveguide stack includes a plurality of waveguides. (Item 14) The head-mounted display includes a first waveguide stack configured to pass light from the world into the user's left eye, and a second waveguide stack configured to pass light from the world into the user's right eye. Each waveguide stack includes a plurality of waveguides, and one or more waveguides of the waveguide stack are configured to output light to the user with a different amount of wavefront divergence than one or more other waveguides of the waveguide stack. The different amounts of wavefront divergence are associated with different focusing adjustments by the eye. The output light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the user. The augmented reality system according to item 1. (Item 15) A method for vertically aligning the content displayed on the left-eye display and the right-eye display of a head-mounted display system using a user's left eye and right eye, the method comprising: providing a left-eye alignment marker using the left-eye display; providing a right-eye alignment marker using the right-eye display; receiving user input and adjusting at least one of the left-eye alignment marker and the right-eye alignment marker; vertically adjusting image content in at least one of the left-eye display and the right-eye display based on the received user input. A method as described above. (Item 16) determining the level of the head-mounted display system with respect to the user's left eye and right eye; before providing the left-eye alignment marker or the right-eye alignment marker, providing feedback to the user regarding whether the head-mounted display system is level with respect to the user's left eye and right eye. The method according to item 15, further comprising the above. (Item 17) The head-mounted display system comprises an eye-tracking system, and determining the level of the head-mounted display system with respect to the user's left eye and right eye includes using the eye-tracking system to determine the user's interpupillary axis extending between the user's left eye and right eye, and determining the level of the head-mounted display system with respect to the interpupillary axis. The method according to item 15. (Item 18) Determining the user's interpupillary axis using the eye-tracking system includes: determining the center of rotation of the user's left eye using the eye-tracking system; determining the center of rotation of the user's right eye using the eye-tracking system; Determining a position of a line extending between rotational centers of a left eye and a right eye of the user, the line constituting an inter-ocular axis The method according to item 17, including this (Item 19) Providing the left-eye alignment marker includes providing a first vertical alignment marker and a first horizontal alignment marker using the left-eye display, and providing the right-eye alignment marker includes providing a second vertical alignment marker and a second horizontal alignment marker using the right-eye display, the method according to item 15 (Item 20) When visually recognized by the user, the first and second vertical alignment markers fuse together within the user's vision, and the first and second horizontal alignment markers remain unfused within the user's vision, the method according to item 19 (Item 21) Receiving user input for adjusting at least one of the left-eye alignment marker and the right-eye alignment marker includes receiving user input for vertically moving at least one of the first and second horizontal alignment markers, the method according to item 19 (Item 22) Receiving user input for adjusting at least one of the left-eye alignment marker and the right-eye alignment marker includes receiving user input for vertically moving at least one of the first and second horizontal alignment markers until the first and second horizontal alignment markers are vertically aligned with each other within the user's vision, the method according to item 19 (Item 23) Determining that the user is wearing the head-mounted display system for a given threshold time In response to the determination that the user is wearing the head-mounted display system for a given threshold time, performing determining a levelness of the head-mounted display system The method according to item 15, further including this

Brief Description of the Drawings

[0063]

Figure 1

[0064]

Figure 2

[0065]

Figure 3

[0066]

Figure 4

[0067]

Figure 5

[0068]

Figure 6

[0069]

Figure 7A

[0070]

Figure 7B

[0071]

Figure 7C

[0072]

Figure 8A

[0073]

Figure 8B

[0074]

Figure 8C

Figure 8D

Figure 8E

[0075]

Figure 9A

Figure 9B

Figure 9C

[0076]

Figure 9D

Figure 9E

Figure 9F

Figure 9G

[0077]

Figure 10

[0078]

Figure 11

[0079]

Figure 12A

Figure 12B

[0080]

Figure 13

[0081]

Figure 14

[0082]

Figure 15

[0083]

Figure 16

[0084]

Figure 17

[0085]

Figure 18

[0086] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0087] The display system may include a head-mounted display (HMD) that can display virtual objects such that the objects appear to be located within the user's surrounding environment. In some embodiments, the virtual objects may be perceived by the user as being three-dimensional (3D).

[0088] The HMD may include two or more distinct displays or a single continuous display. In either case, different images may be output to the left and right user's eyes. As used herein, references to left and right eye displays, or displays associated with the viewer's right and left eyes, can be understood to refer to display devices configured to output different images to the viewer's left and right eyes. Thus, left and right eye displays can refer to physically separated displays, or a single display configured to output different images to each eye. In some embodiments, the different images may be stereoscopic images presenting slightly different views of the same object or scene, and the different views may be fused together by the human visual system, which can create a perception of depth.

[0089] In some situations, the left and right displays may not be properly vertically aligned with each other and with the user's eyes. Various factors can be combined to determine the position of the left and right eye displays with respect to each other and the user's eyes. For example, the HMD may be deformed or deflected over time, leading to a vertical misalignment between the left and right eye displays. The HMD may be deformed or deflected as a result of a number of factors, including thermal cycling, physical shock (e.g., due to being dropped), elastic or inelastic flexure from rough handling, material degradation, etc. In some cases, one of the left or right eye displays may be rotated and / or vertically translated parallel to the other of the left or right eye displays.

[0090] Such vertical misalignment can degrade the user experience and potentially cause user discomfort. Without being limited by theory, the human visual system does not adapt to receiving vertically misaligned light to form corresponding left and right eye stereoscopic images, and it is conceivable that light propagating towards one eye is vertically offset from the light propagating towards the other eye and can form corresponding stereoscopic images. It should be understood that light propagating towards each eye can form a stereoscopic image specific to that eye. As an example, if light from one display is directed horizontally and reaches one eye, and light from another display is directed downward and reaches the other eye, these displays can be considered vertically misaligned. Even a relatively small vertical misalignment between the left and right eyes can potentially cause discomfort such as harmful eye strain and headaches.

[0091] To reduce viewer discomfort due to vertical misalignment, the difference in vertical alignment between the left eye display and the right eye display can be reduced. For example, the left eye display and the right eye display can be vertically aligned by physically moving those displays. In some cases, such physical movement of the displays is not practical or desirable. Advantageously, in some embodiments, the vertical alignment between the left eye display and the right eye display is in fact achieved by vertically shifting the image content up or down on one or both of the displays. In some embodiments, the image shift can enable the left and right eye displays to have a similar vertical relationship with their corresponding left and right eyes.

[0092] The various systems and techniques described herein are at least partially directed to solving problems related to proper vertical alignment of left and right eye displays that may enable a viewer to comfortably view image content. In some embodiments, a head-mounted display system may be configured to determine the position of a user's eyes (e.g., using components such as an inward-facing imaging system that may be an eye-tracking system). The step of determining the eye position may include determining the position of representative points associated with the eye, such as the individual centers of rotation of the eyes and the position and / or orientation of the interocular axis of the user (e.g., an axis extending between corresponding portions of the user's left and right eyes, such as an axis between the center of rotation of the user's left eye and the center of rotation of the user's right eye). The display system may then determine whether the HMD is horizontal with respect to the user's eyes (e.g., horizontal with respect to the user's interocular axis). After making it horizontal, the display system may present a left-eye alignment marker to the user's left eye and a right-eye alignment marker to the user's right eye. Preferably, the alignment markers are not fused by the human visual system and include a horizontal component that becomes apparent in the vertical misalignment between the left-eye display and the right-eye display. The display system may then solicit feedback from the user to identify and / or adjust the vertical alignment difference between the left-eye display and the right-eye display (e.g., by asking the user to align the left-eye and right-eye alignment markers). Using such feedback, the display system may adjust the content displayed through the system by compensating for any vertical alignment differences identified by the user, and thus improve the user's comfort when viewing the HMD.

[0093] In some embodiments, the display system includes a plurality of waveguides formed in a stack for outputting image information to a user. The alignment process may be performed for each waveguide, as discussed herein. For example, different waveguides may be configured to output different primary colors and / or different amounts of wavefront divergence corresponding to different depth planes, and the alignment may be performed using each primary color (for each corresponding primary color waveguide) and / or each depth plane (for each waveguide with the corresponding amount of wavefront divergence).

[0094] Reference is now made to the drawings, where like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic and are not necessarily drawn to scale. (Example of a 3D display of a wearable system)

[0095] A wearable system (also referred to herein as a head-mounted display system or an augmented reality (AR) system) may be configured to present a 2D or 3D virtual image to a user. The image may be a still image, a frame of a video, or a video in a combination or equivalent. At least a portion of the wearable system may be implemented on a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be used synonymously with an AR device (ARD). Further, for the purposes of the present disclosure, the term "AR" is used synonymously with the term "MR".

[0096] FIG. 1 depicts an illustration of a mixed reality scenario with a virtual object and a physical object that are visually recognized by a person. In FIG. 1, an MR scene 100 is depicted, and to a user of MR technology, a real-world park-like setting 110 is visible, featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of MR technology also "sees" a robotic figure 130 standing on the real-world platform 120 and a flying comic-like avatar character 140 that appears as an anthropomorphic bumblebee, but these elements do not exist in the real world.

[0097] It may be desirable for a 3D display to generate a perspective adjustment response corresponding to its virtual depth for each point within the display's field of view in order to generate a true sense of depth, more specifically, a simulated sense of surface depth. If the perspective adjustment response for a display point does not correspond to the virtual depth of that point such that it is determined by both binocular depth cues of convergence and stereopsis, the human eye experiences a vergence conflict, resulting in unstable imaging, harmful eye strain, headaches, and in the absence of vergence information, a nearly complete lack of surface depth.

[0098] VR, AR, and MR experiences may be provided by a display system having a display that provides images corresponding to multiple depth planes to a viewer. The images may vary for each depth plane (e.g., providing slightly different presentations of a scene or object), are separately focused by the viewer's eyes, and thereby serve to provide depth cues to the user based on the eye vergence required to focus on different image features of scenes located on different depth planes or based on observing different image features on different depth planes that are out of focus. As discussed anywhere in this specification, such depth cues provide a believable perception of depth.

[0099] Figure 2 illustrates an embodiment of a wearable system 200, which may be configured to provide an AR / VR / MR scenario. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functions of the display 220. The display 220 may be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. Since the display 220 can be worn on the head of the user 210, it may also be referred to as a head-mounted display (HMD), and the wearable system 200 including the display 220 may also be referred to as a head-mounted display system.

[0100] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's external auditory canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other external auditory canal to provide stereo / formable acoustic control). The display 220 may include an audio sensor (e.g., a microphone) 232 to detect an audio stream from the environment and capture ambient sound. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. Stereo sound reception may be used to determine the location of the sound source. The wearable system 200 may perform voice or speech recognition on the audio stream.

[0101] The wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world in the environment around the user. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system can track either the movement of one eye or the movement of both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may communicate electrically with a processing module 260 or 270 that processes the image information obtained by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user 210's eyes, eye movement, or eye pose. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. The images obtained by the cameras may be used to determine the pupil size or eye pose separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted for that eye.

[0102] As an example, the wearable system 200 may use the outward-facing imaging system 464 or the inward-facing imaging system 462 to obtain an image of the user's pose. The image may be a still image, a video frame, or a video.

[0103] The display 220 may be operably coupled to the local data processing module 260 (250) and may be mounted in various configurations, such as fixed to the frame 230 by a wired conductor or wireless connection, fixed to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt attachment configuration).

[0104] The local processing and data module 260 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in data processing, caching, and storage. The data may include (a) data captured from sensors such as an image capture device (e.g., a camera within an inward-facing imaging system and / or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope (e.g., operably coupled to frame 230 or otherwise attachable to user 210), or (b) data obtained or processed using remote processing module 270 or remote data repository 280, which may, in some cases, pass to display 220 after processing or reading. The local processing and data module 260 may be operably coupled to remote processing module 270 or remote data repository 280 via a communication link 262 or 264, such as a wired or wireless communication link, so that these remote modules are available as resources to the local processing and data module 260. Additionally, remote processing module 280 and remote data repository 280 may be operably coupled to each other.

[0105] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, enabling full autonomy from the remote modules. (Exemplary Components of a Wearable System)

[0106] Figure 3 schematically illustrates exemplary components of a wearable system. Figure 3 shows a wearable system 200, which can include a display 220 and a frame 230. Stretch view 202 schematically illustrates various components of wearable system 200. In one implementation, one or more of the components illustrated in Figure 3 may be part of display 220. The various components may collect various data (e.g., auditory or visual data, etc.) associated with the user of wearable system 200 or the user's environment, either alone or in combination. It should be understood that other embodiments may have additional or fewer components depending on the use for which the wearable system is employed. Note that Figure 3 provides some of the various components and a basic concept of the types of data that can be collected, analyzed, and stored through the wearable system.

[0107] FIG. 3 shows an exemplary wearable system 200, which may include a display 220. The display 220 may include a display lens 226 that can be mounted on a housing or frame 230 corresponding to the user's head or frame 230. The display lens 226 may include one or more transparent mirrors positioned in front of the user's eyes 302, 304 by the housing 230, bounce the projected light 338 into the eyes 302, 304, and be configured to allow transmission of at least some light from the local environment while facilitating beam shaping. The wavefront of the projected light beam 338 may be bent or focused to match the desired focal length of the projected light. As shown, two wide-field-of-view machine vision cameras 316 (also referred to as world cameras) may be coupled to the housing 230 to image the environment around the user. These cameras 316 may be dual-capture visible / non-visible (e.g., infrared) light cameras. The cameras 316 may be part of an outward-facing imaging system 464 shown in FIG. 4. Images obtained by the world cameras 316 may be processed by a pose processor 336. For example, the pose processor 336 may implement one or more object recognition devices 708 (e.g., shown in FIG. 7) to identify the pose of the user or another person in the user's environment, or to identify physical objects in the user's environment.

[0108] Continuing to refer to FIG. 3, shown is a pair of scanning laser shaped wavefront (e.g., for depth) light projector modules with display mirrors and optical systems configured to project light 338 into eyes 302, 304. The depicted figure also shows two small infrared cameras 324 paired with an infrared light source 326 (such as a light emitting diode “LED”, etc.) configured to track the user's eyes 302, 304 and support rendering and user input. Camera 324 may be part of an inward facing imaging system 462 shown in FIG. 4. Wearable system 200 may further feature a sensor assembly 339, which has X, Y, and Z axis accelerometer capabilities as well as magnetic compass and X, Y, and Z axis gyroscope capabilities and preferably can provide data at a relatively high frequency such as 200 Hz. Sensor assembly 339 may be part of an IMU as described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336 such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), or ARM processor (advanced reduced instruction set machine), which may be configured to calculate from wide field of view image information output from capture device 316 the real-time or near real-time user head pose. Head pose processor 336 may be a hardware processor and may be implemented as part of local processing and data module 260 shown in FIG. 2A.

[0109] The wearable system may also include one or more depth sensors 234. Depth sensor 234 may be configured to measure the distance between objects in the environment and the wearable device. Depth sensor 234 may include a laser scanner (e.g., LIDAR), an ultrasonic depth sensor, or a depth sensing camera. In some embodiments where camera 316 has depth sensing capabilities, camera 316 may also be considered a depth sensor 234.

[0110] Also shown is a processor 332 configured to perform digital or analog processing and derive from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be part of the local processing and data module 260 shown in FIG. 2. Wearable system 200 may also include a positioning system, such as, for example, GPS 337 (Global Positioning System), as shown in FIG. 3, to assist in attitude and positioning analysis. Additionally, GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0111] The wearable system may combine data obtained by GPS 337 and a remote computing system (e.g., remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As an example, the wearable system may determine the user's location based on GPS data and read out a world map that includes virtual objects associated with the user's location (e.g., by communicating with remote processing module 270). As another example, wearable system 200 may use world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4) to monitor the environment. Based on the images obtained by world camera 316, wearable system 200 may detect objects within the environment (e.g., by using one or more object recognition devices 708 shown in FIG. 7). The wearable system may further interpret characters using data obtained by GPS 337.

[0112] The wearable system 200 may also include a rendering engine 334, which may be configured to provide rendering information local to the user and facilitate the operation of the scanner and the imaging into the user's eyes for the view of users worldwide. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled to other components of the wearable system 200 (e.g., via a wired or wireless link). For example, the rendering engine 334 may be coupled to the eye camera 324 via the communication link 274 and coupled to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via the communication link 272. The rendering engine 334 may also communicate with other processing units, such as the sensor attitude processor 332 and the image attitude processor 336, via the links 276 and 294, respectively.

[0113] A camera 324 (e.g., a small infrared camera) may be used to track eye pose and support rendering and user input. Some exemplary eye poses may include where the user is looking or the depth of focus (which may be estimated using the vergence of the eyes). The GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide gross or high-speed pose estimation. One or more of the cameras 316 may obtain images and poses, which may be used to map the local environment and share the user view with others in conjunction with data from associated cloud computing resources.

[0114] The exemplary components depicted in FIG. 3 are for illustrative purposes only. A plurality of sensors and other functional modules are shown together for ease of illustration and explanation. Some embodiments may include only one or a subset of these sensors or modules. Further, the locations of these components are not limited to the positions depicted in FIG. 3. Some components may be mounted or stored within other components, such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be located within a belt pack and configured to communicate with other components of the wearable system via wireless communication such as ultra-wideband, Wi-Fi, Bluetooth®, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0115] With respect to the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may generally be utilized to measure the location where the center of the user's eye is geometrically converged, which generally coincides with the focal position or "depth of focus" of the eye. The three-dimensional surface of all points where the eyes converge may be referred to as the "horopter." The focal distance may take on a finite number of depths or may vary infinitely. Light projected from the convergence / divergence movement distance appears to be focused on the subject eyes 302, 304, while light in front of or behind the convergence / divergence movement distance is blurred. Embodiments of the wearable system and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0116] The human visual system is complex and it is difficult to provide a realistic perception of depth. An object viewer can perceive an object in three dimensions due to a combination of convergence-divergence movement and accommodation. The convergence-divergence movement of two eyes relative to each other (for example, the rotation of the pupils such that the pupils move towards or away from each other, converging the viewpoints of the eyes to fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, a change in the focus of the eye's lens or eye accommodation to change the focus from one object to another object at a different distance will automatically cause a coordinated change in convergence-divergence movement under a relationship known as the "accommodation-convergence-divergence reflex" at the same distance. Similarly, a change in convergence-divergence movement will, under normal conditions, induce a coordinated change in accommodation. A display system that provides better coordination between accommodation and convergence-divergence movement can form a more realistic and comfortable simulation of a three-dimensional image.

[0117] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Thus, in order to create an illusion of appropriate depth of focus, the eye's convergence-divergence movement may be tracked using camera 324, and rendering engine 334 and projection subsystem 318 may be utilized to focus and render all objects on or near the single viewing trajectory and render all other objects with a variable degree of defocus (for example, using intentionally created blur). Preferably, system 220 renders to the user at a frame rate of about 60 frames per second or more. As described above, preferably, camera 324 may be utilized for eye tracking and the software may be configured to take into account not only the convergence-divergence geometry but also a focus location queue for serving as a user input. Preferably, such a display system is configured with suitable brightness and contrast for use during the day or at night.

[0118] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute for visual object alignment, which, although not limited by theory, is considered to be approximately the limit of the human eye. The display system 220 may be integrated with a positioning system, which may involve GPS elements, optical tracking, compasses, accelerometers, or other data sources and may assist in position and orientation determination. The positioning information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would facilitate the glasses' understanding of their location relative to the real world).

[0119] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the user's eye accommodation. Unlike conventional 3D display approaches that force the user to focus on where the image is projected, in some embodiments, the wearable system automatically varies the focus of the projected virtual content to enable more comfortable viewing of the one or more images presented to the user. For example, if the user's eye has a current focus of 1 m, the image may be projected to coincide with the user's focus. If the user shifts the focus to 3 m, the image is projected to coincide with the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments allows the user's eye to function in a more natural manner.

[0120] Such a wearable system 200 can eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project virtual images at variable focal distances through one or more variable focus elements (VFE). In one or more embodiments, 3D perception may be achieved through a multi-plane focus system that projects the image onto a fixed focal plane from the user. Other embodiments employ variable plane focus, where the focal plane is reciprocally moved in the z-direction to match the current state of the user's focus.

[0121] In both the multi-plane focus system and the variable plane focus system, the wearable system 200 may employ eye tracking to determine the user's eye convergence / divergence motion, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 comprises a light modulator that projects a variable focus light beam in a raster pattern across the retina through a fiber scanner or other light generating source. Thus, the display capabilities of the wearable system 200 that project images at variable focal distances not only facilitate depth adjustment for the user to visually perceive objects in 3D, but may also be used to compensate for the user's eye abnormalities, as further described in U.S. Patent Publication No. 2016 / 0270656, which is hereby incorporated by reference in its entirety. In some other embodiments, the spatial light modulator may project the image to the user through various optical components. For example, as further described below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user. (Waveguide stack assembly)

[0122] FIG. 4 illustrates an embodiment of a waveguide stack for outputting image information to a user. Wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 may correspond to wearable system 200 of FIG. 2, and FIG. 4A schematically shows some portions of that wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated within display 220 of FIG. 2.

[0123] Continuing to refer to FIG. 4, waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, features 458, 456, 454, 452 may be lenses. In other embodiments, features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., a cladding layer or structure for forming an air gap).

[0124] Waveguides 432b, 434b, 436b, 438b, 440b or a plurality of lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may be utilized to input image information into waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of the image input devices 420, 422, 424, 426, 428 and is input into the corresponding input edges of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output an entire field of cloned collimated beams directed toward the eye 410 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide.

[0125] In some embodiments, image input devices 420, 422, 424, 426, 428 are discrete displays that each generate image information for input into their respective corresponding waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, image input devices 420, 422, 424, 426, 428 are the output ends of a single multiplexed display that can send image information, for example, via one or more optical waveguides (such as optical fiber cables), to each of the image input devices 420, 422, 424, 426, 428.

[0126] The controller 460 controls the operation of the stacked waveguide assemblies 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that adjusts the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. The controller 460 may, in some embodiments, be part of processing module 260 or 270 (illustrated in FIG. 2).

[0127] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved), with a major top surface and a major bottom surface and an edge extending between those major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b each include light extraction optical elements 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguides by redirecting the light, propagating it within each individual waveguide, and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as external coupled light, and the light extraction optical elements may also be referred to as external coupling optical elements. The beam of the extracted light is output by the waveguide at the location where the light propagating within the waveguide impinges on the light redirection element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. For ease of explanation and clarity of the drawings, they are shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of the material forming the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic material components, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the surface of the material component.

[0128] Continuing to refer to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to output collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to generate some convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane that is closer inwardly from optical infinity towards the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to generate another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 436b as originating from a second focal plane that is closer inwardly from optical infinity towards the person than the light from the next upper waveguide 434b was interpreted as originating from.

[0129] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, and using the highest waveguide 440b in the stack, its output is sent through all of the lenses between it and the eye for the aggregated focusing power that represents the focal plane closest to the person. When viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensation lens layer 430 may be disposed at the top of the stack to compensate for the stack of lenses 458, 456, 454, 452. (The compensation lens layer 430 and the stacked waveguide assembly 480 may be configured such that light originating from the world 470 is transmitted to the eye 410 with substantially the same level of divergence (or collimation) as the light had when it was first received by the stacked waveguide assembly 480.) Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0130] Continuing to refer to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have light extraction optical elements of different configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be three-dimensional or surface features configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be a volume hologram, a surface hologram, and / or a diffraction grating. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published Jun. 25, 2015, which is incorporated herein by reference in its entirety.

[0131] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 410 at each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of output emission towards the eye 304 with respect to this particular collimated beam that bounces within the waveguide.

[0132] In some embodiments, one or more DOEs may be switchable between an “on” state where they actively diffract and an “off” state where they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal, in which microdroplets have a diffraction pattern in a host medium, and the refractive index of the microdroplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0133] In some embodiments, the number and distribution of depth planes or depth of field may be varied dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer. As a result, as the pupil size of the viewer's eye decreases, one plane that was indistinguishable because its location in the plane was beyond the depth of focus of the eye becomes distinguishable, and as the pupil size decreases and the depth of field increases correspondingly, it may increase to appear more in focus. Similarly, the number of separated depth planes used to present different images to the viewer may be decreased with a decreased pupil size. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focusing from one depth plane to the other. However, these two depth planes may be sufficient for the user to focus at another pupil size without changing the focusing adjustment at the same time.

[0134] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0135] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition can be difficult in light of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the emitted beam size may be varied using a variable aperture.

[0136] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210, and it encompasses a portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the FOV of the world camera may be larger than the field of view of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 so that the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other contexts, the movement of the wearer may be more restricted, and accordingly, the wearer's FOR may touch a smaller solid angle. Images obtained from the outward-facing imaging system 464 may be used, for example, to track gestures made by the user (e.g., hand or finger gestures) and to detect objects within the world 470 in front of the user.

[0137] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors may be positioned to provide stereo sound reception useful for determining the location of the source of speech. As another example, the audio sensor 232 may comprise a directional microphone, which may also provide such useful directional information regarding the location where the audio source is located. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 when locating the source of speech or determining the active speaker at a particular instant. For example, the wearable system 400 may use voice recognition, alone or in combination with a reflected image of the speaker (as seen in a mirror, for example), to determine the identification of the speaker. As another example, the wearable system 400 may be able to determine the location of a speaker within the environment based on the sound obtained from a directional microphone. The wearable system 400 may use a speech recognition algorithm to analyze the sound resulting from the location of the speaker, determine the content of the speech, and use voice recognition techniques to determine the identification of the speaker (e.g., name or other demographic information).

[0138] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements such as eye movement and face movement. The inward-facing imaging system 466 may capture an image of the eye 410 and may be used to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 may be used to obtain an image for use in determining the direction in which the user is looking (e.g., eye pose), or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be used to separately determine the pupil size or eye pose of each eye independently for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. The image obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye pose or mood, which may be used by the wearable system 400 to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, an accelerometer, a gyroscope, etc. to determine the head pose (e.g., head position or head orientation).

[0139] The wearable system 400 may include a user input device 466 through which a user can input commands to the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a track pad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitance sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), etc. The multi-DOF controller may sense user input in translational (e.g., left / right, forward / backward, or up / down) or rotational (e.g., yaw, pitch, or roll) movements that are possible for some or all of the controller. A multi-DOF controller that supports translational movement may be referred to as 3DOF, while a multi-DOF controller that supports both translational and rotational movement may be referred to as 6DOF. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 either wired or wirelessly. (Other components of the wearable system)

[0140] In many embodiments, the wearable system may include other components in addition to, or instead of, the components of the wearable system described above. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a tactile sensation to the user. For example, the haptic device or component may provide a tactile sensation of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The tactile sensation may reproduce the sensation of a physical object represented by the virtual object, or may reproduce the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some embodiments, the haptic device or component may be held by the user.

[0141] A wearable system may include, for example, one or more physical objects that are operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, table surfaces, etc. In certain embodiments, a totem may not actually have any physical input structures (such as keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so as to appear on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard or virtual trackpad. The user input device 466 (shown in FIG. 4) may be an embodiment of a totem that may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system or other users.

[0142] Examples of haptic devices and totems that can be used with the wearable devices, HMDS, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety. (Example of an eye image)

[0143] FIG. 5 illustrates an image of an eye 500 with an eyelid 504, a sclera 508 (“white of the eye”), an iris 512, and a pupil 516. Curve 516a indicates the pupil boundary between the pupil 516 and the iris 512, and curve 512a indicates the edge boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is illustrated in a natural rest position (e.g., oriented such that both the user's face and line of sight will be directed toward a distant object directly in front of the user). The natural rest position of the eye 500 is in a natural rest position (e.g., out of plane immediately for the eye 500 shown in FIG. 5), and in this embodiment, can be indicated by a natural rest direction 520, which is the direction orthogonal to the surface of the eye 500 when centered within the pupil 516.

[0144] As the eye 500 moves to look at different objects, the eye posture will change relative to the natural rest direction 520. The current eye posture is a direction orthogonal to the surface of the eye (and centered within the pupil 516), but may be determined with reference to an eye posture direction 524, which is oriented toward the object at which the eye is currently directed. Referring to the exemplary coordinate system shown in FIG. 5, the posture of the eye 500 can be represented as two angular parameters that both indicate the azimuthal deviation and the zenith deviation of the eye posture direction 524 of the eye relative to the natural rest direction 520 of the eye. For illustrative purposes, these angular parameters can be represented as θ (azimuthal deviation, determined from a reference azimuth) and φ (zenith deviation, sometimes also referred to as polar deviation). In some embodiments, the angular roll of the eye around the eye posture direction 524 may be included in the determination of the eye posture, and the angular roll may be included in the following analysis. In some other embodiments, other techniques for determining the eye posture may be used, such as, for example, pitch, yaw, and optionally, a roll system.

[0145] The eye image may be obtained from a video using any suitable process, such as a video processing algorithm that can extract the image from one or more sequential frames. The eye pose may be determined from the eye image using various eye tracking techniques. For example, the eye pose may be determined by considering the lens effect of the cornea on the provided light source. Any suitable eye tracking technique may be used to determine the eye pose in the eyelid shape estimation technique described herein. (Example of an eye tracking system)

[0146] FIG. 6 illustrates a schematic diagram of a wearable or head-mounted display system 600 that includes an eye-tracking system. The head-mounted display system 600 may include, in at least some embodiments, components located within a head-mounted unit 602 and components located within a non-head-mounted unit 604. The non-head-mounted unit 604 may be, by way of example, a belt-mounted component, a handheld component, a component within a backpack, a remote component, etc. Incorporating some of the components of the head-mounted display system 600 within the non-head-mounted unit 604 may help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some embodiments, some or all of the functionality described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted 604 may be provided using one or more components included anywhere within the head-mounted display system 600. For example, some or all of the functionality described below in connection with the CPU 612 of the head-mounted unit 602 may be provided using the CPU 616 of the non-head-mounted unit 604, and vice versa. In some examples, some or all of such functionality may be provided using a peripheral device of the head-mounted display system 600. Further, in some embodiments, some or all of such functionality may be provided using one or more cloud computing devices or other remotely located computing devices in a manner similar to that described above with reference to FIG. 2.

[0147] As shown in FIG. 6, the head-mounted display system 600 may include an eye-tracking system that includes a camera 324 that captures an image of the user's eye 610. Optionally, the eye-tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs", etc.). The light sources 326a and 326b may generate a flash (i.e., a reflection from the user's eye that appears in the image of the eye captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, such that the position of the flash in the image captured by the camera 324 may be used in tracking the user's eye (as will be discussed in more detail below in connection with FIGS. 7-11). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In yet another embodiment, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. As a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. As another example, there may be three or more light sources such as light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610.

[0148] The eye tracking module 614 may receive an image from the eye tracking camera 324, analyze the image, and extract various information. As an example, the eye tracking module 614 may detect the user's eye posture, the three-dimensional position of the user's eyes with respect to the eye tracking camera 324 (and the head-mounted unit 602), the direction of one or both of the user's focused eyes 610, the user's convergence / divergence motion depth (i.e., the depth from the user on which the user is focused), the position of the user's pupils, the position of the user's corneas and corneal spheres, the respective centers of rotation of the user's eyes, and the respective centers of the viewpoints of the user's eyes. The eye tracking module 614 may extract such information using the techniques described below in connection with FIGS. 7-11. As shown in FIG. 6, the eye tracking module 614 may be a software module implemented using the CPU 612 within the head-mounted unit 602.

[0149] Data from the eye tracking module 614 may be provided to other components within the wearable system. As an example, such data may be transmitted to components within a non-head-mounted unit 604, such as the CPU 616, including software modules for the light field rendering controller 618 and the alignment observer 620, which may be configured to evaluate whether the display 600 of the head-mounted display system is properly aligned with the user's eyes.

[0150] The rendering controller 618 may adjust the image displayed to the user using the information from the eye tracking module 614 by the rendering engine 622 (which may be a software module within the GPU 621 and may provide an image to the display 220, the rendering engine). As an example, the rendering controller 618 may adjust the image displayed to the user based on the center of rotation or the center of the user's viewpoint. In particular, the rendering controller 618 may use the information regarding the center of the user's viewpoint to simulate a rendering camera (i.e., simulate the collection of an image from the user's viewpoint), and may adjust the image displayed to the user based on the simulated rendering camera.

[0151] Sometimes referred to as a "pinhole perspective projection camera" (or simply, a "perspective projection camera") or a "virtual pinhole camera" (or simply, a "virtual camera"), a "rendering camera" is a simulated camera that may be used to render virtual image content from a database of objects within a virtual world. The objects may have locations and orientations relative to the user or wearer, and potentially, real objects within the environment surrounding the user or wearer. In other words, the rendering camera may represent a viewing point within a rendering space from which the user or wearer is to view 3D virtual content (e.g., virtual objects) of the rendering space. The rendering camera may be managed by a rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eye. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, distortion parameters, etc.) and a specific set of extrinsic parameters (e.g., translation and rotation components relative to the virtual world). The virtual image is taken from the viewpoint of such a camera having the position and orientation of the rendering camera (e.g., the extrinsic parameters of the rendering camera). The system is thus capable of defining and / or adjusting the intrinsic and extrinsic rendering camera parameters. For example, the system may define a specific set of extrinsic rendering camera parameters such that the virtual image is rendered as if it were captured from the viewpoint of a camera having a specific location relative to the user or wearer's eye, so as to provide an image that appears to be from the viewpoint of the user or wearer. The system may later dynamically adjust the extrinsic rendering camera parameters on the fly to maintain alignment with the specific location. Similarly, the intrinsic rendering camera parameters may also be defined and dynamically adjusted over time.In some embodiments, the image is rendered as if it were captured from the perspective of a camera having an aperture (e.g., a pinhole) at a specific location relative to the user's or wearer's eye (such as the center or center of rotation or other location of the viewing point).

[0152] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and a different rendering camera for the user's right eye as the user's eyes are physically separated from each other and thus consistently located at different places. In at least some embodiments, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye may be presented to the user through the left eyepiece on the left side of a head-mounted display (e.g., the head-mounted unit 602), and virtual content rendered from the perspective of the rendering camera associated with the user's right eye may be presented to the user through the right eyepiece on the right side of such a head-mounted display. Further details regarding the creation, adjustment, and use of the rendering camera in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION", which is hereby expressly incorporated by reference in its entirety for all purposes.

[0153] In some embodiments, one or more modules (or components) of system 600 (e.g., light field rendering controller 618, rendering engine 622, etc.) may determine the position and orientation of a rendering camera within a rendering space based on the position and orientation of the user's head and eyes (e.g., as determined based on head pose and eye tracking data, respectively). That is, system 600 effectively maps the position and orientation of the user's head and eyes to a particular location and angular position within the 3D virtual environment, positions and orients the rendering camera at the particular location and angular position within the 3D virtual environment, and may render virtual content for the user as it would be captured by the rendering camera. Further details regarding the real-world / virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE", which is hereby expressly incorporated by reference in its entirety for all purposes. As an example, the rendering controller 618 may adjust the depth at which an image is displayed by selecting the depth plane (or depth planes) to be utilized at any given time for displaying the image. In some embodiments, such depth plane switching may be performed through the adjustment of one or more intrinsic rendering camera parameters.

[0154] The alignment observer 620 may identify whether the head-mounted unit 602 is properly positioned on the user's head using information from the eye tracking module 614. As an example, the eye tracking module 614 may provide eye location information such as the position of the center of rotation of the user's eyes, indicating the three-dimensional position of the user's eyes relative to the camera 324, and the head-mounted unit 602 and the eye tracking module 614 may use the location information to determine whether the display 220 is properly aligned within the user's field of view, or whether the head-mounted unit 602 (or headset) has slipped or is otherwise misaligned with the user's eyes. As an example, the alignment observer 620 may determine whether the head-mounted unit 602 has slipped from the user's nasal bridge and thus moved the display 220 away from and downward from the user's eyes (which may not be desirable), whether the head-mounted unit 602 has moved above the user's nasal bridge and thus moved the display 220 closer to and upward from the user's eyes, whether the head-mounted unit 602 has shifted left or right relative to the user's nasal bridge, whether the head-mounted unit 602 has been lifted above the user's nasal bridge, or whether the head-mounted unit 602 has moved away from the desired position or range of positions in these or other ways. Generally, the alignment observer 620 may generally be able to determine whether the head-mounted unit 602, particularly the display 220, is properly positioned in front of the user's eyes. In other words, the alignment observer 620 may determine whether the left-eye display within the display system 220 is properly aligned with the user's left eye and whether the right-eye display within the display system 220 is properly aligned with the user's right eye. The alignment observer 620 may determine whether the head-mounted unit 602 is properly positioned by determining whether the head-mounted unit 602 is positioned and oriented within the desired range of positions and / or orientations relative to the user's eyes.

[0155] In at least some embodiments, the alignment observer 620 may generate user feedback in the form of an alert, message, or other content. Such feedback may be provided to the user and inform the user of any misalignment of the head-mounted unit 602, along with optional feedback regarding how to correct the misalignment (such as a proposal to adjust the head-mounted unit 602 in a particular manner).

[0156] Exemplary alignment observation and feedback techniques that may be utilized by the alignment observer 620 are described in U.S. Patent Application No. 15 / 717,747, filed September 27, 2017 (Attorney Docket No. MLEAP.052A2), which is incorporated herein by reference in its entirety. (Example of an eye tracking module)

[0157] A detailed block diagram of an exemplary eye tracking module 614 is shown in FIG. 7A. As shown in FIG. 7A, the eye tracking module 614 may include various different sub-modules, may provide various different outputs, and may utilize various available data when tracking the user's eyes. By way of example, the eye tracking module 614 may utilize available data including the geometric arrangement of the light source 326 and the eye tracking camera 324 relative to the head-mounted unit 602, assumed eye dimensions 704 such as a typical distance of about 4.7 mm between the center of the user's corneal curvature and the average center of rotation of the user's eye or a typical distance between the user's center of rotation and the center of the viewpoint, and per-user calibration data 706 such as the interpupillary distance of a particular user. Additional examples of incidental, intrinsic, and other information that may be employed by the eye tracking module 614 are described in U.S. Patent Application No. 15 / 497,726, filed April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0158] The image preprocessing module 710 may receive an image from an eye camera such as the eye camera 324, and may perform one or more preprocessing (i.e., adjustment) operations on the received image. As an example, the image preprocessing module 710 may apply Gaussian blur to the image, may downsample the image to a lower resolution, may apply an unsharp mask, may apply an edge sharpening algorithm, or may apply other suitable filters that assist in subsequent detection, localization, and labeling of flashes, pupils, or other features within the image from the eye camera 324. The image preprocessing module 710 may apply a low-pass filter or a morphological filter such as an open filter that can remove noise, such as high-frequency noise from the pupil boundary 516a (see FIG. 5), thereby interfering with pupil and flash determination. The image preprocessing module 710 may output the preprocessed image to the pupil identification module 712 and the flash detection and labeling module 714.

[0159] The pupil recognition module 712 may receive the pre - processed image from the image pre - processing module 710 and may identify the regions of those images that contain the user's pupils. In some embodiments, the pupil recognition module 712 may determine the coordinates of the position of the user's pupil in the eye - tracking image from the camera 324, i.e., the coordinates of the center or centroid. In at least some embodiments, the pupil recognition module 712 may identify the contour (e.g., the contour of the pupil - iris boundary) in the eye - tracking image, identify the contour moment (i.e., the center of mass), apply the starburst pupil detection and / or Canny edge detection algorithm, exclude outliers based on intensity values, identify sub - pixel boundary points, correct eye camera distortion (i.e., the distortion in the image captured by the eye camera 324), apply the random sample consensus (RANSAC) iterative algorithm, fit an ellipse to the boundary in the eye - tracking image, apply a tracking filter to the image, and identify the sub - pixel image coordinates of the user's pupil centroid. The pupil recognition module 712 may output pupil recognition data, which may indicate the region of the pre - processed image module 712 identified as showing the user's pupil, to the flash detection and labeling module 714. The pupil recognition module 712 may provide the 2D coordinates of the user's pupil (i.e., the 2D coordinates of the user's pupil centroid) in each eye - tracking image to the flash detection module 714. In at least some embodiments, the pupil recognition module 712 may also provide the same type of pupil recognition data to the coordinate system normalization module 718.

[0160] Pupil detection techniques that may be utilized by the pupil recognition module 712 are described in U.S. Patent Publication No. 2017 / 0053165, published on February 23, 2017, and U.S. Patent Publication No. 2017 / 0053166, published on February 23, 2017 (each of which is incorporated herein by reference in its entirety).

[0161] The flash detection and labeling module 714 may receive the pre - processed image from module 710 and the pupil identification data from module 712. The flash detection module 714 may use this data to detect and / or identify a flash (i.e., the reflection of light from the light source 326 from the user's eye) within the region of the pre - processed image indicating the user's pupil. As an example, the flash detection module 714 may search for bright regions in the eye - tracking image that are near the user's pupil and are sometimes also referred to herein as "blobs" or local intensity maxima. In at least some embodiments, the flash detection module 714 may re - scale (e.g., enlarge) the pupil ellipse to include additional flashes. The flash detection module 714 may filter the flashes by size and / or intensity. The flash detection module 714 may also determine the 2D position of each flash in the eye - tracking image. In at least some examples, the flash detection module 714 may determine the 2D position of the flash relative to the user's pupil, which may also be referred to as a pupil - flash vector. The flash detection and labeling module 714 may label the flashes and output the pre - processed image with the labeled flashes to the 3D corneal center estimation module 716. The flash detection and labeling module 714 may also communicate data such as the pre - processed image from module 710 and the pupil identification data from module 712.

[0162] Pupil and flash detection, such as that performed by modules such as modules 712 and 714, may use any suitable technique. As an example, edge detection may be applied to the eye image to identify the flash and the pupil. Edge detection may be applied by various edge detectors, edge detection algorithms, or filters. For example, the Canny edge detector may be applied to the image to detect edges such as lines in the image. The edges may include points located along a line corresponding to a local maximum derivative. For example, the pupil boundary 516a (see FIG. 5) may be located using the Canny edge detector. Once the location of the pupil is determined, various image processing techniques may be used to detect the "pose" of the pupil 116. Determination of the eye pose of the eye image may also be referred to as detection of the eye pose of the eye image. The pose may also be referred to as the line of sight, the direction being looked at, or the orientation of the eye. For example, the pupil may be looking left towards an object, and the pose of the pupil may be classified as a left-facing pose. Other methods may also be used to detect the location of the pupil or the flash. For example, concentric rings may be located within the eye image using the Canny edge detector. As another example, an integral differential operator may be used to find the corneal limbus boundary of the pupil or the iris. For example, the Daugman integral differential operator, the Hough transform, or other iris segmentation techniques may be used to return a curve that estimates the boundary of the pupil or the iris.

[0163] The 3D corneal center estimation module 716 may receive a pre - processed image, including the detected flash data and pupil identification data, from modules 710, 712, 714. The 3D corneal center estimation module 716 may use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 may estimate the 3D position of the center of the corneal curvature of the eye or the center of the user's corneal sphere, i.e., generally, the center of an imaginary sphere having a surface portion co - extensive with the user's cornea. The 3D corneal center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further details of the operation of the 3D corneal center estimation module 716 are provided herein in connection with FIGS. 8A - 8E. Techniques for estimating the position of eye features such as the cornea or corneal sphere, which may be utilized by the 3D corneal center estimation module 716 and other modules within the wearable system of the present disclosure, are discussed in U.S. Patent Application No. 15 / 497,726, filed on Apr. 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0164] The coordinate system normalization module 718 may optionally be included within the eye tracking module 614 (as indicated by its dashed outline). The coordinate system normalization module 718 may receive data indicating the estimated 3D coordinates of the center of the user's cornea (and / or the center of the user's corneal sphere) from the 3D corneal center estimation module 716, and may also receive data from other modules. The coordinate system normalization module 718 may normalize the eye camera coordinate system, which may help to compensate for slippage of the wearable device (e.g., slippage of the head-mounted component from its normal resting position on the user's head, which may be identified by the alignment observer 620). The coordinate system normalization module 718 may rotate the coordinate system and align the z-axis of the coordinate system (i.e., the convergence / divergence motion depth axis) with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716), and may translate the camera center (i.e., the origin of the coordinate system) to a predetermined distance away from the corneal center, such as 30 mm (i.e., the module 718 may enlarge or shrink the eye tracking image depending on whether the eye camera 324 is determined to be closer or farther than the predetermined distance). By using this normalization process, the eye tracking module 614 may be able to establish a consistent orientation and distance within the eye tracking data, relatively independently of variations in the headset positioned on the user's head. The coordinate system normalization module 718 may provide the 3D coordinates of the center of the cornea (and / or corneal sphere), pupil identification data, and the pre-processed eye tracking image to the 3D pupil center locator module 720. Further details of the operation of the coordinate system normalization module 718 are provided herein in connection with FIGS. 9A-9C.

[0165] The 3D pupil center locator module 720 may receive data including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil location data, and pre - processed eye - tracking images, in a normalized or non - normalized coordinate system. The 3D pupil center locator module 720 may analyze such data to determine the 3D coordinates of the center of the user's pupil in a normalized or non - normalized eye camera coordinate system. The 3D pupil center locator module 720 may determine the location of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), assumed eye dimensions 704 such as the size of the typical user's corneal sphere and the typical distance from the corneal center to the pupil center, and the optical properties of the eye such as the refractive index of the cornea (relative to the refractive index of air), or any combination of these. Further details of the operation of the 3D pupil center locator module 720 are provided herein in connection with FIGS. 9D - 9G. Techniques for estimating the position of eye features such as the pupil, which may be utilized by the 3D pupil center locator module 720 and other modules within the wearable system of the present disclosure, are discussed in U.S. Patent Application No. 15 / 497,726, filed on April 26, 2017 (Attorney Docket No. MLEAP.023A7), which is incorporated herein by reference in its entirety.

[0166] The optical axis determination module 722 may receive data indicating the 3D coordinates of the center of the user's cornea and the center of the user's pupil from modules 716 and 720. Based on such data, the optical axis determination module 722 may identify a vector from the location of the corneal center (i.e., from the center of the corneal sphere) to the center of the user's pupil that may define the optical axis of the user's eye. As an example, the optical axis determination module 722 may provide an output defining the user's optical axis to modules 724, 728, 730, and 732.

[0167] The center of rotation (CoR) estimation module 724 may receive data from module 722 that includes parameters of the optical axis of the user's eye (i.e., data indicating the direction of the optical axis within a coordinate system with a known relationship to the head-mounted unit 602). The CoR estimation module 724 may estimate the center of rotation of the user's eye (i.e., the point around which the user's eye rotates as the user's eye rotates left, right, up, and / or down). Assume that a single point may be sufficient even if the eye cannot rotate perfectly around a single point. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eye by moving the center of the pupil (identified by module 720) or the center of curvature of the cornea (as identified by module 716) a specific distance along the optical axis (identified by module 722) toward the retina. This specific distance may be the assumed eye dimension 704. As an example, the specific distance between the center of curvature of the cornea and the CoR may be about 4.7 mm. This distance may be varied for a particular user based on any relevant data, including the user's age, gender, visual prescription, other relevant characteristics, etc.

[0168] In at least some embodiments, the CoR estimation module 724 may refine over time the estimated value of the respective center of rotation of the user's eyes. As an example, over time, the user may eventually rotate the eye (to look at something else, closer, farther, or sometimes left, right, up, or down), which will cause an offset along the respective optical axis of the eye. The CoR estimation module 724 may then analyze the two (or more) optical axes identified by the module 722 and locate the 3D point of the intersection of those optical axes. The CoR estimation module 724 may then determine the center of rotation at the 3D point of the intersection. Such techniques may provide an estimated value of the center of rotation with accuracy that improves over time. Various techniques may be employed to increase the accuracy of the CoR estimation module 724 as well as the determined CoR positions of the left and right eyes. As an example, the CoR estimation module 724 may estimate the CoR by finding the average point of the intersections of the optical axes determined over time for various different eye postures. As an additional example, the module 724 may filter or average the estimated CoR positions over time, calculate a moving average of the estimated CoR positions over time, and / or apply a Kalman filter and the known dynamics of the eye and the eye tracking system to estimate the CoR position over time. As a specific example, the module 724 may calculate a weighted average of the determined point of the optical axis intersection and the assumed CoR position (i.e., 4.7 mm behind the center of the corneal curvature of the eye) such that the determined CoR slowly moves over time to a slightly different location within the user's eye as eye tracking data for the user is acquired, thereby enabling per-user refinement of the CoR position.

[0169] The interpupillary distance (IPD) estimation module 726 may receive data indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes from the CoR estimation module 724. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. Generally, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the centers of the user's pupils when the user is looking at optical infinity (i.e., the optical axes of the user's eyes are approximately parallel to each other), which is the typical definition of the interpupillary distance (IPD). The user's IPD may be used by various components and modules within the wearable system. As an example, the user's IPD may be provided to the alignment observer 620 and used when assessing the extent to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are appropriately spaced according to the user's IPD). As another example, the user's IPD may be provided to the convergence / divergence motion depth estimation module 728 and used when determining the user's convergence / divergence motion depth. The module 726 may employ various techniques such as those discussed in relation to the CoR estimation module 724 to increase the accuracy of the estimated IPD. As an example, the IPD estimation module 724 may apply filtering, temporal averaging, weighted averaging, Kalman filtering, etc., including an assumed IPD distance, as part of the estimation of the user's IPD in an accurate manner.

[0170] The inter-ocular axis estimation module 740 may receive data from one or more modules such as the IPD estimation module 726, the optical axis determination module 722, the 3D corneal center estimation module 716, the 3D pupil center locator module 720, the CoR estimation module 724, and the CoP estimation module 723. In particular, the inter-ocular axis estimation module 740 may receive data indicating the estimated 3D positions of one or more features of the user's left and right eyes, such as the left and right eye rotation centers, the center of the line of sight, the pupil position, the iris position, the optical axis, and the inter-pupillary distance. The inter-ocular axis estimation module 740 may use such information to determine the location of the user's inter-ocular axis. The inter-ocular axis may extend between a feature of the user's left eye and the corresponding feature of the user's right eye. As an example, the inter-ocular axis may extend between the rotation center of the left eye and the rotation center of the right eye. As another example, the inter-ocular axis may extend between the left eye pupil and the right eye pupil. The inter-ocular axis estimation module 740 may be able to determine the position and / or orientation of the inter-ocular axis relative to the wearable system, such as by determining the position of the user's eye (or some feature thereof) relative to the eye camera 324.

[0171] The convergence-divergence motion depth estimation module 728 may receive data from various modules and sub-modules within the eye tracking module 614 (as shown in connection with FIG. 7A). In particular, the convergence-divergence motion depth estimation module 728 may receive data indicating the estimated 3D position of the pupil center (e.g., as provided by module 720 described above), one or more determined parameters of the optical axis (e.g., as provided by module 722 described above), the estimated 3D position of the center of rotation (e.g., as provided by module 724 described above), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as provided by module 726 described above), and / or one or more determined parameters of the optical axis and / or visual axis (e.g., as provided by module 722 and / or module 730 described below). The convergence-divergence motion depth estimation module 728 may detect or otherwise obtain a measurement of the user's convergence-divergence motion depth, which may be the distance from the user at which the user's eyes are focused. As an example, when the user is looking at an object 3 feet from their front, the user's left and right eyes have a convergence-divergence motion depth of 3 feet, while when the user is looking at a distant landscape (i.e., the optical axes of the user's eyes are substantially parallel to each other such that the distance between the centers of the user's pupils can be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have an infinite convergence-divergence motion depth. In some embodiments, the convergence-divergence motion depth estimation module 728 may utilize data indicating the estimated center of the user's pupil (e.g., as provided by module 720) and determine the 3D distance between the estimated centers of the user's pupils. The convergence-divergence motion depth estimation module 728 may obtain a measurement of the convergence-divergence motion depth by comparing such a determined 3D distance between the pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) (e.g., as shown by module 726 described above).In addition to the 3D distance between pupil centers and the estimated IPD, the convergence / divergence motion depth estimation module 728 may calculate the convergence / divergence motion depth using known, assumed, estimated, and / or determined geometries. As an example, module 728 may combine the 3D distance between pupil centers, the estimated IPD, and the 3D CoR position in a triangulation calculation to estimate (i.e., determine) the user's convergence / divergence motion depth. In fact, the evaluation of such a determined 3D distance between pupil centers with respect to the estimated IPD can serve to indicate a measured value of the user's current convergence / divergence motion depth with respect to optical infinity. In some embodiments, the convergence / divergence motion depth estimation module 728 may simply receive or access data indicating the estimated 3D distance between the estimated centers of the user's pupils for the purpose of obtaining such a measured value of the convergence / divergence motion depth. In some embodiments, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by comparing the user's left and right optical axes. In particular, the convergence / divergence motion depth estimation module 728 may estimate the convergence / divergence motion depth by locating the distance from the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes on a plane such as a horizontal plane intersect). Module 728 may utilize the user's IPD in this calculation by setting zero depth as the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the convergence / divergence motion depth estimation module 728 may determine the convergence / divergence motion depth by triangulating eye tracking data with known or derived spatial relationships.

[0172] In some embodiments, the vergence / accommodation depth estimation module 728 may estimate the user's vergence / accommodation depth based on the intersection of the user's visual axes (instead of its optical axis), which may provide a more accurate indication of the distance at which the user is focused. In at least some embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. As will be discussed in more detail in connection with FIG. 10, the user's optical axis and visual axis generally do not coincide. The visual axis is the axis along which a person is looking, while the optical axis is defined by the center of the person's lens and pupil and may pass through the center of the person's retina. In particular, the user's visual axis is generally offset from the center of the user's retina, thereby resulting in different optical and visual axes, as defined by the location of the user's fovea. In at least some of these embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. The optical axis / visual axis mapping module 730 may correct for the difference between the user's optical axis and visual axis and provide information regarding the user's visual axis to other components within the wearable system, such as the vergence / accommodation depth estimation module 728 and the light field rendering controller 618. In some examples, the module 730 may use an assumed eye dimension 704 that includes a typical offset of approximately 5.2° inward (nasally, towards the user's nose) between the optical axis and the visual axis. In other words, the module 730 may shift the user's left optical axis 5.2° to the right nasally (towards the nose) and the user's right optical axis 5.2° to the left nasally (towards the nose) to estimate the directions of the user's left and right optical axes. In other examples, the module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as indicated by the module 722 described above) to the visual axis. As an additional example, the module 730 may shift the user's optical axis nasally by any range formed by, for example, 4.0° - 6.5°, 4.5° - 6.0°, 5.0° - 5.4°, etc., or any of these values.In some arrays, the module 730 may apply an offset, at least in part, based on characteristics of a particular user, such as their age, gender, visual prescription, or other relevant characteristics, and / or at least in part based on a calibration process for a particular user (i.e., to determine the optical axis - visual axis offset for a particular user). In at least some embodiments, the module 730 may also offset the origin of the left and right optical axes and correspond to the user's CoP (as determined by the module 732) instead of the user's CoR.

[0173] When an optional center of perspective (CoP) estimation module 732 is provided, it may estimate the locations of the user's left and right centers of perspective (CoP). The CoP is a useful location for the wearable system and, in at least some embodiments, can be the position directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the locations of the user's left and right centers of perspective based on the 3D location of the user's pupil center, the 3D location of the center of the user's corneal curvature, or such suitable data, or any combination thereof. As an example, the user's CoP can be approximately 5.01 mm in front of the center of the corneal curvature (i.e., 5.01 mm in the direction along the optical axis from the center of the corneal sphere towards the eye's cornea) and can be approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of perspective can be directly in front of the center of their pupil. As an example, the user's CoP can be less than about 2.0 mm from the user's pupil, less than about 1.0 mm from the user's pupil, or less than about 0.5 mm from the user's pupil, or any range between these values. As another example, the center of perspective can correspond to a location within the anterior chamber of the eye. As another example, the CoP can be at 1.0 mm to 2.0 mm, about 1.0 mm, 0.25 mm to 1.0 mm, 0.5 mm to 1.0 mm, or 0.25 mm to 0.5 mm.

[0174] (As the potentially desirable position of the pinhole of the rendering camera and the anatomical position within the user's eye) The center of perspective described herein can be a position that serves to reduce and / or eliminate undesirable parallax shifts. In particular, the optical system of the user's eye closely approximates the theoretical system formed by the front pinhole of the lens projecting onto the screen, with the pinhole, lens, and screen corresponding generally to the user's pupil / iris, lens, and retina, respectively. Further, when two point light sources (or objects) at different distances from the user's eye rotate precisely about the opening of the pinhole (e.g., are rotated along a radius of curvature equal to their individual distance from the opening of the pinhole), it may be desirable for there to be little or no parallax shift. Thus, the CoP would be considered to be located at the center of the pupil of the eye (and such a CoP may be used in some embodiments). However, the human eye includes the cornea, which in addition to the pinhole of the lens and pupil, imparts additional refractive power to the light propagating towards the retina. Thus, the anatomical equivalent of the pinhole within the theoretical system described in this paragraph can be the region of the user's eye located between the outer surface of the user's eye's cornea and the center of the user's eye's pupil or iris. For example, the anatomical equivalent of the pinhole can correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP to such a position within the anterior chamber of the user's eye.

[0175] As discussed above, the eye tracking module 614 may provide data such as the estimated 3D positions of the left and right eye centers of rotation (CoR), the vergence / accommodation depth, the left and right eye optical axes, the 3D positions of the user's eyes, the 3D positions of the left and right centers of the user's corneal curvature, the 3D positions of the user's left and right pupil centers, the 3D positions of the user's left and right fixation centers, the user's IPD, etc., to other components such as the light field rendering controller 618 and the alignment observer 620 within the wearable system. The eye tracking module 614 may also include other sub-modules that detect and generate data associated with other aspects of the user's eyes. As an example, the eye tracking module 614 may include a blink detection module that provides a flag or other alert each time the user blinks, and a saccade detection module that provides a flag or other alert each time the user's eyes saccade (i.e., rapidly shift focus to another point). (Example of a rendering controller)

[0176] A detailed block diagram of an exemplary light field rendering controller 618 is shown in FIG. 7B. As shown in FIGS. 6 and 7B, the rendering controller 618 may receive eye tracking information from the eye tracking module 614 and may provide an output to the rendering engine 622, which may generate an image to be displayed for viewing by a user of the wearable system. As an example, the rendering controller 618 may receive information regarding the vergence / accommodation depth, the left and right eye centers of rotation (and / or fixation centers), and other eye data such as blink data, saccade data, etc.

[0177] The depth plane selection module 750 may receive the convergence / divergence motion depth information, and based on such data, cause the rendering engine 622 to provide the user with the content in a state where the content appears to be located on a specific depth plane (i.e., a specific perspective adjustment or focal length). As discussed in connection with FIG. 4, the wearable system may include a plurality of discrete depth planes formed by a plurality of waveguides that each transmit image information with a variable level of wavefront curvature. In some embodiments, the wearable system may include one or more variable depth planes, such as optical elements, that transmit image information with a level of wavefront curvature that varies over time. In these and other embodiments, the depth plane selection module 750 may cause the rendering engine 622 to transmit the content to the user at a selected depth, at least in part, based on the user's convergence / divergence motion depth (i.e., cause the rendering engine 622 to instruct the display 220 to switch depth planes). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 may render the content at different depths and also generate and / or provide depth plane selection data to a display hardware such as the display 220. The display hardware such as the display 220 may perform electrical depth plane switching in response to depth plane selection data (which may be a control signal) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0178] Generally, it may be desirable for the depth plane selection module 750 to select a depth plane that matches the user's current convergence / divergence motion depth so that the user is provided with an accurate perspective adjustment cue. However, it may also be desirable to switch depth planes in a careful and unobtrusive manner. As an example, it may be desirable to avoid excessive switching between depth planes and / or to switch depth planes at times when the user is less likely to notice the switch, such as during a blink or an eye saccade.

[0179] The hysteresis band intersection detection module 752 can be particularly useful in avoiding excessive switching between depth planes, especially when the depth of the user's convergence / divergence movement varies at the midpoint or transition point between two depth planes. In particular, the module 752 may cause the depth plane selection module 750 to exhibit hysteresis in its selection of depth planes. As an example, the module 752 may cause the depth plane selection module 750 to switch from a first, more distant depth plane to a second, closer depth plane only after the depth of the user's convergence / divergence movement has passed a first threshold. Similarly, the module 752 may cause the depth plane selection module 750 (and thus, can instruct a display such as the display 220) to switch to the first, more distant depth plane only after the depth of the user's convergence / divergence movement has passed a second threshold that is farther from the user than the first threshold. In the overlapping region between the first threshold and the second threshold, the module 750 may cause the depth plane selection module 750 to be maintained as if either depth plane were currently selected as the selected depth plane, and thus, may avoid excessive switching between depth planes.

[0180] The eye event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A, and may delay several depth plane switches to the depth plane selection module 750 until an eye event occurs. As an example, the eye event detection module 750 may delay the planned depth plane switch to the depth plane selection module 750 until a user blink is detected, and may receive data from the blink detection component in the eye tracking module 614 indicating that the user is currently blinking, and in response, may cause the depth plane selection module 750 to perform the planned depth plane switch during the blink event (such as by instructing the module 750 to perform the depth plane switch on the display 220 during the blink event). In at least some embodiments, the wearable system may be able to shift the content onto a new depth plane during the blink event such that the user is less likely to perceive the shift. As another example, the eye event detection module 750 may delay the planned depth plane switch until an eye saccade is detected. As discussed in connection with eye blinks, such an arrangement may facilitate discrete shifts in the depth plane.

[0181] Optionally, the depth plane selection module 750 may delay the planned depth plane switch only for a limited time period before performing the depth plane switch even in the absence of an eye event. Similarly, the depth plane selection module 750 may perform the depth plane switch even in the absence of an eye event when the user's vergence / accommodation depth is substantially outside the currently selected depth plane (i.e., when the user's vergence / accommodation depth exceeds a predetermined threshold that exceeds the normal threshold for depth plane switching). These arrangements may help ensure that the eye event detection module 754 does not indefinitely delay the depth plane switch and does not delay the delayed depth plane switch when there is a large depth of field adjustment error.

[0182] The rendering camera controller 758 may provide information indicating the locations of the user's left and right eyes to the rendering engine 622. The rendering engine 622 may then generate content by simulating the camera at the locations of the user's left and right eyes and generating the content based on the viewpoints of the simulated cameras. As discussed above, the rendering camera may potentially be a simulated camera for use in rendering virtual image content from a database of objects within the virtual world. The objects may have locations and orientations relative to the user or wearer, potentially relative to real objects within the environment surrounding the user or wearer. The rendering camera may be included within the rendering engine and may render a virtual image based on a database of virtual objects to be presented to the eyes. The virtual image may be rendered as if it were captured from the viewpoint of the user or wearer. For example, the virtual image may be rendered as if it were captured by a camera (corresponding to the "rendering camera") having an aperture, lens, and detector for viewing objects within the virtual world. The virtual image is captured from the viewpoint of such a camera having the position of the "rendering camera". For example, the virtual image may be rendered as if it were captured from a camera viewpoint having a specific location relative to the user or wearer's eyes such that the virtual image provides an image that appears to be from the viewpoint of the user or wearer. In some embodiments, the image may be rendered as if it were captured from a camera viewpoint having an aperture at a specific location relative to the user or wearer's eyes (such as a viewpoint center or rotation center or other location as discussed herein).

[0183] The rendering camera controller 758 may determine the positions of the left and right cameras based on the left and right centers of rotation (CoR) determined by the CoR estimation module 724 and / or based on the left and right centers of perspective (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between the CoR location and the CoP location based on various factors. As an example, the rendering camera controller 758, in various modes, may constantly align the rendering camera to the CoR location, constantly align the rendering camera to the CoP location, toggle between aligning the rendering camera to the CoR location and aligning the rendering camera to the CoP location over time based on various factors, or discretely switch, or dynamically align the rendering camera to any of a range of different positions along the optical (or visual) axis between the CoR location and the CoP location over time based on various factors. The CoR and CoP positions may optionally pass through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning), which may average the CoR and CoP locations over time, reduce noise at these locations, and prevent jitter when rendering the simulated rendering camera.

[0184] In at least some embodiments, the rendering camera may be simulated as a pinhole camera with a pinhole placed at the location of the estimated CoR or CoP identified by the eye tracking module 614. Since the CoP is offset from the CoR, whenever the position of the rendering camera is based on the user's CoP, the locations of both the rendering camera and its pinhole are offset as the user's eye rotates. In contrast, whenever the position of the rendering camera is based on the user's CoR, the location of the pinhole of the rendering camera does not move with eye rotation, but the rendering camera (behind the pinhole) may move with eye rotation in some embodiments. In other embodiments where the position of the rendering camera is based on the user's CoR, the rendering camera may not move (i.e., rotate) with the user's eye. (Example of Alignment Observer)

[0185] A block diagram of an exemplary alignment observer 620 is shown in FIG. 7C. As shown in FIGS. 6, 7A, and 7C, alignment observer 620 may receive eye tracking information from eye tracking module 614 (FIGS. 6 and 7A). As an example, alignment observer 620 may receive information regarding the centers of rotation of the user's left and right eyes (e.g., the three-dimensional positions of the centers of rotation of the user's left and right eyes that are on a common coordinate system or that may have a reference common frame with the head-mounted display system 600). As another example, alignment observer 620 may receive display-related properties, fit tolerance, and an eye tracking valid indicator. The display-related properties may include information regarding the display (e.g., display 200 of FIG. 2), such as the field of view of the display, the size of one or more display surfaces, and the position of the display surface relative to the head-mounted display system 600. The fit tolerance may include information regarding the display alignment volume that may indicate the distance by which the user's left and right eyes can move from their nominal positions before the display performance is affected. Additionally, the fit tolerance may indicate the amount of display performance impact that is expected as a function of the position of the user's eyes.

[0186] As shown in FIG. 7C, alignment observer 620 may include a 3D position fit module 770. The position fit module 770 may, as an example, acquire and analyze various data including the left eye center of rotation 3D position (e.g., CoR left), the right eye center of rotation 3D position (e.g., CoR right), display-related properties, and fit tolerance. The 3D position fit module 770 may determine the distances of the user's left and right eyes from their respective left and right eye nominal positions (e.g., may calculate 3D left error and 3D right error) and provide the error distances (e.g., 3D left error and 3D right error) to the device 3D fit module 772.

[0187] The 3D position fit sense module 770 may also compare the error distance to the display-attached properties and the fit sense tolerance to determine whether the user's eye is within the nominal volume, a partially degraded volume (e.g., a volume in which the performance of the display 220 is partially degraded), or a completely degraded or almost completely degraded volume (e.g., a volume in which it is substantially impossible for the display 220 to provide content to the user's eye). In at least some embodiments, the 3D position fit sense module 770 or the 3D fit sense module 772 may provide an output that qualitatively describes the fit sense of the HMD on the user, such as the fit sense quality output shown in FIG. 7C. As an example, the module 770 may provide an output indicating whether the current fit sense of the HMD on the user is good, within tolerance, or a failure. A good fit sense may correspond to a fit sense that enables the user to view at least a certain percentage (such as 90%) of the image, a fit sense within tolerance may enable the user to view at least a lower percentage (such as 80%) of the image, while a failed fit sense may be a fit sense in which only an even lower percentage of the image is visible to the user.

[0188] As another example, the 3D position fit feeling module 770 and / or the device 3D fit feeling module 772 may calculate a visible area metric, which may be a percentage of the overall area (or pixels) of an image displayed by the display 220 visible to the user. Modules 770 and 772 may use one or more models (e.g., mathematical or geometric models), one or more look-up tables, or other techniques for determining the percentage of an image visible to the user as a function of the position of the user's eyes, or a combination of these and other techniques, to calculate the visible area metric by evaluating the positions of the user's left and right eyes relative to the display 220 (e.g., which may be based on the center of rotation of the user's eyes). Additionally, modules 770 and 772 may determine the area or portion of the image displayed by the display 220 that is expected to be visible to the user as a function of the position of the user's eyes.

[0189] The alignment observer 620 may also include the device 3D fit feeling module 772. Module 772 may receive data from the 3D position fit feeling module 770 and may also receive an eye tracking valid indicator, which may be provided by the eye tracking module 614 and may indicate whether the eye tracking system is currently tracking the position of the user's eyes or whether the eye tracking data is unavailable or under error conditions (e.g., determined to be unreliable). The device 3D fit feeling module 772 may, if desired, modify the fit feeling quality data received from the 3D position fit feeling module 770 according to the state of the eye tracking valid data. For example, if data from the eye tracking system indicates that it is not available or has an error, the device 3D fit feeling module 772 may provide a notification that an error exists and / or may not provide an output to the user regarding the quality of the fit feeling or the fit feeling error.

[0190] In at least some embodiments, the alignment observer 620 may provide feedback to the user regarding the quality of the fit as well as details of the nature and magnitude of the error. By way of example, a head-mounted display system may provide feedback to the user during a calibration or fitting process (e.g., as part of a setup procedure), and may also provide feedback during operation (e.g., if the fit degrades due to slippage, the alignment observer 620 may prompt the user to readjust the head-mounted display system). In some embodiments, the alignment analysis may be performed automatically (e.g., during use of the head-mounted display system), and the feedback may be provided without user input. These are merely illustrative examples. (Example of locating the user's cornea using an eye-tracking system)

[0191] FIG. 8A is a schematic view of an eye showing the corneal sphere of the eye. As shown in FIG. 8A, the user's eye 810 may have a cornea 812, a pupil 822, and a lens 820. The cornea 812 may have a generally spherical shape, represented by the corneal sphere 814. The corneal sphere 814 may have a center point 816, also referred to as the corneal center, and a radius 818. The hemispherical cornea of the user's eye may curve around the corneal center 816.

[0192] FIGS. 8B-8E illustrate examples of locating the user's corneal center 816 using the 3D corneal center estimation module 716 and the eye tracking module 614.

[0193] As shown in FIG. 8B, the 3D corneal center estimation module 716 may receive an eye tracking image 852 that includes a corneal flash 854. The 3D corneal center estimation module 716 may then simulate the known 3D positions of the eye camera 324 and the light source 326 (which may be obtained based on data in the eye tracking incidental and intrinsic property database 702, the assumed eye dimension database 704, and / or the per-user calibration data 706) within the eye camera coordinate system 850 in order to project a light ray 856 into the eye camera coordinate system. In at least some embodiments, the eye camera coordinate system 850 may have its origin at the 3D position of the eye tracking camera 324.

[0194] In FIG. 8C, the 3D corneal center estimation module 716 simulates a corneal sphere 814a (which may be obtained based on the assumed eye dimensions from the database 704) and a corneal curvature center 816a at a first position. The 3D corneal center estimation module 716 may then check whether the corneal sphere 814a would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8C, the first position does not match because the light ray 860a does not intersect the light source 326.

[0195] Similar to FIG. 8D, the 3D corneal center estimation module 716 simulates a corneal sphere 814b and a corneal curvature center 816b at a second position. The 3D corneal center estimation module 716 then checks whether the corneal sphere 814b would appropriately reflect light from the light source 326 to the flash position 854. As shown in FIG. 8D, the second position also does not match.

[0196] As shown in FIG. 8E, the 3D corneal center estimation module 716 can ultimately determine that the correct position of the corneal sphere is the corneal sphere 814c and the corneal curvature center 816c. The 3D corneal center estimation module 716 checks that the light from the source 326 will be properly reflected from the corneal sphere and imaged at the correct location of the flash 854 on the image 852 by the camera 324, thereby verifying that the illustrated position is correct. Using this arrangement, and the known 3D positions of the light source 326 and the camera 324, as well as the optical properties (such as focal length) of the camera, the 3D corneal center estimation module 716 can determine the 3D location (with respect to the wearable system) of the center 816 of the curvature of the cornea.

[0197] The processes described herein, at least in relation to FIGS. 8C - 8E, can in fact be an iterative, repetitive, or optimization process for identifying the 3D position of the user's corneal center. Thus, any of a plurality of techniques (e.g., iterative, optimization techniques, etc.) may be used to efficiently and quickly sort or reduce the search space of possible positions. Further, in some embodiments, the system may include two, three, four, or more light sources such as the light source 326, some of all of these light sources being arranged at different positions and resulting in a plurality of flashes such as the flash 854 located at different positions on the image 852 and a plurality of light rays such as the light ray 856 having different origins and directions. Such embodiments can improve the accuracy of the 3D corneal center estimation module 716 because the module 716 can search for corneal positions that result in some or all of the flashes and light rays being properly reflected between their individual light sources and their individual positions on the image 852. In other words, in these embodiments, the positions of some or all of the light sources can rely on the 3D corneal positioning (e.g., iterative, optimization techniques, etc.) process of FIGS. 8B - 8E. (Example of normalizing the coordinate system of the eye tracking image)

[0198] Figures 9A - 9C illustrate an exemplary normalization of the coordinate system of an eye - tracking image by components within a wearable system, such as the coordinate - system normalization module 718 of FIG. 7A. Normalization of the coordinate system of the eye - tracking image with respect to the location of the user's pupil can compensate for slippage of the wearable system relative to the user's face (i.e., headset slippage), and such normalization can establish a consistent orientation and distance between the eye - tracking image and the user's eye.

[0199] As shown in FIG. 9A, the coordinate - system normalization module 718 may receive the estimated 3D coordinates 900 of the center of rotation of the user's cornea and may receive an un - normalized eye - tracking image, such as image 852. The eye - tracking image 852 and the coordinates 900 may, as an example, be in an un - normalized coordinate system 850 based on the location of the eye - tracking camera 324.

[0200] In a first normalization step, the coordinate - system normalization module 718 may rotate the coordinate system 850 into a rotated coordinate system 902 such that the z - axis of the coordinate system (i.e., the vergence / accommodation depth axis) is aligned with the vector between the origin of the coordinate system and the corneal curvature center coordinates 900, as shown in FIG. 9B. In particular, the coordinate - system normalization module 718 may rotate the eye - tracking image 850 into a rotated eye - tracking image 904 until the coordinates 900 of the user's corneal curvature center are normal to the plane of the rotated image 904.

[0201] As a second normalization step, the coordinate - system normalization module 718 may translate the rotated coordinate system 902 into a normalized coordinate system 910 such that the corneal curvature center coordinates 900 are at a standard - normalized distance 906 from the origin of the normalized coordinate system 910, as shown in FIG. 9C. In particular, the coordinate - system normalization module 718 may translate the rotated eye - tracking image 904 into a normalized eye - tracking image 912. In at least some embodiments, the standard - normalized distance 906 may be about 30 millimeters. Optionally, the second normalization step may be performed prior to the first normalization step. (Example of using an eye - tracking system to locate the centroid of the user's pupil)

[0202] Figures 9D - 9G illustrate an example of locating the user's pupil center (i.e., the center of the user's pupil 822 as shown in Figure 8A) using the 3D pupil center locator module 720 and the eye tracking module 614.

[0203] As shown in Figure 9D, the 3D pupil center locator module 720 may receive a normalized eye tracking image 912 that includes the pupil centroid 913 (i.e., the center of the user's pupil as identified by the pupil identification module 712). The 3D pupil center locator module 720 may then simulate the normalized 3D position 910 of the eye camera 324 and project a light ray 914 through the pupil centroid 913 within the normalized coordinate system 910.

[0204] In Figure 9E, the 3D pupil center locator module 720 may simulate a corneal spherical surface such as the corneal spherical surface 901 having a center of curvature 900 based on data from the 3D corneal center estimation module 716 (and as discussed in more detail in relation to Figures 8B - 8E). As an example, the corneal spherical surface 901 may be positioned within the normalized coordinate system 910 based on the location of the center of curvature 816c identified in relation to Figure 8E and based on the normalization process of Figures 9A - 9C. Additionally, the 3D pupil center locator module 720 may identify a first intersection 916 between the light ray 914 (i.e., the light ray between the origin of the normalized coordinate system 910 and the normalized location of the user's pupil) and the simulated cornea as shown in Figure 9E.

[0205] As shown in FIG. 9F, the 3D pupil center locator module 720 may determine the pupil sphere 918 based on the corneal sphere 901. The pupil sphere 918 may share a common center of curvature with the corneal sphere 901 but may have a smaller radius. The 3D pupil center locator module 720 may determine the distance between the corneal center 900 and the pupil sphere 918 (i.e., the radius of the pupil sphere 918) based on the distance between the corneal center and the pupil center. In some embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the assumed eye dimensions 704 of FIG. 7A, from the eye tracking incidental and intrinsic property database 702, and / or from the per-user calibration data 706. In other embodiments, the distance between the pupil center and the corneal center of curvature may be determined from the per-user calibration data 706 of FIG. 7A.

[0206] As shown in FIG. 9G, the 3D pupil center locator module 720 may locate the 3D coordinates of the user's pupil center based on various inputs. As an example, the 3D pupil center locator module 720 uses the 3D coordinates and radius of the pupil sphere 918, the 3D coordinates of the intersection point 916 between the simulated corneal sphere 901 and the ray 914 associated with the pupil centroid 913 in the normalized eye tracking image 912, information regarding the refractive index of the cornea, and other relevant information such as the refractive index of air (which may be stored in the eye tracking incidental and intrinsic property database 702) to determine the 3D coordinates of the center of the user's pupil. In particular, in the simulation, the 3D pupil center locator module 720 may bend the ray 916 into the refracted ray 922 based on the refractive difference between air (at a first refractive index of approximately 1.00) and the corneal material (at a second refractive index of approximately 1.38). After considering the refraction caused by the cornea, the 3D pupil center locator module 720 may determine the 3D coordinates of the first intersection point 920 between the refracted ray 922 and the pupil sphere 918. The 3D pupil center locator module 720 may determine that the user's pupil center 920 is located approximately at the first intersection point 920 between the refracted ray 922 and the pupil sphere 918. By using this arrangement, the 3D pupil center locator module 720 can determine the 3D location of the pupil center 920 (with respect to the wearable system) within the normalized coordinate system 910. Optionally, the wearable system may denormalize the coordinates of the pupil center 920 into the original eye camera coordinate system 850. The pupil center 920 is used together with the corneal curvature center 900 to determine, among other things, the user's optical axis using the optical axis determination module 722 and the user's convergence / divergence motion depth using the convergence / divergence motion depth estimation module 728. (Example of the difference between the optical axis and the visual axis)

[0207] As discussed in connection with the optical axis / view axis mapping module 730 of FIG. 7A, the user's optical axis and view axis are generally not aligned, in part because the user's view axis is defined by its fovea, and the fovea is generally not at the center of a person's retina. Thus, when a person desires to focus on a particular object, the person aligns their view axis with the object, ensuring that light from the object strikes the fovea while their optical axis (defined by the center of their pupil and the center of curvature of their cornea) is actually slightly offset from the object. FIG. 10 is an example of an eye 1000 illustrating the optical axis 1002 of the eye, the view axis 1004 of the eye, and the offset between these axes. Additionally, FIG. 10 illustrates the pupil center 1006 of the eye, the center of curvature 1008 of the eye's cornea, and the eye's average center of rotation (CoR) 1010. In at least some populations, the center of curvature 1008 of the eye's cornea can be approximately 4.7 mm in front of the average center of rotation (CoR) 1010 of the eye, as indicated by dimension 1012. Additionally, the center of the eye's viewpoint 1014 can be approximately 5.01 mm in front of the center of curvature 1008 of the eye's cornea, approximately 2.97 mm behind the outer surface 1016 of the user's cornea, and / or directly in front of the user's pupil center 1006 (e.g., corresponding to a location within the anterior chamber of the eye 1000). As an additional example, dimension 1012 can be any range between 3.0 mm and 7.0 mm, 4.0 and 6.0 mm, 4.5 and 5.0 mm, or 4.6 and 4.8 mm, or any range between any value within these ranges. The center of the eye's viewpoint (CoP) 1014 can be a useful location for a wearable system in at least some embodiments, as aligning a rendering camera to the CoP can help reduce or eliminate parallax artifacts.

[0208] FIG. 10 also illustrates such a location within the human eye 1000 that can be aligned with the pinhole of the rendering camera. As shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 or the visual axis 1004 of the human eye 1000 that is closer to the outer surface of the cornea than both (a) the center of the pupil or iris 1006 and (b) the center 1008 of the corneal curvature of the human eye 1000. For example, as shown in FIG. 10, the pinhole of the rendering camera may be aligned with a location 1014 along the optical axis 1002 of the human eye 1000 that is about 2.97 millimeters rearward from the outer surface of the cornea and about 5.01 millimeters forward from the center 1008 of the corneal curvature. The location 1014 of the pinhole of the rendering camera and / or the anatomical region of the human eye 1000 corresponding to the location 1014 may be regarded as representing the center of the viewing point of the human eye 1000. The optical axis 1002 of the human eye 1000 as shown in FIG. 10 represents the shortest line passing through the center 1008 of the corneal curvature and the center of the pupil or iris 1006. The visual axis 1004 of the human eye 1000 is different from the optical axis 1002 because it represents a line extending from the fovea of the human eye 1000 to the center of the pupil or iris 1006. (Exemplary process for rendering content and checking alignment based on eye tracking)

[0209] FIG. 11 is a process flow diagram of an exemplary method 1100 for providing feedback regarding alignment within a wearable device using eye tracking when rendering content. Method 1100 may be implemented by the wearable system described herein. Embodiments of method 1100 may be used by the wearable system to render content and provide feedback regarding alignment (i.e., the fit of the wearable device with the user) based on data from an eye tracking system.

[0210] In block 1110, the wearable system may capture an image of one or both of the user's eyes. The wearable system may capture an eye image using one or more eye cameras 324, at least as shown in the embodiment of FIG. 3. Optionally, the wearable system may also include one or more light sources 326 configured to shine IR light onto the user's eyes and produce a corresponding flash within the eye image captured by the eye camera 324. As discussed herein, the flash may be used by the eye tracking module 614 to derive various information about the user's eyes, including where the eyes are looking.

[0211] In block 1120, the wearable system may detect the flash and the pupil within the eye image captured in block 1110. As an example, block 1120 may include processing the eye image by a flash detection and labeling module 714 to identify the two-dimensional position of the flash within the eye image, and processing the eye image by a pupil identification module 712 to identify the two-dimensional position of the pupil within the eye image.

[0212] In block 1130, the wearable system may estimate the three-dimensional positions of the user's left and right corneas relative to the wearable system. As an example, the wearable system may estimate the positions of the centers of curvature of the user's left and right corneas and the distances between those centers of curvature and the user's left and right corneas. Block 1130 may be accompanied by a 3D corneal center estimation module 716 that identifies the positions of the centers of curvature, at least as described herein in connection with FIGS. 7A and 8A-8E.

[0213] In block 1140, the wearable system may estimate the three-dimensional positions of the user's left and right pupil centers relative to the wearable system. As an example, the wearable system and the 3D pupil center locator module 720 may estimate the positions of the user's left and right pupil centers as part of block 1140, particularly as described in connection with FIGS. 7A and 9D-9G.

[0214] In block 1150, the wearable system may estimate the three-dimensional positions of the user's left and right centers of rotation (CoR) relative to the wearable system. As an example, the wearable system and the CoR estimation module 724 may estimate the positions of the CoR with respect to the user's left and right eyes, particularly as described in connection with at least FIGS. 7A and 10. As a particular example, the wearable system may find the eye's CoR by tracing back along the optical axis from the center of curvature of the cornea towards the retina.

[0215] In block 1160, the wearable system may estimate the user's IPD, convergence / divergence movement depth, center of perspective (CoP), optical axis, visual axis, and other desired attributes from the eye-tracking data. As an example, as part of block 1160, the IPD estimation module 726 may estimate the user's IPD by comparing the 3D positions of the left and right CoRs, the convergence / divergence movement depth estimation module 728 may estimate the user's depth by finding the intersection (or near intersection) of the left and right optical axes or the intersection of the left and right visual axes, the optical axis determination module 722 may identify the left and right optical axes over time, the optical axis / visual axis mapping module 730 may identify the left and right visual axes over time, and the CoP estimation module 732 may identify the left and right centers of perspective.

[0216] In block 1170, the wearable system may render content and optionally provide feedback regarding alignment (i.e., the fit of the wearable system to the user's head) based at least in part on the eye-tracking data identified in blocks 1120-1160. As an example, the wearable system may identify a suitable location for the rendering camera, as discussed in relation to the light field rendering controller 618 (FIG. 7B) and the rendering engine 622, and then generate content for the user based on the location of the rendering camera. As another example, the wearable system may determine whether it is properly fitted to the user or has slipped from its proper location relative to the user, as discussed in relation to the alignment observer 620, and optionally provide feedback to the user indicating whether adjustment of the fit of the device is needed. In some embodiments, the wearable system may adjust the rendered content based on improper or sub-optimal alignment in an attempt to reduce, minimize, or compensate for the effects of improper or misaligned alignment. (Overview of Device Alignment)

[0217] The wearable system 200 described herein, to output high-perception high-quality images, the display 220 of the wearable system 200 (FIG. 2) is preferably sized to fit the user appropriately (e.g., the input and output of the system 200 interface appropriately with the corresponding parts of the user's head and the device is positioned and oriented with respect to the user's head to be stable and comfortable for wearing and use). As an example, for the display 220 to provide visual content to the user's eyes, the display 220 is preferably positioned in front of the user's eyes and, depending on the relevant properties of the display 220, the user's eyes are preferably positioned within a particular volume. As an additional example, the speaker 240 is preferably positioned near, on, or in the user's ear to provide high-quality audio content to the user, the audio sensor (e.g., microphone) 232 is preferably positioned within a particular area to receive sound from the user, and the inward-facing imaging system 462 (which may include one or more cameras 324 and one or more infrared light sources 326) is preferably positioned in a position and orientation appropriate for obtaining a clear, unobstructed image of the user's eyes (which may be part of an eye-tracking system). These are merely examples of the various reasons why the wearable system 200 is preferably sized to fit the user appropriately.

[0218] To ensure that the wearable system 200 is properly aligned with the user, the wearable system 200 may include an alignment observer such as the alignment observer 620 of FIG. 6. In some embodiments, a properly aligned wearable system 200 includes a display positioned such that one or both eyes of the user receive sufficient image light and can view substantially the entire field of view provided by the display 220 of the wearable display system 200. For example, a properly aligned display may allow an image to be visible across about 80% or more, about 85% or more, about 90% or more, or about 95% or more of the field of view of the display, with a brightness uniformity of 80% or more, about 85% or more, about 90% or more, or about 95% or more. Brightness uniformity may be equal to 100% times the minimum luminance divided by the maximum luminance across the entire field of view of the display when the display shows the same content across the entire field of view (100%×L min / L max ). It should be understood that this can be the case.

[0219] The alignment observer 620 may use various sensors to determine the extent to which the wearable system 200 fits on the user (e.g., whether the display 220 of the wearable system 200 is properly positioned on the user). As an example, the alignment observer 620 may use an inward-facing imaging system 462, which may include an eye-tracking system, to determine the extent to which relevant portions of the wearable system 200 are spatially oriented with respect to the user, particularly the user's eyes, ears, mouth, or other portions that interface with the wearable system 200.

[0220] The alignment observer 620 may assist in the calibration process of an initial or subsequent configuration or setting, etc. of the wearable system 200 for a particular user. As an example, the alignment observer 620 may provide feedback to the user during the configuration or setting of the wearable system 200 for that particular user. Additionally, or alternatively, the alignment observer 620 may continuously or intermittently monitor the alignment of the wearable system 200 on the user, check for continuous proper alignment during use, and may provide on-the-fly user feedback. The alignment observer 620 may provide user feedback indicating when the wearable system 200 is properly aligned and when it is not, either as part of the configuration process or as part of the alignment monitoring during use. The alignment observer 620 may also provide specific recommendations for how the user may correct any alignment misalignment and achieve proper alignment. As an example, the alignment observer 620 may recommend that the user push the wearable device back up after detecting slippage of the wearable device (such as below the user's nasal bridge), or may recommend that the user adjust some adjustable components of the wearable device, etc. (Example of an alignment coordinate system)

[0221] Figures 12A-12B illustrate an exemplary eye position coordinate system that can be used to define the three-dimensional positions of a user's left and right eyes relative to a display of a wearable system described herein. As an example, the coordinate system may include axes x, y, and z. The z-axis of the coordinate system may correspond to depth (e.g., the direction normal to the plane of the front of the user's face), such as the distance between the plane in which the user's eyes are located and the plane in which the display 220 is located. The x-axis of the coordinate system may correspond to the left-right direction, such as the distance between the user's left and right eyes. In other words, the x-axis may generally be parallel to the inter-ocular axis (e.g., the line between corresponding left and right eye features such as the centers of rotation). The y-axis of the coordinate system may correspond to the up-down direction, which may be the vertical direction when the user is standing upright.

[0222] Figure 12A illustrates a side view of the user's eye 1200 and a display surface 1202 (which may be part of the display 220 of FIG. 2), while Figure 12B illustrates a top-down view of the user's eye 1200 and the display surface 1202. The display surface 1202 may be positioned in front of the user's eyes and may output image light to the user's eyes. As an example, the display surface 1202 may comprise one or more externally coupled light elements, active or pixel display elements, and may be part of a stack of waveguides such as the stacked waveguide assembly 480 of FIG. 4. In some embodiments, the display surface 1202 may be planar. In some other embodiments, the display surface 1202 may have other topologies (e.g., be curved). It should be understood that the display surface 1202 may be the physical surface of the display, or simply a plane or other imaginary surface from which it is understood that image light propagates from the display 220 to the user's eyes.

[0223] As shown in FIG. 12A, the user's eye 1200 may have an actual position 1204 that is offset from a nominal position 1206, and the display surface 1202 may be at position 1214. FIG. 12A also illustrates the corneal apex 1212 of the user's eye 1200. The user's line of sight (e.g., its optical axis and / or visual axis) may be substantially along the line between the actual position 1204 and the corneal apex 1212. As shown in FIGS. 12A and 12B, the actual position 1204 may be offset from the nominal position 1206 by a z-offset 1210, a y-offset 1208, and an x-offset 1209. The nominal position 1206 may represent a preferred position (sometimes also referred to as a design position that can generally be centered within a desired volume) for the user's eye 1200 relative to the display surface 1202. As the user's eye 1200 moves away from the nominal position 1206, the performance of the display surface 1202 may degrade.

[0224] It should be understood that a point or volume associated with the user's eye 1200 can be used to represent the position of the user's eye in the alignment analysis herein. The representative point or volume can preferably be any point or volume associated with the eye 1200 that is consistently used. For example, the point or volume may be on or within the eye 1200 or may be disposed away from the eye 1200. In some embodiments, the point or volume is the center of rotation of the eye 1200. The center of rotation can be determined as described herein and is disposed generally symmetrically about various axes within the eye 1200 and may have the advantage of simplifying the alignment analysis to enable a single display alignment volume that is aligned with the optical axis to be utilized for the analysis.

[0225] FIG. 12A also illustrates that the display surface 1202 can be aligned below the user's horizontal line (as seen along the y-axis when the user is looking straight ahead with the optical axis of the user parallel to the ground) and can be tilted (with respect to the y-axis). In particular, the display surface 1202 can be disposed slightly below the user's horizontal line such that the user would need to look down at approximately angle 1216 to view the center of the display surface 1202 when the eye 1200 is at position 1206. This can promote a more natural and comfortable interaction with the display surface 1202, especially when viewing content rendered at a shorter depth (or distance from the user), since the user may find it more comfortable to view content below their horizontal line than above their horizontal line. Additionally, the display surface 1202 can be tilted at an angle 1218 etc. (with respect to the y-axis) such that the display surface 1202 is substantially perpendicular to the user's line of sight when the user is looking at the center of the display surface 1202 (e.g., looking slightly below the user's horizontal line as described below). In at least some embodiments, the display surface 1202 may also be offset to the left or right (e.g., along the x-axis) with respect to the nominal position of the user's eyes. As an example, the left-eye display surface may be offset to the right and the right-eye display surface may be offset to the left (e.g., the display surfaces 1202 may be offset towards each other) such that the user's line of sight strikes the center of the display surface when focused at a distance less than infinity, which can increase user comfort during typical use on a wearable device. (Example of vertically aligning the left and right eye displays on the user's head)

[0226] As described above, proper alignment between the left and right eye displays of a head-mounted display (HMD) (e.g., display 220 within wearable system 200 of FIG. 3) and the user's eyes can be important to ensure a desirable user experience. For example, when the left eye display is vertically offset from the user's left eye by a different amount than the right eye display is vertically offset from the user's right eye due to normal wear of the device over time or by other means of deformation, the user may experience discomfort and eye strain.

[0227] In some embodiments, the head-mounted display system may be configured to determine whether its left and right eye displays are horizontal with respect to the user's eyes (e.g., their interpupillary axis), with the ultimate goal of correcting the calibration metric of the left or right display subsystem (such as a projector input for a corresponding left or right eyepiece lens) so that the images projected through the left and right eye displays are horizontal with respect to the interpupillary axis between the eyes, provide appropriate user feedback, and prompt the user to adjust the images displayed on the left or right eyepiece lens.

[0228] The display system may then provide the user with left and right eye images having alignment markers. The alignment markers may be provided such that any vertical offset between the left eye display and the right eye display is visible to the user. In other words, if there is a vertical offset between the left eye display and the right eye display (e.g., the left eye display is vertically offset with respect to the right eye display), the user may perceive that the images from the left and right eye displays are vertically misaligned and that it is impossible to properly fuse the images (e.g., the offset may be visible in the form of a horizontal portion of the alignment markers that are not aligned with each other). It should be understood that the horizontal portions of the alignment markers may be in the form of other mirror image shapes or arrangements of lines that do not completely overlap when vertically misaligned.

[0229] Figures 13 and 14 illustrate exemplary display screens that may be provided to a user of a head-mounted display (HMD) (e.g., display 220, FIG. 3) as part of a step of leveling the HMD with respect to the user's eyes (e.g., the inter-pupillary axis thereof) (e.g., leveling a horizontal axis extending through the left and right eye displays of the HMD). In particular, FIG. 13 illustrates a display screen 1300 that includes an exemplary guide that may be provided to assist the user in leveling the HMD.

[0230] The display screen 1300 may also include feedback regarding the current orientation of the HMD with respect to the user's eyes. Such feedback may take the form of an avatar 1304 (e.g., a virtual representation of the user) and a virtual depiction 1302 of the HMD. The avatar 1304 may be depicted in a vertical orientation (such that the interpupillary axis is in a horizontal orientation), while the orientation of the virtual depiction 1302 of the HMD may be tilted with respect to the horizontal. The amount and direction of the tilt of the virtual depiction 1302 may be based on the estimated orientation of the interpupillary axis with respect to the HMD (e.g., as determined by module 740 of FIG. 7A). In other embodiments, the avatars 1302 and 1304 may be omitted or may be static. In various embodiments, the amount and direction of the tilt of the HMD may be communicated to the user via other indicia such as alignment markers 1306 and 1308. The alignment marker 1306 may correspond to the user's interpupillary axis (e.g., presented as a horizontal line), while the alignment marker 1308 may indicate the tilt of the HMD with respect to the interpupillary axis. In one example, the marker 1306 may remain static, while the marker 1308 shifts as the user adjusts the HMD on their head towards horizontal. In some embodiments, the orientation of the HMD is determined using various sensors attached to the HMD that image one or more reference points within the external environment and determine the orientation of the HMD based on those reference points, including gyroscopes, accelerometers, and / or imaging systems. Generally, the alignment markers may take any suitable shape or form. By using alignment markers, the user may be able to quickly perceive whether the HMD is tilted on their head, including the direction and amount. Thus, the user can correct the tilt of the HMD until it is properly horizontal.

[0231] As shown in FIG. 14, when the user properly adjusts the inclination of the HMD so that the HMD is horizontal on his or her head (for example, when the interpupillary axis or a part thereof is parallel to the horizontal axis extending between the left-eye display and the right-eye display of the HMD, when the interpupillary axis or a part thereof falls within a predetermined volume defined with respect to the HMD, etc.), the user may be presented with a display screen such as screen 1400 indicating the success of leveling the HMD. In the embodiment of FIG. 14, the user properly levels the HMD, and marker 1402 is shown. As an example, marker 1402 may appear when marks 1306 and 1308 of FIG. 13 overlap.

[0232] Subsequent to leveling the HMD on the user, the system may perform a left-eye and right-eye display alignment protocol to appropriately level the display of the image through a rigid display that is not vertically aligned by itself. In particular, the display alignment process may check and identify any differences in the vertical alignment between the left and right displays and their respective eyes (e.g., automatically or based on user feedback). In other words, the display alignment process may check whether the user's left eye has a first vertical alignment with the left-eye display of the HMD (such as a first offset along the Y-axis in FIG. 12A), while the user's right eye has a second different vertical alignment with the right-eye display of the HMD (such as a second offset of a different magnitude along the Y-axis in FIG. 12) (e.g., automatically or by providing an alignment marker that highlights any misalignment to the user and receiving user feedback regarding any misalignment). Differences in the vertical alignment between the left and right sides may cause one or both of the left-eye and right-eye displays to be offset so that they are not at the same vertical height even when the HMD is determined to be horizontal with respect to the interpupillary axis of the user's eyes. It should be understood that this can occur even when the HMD is mounted horizontally on the user's head due to various factors such as warping or damage to the HMD.

[0233] Figures 15 and 16 illustrate exemplary display screens that may be provided to a user of a head-mounted display (HMD) (e.g., display 220, FIG. 3) as part of a step of identifying vertical misalignments remaining after leveling. As shown in FIG. 15, a display screen such as screen 1500 may include user instructions and alignment markers such as markers 1502, 1504, and 1506 that highlight any vertical misalignments to the user. In some embodiments, the alignment markers may take the form of the letter "T" extending laterally. However, it should be understood that the alignment markers may take other shapes or forms.

[0234] As shown in FIG. 16, the HMD may provide different individual alignment markers to the user's left and right eyes to demonstrate any left-right vertical misalignment. For example, the HMD may display screen 1600a to the user's left eye and screen 1600b to the user's right eye. Screen 1600a may include a left-eye horizontal alignment marker 1502 and a vertical alignment marker 1506, while screen 1600b may include a right-eye horizontal alignment marker 1504 and a vertical alignment marker 1506. In some embodiments, screen 1600a may not include the right-eye horizontal alignment marker 1504, and screen 1600b may not include the left-eye horizontal alignment marker 1502.

[0235] When a user wearing the HMD views screens 1600a and 1600b simultaneously (e.g., using their left and right eyes respectively), the vertical alignment markers 1506a and 1506b may appear to the user as being fused together (e.g., as the mark 1506). However, at least the left and right eye horizontal alignment markers 1502 and 1504 are not spatially aligned with each other, so the user does not perceive the marks 1502 and 1504 as being fused together. Instead, the user will likely be able to determine whether the marks 1502 and 1504 are vertically aligned with each other (or whether there is a left-right vertical misalignment). In other words, an incorrectly aligned system will display the intended image of 1604 as the exemplary misaligned image 1602.

[0236] In at least some embodiments, alignment markers 1502 and 1504 are initially presented at the same location on the left and right eye displays, but variations can encourage the perception of misalignment of the locations. In some embodiments, the system may intentionally introduce a vertical offset between alignment markers 1502 and 1504. In such embodiments, the alignment of markers 1502 and 1504 does not necessarily represent the alignment of the left and right eye displays. For example, the left and right eye displays may exhibit relatively little or no misalignment, but in these embodiments, markers 1502 and 1504 presented by the system may exhibit a relatively large amount of misalignment. Doing so can serve to facilitate user involvement in the display alignment process. That is, some users may feel more compelled to actively participate in the display alignment process when markers 1502 and 1504 presented by the system exhibit a relatively high degree of misalignment. In some examples, the vertical positions of alignment markers 1502 and 1504 may be randomized, pseudo-randomized, or quasi-randomized. For example, the system may randomly, pseudo-randomly, or quasi-randomly select the vertical positions of alignment markers 1502 and 1504 from a predetermined range of vertical positions. In some embodiments, the system may randomly, pseudo-randomly, or quasi-randomly select the offset between alignment markers 1502 and 1504 from a predetermined range of offsets. In some embodiments, the display system may be configured to randomly, pseudo-randomly, or quasi-randomly select an offset based on a pseudo-random number generator such as a Mersenne Twister, Xorshift, etc. In some embodiments, the pseudo-random number generator may directly generate an offset value.In some other embodiments, each offset may be associated with a number or set of numbers, and the generation of numbers by a pseudo-random number generator may be used to select an offset that is associated with the generated number or the set to which the generated number belongs (e.g., each offset may be associated with one or more unique numbers or sets of numbers, the numbers may be generated using a pseudo-random generator, and the offset to be applied may be selected based on the correspondence between the generated number (or set to which the number belongs) and the number (or set of numbers) associated with the offset).

[0237] The system may consider the selected vertical positions of markers 1502 and 1504 and / or the selected offset between markers 1502 and 1504 when the user provides an input for vertically aligning markers 1502 and 1504. After the user provides an input for vertically aligning markers 1502 and 1504, the system may be able to determine the magnitude and direction of the left-right vertical misalignment based on the magnitude and direction of the user input. Preferably, the adjustment of the vertical alignment is performed on only one display at a time.

[0238] Regarding embodiments in which the HMD includes a stack of waveguides, the adjustment of vertical alignment may be performed on only one waveguide at a time. Thus, in some embodiments, the system may perform a display alignment process for each waveguide included within the waveguide stack of the HMD. In some examples, the waveguide stack of the HMD may include a plurality of waveguides, for example, three waveguides per depth plane. The number of waveguides corresponds to the number of primary colors used by the display system and may form full-color content. For example, for an RGB (red, green, blue) display system, the three waveguides provided per depth plane may include a waveguide configured to output red light, a waveguide configured to output green light, and a waveguide configured to output blue light. The system may present markers 1502 and 1504 in each primary color, e.g., red when performing display alignment for a waveguide configured to output red light, green when performing display alignment for a waveguide configured to output green light, and blue when performing display alignment for a waveguide configured to output blue light. In some embodiments, the system may perform display alignment for each waveguide sequentially (e.g., the alignment may be performed sequentially for different waveguides corresponding to different primary colors and / or different waveguides providing different amounts of wavefront divergence corresponding to different depth planes). In some other embodiments, the system may perform display alignment simultaneously for some or all of the waveguides within the waveguide stack of the HMD. In such embodiments, markers 1502 and 1504 may be presented simultaneously in various colors (e.g., red, green, and blue versions of markers 1502 and 1504) and / or at various depths (using waveguides providing different amounts of wavefront divergence), and the user may be able to provide an input to vertically shift one marker at a time.

[0239] It should be understood that user input may be provided using any input device available to the user. For example, gestures, voice activation commands, physical toggles, etc. may be used to provide input. As a specific example, FIG. 17 is a perspective view of a user's hand 1700 and a controller 1701 including various input devices 1702, 1704, 1706. Any of the input devices 1702, 1704, 1706 may be configured to provide input regarding a desired vertical offset for aligning alignment markers 1502 and 1504. (Exemplary process for vertically aligning left and right eye displays on the user's head)

[0240] FIG. 18 is a process flow diagram of an exemplary method 1800 for leveling a head-mounted display system on a user's head and adjusting the vertical alignment of the left and right displays of the display system. Method 1800 may be implemented by the wearable system described herein.

[0241] In block 1802, the wearable system may determine a potential need to adjust the calibration of a head-mounted display (HMD) (e.g., display 220, FIG. 3). The wearable system may determine, individually or in combination, based on various factors including, in response to startup of the HMD, in response to launching an application on the HMD, in response to determining that the HMD has been placed on the user's head, etc., the number of hours since the last calibration, detection of potential or actual deformation of the HMD (e.g., detection of an impact event with an accelerometer or other sensor), detection of misalignment of the HMD (e.g., determination by an eye tracking component that one or both of the user's eyes are not in the designed position relative to the HMD), that the HMD needs to be calibrated. As an additional option, the calibration process may be recommended to the user by a user manual, customer service, or other source, and block 1802 may involve receiving a user request to perform the calibration process.

[0242] In block 1804, the wearable system may perform a device leveling flow, which may include blocks such as blocks 1806 and 1808. In block 1806, the HMD may determine the interpupillary axis of the user and the horizontal axis of the device, which is an axis extending between the left and right eyes displays of the HMD or can be measured relative to the gravity vector of the world coordinate frame of the HMD. Block 1806 may involve determining whether the interpupillary axis and the horizontal axis of the device are parallel or whether there is an inclination between the two axes. In some embodiments, block 1806 may involve determining whether the interpupillary axis or a part thereof falls within a predetermined volume defined with respect to the HMD. In block 1808, the HMD may provide feedback to the user regarding the leveling of the HMD. Such feedback may take the form of displaying the alignment markers 1306 and 1308 of FIG. 13. Blocks 1806 and 1808 may continue until the user finishes the alignment process or levels the HMD properly on their head (e.g., aligns mark 1308 with mark 1306).

[0243] In block 1810, the wearable system may perform a display alignment flow, which may include blocks such as blocks 1812 and 1814. In block 1812, the HMD may provide non-fused left and right eye alignment markers. As an example, the HMD may display the alignment marker 1502 of FIGS. 15 and 16 on the left eye display and the alignment marker 1504 of FIGS. 15 and 16 on the right eye display, as discussed herein. As described above, in some embodiments, the HMD may display one or both of the alignment markers 1502 and 1504 at randomly, pseudo-randomly, or quasi-randomly selected vertical positions. In block 1814, the wearable system may receive user feedback regarding left or right eye display vertical alignment adjustment. In particular, as discussed in relation to FIGS. 15-17, the wearable system may receive user input to shift at least one of the alignment markers 1502 and 1504 until the markers are perpendicularly aligned with each other from the user's perspective. Blocks 1812 and 1814 may continue until the user ends the alignment process or approves any vertical adjustments made to the alignment markers. As described above, in some embodiments, the wearable system may perform one display alignment process per waveguide included in the HMD. In these embodiments, the wearable system may perform operations associated with one or more of blocks 1810, 1812, and 1814 for each waveguide included in the HMD.

[0244] In block 1816, the wearable system may implement any adjustment provided by the user for left - right vertical alignment. As an example, if user feedback indicates that the right - eye display is lower than the left - eye display by a given amount, the wearable system may compensate for the misalignment by shifting the image content displayed via the right - eye display upward by a given amount, by shifting the image content displayed via the left - eye display downward by a given amount, or by shifting an appropriate combination of the right - and left - eye content, following completion of the alignment process of FIG. 18, for example.

[0245] In some embodiments, in block 1816, the wearable system may adjust one or more incidental parameters of a rendering camera associated with the left - eye display and one or more incidental parameters of a rendering camera associated with the right - eye display. For example, in these embodiments, the wearable system may adjust the position and / or orientation of one or both of two rendering cameras (e.g., the left - eye display and the right - eye display) at least partially based on input provided by the user. In some embodiments, the wearable system may skip operations associated with blocks 1804, 1806, and / or 1808. In these embodiments, the wearable system may proceed to block 1810 immediately after performing the operation associated with block 1802. (Computer vision for detecting objects in the surrounding environment)

[0246] As discussed above, the display system may be configured to detect objects or their properties in the environment surrounding the user. The detection may be accomplished using various techniques, including various environmental sensors (e.g., cameras, audio sensors, temperature sensors, etc.) as discussed herein.

[0247] In some embodiments, objects present in the environment may be detected using computer vision techniques. For example, as disclosed herein, a camera facing forward of the display system may be configured to image the surrounding environment, and the display system may be configured to perform image analysis on the image to determine the presence of objects in the surrounding environment. The display system may analyze an image obtained by an outward-facing imaging system and perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. As another example, the display system may be configured to perform face and / or eye recognition to determine the presence and location of faces and / or human eyes within the user's field of view. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), histogram of feature points, various machine learning algorithms (e.g., support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network), or other supervised / unsupervised models, etc.), etc.

[0248] One or more of these computer vision techniques may also be used in combination with data obtained from other environmental sensors (e.g., microphones, etc.) to detect and determine various properties of the objects detected by the sensors.

[0249] As discussed herein, objects in the surrounding environment may be detected based on one or more criteria. When a display system uses a computer vision algorithm or uses data received from one or more sensor assemblies (which may or may not be part of the display system) to detect the presence or absence of a criterion in the surrounding environment, the display system may then signal the presence of the object. (Machine learning)

[0250] Various machine learning algorithms may be used to learn to identify the presence of objects in the surrounding environment. Once trained, the machine learning algorithms may be stored by the display system. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., Apriori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Generalization, etc.), and / or other machine learning algorithms. In some embodiments, the individual models may be customized for individual data sets. For example, a wearable device may generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user), a data set (e.g., a set of additional images to be acquired), a conditional situation, or other variations. In some embodiments, the display system may be configured to generate a model for the analysis of aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0251] The criteria for detecting an object may include one or more threshold conditions. If the analysis of the data obtained by the environmental sensor indicates that the threshold condition has been reached, the display system may provide a signal indicating the detection of the presence of an object in the surrounding environment. The threshold condition may involve quantitative and / or qualitative measurement values. For example, the threshold condition may include a score or percentage associated with the likelihood of reflection and / or the presence of an object in the environment. The display system may compare the score calculated from the data of the environmental sensor with a threshold score. If the score is higher than the threshold level, the display system may detect the presence of reflection and / or an object. In some other embodiments, the display system may signal the presence of an object in the environment if the score is lower than the threshold. In some embodiments, the threshold condition may be determined based on the emotional state of the user and / or the interaction of the user with the surrounding environment.

[0252] In some embodiments, the threshold condition, machine learning algorithm, or computer vision algorithm may be specialized for a specific context. For example, in a diagnostic context, the computer vision algorithm may be specialized to detect a certain response to a stimulus. As another example, the display system may execute a face recognition algorithm and / or an event tracing algorithm as discussed herein to sense the user's reaction to a stimulus.

[0253] The processes, methods, and algorithms described in this specification and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being capable of being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in a dynamic link library that can be compiled and linked into an executable program, or can be written in an interpreted type programming language. In some embodiments, certain operations and methods can be performed by circuits specific to a given function.

[0254] Furthermore, the functional embodiments of the present disclosure are sufficiently mathematically, computationally, or technically complex that a hardware specific to a particular use or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0255] A code module or any type of data can be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same, and / or equivalents. In some embodiments, the non-transitory computer-readable medium may be part of one or more of the local processing and data module (140), the remote processing module (150), and the remote data repository (160). The method and module (or data) may also be transmitted as a data signal generated on various computer-readable transmission media, including wireless-based and wire / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and may take various forms (e.g., as part of a single or multiplexed analog signal, or as a plurality of discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device, or communicated via a computer-readable transmission medium.

[0256] Any process, block, state, step, or functionality in the flow diagrams described and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the embodiments described herein is for purposes of illustration and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. (Other Considerations)

[0257] The processes, methods, and algorithms described herein and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules may be installed in a dynamic link library that can be compiled and linked into an executable program, or may be written in an interpreted type programming language. In some embodiments, certain operations and methods may be implemented by circuits specific to a given function.

[0258] Furthermore, the functional embodiments of the present disclosure are sufficiently mathematically, computationally, or technically complex such that a special purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video or video may include many frames, each frame may have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0259] A code module or any type of data can be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media, including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and can take various forms (e.g., as part of a single or multiplexed analog signal, or as a plurality of discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device, or communicated via a computer-readable transmission medium.

[0260] Any process, block, state, step, or functionality in a flowchart described in and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states may be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many variations of the embodiments are possible.

[0261] The present process, method, and system can be implemented in a network (or distributed) computing environment. The network environment can include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0262] The systems and methods of the present disclosure each have several innovative aspects, none of which alone participate in or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications of the embodiments described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the embodiments shown herein but are to be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0263] In the context of separate implementations, certain features described herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Further, features may be described above as acting in a certain combination and may further be claimed as such initially, but one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination. No single feature or group of features is necessary or essential to every embodiment.

[0264] In particular, conditional clauses used herein such as “~ can”, “~ could”, “~ might”, “~ may”, “e.g.”, and equivalents, generally convey that while one embodiment includes certain features, elements, or steps, other embodiments do not, unless specifically described otherwise or understood otherwise within the context in which they are used. Thus, such conditional clauses are not generally intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms “comprising”, “including”, “having”, and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Additionally, the articles “a”, “an”, and “the” as used in this application and the appended claims should be construed to mean “one or more” or “at least one” unless otherwise defined.

[0265] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in the context in which they are used to convey that the items, terms, etc. can generally be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such connective phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0266] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not necessary for achieving the desired result, and that such operations may be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations need not be performed. Additionally, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated within the exemplary methods and processes schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the illustrated operations. Additionally, the operations may be rearranged or reordered in other embodiments. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described are generally integrated together in a single software product or may be packaged into multiple software products. Additionally, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

Claims

1. 1. An augmented reality system, comprising: a head mounted display configured to present virtual content by outputting light to a user; an imaging system configured to capture an image of the user's eye; at least one processor communicatively coupled to the head mounted display and the imaging system; one or more computer storage media having instructions stored thereon; Equipped with The instructions, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: performing a head mounted display leveling operation prior to performing an image content alignment process for each eye of the user, the head mounted display leveling operation including: determining a position of the user's interocular axis extending between the user's left eye and right eye based at least in part on one or more images captured by the imaging system; determining an orientation of the head mounted display relative to the interocular axis of the user; providing feedback to the user based on the determined orientation of the head mounted display relative to the interocular axis of the user to enable the user to level the head mounted display relative to the interocular axis of the user; augmented reality systems, including

2. Providing feedback to the user based on the determined orientation of the head mounted display relative to the interocular axis of the user includes: The augmented reality system of claim 1 , comprising presenting a horizontal marker having an orientation that varies relative to the determined orientation of the head mounted display relative to the user's left and right eyes.

3. Providing feedback to the user based on the determined orientation of the head mounted display relative to the interocular axis of the user includes: presenting a static horizon marker associated with an orientation of the head mounted display, the static horizon marker being static with respect to the head mounted display; presenting a dynamic horizontal marker associated with an orientation of the interocular axis, the dynamic horizontal marker moving relative to the static horizontal marker of the head mounted display as an orientation of the interocular axis changes relative to the head mounted display when the head mounted display is adjusted on the user's head; The augmented reality system of claim 1 .

4. The augmented reality system of claim 3 , wherein the dynamic horizontal marker merges with the static horizontal marker when the head mounted display is horizontal to the interocular axis.

5. Determining the interocular axis of the user extending between a left eye and a right eye of the user based at least in part on one or more images captured by the imaging system includes: determining a center of rotation for the user's left eye based at least in part on one or more images of the user's left eye captured by the imaging system; determining a center of rotation of the user's right eye based at least in part on one or more images of the user's right eye captured by the imaging system; determining a position of a line extending between centers of rotation of the user's left and right eyes, the line constituting the interocular axis of the user; The augmented reality system of claim 1 .

6. The operations further include, in response to determining that the head mounted display is horizontal with respect to the left and right eyes of the user, performing the image content alignment process for each eye of the user; The image content matching process comprises: providing a left eye alignment marker using a left eye display of the head mounted display; providing a right eye alignment marker using a right eye display of the head mounted display; receiving a user input to adjust at least one of the left eye alignment marker and the right eye alignment marker; vertically adjusting image content on the at least one of the left-eye display and the right-eye display based on the received user input; The augmented reality system of claim 5 .

7. The augmented reality system of claim 6 , wherein the left eye alignment marker comprises a first horizontal line and the right eye alignment marker comprises a second horizontal line.

8. Providing feedback to the user based on the determined orientation of the head mounted display relative to the interocular axis of the user includes: The augmented reality system of claim 5 , further comprising providing feedback to the user indicating whether the head mounted display is level with respect to the user's left and right eyes.

9. The operation further comprises: determining whether the user has been wearing the head mounted display for at least a threshold amount of time; determining the position of the interocular axis based at least in part on one or more images captured by the imaging system in response to determining that the user has been wearing the head mounted display for at least the threshold amount of time; The augmented reality system of claim 1 .

10. The operation further comprises:

2. The augmented reality system of claim 1, further comprising determining whether the head mounted display is horizontal with respect to the user's left and right eyes based on the determined orientation of the head mounted display relative to the user's interocular axis.

11. 2. The augmented reality system of claim 1, wherein the head-mounted display comprises a left eye display and a right eye display, the left eye display comprising a first waveguide stack configured to pass light from the world into the user's left eye, and the right eye display comprising a second waveguide stack configured to pass light from the world into the user's right eye, each waveguide stack comprising a plurality of waveguides.

12. 12. The augmented reality system of claim 11, wherein at least some of the waveguides of the first waveguide stack are configured to output light of a different primary color than other waveguides of the first waveguide stack, and at least some of the waveguides of the second waveguide stack are configured to output light of a different primary color than other waveguides of the second waveguide stack.

13. one or more waveguides in each of the first and second waveguide stacks are configured to output light to the user with a different amount of wavefront divergence than one or more other waveguides in that waveguide stack; The augmented reality system of claim 11 , wherein the output light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the user.

14. 14. The augmented reality system of claim 13, wherein the operations further include performing the image content matching process for each eye of the user, the image content matching process including performing a plurality of matching processes, the matching process for determining a matching between the left eye of the user and a waveguide of the first waveguide stack, a matching process being performed for each waveguide of the first waveguide stack.

15. 15. The augmented reality system of claim 14, wherein the matching process further includes determining a matching between the user's right eye and a waveguide of the second waveguide stack, the matching process being performed for each waveguide of the second waveguide stack.

16. 2. The augmented reality system of claim 1, wherein the operations further include determining whether the head mounted display is level with respect to the user's left and right eyes based on eye tracking data from the imaging system.

17. The augmented reality system of claim 1 , wherein the imaging system comprises a camera.

18. The augmented reality system of claim 17 , wherein the camera is an infrared camera.

19. 1. An augmented reality system, comprising: a head mounted display configured to present virtual content by outputting light to a user, the head mounted display including a left eye display and a right eye display, the left eye display comprising a first waveguide stack configured to pass light from the world into the left eye of the user, and the right eye display comprising a second waveguide stack configured to pass light from the world into the right eye of the user, each waveguide stack comprising a plurality of waveguides, one or more waveguides of each of the first and second waveguide stacks configured to output light to the user with a different amount of wavefront divergence than one or more other waveguides of that waveguide stack, the output light with different amounts of wavefront divergence forming virtual objects at different perceived depths away from the user; an imaging system configured to capture an image of the user's eye; at least one processor communicatively coupled to the head mounted display and the imaging system; one or more computer storage media having instructions stored thereon; Equipped with The instructions, when executed by the one or more processors, cause the one or more processors to perform operations including performing a plurality of matching processes, the matching processes for determining a matching between the user's left eye and a waveguide of the first waveguide stack, a matching process being performed for each waveguide of the first waveguide stack.

20. 20. The augmented reality system of claim 19, wherein the matching process further includes determining a matching between the user's right eye and a waveguide of the second waveguide stack, the matching process being performed for each waveguide of the second waveguide stack.

Images