Determination of the center of eye rotation using one or more eye-tracking cameras

Abstract

To perform determination of an eye center of rotation using one or more eye tracking cameras.SOLUTION: A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints thereon and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of rotation of the user's eye using cornea data derived from the glint images. The display system may render a virtual image content by using a render camera positioned at the determined position of the center of rotation of said eye.SELECTED DRAWING: Figure 9B

Description

Technical Field

[0003] , ,

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 62 / 874,867, filed Jul. 16, 2019, titled "EYE CENTER OF ROTATION DETERMINATION WITH ONE OR MORE EYE TRACKING CAMERAS", the content of which is incorporated herein by reference in its entirety. This application is related to U.S. Application No. 16 / 250,931, filed Jan. 17, 2019, titled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS", and U.S. Patent Publication No. 2018 / 0018515, published Jan. 18, 2018, titled "IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE", the content of which is incorporated herein by reference in its entirety.

[0002] (Field) The present disclosure relates to display systems, virtual reality, and augmented reality imaging and visualization systems, and more particularly to eye tracking using the center of rotation of an eye calculated using corneal data.

Background Art

[0003] (Background) Modern computing and display technologies are driving the development of systems for so-called “virtual reality,” “augmented reality,” or “mixed reality” experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears, or can be perceived, as real. Virtual reality, or “VR,” scenarios typically involve the presentation of digital or virtual image information without transparency to other real-world visual inputs. Augmented reality, or “AR,” scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality, or “MR,” relates to the fusion of the real and virtual worlds to generate a new environment in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it challenging to produce VR, AR, or MR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges related to VR, AR, and MR technologies. [Overview of the Initiative] [Means for solving the problem]

[0004] (summary) Various embodiments of depth plane selection in mixed reality systems are disclosed.

[0005] A display system may be configured to project light onto the user's eye to display virtual image content within the user's field of view. The user's eye may have a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through the lens, pupil, and cornea. The display system may include a frame configured to be supported on the user's head, a head-mounted display positioned on the frame, one or more eye-tracking cameras configured to image the user's eye, and processing electronics communicating with the display and the one or more eye-tracking cameras, which are configured to obtain estimates of the eye's parameters based on images of the eye acquired using the one or more eye-tracking cameras. In some implementations, the eye's parameters include the center of corneal curvature (e.g., the center of curvature measured at the corneal apex). In some implementations, the center of corneal curvature or corneal center refers to the center of curvature of a portion of the cornea or the center of curvature of a spherical surface that coincides with a portion of the corneal surface. For example, in some implementations, the center of corneal curvature or the center of the cornea refers to the center of curvature of the corneal apex or the center of curvature of the spherical surface that coincides with a portion of the surface of the corneal apex. In some implementations, the parameters of the eye include the center of rotation of the eye. Other parameters and information may be determined similarly.

[0006] In some implementations, the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, displaying virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths. In some implementations, the displayed virtual image content appears to originate from different depths at different time periods.

[0007] Various embodiments of a display system that projects light onto one or both of a user's eyes to display virtual image content within the user's field of view are described herein, including the embodiments listed below. Part I

[0008] Example 1: A display system configured to project light into a user's eye for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eye and display virtual image content within the user's field of view; first and second eye-tracking cameras configured to image the user's eye; a plurality of light emitters; and a processing electronic device communicating with the display and the first and second eye-tracking cameras, which receives images of the user's eye captured by the first and second eye-tracking cameras, and is configured to estimate the location of the center of the corneal curvature of the user's eye based on the location of the flash reflections in the image produced by both the first and second eye-tracking cameras and the location of the emitter that produced the individual flash reflections.

[0009] Example 2: A display system configured to project light into a user's eyes for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eyes and display virtual image content within the user's field of view; first and second eye-tracking cameras configured to image the user's eyes; a plurality of light emitters; and a processing electronic device communicating with the display and the first and second eye-tracking cameras, which receives images of the user's eyes captured by the first and second eye-tracking cameras, and is configured to estimate the location of the rotation center of the user's eyes based on the location of the flash reflections in the images produced by both the first and second eye-tracking cameras and the location of the emitters that produced the flash reflections with respect to the location of both the first and second eye-tracking cameras and a plurality of eye poses.

[0010] Example 3: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, the eye comprising the steps of capturing multiple images of the user's eye using a plurality of eye-tracking cameras having a cornea and configured to image the user's eye, and a plurality of light emitters positioned relative to the eye and forming flashes thereon, wherein the images comprise a plurality of flashes; and obtaining an estimate of the center of rotation of the eye based on the plurality of flashes, the step of obtaining an estimate of the center of rotation of the eye comprising the steps of determining a plurality of estimates of the center of corneal curvature of the user's eye based on the plurality of flashes; generating a three-dimensional surface from the plurality of estimates of the center of corneal curvature; and determining an estimate of the center of rotation of the user's eye using the three-dimensional surface.

[0011] Example 4: A display system configured to project light into the user's eyes for displaying virtual image content within the user's field of view, comprising a frame configured to be supported on the user's head, a head-mounted display positioned on the frame, configured to project light into the user's eyes and display virtual image content, first and second eye-tracking cameras configured to image the user's eyes, and a processing electronic device communicating with the display and the first and second eye-tracking cameras, which captures multiple pairs of images of the user's eyes. A display system comprising processing electronics configured to receive images from first and second eye tracking cameras, and, with respect to pairs of images received from the first and second eye tracking cameras, respectively, obtain an estimate of the center of corneal curvature of the user's eye based at least partially on the individual pair of captured images, and to determine a three-dimensional surface, identify the center of curvature of the 3D surface, and obtain an estimate of the rotation center of the user's eye based on the estimated center of corneal curvature of the user's eye obtained based on multiple pairs of captured images of the user's eye received from the individual first and second eye tracking cameras.

[0012] Example 5: A display system configured to project light into a user's eye for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eye and display virtual image content within the user's field of view; an eye-tracking camera configured to image the user's eye; a plurality of light emitters; and a processing electronic device communicating with the display and the eye-tracking camera, which receives images of the user's eye captured by the eye-tracking camera at first and second locations, and is configured to estimate the location of the center of the corneal curvature of the user's eye based on the location of the flash reflections in the image produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the individual flash reflections.

[0013] Example 6: A display system configured to project light into a user's eye for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eye and display virtual image content within the user's field of view; an eye-tracking camera configured to image the user's eye; a plurality of light emitters; and a processing electronic device communicating with the display and the eye-tracking camera, which receives images of the user's eye captured by the eye-tracking camera at a first and a second location, and is configured to estimate the location of the rotation center of the user's eye based on the location of the flash reflections in the image produced by the eye-tracking camera and based on the locations of the emitters that produced the flash reflections with respect to the first and second locations of the eye-tracking camera and a plurality of eye poses.

[0014] Example 7: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, wherein the eye includes capturing a plurality of images of the user's eye using an eye-tracking camera having a cornea and configured to image the user's eye, and a plurality of light emitters positioned relative to the eye and forming flashes thereon, the images comprising a plurality of flashes; and obtaining an estimate of the center of rotation of the eye based on the plurality of flashes, the step of obtaining an estimate of the center of rotation of the eye comprising determining a plurality of estimates of the center of corneal curvature of the user's eye based on the plurality of flashes; generating a three-dimensional surface from the plurality of estimates of the center of corneal curvature; and determining an estimate of the center of rotation of the user's eye using the three-dimensional surface.

[0015] Example 8: A display system configured to project light into the user's eyes for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eyes and display virtual image content; an eye-tracking camera configured to image the user's eyes; and a processing electronic device communicating with the display and the eye-tracking camera, configured to receive multiple pairs of captured images of the user's eyes from the eye-tracking camera, obtain an estimate of the center of corneal curvature of the user's eyes based at least partially on the individual pairs of captured images with respect to the pair of images received from the eye-tracking camera, determine a three-dimensional surface, identify the center of curvature of the 3D surface, and obtain an estimate of the center of rotation of the user's eyes based on the estimated center of corneal curvature of the user's eyes obtained based on multiple pairs of captured images of the user's eyes received from the eye-tracking camera.

[0016] Example 9: A display system configured to project light into a user's eye for displaying virtual image content within the user's field of view, comprising: a frame configured to be supported on the user's head; a head-mounted display positioned on the frame, configured to project light into the user's eye and display virtual image content within the user's field of view; at least one eye-tracking camera configured to image the user's eye; a plurality of light emitters; and a processing electronic device communicating with the display and the eye-tracking camera, which receives images of the user's eye captured by the at least one eye-tracking camera at first and second locations, and is configured to estimate the location of the center of the corneal curvature of the user's eye based on the location of the flash reflections in the image produced by the at least one eye-tracking camera, and based on the location of the at least one eye-tracking camera and the location of the emitter that produced the individual flash reflections, where flash reflections of different light emitters are observable in the image of the eye captured by the eye-tracking camera. Part II

[0017] Example 1: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, First and second eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and first and second eye-tracking cameras, which receives images of the user's eyes captured by the first and second eye-tracking cameras, and in which the flash reflections of different light emitters are observable in the images of the eyes captured by the first and second eye-tracking cameras, Based on the location of the specular reflection in the images produced by both the first and second eye-tracking cameras, and based on the locations of both the first and second eye-tracking cameras and the location of the emitter that produced the individual specular reflection, estimating the location of the center of the corneal curvature of the user's eye, A processing electronic device configured as such, A display system comprising the same.

[0018] Example 2: The processing electronic device Based on the location of the specular reflection in one or more images produced by the first eye-tracking camera, and based on the location of the first eye-tracking camera and the location of the emitter that produced the specular reflection, determining a first direction towards the center of the corneal curvature of the user's eye, Based on the location of the specular reflection in one or more images produced by the second eye-tracking camera, and based on the location of the second eye-tracking camera and the location of the emitter that produced the specular reflection, determining a second direction towards the center of the corneal curvature of the user's eye. The display system according to Example 1, configured as such.

[0019] Example 3: The processing electronic device Defining a first plane including the first eye-tracking camera, the location of the first specular reflection, and the location of the light emitter corresponding to the first specular reflection, Defining a second plane including the first eye-tracking camera, the location of the second specular reflection, and the location of the light emitter corresponding to the second specular reflection, Determining a region of convergence of the first plane and the second plane, which extends along the first direction, Thereby, the display system according to Example 2, configured to determine the first direction.

[0020] Example 4: The processing electronic device Defining a third plane including the second eye-tracking camera, the location of the third specular reflection, and the location of the light emitter corresponding to the third specular reflection, Define a fourth plane including the second eye-tracking camera, the location of the fourth specular reflection, and the location of the light emitter corresponding to the fourth specular reflection. Determine a region of convergence of the third and fourth planes, the region of convergence extending along a second direction. The display system according to Example 3, configured to determine the second direction thereby.

[0021] Example 5: The processing electronic device is the display system according to any of the above examples, configured to estimate the location of the center of the corneal curvature of the user's eye based on the first and second directions towards the center of the corneal curvature of the user's eye.

[0022] Example 6: The processing electronic device Determine the first direction along which the center of the corneal curvature of the user's eye is estimated to be located based on at least one first image received from the first eye-tracking camera, Determine the second direction along which the center of the corneal curvature of the user's eye is estimated to be located based on at least one second image received from the second eye-tracking camera, the first and second directions converging towards a certain region. The display system according to any of the above examples, configured as such.

[0023] Example 7: The processing electronic device is the display system according to any of the above examples, configured to obtain an estimated value of the center of the corneal curvature of the user's eye based on the convergence of the first and second directions.

[0024] Example 8: The processing electronic device is the display system according to any of the above examples, configured to estimate the location of the center of the corneal curvature of the user's eye by identifying the region of convergence of the first and second directions towards the center of the corneal curvature of the user's eye.

[0025] Example 9: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on multiple determinations of the center of corneal curvature of the user's eye for different eye positions.

[0026] Example 10: The display system according to any of the above embodiments, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the center of the corneal curvature of the user's eye for different eye positions.

[0027] Example 11: The display system according to Example 10, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on the trajectory of a point corresponding to an estimate of the center of corneal curvature of the user's eye for different eye positions.

[0028] Example 12: The display system according to Example 10 or 11, wherein the processing electronic device is configured to determine a surface and obtain an estimate of the user's eye's center of rotation based on the trajectory of the point.

[0029] Example 13: The display system according to Example 10 or 11, wherein the processing electronic device is configured to obtain an estimate of the user's eye's center of rotation by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0030] Example 14: The display system according to Example 10 or 11, wherein the processing electronic device is configured to obtain an estimate of the user's eye's rotation center by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0031] Example 15: The display system according to any one of Examples 12, 13, or 14, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0032] Example 16: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0033] Example 17: The display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0034] Example 18: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, First and second eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and first and second eye-tracking cameras, which receives images of the user's eyes captured by the first and second eye-tracking cameras, and in which the flash reflections of different light emitters are observable in the images of the eyes captured by the first and second eye-tracking cameras, Based on the location of the flash reflection in the image produced by both the first and second eye-tracking cameras, and based on the location of both the first and second eye-tracking cameras and the location of the emitter that produced the flash reflection with respect to multiple eye poses, the location of the user's eye's center of rotation is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0035] Example 19: In order to obtain an estimate of the rotation center of the eye, the processing electronic equipment is used. Based on multiple flash reflections related to multiple eye postures, multiple estimates of the center of the user's corneal curvature are determined. Based on multiple estimates of the center of corneal curvature of the user's eye for the given eye postures, an estimate of the user's eye's center of rotation is determined. The display system according to Example 18, configured as described above.

[0036] Example 20: To determine the multiple estimates of the corneal curvature of the user's eye, the processing electronic device is: Based on the individual locations of at least some of the plurality of emitters and the first camera of the eye tracking camera, a first direction toward the center of corneal curvature is determined. Based on the individual locations of at least some of the plurality of emitters and the second camera of the eye tracking camera, a second direction toward the center of corneal curvature is determined. The display system according to Embodiment 19, configured to determine an estimate of the center of the corneal curvature of the user's eye based on the first and second directions.

[0037] Example 21: The processing electronic device is, A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Example 20, configured to determine a first direction by means of the above.

[0038] Example 22: The processing electronic device is, A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 21, configured to determine a second direction by means of the above.

[0039] Example 23: A display system according to any of Examples 20-22, wherein, in order to determine the multiple estimates of the corneal curvature of the user's eye, the processing electronic equipment is configured to determine the region of convergence between a first direction and a second direction and to determine an estimate of the center of the corneal curvature of the user's eye.

[0040] Example 24: In order to obtain an estimate of the rotation center of the eye, the processing electronic equipment, Generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature. Based on a 3D surface, an estimate of the user's eye's center of rotation is determined. A display system as described in any of Examples 19-23.

[0041] Example 25: The display system according to Example 24, wherein the processing electronics are configured to fit the surface to multiple estimates of the center of corneal curvature in order to generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature.

[0042] Example 26: The display system according to Example 24, wherein the processing electronics are configured to fit a spherical surface to multiple estimates of the center of corneal curvature in order to generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature.

[0043] Example 27: To determine an estimate of the user's eye's center of rotation, the processing electronic device is used. Determine two or more normals to a 3D surface, Determine the region where two or more normals converge. A display system according to any of Examples 24-26, configured such that the convergence region includes an estimate of the user's eye rotation center.

[0044] Example 28: A display system according to any of Examples 21-27, comprising one or more images of the user's eye, each associated with a different line of sight vector of the user's eye.

[0045] Example 29: A display system according to any of Examples 21-28, wherein the processing electronic device is configured to map the cornea of ​​the user's eye using a gaze target.

[0046] Example 30: The display system according to any of Examples 18-29, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0047] Example 31: A display system according to any of Examples 18-30, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0048] Example 32: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, wherein the eye has a cornea, and the cornea has a center of curvature, A step of capturing multiple images of a user's eye using multiple eye-tracking cameras configured to image the user's eye and multiple light emitters positioned relative to the eye and forming flashes thereon, wherein the images comprise multiple flashes. A step of obtaining an estimate of the eye's center of rotation based on multiple flashes, The step of obtaining an estimate of the rotation center of the eye includes, The steps include determining multiple estimates of the center of the user's eye's corneal curvature based on multiple flashes, A step of generating a 3D surface from multiple estimates of the center of corneal curvature, The steps include: determining an estimate of the user's eye's center of rotation using a 3D surface; Methods that include...

[0049] Example 33: The step of determining multiple estimates of the corneal curvature of the user's eye is: A step of determining a first vector directed toward the center of corneal curvature based on the locations of at least some of the multiple light emitters and the location of a first camera of multiple eye tracking cameras, The steps include determining a second vector directed toward the center of corneal curvature based on the locations of at least some of the multiple light emitters and the location of a second camera of multiple eye tracking cameras, The steps include determining the region of convergence between the first vector and the second vector, and determining an estimate of the center of the corneal curvature of the user's eye, The method according to Example 32, including the method described in Example 32.

[0050] Example 34: The first direction is, A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The method according to Example 33, determined by...

[0051] Example 35: The second direction is, A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The method according to Example 33, determined by...

[0052] Example 36: The method according to any of Examples 32-35, wherein the step of generating a three-dimensional surface from multiple estimates of the center of corneal curvature includes the step of fitting the surface to multiple estimates of the center of corneal curvature.

[0053] Example 37: The method according to any of Examples 32-35, wherein the step of generating a three-dimensional surface from multiple estimates of the center of corneal curvature includes the step of fitting a sphere to multiple estimates of the center of corneal curvature.

[0054] Example 38: The step of determining the estimated rotation center of the user's eye is: The steps include determining two or more vectors in the direction of the normal to a three-dimensional surface, A step of determining a region of convergence of two or more vectors in the direction normal to a three-dimensional surface, wherein the region of convergence comprises an estimate of the rotation center of the user's eye. The method according to any of Examples 32-37, including the method described above.

[0055] Example 39: The method according to any one of Examples 32-38, comprising images of a user's eye, each image being associated with a different line of sight direction of the user's eye.

[0056] Example 40: The method according to any of Examples 32-39, further comprising the step of mapping the cornea of ​​the user's eye using a gaze target.

[0057] Example 41: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content, First and second eye-tracking cameras configured to image the user's eyes, A processing electronic device that communicates with a display and first and second eye tracking cameras, receiving captured images of multiple pairs of the user's eyes from the first and second eye tracking cameras, With respect to the paired images received from the first and second eye tracking cameras, an estimate of the center of the user's corneal curvature is obtained, at least partially, based on the captured images of the individual pairs. Based on the estimated center of the corneal curvature of the user's eye, obtained from multiple pairs of captured images of the user's eye received from separate first and second eye tracking cameras, the 3D surface is determined. Identify the center of curvature of a 3D surface and obtain an estimate of the user's eye's center of rotation. A processing electronic device configured as follows, A display system equipped with these features.

[0058] Example 42: The display system according to Example 41, wherein the processing electronic device is configured to fit a three-dimensional surface to an estimated center of the corneal curvature of the user's eye, which is acquired based on captured images of multiple pairs of the user's eye received from separate first and second eye tracking cameras.

[0059] Example 43: To obtain an estimate of the center of the corneal curvature of the user's eye, at least partially, based on individual pairs of captured images, the processing electronic device, Based on a first image received from a first eye tracking camera, a first vector is determined, along which the center of the user's eye's corneal curvature is estimated to be located. Based on the second image received from the second eye tracking camera, a second vector is determined along which the center of the user's eye's corneal curvature is estimated to be located, and the first and second images correspond to one of the paired images. Identify the convergence region between paths extending in the directions of the first and second vectors, and obtain an estimate of the center of the corneal curvature of the user's eye. The display system according to Example 41 or 42, configured as described above.

[0060] Example 44: Further comprising multiple light emitters configured to illuminate the user's eyes and form a flash reflection thereon, To determine a first vector based on the first image of the pair of captured images, the processing electronic device performs the following: A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. Identify the convergence region of the first and second planes, where the convergence region extends along the direction of the first vector. The display system according to Example 43, configured as described above.

[0061] Example 45: To determine a second vector based on the second image in each pair of captured images, the processing electronic device performs the following: A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. Determine the convergence regions of the third and fourth planes, where the convergence regions extend along the direction of the second vector. The display system according to Embodiment 44, configured as described above.

[0062] Example 46: The display system according to any of Examples 41-45, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0063] Example 47: A display system according to any one of Examples 41-46, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0064] Example 48: A display system according to any of the above embodiments, wherein at least a portion of the display is transparent, and the transparent portion is positioned in front of the user's eyes when the user wears the head-mounted display, so as to allow light from a portion of the environment in front of the user and the head-mounted display to pass through to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0065] Example 49: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, An eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, The system receives an image of the user's eye captured by an eye-tracking camera, and the flash reflections of different light emitters are observable within the image of the eye captured by the eye-tracking camera. Based on the location of flash reflections in the image produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the individual flash reflections, the location of the center of the corneal curvature of the user's eye is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0066] Example 50: The processing electronic device is Based on the location of flash reflections in one or more images produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection, a first direction toward the center of the corneal curvature of the user's eye is determined. Based on the location of flash reflections in one or more images produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection, a second direction toward the center of the corneal curvature of the user's eye is determined. The display system according to Example 49, configured as described above.

[0067] Example 51: The processing electronic device is A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Example 50, configured to determine a first direction by...

[0068] Example 52: The processing electronic device is A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 51, configured to determine a second direction by means of the above.

[0069] Example 53: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye based on the first and second directions toward the center of the corneal curvature of the user's eye.

[0070] Example 54: The processing electronic device is, Based on at least one first image received from the location of the eye-tracking camera, a first direction is determined along which the center of the corneal curvature of the user's eye is estimated to be located. Based on at least one second image received from the location of the eye-tracking camera, a second direction is determined along which the center of the corneal curvature of the user's eye is estimated to be located, and the first and second directions converge toward a certain region. A display system according to any of the above embodiments, configured as described above.

[0071] Example 55: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the center of the corneal curvature of the user's eye based on the convergence of first and second directions.

[0072] Example 56: A display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye by identifying regions of convergence in the first and second directions toward the center of the corneal curvature of the user's eye.

[0073] Example 57: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on multiple determinations of the center of corneal curvature of the user's eye for different eye positions.

[0074] Example 58: The display system according to any of the above embodiments, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the center of the corneal curvature of the user's eye for different eye positions.

[0075] Example 59: The display system according to Example 58, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on the trajectory of a point corresponding to an estimate of the center of corneal curvature of the user's eye for different eye positions.

[0076] Example 60: The display system according to Example 58 or 59, wherein the processing electronic device is configured to determine a surface and obtain an estimate of the user's eye's center of rotation based on the trajectory of the point.

[0077] Example 61: The display system according to Example 58 or 59, wherein the processing electronic device is configured to obtain an estimate of the user's eye's center of rotation by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0078] Example 62: The display system according to Example 58 or 59, wherein the processing electronic device is configured to obtain an estimate of the user's eye's rotation center by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0079] Example 63: The display system according to any one of Examples 60, 61, or 62, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0080] Example 64: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0081] Example 65: The display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0082] Example 66: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, An eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, The system receives an image of the user's eye captured by an eye-tracking camera, and the flash reflections of different light emitters are observable within the image of the eye captured by the eye-tracking camera. Based on the location of the flash reflection in the image produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection with respect to multiple eye poses, the location of the user's eye's center of rotation is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0083] Example 67: In order to obtain an estimate of the rotation center of the eye, the processing electronic equipment, Based on multiple flash reflections related to multiple eye postures, multiple estimates of the center of the user's corneal curvature are determined. Based on multiple estimates of the center of corneal curvature of the user's eye for the given eye postures, an estimate of the user's eye's center of rotation is determined. The system according to Example 66, configured as described above.

[0084] Example 68: To determine the multiple estimates of the corneal curvature of the user's eye, the processing electronic device is: Based on at least individual locations of some of the plurality of emitters and the location of the eye tracking camera, a first direction toward the center of corneal curvature is determined. Based on at least some of the locations of the plurality of emitters and at least individual locations and the location of the eye tracking camera, a second direction toward the center of the corneal curvature is determined. The system according to Example 67, configured to determine an estimate of the center of the corneal curvature of the user's eye based on the first and second directions.

[0085] Example 69: The processing electronic device is A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Embodiment 68, configured to determine a first direction by...

[0086] Example 70: The processing electronic device is, A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 69, configured to determine a second direction by means of the above.

[0087] Example 71: The system according to any of Examples 68-70, wherein, in order to determine the multiple estimates of the corneal curvature of the user's eye, the processing electronic equipment is configured to determine the region of convergence between a first direction and a second direction and to determine an estimate of the center of the corneal curvature of the user's eye.

[0088] Example 72: In order to obtain an estimate of the rotation center of the eye, the processing electronic equipment, Generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature. Based on a 3D surface, an estimate of the user's eye's center of rotation is determined. The system as described in any of Examples 19-71.

[0089] Example 73: The system according to Example 72, wherein the processing electronics are configured to fit the surface to multiple estimates of the center of corneal curvature in order to generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature.

[0090] Example 74: The system according to Example 73, wherein the processing electronics are configured to fit a sphere to multiple estimates of the center of corneal curvature in order to generate a three-dimensional surface associated with multiple estimates of the center of corneal curvature.

[0091] Example 75: To determine an estimate of the user's eye's center of rotation, the processing electronic device is: Determine two or more normals to a 3D surface, Determine the region where two or more normals converge. The system is configured such that the convergence region includes an estimate of the user's eye's rotation center, as described in any of Examples 72-74.

[0092] Example 76: The system according to any one of Examples 69-75, comprising one or more images of the user's eye, each associated with a different line of sight vector of the user's eye.

[0093] Example 77: The system according to any of Examples 69-76, wherein the processing electronic device is configured to map the cornea of ​​the user's eye using a gaze target.

[0094] Example 78: The display system according to any of Examples 66-77, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0095] Example 79: A display system according to any of Examples 66-78, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0096] Example 80: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, wherein the eye has a cornea, and the cornea has a center of curvature, A step of capturing multiple images of a user's eye using an eye-tracking camera configured to image the user's eye and a plurality of light emitters positioned relative to the eye and forming flashes thereon, wherein the images comprise the plurality of flashes. The process includes the step of obtaining an estimate of the center of rotation of the eye based on multiple flashes, the step of obtaining an estimate of the center of rotation of the eye, The steps include determining multiple estimates of the center of the user's eye's corneal curvature based on multiple flashes, A step of generating a 3D surface from multiple estimates of the center of corneal curvature, The process involves using a 3D surface to determine an estimate of the user's eye's center of rotation. Methods that include...

[0097] Example 81: The step of determining multiple estimates of the corneal curvature of the user's eye is: A step of determining a first vector directed toward the center of corneal curvature based on the locations of at least some of the multiple light emitters and the location of the eye tracking camera, The steps include determining a second vector directed toward the center of corneal curvature based on the locations of at least some of the multiple light emitters and the location of the eye tracking camera, The steps include determining the region of convergence between the first vector and the second vector, and determining an estimate of the center of the corneal curvature of the user's eye, The method according to Example 80, including the method described above.

[0098] Example 82: The first direction is, A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The method according to Example 81, as determined by...

[0099] Example 83: The second direction is, A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The method according to Example 82, as determined by...

[0100] Example 84: The method according to any of Examples 81-83, wherein the step of generating a three-dimensional surface from multiple estimates of the center of corneal curvature includes the step of fitting the surface to multiple estimates of the center of corneal curvature.

[0101] Example 85: The method according to any of Examples 81-83, wherein the step of generating a three-dimensional surface from multiple estimates of the center of corneal curvature includes the step of fitting a sphere to multiple estimates of the center of corneal curvature.

[0102] Example 86: The step of determining the estimated rotation center of the user's eye is: The steps include determining two or more vectors in the direction of the normal to a three-dimensional surface, A step of determining a region of convergence of two or more vectors in the direction normal to a three-dimensional surface, wherein the region of convergence comprises an estimate of the rotation center of the user's eye. The method according to any one of Examples 81-85, including the method described above.

[0103] Example 87: The method according to any one of Examples 81-86, comprising images of a user's eye, each image being associated with a different line of sight direction of the user's eye.

[0104] Example 88: The method according to any of Examples 81-87, further comprising the step of mapping the cornea of ​​the user's eye using a gaze target.

[0105] Example 89: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content, An eye-tracking camera configured to image the user's eyes, A processing electronic device that communicates with a display and an eye-tracking camera, Captured images of multiple pairs of the user's eyes are received from the eye-tracking camera. Each pair of images received from the eye-tracking camera is used to obtain, at least partially, an estimate of the center of the user's corneal curvature based on the captured images of the individual pairs. Based on the estimated center of the corneal curvature of the user's eye, obtained from multiple pairs of captured images of the user's eye received from an eye-tracking camera, the 3D surface is determined. Identify the center of curvature of a 3D surface and obtain an estimate of the user's eye's center of rotation. A processing electronic device configured as follows, A display system equipped with these features.

[0106] Example 90: The display system according to Example 89, wherein the processing electronic device is configured to fit a three-dimensional surface to an estimated center of the corneal curvature of the user's eye, which is obtained based on multiple pairs of captured images of the user's eye received from an eye-tracking camera.

[0107] Example 91: To obtain an estimate of the center of the corneal curvature of the user's eye, at least partially, based on individual pairs of captured images, the processing electronic device... Based on a first image received from the eye-tracking camera, a first vector is determined, along which the center of the user's eye's corneal curvature is estimated to be located. Based on a second image received from the eye-tracking camera, a second vector is determined along which the center of the user's eye's corneal curvature is estimated to be located, and the first and second images correspond to one of the paired images. Identify the convergence region between paths extending in the directions of the first and second vectors, and obtain an estimate of the center of the corneal curvature of the user's eye. The display system according to Example 89 or 90, configured as described above.

[0108] Example 92: Further comprising multiple light emitters configured to illuminate the user's eyes and form a flash reflection thereon, To determine a first vector based on the first image of the pair of captured images, the processing electronic device performs the following: A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. Identify the convergence region of the first and second planes, where the convergence region extends along the direction of the first vector. The display system according to Example 91, configured as described above.

[0109] Example 93: To determine a second vector based on the second image in each pair of captured images, the processing electronic device performs the following: A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. Determine the convergence regions of the third and fourth planes, where the convergence regions extend along the direction of the second vector. The display system according to Example 92, configured as described above.

[0110] Example 94: The display system according to any of Examples 89-93, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0111] Example 95: A display system according to any of Examples 89-94, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0112] Example 96: A display system according to any of the above embodiments, wherein at least a portion of the display is transparent, and the transparent portion is positioned in front of the user's eyes when the user wears the head-mounted display, so as to allow light from a portion of the environment in front of the user and the head-mounted display to pass through to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0113] Example 97: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, At least one eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, Images of the user's eyes are received at first and second locations by at least one eye-tracking camera, and the flash reflections of different light emitters are observable in the images of the eyes captured by the eye-tracking camera. Based on the location of flash reflections in the image produced by the at least one eye-tracking camera, and based on the location of the at least one eye-tracking camera and the location of the emitter that produced the individual flash reflection, the location of the center of the corneal curvature of the user's eye is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0114] Example 98: The processing electronic device is Based on the location of flash reflections in one or more images produced by the at least one eye-tracking camera, and based on a first location of the at least one eye-tracking camera and the location of the emitter that produced the flash reflection, a first direction toward the center of the corneal curvature of the user's eye is determined. Based on the location of flash reflections in one or more images produced by the at least one eye-tracking camera, and based on a second location of the at least one eye-tracking camera and the location of the emitter that produced the flash reflection, a second direction toward the center of the corneal curvature of the user's eye is determined. The display system according to Example 97, configured as described above.

[0115] Example 99: The processing electronic device is A first plane is defined that includes a first location of at least one eye-tracking camera, a first flash reflection location, and a location of the light emitter corresponding to the first flash reflection. A second plane is defined that includes a first location of at least one eye-tracking camera, a second location of a flash reflection, and a location of an optical emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Embodiment 98, configured to determine a first direction by...

[0116] Example 100: The processing electronic device is, A third plane is defined that includes a second location of at least one eye-tracking camera, a third location of a flash reflection, and a location of an optical emitter corresponding to the third flash reflection. A fourth plane is defined, including a second location of at least one eye-tracking camera, a fourth flash reflection location, and a location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 99, configured to determine a second direction by means of the above.

[0117] Example 101: A display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye based on the first and second directions toward the center of the corneal curvature of the user's eye.

[0118] Example 102: The processing electronic device is, Based on at least one first image received from a first location of at least one eye-tracking camera, a first direction is determined along which the center of the corneal curvature of the user's eye is estimated to be located. Based on at least one second image received from a second location of at least one eye-tracking camera, a second direction is determined along which the center of the corneal curvature of the user's eye is estimated to be located, and the first and second directions converge toward a certain region. A display system according to any of the above embodiments, configured as described above.

[0119] Example 103: A display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the center of the corneal curvature of the user's eye based on the convergence of first and second directions.

[0120] Example 104: A display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye by identifying regions of convergence in the first and second directions toward the center of the corneal curvature of the user's eye.

[0121] Example 105: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on multiple determinations of the center of corneal curvature of the user's eye for different eye positions.

[0122] Example 106: The display system according to any of the above embodiments, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the center of the corneal curvature of the user's eye for different eye positions.

[0123] Example 107: The display system according to Example 106, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on the trajectory of a point corresponding to an estimate of the center of corneal curvature of the user's eye for different eye positions.

[0124] Example 108: The display system according to Example 106 or 107, wherein the processing electronic device is configured to determine a surface and obtain an estimate of the user's eye's center of rotation based on the trajectory of the point.

[0125] Example 109: The display system according to Example 106 or 107, wherein the processing electronic device is configured to obtain an estimate of the user's eye's center of rotation by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0126] Example 110: The display system according to Example 106 or 107, wherein the processing electronic device is configured to obtain an estimate of the user's eye's rotation center by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0127] Example 111: The display system according to any one of Examples 108, 109, or 110, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0128] Example 112: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the center of rotation.

[0129] Example 113: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0130] Example 115: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view.

[0131] Example 116: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at different divergent amounts such that the displayed virtual image content appears to originate from different depths, thereby displaying the virtual image content in the user's field of view.

[0132] Example 117: A display system according to any of the above embodiments, wherein the display is configured to project divergent, light into the user's eye and collimate to display virtual image content appearing to originate from different depths within the user's field of view. Part III

[0133] Example 1: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, First and second eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and first and second eye-tracking cameras, which receives images of the user's eyes captured by the first and second eye-tracking cameras, and in which the flash reflections of different light emitters are observable in the images of the eyes captured by the first and second eye-tracking cameras, Based on the location of flash reflections in the images produced by both the first and second eye-tracking cameras, and based on the locations of both the first and second eye-tracking cameras and the emitters that produced the individual flash reflections, the parameters of the eye are estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0134] Example 2: The processing electronic device is, A first direction is determined based on the location of flash reflections in one or more images produced by the first eye-tracking camera, and based on the location of the first eye-tracking camera and the location of the emitter that produced the flash reflection. A second direction is determined based on the location of flash reflections in one or more images produced by the second eye-tracking camera, and based on the location of the second eye-tracking camera and the location of the emitter that produced the flash reflection. The display system according to Example 1, configured to estimate the parameters of the eye by doing so.

[0135] Example 3: The processing electronic device is, A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Embodiment 2, configured to determine a first direction by...

[0136] Example 4: The processing electronic device is, A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 3, configured to determine a second direction by means of the above.

[0137] Example 5: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the parameter of the user's eye based on the first and second directions.

[0138] Example 6: The processing electronic device is Based on at least one first image received from the first eye tracking camera, the first direction is determined. Based on at least one second image received from a second eye tracking camera, the second direction is determined, and the first and second directions converge toward a certain region. A display system according to any of the above embodiments, configured as described above.

[0139] Example 7: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the parameter based on the convergence of the first and second directions.

[0140] Example 8: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the parameter by identifying the convergence regions of the first and second directions.

[0141] Example 9: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye based on multiple determinations of other parameters of the user's eye with respect to different eye postures.

[0142] Example 10: The display system according to any of the above embodiments, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the user's eye parameters for different eye postures.

[0143] Example 11: The display system according to Example 10, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye based on the trajectory of a point, corresponding to estimates of other parameters of the user's eye with respect to different eye postures.

[0144] Example 12: The display system according to Example 10 or 11, wherein the processing electronic device is configured to determine a surface based on the trajectory of the point and to obtain estimates of additional parameters of the user's eye.

[0145] Example 13: The display system according to Example 10 or 11, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0146] Example 14: The display system according to Example 10 or 11, wherein the processing electronic device is configured to obtain an estimate of the user's additional parameters by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0147] Example 15: The display system according to any one of Examples 12, 13, or 14, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0148] Example 16: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, and the rendering camera has a position determined by the additional parameters.

[0149] Example 17: The display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0150] Example 18: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, First and second eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and first and second eye-tracking cameras, which receives images of the user's eyes captured by the first and second eye-tracking cameras, and in which the flash reflections of different light emitters are observable in the images of the eyes captured by the first and second eye-tracking cameras, Based on the location of the flash reflection in the image produced by both the first and second eye-tracking cameras, and based on the location of both the first and second eye-tracking cameras and the location of the emitter that produced the flash reflection with respect to multiple eye poses, the location of the first parameter of the user's eye is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0151] Example 19: To obtain an estimate of the first parameter of the eye, the processing electronic device is used. Based on multiple flash reflexes related to multiple eye postures, multiple estimates of a second parameter of the user's eye are determined. Based on multiple estimates of the user's second eye parameter relating to the multiple eye postures, an estimate of the user's first eye parameter is determined. The display system according to Example 18, configured as described above.

[0152] Example 20: To determine multiple estimates of the second parameter of the user's eye, the processing electronic device, A first direction is determined based on the individual locations of at least some of the plurality of emitters and the first camera of the eye-tracking camera. A second direction is determined based on the individual locations of at least some of the plurality of emitters and the second camera of the eye-tracking camera. Based on the first and second directions, estimates of the second parameter of the user's eye are determined. The display system according to Example 19, configured as described above.

[0153] Example 21: The processing electronic device is, A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Example 20, configured to determine a first direction by means of the above.

[0154] Example 22: The processing electronic device is, A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 21, configured to determine a second direction by means of the above.

[0155] Example 23: A display system according to any of Examples 20-22, wherein, in order to determine multiple estimates of a second parameter of the user's eye, the processing electronic equipment is configured to determine a region of convergence between a first direction and a second direction and to determine estimates of the second parameter of the user's eye.

[0156] Example 24: In order to obtain the estimate of the first parameter of the eye, the processing electronic equipment generates a three-dimensional surface associated with multiple estimates of the second parameter of the eye, Based on a 3D surface, estimates of the first parameter of the user's eye are determined. A display system as described in any of Examples 19-23.

[0157] Example 25: The display system according to Example 24, wherein the processing electronics are configured to fit the surface to multiple estimates of the second parameter, in order to generate a three-dimensional surface associated with multiple estimates of the second parameter.

[0158] Example 26: The display system according to Example 24, wherein the processing electronics are configured to fit a spherical surface to multiple estimates of the second parameter in order to generate a three-dimensional surface associated with multiple estimates of the second parameter.

[0159] Example 27: To determine an estimate of the first parameter of the user's eye, the processing electronic device, Determine two or more normals to a 3D surface, Determine the region where two or more normals converge. A display system according to any of Examples 24-26, configured such that the convergence region comprises an estimate of the first parameter of the user's eye.

[0160] Example 28: A display system according to any of Examples 21-27, comprising one or more images of the user's eye, each associated with a different line of sight vector of the user's eye.

[0161] Example 29: A display system according to any of Examples 21-28, wherein the processing electronic device is configured to use a line-of-sight target.

[0162] Example 30: The display system according to any of Examples 18-29, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the first parameter of the eye.

[0163] Example 31: A display system according to any of Examples 18-30, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0164] Example 32: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, wherein the eye has a cornea, and the cornea has a center of curvature, A step of capturing multiple images of a user's eye using multiple eye-tracking cameras configured to image the user's eye and multiple light emitters positioned relative to the eye and forming flashes thereon, wherein the images comprise multiple flashes. A step of obtaining a first parameter of the eye based on multiple flashes, The step of obtaining a first parameter of the eye includes, The steps include determining multiple estimates of a second parameter of the user's eye based on multiple flashes, The steps include generating a 3D surface from multiple estimates of a second parameter, A method comprising the step of determining an estimate of a first parameter of the user's eye using a three-dimensional surface.

[0165] Example 33: The step of determining multiple estimates of the second parameter of the user's eye is as follows: A step of determining a first vector to be directed, based on the locations of at least some of the optical emitters and the location of a first camera of the multiple eye-tracking cameras, A step of determining a directed second vector based on the locations of at least some of the multiple light emitters and the location of a second camera of multiple eye-tracking cameras, The steps include determining the region of convergence between the first vector and the second vector, and determining an estimate of the second parameter of the user's eye, The method according to Example 32, including the method described in Example 32.

[0166] Example 34: The first direction is, A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The method according to Example 33, determined by...

[0167] Example 35: The second direction is, A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The method according to Example 33, determined by...

[0168] Example 36: The method according to any of Examples 32-35, wherein the step of generating a three-dimensional surface from multiple estimates of a second parameter includes the step of fitting the surface to multiple estimates of the second parameter.

[0169] Example 37: The method according to any of Examples 32-35, wherein the step of generating a three-dimensional surface from multiple estimates of a second parameter includes the step of fitting a sphere to multiple estimates of a second parameter.

[0170] Example 38: The step of determining an estimate of the first parameter of the user's eye is: The steps include determining two or more vectors in the direction of the normal to a three-dimensional surface, A step of determining a region of convergence of two or more vectors in the direction normal to a three-dimensional surface, wherein the region of convergence comprises an estimate of a first parameter of the user's eye. The method according to any of Examples 32-37, including the method described above.

[0171] Example 39: The method according to any one of Examples 32-38, comprising images of a user's eye, each image being associated with a different line of sight direction of the user's eye.

[0172] Example 40: The method according to any of Examples 32-39, further comprising the step of using a gaze target.

[0173] Example 41: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content, First and second eye-tracking cameras configured to image the user's eyes, A processing electronic device that communicates with a display and first and second eye tracking cameras, receiving captured images of multiple pairs of the user's eyes from the first and second eye tracking cameras, With respect to the paired images received from the first and second eye tracking cameras, an estimate of the second parameter of the user's eye is obtained, at least partially, based on the captured images of the individual pairs. Based on estimated second parameters of the user's eye, which are obtained based on captured images of multiple pairs of the user's eyes received from separate first and second eye tracking cameras, the 3D surface is determined. Identify the center of curvature of a 3D surface and obtain an estimate of the first parameter of the user's eye. A processing electronic device configured as follows, A display system equipped with these features.

[0174] Example 42: The display system according to Example 41, wherein the processing electronic device is configured to fit a three-dimensional surface to an estimated second parameter of the user's eye, which is acquired based on captured images of multiple pairs of the user's eye received from separate first and second eye-tracking cameras.

[0175] Example 43: To obtain an estimate of the second parameter of the user's eye, at least partially, based on individual pairs of captured images, the processing electronic device, Based on the first image received from the first eye tracking camera, a first vector is determined. Based on the second image received from the second eye tracking camera, a second vector is determined, and the first and second images correspond to one of the paired images. Identify the convergence region between paths extending in the directions of the first and second vectors, and obtain an estimate of the second parameter of the user's eye. The display system according to Example 41 or 42, configured as described above.

[0176] Example 44: Further comprising multiple light emitters configured to illuminate the user's eyes and form a flash reflection thereon, To determine a first vector based on the first image of the pair of captured images, the processing electronic device performs the following: A first plane is defined, including a first eye tracking camera, the location of a first flash reflection, and the location of a light emitter corresponding to the first flash reflection. A second plane is defined, including the location of a first eye tracking camera, the location of a second flash reflection, and the location of a light emitter corresponding to the second flash reflection. Identify the convergence region of the first and second planes, where the convergence region extends along the direction of the first vector. The display system according to Example 43, configured as described above.

[0177] Example 45: To determine a second vector based on the second image in each pair of captured images, the processing electronic device performs the following: A third plane is defined, including a second eye tracking camera, the location of a third flash reflection, and the location of a light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the second eye tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. Determine the convergence regions of the third and fourth planes, where the convergence regions extend along the direction of the second vector. The display system according to Embodiment 44, configured as described above.

[0178] Example 46: The display system according to any of Examples 41-45, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the first parameter.

[0179] Example 47: A display system according to any one of Examples 41-46, wherein the display is configured to project light into the user's eye at different divergent amounts so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view, or the display is configured to project light into the user's eye, which diverges, and is collimated to display the virtual image content appearing to originate from different depths in the user's field of view.

[0180] Example 48: A display system according to any of the above embodiments, wherein at least a portion of the display is transparent, and the transparent portion is positioned in front of the user's eyes when the user wears the head-mounted display, so as to allow light from a portion of the environment in front of the user and the head-mounted display to pass through to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0181] Example 49: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, An eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, The system receives an image of the user's eye captured by an eye-tracking camera, and the flash reflections of different light emitters are observable within the image of the eye captured by the eye-tracking camera. Based on the location of the flash reflections in the image produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the individual flash reflections, the location of the user's eye parameters is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0182] Example 50: The processing electronic device is A first direction is determined based on the location of flash reflections in one or more images produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection. A second direction is determined based on the location of flash reflections in one or more images produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection. The display system according to Example 49, configured as described above.

[0183] Example 51: The processing electronic device is A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Example 50, configured to determine a first direction by...

[0184] Example 52: The processing electronic device is A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 51, configured to determine a second direction by means of the above.

[0185] Example 53: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the parameter of the user's eye based on the first and second directions.

[0186] Example 54: The processing electronic device is, Determine the first direction based on at least one first image received from the location of the eye-tracking camera, Determine the second direction based on at least one second image received from the location of the eye-tracking camera, and the first and second directions converge towards a certain area, A display system according to any of the above embodiments, configured as such.

[0187] Example 55: The processing electronic device is configured to obtain an estimated value of the parameter of the user's eye based on the convergence of the first and second directions, a display system according to any of the above embodiments.

[0188] Example 56: The processing electronic device is configured to estimate the parameter of the user's eye by identifying the area of convergence of the first and second directions, a display system according to any of the above embodiments.

[0189] Example 57: The processing electronic device is configured to obtain an estimated value of an additional parameter based on a plurality of determinations of the parameters of the user's eye regarding different eye poses, a display system according to any of the above embodiments. <000091I>

[0190] Example 58: The processing electronic device is configured to determine a locus of points corresponding to the estimated values of the parameters of the user's eye regarding different eye poses, a display system according to any of the above embodiments.

[0191] Example 59: The processing electronic device is configured to obtain an estimated value of an additional parameter of the user's eye based on the locus of points corresponding to the estimated values of other parameters of the user's eye regarding different eye poses, a display system according to Example 58.

[0192] Example 60: The processing electronic device is configured to determine a surface based on the locus of points and obtain an estimated value of an additional parameter of the user's eye, a display system according to Example 58 or 59.

[0193] Example 61: The display system according to Example 58 or 59, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0194] Example 62: The display system according to Example 58 or 59, wherein the processing electronic device is configured to obtain an estimate of additional parameters of the user's eye by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0195] Example 63: The display system according to any one of Examples 60, 61, or 62, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0196] Example 64: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the additional parameters.

[0197] Example 65: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at different divergent amounts so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view, or the display is configured to project light into the user's eye, which diverges, and is collimated to display the virtual image content appearing to originate from different depths in the user's field of view.

[0198] Example 66: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, An eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, The system receives an image of the user's eye captured by an eye-tracking camera, and the flash reflections of different light emitters are observable within the image of the eye captured by the eye-tracking camera. Based on the location of the flash reflection in the image produced by the eye-tracking camera, and based on the location of the eye-tracking camera and the location of the emitter that produced the flash reflection with respect to multiple eye poses, the location of a first parameter of the user's eye is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0199] Example 67: To obtain an estimate of the first parameter of the eye, the processing electronic device is used. Based on multiple flash reflexes related to multiple eye postures, multiple estimates of a second parameter of the user's eye are determined. Based on multiple estimates of the user's second eye parameter relating to the multiple eye postures, an estimate of the user's first eye parameter is determined. The system according to Example 66, configured as described above.

[0200] Example 68: To determine multiple estimates of the second parameter of the user's eye, the processing electronic device, Based on the individual locations of at least some of the plurality of emitters and the location of the eye tracking camera, a first direction is determined. Based on at least some of the locations of the plurality of emitters and at least individual locations and the location of the eye tracking camera, a second direction toward the center of the corneal curvature is determined. The system according to Example 67, configured to determine an estimated value of the center of the corneal curvature of the user's eye based on the first and second directions.

[0201] Example 69: The processing electronic device defines a first plane including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. defines a second plane including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. determines a region of convergence of the first plane and the second plane, which extends along the first direction. Thereby, the display system according to Example 68, configured to determine the first direction.

[0202] Example 70: The processing electronic device defines a third plane including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. defines a fourth plane including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. determines a region of convergence of the third plane and the fourth plane, which extends along the second direction. Thereby, the display system according to Example 69, configured to determine the second direction.

[0203] Example 71: To determine a plurality of estimated values of a second parameter of the user's eye, the processing electronic device determines a region of convergence between the first direction and the second direction and is configured to determine an estimated value of the second parameter of the user's eye, the system according to any of Examples 68 - 70.

[0204] Example 72: To obtain an estimated value of the first parameter of the eye, the processing electronic device generates a three-dimensional surface associated with a plurality of estimated values of the second parameter. Based on a 3D surface, estimates of the first parameter of the user's eye are determined. The system as described in any of Examples 19-71.

[0205] Example 73: The system according to Example 72, wherein the processing electronics are configured to fit the surface to multiple estimates of the first parameter in order to generate a three-dimensional surface associated with multiple estimates of the second parameter.

[0206] Example 74: The system according to Example 73, wherein the processing electronics are configured to fit a sphere to multiple estimates of the second parameter in order to generate a three-dimensional surface associated with multiple estimates of the second parameter.

[0207] Example 75: To determine an estimate of the first parameter of the user's eye, the processing electronic device, Determine two or more normals to a 3D surface, Determine the region where two or more normals converge. The system is configured such that the convergence region comprises an estimate of the first parameter of the user's eye, as described in any of Examples 72-74.

[0208] Example 76: The system according to any one of Examples 69-75, comprising one or more images of the user's eye, each associated with a different line of sight vector of the user's eye.

[0209] Example 77: The processing electronic device is configured to use a line-of-sight target, as described in any of Examples 69-76.

[0210] Example 78: The display system according to any of Examples 66-77, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the first parameter.

[0211] Example 79: A display system according to any of Examples 66-78, wherein the display is configured to project light into the user's eye at different divergent amounts so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view, or the display is configured to project light into the user's eye, which diverges, and is collimated to display the virtual image content appearing to originate from different depths in the user's field of view.

[0212] Example 80: A method for determining one or more parameters associated with an eye in a display system configured to project light onto a user's eye for displaying virtual image content within the user's field of view, wherein the eye has a cornea, and the cornea has a center of curvature, A step of capturing multiple images of a user's eye using an eye-tracking camera configured to image the user's eye and a plurality of light emitters positioned relative to the eye and forming flashes thereon, wherein the images comprise the plurality of flashes. A step of obtaining an estimate of a first parameter of the eye based on multiple flashes of light. The step of obtaining an estimate of the first parameter of the eye includes, The steps include determining multiple estimates of a second parameter of the user's eye based on multiple flashes, The steps include generating a 3D surface from multiple estimates of a second parameter, A method comprising the step of determining an estimate of a first parameter of the user's eye using a three-dimensional surface.

[0213] Example 81: The step of determining multiple estimates of the second parameter of the user's eye is as follows: The steps include determining a first vector based on the locations of at least some of the multiple light emitters and the location of the eye tracking camera, The steps include determining a second vector based on the locations of at least some of the multiple light emitters and the location of the eye tracking camera, The steps include determining the region of convergence between the first vector and the second vector, and determining an estimate of the second parameter of the user's eye, The method according to Example 80, including the method described above.

[0214] Example 82: The first direction is, A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The method according to Example 81, as determined by...

[0215] Example 83: The second direction is, A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The method according to Example 82, as determined by...

[0216] Example 84: The method according to any of Examples 81-83, wherein the step of generating a three-dimensional surface from multiple estimates of a second parameter includes the step of fitting the surface to multiple estimates of a first parameter.

[0217] Example 85: The method according to any of Examples 81-83, wherein the step of generating a three-dimensional surface from multiple estimates of a second parameter includes the step of fitting a sphere to multiple estimates of a second parameter.

[0218] Example 86: The step of determining an estimate of the first parameter of the user's eye is: The steps include determining two or more vectors in the direction of the normal to a three-dimensional surface, A step of determining a region of convergence of two or more vectors in the direction normal to a three-dimensional surface, wherein the region of convergence comprises an estimate of a first parameter of the user's eye. The method according to any one of Examples 81-85, including the method described above.

[0219] Example 87: The method according to any one of Examples 81-86, comprising images of a user's eye, each image being associated with a different line of sight direction of the user's eye.

[0220] Example 88: The method of any of Examples 81-87, further using a gaze target.

[0221] Example 89: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content, An eye-tracking camera configured to image the user's eyes, A processing electronic device that communicates with a display and an eye-tracking camera, Captured images of multiple pairs of the user's eyes are received from the eye-tracking camera. Each pair of images received from the eye-tracking camera is used to obtain, at least partially, estimates of the user's eye parameters based on the captured images of the individual pairs. Based on estimated parameters of the user's eye, obtained from captured images of multiple pairs of the user's eyes received from an eye-tracking camera, the 3D surface is determined. Identify the center of curvature of a 3D surface and obtain estimates of additional parameters of the user's eye. A processing electronic device configured as follows, A display system equipped with these features.

[0222] Example 90: The display system according to Example 89, wherein the processing electronic device is configured to fit a three-dimensional surface to estimated parameters of the user's eye, which are acquired based on captured images of multiple pairs of the user's eye received from an eye-tracking camera.

[0223] Example 91: To obtain estimates of the user's eye parameters, at least partially, based on individual pairs of captured images, the processing electronic device, Based on the first image received from the eye-tracking camera, a first vector is determined. Based on the second image received from the eye-tracking camera, a second vector is determined, and the first and second images correspond to one of the paired images. Identify the convergence region between paths extending in the directions of the first and second vectors, and obtain estimates of the user's eye parameters. The display system according to Example 89 or 90, configured as described above.

[0224] Example 92: Further comprising multiple light emitters configured to illuminate the user's eyes and form a flash reflection thereon, To determine a first vector based on the first image of the pair of captured images, the processing electronic device performs the following: A first plane is defined, including the location of the eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection. A second plane is defined, including the location of the eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection. Identify the convergence region of the first and second planes, where the convergence region extends along the direction of the first vector. The display system according to Example 91, configured as described above.

[0225] Example 93: To determine a second vector based on the second image in each pair of captured images, the processing electronic device performs the following: A third plane is defined, including the location of the eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection. A fourth plane is defined, including the location of the eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection. Determine the convergence regions of the third and fourth planes, where the convergence regions extend along the direction of the second vector. The display system according to Example 92, configured as described above.

[0226] Example 94: The display system according to any of Examples 89-93, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the additional parameters.

[0227] Example 95: A display system according to any of Examples 89-94, wherein the display is configured to project light into the user's eye at different divergent amounts so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view, or the display is configured to project light into the user's eye, which diverges, and is collimated to display the virtual image content appearing to originate from different depths in the user's field of view.

[0228] Example 96: A display system according to any of the above embodiments, wherein at least a portion of the display is transparent, and the transparent portion is positioned in front of the user's eyes when the user wears the head-mounted display, so as to allow light from a portion of the environment in front of the user and the head-mounted display to pass through to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0229] Example 97: A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, A frame configured to be supported above the user's head, A head-mounted display positioned on a frame, configured to project light into the user's eyes and display virtual image content in the user's field of view, At least one eye-tracking camera configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with a display and an eye-tracking camera, Images of the user's eyes are received at first and second locations by at least one eye-tracking camera, and the flash reflections of different light emitters are observable in the images of the eyes captured by the eye-tracking camera. Based on the location of flash reflections in the image produced by the at least one eye-tracking camera, and based on the location of the at least one eye-tracking camera and the location of the emitter that produced the individual flash reflection, the location of the user's eye parameters is estimated. A processing electronic device configured as follows, A display system equipped with these features.

[0230] Example 98: The processing electronic device is A first direction is determined based on the location of a flash reflection in one or more images produced by the at least one eye-tracking camera, and based on a first location of the at least one eye-tracking camera and the location of the emitter that produced the flash reflection. A second direction is determined based on the location of a flash reflection in one or more images produced by the at least one eye-tracking camera, and based on a second location of the at least one eye-tracking camera and the location of the emitter that produced the flash reflection. The display system according to Example 97, configured as described above.

[0231] Example 99: The processing electronic device is A first plane is defined that includes a first location of at least one eye-tracking camera, a first flash reflection location, and a location of the light emitter corresponding to the first flash reflection. A second plane is defined that includes a first location of at least one eye-tracking camera, a second location of a flash reflection, and a location of an optical emitter corresponding to the second flash reflection. A region of convergence of the first plane and the second plane, which extends along the first direction, and which determines the region of convergence. The display system according to Embodiment 98, configured to determine a first direction by...

[0232] Example 100: The processing electronic device is, A third plane is defined that includes a second location of at least one eye-tracking camera, a third location of a flash reflection, and a location of an optical emitter corresponding to the third flash reflection. A fourth plane is defined, including a second location of at least one eye-tracking camera, a fourth flash reflection location, and a location of the light emitter corresponding to the fourth flash reflection. A region of convergence of the third and fourth planes, which extends along the second direction, determining the region of convergence. The display system according to Embodiment 99, configured to determine a second direction by means of the above.

[0233] Example 101: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the location of the parameter of the user's eye based on the first and second directions.

[0234] Example 102: The processing electronic device is, Based on at least one first image received from a first location of at least one eye-tracking camera, the first orientation is determined. A second direction is determined based on at least one second image received from a second location of at least one eye-tracking camera, and the first and second directions converge toward a certain region. A display system according to any of the above embodiments, configured as described above.

[0235] Example 103: A display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain an estimate of the parameter based on the convergence of the first and second directions.

[0236] Example 104: The display system according to any of the above embodiments, wherein the processing electronic device is configured to estimate the parameters of the user's eye by identifying the regions of convergence in the first and second directions.

[0237] Example 105: The display system according to any of the above embodiments, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye based on multiple determinations of other parameters of the user's eye with respect to different eye postures.

[0238] Example 106: The display system according to any of the above embodiments, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the user's eye parameters for different eye postures.

[0239] Example 107: The display system according to Example 106, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye based on the trajectory of a point, corresponding to estimates of other parameters of the user's eye for different eye postures.

[0240] Example 108: The display system according to Example 106 or 107, wherein the processing electronic device is configured to determine a surface based on the trajectory of the point and to obtain an estimate of the parameter of the user's eye.

[0241] Example 109: The display system according to Example 106 or 107, wherein the processing electronic device is configured to obtain estimates of additional parameters of the user's eye by determining a surface and estimating the center of curvature of the surface based on the trajectory of the point.

[0242] Example 110: The display system according to Example 106 or 107, wherein the processing electronic device is configured to obtain an estimate of the user's eye parameter by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge.

[0243] Example 111: The display system according to any one of Examples 108, 109, or 110, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface.

[0244] Example 112: The display system according to any of the above embodiments, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eye using a rendering camera, the rendering camera having a position determined by the additional parameters.

[0245] Example 113: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at at least one different amount of divergence and collimation, and to display virtual image content in the user's field of view, so that the displayed virtual image content appears to originate from different depths.

[0246] Example 115: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye so that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view.

[0247] Example 116: A display system according to any of the above embodiments, wherein the display is configured to project light into the user's eye at different divergent amounts such that the displayed virtual image content appears to originate from different depths, thereby displaying the virtual image content in the user's field of view.

[0248] Example 117: A display system according to any of the above embodiments, wherein the display is configured to project divergent, light into the user's eye and collimate to display virtual image content appearing to originate from different depths within the user's field of view.

[0249] Example 118: A display system according to any of the above embodiments, wherein at least a portion of the display is transparent, and the transparent portion is positioned in front of the user's eyes when the user wears the head-mounted display, so as to allow light from a portion of the environment in front of the user and the head-mounted display to pass through to the user's eyes and provide a view of the portion of the environment in front of the user and the head-mounted display.

[0250] Example 119: The display system according to any of the above embodiments, wherein the first parameter includes the center of rotation of the eye.

[0251] Example 120: The display system according to any of the above examples, wherein the second parameter includes the center of corneal curvature.

[0252] Example 121: The display system according to any of the above examples, wherein the parameter includes the center of corneal curvature.

[0253] Example 122: The display system according to any of the above examples, wherein the additional parameter includes the center of rotation of the eye.

[0254] Any of the above embodiments can be combined. In addition, any of the above embodiments can be integrated with a head-mounted display. Furthermore, any of the above embodiments can be implemented using one or more depth surfaces, such as a single depth surface and / or one or more variable depth surfaces (e.g., one or more elements with variable focusing force that provide distance adjustment cues that change over time).

[0255] Furthermore, apparatus and methods for determining various values ​​and parameters such as anatomical, optical, and geometric features, location, and orientation are disclosed herein, though not limited to these. Examples of such parameters include, but are not limited to, the center of rotation of the eye, the center of corneal curvature, the center of the pupil, the boundary of the pupil, the center of the iris, the boundary of the iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, and the center of gaze. In addition, in some implementations, the center of corneal curvature or the center of the cornea refers to the center of curvature of a portion of the cornea or the center of curvature of a spherical surface that coincides with a portion of the surface of the cornea. For example, in some implementations, the center of corneal curvature or the center of the cornea refers to the center of curvature of the corneal apex or the center of curvature of a spherical surface that coincides with a portion of the surface of the corneal apex. Furthermore, the determination of such values, parameters, etc., as enumerated herein, includes estimates and does not necessarily have to precisely match actual values. For example, the determination of the center of rotation of the eye, the center of corneal curvature, the center or boundary of the pupil or iris, the boundary of the limbus, the optical axis of the eye, the visual axis of the eye, the center of viewpoint, etc., may be an estimate, approximation, or near-estimate, and may not be identical to the actual (e.g., anatomical, optical, or geometric) values ​​or parameters. In some cases, for example, a root mean square estimation technique may be used to obtain an estimate of such values. As an example, a technique described herein relates to identifying a place or point where rays or vectors intersect. However, such rays or vectors do not necessarily intersect. In this embodiment, the place or point may be estimated. For example, the place or point may be determined based on the root mean square or other estimation techniques (for example, the place or point may be estimated to be close to or nearest to a ray or vector). Other processes may also be used to estimate, approximate, or otherwise provide values ​​that may not match the actual values. Therefore, the terms “determine” and “estimate” or “determined” and “estimated” are used synonymously herein. A reference to such determined value may therefore include an estimate, an approximation, or a value close to the actual value.Therefore, references to determining parameters or values ​​in any of the locations described above or herein should not be strictly limited to actual values, but may include estimates, approximations, or values ​​close to them.

[0256] Details of one or more implementations of the subject matter described herein are shown in the accompanying drawings and the description below. Other features, aspects, and advantages will be evident from the description, drawings, and claims. Neither this summary nor the following detailed description claims to define or limit the scope of the subject matter of the invention. The present invention provides, for example, the following items: (Item 1) A display system configured to project light onto the user's eyes for displaying virtual image content within the user's field of view, wherein the display system is A frame configured to be supported on the user's head, A head-mounted display positioned on the frame, wherein the display is configured to project light into the user's eyes and display virtual image content in the user's field of view, First and second eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, A processing electronic device that communicates with the display and the first and second eye-tracking cameras, wherein the processing electronic device is Receiving images of the user's eyes captured by the first and second eye-tracking cameras, wherein flash reflections of different light emitters are observable within the images of the eyes captured by the first and second eye-tracking cameras, Estimating the location of the center of the corneal curvature of the user's eye based on the location of the flash reflection in the image produced by both the first and second eye-tracking cameras, and based on the locations of both the first and second eye-tracking cameras and the emitters that produced the individual flash reflections. Processing electronic equipment configured to perform A display system equipped with these features. (Item 2) The aforementioned processing electronic device is Based on the location of the flash reflection in one or more images produced by the first eye-tracking camera, and based on the location of the first eye-tracking camera and the location of the emitter that produced the flash reflection, a first direction toward the center of the corneal curvature of the user's eye is determined. Based on the location of the flash reflection in one or more images produced by the second eye-tracking camera, and based on the location of the second eye-tracking camera and the location of the emitter that produced the flash reflection, a second direction toward the center of the corneal curvature of the user's eye is determined. A display system as described in item 1, configured to perform the following actions. (Item 3) The aforementioned processing electronic device is Defining a first plane including the first eye-tracking camera, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection, Defining a second plane including the first eye-tracking camera, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection, Determining the convergence region of the first plane and the second plane, wherein the convergence region extends along the first direction. The display system according to item 2, configured to determine the first direction by performing the following: (Item 4) The aforementioned processing electronic device is Defining a third plane including the second eye-tracking camera, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection, Defining a fourth plane including the second eye-tracking camera, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection, Determining the convergence region of the third plane and the fourth plane, wherein the convergence region extends along the second direction. The display system according to item 3, configured to determine the second direction by performing the following. (Item 5) The display system according to any of the above items, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye based on the first and second directions toward the center of the corneal curvature of the user's eye. (Item 6) The aforementioned processing electronic device is Based on at least one first image received from the first eye tracking camera, the first direction along which the center of the corneal curvature of the user's eye is estimated to be located is determined; Based on at least one second image received from the second eye tracking camera, a second direction is determined along which the center of the corneal curvature of the user's eye is estimated to be located, wherein the first and second directions converge toward a certain region. A display system as described in any of the above items, configured to perform the following: (Item 7) The display system according to any of the above items, wherein the processing electronic device is configured to obtain an estimate of the center of the corneal curvature of the user's eye based on the convergence of the first and second directions. (Item 8) The display system according to any of the above items, wherein the processing electronic device is configured to estimate the location of the center of the corneal curvature of the user's eye by identifying regions of convergence in the first and second directions toward the center of the corneal curvature of the user's eye. (Item 9) The display system according to any of the above items, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on a plurality of determinations of the center of corneal curvature of the user's eye for different eye positions. (Item 10) The display system according to any of the above items, wherein the processing electronic device is configured to determine the trajectory of a point corresponding to an estimate of the center of the corneal curvature of the user's eye for different eye positions. (Item 11) The display system according to item 10, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye based on the trajectory of the points corresponding to an estimate of the center of corneal curvature of the user's eye for different eye positions. (Item 12) The display system according to item 10 or 11, wherein the processing electronic device is configured to determine a surface based on the trajectory of the point and to obtain an estimate of the rotation center of the user's eye. (Item 13) The display system according to item 10 or 11, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye by determining a surface based on the trajectory of the point and estimating the center of curvature of the surface. (Item 14) The display system according to item 10 or 11, wherein the processing electronic device is configured to obtain an estimate of the rotation center of the user's eye by determining a surface based on the trajectory of the point and determining a region where multiple normals to the surface converge. (Item 15) The display system according to any one of items 12, 13, or 14, wherein the processing electronic device is configured to fit the surface to the trajectory of the point and to acquire the surface. (Item 16) The display system according to any of the above items, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eyes using a rendering camera, and the rendering camera has a position determined by the rotation center. (Item 17) The display system according to any of the above items, wherein the display is configured to project light into the user's eye and display the virtual image content in the user's field of view such that the displayed virtual image content appears to originate from different depths. (Item 18) The display system according to any of the above items, wherein the display is configured to project light into the user's eye at at least one different divergence such that the displayed virtual image content appears to originate from different depths, and to display the virtual image content in the user's field of view. (Item 19) The display system according to any of the above items, wherein the display is configured to project divergent light into the user's eye and collimated light into the user's eye to display virtual image content appearing to originate from different depths in the user's field of view. [Brief explanation of the drawing]

[0257] [Figure 1]Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object visible to a person.

[0258] [Figure 2] Figure 2 schematically illustrates an example of a wearable system.

[0259] [Figure 3] Figure 3 schematically illustrates exemplary components of a wearable system.

[0260] [Figure 4] Figure 4 schematically illustrates an example of a waveguide stack for a wearable device that outputs image information to the user.

[0261] [Figure 5] Figure 5 schematically illustrates an example of an eye.

[0262] [Figure 5A] Figure 5A schematically illustrates an exemplary coordinate system for determining the eye's orientation.

[0263] [Figure 6] Figure 6 is a schematic diagram of a wearable system, including an eye-tracking system.

[0264] [Figure 7A] Figure 7A is a block diagram of a wearable system, which may include an eye-tracking system.

[0265] [Figure 7B] Figure 7B is a block diagram of the rendering controller within the wearable system.

[0266] [Figure 8] Figure 8 illustrates an embodiment of an eye, including the optical axis, visual axis, and center of rotation of the eye.

[0267] [Figure 9A] Figures 9A-E illustrate exemplary configurations of a wearable system for capturing eye image data for use with an eye tracking module. [Figure 9B] Figures 9A-E illustrate exemplary configurations of a wearable system for capturing eye image data for use with an eye tracking module. [Figure 9C] Figures 9A-E illustrate exemplary configurations of a wearable system for capturing eye image data for use with an eye tracking module. [Figure 9D] Figures 9A-E illustrate exemplary configurations of a wearable system for capturing eye image data for use with an eye tracking module. [Figure 9E] Figures 9A-E illustrate exemplary configurations of a wearable system for capturing eye image data for use with an eye tracking module.

[0268] [Figure 10] Figure 10 shows a graphic diagram of the exemplary center of rotation (CoR) determination process that can be performed by the eye tracking module.

[0269] [Figure 11] Figure 11 shows an exemplary image of a flash on an arbitrary eye, used by the eye-tracking module to determine the estimated center of rotation.

[0270] [Figure 12A] Figures 12A-D illustrate the steps in an exemplary determination of a first plane, including the location of the first flash, a camera that captures an image of the flash, and the location of the illumination source that produces the first flash. [Figure 12B] Figures 12A-D illustrate the steps in an exemplary determination of a first plane, including the location of the first flash, a camera that captures an image of the flash, and the location of the illumination source that produces the first flash. [Figure 12C]Figures 12A-D illustrate the steps in an exemplary determination of a first plane, including the location of the first flash, a camera that captures an image of the flash, and the location of the illumination source that produces the first flash. [Figure 12D] Figures 12A-D illustrate the steps in an exemplary determination of a first plane, including the location of the first flash, a camera that captures an image of the flash, and the location of the illumination source that produces the first flash.

[0271] [Figure 13A] Figures 13A-D illustrate the steps in an exemplary determination of the second plane, including the location of the second flash, the camera, and the illuminator that produces the second flash. [Figure 13B] Figures 13A-D illustrate the steps in an exemplary determination of the second plane, including the location of the second flash, the camera, and the illuminator that produces the second flash. [Figure 13C] Figures 13A-D illustrate the steps in an exemplary determination of the second plane, including the location of the second flash, the camera, and the illuminator that produces the second flash. [Figure 13D] Figures 13A-D illustrate the steps in an exemplary determination of the second plane, including the location of the second flash, the camera, and the illuminator that produces the second flash.

[0272] [Figure 14A] Figures 14A-C illustrate the intersection between the first plane in Figures 12A-D and the second plane in Figures 13A-D. This intersection corresponds to a vector along which the corneal center may be located. [Figure 14B] Figures 14A-C illustrate the intersection between the first plane in Figures 12A-D and the second plane in Figures 13A-D. This intersection corresponds to a vector along which the corneal center may be located. [Figure 14C] Figures 14A-C illustrate the intersection between the first plane in Figures 12A-D and the second plane in Figures 13A-D. This intersection corresponds to a vector along which the corneal center may be located.

[0273] [Figure 15A] Figures 15A-15B illustrate multiple vectors acquired using multiple cameras, along which the corneal center may be located. These vectors may converge or intersect at locations corresponding to or near the corneal center. [Figure 15B] Figures 15A-15B illustrate multiple vectors acquired using multiple cameras, along which the corneal center may be located. These vectors may converge or intersect at locations corresponding to or near the corneal center.

[0274] [Figure 16A] Figures 16A–16C illustrate exemplary steps in the exemplary determination of a vector along which the corneal center may be located, using a shared illumination source between multiple cameras. [Figure 16B] Figures 16A–16C illustrate exemplary steps in the exemplary determination of a vector along which the corneal center may be located, using a shared illumination source between multiple cameras. [Figure 16C] Figures 16A–16C illustrate exemplary steps in the exemplary determination of a vector along which the corneal center may be located, using a shared illumination source between multiple cameras.

[0275] [Figure 17A] Figures 17A-17B show the estimated 3D surface based on the calculated corneal center. [Figure 17B] Figures 17A-17B show the estimated 3D surface based on the calculated corneal center.

[0276] [Figure 18A] Figures 18A and 18B show illustrative estimations of the rotation center in the convergence of multiple surface normal vectors normal to the 3D surface, calculated based on the corneal center. [Figure 18B] Figures 18A and 18B show illustrative estimations of the rotation center in the convergence of multiple surface normal vectors normal to the 3D surface, calculated based on the corneal center.

[0277] [Figure 19-1] Figures 19A-1 and 19A-2 illustrate exemplary surface fit to the selected estimated corneal center.

[0278] [Figure 19-2] Figures 19B-1 and 19B-2 show exemplary surface normal vectors that may be normal to the surface fitted to the estimated corneal center.

[0279] [Figure 19-3] Figures 19C-1 and 19C-2 illustrate the CoR region estimated based on the intersection points of the surface normal vectors.

[0280] [Figure 19-4] Figures 19D-1 and 19D-2 illustrate exemplary fitting surfaces to alternative selections of the corneal center.

[0281] [Figure 20] Figure 20 illustrates an exemplary rotation center extraction process that can be implemented by an eye tracking module.

[0282] [Figure 21] Figure 21 illustrates an exemplary eye-tracking process that may use the process in Figure 20 to determine the estimated center of rotation using the center of corneal curvature. [Modes for carrying out the invention]

[0283] Throughout the drawings, reference numbers may be reused to indicate correspondences between the referenced elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure. (Detailed explanation)

[0284] Here, we refer to the drawings, where similar reference numbers refer to the same parts throughout. Unless otherwise indicated, the drawings are schematic and not necessarily drawn to exact scale. A. An example of a 3D display for a wearable system

[0285] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a user with 2D or 3D virtual images. The images may be still images, video frames, or videos in combination or equivalent. At least a portion of the wearable system can be implemented on a wearable device capable of presenting a VR, AR, or MR environment, either alone or in combination, for user interaction. The term wearable device may be used synonymously with AR device (ARD). Furthermore, for the purposes of this disclosure, the term "AR" is used synonymously with the term "MR".

[0286] Figure 1 illustrates an example of a mixed reality scenario involving a virtual reality object and a physical object that are visible to a person. In Figure 1, MR scene 100 is depicted, and the user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives "seeing" a robotic figure 130 standing on the real-world platform 120 and a flying cartoonish avatar character 140 that appears to be a personification of a bumblebee, although these elements do not exist in the real world.

[0287] It may be desirable for a 3D display to generate a distance-accommodative response corresponding to the virtual depth of each point within the display's field of view, in order to produce a true sense of depth, more specifically, a simulated sense of surface depth. If the distance-accommodative response for a display point does not correspond to the virtual depth of that point, as determined by the binocular depth cues for convergence and stereopsis, the human eye may experience distance-accommodative collision, which can result in unstable image formation, harmful eye strain, headaches, and, in the absence of distance-accommodative information, a near-complete lack of surface depth.

[0288] VR, AR, and MR experiences can be provided by a display system having a display that provides the viewer with images corresponding to multiple depth planes. The images may differ for each depth plane (e.g., providing slightly different presentations of a scene or object) and can be individually focused by the viewer's eyes, thereby helping to provide the user with depth cues based on the eye's accommodation required to focus on different image features relating to a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a credible perception of depth.

[0289] Figure 2 illustrates an embodiment of the wearable system 200, which can be configured to provide AR / VR / MR scenes. The wearable system 200 may also be referred to as the AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230, which can be worn by a user, wearer, or viewer 210. The display 220 can be positioned in front of the user's eyes. The display 220 can present AR / VR / MR content to the user. The display 220 may comprise a head-mounted display (HMD) that is worn on the user's head.

[0290] In some embodiments, speaker 240 is coupled to frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). Display 220 may include audio sensors (e.g., microphones) 232 to detect audio streams from the environment and capture ambient sounds. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. Stereo sound reception can be used to determine the location of sound sources. The wearable system 200 can perform voice or speech recognition on the audio stream.

[0291] The wearable system 200 may include an outward-facing imaging system 464 (shown in Figure 4) that observes the world within the user's surrounding environment. The wearable system 200 may also include an inward-facing imaging system 462 (shown in Figure 4) that can track the user's eye movements. The inward-facing imaging system can track the movement of one eye or both eyes. The inward-facing imaging system 462 may be mounted on the frame 230 and may communicate with a processing module 260 or 270 that processes the image information obtained by the inward-facing imaging system and can determine, for example, the pupil diameter or orientation of the user's eyes, eye movements, or eye posture. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. Images acquired by the camera may be used to determine the pupil size or eye orientation separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted for that eye.

[0292] As an example, the wearable system 200 can obtain an image of the user's posture using an outward-facing imaging system 464 or an inward-facing imaging system 462. The image may be a still image, a video frame, or a video.

[0293] The display 220 can be operably coupled to a local data processing module 260 (250), which can be mounted in various configurations, such as being fixedly attached to the frame 230 by wired or wireless connections, fixed to a helmet or hat worn by the user, built into headphones, or otherwise detachably attached to the user 210 (for example, in a backpack configuration, in a belt-connected configuration).

[0294] The local processing and data module 260 may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be used to assist in data processing, caching, and storage. The data may include (a) data captured from sensors (e.g., cameras in inward-facing imaging systems and / or outward-facing imaging systems), audio sensors (e.g., microphones), inertial measuring units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, or gyroscopes (e.g., operably coupled to frame 230 or otherwise attached to user 210), or (b) data potentially obtained or processed using the remote processing module 270 or remote data repository 280 for transmission to display 220 after processing or reading. The local processing and data module 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 by communication links 262 or 264 via wired or wireless communication links, etc., so that these remote modules can be used as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0295] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may comprise digital data storage equipment, which may be available through the internet or other networking configurations in a “cloud” resource configuration. In some embodiments, all data is stored, and all calculations are performed in the local processing and data module, enabling fully autonomous use from the remote module. B. Exemplary components of a wearable system

[0296] Figure 3 schematically illustrates exemplary components of a wearable system. Figure 3 shows a wearable system 200, which may include a display 220 and a frame 230. Enlarged Figure 202 schematically illustrates various components of the wearable system 200. In one implementation, one or more of the components illustrated in Figure 3 may be part of the display 220. The various components, individually or in combination, can collect various data (e.g., auditory or visual data) associated with the user or the user's environment of the wearable system 200. It should be understood that other embodiments may have additional or fewer components depending on the application in which the wearable system is used. Figure 3 provides some of the various components and a basic concept of the types of data that may be collected, analyzed, and stored through the wearable system.

[0297] Figure 3 shows an exemplary wearable system 200, which may include a display 220. The display 220 may comprise a display lens 226, which may be mounted on the housing or frame 230, corresponding to the user's head or frame 230. The display lens 226 may comprise one or more transparent mirrors, positioned by the housing 230 in front of the user's eyes 302, 304, and may be configured to bounce the projected light 338 into the eyes 302, 304, facilitating beam shaping, while also allowing the transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may be bent or focused to match a desired focal length of the projected light. As shown, two wide-field machine vision cameras 316 (also referred to as world cameras) are coupled to the housing 230 and can image the environment around the user. These cameras 316 may be dual-capture visible / invisible (e.g., infrared) light cameras. Camera 316 may be part of an outward-facing imaging system 464 shown in Figure 4. Images acquired by the world camera 316 can be processed by the attitude processor 336. For example, the attitude processor 336 may implement one or more object recognition devices 708 to identify the attitude of the user or another person in the user's environment, or to identify physical objects in the user's environment.

[0298] Continuing with reference to Figure 3, a pair of scanning laser-shaped wavefront (for, e.g., depth) light projection modules are shown, with display mirrors and optics configured to project light 338 into eyes 302, 304. The depicted figures also show two miniature infrared cameras 324 paired with infrared light sources 326 (light-emitting diodes "LEDs" etc.) configured to track the user's eyes 302, 304 and to support rendering and user input. Cameras 324 may be part of an inward-facing imaging system 462, as shown in Figure 4. The wearable system 200 may further feature a sensor assembly 339, which has X, Y, and Z-axis accelerometer capabilities and magnetic compass and X, Y, and Z-axis gyroscope capabilities, preferably capable of providing data at relatively high frequencies such as 200 Hz. The sensor assembly 339 may be part of an IMU, as described with reference to Figure 2. The described system 200 may also include a head attitude processor 336, such as an ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or ARM processor (Advanced Reduced Instruction Set Machine), which may be configured to calculate real-time or near real-time user head attitude from wide-field image information output from the capture device 316. The head attitude processor 336 may be a hardware processor and may be implemented as part of the local processing and data module 260 shown in Figure 2.

[0299] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure the distance between an object in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., LiDAR), an ultrasonic depth sensor, or a depth-sensing camera. In some implementations where the camera 316 has depth-sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0300] Also shown is a processor 332, configured to perform digital or analog processing and derive attitude from gyroscope, compass, or accelerometer data from sensor assembly 339. The processor 332 may also be part of a local processing and data module 260, as shown in Figure 2. The wearable system 200 may also include a positioning system, such as GPS 337 (Global Positioning System), as shown in Figure 3, to assist in attitude and positioning analysis. In addition, GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0301] The wearable system may combine data obtained by GPS337 and a remote computing system (e.g., a remote processing module 270, another user's ARD, etc.) which can provide more information about the user's environment. In one embodiment, the wearable system can determine the user's location based on GPS data and retrieve a world map containing virtual objects associated with the user's location (e.g., by communicating with the remote processing module 270). In another embodiment, the wearable system 200 can monitor the environment using a world camera 316 (which may be part of an outward-facing imaging system 464 shown in Figure 4). Based on images obtained by the world camera 316, the wearable system 200 can detect objects in the environment (e.g., by using one or more object recognition devices). The wearable system can further interpret characters using data obtained by GPS337.

[0302] The wearable system 200 may also include a rendering engine 334, which can be configured to provide the user with local rendering information for a global user view and to facilitate scanner operation and imaging into the user's eyes. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or graphics processing unit). In some embodiments, the rendering engine is part of a local processing and data module 260. The rendering engine 334 can be communicatively coupled to other components of the wearable system 200 (e.g., via wired or wireless links). For example, the rendering engine 334 can be coupled to an eye camera 324 via communication link 274 and to a projection subsystem 318 (which can project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via communication link 272. The rendering engine 34 can also communicate with other processing units, such as a sensor pose processor 32 and an image pose processor 336, via links 276 and 294, respectively.

[0303] Camera 324 (e.g., a small infrared camera) may be used to track eye posture and support rendering and user input. Some exemplary eye postures may include the location or depth of focus the user is looking at (which may be estimated using eye convergence and divergence movements). GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide rough or fast posture estimation. One or more of the cameras 316 can obtain images and postures, which, in conjunction with data from associated cloud computing resources, may be used to map the local environment and share the user view with others.

[0304] The exemplary components depicted in Figure 3 are for illustrative purposes only. Multiple sensors and other functional modules are shown together for ease of illustration and explanation. Some embodiments may include only one or a subset of these sensors or modules. Furthermore, the location of these components is not limited to the positions depicted in Figure 3. Some components may be mounted or housed within other components, such as belt-mounted components, handheld components, or helmet components. In one embodiment, the image pose processor 336, sensor pose processor 332, and rendering engine 334 may be located within a belt pack and configured to communicate with other components of the wearable system via wireless communication such as ultra-wideband, Wi-Fi, Bluetooth®, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by the user. However, some components of the wearable system 200 may be mounted on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0305] With regard to the projection of light 338 into the user's eyes 302, 304, in some embodiments, a camera 324 may be used to measure the location where the center of the user's eyes geometrically converges, generally coinciding with the focal point of the eyes or “depth of focus”. The three-dimensional surface of all points where the eyes converge may be referred to as the “single-vision trajectory”. The focal length can take a finite number of depths or can vary infinitely. Light projected from the convergence-divergence distance appears to converge onto the target eyes 302, 304, while light in front of or behind the convergence-divergence distance is blurred. Embodiments of the wearable systems and other display systems of this disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0306] The human visual system is complex, making it difficult to provide a realistic perception of depth. Viewers of objects may perceive them as three-dimensional due to a combination of convergence / divergence movements and accommodation. The convergence / divergence movements of two eyes relative to each other (e.g., rotational movement of the pupils toward or away from each other to converge the lines of sight and fixate on an object) are closely related to the focusing (or "accommodation") of the eye's lens. Under normal conditions, a change in the focal point of the eye's lens or the eye's accommodation to change focus from one object to another at a different distance will automatically produce a corresponding change in convergence / divergence to the same distance, under a relationship known as the "accommodation-convergence / divergence reflex." Similarly, a change in convergence / divergence will, under normal conditions, induce a corresponding change in accommodation. Display systems that provide a better match between accommodation and convergence / divergence can form a more realistic and comfortable simulation of three-dimensional images.

[0307] Furthermore, spatially coherent light with a beam diameter of less than approximately 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Therefore, to create the illusion of appropriate depth of field, the convergence and divergence movements of the eye may be tracked using camera 324, and the rendering engine 334 and projection subsystem 318 may be used to render all objects on or near the monosight trajectory in focus, and all other objects to a variable degree of out-of-focus rendering (e.g., using intentionally created blur). Preferably, system 220 renders to the user at a frame rate of approximately 60 frames / second or higher. As described above, preferably, camera 324 may be used for eye tracking, and the software may be configured to take up not only the convergence and divergence motion geometry but also a focus location cue to serve as user input. Preferably, such a display system is configured with brightness and contrast suitable for daytime or nighttime use.

[0308] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds for visual object alignment, angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc, which is considered to be near the limit of the human eye, although not limited by theory. The display system 220 may be integrated with a positioning system, which may include GPS elements, optical tracking, a compass, an accelerometer, or other data sources to assist in position and orientation determination. Positioning information may be used to facilitate accurate rendering in the user's view of the relevant world (for example, such information would help the glasses grasp their location relative to the real world).

[0309] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the user's eye accommodation. Unlike conventional 3D display approaches that force the user to focus on the location where the image is projected, in some embodiments, the wearable system is configured to automatically adjust the focus of the projected virtual content, enabling more comfortable viewing of one or more images presented to the user. For example, if the user's eye has a current focus of 1 m, the image may be projected to match the user's focus. If the user shifts their focus to 3 m, the image is projected to match the new focus. Thus, rather than forcing the user to a predetermined focus, the wearable system 200 in some embodiments allows the user's eye to function in a more natural manner.

[0310] Such a wearable system 200 can eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 are configured to project a virtual image to a variable focal length through one or more variable focal elements (VFEs). In one or more embodiments, 3D perception may be achieved through a multi-plane focal system that projects the image to a fixed focal plane away from the user. Other embodiments employ a variable plane focal, where the focal plane is moved back and forth in the z-direction to match the current state of the user's focus.

[0311] In both multi-plane focal systems and variable-plane focal systems, the wearable system 200 may employ eye tracking to determine the convergence and divergence movements of the user's eyes, determine the user's current focus, and project a virtual image onto the determined focus. In other embodiments, the wearable system 200 includes an optical modulator that projects a variable-focus light beam in a variable raster pattern across the retina through a fiber scanner or other light-generating source. Thus, the ability of the wearable system 200's display to project an image to a variable focal length not only facilitates the user's accommodation for viewing objects in 3D, but may also be used to compensate for the user's eye abnormalities, as further described in U.S. Patent Publication No. 2016 / 0270656 (which is incorporated herein in whole by reference). In some other embodiments, the spatial light modulator may project an image to the user through various optical components. For example, as further described below, the spatial light modulator may project an image onto one or more waveguides, which then transmit the image to the user. C. Waveguide Stack Assembly

[0312] Figure 4 illustrates an embodiment of a waveguide stack for outputting image information to the user. The wearable system 400 includes a waveguide stack or stacked waveguide assembly 480, which may be used to provide three-dimensional perception to the eyes / brain using a plurality of waveguides 432b, 434b, 436b, 438b, and 4400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of Figure 2, and Figure 4A shows some parts of that wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of Figure 2.

[0313] Continuing with Figure 4, the waveguide assembly 480 may also include several features 458, 456, 454, and 452 between the waveguides. In some embodiments, features 458, 456, 454, and 452 may be lenses. In other embodiments, features 458, 456, 454, and 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures for forming air gaps).

[0314] Waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using varying levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a specific depth plane and configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may be used to input image information into waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surfaces of image input devices 420, 422, 424, 426, and 428 and is fed into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be fed into each waveguide and output the entire field of cloned collimated beams, directed toward the eye 410 at a specific angle (and divergence) corresponding to a depth plane associated with a particular waveguide.

[0315] In some embodiments, the image input devices 420, 422, 424, 426, and 428 are discrete displays, each generating image information for input into the corresponding waveguides 440b, 438b, 436b, 434b, and 432b, respectively. In some other embodiments, the image input devices 420, 422, 424, 426, and 428 are output terminals of a single multiplexed display, which can send image information to each of the image input devices 420, 422, 424, 426, and 428, for example, via one or more optical conduits (such as optical fiber cables).

[0316] The controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, and 428. The controller 460 includes programming (e.g., instructions in a non-transient computer-readable medium) to coordinate the timing and delivery of image information to the waveguides 440b, 438b, 436b, 434b, and 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 460 may be part of a processing module 260 or 270 (illustrated in Figure 2).

[0317] Waveguides 440b, 438b, 436b, 434b, and 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, and 432b may each be planar or have another shape (e.g., curved), with a main upper surface and a bottom surface and a rim extending between their main upper and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, and 432b may each include light extraction optical elements 440a, 438a, 436a, 434a, and 432a, respectively, configured to extract light from the waveguide by redirecting the light propagating within each individual waveguide out of the waveguide and outputting image information to the eye 410. The extracted light may also be referred to as externally coupled light, and the light extraction optical elements may also be referred to as externally coupled optical elements. The beam of extracted light is output by the waveguide at the point where light propagating within the waveguide strikes the light redirection element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffracting optical features. For the sake of clarity of the illustration and to facilitate the explanation, they are shown positioned on the bottom main surface of waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be positioned on the top or bottom main surface, or directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be mounted on a transparent substrate and formed within a layer of material that forms the waveguides 440b, 438b, 436b, 434b, and 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, and 432b may be monolithic pieces of material, and the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be formed on or inside the surface of the material piece.

[0318] Continuing with reference to Figure 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, and 432b is configured to emit light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is introduced into such waveguide 432b. The collimated light may represent the optical infinity focal plane. The next upper waveguide 434b may be configured to emit collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate some convex wavefront curvature so that the eye / brain interprets the light emanating from the next upper waveguide 434b as emanating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce a different, gradually increasing wavefront curvature so that the eye / brain interprets the light emanating from the third waveguide 436b as originating from a second focal plane that is closer inward toward the person from optical infinity than the light from the next upper waveguide 434b.

[0319] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are configured similarly, and the highest waveguide in the stack, 440b, is used to transmit its output through all the lenses between it and the eye for a convergent focusing force representing the focal plane closest to the person. When viewing / interpreting light from the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be positioned on top of the stack to compensate for the stack of lenses 458, 456, 454, 452 and to compensate for the convergent force of the lower lens stacks 458, 456, 454, 452. (The compensating lens layer 430 and the stacked waveguide assembly 480 may be configured as a whole to transmit light originating from the world 470 to the eye 410 with substantially the same level of divergence (or collimation) as the light had when it was first received by the stacked waveguide assembly 480.) Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light-extracting optical elements of the waveguide and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0320] Continuing with reference to Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be configured to redirect light out of their respective waveguides for specific depth planes associated with the waveguides, and to output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have different configurations of the light extraction optical elements, outputting light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be three-dimensional or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a may be three-dimensional holograms, surface holograms, and / or diffraction gratings. Optical elements for light extraction, such as diffraction gratings, are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015 (which is incorporated herein by reference as a whole).

[0321] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, and 432a are diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOEs have relatively low diffraction efficiency such that only a portion of the beam light is deflected toward the eye 410 at each intersection of the DOEs, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is therefore split into several associated emission beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of emission toward the eye 304 with respect to this particular collimated beam bouncing within the waveguide.

[0322] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer-dispersed liquid crystal in which droplets have a diffraction pattern within a host medium, and the refractive index of the droplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplets may be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0323] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may change inversely with the viewer's pupil size. As a result, as the pupil size of the viewer's eye decreases, the depth of field increases so that one plane that is indistinguishable because its location is beyond the eye's depth of field becomes discernible and appears more in focus with the decrease in pupil size and the corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may decrease with the decrease in pupil size. For example, it may not be possible for a viewer to clearly perceive the details of both a first and a second depth plane at one pupil size without adjusting the eye's accommodation from one depth plane to the other. However, these two depth planes can simultaneously be sufficiently in focus for the user at another pupil size without changing accommodation.

[0324] In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size or orientation, or in response to the reception of an electrical signal indicating a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing load on the system and thereby increase the system's responsiveness. In embodiments where the DOE for a waveguide is switchable between on and off states, the DOE may be switched off when the waveguide receives image information.

[0325] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be difficult in light of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to the determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the size of the emitted beam may be varied using a variable aperture.

[0326] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 may sometimes be referred to as the FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210, and this encompasses a portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the FOV of the world camera may be larger than that of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the eye-moving field of view (FOR). FOR may include a solid angle of 4π steradians surrounding the wearable system 400, as the wearer can move their body, head, or eyes and perceive substantially any direction in space. In other contexts, the wearer's movement may be more restrained, and accordingly, the wearer's FOR may be tangent to a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures made by the user (e.g., hand or finger gestures), to detect objects in the world 470 in front of the user, and so on.

[0327] The wearable system 400 includes an audio sensor 232, for example, a microphone, which can capture ambient sound. As described above, in some embodiments, one or more other audio sensors can be positioned to provide stereo sound reception useful for determining the location of a speech source. In another embodiment, the audio sensor 232 may comprise a directional microphone, which can also provide such useful directional information regarding the location of the audio source. The wearable system 400 can use information from both the outward-facing imaging system 464 and the audio sensor 230, for example, to locate a speech source or to determine the active speaker at a particular moment. For example, the wearable system 400 can use speech recognition, either alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror), to determine the speaker's identification. In another embodiment, the wearable system 400 can determine the speaker's location in the environment based on the sound obtained from the directional microphone. The wearable system 400 can use a speech recognition algorithm to analyze sounds originating from the speaker's location, determine the content of the speech, and use speech recognition techniques to identify the speaker (e.g., name or other demographic information).

[0328] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inward-facing imaging system 466 may be used to capture an image of the eye 410 and to determine the pupil size and / or orientation of the eye 304. The inward-facing imaging system 466 may be used to determine the direction the user is looking (e.g., eye posture) or to obtain an image for the user's biometric identification (e.g., via iris recognition). In some embodiments, at least one camera may be used for each eye, independently, to separately determine the pupil size or eye posture of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only one eye 410 (e.g., using only one camera per pair of eyes) was determined and assumed to be similar with respect to both of the user's eyes. Images obtained by the inward-facing imaging system 466 may be used by the wearable system 400 to determine the user's eye posture or mood, or to determine the audio or visual content to be presented to the user. The wearable system 400 may also use sensors such as an IMU, accelerometer, and gyroscope to determine the head posture (e.g., head position or head orientation).

[0329] The wearable system 400 may include a user input device 466 that allows the user to input commands into a controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, touchscreen, joystick, multi-degree-of-freedom (DOF) controller, capacitive sensing device, game controller, keyboard, mouse, directional pad (D-pad), wand, tactile device, totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller may sense user input in translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll), which can be some or all of the controller's possible movements. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports both translation and rotation may be referred to as 6DOF. In some cases, the user may use a finger (e.g., thumb) to press or swipe over a touch sensor input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held in the user's hand while using the wearable system 400. The user input device 466 can be connected to the wearable system 400 via wired or wireless communication. D. Other components of the wearable system

[0330] In many implementations, the wearable system may include other components in addition to, or as alternatives to, the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a sense of touch to the user. For example, the tactile devices or components may provide a sense of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The tactile sensation may replicate the feeling of a physical object represented by the virtual object, or the feeling of an imaginary object or character represented by the virtual content (e.g., a dragon). In some implementations, the tactile devices or components may be worn by the user (e.g., user-wearable gloves). In some implementations, the tactile devices or components may be held by the user.

[0331] A wearable system may include, for example, one or more physical objects that are operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as pieces of metal or plastic, walls, or table surfaces. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so that it appears to the user as being on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and trackpad so that they appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so that they appear to be on the surface of a thin rectangular aluminum plate that acts as a totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect user operation or interaction or touch using the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. The user input device 466 (shown in Figure 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with posture to interact with the wearable system or other users.

[0332] Examples of wearable devices, HMDs, and display systems and usable tactile devices and totems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777 (which is incorporated herein in whole by reference). E. Examples of eye images

[0333] Figure 5 illustrates an image of an eye 500, including the eyelid 504, sclera 508 ("white of the eye"), iris 512, and pupil 516. Curve 516a indicates the pupillary boundary between the pupil 516 and the iris 512, and curve 512a indicates the marginal boundary between the iris 512 and the sclera 508. The eyelid 504 includes an upper eyelid 504a and a lower eyelid 504b. The eye 500 is illustrated in a natural, resting position (for example, oriented so that both the user's face and line of sight are directed toward a distant object directly in front of the user). The natural resting position of the eye 500 may be indicated by a natural resting direction 520, which is perpendicular to the surface of the eye 500 when it is in the natural resting position (for example, directly outward from the surface facing the eye 500 as shown in Figure 5), and in this embodiment, is centered within the pupil 516.

[0334] As the eye 500 moves to view a different object, the eye orientation will change with respect to the natural resting direction 520. The current eye orientation is in a direction perpendicular to the surface of the eye (and centered within the pupil 516), but can be determined by referring to the eye orientation direction 524, which the eye is oriented toward the object it is currently pointing at. Referring to the exemplary coordinate system shown in Figure 5A, the orientation of the eye 500 can be expressed as two angular parameters, both indicating the azimuth and zenith deflection of the eye's eye orientation direction 524 relative to the eye's natural resting direction 520. For illustrative purposes, these angular parameters can be expressed as θ (azimuth deflection, determined from the base azimuth) and φ (zenith deflection, sometimes also referred to as polar deflection). In some implementations, the angular roll of the eye around the eye orientation direction 524 can be included in the determination of the eye orientation, and the angular roll can be included in the analysis below. In other implementations, other techniques for determining eye posture may be used, such as pitch, yaw, and, voluntarily, roll systems.

[0335] Eye images can be obtained from video using any appropriate process, for example, by using a video processing algorithm that can extract images from one or more sequential frames. Eye pose can be determined from eye images using various eye tracking techniques. For example, eye pose can be determined by considering the lens effect of the cornea on a given light source. Any suitable eye tracking technique can be used to determine eye pose. F. Examples of eye-tracking systems

[0336] Figure 6 illustrates a schematic diagram of a wearable system 600, including an eye-tracking system. In at least some embodiments, the wearable system 600 may include components located within a head-mounted unit 602 and components located within a non-head-mounted unit 604. The non-head-mounted unit 604 may, in some embodiments, be a belt-mounted component, a handheld component, a backpack component, a remote component, etc. Integrating some of the components of the wearable system 600 into the non-head-mounted unit 604 may help reduce the size, weight, complexity, and cost of the head-mounted unit 602. In some implementations, some or all of the functionality described as being performed by one or more components of the head-mounted unit 602 and / or the non-head-mounted unit 604 may be provided using one or more components included in either of the wearable system 600. For example, some or all of the functionality described below in relation to the CPU 612 of the head-mounted unit 602 may be provided using the CPU 616 of the non-head-mounted unit 604, and vice versa. In some embodiments, some or all of such functionality may be provided using peripheral devices of the wearable system 600. Furthermore, in some implementations, some or all of such functionality may be provided using one or more cloud computing devices or other remotely located computing devices in a manner similar to that described above with reference to Figure 2.

[0337] As shown in Figure 6, the wearable system 600 may include an eye-tracking system, which includes a camera 324 that captures an image of the user's eye 610. Optionally, the eye-tracking system may also include light sources 326a and 326b (such as light-emitting diodes "LEDs"). The light sources 326a and 326b may generate flashes of light (e.g., reflections from the user's eye appearing in the image of the eye captured by the camera 324). The positions of the light sources 326a and 326b relative to the camera 324 may be known, and consequently, the positions of the flashes in the image captured by the camera 324 may be used when tracking the user's eye (as will be discussed in more detail below in relation to Figure 7). In at least one embodiment, there may be one light source 326 and one camera 324 associated with one of the user's eyes 610. In another embodiment, there may be one light source 326 and one camera 324 associated with each of the user's eyes 610. In yet another embodiment, there may be one or more cameras 324 and one or more light sources 326 associated with one or each of the user's eyes 610. In a specific example, there may be two light sources 326a and 326b and one or more cameras 324 associated with each of the user's eyes 610. In yet another embodiment, there may be three or more light sources, such as light sources 326a and 326b, and one or more cameras 324 associated with each of the user's eyes 610. In some implementations described herein, two or more cameras may be employed to image a given eye.

[0338] The eye tracking module 614 may receive images from the eye tracking camera 324, analyze the images, and extract various information. For example, the eye tracking module 614 may detect the user's eye posture, the three-dimensional position of the user's eyes relative to the eye tracking camera 324 (and the head-mounted unit 602), the direction of one or both of the user's eyes 610 in focus, the user's convergence / divergence motion depth (e.g., the depth from which the user is in focus), the position of the user's pupils, the position of the user's cornea and / or corneal sphere, the rotation center of one or each of the user's eyes, and the viewpoint center of one or each of the user's eyes, or any combination thereof. The eye tracking module 614 may extract such information using techniques described below in relation to Figures 7-11 and / or 12-21. As shown in Figure 6, the eye tracking module 614 may be a software module implemented using the CPU 612 in the head-mounted unit 602.

[0339] One camera 324 is shown in Figure 6 to image the eye, but in some implementations, as discussed herein, multiple cameras may be used to image the eye and for measurements such as corneal center and / or rotation center measurement, or otherwise for eye tracking or other purposes.

[0340] Data from the eye-tracking module 614 may be provided to other components within the wearable system. In one embodiment, such data may be transmitted to components within a non-head-mounted unit 604, such as a CPU 616, which includes software modules for a light field rendering controller 618 and an alignment observer 620.

[0341] The rendering controller 618 may adjust the image displayed to the user using information from the eye tracking module 614, with the help of the rendering engine 622 (which may be a software module within the GPU 620 and capable of providing images to the display 220). In one embodiment, the rendering controller 618 may adjust the image displayed to the user based on the user's center of rotation or center of viewpoint. In particular, the rendering controller 618 may use information about the user's center of viewpoint to simulate a rendering camera (for example, simulating image acquisition from the user's viewpoint) and adjust the image displayed to the user based on the simulated rendering camera.

[0342] A rendering camera, sometimes also called a "pinhole perspective projection camera" (or simply a "perspective projection camera") or a "virtual pinhole camera" (or simply a "virtual camera"), is a simulated camera used to render virtual image content from a database of objects in a virtual world. Objects may have a location and orientation relative to the user or wearer, and possibly relative to real objects in the environment surrounding the user or wearer. In other words, a rendering camera may represent a viewpoint in rendering space from which the user or wearer should view 3D virtual content (e.g., virtual objects) in the rendering space. The rendering camera may be managed by a rendering engine and render virtual images based on a database of virtual objects to be presented to the eye. The virtual images may be rendered as if they were taken from the user's or wearer's viewpoint. For example, a virtual image may be rendered as if captured by a pinhole camera (corresponding to the "rendering camera") having a specific set of intrinsic parameters (e.g., focal length, camera pixel size, principal point coordinates, distortion / distortion parameters, etc.) and a specific set of auxiliary parameters (e.g., translation and rotation components relative to the virtual world). The virtual image is captured from the viewpoint of such a camera, having the position and orientation of the rendering camera (e.g., the auxiliary parameters of the rendering camera). The system can define and / or adjust the intrinsic and auxiliary rendering camera parameters. For example, the system may define a specific set of auxiliary rendering camera parameters so that the virtual image is rendered as if captured from the viewpoint of a camera having a specific location relative to the user's or wearer's eyes, so as to provide an image that appears as if it were from the viewpoint of the user or wearer. The system may later dynamically adjust the auxiliary rendering camera parameters on the fly to maintain alignment with the specific location. Similarly, the intrinsic rendering camera parameters may also be defined and dynamically adjusted over time.In some implementations, images are rendered as if they were captured from the viewpoint of a camera that has an aperture (e.g., a pinhole) at a specific location relative to the user's or wearer's eye (such as the center of the viewpoint, the center of rotation, or another location).

[0343] In some embodiments, the system may create or dynamically reposition and / or reorient one rendering camera for the user's left eye and another rendering camera for the user's right eye as the user's eyes are physically separated from each other and therefore consistently positioned in different locations. In at least some implementations, virtual content rendered from the viewpoint of the rendering camera associated with the viewer's left eye may be presented to the user through the left eyepiece of a head-mounted display (e.g., head-mounted unit 602), and virtual content rendered from the viewpoint of the rendering camera associated with the user's right eye may be presented to the user through the right eyepiece of such a head-mounted display. Further details discussing the creation, adjustment, and use of rendering cameras in the rendering process are provided in U.S. Patent Application No. 15 / 274,823, entitled "METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN 3D RECONSTRUCTION" (expressly incorporated herein in whole by reference for any purpose).

[0344] In some embodiments, one or more modules (or components) of System 600 (e.g., a light field rendering controller 618, a rendering engine 620, etc.) may determine the position and orientation of the rendering camera in rendering space based on the position and orientation of the user's head and eyes (e.g., determined based on head pose and eye tracking data, respectively). That is, System 600 can, in effect, map the position and orientation of the user's head and eyes to specific locations and angles in the 3D virtual environment, position and orient the rendering camera to those locations and angles in the 3D virtual environment, and render virtual content for the user as it will be captured by the rendering camera. Further details discussing the real-world / virtual-world mapping process are provided in U.S. Patent Application No. 15 / 296,869, entitled "SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE" (expressly incorporated herein in whole by reference for any purpose). As an example, the rendering controller 618 may adjust the depth to which the image is displayed by selecting a depth plane (or multiple depth planes) to be used for displaying the image at any given time. In some implementations, such depth plane switching may be performed through the adjustment of one or more intrinsic rendering camera parameters. For example, the light field rendering controller 618 may adjust the focal length of the rendering camera when performing depth plane switching or adjustment. As will be described in more detail below, the depth planes may be switched based on user-determined convergence / divergence motion or fixation depth.

[0345] The alignment observer 620 may use information from the eye tracking module 614 to determine whether the head-mounted unit 602 is properly positioned on the user's head. In one embodiment, the eye tracking module 614 may provide eye location information, such as the position of the user's eye's center of rotation, indicating the three-dimensional position of the user's eye relative to the camera 324. The head-mounted unit 602 and the eye tracking module 614 may use the location information to determine whether the display 220 is properly aligned within the user's field of view, or whether the head-mounted unit 602 (or headset) is detached or otherwise misaligned with the user's eye. As an example, the alignment observer 620 may be able to determine whether the head-mounted unit 602 is detached from the user's nasal bridge and therefore moves the display 220 away from the user's eyes and downward (which is undesirable), whether the head-mounted unit 602 is moved above the user's nasal bridge and therefore moves the display 220 closer to the user's eyes and upward, whether the head-mounted unit 602 is shifted to the left or right relative to the user's nasal bridge, whether the head-mounted unit 602 is lifted above the user's nasal bridge, or whether the head-mounted unit 602 is moved away from a desired position or range of positions in any other way. Generally, the alignment observer 620 may be able to determine whether the head-mounted unit 602, in particular the display 220, is properly positioned in front of the user's eyes. In other words, the alignment observer 620 may determine whether the left display in the display system 220 is properly aligned with the user's left eye and whether the right display in the display system 220 is properly aligned with the user's right eye. The alignment observer 620 may also determine whether the head-mounted unit 602 is properly positioned by determining whether the head-mounted unit 602 is positioned and oriented within a desired range of position and / or orientation relative to the user's eyes.

[0346] In at least some embodiments, the alignment observer 620 may generate user feedback in the form of alerts, messages, or other content. Such feedback may be provided to the user and may inform the user of any misalignment of the head-mounted unit 602, along with optional feedback on how to correct the misalignment (such as suggestions for adjusting the head-mounted unit 602 in a particular manner).

[0347] Exemplary alignment observation and feedback techniques that may be utilized by the alignment observer 620 are described in U.S. Patent Application No. 15 / 717,747 (Patent Attorney No. MLEAP.052A2), filed September 27, 2017, and U.S. Provisional Patent Application No. 62 / 644,321 (Patent Attorney No. MLEAP.195PR), filed March 16, 2018 (both are incorporated herein by reference as a whole). G. Examples of eye tracking modules

[0348] A detailed block diagram of an exemplary eye-tracking module 614 is shown in Figure 7A. As shown in Figure 7A, the eye-tracking module 614 may include various different submodules, provide various different outputs, and utilize various available data when tracking the user's eyes. In an embodiment, the eye-tracking module 614 may utilize available data including the geometric arrangement of the eye-tracking camera 324 relative to the light source 326 and the head-mounted unit 602, assumed eye dimensions 704 such as a typical distance of approximately 4.7 mm between the center of the user's corneal curvature and the mean center of rotation of the user's eye or a typical distance between the user's center of rotation and the center of viewpoint, and user-specific calibration data 706 such as the interpupillary distance of a particular user. Additional embodiments of incidental properties, inherent properties, and other information that may be employed by the eye tracking module 614 are described in U.S. Patent Application No. 15 / 497,726 (Patent Attorney Reference No. MLEAP.023A7), filed on 26 April 2017 (which is incorporated herein by reference in its entirety).

[0349] The image preprocessing module 710 may receive an image from an eye camera such as the eye camera 324 and perform one or more preprocessing (e.g., adjustment) operations on the received image. For example, the image preprocessing module 710 may apply Gaussian blur to the image, downsample the image to a lower resolution, apply an unsharp mask, apply an edge sharpening algorithm, or apply other suitable filters to assist in subsequent detection, localization, and labeling of flashes, pupils, or other features in the image from the eye camera 324. The image preprocessing module 710 may apply a low-pass filter, such as an open filter, or a morphological filter that can remove high-frequency noise from the pupil boundary 516a (see Figure 5), thereby removing noise that may interfere with pupil and flash determination. The image preprocessing module 710 may output the preprocessed image to the pupil identification module 712 and the flash detection and labeling module 714.

[0350] The pupil identification module 712 may receive preprocessed images from the image preprocessing module 710 and may identify regions of those images that include the user's pupil. In some embodiments, the pupil identification module 712 may determine the coordinates of the user's pupil's position in the eye-tracking image from the camera 324, i.e., the coordinates of its center or centroid. In at least some embodiments, the pupil identification module 712 may identify contours in the eye-tracking image (e.g., the contour of the pupil-iris boundary), identify contour moments (e.g., the center of mass), apply starburst pupil detection and / or Canny edge detection algorithms, exclude outliers based on intensity values, identify subpixel boundary points, correct eye-camera distortion (e.g., distortion in images captured by the eye camera 324), apply a random sample consensus (RANSAC) iterative algorithm, fit ellipses to the boundaries in the eye-tracking image, apply a tracking filter to the image, and identify the subpixel image coordinates of the user's pupil centroid. The pupil identification module 712 may output pupil identification data (which may indicate a region of the preprocessed image module 712 identified as representing the user's pupil) to the flash detection and labeling module 714. The pupil identification module 712 may also provide the flash detection module 714 with the 2D coordinates of the user's pupil in each eye tracking image (e.g., the 2D coordinates of the centroid of the user's pupil). In at least some embodiments, the pupil identification module 712 may also provide the same type of pupil identification data to the coordinate system normalization module 718.

[0351] The pupil detection techniques that may be utilized by the pupil identification module 712 are described in U.S. Patent Publication No. 2017 / 0053165 and U.S. Patent Publication No. 2017 / 0053166, both published on February 23, 2017 (which are incorporated herein by reference in their entirety).

[0352] The flash detection and labeling module 714 may receive a pre-processed image from module 710 and pupil identification data from module 712. Using this data, the flash detection module 714 may detect and / or identify flashes (i.e., reflections of light from the light source 326 from the user's eye) within a region of the pre-processed image showing the user's pupil. In an embodiment, the flash detection module 714 may search for bright areas in the eye-tracking image near the user's pupil, sometimes referred herein as "blobs" or local intensity maximums. In at least some embodiments, the flash detection module 714 may rescale (e.g., enlarge) the pupil ellipse to include additional flashes. The flash detection module 714 may filter flashes by size and / or intensity. The flash detection module 714 may also determine the 2D location of each flash in the eye-tracking image. In at least some embodiments, the flash detection module 714 may determine the 2D position of the flash relative to the user's pupil, which may also be referred to as the pupil-flash vector. The flash detection and labeling module 714 may label the flash and output a preprocessed image with the labeled flash to the 3D corneal center estimation module 716. The flash detection and labeling module 714 may also transmit data such as the preprocessed image from module 710 and pupil identification data from module 712. In some implementations, the flash detection and labeling module 714 may determine the light source (e.g., among several light sources in the system, including infrared light sources 326a and 326b) that produced each identified flash. In these embodiments, the flash detection and labeling module 714 may label the flash with information identifying the associated light source and output a preprocessed image with the labeled flash to the 3D corneal center estimation module 716.

[0353] Pupil and flash detection, as performed by modules such as modules 712 and 714, can use any preferred technique. For example, edge detection can be applied to an eye image to identify the flash and pupil. Edge detection can be applied by various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector can be applied to an image to detect edges such as lines in the image. Edges may include points located along lines corresponding to the local maximum derivative. For example, the pupil boundary 516a (see Figure 5) can be localized using the Canny edge detector. Once the location of the pupil is determined, various image processing techniques can be used to detect the “orientation” of the pupil 116. Determining the eye orientation of an eye image may also be referred to as eye orientation detection of an eye image. Orientation may also be referred to as gaze direction, direction of orientation, or eye orientation. For example, a pupil may be looking to the left toward an object, and the pupil orientation may be classified as a leftward orientation. Other methods can also be used to detect the location of the pupil or flash. For example, concentric rings may be located within an eye image using a Canny edge detector. In another embodiment, integral-differential operators may be used to locate the limbal boundary of the pupil or iris. For example, the Daugman integral-differential operator, the Hough transform, or other iris segmentation techniques may be used to return a curve that estimates the boundary of the pupil or iris.

[0354] The 3D corneal center estimation module 716 may receive preprocessed images from modules 710, 712, and 714, including detected flash data and pupil identification data. The 3D corneal center estimation module 716 may use this data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 716 may estimate the 3D position of the center of the corneal curvature of the eye or the center of the user's corneal sphere, for example, the center of an imaginary sphere having a surface portion extending generally with the user's cornea. The 3D corneal center estimation module 716 may provide data indicating the estimated 3D coordinates of the corneal sphere and / or the user's cornea to the coordinate system normalization module 718, the optical axis determination module 722, and / or the light field rendering controller 618. Further details of the operation of the 3D corneal center estimation module 716 are provided herein in reference to Figures 11A–16C. Exemplary techniques for estimating the location of ocular features such as the cornea or corneal sphere, which may be utilized by the 3D corneal center estimation module 716 and other modules in the wearable system of this disclosure, are discussed in U.S. Patent Application No. 15 / 497,726 (Patent Attorney Reference No. MLEAP.023A7), filed on 26 April 2017 (incorporated herein in whole by reference).

[0355] The coordinate system normalization module 718 may optionally be contained within the eye tracking module 614 (as indicated by its dashed contour). The coordinate system normalization module 718 may receive data indicating the estimated 3D coordinates of the user's corneal center (and / or the user's corneal sphere) from the 3D corneal center estimation module 716, or it may receive data from other modules. The coordinate system normalization module 718 may normalize the eye camera coordinate system, which may help compensate for slippage of the wearable device (e.g., slippage of a head-mounted component from its normally stationary position on the user's head, which can be identified by the alignment observer 620). The coordinate system normalization module 718 may rotate the coordinate system to align the z-axis of the coordinate system (e.g., the convergence-divergence motion depth axis) with the corneal center (e.g., as indicated by the 3D corneal center estimation module 716), and may translate the camera center (e.g., the origin of the coordinate system) to a predetermined distance away from the corneal center, such as 30 mm (for example, module 718 may enlarge or shrink the eye tracking image depending on whether the eye camera 324 is determined to be closer or further than a predetermined distance). By using this normalization process, the eye tracking module 614 may be able to establish consistent orientation and distance in the eye tracking data, relatively independently of variations in the headset positioned on the user's head. The coordinate system normalization module 718 may provide the 3D coordinates of the corneal (and / or corneal sphere) center, pupil identification data, and preprocessed eye tracking images to the 3D pupil center locator module 720.

[0356] The 3D pupil center locator module 720 may receive data including the 3D coordinates of the center of the user's cornea (and / or corneal sphere), pupil location data, and preprocessed eye tracking images in a normalized or denormalized coordinate system. The 3D pupil center locator module 720 may analyze such data to determine the 3D coordinates of the user's pupil center in a normalized or denormalized eye camera coordinate system. The 3D pupil center locator module 720 may determine the location of the user's pupil in three dimensions based on the 2D position of the pupil centroid (as determined by module 712), the 3D position of the corneal center (as determined by module 716), assumed eye dimensions 704 such as the size of a typical user's corneal sphere and the typical distance from the corneal center to the pupil center, and optical properties of the eye such as the refractive index of the cornea (relative to the refractive index of air), or any combination thereof. Techniques for estimating the location of ocular features such as the pupil, which may be utilized by the 3D pupil center locator module 720 and other modules in the wearable system of this disclosure, are discussed in U.S. Patent Application No. 15 / 497,726 (Patent Attorney Reference No. MLEAP.023A7), filed on 26 April 2017 (which is incorporated herein by reference in its entirety).

[0357] The optical axis determination module 722 may receive data from modules 716 and 720 indicating the 3D coordinates of the user's cornea and the user's pupil center. Based on such data, the optical axis determination module 722 may identify a vector from the corneal center (e.g., from the center of the corneal sphere) to the user's pupil center that can define the optical axis of the user's eye. In an embodiment, the optical axis determination module 722 may provide an output defining the user's optical axis to modules 724, 728, 730, and 732.

[0358] The Center of Rotation (CoR) estimation module 724 may receive data from module 722 that includes parameters of the user's eye's optical axis (e.g., data indicating the direction of the optical axis in a coordinate system with a known relationship to the head-mounted unit 602). For example, the CoR estimation module 724 may estimate the center of rotation of the user's eye. The center of rotation may indicate a point around which the user's eye rotates when the eye rotates left, right, up, and / or down. It is assumed that a single point is sufficient even if the eye cannot rotate perfectly around that single point. In at least some embodiments, the CoR estimation module 724 may estimate the center of rotation of the eye by moving the pupil center (identified by module 720) or the center of corneal curvature (as identified by module 716) a specific distance along the optical axis (identified by module 722) toward the retina. This specific distance may be the assumed eye dimension 704. In one embodiment, the specific distance between the center of corneal curvature and the CoR may be approximately 4.7 mm. This distance may vary for a particular user based on any relevant data, including the user's age, gender, visual prescription, and other relevant characteristics. As discussed above, in some implementations, the center of corneal curvature or the center of the cornea refers to the center of curvature of a portion of the cornea or the center of curvature of a spherical surface that coincides with a portion of the corneal surface. For example, in some implementations, the center of corneal curvature or the center of the cornea refers to the center of curvature of the corneal apex or the center of curvature of a spherical surface that coincides with a portion of the surface of the corneal apex.

[0359] In at least some embodiments, the CoR estimation module 724 may refine its estimate of the rotation center of each of the user's eyes over time. For example, over time, the user will eventually rotate their eyes (to look elsewhere, something closer or farther away, or at some point left, right, up, or down), causing a deviation in each of the optical axes of their eyes. The CoR estimation module 724 may then analyze two (or more) optical axes identified by module 722 and locate a 3D point at the intersection of those optical axes. The CoR estimation module 724 may then determine the rotation center at that 3D point at the intersection. Such a technique can provide an estimate of the rotation center with accuracy that improves over time.

[0360] Various techniques may be employed to increase the accuracy of the CoR estimation module 724 and the determined CoR positions of the left and right eyes. In one embodiment, the CoR estimation module 724 may estimate the CoR by finding the average point of the intersection of optical axes determined over time for various different eye poses. In an additional embodiment, the module 724 may filter or average the estimated CoR positions over time, calculate a moving average of the estimated CoR positions over time, and / or apply a Kalman filter and known dynamics of the eye and eye-tracking system to estimate the CoR positions over time. In some implementations, a least-squares approach may be taken to determine one or more points of the intersection of optical axes. In such implementations, the system may identify a location where, at a given time, the sum of the squared distances to a given set of optical axes is reduced or minimized as a point of the optical axis intersection. As a specific embodiment, module 724 may calculate a weighted average of the determined point of the optical axis intersection and the assumed CoR position (e.g., 4.7 mm behind the center of the corneal curvature of the eye) so that the determined CoR slowly shifts over time from an assumed CoR position (e.g., 4.7 mm behind the center of the corneal curvature of the eye) to a slightly different location within the user's eye as eye tracking data for the user is acquired, thereby enabling user-specific refinement of the CoR position.

[0361] Under ideal conditions, the 3D position of the true CoR of the user's eye relative to the HMD should change over time by a negligible or minimal amount as the user moves their eye (e.g., as the user's eye rotates around its center of rotation). In other words, with respect to a given set of eye movements, the 3D position of the true CoR of the user's eye (relative to the HMD) should, hypothetically, not vary over time as much as any other point along the optical axis of the user's eye. Thus, the further a point along the optical axis is from the true CoR of the user's eye, the more its 3D position will exhibit over time a greater amount of variation or dispersion as the user moves their eye. In some embodiments, other submodules of the CoR estimation module 724 and / or eye tracking module 614 may utilize this statistical relationship to improve the accuracy of the CoR estimation. In such embodiments, other submodules of the CoR estimation module 724 and / or eye tracking module 614 may refine their estimate of the CoR 3D position over time by identifying the variability of their CoR estimate, which has low variability (e.g., low variance or standard deviation).

[0362] In a first embodiment, the CoR estimation module 724 estimates the CoR based on the intersections of several different optical axes (each associated with a user looking in a different direction). In this embodiment, the CoR estimation module 724 may utilize this statistical relationship (the true CoR should have low variance) by introducing a common offset in each direction of the optical axes (e.g., shifting each axis by a uniform amount) and determining whether the offset optical axes intersect each other at intersections having low variability, e.g., low dispersion or standard deviation. This can help correct small systematic errors in the calculation of the optical axis directions and refine the estimated location of the CoR to be closer to the true CoR.

[0363] In a second embodiment, the CoR estimation module 724 estimates the CoR by moving a specific distance (e.g., the distance between the center of corneal curvature and the CoR) along an optical axis (or another axis). In this embodiment, the system may vary, optimize, adjust, or otherwise adjust the specific distance between the center of corneal curvature and the CoR over time in such a way as to reduce or minimize the amount of variation in the estimated CoR position, e.g., variance and / or standard deviation (e.g., with respect to a large group of images of an eye captured at different times). For example, if the CoR estimation module 724 initially uses a specific distance value of 4.7 mm (from the center of corneal curvature, along the optical axis) to obtain a CoR position estimate, but the true CoR of a given user's eye may be located 4.9 mm behind the center of the eye's corneal curvature (along the optical axis), then the initial set of CoR position estimates obtained by the CoR estimation module 724 may exhibit a relatively high amount of variation, e.g., variance or standard deviation. In response to the detection of such relatively high amounts of variation (e.g., variance or standard deviation), the CoR estimation module 724 may search for and identify one or more points along the optical axis that have lower amounts of variation (e.g., variance or standard deviation), or it may identify a distance of 4.9 mm that has the lowest amount of variation (e.g., variance or standard deviation), and therefore the specific distance value used may be adjusted to 4.9 mm.

[0364] The CoR estimation module 724 may, in response to detecting that the current CoR estimate has a relatively high amount of variability (e.g., variance or standard deviation), search for alternative CoR estimates with lower variability (e.g., variance and / or standard deviation), or, naturally after obtaining the initial CoR estimate, search for alternative CoR estimates with lower variability (e.g., variance or standard deviation). In some embodiments, such optimization / adjustment may occur gradually over time, while in other embodiments, such optimization / adjustment may be performed during the initial user calibration session. In embodiments where such a procedure is performed during the calibration procedure, the CoR estimation module 724 may first not agree / adhere to any particular assumed distance, but rather collect a set of eye-tracking data over time, perform a statistical analysis on the set of eye-tracking data, and determine a particular distance value that, based on the statistical analysis, yields a CoR position estimate with a variability (e.g., variance or standard deviation) of the least likely amount (e.g., global minimum).

[0365] The interpupillary distance (IPD) estimation module 726 may receive data from the CoR estimation module 724 indicating the estimated 3D positions of the rotation centers of the user's left and right eyes. The IPD estimation module 726 may then estimate the user's IPD by measuring the 3D distance between the rotation centers of the user's left and right eyes. Generally, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye may be approximately equal to the distance between the user's pupil centers when the user is looking at optical infinity (e.g., the optical axes of the user's eyes are approximately parallel to each other), and this is a typical definition of interpupillary distance (IPD). The user's IPD may be used by various components and modules within the wearable system. In an embodiment, the user's IPD may be provided to an alignment observer 620 and used to assess the degree to which the wearable device is aligned with the user's eyes (e.g., whether the left and right display lenses are properly spaced according to the user's IPD). In another embodiment, the user's IPD may be provided to the condensation-divergence motion depth estimation module 728 and used in determining the user's condensation-divergence motion depth. Module 726 may employ various techniques, such as those discussed in relation to the CoR estimation module 724, to increase the accuracy of the estimated IPD. In an embodiment, the IPD estimation module 724 may apply filtering, averaging over time, weighted averaging, Kalman filtering, etc., including assumed IPD distance, as part of the estimation of the user's IPD in an accurate manner.

[0366] The convergence-divergence depth estimation module 728 may receive data from various modules and submodules within the eye tracking module 614 (as shown in relation to Figure 7A). In particular, the convergence-divergence depth estimation module 728 may employ data indicating the estimated 3D position of the pupillary center (e.g., as provided by module 720 described above), one or more determined parameters of the optical axis (e.g., as provided by module 722 described above), the estimated 3D position of the rotation center (e.g., as provided by module 724 described above), the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the rotation center) (e.g., as provided by module 726 described above), and / or one or more determined parameters of the optical axis and / or visual axis (e.g., as provided by modules 722 and / or module 730 described below). The convergence-divergence depth estimation module 728 may detect or otherwise obtain a measurement of the user's convergence-divergence depth, which may be the distance from the user at which the user's eyes are in focus. For example, when a user is looking at an object 3 feet in front of them, the user's left and right eyes have a convergence-divergence depth of 3 feet, while when the user is looking at a distant landscape (e.g., the optical axes of the user's eyes are approximately parallel to each other so that the distance between the centers of the user's pupils may be approximately equal to the distance between the centers of rotation of the user's left and right eyes), the user's left and right eyes have a convergence-divergence depth of infinity. In some implementations, the convergence-divergence depth estimation module 728 may utilize data indicating the estimated centers of the user's pupils (e.g., as provided by module 720) to determine the 3D distance between the estimated centers of the user's pupils. The convergence-divergence depth estimation module 728 may obtain a measure of convergence-divergence depth by comparing such determined 3D distance between pupil centers with the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the center of rotation) (e.g., as shown by module 726 described above).In addition to the 3D distance between pupil centers and the estimated IPD, the convergence-divergence depth estimation module 728 may calculate the convergence-divergence depth using known, assumed, estimated, and / or determined geometric shapes. In one embodiment, module 728 may estimate (e.g., determine) the user's convergence-divergence depth by combining the 3D distance between pupil centers, the estimated IPD, and the 3D CoR position in trigonometric calculations. Indeed, an evaluation of such a determined 3D distance between pupil centers relative to the estimated IPD may serve to indicate the user's current convergence-divergence depth measurement relative to optical infinity. In some embodiments, the convergence-divergence depth estimation module 728 may simply receive or access data indicating the estimated 3D distance between the estimated pupil centers of the user, for the purpose of obtaining such a measurement of the convergence-divergence depth. In some embodiments, the convergence-divergence depth estimation module 728 may estimate the convergence-divergence depth by comparing the user's left and right optical axes. In particular, the convergence-divergence depth estimation module 728 may estimate the convergence-divergence depth by locating the distance from the user where the user's left and right optical axes intersect (or where the projections of the user's left and right optical axes on a plane such as a horizontal plane intersect). The module 728 may utilize the user's IPD in this calculation by setting zero depth as the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the convergence-divergence depth estimation module 728 may determine the convergence-divergence depth by triangulation of eye-tracking data with known or derived spatial relationships.

[0367] In some embodiments, the convergence-divergence depth estimation module 728 may estimate the user's convergence-divergence depth based on the intersection of the user's visual axes (instead of its optical axes), which can provide a more accurate indication of the distance the user is in focus. In at least some embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. As will be discussed in more detail in relation to Figure 10, the user's optical axis and visual axis are generally not aligned. The visual axis is the axis along which a person looks, while the optical axis is defined by the center of the person's lens and pupil and can proceed through the center of the person's retina. In particular, the user's visual axis is generally defined by the location of the user's fovea, which is offset from the center of the user's retina and can thereby result in different optical and visual axes. In at least some of these embodiments, the eye tracking module 614 may include an optical axis / visual axis mapping module 730. The optical axis / visual axis mapping module 730 may correct the difference between the user's optical axis and visual axis and provide information about the user's visual axis to other components in the wearable system, such as the convergence / divergence motion depth estimation module 728 and the light field rendering controller 618. In some embodiments, module 730 may use assumed eye dimensions 704, which include a typical inward (nasal, towards the user's nose) offset of approximately 5.2° between the optical axis and visual axis. In other embodiments, module 730 may shift the user's left optical axis 5.2° to the right towards the nose and the user's right optical axis 5.2° to the left towards the nose in order to estimate the direction of the user's left and right optical axes. In other embodiments, module 730 may utilize per-user calibration data 706 when mapping the optical axis (e.g., as shown by module 722 described above) to the visual axis. In an additional embodiment, module 730 may shift the user's optical axis nasally by 4.0° to 6.5°, 4.5° to 6.0°, 5.0° to 5.4°, or any range formed by any of these values.In some arrays, module 730 may apply a shift at least in part based on the characteristics of a particular user, such as their age, gender, visual prescription, or other relevant characteristics, and / or at least in part based on a calibration process for a particular user (e.g., to determine the optical axis-visual axis offset for a particular user). In at least some embodiments, module 730 may also shift the origins of the left and right optical axes to correspond to the user's CoP (as determined by module 732) instead of the user's CoR.

[0368] When a voluntary center of gaze (CoP) estimation module 732 is provided, the locations of the user's left and right centers of gaze (CoP) may be estimated. The CoP is a useful location for a wearable system and, in at least some embodiments, may be directly in front of the pupil. In at least some embodiments, the CoP estimation module 732 may estimate the locations of the user's left and right centers of gaze based on the 3D location of the user's pupil center, the 3D location of the user's corneal curvature center, or such preferred data, or any combination thereof. As an example, the user's CoP may be about 5.01 mm in front of the corneal curvature center (e.g., 5.01 mm from the corneal spherical center toward the cornea of ​​the eye, along the optical axis) and about 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. The user's center of gaze may be directly in front of its pupil center. In one embodiment, the user's CoP may be less than approximately 2.0 mm from the user's pupil, less than approximately 1.0 mm from the user's pupil, less than approximately 0.5 mm from the user's pupil, or any range between any of these values. In another embodiment, the center of gaze may correspond to a location within the anterior chamber of the eye. In yet another embodiment, the CoP may be 1.0 mm to 2.0 mm, approximately 1.0 mm, 0.25 mm to 1.0 mm, 0.5 mm to 1.0 mm, or 0.25 mm to 0.5 mm from the user's pupil.

[0369] The center of view (CP) described herein (as a potentially desirable position for the pinhole of the rendering camera and an anatomical position within the user's eye) may be a position that plays a role in reducing and / or eliminating undesirable parallax deviation. In particular, the optical system of the user's eye roughly corresponds to a theoretical system formed by the projection of a pinhole in front of a lens onto the screen, where the pinhole, lens, and screen roughly correspond to the user's pupil / iris, crystalline lens, and retina, respectively. Furthermore, it may be desirable that there be little to no parallax deviation when two point light sources (or objects) at different distances from the user's eye rotate strictly around the opening of the pinhole (e.g., rotated along a radius of curvature equal to their respective distances from the opening of the pinhole). Thus, it would be considered that the CoP should be located at the center of the pupil of the eye (and such a CoP may be used in some embodiments). However, the human eye includes the cornea, in addition to the crystalline lens and the pinhole of the pupil, which imparts additional refractive power to the light propagating toward the retina. Therefore, the anatomical equivalent of the pinhole in the theoretical system described in this paragraph may be a region of the user's eye located between the outer surface of the cornea and the center of the pupil or iris of the user's eye. For example, the anatomical equivalent of the pinhole may correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP in such a location within the anterior chamber of the user's eye.

[0370] As discussed above, the eye tracking module 614 may provide data such as the estimated 3D positions of the left and right eye rotation centers (CoR), convergence and divergence depth, left and right eye optical axes, 3D position of the user's eye, 3D positions of the left and right centers of the user's corneal curvature, 3D positions of the user's left and right pupil centers, 3D positions of the user's left and right viewpoint centers, and the user's IPD to other components in the wearable system, such as the light field rendering controller 618 and the alignment observer 620. The eye tracking module 614 may also include other submodules that detect and generate data associated with other aspects of the user's eye. In an embodiment, the eye tracking module 614 may include a blink detection module that provides a flag or other alert each time the user blinks, and a saccade detection module that provides a flag or other alert each time the user's eye saccades (e.g., rapidly shifts focus to another point).

[0371] Other methods for determining eye tracking and rotation centers are also possible. Therefore, the eye tracking module 614 may vary. In the various implementations of the eye tracking module described below, for example, the estimated rotation center is determined based on the center values ​​of multiple corneal curvatures. In some implementations, for example, as discussed with reference to Figures 17A-19D, the eye tracking module 614 may estimate the rotation center of the eye by determining the convergence or intersection of surface normal vectors of surfaces fitted to the centers of multiple corneal curvatures for different eye poses. Note that one or more features from the eye tracking module 614 described above or anywhere in this specification may be included in other implementations of the eye tracking module. H. Rendering Controller Example

[0372] A detailed block diagram of an exemplary light field rendering controller 618 is shown in Figure 7B. As shown in Figures 6 and 7B, the rendering controller 618 may receive eye tracking information from the eye tracking module 614 and provide output to the rendering engine 622, which may generate an image to be displayed for viewing by the user of the wearable system. In an embodiment, the rendering controller 618 may receive convergence and divergence depth, left and right eye rotation centers (and / or viewpoint centers), and other eye data such as blink data and saccade data.

[0373] The depth plane selection module 750 may receive convergence-divergence motion depth information and other eye data, and based on such data, may cause the rendering engine 622 to deliver content to the user with a specific depth plane (e.g., a specific accommodation or focal length). As discussed in relation to Figure 4, the wearable system may include a plurality of discrete depth planes, each formed by a plurality of waveguides, each transmitting image information with a variable level of wavefront curvature. In some embodiments, the wearable system may include one or more variable depth planes, such as optical elements, that transmit image information with a level of wavefront curvature that varies over time. In these and other embodiments, the depth plane selection module 750 may cause the rendering engine 622 to deliver content to the user at a selected depth, partly based on the user's convergence-divergence motion depth (e.g., instructing the rendering engine 622 to switch the depth plane on the display 220). In at least some embodiments, the depth plane selection module 750 and the rendering engine 622 may render content at different depths and generate and / or provide depth plane selection data to display hardware such as the display 220. The display hardware such as the display 220 may perform electrical depth plane switching in response to the depth plane selection data (which may be control signals) generated and / or provided by modules such as the depth plane selection module 750 and the rendering engine 622.

[0374] Generally, it may be desirable for the depth plane selection module 750 to select a depth plane that matches the user's current convergence-divergence motion depth so that the user is provided with accurate near-far accommodation cues. However, it may also be desirable to switch depth planes in a discreet and unobtrusive manner. For example, it may be desirable to avoid excessive switching between depth planes and / or to switch depth planes at times when the user is unlikely to notice the switch, such as during blinks or eye saccades.

[0375] The hysteresis band intersection detection module 752 can be particularly useful in avoiding excessive switching between depth planes when the user's convergence-divergence motion depth fluctuates at the midpoint or transition point between two depth planes. In particular, module 752 may cause the depth plane selection module 750 to exhibit hysteresis in its selection of depth planes. As an example, module 752 may cause the depth plane selection module 750 to switch from a first more distant depth plane to a second more nearby depth plane only after the user's convergence-divergence motion depth has passed a first threshold. Similarly, module 752 may cause the depth plane selection module 750 (and thus, as indicated on a display such as display 220) to switch to a first more distant depth plane only after the user's convergence-divergence motion depth has passed a second threshold that is further from the user than the first threshold. In the overlapping region between the first threshold and the second threshold, module 750 may cause the depth surface selection module 750 to maintain that either depth surface is currently selected as the selected depth surface, thereby avoiding excessive switching between depth surfaces.

[0376] The eye event detection module 750 may receive other eye data from the eye tracking module 614 in Figure 7A, and may delay several depth plane switchings in the depth plane selection module 750 until an eye event occurs. In one embodiment, the eye event detection module 750 may delay a planned depth plane switching in the depth plane selection module 750 until a user blink is detected, and may receive data from the blink detection component in the eye tracking module 614 indicating that the user is currently blinking, and in response, may cause the depth plane selection module 750 to perform a planned depth plane switching during the blink event (e.g., by instructing module 750 to perform a depth plane switching during the blink event). In at least some embodiments, the wearable system may be able to shift content onto a new depth plane during a blink event so that the user is less likely to perceive the shift. In another embodiment, the eye event detection module 750 may delay a planned depth plane switching until an eye saccade is detected. As discussed in relation to eye blinking, such arrangements can facilitate discrete shifts in the depth plane.

[0377] If desired, the depth plane selection module 750 may delay a planned depth plane switch for a limited time period before performing a depth plane switch, even in the absence of an eye event. Similarly, the depth plane selection module 750 may perform a depth plane switch even in the absence of an eye event when the user's convergence-divergence depth is substantially outside the currently selected depth plane (for example, when the user's convergence-divergence depth exceeds a predetermined threshold that is above the normal threshold for a depth plane switch). These arrangements can help ensure that the eye event detection module 754 does not delay the depth plane switch indefinitely and does not delay the delayed depth plane switch when a large near-far accommodation error exists.

[0378] The rendering camera controller 758 may provide the rendering engine 622 with information indicating the locations of the user's left and right eyes. The rendering engine 622 may then generate content by simulating cameras at the positions of the user's left and right eyes and generating content based on the viewpoint of the simulated cameras. As discussed above, the rendering camera is, possibly, a simulated camera for use in rendering virtual image content from a database of objects in the virtual world. The objects may have a location and orientation relative to the user or wearer, and possibly to real objects in the environment surrounding the user or wearer. The rendering camera may be included in the rendering engine and render virtual images based on a database of virtual objects to be presented to the eyes. The virtual images may be rendered as if they were taken from the viewpoint of the user or wearer. For example, the virtual images may be rendered as if they were captured by a camera (corresponding to the “rendering camera”) having an aperture, lens, and detector that views objects in the virtual world. The virtual images are taken from the viewpoint of such a camera having the location of the “rendering camera”. For example, the virtual image may be rendered as if captured from a camera viewpoint having a specific location relative to the user's or wearer's eyes, so as to provide an image that appears as if it were from the user's or wearer's viewpoint. In some implementations, the image is rendered as if captured from a camera viewpoint having an aperture at a specific location relative to the user's or wearer's eyes (such as the viewpoint center, rotation center, or other location as discussed herein).

[0379] The rendering camera controller 758 may determine the positions of the left and right cameras based on the left and right eye rotation centers (CoR) determined by the CoR estimation module 724 and / or the left and right eye viewpoint centers (CoP) determined by the CoP estimation module 732. In some embodiments, the rendering camera controller 758 may switch between CoR locations and CoP locations based on various factors. For example, the rendering camera controller 758 may, in various modes, permanently align the rendering camera to the CoR location, permanently align the rendering camera to the CoP location, toggle or discretely switch between aligning the rendering camera to the CoR location and aligning the rendering camera to the CoP location over time based on various factors, or dynamically align the rendering camera to any range of different positions along the optical (or visual) axis between the CoR location and the CoP location over time based on various factors. The CoR and CoP locations may optionally pass through a smoothing filter 756 (in any of the aforementioned modes for rendering camera positioning), which averages the CoR and CoP locations over time, reduces noise at these locations, and prevents jitter when rendering the simulated rendering camera.

[0380] In at least some embodiments, the rendering camera may be simulated as a pinhole camera with a pinhole positioned at the estimated CoR or CoP location identified by the eye-tracking module 614. Since the CoP is offset from the CoR, when the rendering camera's position is based on the user's CoP, the location of both the rendering camera and its pinhole shifts as the user's eye rotates. In contrast, when the rendering camera's position is based on the user's CoR, the location of the rendering camera's pinhole does not move with eye rotation, but the rendering camera (behind the pinhole) may move with eye rotation in some embodiments. In other embodiments where the rendering camera's position is based on the user's CoR, the rendering camera does not have to move (i.e., rotate) with the user's eye. I. Examples of differences between the optical axis and the visual axis

[0381] As discussed in relation to the optical axis / visual axis mapping module 730 in Figure 7A, the user's optical axis and visual axis are generally not aligned, partly because the user's visual axis is defined by its fovea, which is generally not at the center of the person's retina. Therefore, when a person desires to focus on a particular object, the person aligns its visual axis with that object, ensuring that light from the object strikes its fovea, while its optical axis (defined by its pupillary center and the center of its corneal curvature) is actually slightly offset from the object. Figure 8 is an embodiment of an eye 900 illustrating the optical axis 902 of the eye, the visual axis 904 of the eye, and the offset between these axes. In addition, Figure 8 illustrates the pupillary center 906 of the eye, the center of the corneal curvature 908 of the eye, and the mean center of rotation (CoR) 910 of the eye. In at least some populations, the center of corneal curvature of the eye 908 may be approximately 4.7 mm in front of the mean center of rotation (CoR) 910 of the eye, as indicated by dimension 912. In addition, the center of gaze 914 of the eye may be approximately 5.01 mm in front of the center of corneal curvature 908, approximately 2.97 mm behind the outer surface of the user's cornea 916, and / or directly in front of the center of the user's pupil 906 (e.g., corresponding to a location within the anterior chamber of the eye 900). In additional embodiments, dimension 912 may be 3.0 mm to 7.0 mm, 4.0 to 6.0 mm, 4.5 to 5.0 mm, or 4.6 to 4.8 mm, or any range between any value within any of these ranges. The center of gaze (CoP) 914 of the eye may be a useful location for wearable systems, as aligning the rendering camera to the CoP may help reduce or eliminate parallax artifacts in at least some embodiments.

[0382] Figure 8 also illustrates such a location within a human eye 900 where the rendering camera pinhole may be aligned. As shown in Figure 8, the rendering camera pinhole may be aligned with a location 914 along the optical axis 902 or visual axis 904 of the human eye 900, closer to the outer surface of the cornea than both (a) the center of the pupil or iris 906 and (b) the center of corneal curvature 908. For example, as shown in Figure 8, the rendering camera pinhole may be aligned with a location 914 along the optical axis 902 of the human eye 900, which is approximately 2.97 mm posterior to the outer surface of the cornea 916 and approximately 5.01 mm anterior to the center of corneal curvature 908. The location 914 of the rendering camera pinhole and / or the anatomical region of the human eye 900 to which location 914 corresponds may be considered to represent the viewpoint center of the human eye 900. As shown in Figure 8, the optical axis 902 of a human eye 900 represents the shortest line passing through the center of the corneal curvature 908 and the center of the pupil or iris 906. The visual axis 904 of a human eye 900 differs from the optical axis 902 because it represents a line extending from the fovea of ​​the human eye 900 to the center of the pupil or iris 906. J. Exemplary Wearable Device Configurations

[0383] Figures 9A-E illustrate exemplary configurations of components of an exemplary wearable device 2000 for capturing eye image data for use by an eye tracking module 614. For example, as shown in Figure 9A, the wearable device 2000 may be part of a wearable system as described above with reference to Figure 2-4. The wearable device 2000 may include a left eyepiece 2010A and a right eyepiece 2010B. The left eyepiece 2010A may be capable of imaging the user's left eye, and the right eyepiece 2010B may be capable of imaging the user's right eye.

[0384] As shown in Figure 9B, the left eyepiece 2010A may contain one or more illumination sources 2022. Similarly, the right eyepiece 2010B may contain one or more illumination sources 2024. For example, there may be four illumination sources 2022 and four illumination sources 2024. The illumination sources 2022 may be positioned within the left eyepiece 2010A and emit light toward the user's left eye 2012A. The illumination sources 2022 may be positioned so as not to obstruct the user's view through the left eyepiece 2010A. For example, the illumination sources 2022 may be positioned around the edge of the display within the left eyepiece 2010A so as not to obstruct the user's view through the display. Similarly, the illumination sources 2024 may be positioned within the right eyepiece 2010B and emit light toward the user's right eye 2012B. Illuminator 2024 may be positioned so as not to obstruct the user's view through the right eyepiece 2010B. For example, illuminator 20204 may be positioned around the edge of the display within the right eyepiece 2010B so as not to obstruct the user's view through the display. Illuminators 2022 and 2024 may emit light in the visible or invisible light spectrum. For example, illuminators 2022 and 2024 may be infrared (IR) LEDs. Illuminators may also be positioned or configured differently.

[0385] As illustrated in Figure 9B, the left eyepiece 2010A may include a left eye imaging system. The left eye imaging system may include one or more inward-facing cameras (2014, 2016). For example, the left eye imaging system may include a left eye tracking camera 2014 for the left eyepiece 2010A and a right eye tracking camera 2016 for the left eyepiece 2010A. The left eye tracking camera 2014 and the right eye tracking camera 2016 may be located to the left and right of each other, and possibly to the left and right of the center of the left eyepiece, respectively. Similarly, the right eyepiece 2010B may include a right eye imaging system. The right eye imaging system may include one or more inward-facing cameras (2018, 2020). For example, the right eye imaging system may include a left eye tracking camera 2018 for the right eyepiece 2010B and a right eye tracking camera 2020 for the right eyepiece 2010B. The left eye tracking camera 2018 and the right eye tracking camera 2020 may be positioned to the left and right of each other, and possibly to the left and right of the center of the right eyepiece, respectively. One or more cameras in the left eye tracking system 2010A and one or more cameras in the right eye tracking system 2010B may be placed within the wearable device 2000 so as not to interfere with capturing images of the user's eyes. Other configurations are also possible.

[0386] The field of view of the imaging system for the left eyepiece 2010A may be capable of imaging all or a useful portion of the user's left eye 2012A at many different eye positioning positions (not necessarily imaging the right eye or part thereof that is useful for eye tracking). Similarly, the field of view of the imaging system for the right eyepiece 2010B may be capable of imaging all or a useful portion of the user's right eye 2012B at many different eye positioning positions (not necessarily imaging the left eye or part thereof that is useful for eye tracking). For example, a user may be able to move their eye in any direction up to 50 degrees from the central line of sight during normal movement. The imaging system may be collectively configured to image substantially the entire full range of motion (e.g., 50 degrees) of the user's eye during its normal movement.

[0387] Figure 9C illustrates the exemplary field of view 2030 of the right eye tracking camera 2016 with the left eyepiece 2010A and the exemplary field of view 2032 of the right eye tracking camera 2020 with the right eyepiece 2010B. Figure 9D illustrates the exemplary field of view 2040 of the right eye tracking camera 2014 with the left eyepiece 2010A and the exemplary field of view 2042 of the right eye tracking camera 2018 with the right eyepiece 2010B. Figure 9E illustrates how the fields of view 2030 and 2040 from the left eye tracking camera 2014 and the right eye tracking camera 2016 of the left eyepiece 2010A can overlap, respectively, to image substantially all of the user's left eye 2012A. In addition, Figure 9E illustrates how the fields of view 2040 and 2042 from the left eye tracking camera 2018 and the right eye tracking camera 2020 of the right eyepiece 2010B may overlap, respectively, to image substantially the entirety of the image user's right eye 2012B. Variations are also possible. For example, the number and location of cameras may differ. Other types of imaging systems may also be used. K. An example of locating the center of rotation using an eye-tracking system.

[0388] To simplify the eye tracking system (or the process within the eye tracking module 614), it may be desirable to reduce the number of variables required to determine the center of rotation (CoR) of the human eye. Advantageously, reducing the number of variables used to determine the CoR can also improve eye tracking accuracy. For example, since the CoR may be used to determine the line-of-sight vector for use in eye tracking, an increased error in the CoR can result in less accurate eye tracking. Errors in the CoR can arise from errors introduced during the determination of the variables used to calculate the CoR. For example, the CoR calculation may involve steps to extract the pupillary center and model the corneal sphere. Both of these processes can introduce errors and contribute to inaccuracy. Therefore, it may be advantageous to extract the CoR using a limited number of variables.

[0389] Described herein are systems and methods for extracting CoR from corneal data, primarily or entirely. Advantageously, for reasons similar to those discussed above, the system can improve the accuracy of eye-tracking systems. For example, the system may require certain assumptions and thus reduce the potential for introducing errors. In addition to, or as an alternative to, improved accuracy, the system can improve other aspects of eye-tracking systems. For example, the system may rely on shorter eye exposures to the light source. Shorter eye exposures, associated with longer eye exposures to the light source, can reduce risks, lower lighting power consumption, and provide a higher ambient light blocking rate. In another embodiment, the system may not require a wide field of view. Reduced field of view requirements can allow for greater flexibility in the hardware design of the wearable system.

[0390] In some embodiments, the center of rotation (CoR) of a human eye can be extracted from corneal data. Figure 10 shows a graphic diagram of an exemplary CoR extraction system 1000 that may be implemented by an eye tracking module 614. For example, a wearable system may use an illumination system comprising one or more light sources to generate two or more flashes on the cornea 1020 of the user's eye 1010. In various implementations, the illumination system comprises multiple separate regions from which light is emitted. These regions may correspond to isolated light emitters or light sources. A flash detection and labeling module 714 may extract the flash locations on the cornea 1020 of the eye 1010. As described below, a 3D corneal center estimation module 716 may determine an approximate corneal curvature 1018 based on the flash locations and calculate the estimated center 1012 of the approximated corneal curvature 1018. Different eye orientations may provide associated estimated centers for different approximated corneal curvatures 1018 and 1012. The CoR estimation module 724 may determine an estimated CoR based on multiple estimated centers of corneal curvature 1012 by applying a surface to the estimated center 1012 and determining a region 1016 of convergence or intersection of a set of surface normal vectors 1014 normal to the surface. The estimated CoR may be derived from this region 1016, for example, the estimated CoR may be in or within this region 1016.

[0391] Optionally, the CoR estimation may be further checked using the eye tracking module 614. For example, if the CoR has moved relative to the wearable device during use, as described in more detail below, a new measurement of the corneal center 1012 may be tested by measuring the distance between the newly calculated corneal center 1012 and a surface fitted to the calculated set of corneal centers 1012. If the distance is too large, the eye tracking module 614 may pause eye tracking or switch to a different method or a different eye tracking method for determining the CoR. In some embodiments, the switching may be temporary until sufficient data is collected to reduce the overall error.

[0392] Advantageously, CoR extraction 1000 may make one or more assumptions. For example, CoR extraction 1000 may assume that flash extraction is accurate, the geometry of the eye 1010 is known, the corneal radius (or two radii in the case of corneal astigmatism) is known, or that the data were collected during normal or random movement of the user's gaze. L. Exemplary eye-tracking environment

[0393] As discussed above, the CoR may be determined from a plurality of estimated centers of corneal curvature 1012. For example, a surface may be fitted to an estimated center of corneal curvature 1012, and a plurality of surface normal vectors 1014 normal to this surface may be obtained. A region 1016 of convergence of these sets of surface normal vectors 1014 may be identified. The estimated CoR may be obtained from this convergence region 1016, for example, the estimated CoR may be in or within this region 1016.

[0394] To obtain multiple estimated centers of corneal curvature 1012, flashes may be produced on the eye using an illumination source and imaged by a camera as described above. Figure 11 shows an exemplary image of flashes on an arbitrary eye used by an eye-tracking module to determine estimated centers of rotation. For example, as discussed above, the wearable system may include an imaging system. The imaging system may image the user's eye 1110 and produce an eye image 1101. The wearable system may include one or more illumination sources 1102 having spatially distinct regions that emit light. Thus, light from the illumination sources 1102 may produce one or more flashes 1104 on the user's eye 1110, which are reflections of these spatially distinct regions that emit light.

[0395] The imaging system of a wearable system may be part of an eye-tracking assembly (for example, as shown in Figures 9A-E). The imaging system may include one or more cameras. For example, the imaging system may include a single camera at a location 1106 associated with the user's eye 1110. In another embodiment, the imaging system may include multiple cameras that may be located at different locations associated with the user's eye 1110.

[0396] The illumination source 1102 may include one or more light sources, such as light-emitting diodes (LEDs). The illumination source may emit light within the visible or invisible light spectrum (e.g., infrared (IR) light). For example, the illumination source 1102 may be an infrared (IR) LED. The illumination source 1102 may be part of an eye tracking assembly (e.g., as shown in Figures 9A-E).

[0397] The illumination source 1102 may produce one or more specular reflections 1104 on the cornea of ​​the user's eye 1110. The specular reflections 1104 may also be referred to as flashes. For example, there may be two illumination sources (1102A, 1102B). The illumination source may be configured to produce two or more discrete flashes (1104A, 1104B) on the user's eye 1110. Figure 11 shows an image of the user's eye with flashes on it. Figure 11 also shows a diagram of the camera 1106 (represented by the origin of the coordinate system) in comparison to the locations of the eye 1110 and the flashes 1104A, 1104B on it, and relative to the illumination sources 1102A, 1102B at their relative locations. Exemplary extraction of a vector along which the corneal center is located, using a single camera.

[0398] As discussed above, a camera at location 1106 may image flashes 1104A, 1104B on the user's eye 1110, produced by illumination sources 1102A, 1102B. In Figures 12A-D, a first plane 1220 can be determined, including the location of flash 1104A, the camera capturing an image of the flash at location 1106, and the illumination source 1102A producing the flash. In particular, the module may determine a first plane 1220 including the first illumination source 1102A and the first flash 1104A. Similarly, as shown in Figures 13A-D, a second plane 1320 can be determined, including the location of flash 1104B, the camera capturing an image of the flash at location 1106, and the illumination source 1102B producing the flash. In particular, module 716 may determine the second plane 1320 based on the second illumination source 1102B and the second flash 1104B. As shown in Figures 14A-C, module 716 may determine the intersection between the first plane 1220 and the second plane 1320. The intersection between the first plane 1220 and the second plane 1320 may define a vector 1410 that is directed along the location where the corneal center is located. As shown in Figures 14A-C, this vector 1410 may also extend along a direction including the camera location 1106.

[0399] In some implementations, module 716 may determine the first plane 1220 by determining a set of lines 1210, 1212, and 1214 between the first illumination source 1102A, the first flash 1104A, and the camera location 1106. As shown in Figure 12A, module 716 may determine a first line 1210 extending between the camera location 1106 and the location in the image plane 1101A of the first flash 1104A that can be produced by the first illumination source 1102A. As shown in Figure 12B, module 716 may determine a second line 1212 extending between the camera location 1106 and the location of the illumination source 1102A that produced the first flash 1104A. As shown in Figure 12C, module 716 may determine a third line 1214 projected between a location in the image plane 1101A of the illumination source 1102A and the first flash 1104A. As shown in Figure 12D, any two of these lines 1210, 1210, and 1214 may define a plane 1220 in which the corneal center may be located.

[0400] Similarly, in some implementations, module 716 may determine the second plane 1320 by determining a set of lines 1310, 1312, and 1314 between the second illumination source 1102B, the second flash 1104B, and the camera location 1106. As shown in Figure 13A, module 716 may determine a first line 1310 extending between the camera location 1106 and the location in the image plane 1101A of the second flash 1104B that can be produced by the second illumination source 1102B. As shown in Figure 13B, module 716 may determine a second line 1313 extending between the camera location 1106 and the location of the second illumination source 1102B that produced the second flash 1104A. As shown in Figure 13C, module 716 may determine a third line 1314 extending between the location of the second flash 1104B in the image plane 1101A and the second illumination source 1102B. As shown in Figure 13D, lines 1310, 1310, and 1314 may define a plane 1320 in which the corneal center may be located.

[0401] However, in some implementations, the first plane 1220 can be determined directly from the locations of the first illumination source 1102A and the first flash 1104A and the camera location 1106, without necessarily defining the lines 1210, 1210, and 1214 separately. Similarly, the second plane 1320 can be determined directly from the locations of the second illumination source 1102B and the second flash 1104B and the camera location 1106, without necessarily defining the lines 1310, 1310, and 1314 separately.

[0402] Module 716 may identify the intersection between the first and second planes 1220 and 1320. As shown in Figures 14A and 14B, the intersection of the first plane 1220 and the second plane 1320 may define a vector 1410 that is associated with the origin of camera location 1106, or otherwise extends along a direction that may include the camera location. As shown in Figure 14C, vector 1410 may point toward the corneal center location.

[0403] Module 716 may repeat the estimation process multiple times to generate one or more corneal vectors 1410. For example, Module 716 may determine a first plane 1220, which it uses to define vectors based on a first illumination source 1102A and a first flash 1104A, with a number of different camera locations 1106. The camera locations 1106 can vary in relation to the user's eye 1110 (e.g., distance to the user's eye 1110, or horizontal or vertical position relative to the eye, or any combination thereof) or with respect to the location of the illumination source (1102A, 1102B). Module 716 may determine vectors 1410 with respect to one or more of the camera locations 1106. Module 716 may then determine the corneal center from the intersection of two or more vectors, as described above. If two or more vectors do not intersect, the corneal center may be interpolated or otherwise extrapolated from the vector data. In addition, or alternatively, the eye tracking module 614 may collect and analyze more data and determine the corneal center.

[0404] Module 716 may repeat the estimation process while varying one or more parameters associated with the eye-tracking environment 1100. For example, Module 716 may repeat the process using different camera locations or with respect to different gaze directions of the user's eyes. Eye-tracking module 614 may use gaze targets to ensure that the user maintains their eye posture while the parameters are varied. For example, eye-tracking module 614 may estimate one or more vectors 1410 while the user directs their gaze towards gaze targets while varying parameters such as the location of the camera 1106 or the location of the light source 1102. In addition, or alternatively, eye-tracking module 614 may estimate one or more vectors 1410 while the user naturally moves their gaze while using the wearable device. For example, eye-tracking module 614 may capture data associated with different parameters during the natural movement of the user's eyes.

[0405] The repeated estimation process may yield multiple vectors 1410, each pointed towards a corneal center associated with a specific eye posture. Module 716 may determine the region of intersection or convergence of the multiple vectors 1410 to generate an estimated center of corneal curvature. N. Exemplary extraction of a vector along which the corneal center is located, using multiple cameras.

[0406] In various implementations, multiple cameras may be employed to image the eye, and images from multiple cameras may be used to determine the center of the corneal curvature of that eye. In particular, module 716 may determine a vector (1510, 1530) along which the corneal center may be located. Figures 15A-16C illustrate steps in an exemplary process for determining such a vector using multiple cameras. For example, as shown in Figure 15A, a first camera at a first location 1506 may image flashes 1504A, 1504B on the user's eye 1501 produced by illumination sources 1502A, 1502B, and a second camera at location 1526 may image flashes 1524A, 1524B on the user's eye 1501 produced by illumination sources 1522A, 1522B. Module 716 may determine a first vector 1510 based on data associated with a first camera and illumination source 1502A, 1502B at location 1506, and may determine a second vector 1530 associated with a second camera and illumination source 1522A, 1522B at location 1526. As shown in Figure 15B, module 716 may estimate the corneal center 1520 by determining the convergence or intersection between the first vector 1510 and the second vector 1530.

[0407] To obtain the first vector 1510, module 716 may identify a first plane 1512 by determining a set of lines (not shown) between a first illumination source 1502A, a first flash location 1504A in the image plane 1503A, and a first camera at the first location 1506. Module 716 may determine a second plane 1514 by determining a set of lines (not shown) between a second illumination source 1502B, a second flash location 1504B in the image plane 1503A, and a second camera location 1506. Module 716 may determine the vector 1510 by determining the intersection of these first and second planes 1512 and 1514. The intersection of these planes 1512 and 1514 can define the vector 1510, with an origin at camera location 1506 that points toward the center location of corneal curvature.

[0408] However, in some implementations, the first plane 1512 can be determined directly from the locations of the first illumination source 1502A, the first flash 1504A, and the first camera 1106, without necessarily defining one or more lines separately. Similarly, the second plane 1514 can be determined directly from the locations of the second illumination source 1502B, the second flash 1504B, and the first camera 1506, without necessarily defining one or more lines separately.

[0409] Module 716 may similarly determine a first plane 1532 by determining a set of lines (not shown) between a first illumination source 1522A, a first flash location 1524A in the image plane 1503B, and a first camera at location 1526. Module 716 may determine a second plane 1534 by determining a set of lines (not shown) between a second illumination source 1522B, a second flash location 1524B in the image plane 1503B, and a camera location 1526. Module 716 may determine a second vector 1530 by determining the intersection of these first and second planes 1532 and 1534. The intersection of planes 1532 and 1534 can define a vector 1530 with its origin at camera location 1526, which may point toward the center location of corneal curvature. However, in some implementations, the first plane 1532 can be determined directly from the locations of the first illumination source 1522A, the first flash 1524A, and the second camera 1526, without necessarily defining one or more lines separately. Similarly, the second plane 1534 can be determined directly from the locations of the second illumination source 1522B, the second flash 1524B, and the second camera 1526, without necessarily defining one or more lines separately.

[0410] As illustrated in Figure 15B, module 716 may determine the center location of corneal curvature based on these first and second vectors 1510 and 1530. For example, module 716 may determine the convergence or intersection 1520 of these vectors 1510 and 1530. The convergence or intersection 1520 may correspond to an approximate corneal center location. If vectors 1510 and 1530 do not intersect, the center of corneal curvature may be interpolated or otherwise extrapolated from the vector data. In addition, or alternatively, eye tracking module 614 may collect and analyze more data to determine the center of corneal curvature 1520.

[0411] Figures 16A–16C illustrate another exemplary process for determining the center of corneal curvature using multiple cameras. As shown in Figure 16A, the wearable system may have a set of shared illumination sources 1602A, 1602B, which can be used in conjunction with multiple eye cameras. The shared illumination sources 1602A, 1602B may also be a separate set of illumination sources associated with one or more cameras, in addition to or alternatively. The set of shared illumination sources 1602A, 1602B may produce flashes 1604A, 1604B, 1604C, 1604D on the user's eye.

[0412] As shown in Figure 16B, module 716 may use shared illumination sources 1602A and 1602B to determine a set of planes. For example, module 716 may determine a first plane 1630 by determining a set of lines (not shown) between a first illumination source 1602A, a first flash location 1604A in the image plane 1503A, and a first camera at location 1506. Module 716 may determine a second plane 1632 by determining a set of lines (not shown) between a second illumination source 1602B, a second flash location 1604B in the first image plane 1503A, and a first camera location 1506.

[0413] However, in some implementations, the first plane 1630 can be determined directly from the locations of the first illumination source 1602A, the first flash 1604A in the first image plane 1503A, and the first camera 1506, without necessarily defining one or more lines separately. Similarly, the second plane 1632 can be determined directly from the locations of the second illumination source 1602B, the second flash 1604B, and the first camera 1506, without necessarily defining one or more lines separately.

[0414] Module 716 may determine a different first plane 1634 by determining a set of lines (not shown) between a first illumination source 1602A, a first flash location 1604C in the image plane 1503B, and a second camera at location 1526. Module 716 may determine a separate different plane 1636 by determining a set of lines (not shown) between a second illumination source 1602B, a second flash location 1604D in the second image plane 1503B, and a second camera location 1526.

[0415] However, in some implementations, a different first plane 1634 can be determined directly from the locations of the first illumination source 1602A, the first flash 1604C in the image plane 1503B, and the second camera 1526, without necessarily defining one or more lines separately. Similarly, a different second plane 1636 can be determined directly from the locations of the second illumination source 1602B, the second flash 1604D, and the second camera 1526, without necessarily defining one or more lines separately.

[0416] As shown in Figure 16C, module 614 may determine the intersection of planes 1630 and 1632 and determine vector 1610. The intersection of planes 1630 and 1632 may define vector 1610, with its origin at camera location 1506, which can point toward the corneal center location. Similarly, module 614 may determine the intersection of planes 1634 and 1636 and determine vector 1630. The intersection of planes 1634 and 1636 may define vector 1630, with its origin at camera location 1526, which can point toward the corneal center location.

[0417] Continuing with Figure 16C, module 716 may determine the center location of corneal curvature based on vectors 1610 and 1630. For example, module 716 may determine the convergence or intersection 1620 of the first and second vectors 1610 and 1630. The convergence or intersection 1620 may correspond to the approximate center location of corneal curvature. If the first and second vectors 1610 and 1630 do not intersect, the center of corneal curvature may be interpolated or otherwise extrapolated from the vector data. In addition, or alternatively, eye tracking module 614 may collect and analyze more data to determine the center of corneal curvature.

[0418] Module 716 may repeat the estimation process for multiple gaze directions of the user's eyes. For example, the wearable system may display one or more gaze targets to which the user can direct their gaze. The eye tracking module 614 may estimate one or more vectors 1410 while the user directs their gaze to the gaze targets. In addition, or alternatively, the eye tracking module 614 may estimate one or more vectors 1410 while the user naturally moves their gaze while using the wearable device. For example, the eye tracking module 614 may capture data associated with different parameters during the natural movement of the user's eyes. As described below, the data captured at different eye poses or gaze vectors of the user's eyes may be used to calculate multiple corneal centers, which may be used by the CoR estimation module 724 to estimate the CoR. O. Estimation of the center of rotation

[0419] The Center of Rotation (CoR) estimation module 724 may determine the estimated center of rotation based on the estimated center of the corneal curvature 10¹². For example, the CoR estimation module 724 may fit a surface to one or more estimated corneal centers of curvature and determine a set of surface normal vectors in the direction normal to the fitted surface. The surface normal vectors may converge or intersect at points or regions that may correspond to the estimated CoR.

[0420] To determine the surface, module 614 may analyze multiple eye images. For example, the wearable system may image the user's eye 1501 while it is in one or more eye postures (e.g., using an inwardly facing imaging system 462). In some implementations, module 614 may prompt for one or more eye postures or gaze directions through the display of gaze targets on the wearable device's display. In addition, or alternatively, module 614 may collect data associated with one or more eye postures that occur naturally during use of the wearable device.

[0421] As illustrated in Figures 17A and 17B, module 614 may determine multiple corneal centers of curvature 1712 based on data collected by the wearable system while the user's eye is in one or more eye positions. For example, module 614 may perform the corneal curvature center estimation process multiple times (e.g., with respect to different gaze directions or eye positions of the user's eye 1501) using one or more cameras as part of module 716, as described above. The output of the corneal center estimation process of module 716 may include multiple estimated corneal centers of curvature 1712.

[0422] Multiple corneal centers with curvature 1712 can be placed within a region 1710 in three-dimensional (3D) space. Region 1710 may correspond to the corneal sphere 1022. Without endorsement of any particular scientific theory, multiple corneal centers with curvature 1712 can be approximately aligned within region 1710 according to the shape of the corneal curvature 1018. For example, multiple corneal centers with curvature 1712 can be aligned within region 1710 so as to trace a contour approximately parallel to or substantially identical in shape to the shape of the cornea 1020. If the cornea is approximately spherical, multiple corneal centers 1712 can approximately follow the corneal curvature 1018 at a distance approximately corresponding to the radius of the cornea. In the case of astigmatism (or if the cornea is not approximately spherical), multiple corneal centers 1712 can approximately follow the corneal curvature 1018 at a distance approximately corresponding to one or more radii of the corneal geometry.

[0423] In various implementations, module 614 may determine whether multiple corneal centers 1712 fall within a determined tolerance of the expected distance from the corneal surface 1022 to the center 1022 of the corneal sphere. For example, the corneal sphere 1022 can be spherical or astigmatic (e.g., have a geometric shape other than spherical). The expected distance may correspond to the distance to the center of the geometric shape of the corneal sphere 1022. For example, if the corneal geometry is spherical, the expected distance may be the radius of the corneal sphere 1022. If a corneal center 1712 falls outside the determined tolerance, module 614 may reduce the outlier contribution in further analysis. For example, module 614 may exclude outlier data points from further analysis. In addition, or alternatively, if the threshold number of corneal centers 1712 falls outside the determined tolerance, module 614 may halt the analysis until further data is obtained or a different method for determining the rotation center is switched to.

[0424] As shown in Figure 17B, module 724 may fit a 3D surface 1714 to a plurality of corneal centers 1712. Module 724 may fit the 3D surface using, for example, regression analysis. Module 724 may determine the fit using a preferred surface or curve fitting technique. Module 724 may fit the corneal centers 1712 to a low-order polynomial 3D surface 1714 using, for example, polynomial regression. In another embodiment, module 724 may apply a geometric fit to the corneal centers 1712 (e.g., total least squares fit). In some embodiments, the surface 1714 may have a curvature similar to the corneal curvature 1018. In other embodiments, the surface 1714 may have a shape different from the corneal curvature 1018.

[0425] Module 724 may determine a set of surface normal vectors that are normal to surface 1714. Figure 18A illustrates an exemplary calculation of CoR (or center of the eye "EBC") using surface normal vectors 1814. For example, module 716 may determine a set of estimated corneal centers 1812. Module 724 may fit surface 1714 (as shown in Figure 17B) to the estimated corneal centers 1812. Module 724 may then determine one or more surface normal vectors 1814 that are normal to surface 1714. Surface normal vectors 1814 can arise from estimated centers of corneal curvature 1812. For example, module 724 may determine a surface normal vector 1814 for each estimated center of corneal curvature 1812 used to determine surface 1714. Fewer surface normals may be used in some implementations. In addition, or alternatively, the surface normal vector 1814 may originate from other points on the surface 1714.

[0426] Module 724 may determine the region 1802 of convergence of the surface normal vectors 1814. For example, as shown in the inset figure 1801 of Figure 18A, some or all of the surface normal vectors 1814 may converge or intersect within the region 1802 in 3D space. The region 1802 in 3D space may be the intersection point or the volume in 3D space (e.g., volume 1920 in Figures 19C and 19D) where the normal vectors intersect and / or converge. The volume in 3D space may be centered around the midpoint of the intersection or convergence of the surface normal vectors 1814. The volume in 3D space may be large enough to encompass most of the intersection point.

[0427] The convergence region 1802 may include different areas of convergence or intersection corresponding to different line-of-sight directions or eye orientations. For example, the convergence region 1802 may include a sub-region 1820 corresponding to a first line-of-sight direction (e.g., downward line of sight) and a sub-region 1822 corresponding to a second line-of-sight direction (e.g., upward line of sight). In some embodiments, the sub-regions 1820 and 1822 may correspond to approximate CoRs associated with regions of the display of a wearable device. For example, the first sub-region 1820 may correspond to the upper region of the display, and the second sub-region 1822 may correspond to the lower region of the display.

[0428] Module 724 may determine the CoR by analyzing the convergence region 1802. For example, Module 724 may determine the CoR by determining the mode or median of the convergence or intersections of vector 1814. In addition, or alternatively, Module 724 may first determine line-of-sight based convergence or intersections, such as the mode or median of the convergence or intersections of vector 1814 in sub-regions 1820, 1822, and then determine the CoR by determining the mode or median based on those line-of-sight based convergence or intersections. In addition, or alternatively, Module 724 may perform a different analysis of the convergence or intersections to determine the CoR. For example, Module 724 may use a machine learning algorithm to determine the CoR.

[0429] In some embodiments, variations in the calculated curvature of the corneal center can result in a wider convergence region 1824, as opposed to a single point at the intersection. Figure 18B illustrates an exemplary CoR calculation 1803 using region 1824. For example, a calculated corneal center with curvature 1832 may be noisy with respect to the fitted 3D surface 1830. A noisy corneal center 1832 may result in a region 1824 in which the CoR or eye center (EBC) is likely based on the intersection of vectors (not shown) with origins at the corneal center 1832. In some implementations, module 614 may use region 1824 when calculating the line of sight direction. For example, module 614 may determine the CoR as the center of region 1824 or some other location within, on, or otherwise based on region 1824.

[0430] In various implementations, module 724 may select a portion of the estimated corneal centers 1910 to determine the CoR. Figures 19A-1 and 19A-2 illustrate exemplary surfaces 1912 fitted to a portion of the estimated corneal centers 1910 that may be selected using a data reduction process. Figures 19B-1 and 19B-2 show exemplary vectors 1916 that may be normal to the surface 1912. Vectors 1916 may occur at the selected estimated corneal centers 1910. Figures 19C-1 and 19C-2 illustrate estimated CoR regions 1920 based on points of regional convergence or intersection of vectors 1916. If module 724 does not select a corneal center 1910 to fit to the surface 1912, many of the vectors 1922 may not converge or intersect within the region 1920, as shown in Figures 19D-1 and 19D-2.

[0431] In various implementations, module 724 may select estimated corneal centers 1910 based on a determined region of convergence of normal vectors 1916. For example, module 724 may determine a large region in which normal vectors 1922 intersect. In some implementations, if the large region has a volume exceeding the threshold volume, module 724 may determine a smaller set of corneal centers 1910 and use it to determine the CoR. In some implementations, the threshold volume may include a suitable volume for determining the CoR, which is associated with a threshold accuracy of eye-tracking based on its CoR. For example, a volume of 30 percent of the user's eye volume may be associated with an 80% decrease in accuracy in eye-tracking. If the determined volume exceeds the threshold volume, module 724 may select a smaller set of corneal centers 1910 based on any number of suitable data selection criteria, as described below.

[0432] In addition, or alternatively, module 724 may use any number of data reduction processes, such as machine learning algorithms or filtering processes, to select the estimated corneal centers 1910 for analysis. For example, module 724 may filter the data to remove outliers. The filter may include the step of determining a confidence score associated with a given corneal center 1910 and selecting the corneal center 1910 based on that confidence score. In some embodiments, the confidence score may be determined based on the deviation of the corneal center at curvature 1910 from a quadratic calculation or the determination of the corneal center at curvature 1910 or surface 1912. In some embodiments, the confidence score may be based on the location of the corneal center at curvature 1910 relative to the fitted surface 1912 (e.g., the deviation of the corneal center 1910 from the fitted surface 1912). In some embodiments, the confidence score may be determined based on the error calculated in flash extraction, which is used to determine the corneal center at curvature 1910. For example, flash extraction can have high error if there are errors in the eye image being analyzed for flash extraction (e.g., due to blur, obstacles in the image, distortion, or other noise sources...

Claims

1. A display system configured to display virtual image content within the user's field of view by projecting light onto the user's eyes, wherein the display system is A frame configured to be supported on the user's head, A head-mounted display positioned on the frame, the head-mounted display is configured to display virtual image content in the user's field of view by projecting light into the user's eyes, One or more eye-tracking cameras configured to image the user's eyes, Multiple optical emitters, The head-mounted display and the processing electronic device that communicates with the one or more eye-tracking cameras. Equipped with, The aforementioned processing electronic device is Receiving an image of the user's eye captured by one or more eye-tracking cameras, wherein the flash reflections of the multiple different light emitters are observable in the image of the eye captured by the one or more eye-tracking cameras. The location of the user's eye rotation center is estimated based on the location of the flash reflection in the image generated by both of the one or more eye-tracking cameras, and based on the location of the one or more eye-tracking cameras and the locations of the multiple light emitters that generated the flash reflection with respect to multiple eye poses. It is configured to do the following: Estimating the location of the center of rotation of the eye is Based on multiple flash reflections related to multiple eye postures, multiple estimates of the center of the corneal curvature of the user's eye are determined, Determining an estimated location of the center of rotation of the user's eye based on a plurality of estimated values ​​of the center of corneal curvature of the user's eye for a plurality of eye postures, wherein determining the estimated location of the center of rotation of the user's eye based on a plurality of estimated values ​​of the center of corneal curvature of the user's eye for a plurality of eye postures is done by generating a three-dimensional curvilinear surface associated with the plurality of estimated values ​​of the center of corneal curvature, and determining the estimated location of the center of rotation of the user's eye based on the three-dimensional curvilinear surface. A display system in which the processing is performed by the aforementioned electronic device.

2. In order to determine the plurality of estimates of the corneal curvature of the user's eye, the processing electronic device, A first direction toward the center of the corneal curvature is determined based on the location of at least some of the plurality of light emitters and one of the one or more eye tracking cameras, A second direction toward the center of the corneal curvature is determined based on the location of at least some of the plurality of light emitters and one of the one or more eye tracking cameras, Based on the first and second directions, an estimated value of the center of the corneal curvature of the user's eye is determined. The display system according to claim 1, configured to perform the following:

3. The aforementioned processing electronic device is Defining a first plane that includes one of the one or more eye-tracking cameras, the location of the first flash reflection, and the location of the light emitter corresponding to the first flash reflection, Defining a second plane that includes one of the one or more eye-tracking cameras, the location of the second flash reflection, and the location of the light emitter corresponding to the second flash reflection, The first intersection of the first plane and the second plane is determined such that the first intersection defines a first vector, and the first vector extends along the first direction. The display system according to claim 2, configured to determine the first direction by performing the following.

4. The aforementioned processing electronic device is Defining a third plane that includes one of the one or more eye-tracking cameras, the location of the third flash reflection, and the location of the light emitter corresponding to the third flash reflection, Defining a fourth plane that includes one of the one or more eye-tracking cameras, the location of the fourth flash reflection, and the location of the light emitter corresponding to the fourth flash reflection, Determining a second intersection of the third plane and the fourth plane, wherein the second intersection defines a second vector, and the second vector extends along the second direction. The display system according to claim 3, configured to determine the second direction by performing the following.

5. In order to determine the plurality of estimates of the corneal curvature of the user's eye, the processing electronic device, By determining the convergence region between the first direction and the second direction, an estimate of the center of the corneal curvature of the user's eye is determined. The display system according to claim 2, configured to perform the following:

6. In order to determine the plurality of estimates of the corneal curvature of the user's eye, the processing electronic device, Based on the location of at least some of the plurality of light emitters and the first camera among the one or more eye tracking cameras, a first direction toward the center of the corneal curvature is determined, Based on the location of at least some of the plurality of light emitters and the second camera among the one or more eye tracking cameras, a second direction toward the center of the corneal curvature is determined, Based on the first and second directions, an estimated value of the center of the corneal curvature of the user's eye is determined. The display system according to claim 1, configured to perform the following:

7. The aforementioned processing electronic device is Defining a first plane that includes the location of the first camera and the first flash reflection and the location of the light emitter corresponding to the first flash reflection, Defining a second plane that includes the locations of the first camera and the second flash reflection and the location of the light emitter corresponding to the second flash reflection, The first intersection of the first plane and the second plane is determined such that the first intersection defines a first vector, and the first vector extends along the first direction. The display system according to claim 6, configured to determine the first direction by performing the following.

8. The aforementioned processing electronic device is Defining a third plane including the location of the second camera and the third flash reflection and the location of the light emitter corresponding to the third flash reflection, Defining a fourth plane including the location of the second camera and the fourth flash reflection and the location of the light emitter corresponding to the fourth flash reflection, Determining a second intersection of the third plane and the fourth plane, wherein the second intersection defines a second vector, and the second vector extends along the second direction. The display system according to claim 7, configured to determine the second direction by performing the following.

9. The display system according to claim 1, wherein the processing electronic equipment is configured to fit the curved surface to the plurality of estimates of the center of corneal curvature in order to generate the three-dimensional curved surface associated with the plurality of estimates of the center of corneal curvature.

10. The display system according to claim 1, wherein the processing electronic equipment is configured to fit a spherical surface to the plurality of estimates of the center of corneal curvature in order to generate the three-dimensional curved surface associated with the plurality of estimates of the center of corneal curvature.

11. In order to determine the estimated location of the center of rotation of the user's eye, the processing electronic device: Determining two or more normals to the three-dimensional curved surface, Determining the convergence region of the two or more normals mentioned above. It is configured to do the following: The display system according to claim 1, wherein the convergence region comprises the estimated location of the center of rotation of the user's eye.

12. The display system according to claim 1, wherein the one or more images of the user's eye include one or more images associated with a plurality of different line-of-sight vectors of the user's eye.

13. The display system according to claim 1, wherein the processing electronic device is configured to map the cornea of ​​the user's eye using a line-of-sight target.

14. The display system according to claim 1, wherein the processing electronic device is configured to render a virtual image to be presented to the user's eyes by using a rendering camera, and the rendering camera has a position determined by the center of rotation.

15. The display system according to claim 1, wherein the head-mounted display is configured to display virtual image content in the user's field of view by projecting light into the user's eyes in at least one different amount of divergence and collimation, so that the displayed virtual image content appears to originate from a plurality of different depths.

Images