Method and system for tracking eye movements in conjunction with an optical scanning projector - Patents.com

The eye tracking system using a light scanning projector and photodetectors with visible and non-visible wavelengths effectively tracks eye movements in augmented reality systems, addressing the need for improved display systems in wearable technology.

JP7581413B2Active Publication Date: 2024-11-12MAGIC LEAP INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023074415
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-03-21
Filing Date
2023-04-28
Publication Date
2024-11-12
Estimated Expiration
2038-03-21

AI Technical Summary

Technical Problem

There is a need for improved methods and systems in augmented reality systems, particularly in display systems, to enhance eye tracking capabilities, especially in conjunction with wearable displays and waveguide displays driven by fiber scanning projectors.

Method used

An eye tracking system is implemented using a light scanning projector coupled with eyepieces and photodetectors, which includes an optical scanning projector, an eyepiece with an exit pupil expander, and a processor to track eye movement by correlating reflected light intensities with pixel positions, utilizing visible and non-visible wavelengths for accurate eye orientation determination.

Benefits of technology

Enables efficient eye tracking with reduced system complexity, allowing for small form factor wearable displays that can adapt to eye movements in real-time, enhancing user interaction in augmented reality environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581413000001
    Figure 0007581413000001
  • Figure 0007581413000002
    Figure 0007581413000002
  • Figure 0007581413000003
    Figure 0007581413000003
Patent Text Reader

Abstract

To provide a suitable method and system for tracking eye movement in conjunction with a light scanning projector.SOLUTION: The present invention generally relates to a method and system associated with a projection display system that includes a wearable display. More specifically, the present invention can provide a method and system for tracking eye movement in association with a light scanning projector system. A method and system for determining eye orientation and motion as a time function (i.e., eye tracking) can be implemented in combination with a waveguide display that is driven by a fiber scanning projector. The present invention can be applied to various uses in a computer vision and an image display system. An eye tracking system includes a light scanning projector and eye lenses optically coupled to the light scanning projector. The eye tracking system also includes an optical sensor and a processor coupled to the light scanning projector and the optical sensor.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 474,497, filed March 21, 2017, and entitled “Method and System for Tracking Eye Movement in Conjunction with a Fiber Scanning Projector,” the disclosure of which is incorporated herein in its entirety for all purposes. [Background technology]

[0002] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a viewer in a manner that appears or may be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information in an augmentation to the visualization of the real world around the viewer.

[0003] Despite the advances made in these display technologies, there remains a need in the art for improved methods and systems relating to augmented reality systems, and in particular display systems. Summary of the Invention [Means for solving the problem]

[0004] The present invention generally relates to methods and systems related to projection display systems, including wearable displays. More specifically, embodiments of the present invention provide methods and systems for tracking eye movement in association with optical scanning projector systems. In certain embodiments, methods and systems for determining eye orientation and movement as a function of time (i.e., eye tracking) are implemented in conjunction with a waveguide display driven by a fiber scanning projector. The present invention is applicable to a variety of applications in computer vision and image display systems.

[0005] According to an embodiment of the present invention, an eye tracking system is provided. The eye tracking system includes a pair of glasses including two frames, and a light scanning projector coupled to the glasses and operable to scan a beam of light. The eye tracking system also includes an eyepiece mounted in one of the two frames and optically coupled to the light scanning projector. The eyepiece includes an exit pupil expander operable to direct at least a portion of the beam of light toward a user's eye. The eye tracking system further includes one or more light detectors coupled to the glasses, and a processor coupled to the light scanning projector and the one or more light detectors.

[0006] According to a specific embodiment of the present invention, an eye tracking system is provided. The eye tracking system includes an optical scanning projector and an eyepiece optically coupled to the optical scanning projector. The eye tracking system also includes an optical sensor and a processor coupled to the optical scanning projector and the optical sensor.

[0007] According to another embodiment of the present invention, there is provided a method of operating an eye tracking system including an optical scanning projector, an eyepiece, and an optical sensor. The method includes generating an optical signal using the optical scanning projector and coupling at least a portion of the optical signal into the eyepiece. The method also includes projecting an image viewable by a user's eye from the eyepiece, detecting light reflected from the user's eye using the optical sensor, and tracking the position of the user's eye.

[0008] According to another specific embodiment of the present invention, a method for tracking eye movement from a first orientation to a second orientation is provided. The method includes scanning a waveguide of a scanning waveguide projector to form a first image frame of a video image. A first series of pixels defines the first image frame. The method also includes coupling the first image frame into an eyepiece lens including a plurality of planar waveguides, transmitting at least a portion of the first image frame to a user's eye, and detecting a first series of reflected intensities, each of which is associated with a pixel of the first series of pixels. The method further includes correlating the first series of detected reflected intensities with the first series of pixels, determining a first orientation of the eye, and scanning the waveguide to form a second image frame of the video image. A second series of pixels defines the second image frame. Additionally, the method includes coupling the second image frame into the eyepiece, transmitting at least a portion of the second image frame to an eye of a user, and detecting a second series of reflected intensities, each of which is associated with a pixel of the second series of pixels. The method further includes correlating the second series of detected reflected intensities with the second series of pixels, and determining a second orientation of the eye.

[0009] According to another specific embodiment of the present invention, a method for characterizing eye orientation is provided. The method includes scanning a fiber of a fiber scanning projector to form a first portion of an image frame of a video image. A first series of pixels defines the first portion of the image frame. The method also includes coupling the first portion of the image frame into an eyepiece including a plurality of planar waveguides, transmitting at least a portion of the first portion of the image frame to an eye of a user, and detecting a temporal reflection signal at each of a plurality of optical detectors. The method further includes correlating the plurality of temporal reflection signals with the first series of pixels, and scanning the fiber to form a second portion of the image frame of the video image. A second series of pixels defines the second portion of the image frame. Additionally, the method includes coupling the second portion of the image frame into the eyepiece and transmitting at least a portion of the second portion of the image frame to an eye of a user. The method further includes detecting a second temporal reflection signal at each of the plurality of optical detectors, correlating the plurality of second temporal reflection signals with a second series of pixels, and determining an eye orientation.

[0010] According to certain embodiments of the present invention, a method for performing eye tracking is provided. The method includes providing an optical scanning projector, an eyepiece mounted in a frame, and a plurality of optical detectors aligned around a periphery of the frame. The method also includes projecting a first image frame onto the eye using the optical scanning projector and the eyepiece, detecting a plurality of time-varying reflection signals using the plurality of optical detectors, and determining a first eye orientation. The method further includes projecting a second image frame onto the eye using the optical scanning projector and the eyepiece, detecting a second plurality of time-varying reflection signals using the plurality of optical detectors, and determining a second eye orientation.

[0011] Numerous benefits over conventional techniques are achieved by the methods of the present invention. For example, embodiments of the present invention provide methods and systems that enable eye tracking in conjunction with a waveguide display having a small form factor. In addition, some embodiments utilize light provided to drive the display as input radiation and track the eyes based on reflections of this display light, thereby reducing system complexity. These and other embodiments of the present invention, along with many of their advantages and features, are described in more detail in conjunction with the following text and accompanying figures. The present specification also provides, for example, the following items: (Item 1) 1. An eye tracking system comprising: An optical scanning projector; an eyepiece optically coupled to the optical scanning projector; An optical sensor; a processor coupled to the optical scanning projector and the optical sensor; An eye tracking system comprising: (Item 2) 2. The eye tracking system of claim 1, wherein the optical scanning projector comprises a scanning point light source. (Item 3) 3. The eye tracking system of claim 2, wherein the optical scanning projector comprises a scanning waveguide projector. (Item 4) 4. The eye tracking system of claim 3, wherein the scanning waveguide projector comprises a fiber scanning projector. (Item 5) 2. The eye tracking system of claim 1, further comprising a pair of glasses having a frame, wherein the optical scanning projector, the eyepiece, and the optical sensor are mounted within the frame. (Item 6) 2. The eye tracking system of claim 1, wherein the eyepiece comprises a plurality of planar waveguides. (Item 7) 7. The eye tracking system of claim 6, wherein the plurality of planar waveguides comprises four planar waveguides. (Item 8) 2. The eye tracking system of claim 1, wherein the optical sensor comprises a photodetector. (Item 9) 2. The eye tracking system of claim 1, wherein the optical scanning projector is operable to output a display signal and a characterization signal. (Item 10) 10. The eye tracking system of claim 9, wherein the display signal includes one or more visible wavelengths and the characterization signal includes one or more invisible wavelengths. (Item 11) Item 11. The eye tracking system of item 10, wherein the one or more visible wavelengths include red, green, and blue wavelengths, and the one or more invisible wavelengths include infrared wavelengths. (Item 12) the eyepiece comprises a plurality of planar waveguides; The plurality of planar waveguides include a first planar waveguide corresponding to red wavelengths; a second planar waveguide corresponding to green wavelengths; and a third planar waveguide corresponding to blue wavelengths; and A fourth planar waveguide for infrared wavelengths; Item 12. The eye tracking system of item 11, comprising: (Item 13) 1. A method of operating an eye tracking system including an optical scanning projector, an eyepiece, and an optical sensor, the method comprising: generating an optical signal using the optical scanning projector; coupling at least a portion of the optical signal into the eyepiece; projecting an image viewable by an eye of a user from the eyepiece; detecting light reflected from an eye of the user using the optical sensor; Tracking the position of the user's eyes; A method comprising: (Item 14) Item 14. The method of item 13, wherein the optical scanning projector comprises a fiber scanning projector. (Item 15) Item 14. The method of item 13, wherein the optical signals include an indication signal and a characterization signal. (Item 16) Item 16. The method of item 15, wherein the indication signal includes one or more visible wavelengths and the characterization signal includes one or more non-visible wavelengths. (Item 17) Item 17. The method of item 16, wherein the one or more visible wavelengths include red, green, and blue wavelengths, and the one or more invisible wavelengths include infrared wavelengths. (Item 18) Item 14. The method of item 13, wherein the eyepiece lens comprises a plurality of planar waveguides. (Item 19) Item 19. The method of item 18, wherein the plurality of planar waveguides comprises four planar waveguides. (Item 20) Item 14. The method of item 13, wherein the optical sensor comprises a photodetector. (Item 21) Item 14. The method of item 13, wherein tracking the position of the user's eye includes correlating a sequence of pixel positions of the image with a sequence of intensities of the light reflected from the user's eye. (Item 22) 1. A method for tracking eye movements from a first orientation to a second orientation, the method comprising: scanning an output beam of an optical scanning projector to form a first image frame of a video image, a first series of pixels defining the first image frame; coupling the first image frame into an eyepiece including a plurality of planar waveguides; transmitting at least a portion of the first image frame to an eye of a user; detecting a first series of reflected intensities, each of the first series being associated with a pixel of the first series of pixels; correlating the first series of reflected intensities with the first series of pixels; determining a first orientation of the eye; scanning the output beam to form a second image frame of the video image, a second series of pixels defining the second image frame; and coupling the second image frame into the eyepiece; transmitting at least a portion of the second image frame to an eye of the user; detecting a second series of reflected intensities, each of the second series being associated with a pixel of the second series of pixels; correlating the second series of reflected intensities with the second series of pixels; determining a second orientation of the eye; and A method comprising: (Item 23) 23. The method of claim 22, further comprising outputting trajectories associated with the first orientation and the second orientation. (Item 24) The plurality of planar waveguides include a first planar waveguide corresponding to red wavelengths; a second planar waveguide corresponding to green wavelengths; and a third planar waveguide corresponding to blue wavelengths; and A fourth planar waveguide for infrared wavelengths; 23. The method according to item 22, comprising: (Item 25) 23. The method of claim 22, wherein the first image frame and the second image frame are consecutive frames of the video image. (Item 26) Item 23. The method of item 22, wherein one or more additional image frames are positioned between the first image frame and the second image frame. (Item 27) 23. The method of claim 22, wherein coupling the first image frame into the eyepiece and coupling the second image frame into the eyepiece includes diffracting the first image frame and the second image frame using a diffractive optical element. (Item 28) 23. The method of claim 22, wherein transmitting at least a portion of the first image frame to the user's eye and transmitting at least a portion of the second image frame to the user's eye comprises diffracting at least a portion of the first image frame and at least a portion of the second image frame using one or more diffractive optical elements. (Item 29) 30. The method of claim 28, wherein the one or more diffractive optical elements comprise a first diffractive optical element operable to diffract light within the plane of the eyepiece and a second diffractive optical element operable to diffract light out of the plane of the eyepiece. (Item 30) 1. A method for characterizing eye orientation, the method comprising: scanning a fiber of a fiber scanning projector to form a first portion of an image frame of a video image, a first series of pixels defining the first portion of the image frame; coupling a first portion of the image frame into an eyepiece including a plurality of planar waveguides; transmitting at least a portion of a first portion of the image frame to an eye of a user; detecting a temporal reflection signal at each of a plurality of optical detectors; correlating the plurality of temporal reflectance signals with the first series of pixels; scanning the fiber to form a second portion of an image frame of the video image, a second series of pixels defining the second portion of the image frame; coupling a second portion of the image frame into the eyepiece; transmitting at least a portion of a second portion of the image frame to an eye of the user; detecting a second temporal reflected signal at each of the plurality of optical detectors; correlating the plurality of second temporal reflectance signals with the second set of pixels; determining said eye orientation; and A method comprising: (Item 31) Item 31. The method of item 30, wherein the plurality of optical detectors comprises a plurality of photodetectors. (Item 32) Item 32. The method of item 31, wherein the plurality of photodetectors are aligned around a peripheral region of the eye. (Item 33) Item 32. The method of item 31, wherein the eyepiece is mounted in a frame having a periphery. (Item 34) Item 34. The method of item 33, wherein the plurality of photodetectors are arranged on a periphery of the frame. (Item 35) Item 34. The method according to item 33, wherein the plurality of photodetectors are disposed within a temple coupled to the frame. (Item 36) The plurality of planar waveguides include a first planar waveguide corresponding to red wavelengths; a second planar waveguide corresponding to green wavelengths; and a third planar waveguide corresponding to blue wavelengths; and A fourth planar waveguide for infrared wavelengths; 31. The method according to item 30, comprising: (Item 37) 1. An eye tracking system comprising: A pair of glasses including two frames; a light scanning projector coupled to the glasses and operable to scan a beam of light; an eyepiece mounted in one of the two frames and optically coupled to the optical scanning projector, the eyepiece including an exit pupil expander operable to direct at least a portion of the beam of light towards an eye of a user; one or more photodetectors coupled to the glasses; a processor coupled to the optical scanning projector and the one or more optical detectors; An eye tracking system comprising: (Item 38) Item 38. The eye tracking system of item 37, wherein the optical scanning projector comprises a fiber scanning projector. (Item 39) Item 38. The eye tracking system of item 37, wherein the glasses include temples, the optical scanning projector is mounted in one of the two frames, and the one or more optical detectors are mounted within the temples. (Item 40) 40. The eye tracking system of claim 39, wherein the one or more optical detectors are mounted in the temple at a vertical position between the eyepiece and the user's eye. (Item 41) Item 38. An eye tracking system as described in item 37, wherein the eyepiece lens comprises a plurality of planar waveguides. (Item 42) Item 38. The eye tracking system of item 37, wherein the optical scanning projector is operable to output a display signal and a characterization signal. (Item 43) 43. The eye tracking system of claim 42, wherein the indication signal comprises one or more visible wavelengths and the characterization signal comprises one or more invisible wavelengths. (Item 44) 1. A method for performing eye tracking, the method comprising: providing an optical scanning projector, an eyepiece mounted within a frame, and one or more optical detectors aligned around a periphery of the frame; projecting a first image frame onto an eye using the optical scanning projector and the eyepiece; detecting a plurality of time-varying reflected signals using the one or more optical detectors; Determining a first eye orientation; projecting a second image frame to the eye using the optical scanning projector and the eyepiece; and detecting a second plurality of time-varying reflected signals using the one or more optical detectors; and Determining a second eye orientation. A method comprising: (Item 45) Item 45. The method of item 44, wherein the one or more optical detectors are positioned at a longitudinal position between the eyepiece and the eye. (Item 46) Item 46. The method of item 45, wherein the one or more optical detectors are mounted in a temple that is mechanically coupled to the frame. (Item 47) Item 45. The method of item 44, further comprising outputting trajectories associated with the first eye orientation and the second eye orientation. (Item 48) Item 45. The method of item 44, wherein the plurality of time-varying reflected signals are associated with increasing scan angles of the optical scanning projector during the first image frame. (Item 49) Item 45. The method of item 44, wherein the second plurality of time-varying reflected signals are associated with increasing scan angles of the optical scanning projector during the second image frame. (Item 50) Item 45. The method of item 44, wherein the eyepiece lens comprises a plurality of planar waveguides. (Item 51) Projecting the first image frame to the eye includes: coupling a portion of the first image frame into each of the plurality of planar waveguides; transmitting at least a portion of the conjugated moiety into the eye; and 51. The method according to item 50, comprising: (Item 52) Projecting the second image frame to the eye includes: coupling a portion of the second image frame into each of the plurality of planar waveguides; transmitting at least a portion of the conjugated moiety into the eye; and 51. The method according to item 50, comprising: (Item 53) Item 45. The method of item 44, wherein the first image frame and the second image frame are consecutive frames of a video image. (Item 54) Item 45. The method of item 44, wherein one or more additional image frames are positioned between the first image frame and the second image frame. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 diagrammatically illustrates light paths in a viewing optical assembly (VOA) that may be used to present a digital or virtual image to a viewer, according to one embodiment of the present invention.

[0013] [Diagram 2] FIG. 2 is a simplified schematic diagram illustrating eyeglasses incorporating an eye tracking system according to one embodiment of the present invention.

[0014] [Diagram 3] FIG. 3 is a simplified diagram illustrating several spectral profiles according to one embodiment of the present invention.

[0015] [Figure 4] FIG. 4 is a simplified side view illustrating an eyepiece lens layer according to one embodiment of the present invention.

[0016] [Diagram 5] FIG. 5 is a simplified side view illustrating an eyepiece lens layer according to another embodiment of the present invention.

[0017] [Figure 6] FIG. 6 is a simplified schematic diagram of an eye tracking system according to one embodiment of the present invention.

[0018] [Figure 7] FIG. 7 is a simplified flowchart illustrating a method of operating an eye tracking system according to one embodiment of the present invention.

[0019] [Figure 8] FIG. 8 is a simplified flowchart illustrating a method for tracking eye movement from a first orientation to a second orientation according to an embodiment of the present invention.

[0020] [Figure 9] FIG. 9 is a simplified schematic diagram illustrating a side view of an eye tracking system using corneal phosphenes, according to one embodiment of the present invention.

[0021] [Figure 10] FIG. 10 is a simplified flowchart illustrating a method for characterizing eye orientation, according to an embodiment of the present invention.

[0022] [Figure 11] FIG. 11 is a simplified flowchart illustrating a method for performing eye tracking according to one embodiment of the present invention.

[0023] [Figure 12A] FIG. 12A is a simplified schematic diagram illustrating a side view of an eye tracking system according to one embodiment of the present invention.

[0024] [Figure 12B] FIG. 12B is a simplified schematic diagram illustrating a side view of an eye tracking system in accordance with an alternative embodiment of the present invention.

[0025] [Figure 12C] FIG. 12C is a simplified schematic diagram illustrating a side view of an eye tracking system in accordance with another alternative embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The present invention generally relates to methods and systems related to projection display systems, including wearable displays. More specifically, embodiments of the present invention provide methods and systems for tracking eye movement in association with optical scanning projector systems (also referred to as beam scanning projector systems). In certain embodiments, methods and systems for determining eye orientation and movement as a function of time (i.e., eye tracking) are implemented in conjunction with a waveguide display driven by a fiber scanning projector. The present invention is applicable to a variety of applications in computer vision and image display systems.

[0027] As described herein, embodiments of the present invention leverage elements of an optical scanning projector (e.g., a fiber scanning projector) to enable eye tracking. For example, a beam of light can be scanned in time onto the eye. Light reflected from either the cornea or retina of the eye is detected using one or more detectors. As the position of the beam of light as a function of time is known, temporal variations in reflection can be correlated with eye position, enabling gaze detection.

[0028] 1 diagrammatically illustrates light paths in a viewing optical assembly (VOA) that may be used to present digital or virtual images to a viewer, according to an embodiment of the present invention. The VOA includes a projector 101 and an eyepiece 100 that may be worn around or in front of the viewer's eye. As discussed herein, the VOA may be integrated with the frames of eyeglasses to present digital or virtual images to a viewer wearing these eyeglasses.

[0029] Referring to FIG. 1, a fiber scanning projector 101 is illustrated. The fiber scanning projector 101, which may have dimensions of approximately 2 mm×2 mm×7 mm, includes a fiber input 110, a fiber oscillation region 120, and an optical assembly section 130. A piezoelectric actuator 150 is supported by a retaining collar 152 and driven by an electrical signal from a wire (not shown). An optical fiber 154, also referred to as a scanning fiber, is mechanically coupled to the piezoelectric actuator 150 and oscillates within the fiber oscillation region 120, for example, in a helical configuration with each deflection increasing during the projection of light over a given frame time. Input light to the fiber scanning projector is provided through the fiber input 110, and output light from the fiber scanning projector 101 is provided through one or more of the surfaces of the optical assembly section 130. The various elements of the fiber scanning projector are more fully described in U.S. Patent Application Serial No. __, filed March 21, 2018 (Attorney Docket No. 101782-1075067 (003310US)), the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0030] 1 illustrates light from fiber scanning projector 101 as being directed directly into eyepiece 100, in other embodiments an optional projector relay 103 is utilized although this is not required by the invention and other optical configurations can also be utilized in accordance with embodiments of the invention. In the illustrated embodiment, light exits the optical assembly section in a direction approximately perpendicular to the longitudinal axis of the mechanical enclosure 156 of fiber scanning projector 101.

[0031] With reference to FIG. 1, a fiber scanning projector 101 is illustrated. However, it should be understood that other scanning light or beam systems, which may be implemented, for example, as a scanning waveguide system, may also be used with embodiments of the present invention. Thus, while optical fiber as one implementation for guiding light is illustrated, in some embodiments, the present invention is not limited to fiber scanning systems, and other waveguide scanning systems may also be utilized in accordance with other embodiments. Examples of other waveguide systems include microelectromechanical systems (MEMS) that integrate waveguide features, for example, silicon waveguides integrated with cantilevered beams, into an optical scanning system. Additionally, scanning mirror systems, in which a beam of light is steered by a projector, may be used with embodiments of the present invention as described herein. Additionally, scanning point sources, for example, light emitting diodes (LEDs) or organic LEDs (OLEDs), may be utilized in embodiments of the present invention.

[0032] During operation, the optical fiber 154, which is mechanically attached to the piezoelectric actuator 150, oscillates within the fiber oscillation region 120. In one embodiment, the piezoelectric actuator 150 includes four electrodes distributed at circumferential positions offset 90° relative to each other. Thus, positive and negative voltages applied to opposite sides of the piezoelectric actuator can deflect the actuator and the scanning fiber in the plane of the electrodes. By synchronously driving all four electrodes, fiber oscillation can be achieved. As light exits the optical fiber 154 as it scans, it is coupled into the optical assembly section 130, which redirects the light towards the eyepiece 100.

[0033] The fiber scanning projector 101 can provide multiple colors, including the three primary colors red, green, and blue (RBG), to form a full-color display. Thus, the eyepiece 100 may include one or more eyepiece layers. In one embodiment, the eyepiece 100 includes three eyepiece layers, i.e., one eyepiece layer for each of the three primary colors red, green, and blue. In another embodiment, the eyepiece 100 may include six eyepiece layers, i.e., one set of eyepiece layers for each of the three primary colors configured to form a virtual image in one depth plane, and another set of eyepiece layers for each of the three primary colors configured to form a virtual image in another depth plane. In other embodiments, the eyepiece 100 may include three or more eyepiece layers for each of the three primary colors for three or more different depth planes. Each eyepiece lens layer comprises a planar waveguide and may include an internal coupling grating 107, an orthogonal pupil expander (OPE) region 108, and an exit pupil expander (EPE) region 109.

[0034] Still referring to FIG. 1 , the projector 101 projects image light onto an internal coupling grating 107 in the eyepiece layer 100. The internal coupling grating 107 couples the image light from the projector 101 into a planar waveguide that propagates in a direction toward the OPE region 108. The waveguide propagates the image light in a horizontal direction by total internal reflection (TIR). The OPE region 108 of the eyepiece layer 100 also includes diffractive elements that couple and redirect a portion of the image light propagating in the waveguide toward the EPE region 109. The EPE region 109 includes diffractive elements that couple and direct a portion of the image light propagating in the waveguide toward the viewer's eye 102 in a direction substantially perpendicular to the plane of the eyepiece layer 100. In this way, the image projected by the projector 101 can be viewed by the viewer's eye 102.

[0035] As described above, the image light generated by the projector may include light of three primary colors, namely, blue (B), green (G), and red (R). Such image light can be separated into its constituent colors, e.g., in time or space, such that the image light of each constituent color can be coupled into a separate waveguide within the eyepiece.

[0036] FIG. 2 is a simplified schematic diagram illustrating glasses incorporating an eye tracking system according to an embodiment of the present invention. As described more fully herein, a compact form factor comparable to standard glasses is enabled by embodiments of the present invention. By utilizing embodiments of the present invention, a display with a desired field of view, depth of resolution, integrated inertial motion unit (IMU), camera, audio components, and the like are provided. FIG. 2 illustrates glasses 210 and a frame 212. As illustrated in FIG. 2, the fiber scanning projector 101 can be mounted within the frame 212 such that the projected light is directed toward the internal coupling grating 107. In other embodiments, the projector can be mounted within the temples of the glasses. As discussed in connection with FIG. 1, the fiber scanning projector 101 works in combination with an eyepiece 100 disposed within the frame 212 to direct the projected light toward the user's eye. The small size of the fiber scanning projector 101 allows for the integration of multiple fiber scanning projectors that can direct light toward the viewer's eyes, although this is not required by the present invention and a single projector per eye can be utilized. Although the eye tracking system is only discussed in connection with the right eye, it should be understood that a similar system can be used with the other eye. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0037] In addition to integrating the projector 101 into the frame 212, according to an embodiment of the present invention, one or more sensors can be integrated into the glasses as additional elements of an eye tracking system suitable for tracking the movement of the viewer's eyes. As illustrated in FIG. 2, a photodetector 220 is mounted in the frame opposite the fiber scanning projector 101. As described more fully below, light emitted from the eyepiece 100 can be reflected from the eye and incident on the photodetector 220, which can be a high-speed photodetector operating, for example, at a few kilohertz speed, at tens of kilohertz speed, or at higher frequencies. In addition to the optical sensor mounted in the frame, other optical configurations are also included within the scope of the present invention, including a reflector or folding mirror that directs the reflected light to one or more optical sensors. In an alternative embodiment, a refractive element is utilized to capture and route the reflected light to the optical sensor. In yet another alternative embodiment, an optical fiber, for example a multimode optical fiber, can be used to capture the reflected light. In all of these embodiments, multiple optical sensors may be used and multiple optical elements (e.g., reflectors, folding mirrors, refractive elements, fibers, or the like) may be used to implement the methods and systems described herein. Thus, the above description relating to a single optical element and illustration of a single photodetector 220 mounted within a frame is by way of example only and not limitation.

[0038] As the fiber scanning projector emits light into the projection path, the position of the emitted light as a function of time is known. For example, during the scanning of a spiral scan pattern associated with a frame of a video, the spatial position of the emitted light as a function of time can be determined and recorded. As an example, within a 33 millisecond frame time associated with the presentation of a display frame at 30 Hz, the fiber can be scanned at a speed in the range of tens of kilohertz, moving from a central portion of the display frame to a peripheral portion within the 33 millisecond frame time. As the light is reflected from the eye and received at a photodetector, the photodetector can be used to measure the intensity of the reflected light as a function of time (e.g., at a speed of tens of kilohertz). By correlating the spatial position of the emitted light as a function of time and the measured photodetector intensity as a function of time, the spatial position associated with the reflected light can be determined. Thus, since the position of the emitted light at a given time is known, the light reflected at this given time can be correlated with a position. As a result, a spatial map of the reflected light can be generated and correlated with the optical properties of the structure reflecting the light. For the case of the eye, the spatial map of the reflected light can be correlated with the position and / or orientation of the eye. By scanning the light emitted by the fiber and forming spatial maps at different times, the system is able to track the position of the eye as a function of time.

[0039] As described herein, embodiments of the present invention utilize a scanning beam system in which pixels are scanned into the eye one by one. In other words, the pixels are encoded as a function of angle. Considering the retina, an image is projected across the retina. As described below, by using an eyepiece to direct the scanning light into the user's eye, both visible light and characterization light are output in a superimposed manner. Reflection of the characterization light can be used to produce an image of the retina or other elements of the eye. Given that an element of the eye, e.g., the fovea, is aligned with a given pixel, the orientation of the eye can be determined. As this orientation changes over time, eye tracking is performed.

[0040] In some implementations, the optical scanning projector is used not only to display an image to a user, but also to receive light reflected from the user's eye, which then propagates in a return path through the eyepiece and impinges on the optical assembly section 130. Thus, in these embodiments, the optical scanning projector (e.g., a fiber scanning projector) is an element of the eye tracking system where light produced by the optical scanning projector is reflected and received by the optical scanning projector for subsequent use in the eye tracking calculations.

[0041] In embodiments where light reflected from the eye is coupled back into the fiber of the fiber scanning projector, the light can be coupled not only into the core of the fiber, but into the cladding of the fiber as well.

[0042] FIG. 3 is a simplified diagram illustrating several spectral profiles according to an embodiment of the present invention. In FIG. 3, red (R), green (G), and blue (B) spectral profiles associated with the emitted light are illustrated as curves 310, 312, and 314. These emission profiles provide a full-color display experience for the user. FIG. 3 also illustrates an infrared (IR) spectral profile 320, which is not visible to the eye but reflects off the cornea, retina, or other parts of the user's eye. Light in the IR region of the electromagnetic spectrum can be projected along with the RGB signal as the fiber scanning projector scans through the field of view of the display. In some embodiments, the light projected into the eyepiece can be classified as display light (e.g., RGB light) and characterization light (e.g., IR light). In certain embodiments, the display light is visible light and the characterization light is invisible light. In other implementations, different wavelengths of visible light can be used for both the display light and the characterization light. The fundus of the eye, including the retina, is characterized by different reflectivities as a function of position. Thus, as the projected light is scanned across different portions of the eye, the variations in reflected light intensity measured using an optical sensor (e.g., one or more high-speed photodiodes) can be used to generate a fundus map of the user's eye.

[0043] Other wavelengths can be utilized to provide reflectance from the cornea in addition to a single IR band as illustrated in Figure 3. Additionally, multiple bands in the IR, for example, can be utilized as appropriate for a particular application.

[0044] 4 is a simplified side view illustrating an eyepiece layer according to an embodiment of the present invention. An additional eyepiece layer can be provided in addition to the RGB layer to couple display and characterization light into the eyepiece. In FIG. 4, an eyepiece with matched internal coupling gratings (i.e., diffractive optical elements) is illustrated in side view.

[0045] The eyepiece 400, which may be used to project an image to a viewer's eye, includes a first planar waveguide 410 positioned in a first lateral plane (i.e., at a first longitudinal position disposed along the z-axis). The first planar waveguide 410 comprises a first diffractive optical element (DOE) 412 disposed at a first lateral position (i.e., a first xy coordinate position). The first DOE 412 is associated with a first wavelength range (e.g., red wavelengths) and is operable in this example to diffract red light into the first planar waveguide. The first DOE 412 may be implemented using a transmission grating that couples the first wavelength into the first planar waveguide but substantially passes wavelengths outside the first wavelength range.

[0046] The eyepiece also includes a second planar waveguide 420 positioned in a second lateral plane adjacent to the first lateral plane. The second planar waveguide includes a second DOE 422 disposed in a first lateral position below the first DOE 412. The second DOE 422 is associated with a second wavelength range (e.g., green wavelengths) and is operable in this embodiment to diffract green light into the second planar waveguide. The second DOE 422 can be implemented using a transmission grating that couples the second wavelength into the second planar waveguide but substantially passes wavelengths outside the second wavelength range.

[0047] A third planar waveguide 430 is positioned in a third lateral plane adjacent to the second lateral plane. The third planar waveguide includes a third DOE 432 disposed in a first lateral position below the first DOE and the second DOE and aligned along a vertical direction (i.e., aligned with the z-axis). The third DOE 422 is associated with a third wavelength range (e.g., blue wavelengths) and is operable in this embodiment to diffract blue light into the third planar waveguide. The third DOE 432 can be implemented using a transmission grating that couples the third wavelength into the third planar waveguide but substantially passes wavelengths outside the third wavelength range. Using the first three planar waveguides, visible radiation is directed toward a viewer.

[0048] In addition to the three planar waveguides 410, 420, and 430 that can be used to direct visible display light to a user, a fourth planar waveguide 440 is illustrated in FIG. 4. The fourth planar waveguide 440 is positioned in a fourth lateral plane adjacent to the third lateral plane. The fourth planar waveguide includes a fourth DOE 442 that is disposed in a first lateral position below the first DOE, the second DOE, and the third DOE and is aligned along the vertical direction (i.e., aligned with the z-axis). The fourth DOE 442 is associated with a fourth wavelength range (e.g., IR wavelengths) that is not visible to the user, and in this embodiment is operable to diffract IR light into the fourth planar waveguide. The fourth DOE 432 can be implemented using a transmission grating that couples the fourth wavelength into the fourth planar waveguide but substantially passes wavelengths outside the fourth wavelength range, or can be implemented as a reflective grating with a mirror coating for higher diffraction efficiency.

[0049] Although all four DOEs are aligned in the embodiment illustrated in FIG. 4, this is not required by the present invention, and the DOEs can be spatially separated at different lateral positions. Furthermore, the order in which the planar waveguides are aligned can be varied as appropriate for a particular application. As an example, the second DOE (e.g., for diffracting green light) can be spatially separated from the first and third DOEs that can be aligned. In this example, since green light is in the middle of the visible spectrum, it is spatially separated from blue and red light that is not strongly diffracted in the DOEs for other colors, allowing the blue and red DOEs to be spatially aligned. Similarly, the fourth DOE can be spatially separated from the DOEs associated with red wavelengths, e.g., aligned with the blue or green DOEs. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0050] Given the spatial location of light emitted by the fiber scanning projector as a function of time, which is associated with pixel locations of an image frame, as discussed above, the intensity of light reflected from the eye, for example in the IR spectrum, can be correlated with pixel locations to form a map of reflected intensity. Because reflectance varies as a function of position on the eye, an eye fingerprint can be formed for each eye position or orientation. Thus, embodiments of the present invention can track eye movement as a function of time, which can then be used to modify the manner in which virtual content is displayed, or the like. Although some embodiments are discussed in terms of image frames (i.e., video frames) of a video signal, it should be understood that image frames are not required by the present invention. In addition to images that are composited into frames, embodiments of the present invention are applicable more generally to known sequences of fiber positions and light outputs. For example, embodiments of the present invention are applicable to scanning displays that, rather than rendering frames, utilize a virtual point cloud at time t and calculate the desired pixel intensity given the current position of the scanning device. Knowing the position and intensity as a function of time allows for interpretation of eye tracking sensor signals as described herein.

[0051] FIG. 5 is a simplified side view illustrating an eyepiece layer according to an alternative embodiment of the present invention. FIG. 5 shares common elements with elements illustrated in FIG. 4, and the description provided in connection with FIG. 4 is applicable to FIG. 5, where appropriate. In the eyepiece illustrated in FIG. 5, a portion of the display light and the characterization light are coupled into one of the planar waveguides through a diffractive optical element. In the particular embodiment illustrated in FIG. 5, light in one of the visible wavelength ranges (e.g., the red wavelength range) and light in the invisible wavelength range (e.g., the IR wavelength range) are coupled into a first planar waveguide for transmission to the user's eye.

[0052] 5, a first planar waveguide 510 comprises a first diffractive optical element (DOE) 512 disposed at a first lateral position associated with a first wavelength range (e.g., red wavelengths) and a characterization wavelength range (e.g., IR wavelengths) and is operable to diffract light within the first wavelength range and the characterization wavelength range into the first planar waveguide. The first DOE 512 may be implemented using a transmission grating that couples light within the first wavelength and the characterization wavelength range into the first planar waveguide but substantially passes wavelengths outside the first wavelength range and the characterization wavelength range.

[0053] The eyepiece also includes a second planar waveguide 520 positioned in a second lateral plane adjacent to the first lateral plane. The second planar waveguide includes a second DOE 522 disposed in a first lateral position below the first DOE 512. The second DOE 522 is associated with a second wavelength range (e.g., green wavelengths) and is operable in this embodiment to diffract green light into the second planar waveguide. The second DOE 522 can be implemented using a transmission grating that couples the second wavelength into the second planar waveguide but substantially passes wavelengths outside the second wavelength range.

[0054] A third planar waveguide 530 is positioned in a third lateral plane adjacent to the second lateral plane. The third planar waveguide includes a third DOE 532 disposed at a first lateral position below the first DOE and the second DOE and aligned along a vertical direction (i.e., aligned with the z-axis). The third DOE 522 is associated with a third wavelength range (e.g., blue wavelengths) and is operable in this embodiment to diffract blue light into the third planar waveguide. The third DOE 532 can be implemented using a transmission grating that couples the third wavelength into the third planar waveguide but substantially passes wavelengths outside the third wavelength range.

[0055] Using these three planar waveguides, both visible radiation and radiation within the characterization wavelength range are directed towards the viewer. Light within the characterization wavelength range reflects off the viewer's eye and can be captured by one or more photodetectors as discussed above. Thus, using the three planar waveguides in this embodiment, display light and characterization light are delivered to the viewer.

[0056] 6 is a simplified schematic diagram of an eye tracking system according to an embodiment of the present invention. The eye tracking system 600 includes an image processor 610, at least one fiber scanning projector 620, an eyepiece 630, and an optical sensor 640. The image processor may include a driver circuit 612 for driving the fiber scanning projector 620, an eye tracking subsystem 614, a processor 616, and an input / output device 618. The eyepiece 630 includes a number of planar waveguides 632, each of which may incorporate a diffractive optical element 634. In the illustrated embodiment, the eyepiece 630 is mounted within a frame of glasses and includes four planar waveguides operating in conjunction with three display wavelength ranges (e.g., RGB) and one characterization wavelength range (e.g., infrared).

[0057] As illustrated in FIG. 6, the driver circuit 612 may include electronics that communicate with the fiber scanning projector. The image processor 610 may generate the virtual content that is ultimately displayed to the user. The image processor 610 may convert images or videos associated with the virtual content into a format that can be projected to the user in 3D through the use of an eyepiece that provides multiple depth planes. Content can be received through the input / output device 616 and processed using the processor 616 of the image processor 610.

[0058] In some embodiments, the eye tracking system 600 utilizes coupling optics 650 to direct light from the fiber scanning projector 620 to the eyepiece 630. Coupling optics 650 may refer to one or more conventional lenses used to direct light into the eyepiece, for example, a diffractive optical element in each planar waveguide.

[0059] In operation, the light output from the fiber scanning projector 650 can include both the display light 622 and the characterization light 624. In an embodiment utilizing the combining optics 650, the combined display light 626 and the combined characterization light 628 impinge on the diffractive optical element 634 of the planar waveguide layer 632. The output of the eyepiece 630 includes both the image display light 636 and the eyepiece characterization light 638 that propagate to the user's eye 605. Although both the image display light 636 and the eyepiece characterization light 638 are illustrated in FIG. 6 by single rays, it should be understood that this light may exit the eyepiece 630 as a diverging cone of light. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0060] Eyepiece characterization light reflecting from the eye is illustrated as reflected light 642 and impinges on an optical sensor 640, which may be mounted within the frame of eyeglasses. Due to the shape of the eye, the light reflecting from the eye, for example from the cornea, is characterized by a cone of light having a larger angular range than the cones of light associated with either the image display light 636 or the eyepiece characterization light 638. In applications using a photodetector, the electrical signal output by the photodetector is transmitted to the image processor 610 so that it may be processed by the eye tracking subsystem 614 or other suitable processor.

[0061] Eye tracking can be performed as the eye moves over time. As an example, a first orientation of the eye during a first frame of a video image can be determined and compared with a second orientation of the eye during a subsequent second frame of the video image. Thus, over time, the orientation of the eye can be determined as follows:

[0062] The fiber of the fiber scanning projector is scanned to form a first image frame of a video image. The scan may be characterized by a helical pattern with increasing deflection from a central axis as a function of time. As the fiber sweeps the scan pattern, a first series of pixels is generated that defines a first image frame. The first image frame is coupled into an eyepiece that includes a plurality of planar waveguides. In some embodiments, both display light (e.g., visible light) and characterization light (e.g., infrared light) can be generated using the fiber scanning projector, and different planar waveguides can be used to direct the light to the user's eye. Thus, the embodiment transmits at least a portion of the first image frame to the user's eye. Using an optical sensor, a first series of reflected intensities is detected. Each of the reflected intensities of the series is associated with a pixel of the first series of pixels.

[0063] Considering a fiber, as it scans through the pixels of an image frame, at a first time, a first pixel will be illuminated. When this first pixel is displayed, a certain amount of light will be reflected from the eye, resulting in a first measurement of reflected intensity. For a second pixel at a later time, a second measurement of reflected intensity will be provided. Thus, over time, a series of reflected intensities will be measured, each associated with a different pixel. The amount of reflection and the value of the reflected intensity in the series will depend on the shape (e.g., radius) of the cornea at which the pixel is displayed, the location where the pixel is located on the cornea, and the like.

[0064] Thus, as the fiber sweeps through the pattern that defines the pixels of the first image frame, the timing of pixel generation is known. This provides a spatial mapping of the pixels as a function of time. The timing of the detection of a sequence of reflected intensities can be correlated with the timing of pixel generation to generate a spatial map of reflected intensities. This spatial map can be used to characterize the orientation of the eye during the first image frame.

[0065] For subsequent image frames (i.e., the second image frame), this process can be repeated to generate a second spatial map of the reflection intensity measured during the second image frame. Because the spatial map will vary with eye position, by comparing spatial maps collected at two different times as the video progresses, eye position can be tracked as a function of time. In some embodiments, a calibration process is used when images are displayed and used to direct the user to look at different parts of the field of view. Measurements of the reflection intensity at each of the different eye orientations can be used to generate a spatial map associated with each of the different eye orientations. This library of spatial maps can then be used to locate the eye at any time during the video display. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0066] FIG. 7 is a method of operating an eye tracking system according to an embodiment of the present invention. The eye tracking system includes a fiber scanning projector, an eyepiece, and an optical sensor. Method 700 includes generating an optical signal using the fiber scanning projector (710). The fiber scanning projector can utilize elements as illustrated in FIG. 6 and other figures herein. The optical signal can include a display signal (e.g., one or more visible wavelengths, such as red, green, and blue wavelengths) used to generate light displayed to a user, and a characterization signal (e.g., one or more non-visible wavelengths, such as infrared wavelengths) used to provide a signal for use in eye tracking. An eyepiece as illustrated in FIGS. 4 and 5 may include multiple planar waveguides, for example, four planar waveguides for three visible colors and an infrared characterization signal. In some embodiments, multiple depth planes are provided by combining sets of eyepieces with different characteristics.

[0067] The method also includes coupling (712) at least a portion of the optical signal into an eyepiece and projecting (714) an image from the eyepiece that is viewable by the user's eye. The method further includes detecting (716) light reflected from the user's eye using an optical sensor (e.g., a photodetector) and tracking (718) the position of the user's eye. While a single photodetector can be utilized in some embodiments, other embodiments utilize multiple photodetectors, and the invention is not limited to the use of a single photodetector. Tracking the position of the user's eye can include correlating a sequence of pixel positions of the image with a sequence of intensities of light reflected from the user's eye.

[0068] In one embodiment, the optical signal includes a display signal (e.g., comprised of one or more visible wavelengths) and a characterization signal (e.g., comprised of one or more non-visible wavelengths, e.g., infrared wavelengths). Using an eyepiece comprising multiple planar waveguides arranged adjacent to one another, the various wavelengths can be coupled into the planar waveguides as a function of wavelength. For example, four planar waveguides corresponding to red, green, blue, and infrared wavelengths can be utilized.

[0069] It should be understood that the specific steps illustrated in FIG. 7 provide a particular method of operating an eye tracking system according to an embodiment of the present invention. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, each step illustrated in FIG. 7 may include multiple sub-steps that may be performed in various sequences, as appropriate for each step. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0070] FIG. 8 is a simplified flow chart illustrating a method of tracking eye movement from a first orientation to a second orientation according to an embodiment of the present invention. The method 800 includes scanning a fiber of a fiber scanning projector to form a first image frame of a video image (810). A first series of pixels defines the first image frame. The method also includes coupling the first image frame into an eyepiece including a plurality of planar waveguides (812) and transmitting at least a portion of the first image frame to a user's eye (814). In one example, the plurality of planar waveguides can include a first planar waveguide corresponding to a red wavelength, a second planar waveguide corresponding to a green wavelength, a third planar waveguide corresponding to a blue wavelength, and a fourth planar waveguide corresponding to an infrared wavelength. Coupling the first image frame into the eyepiece can be accomplished by diffracting the first image frame into one or more of the planar waveguides using a diffractive optical element, for example an internal coupling grating.

[0071] The method further includes detecting a first series of reflected intensities (816), each of which is associated with a pixel of the first series of pixels. Additionally, the method includes correlating (818) the first series of detected reflected intensities with the first series of pixels and determining (820) a first orientation of the eye.

[0072] The method also includes scanning the fiber (822) to form a second image frame of the video image, where a second series of pixels defines the second image frame, coupling the second image frame into the eyepiece (824), and transmitting at least a portion of the second image frame to the user's eye (826). Coupling the second image frame into the eyepiece can be accomplished by diffracting the second image frame into one or more of the planar waveguides using a diffractive optical element, for example an internal coupling grating.

[0073] The steps of transmitting at least a portion of the first image frame to the user's eye and transmitting at least a portion of the second image frame to the user's eye may be accomplished by diffracting at least a portion of the first image frame and at least a portion of the second image frame using one or more diffractive optical elements, for example a first diffractive optical element (e.g., an orthogonal pupil expander) operable to diffract light within the plane of the eyepiece and a second diffractive optical element (e.g., an exit pupil expander) operable to diffract light out of the plane of the eyepiece.

[0074] The method further includes detecting (828) a second series of reflected intensities, each of which is associated with a pixel of the second series of pixels, correlating (830) the second series of detected reflected intensities with the second series of pixels, and determining (832) a second orientation of the eye. In an alternative embodiment, the method further includes outputting (834) trajectories associated with the first orientation and the second orientation.

[0075] The first and second image frames may be consecutive frames of the video image. Alternatively, one or more additional image frames may be located between the first and second image frames, which may be temporally separated in time by a multiple of the frame period.

[0076] As part of the implementation of the method illustrated in FIG. 8, a training process can be utilized, in which the display presents a series of objects to be viewed to the user. The user is tasked with viewing a series of objects, each of which is associated with a different eye gaze. As the user views each of the series of objects, a predetermined eye movement is produced, and the scanning of the light beam and the corresponding reflection detected are used to train the system, thereby accumulating a database of time-varying reflections as a function of time per eye gaze. Machine learning and other suitable techniques can be utilized in implementing such a training process. In addition to measurements made while the user is viewing a static object, the tracking of a moving object by the user can be utilized as a component of the training process.

[0077] It should be understood that the specific steps illustrated in FIG. 8 provide a particular method of tracking eye movement from a first orientation to a second orientation according to an embodiment of the present invention. Other sequences of steps may also be implemented in accordance with alternative embodiments. For example, alternative embodiments of the present invention may implement the steps outlined above in a different order. Furthermore, the individual steps illustrated in FIG. 8 may include multiple sub-steps that may be implemented in various sequences, as appropriate for the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0078] 9 is a simplified schematic diagram illustrating a side view of an eye tracking system using corneal phosphenes, according to an embodiment of the present invention. As illustrated in FIG. 9, embodiments of the present invention provide methods and systems that are robust in high brightness applications, for example outdoors, when the signal-to-noise ratio for reflected light from the eye may be lower than desired.

[0079] Referring to FIG. 9, the cornea 910 of the eye 905 is positioned adjacent to an eyepiece 920. The eyepiece 920 may share common elements with the eyepiece 500 illustrated in FIG. 5. A fiber scanning projector is used to drive the eyepiece, and light emitted from the eyepiece 920 is illustrated as output rays 922, which may also be referred to as beamlets. In the side view illustrated in FIG. 9, a pair of eyeglass frames 930 is shown supporting the eyepiece. For purposes of clarity, the output rays 922 that impinge on the cornea 910 are illustrated using solid lines, and the reflected rays 924 that reflect from the cornea are illustrated using dashed lines.

[0080] The frame 930 supports a number of optical detectors 932, which may be high-speed photodetectors. The multiple optical detectors 932 are aligned around the periphery of the frame and are illustrated in FIG. 9 as detectors 932a and 932b positioned on opposing edges of the frame 930. Although the detectors 932a and 932b are illustrated at the top and bottom of the frame, it should be understood that they may be integrated in other peripheral portions, including near the temples of the glasses. When the output light rays or beamlets impinge on the cornea, which is generally spherical, the light will be reflected back toward the eyepiece / frame / optical detector at different angles from the corneal surface. In turn, some of the reflected light will reach the multiple optical detectors, which may be embedded or otherwise mounted within the frame 930. In one embodiment, the multiple optical detectors are aligned at fixed intervals around the periphery, while in other embodiments, the array pattern may use non-fixed intervals as appropriate for a particular application. The optical detectors can be oriented such that they collect light reflected from the eye with high efficiency. Although high speed photodetectors are discussed herein as optical detectors, the invention is not limited to photodetectors and optical fibers, multimode optical fibers, and other optical elements can be utilized in place of or in conjunction with various optical detectors.

[0081] Consider a non-moving eye. During a first time, e.g., during a first portion of a frame of video data, a first portion of an image presented to the user, defined by an output light beam, will result in a first amount of reflection being detected at the multiple optical detectors. During a second time, e.g., during a second portion of a frame of video data, the portion of the image presented to the user and the corresponding output light beam will be changed with respect to the output light beam that defines the first portion of the image. This results from the fact that a scanning fiber projector is utilized to drive the eyepiece, and the angle swept by the scanning fiber varies as a function of time. Thus, during a second time, the amount of reflection detected at the multiple optical detectors will vary with respect to the first amount of reflection detected by the multiple optical detectors. Thus, with multiple optical detectors aligned around the periphery of the frame, even with a non-moving eye, a time-varying amount of light will be detected as a function of the scan angle. The dynamic nature of the scanning beam effectively provides a dynamic flash generator, since the reflection from the cornea, which varies as a function of time, can be viewed as a flash that varies as a function of time. Gaze detection is made possible by correlating eye gaze with patterns of reflections / flashes detected as a function of time.

[0082] The system can be calibrated to provide a unique signature, similar to a fingerprint, for several eye gazes or orientations, as discussed above in connection with FIG. 8. By having the user look in a first direction, the time-varying change in the detected intensity at each of the multiple optical detectors can be characterized to define a unique signature associated with this first direction. When the user looks in a second, different direction, a different time-varying change in the detected intensity can be measured and used to define a second unique signature associated with this second direction. The calibration process can perform these signatures for several eye orientations, which can facilitate the formation of a lookup table or other suitable pattern identifier for correlating eye gaze or orientation with the unique signature. As mentioned above, in addition to lookup tables, machine learning techniques can be employed, which may be more computationally efficient.

[0083] In operation, time-varying changes in the detected intensity can be measured and matched to a unique characterization for a given orientation at a sequence of times to provide eye tracking.

[0084] FIG. 10 is a simplified flow chart illustrating a method for performing eye tracking using corneal phosphenes, according to an embodiment of the present invention. The method 1000 includes scanning a beam of an optical scanning projector (e.g., a fiber scanning projector) to form a first portion of an image frame of a video image (1010). A first series of pixels defines the first portion of the image frame. The method also includes coupling the first portion of the image frame into an eyepiece lens including a plurality of planar waveguides (1012), transmitting at least a portion of the first portion of the image frame to a user's eye (1014), and detecting a temporal reflection signal at each of a plurality of optical detectors (1016). In some embodiments, a single detector is utilized rather than multiple optical detectors, for example mounted on a temple adjacent to a frame including the eyepiece lens.

[0085] The plurality of planar waveguides may include a first planar waveguide corresponding to red wavelengths, a second planar waveguide corresponding to green wavelengths, a third planar waveguide corresponding to blue wavelengths, and a fourth planar waveguide corresponding to infrared wavelengths. The plurality of optical detectors may be a plurality of photodetectors that may be aligned around a peripheral region of the eye. As illustrated in FIG. 2, the eyepiece may be mounted in a frame, and the plurality of photodetectors may be positioned around a periphery of the frame. In other embodiments, one or more detectors are mounted in a temple that is connected to the frame. The method further includes correlating the plurality of temporal reflection signals with the first series of pixels (1018).

[0086] Additionally, the method includes scanning the beam to form a second portion of an image frame of the video image (1020). A second series of pixels defines the second portion of the image frame. The method also includes coupling the second portion of the image frame into an eyepiece (1022), transmitting at least a portion of the second portion of the image frame to a user's eye (1024), and detecting a second temporal reflection signal at each of a plurality of optical detectors (1026). Further, the method includes correlating the plurality of second temporal reflection signals with the second series of pixels (1028) and determining an eye orientation (1030). As discussed above, the multiple optical detectors can be replaced with a single detector, e.g., a high-speed photodetector, per eyepiece. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0087] It should be understood that the specific steps illustrated in FIG. 10 provide a particular method of characterizing eye orientation according to an embodiment of the present invention. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps illustrated in FIG. 10 may include multiple sub-steps that may be performed in various sequences, as appropriate for the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0088] 11 is a simplified flow chart illustrating a method for performing eye tracking according to an embodiment of the present invention. Method 1100 includes providing a fiber scanning projector, an eyepiece mounted in a frame, and a number of optical detectors aligned around a periphery of the frame. The eyepiece can include a number of planar waveguides.

[0089] The method also includes projecting 1110 a first image frame to the eye using a fiber scanning projector and an eyepiece, detecting 1112 a plurality of time-varying reflected signals using a plurality of optical detectors, and determining 1114 a first eye orientation. Projecting 1110 the first image frame to the eye can include coupling a portion of the first image frame into each of a plurality of planar waveguides and transmitting at least a portion of the coupled portion to the user's eye.

[0090] The method further includes projecting (1116) a second image frame to the eye using the fiber scanning projector and eyepiece, detecting (1118) a second plurality of time-varying reflected signals using a plurality of optical detectors, and determining (1120) a second eye orientation. Projecting the second image frame to the eye can include coupling a portion of the second image frame into each of a plurality of planar waveguides and transmitting at least a portion of the coupled portion to the user's eye. In an alternative embodiment, the method further includes outputting trajectories associated with the first eye orientation and the second eye orientation.

[0091] As discussed in connection with FIG. 10, the multiple time-varying reflected signals can be associated with increasing scan angles of the fiber scanning projector during a first image frame as the fiber scans, for example, at a larger angle between frames. The second multiple time-varying reflected signals can also be associated with increasing scan angles of the fiber scanning projector during a second image frame. The first and second image frames can be successive frames of a video image. Alternatively, one or more additional image frames can be positioned between the first and second image frames.

[0092] It should be understood that the specific steps illustrated in FIG. 11 provide a particular method of performing eye tracking according to an embodiment of the present invention. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps illustrated in FIG. 11 may include multiple sub-steps that may be performed in various sequences, as appropriate for the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0093] 12A is a simplified schematic diagram illustrating a side view of an eye tracking system according to an embodiment of the present invention. As illustrated in FIG. 12A, embodiments of the present invention provide methods and systems that are robust in high brightness applications, for example outdoors, when the signal to noise ratio for reflected light from the eye may be lower than desired. The embodiments of the present invention and the elements illustrated in FIG. 12 can be considered in the context of a scanning laser ophthalmoscope.

[0094] Referring to FIG. 12A, eye 1205 is positioned adjacent to eyepiece 1220. Eyepiece 1220 may share common elements with eyepiece 500 illustrated in FIG. 5. A fiber scanning projector is used to drive the eyepiece, and light emitted from eyepiece 1220 is illustrated as output rays 1222, which may also be referred to as beamlets. In the side view illustrated in FIG. 12, a frame 1230 of glasses is shown supporting the eyepiece. For purposes of clarity, output rays 1222 passing through cornea 1208 and impinging on cornea 1210 are illustrated using solid lines, and reflected rays 1224 reflecting from the retina are illustrated using dashed lines. As illustrated in FIG. 12A, refraction by the lens and cornea (not shown) results in the beamlets being focused to a point on the retina, which correlates with a pixel of the display. If output beam 1222 includes wavelengths at which the retina is highly reflective, e.g., infrared wavelengths, reflected light at these wavelengths will tend to propagate substantially on-axis through the eye and cornea after reflection from the retina. Following a substantially reverse optical path, reflected beam 1224 is shown to be slightly diverging for purposes of clarity. Because reflected beam 1224 propagates along a path that is substantially the same axis as output beam 1222, the detector is preferably within the field of view, rather than at the periphery of the field of view.

[0095] After reflection from the retina 1210, the reflected light rays 1224 impinge on the eyepiece 1220 and are coupled into the eyepiece (e.g., by diffraction), where they propagate inside the eyepiece by total internal reflection. In the illustrated embodiment, the frame 1230 supports one or more optical detectors 1232, which may be high-speed photodetectors. The one or more optical detectors 1232 may be aligned around the periphery of the eyepiece and frame to capture light reflected at several angles. Light may be coupled from the eyepiece to the optical detector without the use of an orthogonal coupling element, e.g., an external coupling grating, but may pass through the edge of the eyepiece to the optical detector, as shown in FIG. 12A. The efficiency of coupling from the eyepiece to the optical detector may thus be higher than that typically achieved using an orthogonal coupling element. In this embodiment, the eyepiece is thus used as a collecting element, which may be coupled with one or more optical detectors integrated with the eyepiece and operable to detect reflected light located outside the user's field of view. The operation of the detector may share similar functionality as the detectors discussed in connection with other embodiments herein.

[0096] FIG. 12B is a simplified schematic diagram illustrating a side view of an eye tracking system according to an alternative embodiment of the present invention. In this alternative embodiment, an additional diffractive structure 1221 is integrated with the eyepiece 1220 to diffract light reflected from the retina to one or more detectors 1252. As an example, an additional diffractive structure 1221 that is efficient at infrared wavelengths may be integrated with the eyepiece 1220 (e.g., stacked behind the eyepiece) to direct reflected light 1224 to detectors 1252a and 1252b. Although detectors 1252a and 1252b are illustrated at the top and bottom of the frame, it should be understood that they may be integrated at other peripheral portions, including near the temples of the glasses. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0097] As discussed in connection with Fig. 12A, light propagating in the additional diffractive structure 1221 by TIR can be coupled from the additional diffractive structure to the optical detector without the use of an orthogonal coupling element, e.g., an external coupling grating, but can pass through an edge of the additional diffractive structure to the optical detector, as shown in Fig. 12B. As a result, the efficiency of coupling from the additional diffractive structure to the optical detector 1252a / 1252b can thus typically be higher than that achieved using an orthogonal coupling element.

[0098] 12A and 12B, an optical detector may be placed to the right (i.e., in front) of the eyepiece 920 and used to detect light reflected from the retina. Additionally, a beam splitter or mirror that preferentially reflects infrared light (i.e., a hot mirror) may be utilized to separate the reflected infrared light from reflected light in other wavelength bands. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0099] The diffractive properties of either the eyepiece or the additional diffractive structures can be spatially varied to increase the information content available using the system. Note that the diffractive structures can be either image relaying (i.e., image storing) or will not store imaging effects. As an example, a multimode fiber coupled to a detector can be optically coupled to the eyepiece at a peripheral location for non-image storing applications. As another example, in connection with the spatial variation discussed above, an array of spatially separated fibers can be optically coupled to the eyepiece. As the eye / cornea rotates, the distribution of reflected light impinging on the spatially separated array varies, and this temporal / spatial variation as the eye / cornea rotates can be used to encode the orientation of the eye.

[0100] For example, the spatial variation of the eyepiece of the additional diffractive structures may be implemented as a set of radial cross sections, each mounted on the periphery, diffracting light toward a detector associated with each of the radial cross sections. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0101] FIG. 12C is a simplified schematic diagram illustrating a side view of an eye tracking system according to another alternative embodiment of the present invention. The embodiment illustrated in FIG. 12C shares common elements with the embodiment illustrated in FIG. 12A and 12B, and the description provided in connection with FIG. 12A and 12B is applicable to FIG. 12C, where appropriate. In the embodiment illustrated in FIG. 12C, a diffractive reflector 1240 is integrated with the eyepiece 1220 to reflect light reflected from the retina to one or more detectors 1252 through diffraction. As an example, a diffractive reflector 1240, which may be effective at reflecting infrared wavelengths, may be integrated with the eyepiece 1220 (e.g., stacked behind the eyepiece) to reflect light 1224 to detectors 1252a and 1252b, which may be positioned peripherally and at a vertical position between the eyepiece and the eye (i.e., along the z-axis). For example, detectors 1252a and 1252b may be mounted within the temples of glasses. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0102] Thus, the diffractive reflector illustrated in Figure 12C provides an alternative to the waveguiding implementation illustrated in Figures 12A and 12B. In addition, the use of a diffractive reflector allows flexibility in detector placement since the detector can be moved away from the eyeglass frame, for example to one or more predefined locations along the eyeglass temples. As discussed above, the detector does not need to store images and can be implemented as a high speed photodiode, reducing system complexity and cost.

[0103] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and that in light thereof, various modifications or changes will be suggested to those skilled in the art and are to be included within the spirit and scope of the present application and the appended claims.

Claims

1. An eye tracking system operable to track eye movements of a user, the eye tracking system comprising: a light scanning projector operable to scan a light beam to project an image; an eyepiece optically coupled to the optical scanning projector, the eyepiece including an inner surface facing an eye of the user and an outer surface opposite the inner surface; An optical sensor; an image processor coupled to the optical scanning projector and the optical sensor; Including, An eye tracking system, wherein the eyepiece includes a diffractive reflector overlying the eyepiece, the diffractive reflector being formed as a planar layer laminated to the inner surface of the eyepiece and operable to reflect one or more time-varying signals reflected from the user's eye to the optical sensor by diffraction.

2. The eye tracking system of claim 1, wherein the image processor is operable to correlate positions of a series of pixels of the image with a series of intensities of the one or more time-varying signals reflected from the user's eye to track eye movements of the user.

3. 3. The eye tracking system of claim 1 or 2, wherein the optical scanning projector comprises a fiber scanning projector configured to scan in a spiral pattern with increasing angular deflection as a function of time during the scanning time of an image frame.

4. The eye tracking system of claim 1 or 2, further comprising a pair of glasses having a frame, the optical scanning projector, the eyepiece, and the optical sensor being mounted in the frame.

5. The eye tracking system of claim 1 or 2, wherein the eyepiece lens comprises a plurality of planar waveguides.

6. The eye tracking system of claim 5 , wherein the plurality of planar waveguides includes four planar waveguides.

7. The eye tracking system of claim 1 or 2, wherein the optical sensor comprises a photodetector.

8. The eye tracking system of claim 1 or 2, wherein the optical scanning projector is operable to output a display signal and a characterization signal.

9. The eye tracking system of claim 8 , wherein the indication signal comprises one or more visible wavelengths and the characterization signal comprises one or more non-visible wavelengths.

10. 10. The eye tracking system of claim 9, wherein the one or more visible wavelengths include red, green, and blue wavelengths, and the one or more invisible wavelengths include infrared wavelengths.

11. The eyepiece lens is a first planar waveguide corresponding to red wavelengths; a second planar waveguide corresponding to green wavelengths; a third planar waveguide corresponding to blue wavelengths; and a fourth planar waveguide corresponding to infrared wavelengths; and 11. The eye tracking system of claim 10, comprising a plurality of planar waveguides including:

12. 1. A method of operating an eye tracking system including an optical scanning projector, an eyepiece, and an optical sensor, the method comprising: generating a time-varying optical signal using the optical scanning projector; coupling at least a portion of the time-varying optical signal into the eyepiece, the eyepiece including an inner surface facing an eye of a user and an outer surface opposite the inner surface; projecting an image from the eyepiece viewable by an eye of the user; reflecting, by diffraction, one or more time-varying signals reflected by the user's eye to the optical sensor using a diffractive reflector overlying the eyepiece, the diffractive reflector being formed as a planar layer laminated to the inner surface of the eyepiece; detecting the one or more time-varying signals using the optical sensor; Tracking the position of the user's eyes; A method comprising:

13. The method of claim 12, wherein tracking the position of the user's eye includes correlating a series of pixel positions of the image with a series of intensities of the one or more time-varying signals reflected from the user's eye.

14. 14. The method of claim 12 or 13, wherein the optical scanning projector comprises a fiber scanning projector configured to scan in a helical pattern with increasing angular deflection as a function of time during the scanning time of the image.

15. The method of claim 12 or 13, wherein the time-varying optical signals include an indication signal and a characterization signal.

16. The method of claim 15 , wherein the indication signal includes one or more visible wavelengths and the characterization signal includes one or more non-visible wavelengths.

17. The method of claim 16 , wherein the one or more visible wavelengths include red, green, and blue wavelengths, and the one or more invisible wavelengths include infrared wavelengths.

18. The method of claim 12 or 13, wherein the eyepiece comprises a plurality of planar waveguides.

19. The method of claim 18 , wherein the plurality of planar waveguides comprises four planar waveguides.

20. The method of claim 12 or 13, wherein the optical sensor comprises a photodetector.

Citation Information

Patent Citations

  • Holographic Substrate-Guided Wave-Based See-Through Display

    US20100157400A1

  • Systems, devices, and methods that integrate eye tracking and scanning laser projection in wearable heads-up displays

    US20160349514A1

  • Systems, devices, and methods that integrate eye tracking and scanning laser projection in wearable heads-up displays

    US20160349516A1

  • Optical system

    WO2016203212A2

  • Collimating fiber scanner design with inward pointing angles in virtual / augmented reality system

    WO2017015302A1