Bright pupil eye tracking system

By using a light source array and pupil brightness detection technology in the eye tracking system, the problem of inaccurate eye position determination in existing technologies has been solved, achieving higher-precision eye tracking and improving the content presentation effect of head-mounted displays.

CN115053164BActive Publication Date: 2024-11-15CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180010061.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-04-09
Filing Date
2021-01-25
Publication Date
2024-11-15
Estimated Expiration
2041-01-25

AI Technical Summary

Technical Problem

Existing eye-tracking systems suffer from insufficient accuracy in determining the position and focus of a user's eyes, especially in head-mounted displays, making it difficult to accurately adjust the content presented to the user.

Method used

An array of light sources is used to emit invisible light to illuminate the eye. By capturing and analyzing images of the eye, the position of the eye is determined using pupil brightness detection technology, and then precise positioning is achieved using computing devices.

Benefits of technology

It improves the precision and accuracy of eye tracking, enabling better adjustment of content presentation in head-mounted displays and enhancing the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115053164B_ABST
    Figure CN115053164B_ABST
Patent Text Reader

Abstract

An eye tracking method includes: enabling at least one light source in an array of light sources to emit invisible light to illuminate an eye. The method also includes: obtaining at least one image of the eye while the at least one light source is enabled. When the at least one light source is enabled, in response to determining that the at least one image indicates a bright pupil condition, the position of the eye can then be determined based on the position of the at least one light source within the array of light sources.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 983,352, entitled “Bright Pupil Eye-Tracking System,” filed on February 28, 2020. U.S. Patent Application No. 62 / 983,352 is expressly incorporated herein by reference in its entirety. Technical Field

[0003] Various aspects of the present disclosure generally relate to eye tracking systems. Background Art

[0004] Eye tracking may refer to the process of detecting the direction of a user's gaze, which may include detecting the angular orientation of the eye in 3-dimensional (3D) space. Eye tracking may also include detecting the position and movement of the eye (e.g., the center of the eye), the torsion of the eye (i.e., the rotation of the eye around the pupil axis), the shape of the eye, the current focal length of the eye, the dilation of the pupil, other characteristics of the eye state, or some combination thereof.

[0005] Eye tracking systems may be used in a variety of contexts, such as medical research or diagnosis, human-computer interaction, etc. In some contexts, such as using a head mounted display (HMD), it may be advantageous for the HMD to determine the location of a user's eyes and / or determine where the user's eyes are focused to alter content presented to the user. Summary of the invention

[0006] Accordingly, the present invention relates to a method, a system and a head mounted display according to the appended claims.

[0007] In one aspect, the present invention relates to an eye tracking method comprising:

[0008] enabling at least one light source in the array of light sources to emit invisible light to illuminate the eye;

[0009] While the at least one light source is enabled, obtaining at least one image of the eye; and

[0010] In response to determining that at least one image indicates a bright pupil condition when at least one light source is enabled, a position of the eye is determined based on a position of at least one light source within the array of light sources.

[0011] In an embodiment of the method according to the present invention, determining that at least one image indicates a bright pupil condition may also include: comparing the brightness of the pupil in the at least one image with the brightness of the pupil in another image corresponding to when another light source in the light source array is enabled to emit invisible light to illuminate the eye.

[0012] In an embodiment of the method according to the invention, determining that the at least one image indicates a bright pupil condition may further comprise comparing the brightness of the pupil in the at least one image with a brightness threshold.

[0013] In an embodiment of the method according to the present invention, the method may further include: enabling a first light source in a light source array to emit invisible light to illuminate the eye; while the first light source is enabled, obtaining a first image of the eye; enabling a second light source in a light source array to emit invisible light to illuminate the eye; while the second light source is enabled, obtaining a second image of the eye; and in response to determining that the pupil of the eye in the first image is brighter than the pupil of the eye in the second image, determining the position of the eye based on the position of the first light source within the light source array. In addition, the method may further include: in response to determining that the pupil of the eye in the second image is brighter than the pupil of the eye in the first image, determining the position of the eye based on the position of the second light source within the light source array.

[0014] In an embodiment of the method according to the invention, the invisible light may comprise infrared or near infrared light.

[0015] In an embodiment of the method according to the present invention, the method may further include: enabling a first light source grouping in the light source array; obtaining a first image of the eye while the first light source grouping is enabled; enabling a second light source grouping in the light source array; obtaining a second image of the eye while the second light source grouping is enabled; comparing the brightness of the pupil in the first image with the brightness of the pupil in the second image; and in response to determining that the pupil in the first image is brighter than the pupil in the second image, determining that the pupil is aligned with the first light source grouping. In addition, the method may further include: in response to determining that the pupil is aligned with the first light source grouping, individually enabling each light source in the first grouping; obtaining corresponding images of the eye while each light source in the first light source grouping is individually enabled; and determining that one of the corresponding images indicates a bright pupil condition, wherein determining the position of the eye is based on the position of one of the light sources included in the first grouping, the one light source corresponding to one image indicating the bright pupil condition.

[0016] In an embodiment of the method according to the present invention, the method may further include: enabling a first light source grouping in the light source array; obtaining a first image of the eye while the first light source grouping is enabled; enabling a second light source grouping in the light source array; obtaining a second image of the eye while the second light source grouping is enabled; comparing the brightness of the pupil in the first image with the brightness of the pupil in the second image; and in response to determining that the pupil in the first image is brighter than the pupil in the second image, determining that the pupil is aligned with the first light source grouping. In addition, the method may further include: in response to determining that the pupil in the second image is brighter than the pupil in the first image, determining that the pupil is aligned with the second light source grouping; in response to determining that the pupil is aligned with the second light source grouping, individually enabling each light source in the second grouping; obtaining corresponding images of the eye while each light source in the second light source grouping is individually enabled; and determining that one of the corresponding images indicates a bright pupil condition, wherein determining the position of the eye is based on the position of one of the light sources included in the second grouping, the one light source corresponding to one image indicating a bright pupil condition.

[0017] In one aspect, the present invention relates to an eye tracking system, the system comprising: a light source array, wherein each light source in the light source array is configured to be selectively enabled to emit invisible light to an eye-facing side of the eye tracking system to illuminate the eye; a camera configured to capture an image of the eye based on the invisible light; and a computing device communicatively coupled to the light source array and the camera, wherein the computing device comprises: a light source control module configured to enable at least one light source in the light source array; a camera interface module configured to obtain at least one image of the eye captured by the camera while at least one light source is enabled; a bright pupil detection module configured to detect whether at least one image indicates a bright pupil condition; and a position determination module configured to determine a position of the eye based on a position of at least one light source within the light source array in response to the bright pupil detection module determining that at least one image indicates a bright pupil condition.

[0018] In an embodiment of an eye tracking system according to the present invention, the bright pupil detection module can be configured to compare the brightness of the pupil in at least one image with the brightness of the pupil in another image corresponding to when another light source in the light source array is enabled to emit invisible light to illuminate the eye.

[0019] In an embodiment of the eye tracking system according to the invention, the bright pupil detection module may be configured to compare the brightness of the pupil in the at least one image with a brightness threshold.

[0020] In an embodiment of the eye tracking system according to the present invention, the invisible light may include infrared or near infrared light.

[0021] In an embodiment of the eye tracking system according to the present invention, the system may further include: an optical combiner disposed between the light source array and the eye-facing side of the eye tracking system, wherein the optical combiner is configured to receive the reflected invisible light reflected by the eye to guide the reflected invisible light to the camera to generate an image of the eye. In addition, the light source array may further include a transparent substrate, and wherein each light source in the light source array is disposed on the transparent substrate.

[0022] In an embodiment of the eye tracking system according to the present invention, the system may further include a plurality of beam shaping optical devices, each of the plurality of beam shaping optical devices being disposed on a corresponding light source in the light source array to guide the invisible light emitted by each corresponding light source along a corresponding optical axis. In addition, each of the plurality of beam shaping optical devices may be configured to collimate the invisible light into collimated invisible light. In addition, the system may further include: an optical element disposed between the plurality of beam shaping optical devices and the eye-facing side of the eye tracking system to focus the collimated invisible light onto the pupil plane of the eye.

[0023] In one aspect, the present invention relates to a head mounted display (HMD) comprising: an array of light sources, wherein each light source in the array of light sources is configured to be selectively enabled to emit invisible light to an eye-facing side of the HMD to illuminate the eye; a camera configured to capture an image of the eye; an optical combiner disposed between the array of light sources and the eye-facing side, wherein the optical combiner is configured to receive reflected invisible light reflected by the eye and direct the reflected invisible light to the camera to generate an image of the eye; and a display layer disposed between the optical combiner and a back side of the HMD, the display layer being configured to provide a display image transmitted by the optical combiner. The invention relates to a method for controlling the display light of the HMD to be directed to the eye-facing side of the HMD; and a computing device communicatively coupled to the light source array and the camera, wherein the computing device includes: a light source control module configured to enable at least one light source in the light source array; a camera interface module configured to obtain at least one image of the eye captured by the camera while the at least one light source is enabled; a bright pupil detection module configured to detect whether the at least one image indicates a bright pupil condition; and a position determination module configured to determine a position of the eye based on a position of the at least one light source within the light source array in response to the bright pupil detection module determining that the at least one image indicates a bright pupil condition.

[0024] In an embodiment of the HMD according to the invention, each light source in the light source array may be arranged on the display layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Non-limiting and non-exhaustive aspects of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0026] Figure 1A An example eye tracking system in accordance with various aspects of the present disclosure is illustrated.

[0027] Figure 1B Another example eye tracking system in accordance with various aspects of the present disclosure is illustrated.

[0028] Figure 2 An array of light sources and a corresponding image of an eye captured when at least one light source is enabled are illustrated in accordance with various aspects of the present disclosure.

[0029] Figure 3A Illustrated are arrays of light sources and corresponding images of an eye captured when groupings of light sources are enabled in accordance with various aspects of the present disclosure.

[0030] Figure 3B Illustrated are arrays of light sources and corresponding images of an eye captured when individual light source groupings are enabled in accordance with various aspects of the present disclosure.

[0031] Figure 4 Computing devices in accordance with various aspects of the present disclosure are illustrated.

[0032] Figure 5 A flow chart illustrating an eye tracking process in accordance with various aspects of the present disclosure is illustrated.

[0033] Figure 6 is a flow chart illustrating another eye tracking process including enabling a first light source and a second light source in accordance with various aspects of the present disclosure.

[0034] Figure 7 is a flow chart illustrating an eye tracking process including enabling a first light source grouping and a second light source grouping in accordance with various aspects of the present disclosure.

[0035] Figure 8 A head mounted display (HMD) according to various aspects of the present disclosure is illustrated. DETAILED DESCRIPTION

[0036] Various aspects and embodiments are disclosed in the following description and related drawings to illustrate specific examples related to eye tracking methods and systems. Upon reading this disclosure, alternate aspects and embodiments will be apparent to those skilled in the relevant art and may be constructed and practiced without departing from the scope or spirit of the disclosure. Additionally, well-known elements will not be described in detail or may be omitted so as not to obscure the relevant details of the various aspects and embodiments disclosed herein.

[0037] In some embodiments of the present disclosure, the term "near eye" may be defined as including an element that is configured to be placed within 50 mm of a user's eye when a near eye device is in use. Thus, a "near eye optical element" or "near eye system" will include one or more elements that are configured to be placed within 50 mm of a user's eye.

[0038] In various aspects of the present disclosure, visible light may be defined as having a wavelength range of about 380 nm to 700 nm. Invisible light may be defined as light with a wavelength outside the visible light range, such as ultraviolet light and infrared light. Infrared light having a wavelength range of about 700 nm to 1 mm includes near infrared light. In various aspects of the present disclosure, near infrared light may be defined as having a wavelength range of about 700 nm to 1.4 μm.

[0039] Figure 1A An example eye tracking system 100A is illustrated in accordance with various aspects of the present disclosure. The illustrated example of eye tracking system 100A includes light source arrays 102A-102G, layer 104, beam shaping optics 106A-106G, optical element 108, optical combiner 110, camera 120, and computing device 122. Figure 1A Also shown is an eye 114 including a fundus 112 , a pupil 118 , and a pupil plane 116 .

[0040] like Figure 1A As shown, light sources 102A to 102G are disposed on layer 104. Layer 104 can be a transparent substrate, such as glass or plastic. In one example, light sources 102A to 102G can be encapsulated in a transparent substrate. The transparent substrate can be transmissive for visible light (e.g., 400nm to 750nm), and can be configured to be placed on a display plane of an electronic or optical display layer (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a micro-LED display, a waveguide, etc.), which is configured to generate display light for presentation to a user. In another example, layer 104 itself is an electronic or optical display layer, in which light sources 102A to 102G are disposed on the display layer and dispersed in the display light generated by the display layer (e.g., in the user's field of view).

[0041] Each light source 102A-102G can be a micro light emitting diode (microLED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a super luminescent diode (SLED). In addition, each light source 102A-102G can be individually enabled to emit invisible light 107 to illuminate the eye 114. In some examples, the invisible light 107 is infrared light or near infrared light. In some aspects, each light source 102A-102G is arranged on layer 104 in a two-dimensional (2D) array of columns and rows. In some examples, each light source 102A-102G can be referred to as a point light source, where only one of the light sources 102A-102G is enabled at the same time to emit invisible light 107 (e.g., at a certain time). Figure 1A In the illustrated example of FIG. 1 , only a single light source 102G is currently enabled to emit invisible light 107).

[0042] Figure 1A The illustrated example of also illustrates a plurality of beam shaping optics 106A to 106G. Each beam shaping optic 106A to 106G can be disposed on a corresponding light source 102A to 102G to direct the invisible light emitted by each light source along a corresponding optical axis. Thus, in some aspects, each light source 102A to 102G can be configured to emit invisible light 107 along a different optical axis through a corresponding beam shaping optic 106A to 106G. For example, the beam shaping optic 106A can be configured to direct the invisible light generated by the light source 102A along a first optical axis, while the beam shaping optic 106B can be configured to direct the invisible light generated by the light source 102B along a second (i.e., different) optical axis. In some examples, each of the beam shaping optics 106A to 106G can also be configured to collimate the invisible light 107 generated by the corresponding light source 102A to 102G.

[0043] like Figure 1A As shown, the eye tracking system 100A also includes an optical element 108 disposed between the beam shaping optics 106A-106G and an eyeward side 103 of the eye tracking system 100A. Figure 1A Also illustrated is an optical combiner 110 disposed between the light sources 102A to 102G and the eye-facing side 103, and specifically may be disposed between the optical element 108 and the eye-facing side 103. In some examples, the optical element 108 includes one or more lenses configured to receive the collimated invisible light 107 and focus the collimated invisible light 107 to a pupil plane 116 of the eye 114 through the optical combiner 110. Thus, in some examples, the optical element 108 provides a "Maxwell view" of the invisible light 107. Figure 1AAs shown, the invisible light 107 then expands toward the back of the eye 114 as it exits the pupil 118 to illuminate a large area of ​​the fundus 112 .

[0044] In some aspects, optical combiner 110 is configured to receive reflected invisible light 109 (having wavelengths emitted by light sources 102A-102G) reflected / scattered by fundus 112 of eye 114 and direct reflected invisible light 109 to camera 120. Camera 120 may be located at a different location than that illustrated. In some aspects, optical combiner 110 is transmissive to visible light (e.g., approximately 400nm to 700nm), such as scene light (e.g., from the environment) incident on back side 105 of eye tracking system 100A. Even so, in some examples, optical combiner 110 is transmissive to visible light generated by display layers (e.g., layer 104). In some examples, optical combiner 110 may be configured as a holographic optical element (HOE) or volume hologram, which may include one or more Bragg gratings for directing reflected invisible light 109 toward camera 120. In some examples, the optical combiner 110 includes a polarization-selective volume hologram (aka polarization volume hologram) that diffracts (reflects) a specific polarization orientation of incident light having a specific wavelength toward the camera 120 while passing other polarization orientations.

[0045] Camera 120 is configured to generate one or more images 123 of eye 114, where image 123 is an image of reflected invisible light 109. In some examples, camera 120 can be configured to filter out light other than invisible light 107 / reflected invisible light 109, such that camera 120 only images wavelengths of reflected invisible light 109.

[0046] In some examples, computing device 122 may be configured to determine eye tracking information (e.g., location, orientation, gaze angle, etc.) of eye 114 based on image 123 captured by camera 120. As will be described in greater detail below, computing device 122 may then process image 123 to detect a bright pupil condition to determine eye tracking information (e.g., location, orientation, gaze angle, etc. of eye 114). For example, computing device 122 may determine whether eye 114 is looking in a straight ahead, leftward, rightward, upward, or downward direction.

[0047] In some embodiments, the computing device 122 may include a light source control module that is communicatively coupled to the light source array 102A to 102G. As discussed above, each of the light sources 102A to 102G may emit invisible light 107 along a corresponding optical axis. If the eye 114 is not aligned with the currently enabled light sources 102A to 102G, the pupil 118 of the eye may bloom the invisible light 107, which may reduce or prevent the light from reaching the fundus 112, which will darken the appearance of the pupil 118 in the resulting image 123. However, if the eye 114 is aligned with the currently enabled light sources 102A to 102G (e.g., along the same optical axis as the emitted invisible light 107), the pupil 118 will appear brighter in the resulting image 123.

[0048] Thus, the control module of the computing device 122 can generate one or more control signals 124 to selectively enable at least one of the light sources 102A-102G, and analyze the resulting images 123 to detect a bright pupil condition. If the images 123 captured while a particular light source (e.g., light source 102G) is enabled indicate a bright pupil condition, the computing device 122 can then determine the position / gaze angle of the eye 114 based on the position of the light source (e.g., light source 102G) within the light source array.

[0049] As mentioned above, the beam shaping optics 106A to 106G can be configured to collimate the invisible light 107 emitted by the light sources 102A to 102G, wherein the optical element 108 then focuses the collimated invisible light onto the pupil plane 116. However, in other examples of the present disclosure, the beam shaping optics 106A to 106G that collimate the invisible light can be omitted. For example, Figure 1B Another example eye tracking system 100B according to various aspects of the present disclosure is illustrated. Figure 1B In the illustrated example of , invisible (and non-collimated) light 107 emitted by light source 102G is received by optical element 108. In this example, optical element 108 can then collimate the invisible light and direct the collimated invisible light to eye 114. Eye 114 then focuses the collimated invisible light (e.g., by the lens of eye 114) onto fundus 112. In this example, fundus 112 can act as a retroreflector, where the reflected invisible light is reflected back at substantially the same angle as the invisible light was incident on fundus 112. Similar to Figure 1A In the example of , the reflected invisible light 109 is received by the optical combiner 110 , which then directs the reflected invisible light 109 to the camera 120 to generate the image 123 .

[0050] Figure 22 illustrates an array of light sources 206 and corresponding images 202A to 202G of an eye captured when at least one light source is enabled, in accordance with various aspects of the present disclosure. The array of light sources 206 is Figure 1A and 1B Similarly, images 202A to 202G are possible examples of image 123. As discussed above, each of the light sources in light source array 206 can be individually enabled to emit invisible light to illuminate the eye, where a corresponding image of the eye can then be captured while the light source is enabled. By way of example, Figure 2 The image 202A is illustrated as being captured when light source 208A is enabled, image 202B is an image of the eye when light source 208B is enabled, image 202C is an image of the eye when light source 208C is enabled, image 202D is an image of the eye when light source 208D is enabled, image 202E is an image of the eye when light source 208E is enabled, and image 202F is an image of the eye when light source 208F is enabled, image 202G is an image of the eye when light source 208G is enabled, and so on. Figure 1A The images captured by the camera 120 can then be used by a computing device (e.g. Figure 1A In one example, analyzing images 202A to 202G includes determining whether any image indicates a bright pupil condition. Figure 2 In the illustrated example of , image 202E indicates a bright pupil condition of pupil 204. In response to determining that the image indicates a bright pupil condition, the computing device determines that the eye is aligned with a light source that was enabled when the image was obtained (e.g., the eye was aligned with light source 208E when image 202E was captured). The computing device can then determine the position of the eye based on the known position of light source 208E within array 206. In some examples, determining the position of the eye includes converting the position of light source 208E into a calculated eye position and / or gaze angle.

[0051] In some examples, determining whether a bright pupil condition exists includes comparing the brightness of the pupil 204 in one image to the brightness of the pupil 204 in another image. In some aspects, this may include identifying a pupil region for each image using one or more computer vision techniques, and determining the brightness of the pupil region (e.g., the average brightness of all pixels within the pupil region). The determined brightness of the pupil region may then be compared to the brightness determined in the other images, with the image having the brightest pupil region being determined to indicate a bright pupil condition.

[0052] In other examples, a bright pupil condition can be determined by comparing the brightness of the pupil region of the image to a brightness threshold. That is, in this example, the brightness of the pupil region of one image can be compared to a fixed or dynamically created threshold indicating a bright pupil condition, rather than comparing the images together.

[0053] In some aspects, each light source in the light source array 206 may be enabled individually and a corresponding image captured. However, in other examples, two or more light source groups may be enabled, with corresponding images acquired to determine whether the eye is aligned with a particular light source group. By way of example, Figure 3A 2 illustrates an array of light sources 206 and corresponding images 302A and 302B of an eye captured when light source groupings are enabled in accordance with various aspects of the present disclosure. Figure 3A As shown, image 302A is an image of the eye captured when the first light source grouping 304 is enabled, while image 302B is an image of the eye captured when the second light source grouping 306 is enabled. Figure 3A Only two groups of light sources are illustrated, but other embodiments may include multiple groups in the light source array 206, including two or more.

[0054] In some examples, enabling the light source grouping includes enabling multiple light sources in light source array 206 simultaneously, wherein the captured image includes an image of the eye when all light sources in the grouping are enabled. The resulting images 302A and 302B can then be analyzed (e.g., by computing device 122) to determine whether a bright pupil condition exists by comparing the brightness of pupil 204 in image 302A to the brightness of pupil 204 in image 302B, or by comparing the brightness of pupil 204 in each image to a brightness threshold. In the illustrated example, image 302B indicates a bright pupil condition.

[0055] An image indicating a bright pupil condition when a light source grouping is enabled means that one of the light sources in light source grouping 206 is aligned with the eye. Figure 3A Each light source in the group 306) can then be individually enabled (and corresponding images captured) to determine the specific light source that caused the bright pupil condition.

[0056] For example, Figure 3B The light source array 206 and corresponding images 302x to 302z of the eye captured when the individual light source groups 306 are enabled are illustrated in accordance with various aspects of the present disclosure. Image 302x is an image of the eye captured when light source 308A is enabled, image 302y is an image of the eye when light source 308B is enabled, and image 302z is an image of the eye when light source 308C is enabled. Figure 3BOnly three images 302x to 302y are illustrated, and as mentioned above, various aspects of the present disclosure may include individually enabling each of the light sources included in group 306. Figure 3B As shown, image 302y indicates a bright pupil condition, where the computing device can then determine the position of the eye based on the position of light source 308B within light source array 206.

[0057] Figure 4 A computing device 402 is illustrated in accordance with various aspects of the present disclosure. The illustrated example of computing device 402 is shown to include a communication interface 404, one or more processors 406, hardware 408, and memory 410. Figure 4 The computing device 402 is Figure 1A One possible implementation of the computing device 122.

[0058] Communication interface 404 may include wireless and / or wired communication components that enable computing device 402 to transmit data to and receive data from other devices / components. Hardware 408 may include additional hardware interfaces, data communication, or data storage hardware. For example, a hardware interface may include a data output device and one or more data input devices.

[0059] Memory 410 can be implemented using computer-readable media (such as computer storage media). In some aspects, computer-readable media may include volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), high-definition multimedia / data storage disk or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device or any other non-transmission medium that can be used to store information for access by a computing device.

[0060] The processor 406 and memory 410 of the computing device 402 may implement a light source control module 412, a camera interface module 414, a bright pupil detection module 416, and a position determination module 418. The light source control module 412, the camera interface module 414, the bright pupil detection module 416, and the position determination module 418 may include routines, program instructions, objects, and / or data structures that perform specific tasks or implement specific abstract data types. The memory 410 may also include a data repository (not shown) used by the light source control module 412, the camera interface module 414, the bright pupil detection module 416, and / or the position determination module 418.

[0061] The light source control module 412 may be configured to generate one or more control signals (eg Figure 1A The camera interface module 414 may be configured to receive an image (e.g., a control signal 124) to enable and / or disable one or more light sources included in the light source arrays 102A to 102G. Figure 1A The camera interface module 414 may optionally be configured to trigger the camera 120 to capture one or more images in response to the light source control module 412 enabling a light source in the light source array. The bright pupil detection module 416 is configured to analyze one or more images to determine whether the image indicates a bright pupil condition. As mentioned above, the bright pupil detection module 416 may implement one or more computer vision techniques to identify a pupil region and determine a corresponding brightness of the pupil region. The bright pupil detection module 416 may detect a bright pupil condition by comparing the brightness of the pupil region between two or more images, or alternatively, may detect a bright pupil condition by comparing the brightness of the pupil region on the image with a brightness threshold. In response to the bright pupil detection module 416 detecting a bright pupil condition, the position determination module 418 may determine the position of the eye based on the position of the light source enabled for the image indicating such a bright pupil condition. As mentioned above, the position of the eye may be determined by converting the position of the light source into a calculated eye position and / or gaze angle.

[0062] Figure 5 FIG. 5 is a flowchart illustrating an eye tracking process 500 according to various aspects of the present disclosure. The process 500 is performed by Figure 1A One possible process of eye tracking performed by the eye tracking system 100A. In process block 502, Figure 1A The computing device 122 may generate a control signal 124 to enable at least one light source 102G in the light source array 102A-102G to emit invisible light 107 to illuminate the eye 114. In process block 504, while the at least one light source 102G is enabled, the computing device 122 may obtain (e.g., receive) at least one image 123 of the eye 114 captured by the camera 120. Next, in process block 508, in response to determining that the image 123 indicates a bright pupil condition, the computing device 122 may determine a position of the eye 114 based on the position of the at least one light source 102G within the light source array 102A-102G.

[0063] Figure 6 is a flow chart illustrating another eye tracking process 600 including activating a first light source and a second light source according to various aspects of the present disclosure. The process 600 refers to Figure 2Description. In process block 602, a first light source 208A in the light source array 206 is enabled to emit invisible light to illuminate the eye. In process block 604, a first image 202A is obtained, which is an image of the eye captured when the first light source 208A is enabled. Next, in process block 606, a second light source 208E is enabled to emit invisible light to illuminate the eye (e.g., the first light source 208A is disabled). Process block 608 includes obtaining an image 202E captured by a camera (e.g., camera 120) while the second light source 208E is enabled.

[0064] In decision block 610, a computing device (e.g., computing device 122) determines whether pupil 204 in first image 202A is brighter than pupil 204 in second image 202E. If pupil 204 in first image 202A is determined to be brighter than pupil 204 in second image 202E, process 600 proceeds to process block 612, where the computing device determines the position of the eye based on the position of first light source 208A within light source array 206. However, if in decision block 610, it is determined that pupil 204 in first image 202A is not brighter than pupil 204 in second image 202E (i.e., pupil 204 in second image 202E is brighter), process 600 proceeds to process block 614, where the position of the eye is determined based on the position of second light source 208E within light source array 206.

[0065] Although process 600 is described above with reference to enabling only two light sources (and obtaining corresponding two images), as described above, various aspects of the present disclosure may include individually enabling two or more light sources 102A to 102G included in light source arrays 102A to 102G, wherein the brightness of each of the obtained images is compared to each other to determine which image indicates a bright pupil condition.

[0066] Figure 7 is a flow chart illustrating an eye tracking process 700 including enabling a first light source grouping and a second light source grouping according to various aspects of the present disclosure. The process 700 refers to Figure 3A and 3B describe.

[0067] In process block 702, a first light source grouping 304 is enabled to emit invisible light to illuminate an eye. In process block 704, a first image 302A is obtained, which is an image of the eye when all light sources included in the first grouping 304 are simultaneously enabled. In process block 706, a second light source grouping 306 is enabled to emit invisible light. Process block 708 includes obtaining a second image 302B, which is an image of the eye when all light sources included in the second grouping 306 are simultaneously enabled (e.g., and the light sources in the first grouping 304 are disabled).

[0068] Next, in decision block 710, a computing device (e.g., computing device 122) determines whether pupil 204 in first image 302A is brighter than pupil 204 in second image 302B. If pupil 204 in first image 302A is determined to be brighter than pupil 204 in second image 302B, process 700 proceeds to process block 712, where each light source within first grouping 304 is individually enabled and a corresponding image of the eye is obtained (e.g., captured by camera 120). In process block 714, the computing device then determines the position of the eye based on the position of the light sources in first grouping 304 that correspond to the image indicating a bright pupil condition (i.e., obtained in process block 712).

[0069] Returning to decision block 710, if the pupil 204 in the first image 302A is not brighter than the pupil 204 in the second image 302B, then process 700 proceeds to process block 716, where each of the light sources included in the second grouping 306 is individually enabled and a corresponding image of the eye is obtained. In process block 718, the computing device may then determine the position of the eye based on the position of the light sources in the second grouping 306 that correspond to the image indicating a bright pupil condition (i.e., obtained in process block 716).

[0070] In some embodiments, various aspects of the present disclosure can be used in a head-mounted device, such as a virtual reality (VR) or augmented reality (AR) device. In some aspects, the head-mounted device can include an eye tracking system to enhance the user's viewing experience. In some instances, eye tracking can be assisted by determining the position and / or movement of the eyes. For example, when the gaze angle is determined, the virtual image presented to the user by the display of the head-mounted device can be adjusted in response to the determined gaze angle.

[0071] By example, Figure 8A head mounted display (HMD) 800 according to various aspects of the present disclosure is illustrated. An HMD such as HMD 800 is a head mounted device that is typically worn on a user's head to provide artificial reality content to the user. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, and it may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality, or some combination and / or derivative thereof. The illustrated example of HMD 800 is shown to include a viewing structure 840, a top stabilizing structure 841, a side stabilizing structure 842, a rear stabilizing structure 843, and a front rigid body 844. In some examples, HMD 800 is configured to be worn on the head of a user of HMD 800, wherein the top stabilizing structure 841, the side stabilizing structure 842, and / or the rear stabilizing structure 843 may include a fabric strap that includes an elastic structure and one or more rigid structures (e.g., plastic) to secure HMD 800 to the user's head. The HMD 800 may also optionally include one or more headphones 820 to deliver audio to the ear(s) of the user of the HMD 800 .

[0072] The illustrated example of HMD 800 also includes an interface film 818 for contacting the face of a user of HMD 800 , where the interface film 818 is used to block at least some ambient light from reaching the eyes of the user of HMD 800 .

[0073] The example HMD 800 may also include a chassis ( Figure 8 The viewing structures 840 may include any of processing logic, wired and / or wireless data interfaces for sending and receiving data, a graphics processor, and one or more memories for storing data and computer executable instructions. In one example, the viewing structures 840 may be configured to receive wired power and / or may be configured to be powered by one or more batteries. Additionally, the viewing structures 840 may be configured to receive wired and / or wireless data including video data.

[0074] The viewing structures 840 may include a display system having one or more electronic displays for directing light to the eye(s) of a user of the HMD 800. The display system may include one or more of a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a micro-LED display, etc., for emitting light (e.g., content, images, video, etc.) to the user of the HMD 800. The viewing structures 840 may also include an optical assembly configured to receive image light from the display system and generate a virtual image (e.g., by collimating the image light) for viewing by the eyes of the wearer of the HMD 800.

[0075] In some examples, the viewing structure includes an eye tracking system 845 for tracking the movement and / or determining the position of the user's eyes. The eye tracking system 845 can be implemented by any of the embodiments discussed herein, including Figure 1A Eye tracking system 100A.

[0076] Embodiments of the present invention may include an artificial reality system, or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect on the viewer). Additionally, in some embodiments, artificial reality may also be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or is otherwise used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0077] The above description of the illustrated embodiments of the present invention (including what is described in the Abstract) is not intended to be exhaustive or to limit the present invention to the precise forms disclosed. Although specific embodiments and examples of the present invention are described herein for illustrative purposes, various modifications are possible within the scope of the present invention, as will be appreciated by those skilled in the relevant art.

[0078] These modifications may be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in this specification. Instead, the scope of the invention will be determined entirely by the following claims, which will be interpreted in accordance with established doctrines of claim interpretation.

Claims

1. An eye tracking method, include: enabling at least one light source in an array of light sources to emit invisible light to illuminate an eye, wherein the invisible light emitted by the at least one light source propagates through an optical combiner disposed between the array of light sources and the eye, wherein the optical combiner is configured to receive the invisible light reflected by a fundus of the eye and direct the invisible light reflected by the fundus to a camera, and wherein the optical combiner is configured as a holographic optical element or a volume hologram; generating, with the camera, at least one image of the eye while the at least one light source is enabled, wherein the at least one image is the invisible light reflected by the fundus; as well as In response to determining that the at least one image indicates a bright pupil condition when the at least one light source is enabled, a position of the eye is determined based on a position of the at least one light source within the array of light sources.

2. The eye tracking method of claim 1 , wherein determining that the at least one image indicates the bright pupil condition include: comparing the brightness of the pupil in the at least one image with the brightness of the pupil in another image corresponding to another light source in the array of light sources being enabled to emit invisible light to illuminate the eye, or The brightness of the pupil in the at least one image is compared to a brightness threshold.

3. The eye tracking method according to claim 1, further comprising: include: activating a first light source in the array of light sources to emit invisible light to illuminate the eye; While the first light source is enabled, obtaining a first image of the eye; activating a second light source in the array of light sources to emit invisible light to illuminate the eye; while the second light source is enabled, obtaining a second image of the eye; as well as In response to determining that the pupil of the eye in the first image is brighter than the pupil of the eye in the second image, the position of the eye is determined based on the position of the first light source within the array of light sources.

4. The eye tracking method according to claim 3, further comprising: include: In response to determining that the pupil of the eye in the second image is brighter than the pupil of the eye in the first image, the position of the eye is determined based on the position of the second light source within the array of light sources. The eye tracking method according to claim 1 , wherein the invisible light comprises infrared or near-infrared light.

6. The eye tracking method according to claim 1, further comprising: include: activating a first light source group in the light source array; While the first light source group is enabled, obtaining a first image of the eye; activating a second light source group in the light source array; while the second light source group is enabled, obtaining a second image of the eye; comparing the brightness of the pupil in the first image with the brightness of the pupil in the second image; as well as In response to determining that the pupil in the first image is brighter than the pupil in the second image, determining that the pupil is aligned with the first light source grouping.

7. The eye tracking method according to claim 6, further comprising: include: In response to determining that the pupil is aligned with the first grouping of light sources, individually enabling each light source in the first grouping of light sources; while each light source in the first light source group is individually enabled, obtaining a corresponding image of the eye; as well as determining that one of the corresponding images indicates the bright pupil condition, wherein determining the position of the eye is based on a position of one of the light sources included in the first light source grouping, the one light source corresponding to the one image indicating the bright pupil condition, or further comprising: In response to determining that the pupil in the second image is brighter than the pupil in the first image, determining that the pupil is aligned with the second light source grouping; In response to determining that the pupil is aligned with the second grouping of light sources, individually enabling each light source in the second grouping of light sources; while each light source in the second light source grouping is individually enabled, obtaining a corresponding image of the eye; and Determining that one of the corresponding images indicates the bright pupil condition, wherein determining the position of the eye is based on a position of one of the light sources included in the second light source grouping, the one light source corresponding to the one image indicating the bright pupil condition.

8. An eye tracking system, include: an array of light sources, wherein each light source in the array of light sources is configured to be selectively enabled to emit invisible light toward an eye-facing side of the eye tracking system to illuminate the eye; a camera configured to capture an image of the eye based on the invisible light; an optical combiner disposed between the light source array and an eye-facing side of the eye tracking system, wherein the optical combiner is configured to receive reflected invisible light reflected by a fundus of the eye to direct the reflected invisible light to the camera to generate the image of the eye, wherein the optical combiner is configured as a holographic optical element or a volume hologram; as well as a computing device communicatively coupled to the light source array and the camera, wherein the computing device comprises: a light source control module, configured to enable at least one light source in the light source array; a camera interface module configured to obtain at least one image of the eye captured by the camera while the at least one light source is enabled, wherein the at least one image is the invisible light reflected by the fundus; a bright pupil detection module configured to detect whether the at least one image indicates a bright pupil condition; and A position determination module is configured to determine the position of the eye based on the position of the at least one light source within the light source array in response to the bright pupil detection module determining that the at least one image indicates the bright pupil condition.

9. The eye tracking system according to claim 8, wherein the bright pupil detection module is configured to: The brightness of the pupil in the at least one image is compared with the brightness of the pupil in another image corresponding to when another light source in the light source array is enabled to emit invisible light to illuminate the eye, or the brightness of the pupil in the at least one image is compared with a brightness threshold.

10. The eye tracking system of claim 8, wherein the invisible light comprises infrared or near-infrared light.

11. The eye tracking system according to claim 8, The light source array comprises a transparent substrate, and each light source in the light source array is disposed on the transparent substrate.

12. The eye tracking system according to claim 8 further includes a plurality of beam shaping optical devices, each of the plurality of beam shaping optical devices being arranged on a corresponding light source in the light source array to guide the invisible light emitted by each corresponding light source along a corresponding optical axis.

13. The eye tracking system of claim 12, wherein each of the plurality of beam shaping optics is configured to collimate the invisible light into collimated invisible light.

14. The eye tracking system according to claim 13, further comprising: include: An optical element is disposed between the plurality of beam shaping optical devices and the eye-facing side of the eye tracking system to focus the collimated invisible light onto a pupil plane of the eye.

15. A head mounted display (HMD), include: The eye tracking system according to claim 7, wherein each light source in the light source array is configured to be selectively enabled to emit invisible light toward the eye-facing side of the HMD to illuminate the eye; as well as A display layer is disposed between the optical combiner and a back side of the HMD, the display layer being configured to provide display light that propagates through the optical combiner to the eye-facing side of the HMD. 16 . The HMD of claim 15 , wherein each light source in the light source array is disposed on the display layer.