Eye tracking system and method, wearable display device, and electronic device

Abstract

The present invention provides an eye tracking system, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light, from the at least one metasurface, reflected by eyes of a user, and to convert the received light into an electric signal; and a controller configured to control the direction of light reflected by the at least one metasurface and emitted from the at least one light source, such that the reflected light can reach the at least one detector after being re-reflected by the eyes of the user. According to embodiments of the present invention, all-solid-state light beam directing control based on a metasurface has the characteristics of rapid scanning and wide-angle scanning, such that the light beam directing has better accuracy, and the system has higher consistency and reliability and a smaller physical size.

Description

System, method, wearable display device and electronic device for eye tracking

[0001] This application claims priority to the Chinese invention patent application entitled “Systems, methods, wearable display devices and electronic devices for eye tracking” and application number 2023108184464, filed on July 4, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to an eye tracking system, a wearable display device, an eye tracking method, an electronic device, and a computer-readable storage medium. Background Art

[0003] With the rise of augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies, the use of wearable display devices will become a trend. Providing stable eye tracking capabilities and accurate eye movement data for wearable display devices can allow users to move freely and interact naturally with the environment.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a system for eye tracking is provided, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light from the at least one metasurface that is reflected by a user's eyes and convert the received light into an electrical signal; and a controller configured to control the reflection direction of the light irradiated from the at least one light source by the at least one metasurface, so that the reflected light can reach the at least one detector via re-reflection from the user's eyes.

[0006] In a second aspect of the present disclosure, a wearable display device is provided, which includes the system of the first aspect.

[0007] In a third aspect of the present disclosure, a method for eye tracking is provided, comprising: controlling at least one light source to emit light; controlling at least one metasurface to reflect the light emitted by the at least one light source toward a user's eye; controlling at least one detector to receive the light from the at least one metasurface reflected by the user's eye; and converting the received light into an electrical signal.

[0008] In a fourth aspect of the present disclosure, a method for eye tracking is provided, comprising: controlling at least one light source to emit light using a modulation signal; controlling at least one metasurface to reflect light emitted by the at least one light source toward a user's eye; controlling at least one detector to receive light from the at least one metasurface reflected by the user's eye using the modulation signal; and converting the received light into an electrical signal.

[0009] In a fifth aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the third aspect or the fourth aspect.

[0010] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the third aspect or the fourth aspect.

[0011] It should be understood that the contents described in the content of this disclosure are not intended to limit the key features or important features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] FIG1 shows a schematic diagram of an example of an eye tracking system according to some embodiments of the present disclosure;

[0014] FIG2 shows a schematic diagram of an example architecture for controlling an eye tracking system according to some embodiments of the present disclosure;

[0015] FIG3 shows a schematic diagram of an example of a metasurface according to some embodiments of the present disclosure;

[0016] FIG4 shows a schematic diagram of an example of a one-dimensional metasurface according to some embodiments of the present disclosure;

[0017] FIG5 shows a schematic diagram of an example of a two-dimensional metasurface according to some embodiments of the present disclosure;

[0018] FIG6 is a schematic diagram showing an example of a direction in which a metasurface reflects light according to some embodiments of the present disclosure;

[0019] FIG7 is a schematic diagram showing an example of processing an electrical signal output by a detector according to some embodiments of the present disclosure;

[0020] FIG8 shows a schematic diagram of an example of a wearable display device according to some embodiments of the present disclosure;

[0021] FIG9 is a schematic diagram showing a process of an eye tracking method according to some embodiments of the present disclosure;

[0022] FIG10 is a schematic diagram showing the process of an eye tracking method according to some embodiments of the present disclosure; and

[0023] FIG11 shows a block diagram of a device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0029] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0031] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0032] As briefly mentioned above, VR, AR, and MR technologies combine hardware devices and various technical means to fuse virtual content with real scenes, providing users with a unique sensory experience. VR uses computers to simulate a three-dimensional virtual world, providing users with an immersive experience in vision, hearing, and touch. AR allows the real environment and virtual objects to be superimposed in real time and coexist in the same space. MR is a new visual environment that merges the real and virtual worlds, allowing objects in the physical real-world scene to coexist with objects in the virtual world in real time.

[0033] Eye tracking refers to tracking eye movements by measuring the position of the eye's gaze point or the movement of the eye relative to the head. In eye tracking systems, optical switches and mirrors are typically used to redirect a light beam from a light source to scan the eye. Eye tracking can be applied to extended reality (XR) applications, including AR, VR, and MR.

[0034] Some eye-tracking systems use micro-electro-mechanical systems (MEMS) technology, which uses the motion of a scanning mirror to deflect and scan the beam. However, MEMS scanning mirrors are limited in their scanning range, speed, and pattern. Particularly in open-loop systems, they lack feedback and correction for output deviations and disturbances, reducing beam guidance accuracy. Furthermore, the scanning mirror's physical size, the reliability of its moving parts, and process consistency can all hinder its application in near-eye display devices.

[0035] An embodiment of the present disclosure proposes a system for eye tracking. According to various embodiments of the present disclosure, the eye tracking system includes: a light source, a metasurface, a detector, and a controller. The controller controls the reflection direction of the metasurface to light irradiated from at least one light source, so that the reflected light can reach the detector through re-reflection by the user's eyes. The detector receives light from at least one metasurface reflected by the user's eyes and converts the received light into an electrical signal. Thus, the all-solid-state characteristic of the metasurface is utilized to achieve beam guidance with better accuracy and higher reliability. In addition, a beam can be formed in any direction of the field of view at any time, without being restricted by the scanning mode and without the need to operate in resonance, with the advantages of fast scanning and wide scanning angles.

[0036] Example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0037] FIG1 shows a schematic diagram of an example system 100 of an eye tracking system according to some embodiments of the present disclosure. System 100 includes a light source 110, a metasurface 120, a detector 130, and a controller 150. Light emitted by light source 110 is received and reflected by metasurface 120. The reflected light is then reflected again by a user's eye 140 and reaches detector 130.

[0038] The light source 110 can be an infrared or near-infrared light source. Due to the invisibility of infrared light, the use of an infrared light source in an eye tracking system will not interfere with the user's eyes, and the boundaries between the pupil and iris and the iris and sclera in the eye can be accurately distinguished. In some embodiments, the light source 110 can use a vertical cavity surface emitting laser (VCSEL) chip to achieve the emission of a near-infrared laser beam. The VCSEL chip includes an array of emitting units, each emitting unit emits a separate laser beam, and the laser beams emitted by multiple emitting units are incoherent and spatially dispersed. In some embodiments, the system 100 also includes a beam adjuster, which is configured to adjust the direction or shape of the light beam emitted by the light source 110. The beam adjuster is, for example, a collimating lens, which is used to receive the light emitted by the light source 110, collimate it, and then emit it. The beam adjuster is, for example, a diffractive optical element (DOE), which is used to receive the light emitted by the light source 110, homogenize it, and then emit it.

[0039] The metasurface 120 can be implemented as a tunable metasurface. The controller 150 can control the metasurface 120 to dynamically control optical parameters (such as amplitude, phase, and polarization) based on schemes such as mechanical drive, carrier concentration modulation, and phase change materials. For example, the structural units in the metasurface 120 are driven by microelectromechanical or nanoelectromechanical systems to reconfigure the physical shape and spatial arrangement of these structural units to achieve dynamic control. For another example, dynamic control can be achieved by changing the carrier concentration of the structural units in the metasurface 120. For another example, the phase change material is caused to undergo phase change through external stimulation, and the refractive index difference between different phases is used for control. Through these schemes, the tunable metasurface introduces sudden changes in optical parameters to achieve dynamic control, thereby breaking the dependence of traditional optical elements on the propagation optical path. In some embodiments, the metasurface 120 is a reflective metasurface. The metasurface 120 is used to manipulate laser beams. Specifically, the metasurface 120 receives near-infrared light from the light source 110 , and the controller 150 controls the reflection direction of the near-infrared light by controlling the metasurface 120 , thereby reflecting the near-infrared light toward the user's eyes 140 .

[0040] The controller 150 controls the metasurface 120 to control the light source 110 to emit light to scan the eye 140. In some embodiments, the controller 150 controls the light source 110 to emit light to scan the iris region to obtain the gaze direction and estimate the pupil position.

[0041] System 100 may include one or more detectors 130. Detector 130 may be a photodiode (PD) for receiving light reflected from the metasurface 120 by the user's eye 140 and converting the received light into an electrical signal. Detector 130 may also be a position sensing detector (PSD) for providing position information to controller 150. Eye 140 includes pupil 141, iris 142, and cornea 143. Iris 142 is the colored portion surrounding pupil 141. Cornea 143 is the transparent portion covering the outer layer of pupil 141 and iris 142. Light striking different parts of eye 140 produces different types of reflected light. For example, when light strikes cornea 143, specular reflection occurs, forming a light spot 144 on cornea 143. For another example, when light strikes iris 142, diffuse reflection occurs. According to such reflection characteristics, the controller 150 can determine the position of the pupil / iris based on the area or range of the diffusely reflected light. The controller 150 can also determine the position of the cornea based on the position of the specularly reflected light spot.

[0042] In some embodiments, system 100 further includes a lens 160. Lens 160 may be a VR display lens or an AR lens, thereby enabling system 100 to be applied in VR or AR fields. Metasurface 120 and detector 130 may be arranged along lens 160 or within lens 160.

[0043] Figure 2 illustrates an exemplary architecture 200 for controlling an eye-tracking system according to some embodiments of the present disclosure. In architecture 200, controller 150 controls metasurface 120 using scanning signals 210. Furthermore, controller 150 controls light source 110 and detector 130 using modulation signals 230. Furthermore, detector 130 receives light reflected from user's eye 140 and converts it into an electrical signal, which controller 150 then processes. These control and processing processes are described separately below.

[0044] Depending on the specific implementation of metasurface 120, controller 150 may employ an appropriate control method. In some embodiments, metasurface 120 may be implemented as an electrically tunable metasurface. By varying the carrier concentration within metasurface 120, the phase shift can be controlled, thereby controlling the direction of light reflection.

[0045] Figure 3 shows a schematic diagram of an example metasurface 300 according to some embodiments of the present disclosure. Example 300 is an electrically tunable metasurface that combines a transparent conductive oxide (TCO) material with a metal structure. Figure 3 shows a cross-sectional view of the electrically tunable metasurface, including a substrate 310 and a structural unit 320.

[0046] Substrate 310 can include multiple layers of material to provide electrical isolation, carrier accumulation, physical support, and other functions. As an example, substrate 310 includes an insulating layer 301, a TCO layer 302, and a base layer 303. Insulating layer 301, for example, is made of aluminum oxide (Al2O3), which electrically insulates structural unit 320 from TCO layer 302. TCO layer 302 has a conductivity close to that of metal, for example, indium tin oxide (ITO), which has a near-zero dielectric constant in the near-infrared band. Base layer 303, for example, is made of gold (Au). Structural unit 320, which can also be referred to as a nanoantenna, is made, for example, of the same metallic material, Au, as base layer 303.

[0047] By applying an external electric field, the concentration of free carriers in the conductive material can be changed, thereby realizing an electrically tunable metasurface. In the example of FIG3 , a bias voltage V is applied between the structural unit 320 and the base layer 303. bias The charge accumulation layer 330 can be formed by adjusting the bias voltage V bias This can increase the carrier concentration, thereby controlling the direction of light reflection. Based on this principle, the controller 150 uses the scan signal 210 to provide a bias voltage to the structural unit 320. Example 300 includes only one structural unit 320. For a metasurface including multiple structural units, the controller 150 can provide both one-dimensional and two-dimensional biasing schemes.

[0048] FIG4 shows a schematic diagram of an example 400 of a one-dimensional metasurface according to some embodiments of the present disclosure. A one-dimensional metasurface is also known as a metasurface employing a one-dimensional biasing scheme. Example 400 includes a substrate 410, structural units 420, and control lines 430. The structural units 420 and the base layer 401 in the substrate 410 are both made of metal, such as gold, silver, or aluminum. The materials of the two can be the same or different. The base layer is grounded. The control lines 430 are metal conductors. Each control line 430 extends in the Y direction to connect multiple structural units 420, thereby forming a metasurface arranged in a one-dimensional linear periodic pattern. The array period and height of the structural units 420 are both subwavelength-scale. The controller 150 transmits a scanning signal 210 through each control line 430 to simultaneously provide a bias voltage to the multiple structural units 420 connected thereto, thereby implementing a one-dimensional biasing scheme. It should be understood that the multiple structural units 420 in Example 400 can be arranged in a regular, evenly spaced pattern or in a staggered pattern, and this is not a limitation of the present disclosure.

[0049] Figure 5 shows a schematic diagram of an example 500 of a two-dimensional metasurface according to some embodiments of the present disclosure. A two-dimensional metasurface is also known as a metasurface that utilizes a two-dimensional biasing scheme. Example 500 includes a substrate 510, structural units 520, and control lines 530. The base layer 501, structural units 520, and control lines 530 in substrate 510 are all made of metal, such as gold, silver, copper, aluminum, or the like. These materials can be the same or different. The base layer 501 is grounded. Each control line 530 can extend in the Y and X directions and connect to a structural unit 520, thereby forming a two-dimensional linear periodic metasurface. The array period and height of the structural units 520 are both subwavelength-scale. The control lines 530 are metal conductors. The controller 150 can use scanning signals to provide bias voltages to each structural unit 520, enabling more precise control of the metasurface. It should be understood that the multiple structural units 520 in example 500 can be arranged in a regular, evenly spaced pattern or in a staggered pattern, and this is not a limitation of the present disclosure.

[0050] For an eye tracking system, it is necessary to direct light in the direction of the user's eyes. Regardless of whether a one-dimensional metasurface or a two-dimensional metasurface is used, the controller 150 can achieve two-dimensional directional control of the reflected light by controlling the bias voltage applied to the metasurface, thereby achieving optical scanning of the user's eyes (as shown by the curved dotted arrow in Figure 1).

[0051] FIG6 is a schematic diagram of an example 600 of a metasurface reflecting light according to some embodiments of the present disclosure. The structural unit 610 in the example 600 receives and reflects light emitted from the light source 110. The controller 150 can control the light reflection direction to rotate about the X axis, such as the reflection direction 620, with a corresponding pitch angle of θ. The controller 150 can also control the light reflection direction to rotate about the Z axis, such as the reflection direction 630, with a corresponding heading angle of Through these two methods, the controller 150 realizes one-dimensional direction control of the reflected light. The controller 150 can also control the reflection direction of the light to rotate around the X-axis and the Z-axis at the same time, such as the reflection direction 640, thereby realizing two-dimensional direction control of the reflected light.

[0052] The above describes various embodiments of the controller 150 controlling the metasurface 120 using the scanning signal 210. Returning to FIG2 , additionally or alternatively, the controller 150 processes the electrical signal 220 output by the detector 130 to determine the pupil position and the gaze direction.

[0053] As described above, the position of the pupil / iris can be determined based on the diffusely reflected light, and the position of the cornea can be determined based on the specularly reflected light. In order to separate the diffusely reflected light from the specularly reflected light, different detectors 130 can be used.

[0054] 7 shows a schematic diagram of an example architecture 700 for controlling an eye tracking system according to some embodiments of the present disclosure. Example architecture 700 is extended based on example architecture 200 , for example, detector 130 further includes a first detector 705 and a second detector 715 .

[0055] The first detector 705 can be, for example, a diffuse reflection detector. During a period of scanning of the eye 140 by the metasurface 120 controlled by the controller 150, the first detector 705 continuously receives diffusely reflected first light L1. This light may include infrared light from the light source 110 and may also include ambient light. The controller 150 performs peak signal detection 725 and signal processing 730 (e.g., noise removal) on the first electrical signal 710 output by the first detector 705 to generate a pixel-level pupil / iris image 735. Pixels in the image with peak electrical signals less than a first threshold correspond to the pupil area, while pixels with peak electrical signals greater than the first threshold correspond to the iris area. Based on this image, eye feature detection 765 can be performed. The detected eye features can be used to determine the location of the pupil center, pupil size, and other information. Accurate pupil information can be obtained using a small number (e.g., 1-2) of such first detectors 705. This facilitates system miniaturization and reduces power consumption. In some embodiments, the controller 150 may use a related algorithm to determine pupil position and gaze direction 740 based on the result of eye feature detection 765 (i.e., the detected eye features). Pupil position and gaze direction may be used for eye tracking, identity recognition, etc.

[0056] In some embodiments, the light spot / pupil / iris image 735 can be used to reconstruct an eye image 770. For example, the controller 150 uses the scanning signal 750 to control the direction in which the metasurface 120 reflects light emitted by the light source 110, thereby scanning the eye 140 using the reflected light. The controller 150 reconstructs a three-dimensional image of the eye based on the light spot / pupil / iris image 735 generated during the scanning process. Additionally or alternatively, the controller 150 can scan the eye 140 based on a preset scanning pattern. The scanning pattern is determined, for example, based on parameters such as a scanning route, a scanning range, or light intensity.

[0057] The second light L2 reflected from the mirror is received directionally by the second detector 715. The second detector 715 is, for example, a photodiode. The controller 150 can generate a light spot image 735 on the cornea based on the second electrical signal 720 output by the second detector 715. Pixels in the image with electrical signal peaks greater than a second threshold correspond to the center of the light spot. Since the center of corneal curvature lies on the bisector (normal) of the angle between the incident and reflected light directions, the controller 150 uses the scanning signal to control the bias voltage of the metasurface 120, thereby controlling the direction of light reflection, i.e., the direction of light incident on the eye 140. Therefore, the direction of light incident on the cornea can be determined based on the position information 755 in the scanning signal. The second detector 715 can only receive light reflected from the cornea in a directionally direction, so the direction of light reflected from the cornea can be determined based on the physical position of the second detector 715. In some embodiments, the second light L2 reflected from the mirror is received by a position sensing detector. A position sensing detector is an optical detector that can measure the continuous position of a light spot on a detector surface and convert the position of a light spot on a photosensitive surface into an electrical signal. Such a detector can provide position information to controller 150, replacing the physical position of the detector. This eliminates the need to calibrate and store the physical position of the detector during system calibration. During eye tracking, controller 150 uses the position information in the electrical signal output by the position sensing detector to determine the direction of light reflected from the cornea, ultimately determining the direction of gaze.

[0058] In some embodiments, if the radius of the cornea is known, the center of the cornea can be determined based on the distance between the center of the cornea and the light spot being equal to the radius of the cornea. In other words, the center of the cornea can be determined using a single light spot detected by a single second detector 715. This significantly reduces the number of detectors required, facilitating system miniaturization and reducing power consumption.

[0059] In other embodiments, if the radius of the cornea is unknown, multiple second light beams L2 reflected by the mirror are sequentially received by a plurality of dispersed second detectors 715. The controller 150 determines multiple normal directions based on the center positions of these light spots, the direction of the incident light, and the direction of the reflected light. Further, the center position of the cornea is determined based on the intersection of the multiple normal directions. Considering the curvature of the cornea, a relatively small number of photodiodes (e.g., 3-4) can be used to obtain a very accurate corneal center position. This is conducive to system miniaturization and reduced power consumption.

[0060] Through the above method, the controller 150 can determine the line of sight of the user's eye 140 based on the line connecting the center position of the pupil and the center position of the cornea, thereby realizing the eye tracking function.

[0061] In some embodiments, the controller 150 may generate a region of interest (ROI) 745 based on the output of the pupil position and gaze direction 740. The controller 150 may adjust the scanning range of the scanning signal 750 for the user's eye 140 based on the ROI 745, thereby improving the speed and efficiency of eye tracking.

[0062] For eye tracking systems, the continued pursuit of small size and low power consumption facilitates device miniaturization. Additionally or alternatively, in the example of FIG2 , controller 150 utilizes modulation signal 230 to control light source 110 and detector 130. In the example of FIG7 , controller 150 utilizes modulation signal 760 to control light source 110, first detector 705, and second detector 715. While controlling light source 110 to emit light, enabling the corresponding detectors to receive light simultaneously helps reduce system power consumption.

[0063] In the example of FIG2 , the controller 150 uses a modulation signal 230 to provide a driving voltage to the light source 110 to control the light source 110 to emit light. Simultaneously, the controller 150 uses the modulation signal 230 to provide a bias voltage to the detector 130 to turn on the detector 130 to receive light reflected from the user's eye 140 and from the metasurface 120. If the light source 110 is not emitting light, the detector 130 is turned off, thereby reducing system power consumption. It should be understood that, although not shown in FIG2 , the modulation signal 230 provided by the controller 150 can be converted by any appropriate driver circuit before providing the driving voltage to the light source 110. The modulation signal 230 can also be converted by any appropriate switching circuit before providing the bias voltage to the detector 130.

[0064] In the example of FIG7 , the controller 150 uses the modulation signal 760 to provide a driving voltage to the light source 110 to control the light source 110 to emit light. Simultaneously, the controller 150 uses the modulation signal 760 to provide a bias voltage to the first detector 705 to turn on the first detector 705 to receive the first light L1. As described above, the first light L1 may include infrared light from the light source 110, or may include ambient light. Additionally or alternatively, the controller 150 continuously provides a bias voltage to the first detector 705 to ensure that the first detector 705 is always in the on state. Further, the controller 150 uses the modulation signal 760 to provide a bias voltage to the second detector 715 to turn on the second detector 715 to receive the second light L2. As described above, the second detector 715 directionally receives the second light L2 specularly reflected from the cornea. Thus, when the reflected light from the metasurface 120 reaches or near a corresponding location on the cornea, the second detector 715 is activated to receive the reflected light. When the peak value of the second electrical signal 720 output by the second detector 715 exceeds a second threshold, the corresponding location is determined to be the center of the light spot. It should be understood that, although not shown in FIG7 , the modulation signal 760 provided by the controller 150 can be converted by any appropriate drive circuit before providing a drive voltage to the light source 110. The modulation signal 760 can also be converted by any appropriate switch circuit before providing a bias voltage to the first detector 705 and the second detector 715.

[0065] Through the various embodiments described above, control methods for an eye-tracking system have been introduced. It should be understood that, although the control methods are described above using controller 150 as the execution entity, these control methods can be implemented using different controllers. In other words, the eye-tracking system can include multiple controllers 150, each used to control the light source, metasurface, detector, and various signal processing functions.

[0066] An embodiment of the present disclosure also provides a wearable display device, which may include the system for eye tracking described above.

[0067] Figure 8 shows a schematic diagram of an example wearable display device 800 according to some embodiments of the present disclosure. Wearable display device 800 may be, for example, an AR, VR, or MR display device, including but not limited to a head-mounted display, smart glasses, or a user behavior analysis device. The following describes the positional relationships of the various components of wearable display device 800 from a structural perspective.

[0068] Taking the components corresponding to a single eye as an example, in some embodiments, the wearable display device 800 includes a light source 110, a metasurface 120, multiple detectors 130, and a lens 160. The lens 160 may be, for example, a VR display lens or an AR lens. The multiple detectors 130 are positioned adjacent to the edge of the lens 160, so as not to block the user's line of sight while being as close as possible to the user's eye 140, such as the right eye, to facilitate receiving light reflected from the eye 140. Furthermore, the multiple detectors 130 may be arranged around the lens 160. The light source 110 and the metasurface 120 are positioned adjacent to the ends of the lens 160 along the transverse axis H of the lens 160. When the lens 160 is a common elliptical shape, the transverse axis H may generally be the long axis of the ellipse. When the user wears the wearable display device 800, the light source 110 and the metasurface 120 are positioned, for example, near the user's nose.

[0069] Continuing with the example of the components corresponding to a single eye, in some embodiments, the wearable display device 800 further includes a light source 810 and a metasurface 820. The light source 110 and the metasurface 120 are disposed adjacent to one end of the lens 160 along the transverse axis H of the lens 160, and the light source 810 and the metasurface 820 are disposed adjacent to the other end of the lens 160 along the transverse axis H of the lens 160. When a user wears the wearable display device 800, the light source 110 and the metasurface 120 are, for example, near the user's nose, and the light source 810 and the metasurface 820 are, for example, near the user's temple.

[0070] Example Process

[0071] FIG9 shows a flow chart of a process 500 for eye tracking according to some embodiments of the present disclosure. The process 900 may be implemented at the system 100. The process 900 is described below with reference to FIG1.

[0072] At block 910 , the system 100 controls at least one light source to emit light.

[0073] At block 920 , at least one metasurface is controlled to reflect light emitted by at least one light source toward an eye of a user.

[0074] At block 930 , the system 100 controls at least one detector to receive light from at least one metasurface that is reflected by the user's eyes.

[0075] At block 940 , the system 100 converts the received light into an electrical signal.

[0076] In some embodiments, the system 100 controls the bias voltage of at least one structural unit in the metasurface using a scanning signal, thereby controlling the reflection direction of light.

[0077] In some embodiments, the system 100 determines the pupil position and / or the gaze direction of the user's eye by at least one of: position information in the scanning signal, the electrical signal, and the position of the detector that outputs the electrical signal.

[0078] In some embodiments, the at least one detector includes a first detector, and the system 100 controls the first detector to receive first light from the at least one metasurface reflected by the pupil and / or iris of the user's eye; convert the first light into a first electrical signal; and determine the pupil position of the user's eye based at least on the first electrical signal.

[0079] In some embodiments, the at least one detector includes a plurality of second detectors, and the system 100 controls the plurality of second detectors to receive second light reflected from the at least one metasurface by the cornea of ​​the user's eye; convert the second light into a second electrical signal; and determine the direction of the user's eye's gaze based at least on the second electrical signal.

[0080] In some embodiments, the system 100 controls the plurality of second detectors to provide position information of the detectors.

[0081] FIG10 shows a flow chart of a process 1000 for eye tracking according to some embodiments of the present disclosure. The process 1000 may be implemented at the system 100. The process 1000 is described below with reference to FIG1.

[0082] At block 1010 , the system 100 controls at least one light source to emit light using a modulation signal.

[0083] At block 1020 , the system 100 controls at least one metasurface to reflect light emitted by at least one light source toward an eye of a user.

[0084] At block 1030 , the system 100 controls at least one detector to receive light from at least one metasurface reflected by the user's eyes using a modulation signal.

[0085] At block 1040 , the system 100 converts the received light into an electrical signal.

[0086] In some embodiments, at least one light source includes a first light source and a second light source, and at least one metasurface includes a first metasurface associated with the first light source and a second metasurface associated with the second light source. The system 100 controls the first light source and the second light source to emit light respectively using different modulation signals with a phase difference; and simultaneously controls at least one detector to receive light from the first metasurface and from the second metasurface reflected by the user's eyes using different modulation signals with a phase difference.

[0087] In some embodiments, the at least one detector includes a first detector, and the system 100 controls the first detector to receive first light from the at least one metasurface reflected by the pupil and / or iris of the user's eye; convert the first light into a first electrical signal; and determine the pupil position of the user's eye based at least on the first electrical signal.

[0088] In some embodiments, the at least one detector includes a plurality of second detectors, and the system 100 controls the plurality of second detectors to receive second light reflected from the at least one metasurface by the cornea of ​​the user's eye; convert the second light into a second electrical signal; and determine the direction of the user's eye's gaze based at least on the second electrical signal.

[0089] In some embodiments, the system 100 controls the plurality of second detectors to provide position information of the detectors.

[0090] Example device

[0091] FIG11 is a block diagram illustrating an electronic device 1100 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 1100 shown in FIG11 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 1100 shown in FIG11 may be used to implement the system 100 of FIG1 .

[0092] As shown in FIG11 , electronic device 1100 is in the form of a general-purpose computing device. Components of electronic device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1120. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 1100.

[0093] The electronic device 1100 typically includes a plurality of computer storage media. Such media can be any accessible media that is accessible to the electronic device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1120 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1130 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 1100.

[0094] The electronic device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 11 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1120 may include a computer program product 1125 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0095] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the electronic device 1100 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 1100 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0096] Input device 1150 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1160 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 1100 may also communicate with one or more external devices (not shown) via communication unit 1140 as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with electronic device 1100, or with any device that allows electronic device 1100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0097] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0098] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0100] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0101] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0102] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A system for eye tracking, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light from the at least one metasurface reflected by an eye of a user and convert the received light into an electrical signal; as well as A controller is configured to control a reflection direction of the at least one metasurface to the light irradiated from the at least one light source, so that the reflected light can reach the at least one detector via re-reflection by the eyes of the user.

2. The system of claim 1, wherein the controller is configured to: The bias voltage of the structural unit in the at least one metasurface is controlled by using a scanning signal, thereby controlling the reflection direction of the light.

3. The system of claim 2, wherein the controller is further configured to: The pupil position and / or the sight direction of the user's eyes are determined by at least one of the following: The position information in the scanning signal, The electrical signal, The location of the detector that outputs the electrical signal.

4. The system of claim 1, wherein the controller is configured to: controlling the at least one light source to emit light using a modulation signal; and The at least one detector is controlled by the modulation signal to receive light from the at least one metasurface reflected by the user's eyes.

5. The system of claim 1, wherein the at least one light source comprises a first light source and a second light source, and the at least one metasurface comprises a first metasurface associated with the first light source and a second metasurface associated with the second light source, the controller being further configured to: The first light source and the second light source are respectively controlled by different modulation signals with phase difference. A second light source emits light; and The at least one detector is simultaneously controlled to receive the light from the first metasurface and the light from the second metasurface reflected by the user's eyes using the different modulation signals with the phase difference.

6. The system of claim 1, wherein the at least one detector comprises: a first detector configured to receive first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of the user; converting the first light into a first electrical signal; as well as The first electrical signal is provided to the controller to determine the pupil position of the user's eye.

7. The system of claim 1, wherein the at least one detector comprises: a plurality of second detectors configured to receive second light from the at least one metasurface reflected by a cornea of ​​an eye of the user; converting the second light into a second electrical signal; as well as The second electrical signal is provided to the controller to determine the gaze direction of the user's eyes.

8. The system of claim 7, wherein the plurality of second detectors are further configured to provide position information of the detectors to the controller.

9. A wearable display device comprising the system according to any one of claims 1 to 8.

10. The wearable display device according to claim 9, further comprising: lens, The at least one detector is disposed adjacent to an edge of the lens, and the at least one light source and the at least one metasurface are disposed adjacent to an end of the lens along a transverse axis of the lens.

11. A method for eye tracking, comprising: controlling at least one light source to emit light; controlling at least one metasurface to reflect light emitted by the at least one light source toward eyes of a user; controlling at least one detector to receive light from the at least one metasurface reflected by an eye of the user; as well as The received light is converted into an electrical signal.

12. The method of claim 11, wherein controlling at least one metasurface to reflect light emitted by at least one light source toward eyes of the user comprises: The bias voltage of the structural unit in the at least one metasurface is controlled by using a scanning signal, thereby controlling the reflection direction of the light.

13. The method according to claim 12, further comprising: The pupil position and / or the sight direction of the user's eyes are determined by at least one of the following: The position information in the scanning signal, The electrical signal, The location of the detector that outputs the electrical signal.

14. The method of claim 11, wherein the at least one detector comprises a first detector, the method further comprising: Controlling the first detector to receive first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of the user; converting the first light into a first electrical signal; as well as Based at least on the first electrical signal, a pupil position of the user's eye is determined.

15. The method of claim 11, wherein the at least one detector comprises a plurality of second detectors, the method further comprising: controlling the plurality of second detectors to receive second light from the at least one metasurface reflected by a cornea of ​​an eye of the user; converting the second light into a second electrical signal; as well as The gaze direction of the user's eyes is determined based at least on the second electrical signal.

16. The method according to claim 15, further comprising: The plurality of second detectors are controlled to provide position information of the detectors.

17. A method for eye tracking, comprising: controlling at least one light source to emit light using a modulation signal; controlling at least one metasurface to reflect light emitted by the at least one light source toward eyes of a user; controlling at least one detector to receive light from the at least one metasurface reflected by the user's eyes using the modulation signal; as well as The received light is converted into an electrical signal.

18. The method of claim 17, wherein the at least one light source comprises a first light source and a second light source, and the at least one metasurface comprises a first metasurface associated with the first light source and a second metasurface associated with the second light source, the method further comprising: Using different modulation signals with phase differences to control the first light source and the second light source to emit light respectively; as well as The at least one detector is simultaneously controlled to receive the light from the first metasurface and the light from the second metasurface reflected by the user's eyes using the different modulation signals with the phase difference.

19. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 11 to 16 or the method according to any one of claims 17 to 18.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 11 to 16 or the method according to any one of claims 17 to 18.

Images