System, method, wearable display device, and electronic device for eye tracking

Abstract

The present disclosure provides a system for eye movement tracking, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; the at least one detector is configured to receive the light reflected by the eyes of the user from the at least one metasurface and convert the received light into an electric signal; and a controller configured to control a reflection direction of the at least one metasurface to the light irradiated from the at least one light source such that the reflected light can reach the at least one detector via re-reflection of the user's eyes. According to the embodiment of the invention, the all-solid-state light beam guiding control based on the metasurface has the characteristics of rapid scanning and wide-angle scanning, so that the light beam guiding has better accuracy, the consistency and reliability of the system are higher, and the physical size is smaller.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to an eye tracking system, a wearable display device, an eye tracking method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rise of augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies, the use of wearable display devices will become a trend. Providing stable eye tracking capabilities and accurate eye movement data for wearable display devices can allow users to move freely and interact naturally with the environment. Summary of the invention

[0003] In a first aspect of the present disclosure, a system for eye tracking is provided, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light from the at least one metasurface reflected by a user's eyes and convert the received light into an electrical signal; and a controller configured to control a reflection direction of the at least one metasurface with respect to light irradiated from the at least one light source, so that the reflected light can reach the at least one detector via re-reflection by the user's eyes.

[0004] In a second aspect of the present disclosure, a wearable display device is provided, which includes the system of the first aspect.

[0005] In a third aspect of the present disclosure, there is provided an eye tracking method, comprising: controlling at least one light source to emit light; controlling at least one metasurface to reflect light emitted by at least one light source to eyes of a user; controlling at least one detector to receive light from at least one metasurface reflected by eyes of the user; and converting the received light into an electrical signal.

[0006] In a fourth aspect of the present disclosure, there is provided an eye tracking method, comprising: controlling at least one light source to emit light using a modulation signal; controlling at least one metasurface to reflect light emitted by at least one light source to eyes of a user; controlling at least one detector to receive light from at least one metasurface reflected by eyes of the user using a modulation signal; and converting the received light into an electrical signal.

[0007] In a fifth aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the third aspect or the fourth aspect.

[0008] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the third aspect or the fourth aspect.

[0009] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram showing an example of an eye tracking system according to some embodiments of the present disclosure;

[0012] Figure 2 A schematic diagram showing an example architecture for controlling an eye tracking system according to some embodiments of the present disclosure;

[0013] Figure 3 A schematic diagram showing an example of a metasurface according to some embodiments of the present disclosure;

[0014] Figure 4 A schematic diagram showing an example of a one-dimensional metasurface according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram showing an example of a two-dimensional metasurface according to some embodiments of the present disclosure;

[0016] Figure 6 A schematic diagram showing an example of a reflection direction of light by a metasurface according to some embodiments of the present disclosure;

[0017] Figure 7 A schematic diagram showing an example of processing an electrical signal output by a detector according to some embodiments of the present disclosure;

[0018] Figure 8A schematic diagram showing an example of a wearable display device according to some embodiments of the present disclosure;

[0019] Fig. 9 A schematic diagram showing the process of an eye tracking method according to some embodiments of the present disclosure;

[0020] Fig.10 A schematic diagram showing the process of an eye tracking method according to some embodiments of the present disclosure; and

[0021] Fig.11 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0029] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0030] As briefly mentioned above, VR technology, AR technology and MR technology combine virtual content and real scenes by using hardware equipment and a combination of various technical means to provide users with a unique sensory experience. VR is a virtual world that uses computers to simulate three-dimensional space, providing users with an immersive experience in vision, hearing, touch, etc. AR allows the real environment and virtual objects to be superimposed in the same space in real time and exist simultaneously. MR is a new visualization environment that integrates the real world and the virtual world, where objects in the physical real world scene coexist with objects in the virtual world in real time.

[0031] Eye tracking refers to tracking eye movements by measuring the position of the eye's gaze point or the movement of the eye relative to the head. In an eye tracking system, optical switches and mirrors are usually used to change the direction of the light beam from the light source to complete the scanning of the human eye. Eye tracking can be applied to the field of Extended Reality (XR), including AR, VR, and MR.

[0032] Some eye tracking systems use micro-electro-mechanical system (MEMS) technology to achieve deflection scanning of the beam direction through the movement of the scanning mirror. However, the scanning range, scanning speed and scanning mode of the MEMS scanning mirror are limited. Especially in the open-loop system, it cannot provide feedback and correction for output deviations and disturbances, which reduces the accuracy of beam guidance. In addition, the physical size of the scanning mirror, the reliability of the moving parts or the process consistency may become the limiting factors for its application in near-eye display devices.

[0033] An embodiment of the present disclosure proposes a system for eye tracking. According to various embodiments of the present disclosure, the eye tracking system includes: a light source, a metasurface, a detector, and a controller. The controller controls the reflection direction of the metasurface to light irradiated from at least one light source, so that the reflected light can reach the detector through re-reflection by the user's eyes. The detector receives light from at least one metasurface reflected by the user's eyes, and converts the received light into an electrical signal. Thus, the beam guidance is achieved by utilizing the all-solid-state characteristics of the metasurface, with better accuracy and higher reliability. In addition, a beam can be formed in any direction of the field of view at any time, without being restricted by the scanning mode, and does not need to operate in resonance, with the advantages of fast scanning and wide scanning angles.

[0034] Example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0035] Figure 1 A schematic diagram of an example system 100 of an eye tracking system according to some embodiments of the present disclosure is shown. The system 100 includes a light source 110, a metasurface 120, a detector 130, and a controller 150. Light emitted by the light source 110 is received and reflected by the metasurface 120. The reflected light is reflected again by the user's eye 140 and reaches the detector 130.

[0036] The light source 110 may be an infrared or near-infrared light source. Due to the invisibility of infrared light, the use of an infrared light source in an eye tracking system will not interfere with the user's eyes, and the boundaries between the pupil and the iris and the iris and the sclera in the eye can be accurately distinguished. In some embodiments, the light source 110 may use a vertical cavity surface emitting laser (VCSEL) chip to achieve the emission of a near-infrared laser beam. The VCSEL chip includes an array of emitting units, each emitting unit emits a separate laser beam, and the laser beams emitted by multiple emitting units are incoherent and spatially dispersed. In some embodiments, the system 100 also includes a beam adjuster, which is configured to adjust the direction or shape of the light beam emitted by the light source 110. The beam adjuster is, for example, a collimating lens, which is used to receive the light emitted by the light source 110, collimate it, and then emit it. The beam adjuster is, for example, a diffractive optical element (DOE), which is used to receive the light emitted by the light source 110, homogenize it, and then emit it.

[0037] The metasurface 120 can be implemented as a tunable metasurface. The controller 150 can control the metasurface 120 to dynamically regulate optical parameters (such as amplitude, phase, polarization) based on mechanical drive, carrier concentration modulation, phase change materials, etc. For example, the structural units in the metasurface 120 are driven by micro-electromechanical or nano-electromechanical systems, and the physical shape and spatial arrangement of these structural units are reconfigured to achieve dynamic regulation. For another example, dynamic regulation is achieved by changing the carrier concentration of the structural units in the metasurface 120. For another example, the phase change material is caused to undergo a phase change through external excitation, and the refractive index difference between different phases is used for regulation. Through these schemes, the tunable metasurface introduces a sudden change in optical parameters to achieve dynamic regulation, thereby breaking the dependence of traditional optical elements on the propagation optical path. In some embodiments, the metasurface 120 is a reflective metasurface. The metasurface 120 is used to manipulate a laser beam. Specifically, the metasurface 120 receives the near-infrared light from the light source 110 , and the controller 150 controls the reflection direction of the near-infrared light by controlling the metasurface 120 , thereby reflecting the near-infrared light to the user's eyes 140 .

[0038] The controller 150 controls the light emitted by the light source 110 to scan the eye 140 by controlling the metasurface 120. In some embodiments, the controller 150 controls the light emitted by the light source 110 to scan the iris area to obtain the sight direction and estimate the pupil position.

[0039] The system 100 may include one or more detectors 130. The detector 130 may be a photodiode (PD) for receiving light from the metasurface 120 reflected by the user's eye 140 and converting the received light into an electrical signal. The detector 130 may also be a position sensing detector (PSD) for providing position information to the controller 150. The eye 140 includes a pupil 141, an iris 142, and a cornea 143. The iris 142 is a colored part surrounding the pupil 141. The cornea 143 is a transparent part covering the outer layer of the pupil 141 and the iris 142. When light is irradiated to different parts of the eye 140, different types of reflected light will be generated. For example, when light is irradiated to the cornea 143, specular reflection will occur, and a light spot 144 will be formed on the cornea 143. For another example, when light is irradiated to the iris 142, diffuse reflection will occur. According to such reflection characteristics, the controller 150 can determine the position of the pupil / iris based on the area or range of diffusely reflected light, and the controller 150 can also determine the position of the cornea based on the position of the specularly reflected light spot.

[0040] In some embodiments, the system 100 further includes a lens 160. The lens 160 may be a VR display lens or include an AR lens, so that the system 100 may be applied in the VR or AR field. The metasurface 120 and the detector 130 may be arranged along the lens 160, or may be arranged within the lens 160.

[0041] Figure 2 A schematic diagram of an example architecture 200 for controlling an eye tracking system according to some embodiments of the present disclosure is shown. In architecture 200, controller 150 controls metasurface 120 using scanning signal 210. Further, controller 150 controls light source 110 and detector 130 using modulation signal 230. Further, detector 130 receives light reflected by user's eye 140 and converts it into an electrical signal, and controller 150 processes electrical signal 220. These control processes and processing processes are introduced separately below.

[0042] Depending on the specific implementation of the metasurface 120, the controller 150 may adopt a suitable control method. In some embodiments, the metasurface 120 may be implemented as an electrically tunable metasurface. The phase shift may be controlled by changing the carrier concentration in the metasurface 120, thereby achieving control over the reflection direction of light.

[0043] Figure 3A schematic diagram of an example 300 of a metasurface according to some embodiments of the present disclosure is shown. Example 300 is an electrically tunable metasurface combining a transparent conductive glass (TCO) material with a metal structure. Figure 3 Shown is a cross-sectional view of an electrically tunable metasurface, example 300 including a substrate 310 and a structural unit 320 .

[0044] The substrate 310 may include multiple layers of materials for providing electrical isolation, accumulating carriers, physical support, etc. As an example, the substrate 310 includes an insulating layer 301, a TCO layer 302, and a base layer 303. The insulating layer 301 may be made of, for example, aluminum oxide (Al 2 O 3 ), the structural unit 320 can be electrically insulated from the TCO layer 302. The TCO layer 302 has an electrical conductivity close to that of metal, for example, indium tin oxide (ITO) is selected, which has a near-zero dielectric constant point in the near-infrared band. The base layer 303 is, for example, gold (Au). The structural unit 320 can also be called a nanoantenna, for example, the same metal material Au as the base layer 303 is selected.

[0045] The concentration of free carriers in a conductive material can be changed by applying an external electric field, thereby achieving electrically tunable metasurfaces. Figure 3 In the example of FIG. 1 , a bias voltage V is applied between the structural unit 320 and the base layer 303 bias The charge accumulation layer 330 can be formed by adjusting the bias voltage V bias The carrier concentration can be increased, thereby achieving control over the reflection direction of light. Based on this principle, the controller 150 uses the scanning signal 210 to provide a bias voltage to the structure unit 320. Example 300 includes only one structure unit 320. For a metasurface including multiple structure units, the controller 150 can provide a one-dimensional bias scheme and a two-dimensional bias scheme.

[0046] Figure 4A schematic diagram of an example 400 of a one-dimensional metasurface according to some embodiments of the present disclosure is shown. A one-dimensional metasurface is a metasurface that adopts a one-dimensional biasing scheme. Example 400 includes a substrate 410, a structural unit 420, and a control line 430. The structural unit 420 and the base layer 401 in the substrate 410 are both metals, such as gold, silver, aluminum, etc. The materials of the two can be the same or different. The base layer is grounded. The control line 430 is a metal wire. Each control line 430 extends in the Y direction to connect multiple structural units 420, thereby forming a metasurface arranged in a one-dimensional linear periodic manner. The array period and height of the structural unit 420 are both sub-wavelength scales. The controller 150 transmits a scanning signal 210 through each control line 430 to simultaneously provide a bias voltage to the multiple structural units 420 connected thereto, thereby realizing a one-dimensional biasing scheme. It should be understood that the multiple structural units 420 in Example 400 can be arranged regularly at equal intervals or staggered, and the present disclosure is not limited to this.

[0047] Figure 5 A schematic diagram of an example 500 of a two-dimensional metasurface according to some embodiments of the present disclosure is shown. A two-dimensional metasurface is a metasurface that adopts a two-dimensional biasing scheme. Example 500 includes a substrate 510, a structural unit 520, and a control line 530. The materials of the base layer 501, the structural unit 520, and the control line 530 in the substrate 510 are all metals, such as gold, silver, copper, aluminum, etc. The materials of the three can be the same or different. The base layer 501 is grounded. Each control line 530 can extend in the Y direction and the X direction and connect a structural unit 520, thereby forming a two-dimensional linear periodically arranged metasurface. The array period and height of the structural unit 520 are both sub-wavelength scales. The control line 530 is a metal wire. The controller 150 can use a scanning signal to provide a bias voltage to each structural unit 520 respectively, so as to achieve more precise control of the metasurface. It should be understood that the multiple structural units 520 in Example 500 can be arranged regularly at equal intervals or staggered, and the present disclosure is not limited to this.

[0048] For an eye tracking system, it is necessary to direct light in the direction of the user's eyes. Whether a one-dimensional metasurface or a two-dimensional metasurface is used, the controller 150 can control the two-dimensional direction of the reflected light by controlling the bias voltage applied to the metasurface, thereby achieving optical scanning of the user's eyes (e.g., Figure 1 (indicated by the curved dashed arrow in the middle).

[0049] Figure 6Schematic diagram of example 600 of the reflection direction of light by a metasurface according to some embodiments of the present disclosure. The structural unit 610 in example 600 receives and reflects light emitted from the light source 110. The controller 150 can control the reflection direction of the light to rotate around the X axis, such as the reflection direction 620, and the corresponding pitch angle is θ. The controller 150 can also control the reflection direction of the light to rotate around the Z axis, such as the reflection direction 630, and the corresponding heading angle is Through these two methods, the controller 150 realizes one-dimensional direction control of the reflected light. The controller 150 can also control the reflection direction of the light to rotate around the X-axis and the Z-axis at the same time, such as the reflection direction 640, thereby realizing two-dimensional direction control of the reflected light.

[0050] In the above, various embodiments are used to introduce how the controller 150 uses the scanning signal 210 to control the metasurface 120. Figure 2 Additionally or alternatively, the controller 150 performs signal processing on the electrical signal 220 output by the detector 130 to determine the pupil position and the line of sight direction.

[0051] As described above, the position of the pupil / iris can be determined based on the diffusely reflected light, and the position of the cornea can be determined based on the specularly reflected light. To separate the diffusely reflected light from the specularly reflected light, different detectors 130 can be used.

[0052] Figure 7 A schematic diagram of an example architecture 700 for controlling an eye tracking system according to some embodiments of the present disclosure is shown. The example architecture 700 is extended on the basis of the example architecture 200 , for example, the detector 130 further includes a first detector 705 and a second detector 715 .

[0053] The first detector 705 may be, for example, a diffuse reflection detector. During the process of the controller 150 controlling the metasurface 120 to perform a cycle of scanning on the eye 140, the diffusely reflected first light L1 is continuously received by the first detector 705. Such light may include infrared light from the light source 110 and may also include ambient light. The controller 150 performs peak signal detection 725 and signal processing 730 (e.g., noise removal) on the first electrical signal 710 output by the first detector 705 to generate a pixel-level pupil / iris image 735. Pixels in the image whose electrical signal peak value is less than the first threshold correspond to the pupil area, and pixels whose electrical signal peak value is greater than the first threshold correspond to the iris area. Based on the image, eye feature detection 765 may be performed, and the position of the pupil center, the pupil size, etc. may be determined based on the detected eye features. Accurate pupil information can be obtained by using a small number (e.g., 1-2) of such first detectors 705. This is conducive to the miniaturization of the system and the reduction of power consumption. In some embodiments, the controller 150 may use a related algorithm to determine the pupil position and the gaze direction 740 based on the result of the eye feature detection 765 (ie, the detected eye features). The pupil position and the gaze direction may be used for eye tracking, identity recognition, and the like.

[0054] In some embodiments, the light spot / pupil / iris image 735 can be used for eye image reconstruction 770. For example, the controller 150 controls the reflection direction of the light emitted by the light source 110 by the metasurface 120 using the scanning signal 750, and scans the eye 140 through the reflected light. The controller 150 reconstructs a three-dimensional image of the eye based on the light spot / pupil / iris image 735 generated during the scanning process. Additionally or alternatively, the controller 150 can scan the eye 140 based on a preset scanning mode. The scanning mode is determined based on parameters such as a scanning route, a scanning range, or light intensity.

[0055] The second light L2 reflected by the mirror is received in a directional manner by the second detector 715. The second detector 715 is, for example, a photodiode. The controller 150 can generate a spot image 735 on the cornea according to the second electrical signal 720 output by the second detector 715. The pixel in the image whose electrical signal peak value is greater than the second threshold corresponds to the center position of the spot. Since the center position of the corneal curvature is on the bisector (normal) of the angle between the incident light direction and the reflected light direction. The controller 150 uses the scanning signal to control the bias voltage of the metasurface 120, thereby controlling the reflection direction of the light, that is, the incident direction reaching the eye 140. Therefore, the incident light direction of the cornea can be determined based on the position information 755 in the scanning signal. The second detector 715 can only receive the reflected light from the cornea in a directional manner, so the reflected light direction of the cornea can be determined based on the physical position of the second detector 715. In some embodiments, the second light L2 reflected by the mirror is received by a position sensing detector. The position sensing detector is an optical detector that can measure the continuous position of a light spot on the detector surface, and can convert the position of a light spot on the photosensitive surface into an electrical signal. Such a detector can provide position information to the controller 150 to replace the physical position of the detector. Therefore, when the system is calibrated, it is not necessary to calibrate and store the physical position of the detector. When tracking the eye movement, the controller 150 determines the direction of the reflected light from the cornea through the position information in the electrical signal output by the position sensing detector, so as to finally determine the line of sight direction.

[0056] In some embodiments, if the radius of the cornea is known, the center position of the cornea can be determined based on the distance between the center position of the cornea and the light spot being equal to the radius of the cornea. That is, the position of the center of the cornea can be determined by a light spot detected by a single second detector 715. This greatly reduces the number of detectors required, which is conducive to miniaturization of the system and reduction of power consumption.

[0057] In other embodiments, if the radius of the cornea is an unknown quantity, multiple second lights L2 reflected by the mirror are received in sequence by multiple second detectors 715 that are dispersedly arranged. The controller 150 determines multiple normal directions based on the center positions of these light spots, the direction of the incident light, and the direction of the reflected light. Further, the center position of the cornea is determined based on the intersection of the multiple normal directions. Considering the curvature of the cornea, a very accurate center position of the cornea can be obtained by using a smaller number (e.g., 3-4) of photodiodes. This is conducive to the miniaturization of the system and the reduction of power consumption.

[0058] Through the above method, the controller 150 can determine the sight direction of the user's eye 140 according to the line between the center position of the pupil and the center position of the cornea, thereby realizing the eye tracking function.

[0059] In some embodiments, the controller 150 may generate a region of interest (ROI) 745 based on the output of determining the pupil position and the gaze direction 740. The controller 150 may adjust the scanning range of the scanning signal 750 for the user's eye 140 based on the region of interest 745, thereby improving the speed and efficiency of eye tracking.

[0060] For eye tracking systems, the continued pursuit of small size and low power consumption is conducive to the miniaturization of devices. Additionally or alternatively, in Figure 2 In the example of FIG. 1 , the controller 150 controls the light source 110 and the detector 130 using the modulation signal 230. Figure 7 In the example of FIG. 7 , the controller 150 uses the modulation signal 760 to control the light source 110, the first detector 705, and the second detector 715. While controlling the light source 110 to emit light, turning on the corresponding detector for receiving light is beneficial to reducing system power consumption.

[0061] exist Figure 2 In the example of FIG. 1 , the controller 150 uses the modulation signal 230 to provide a driving voltage to the light source 110 to control the light source 110 to emit light. Simultaneously, the controller 150 uses the modulation signal to provide a bias voltage to the detector 130 to turn on the detector 130 to receive light from the metasurface 120 reflected by the user's eyes 140. If the light source 110 does not emit light, the detector 130 is turned off, thereby reducing system power consumption. It should be understood that although Figure 2 Not shown, the modulation signal 230 provided by the controller 150 may be converted via any appropriate driving circuit to provide a driving voltage to the light source 110 , and the modulation signal 230 may also be converted via any appropriate switching circuit to provide a bias voltage to the detector 130 .

[0062] exist Figure 7In the example, the controller 150 uses the modulation signal 760 to provide a driving voltage to the light source 110 to control the light source 110 to emit light. Synchronously, the controller 150 uses the modulation signal 760 to provide a bias voltage to the first detector 705 to turn on the first detector 705 to receive the first light L1. As described above, the first light L1 may include infrared light from the light source 110, and may also include ambient light. Additionally or alternatively, the controller 150 continuously provides a bias voltage to the first detector 705 to ensure that the first detector 705 is always in the on state. Further, the controller 150 uses the modulation signal 760 to provide a bias voltage to the second detector 715 to turn on the second detector 715 to receive the second light L2. As described above, the second detector 715 receives the second light L2 reflected from the cornea in a directional manner. Therefore, when the reflected light from the metasurface 120 reaches the corresponding position or vicinity of the cornea, the second detector 715 is turned on to receive the light, and when the peak value of the second electrical signal 720 outputted by the second detector 715 is greater than the second threshold value, the corresponding position is determined to be the center position of the light spot. It should be understood that although Figure 7 Not shown, the modulation signal 760 provided by the controller 150 may be converted via any appropriate driving circuit to provide a driving voltage to the light source 110 , and the modulation signal 760 may also be converted via any appropriate switching circuit to provide a bias voltage to the first detector 705 and the second detector 715 .

[0063] Through the above various embodiments, the control method of the eye tracking system is introduced. It should be understood that although the above control method is described by taking the controller 150 as the execution subject, these control methods can be implemented by different controllers, or in other words, the eye tracking system can include multiple controllers 150, which are respectively used to control the light source, the metasurface, the detector and various signal processing.

[0064] An embodiment of the present disclosure also provides a wearable display device, which may include the system for eye tracking described above.

[0065] Figure 8 A schematic diagram of an example 800 of a wearable display device according to some embodiments of the present disclosure is shown. The wearable display device 800 may be, for example, an AR, VR, or MR display device, including but not limited to a head-mounted display, smart glasses, a user behavior analysis device, etc. The positional relationship of each component in the wearable display device 800 is described below from a structural perspective.

[0066] Taking the components corresponding to a single eye as an example, in some embodiments, the wearable display device 800 includes a light source 110, a metasurface 120, a plurality of detectors 130, and a lens 160. The lens 160 is, for example, a VR display lens or includes an AR lens. The plurality of detectors 130 are arranged adjacent to the edge of the lens 160, so as not to block the line of sight and to be as close as possible to the user's eye 140, such as the right eye, so as to receive light reflected from the eye 140. Further, the plurality of detectors 130 can be arranged around the lens 160. The light source 110 and the metasurface 120 are arranged adjacent to the end of the lens 160 along the horizontal axis H of the lens 160. In the case where the lens 160 is a common ellipse-like shape, the horizontal axis H can generally be the long axis of the ellipse. In the case where the user wears the wearable display device 800, the light source 110 and the metasurface 120 are arranged, for example, close to the user's nose.

[0067] Continuing with the example of the components corresponding to a single eye, in some embodiments, the wearable display device 800 further includes a light source 810 and a metasurface 820. The light source 110 and the metasurface 120 are disposed adjacent to one end of the lens 160 along the transverse axis H of the lens 160, and the light source 810 and the metasurface 820 are disposed adjacent to the other end of the lens 160 along the transverse axis H of the lens 160. When the user wears the wearable display device 800, the light source 110 and the metasurface 120 are, for example, close to the user's nose, and the light source 810 and the metasurface 820 are, for example, close to the user's temple.

[0068] Example Process

[0069] Fig. 9 900 can be implemented at system 100. Figure 1 Process 900 is described.

[0070] At block 910 , the system 100 controls at least one light source to emit light.

[0071] At block 920 , at least one metasurface is controlled to reflect light emitted by at least one light source toward an eye of a user.

[0072] At block 930 , the system 100 controls at least one detector to receive light from at least one metasurface reflected by an eye of the user.

[0073] At block 940 , the system 100 converts the received light into an electrical signal.

[0074] In some embodiments, the system 100 controls the bias voltage of at least one structural unit in the metasurface using a scanning signal, thereby controlling the reflection direction of light.

[0075] In some embodiments, the system 100 determines the pupil position and / or the gaze direction of the user's eye by at least one of the following: position information in a scanning signal, an electrical signal, and a position of a detector that outputs the electrical signal.

[0076] In some embodiments, the at least one detector includes a first detector, and the system 100 controls the first detector to receive a first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of a user; convert the first light into a first electrical signal; and determine a pupil position of the eye of the user based at least on the first electrical signal.

[0077] In some embodiments, the at least one detector includes a plurality of second detectors, and the system 100 controls the plurality of second detectors to receive a second light from the at least one metasurface reflected by the cornea of ​​the user's eye; converts the second light into a second electrical signal; and determines the line of sight direction of the user's eye based at least on the second electrical signal.

[0078] In some embodiments, the system 100 controls a plurality of second detectors to provide position information of the detectors.

[0079] Fig.10 1 is a flowchart of a process 1000 for eye tracking according to some embodiments of the present disclosure. The process 1000 may be implemented at the system 100. Figure 1 Process 1000 is described.

[0080] At block 1010 , the system 100 controls at least one light source to emit light using a modulation signal.

[0081] At block 1020 , the system 100 controls at least one metasurface to reflect light emitted by at least one light source toward an eye of a user.

[0082] At block 1030 , the system 100 controls at least one detector to receive light from at least one metasurface reflected by an eye of a user using a modulation signal.

[0083] At block 1040 , the system 100 converts the received light into an electrical signal.

[0084] In some embodiments, at least one light source includes a first light source and a second light source, and at least one metasurface includes a first metasurface associated with the first light source and a second metasurface associated with the second light source, the system 100 uses different modulation signals with a phase difference to respectively control the first light source and the second light source to emit light; and uses different modulation signals with a phase difference to simultaneously control at least one detector to receive light from the first metasurface and from the second metasurface reflected by the user's eyes.

[0085] In some embodiments, the at least one detector includes a first detector, and the system 100 controls the first detector to receive a first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of a user; convert the first light into a first electrical signal; and determine a pupil position of the eye of the user based at least on the first electrical signal.

[0086] In some embodiments, the at least one detector includes a plurality of second detectors, and the system 100 controls the plurality of second detectors to receive a second light from the at least one metasurface reflected by the cornea of ​​the user's eye; converts the second light into a second electrical signal; and determines the line of sight direction of the user's eye based at least on the second electrical signal.

[0087] In some embodiments, the system 100 controls a plurality of second detectors to provide position information of the detectors.

[0088] Example Device

[0089] Fig.11 1 shows a block diagram of an electronic device 1100 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig.11 The electronic device 1100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig.11 The electronic device 1100 shown can be used to implement Figure 1 system 100.

[0090] like Fig.11 As shown, the electronic device 1100 is in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to, one or more processors or processing units 1110, a memory 1120, a storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. The processing unit 1110 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 1120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 1100.

[0091] The electronic device 1100 typically includes a plurality of computer storage media. Such media may be any accessible media accessible to the electronic device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1120 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1130 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 1100.

[0092] The electronic device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.11 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1120 may include a computer program product 1125 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0093] The communication unit 1140 implements communication with other computing devices through a communication medium. Additionally, the functions of the components of the electronic device 1100 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 1100 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0094] The input device 1150 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 1160 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 1100 may also communicate with one or more external devices (not shown) through the communication unit 1140 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 1100, or communicate with any device that allows the electronic device 1100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0095] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0096] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0099] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0100] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A system for eye tracking, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light from the at least one metasurface reflected by an eye of a user and convert the received light into an electrical signal; as well as A controller is configured to control a reflection direction of the at least one metasurface to the light irradiated from the at least one light source, so that the reflected light can reach the at least one detector via re-reflection by the eyes of the user.

2. The system of claim 1, wherein the controller is configured to: The bias voltage of the structural unit in the at least one metasurface is controlled by using a scanning signal, thereby controlling the reflection direction of the light.

3. The system of claim 2, wherein the controller is further configured to: The pupil position and / or the sight direction of the user's eyes are determined by at least one of the following: The position information in the scanning signal, The electrical signal, The location of the detector that outputs the electrical signal.

4. The system of claim 1, wherein the controller is configured to: controlling the at least one light source to emit light using a modulation signal; and The at least one detector is controlled by the modulation signal to receive light from the at least one metasurface reflected by the user's eyes.

5. The system of claim 1, wherein the at least one light source comprises a first light source and a second light source, and the at least one metasurface comprises a first metasurface associated with the first light source and a second metasurface associated with the second light source, the controller being further configured to: Using different modulation signals with phase differences to control the first light source and the second light source to emit light respectively; and The at least one detector is simultaneously controlled to receive the light from the first metasurface and the light from the second metasurface reflected by the user's eyes using the different modulation signals with the phase difference.

6. The system of claim 1, wherein the at least one detector comprises: a first detector configured to receive first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of the user; converting the first light into a first electrical signal; as well as The first electrical signal is provided to the controller to determine the pupil position of the user's eye.

7. The system of claim 1, wherein the at least one detector comprises: a plurality of second detectors configured to receive second light from the at least one metasurface reflected by a cornea of ​​an eye of the user; converting the second light into a second electrical signal; as well as The second electrical signal is provided to the controller to determine the gaze direction of the user's eyes.

8. The system of claim 7, wherein the plurality of second detectors are further configured to provide position information of the detectors to the controller.

9. A wearable display device comprising the system according to any one of claims 1 to 8.

10. The wearable display device according to claim 9, further comprising: lens, The at least one detector is disposed adjacent to an edge of the lens, and the at least one light source and the at least one metasurface are disposed adjacent to an end of the lens along a transverse axis of the lens.

11. A method for eye tracking, comprising: controlling at least one light source to emit light; controlling at least one metasurface to reflect light emitted by the at least one light source toward eyes of a user; controlling at least one detector to receive light from the at least one metasurface reflected by the user's eyes; as well as The received light is converted into an electrical signal.

12. The method of claim 11, wherein controlling at least one metasurface to reflect light emitted by at least one light source toward eyes of the user comprises: The bias voltage of the structural unit in the at least one metasurface is controlled by using a scanning signal, thereby controlling the reflection direction of the light.

13. The method according to claim 12, further comprising: The pupil position and / or the sight direction of the user's eyes are determined by at least one of the following: The position information in the scanning signal, The electrical signal, The location of the detector that outputs the electrical signal.

14. The method of claim 11, wherein the at least one detector comprises a first detector, the method further comprising: Controlling the first detector to receive first light from the at least one metasurface reflected by a pupil and / or an iris of an eye of the user; converting the first light into a first electrical signal; as well as Based at least on the first electrical signal, a pupil position of the user's eye is determined.

15. The method of claim 11, wherein the at least one detector comprises a plurality of second detectors, the method further comprising: controlling the plurality of second detectors to receive second light from the at least one metasurface reflected by a cornea of ​​an eye of the user; converting the second light into a second electrical signal; as well as The gaze direction of the user's eyes is determined based at least on the second electrical signal.

16. The method according to claim 15, further comprising: The plurality of second detectors are controlled to provide position information of the detectors.

17. A method for eye tracking, comprising: controlling at least one light source to emit light using a modulation signal; controlling at least one metasurface to reflect light emitted by the at least one light source toward eyes of a user; controlling at least one detector to receive light from the at least one metasurface reflected by the user's eyes using the modulation signal; as well as The received light is converted into an electrical signal.

18. The method of claim 17, wherein the at least one light source comprises a first light source and a second light source, and the at least one metasurface comprises a first metasurface associated with the first light source and a second metasurface associated with the second light source, the method further comprising: Using different modulation signals with phase differences to control the first light source and the second light source to emit light respectively; as well as The at least one detector is simultaneously controlled to receive the light from the first metasurface and the light from the second metasurface reflected by the user's eyes using the different modulation signals with the phase difference.

19. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 11 to 16 or the method according to any one of claims 17 to 18.

20. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 11 to 16 or the method according to any one of claims 17 to 18.

Images