An eye tracking method and system

Abstract

An eye tracking method and system, the method comprising: acquiring a first image containing a user's eyeball when a first illumination light source of at least one illumination light source is turned on and other illumination light sources are turned off; the first image containing a first glint within the user's eyeball, the first glint being formed based on light emitted by the first illumination light source; determining first glint position information according to the first image, the first glint position information being used to represent a position of the first glint relative to a pupil of the user's eyeball; determining a target position of a gaze of the user's eyeball according to the first glint position information and mapping information, the mapping information being used to represent a mapping relationship between glint position information and positions of gazes of the user's eyeball, the mapping information being obtained based on training of a plurality of sample data. Therefore, by the method, a position of a gaze of a user's eye can be accurately and quickly determined.

Description

Technical Field

[0001] This application relates to the field of visual tracking technology, and more particularly to an eye-tracking method and system. Background Technology

[0002] With the widespread application of eye-tracking technology, such as in the medical and aerospace fields, as well as virtual reality technology, research into eye-tracking technology is becoming increasingly important. Eye-tracking technology uses sensors to capture and extract eye information, measure eye movements, and thus estimate the user's gaze direction or the position of the eye's fixation point.

[0003] Currently, existing eye-tracking methods include Pugin image method, contact lens method, scleral reflection method, pupil-corneal reflection method, and image-based methods. The principle behind current eye-tracking methods is based on determining the user's gaze position or direction by accurately locating the light source closest to the center of the user's pupil. However, due to differences in eye structure among users, the mapping relationship between the position of the bright spot formed by the light source on the eyeball relative to the pupil center and the user's gaze position varies from person to person. Therefore, before performing eye tracking, this mapping relationship needs to be statistically determined, resulting in a relatively large time delay in eye tracking. Furthermore, when the change in the user's gaze position or direction is small, the position of the bright spot on the user's eyeball is also relatively close to the pupil center, making it difficult to accurately determine the user's gaze position or direction. Summary of the Invention

[0004] An eye-tracking method and system that can quickly and accurately locate the position or direction of a user's gaze.

[0005] In a first aspect, this application provides an eye-tracking method, comprising: acquiring a first image of a user's eye portion when a first illumination source in at least one illumination source is turned on and other illumination sources are turned off; wherein the first image contains a first bright spot in the user's eyeball, the first bright spot being formed based on light emitted by the first illumination source; determining the position information of the first bright spot based on the first image, the first bright spot position information being used to represent the position of the first bright spot relative to the pupil of the user's eyeball; determining the target position of the user's eyeball gaze based on the first bright spot position information and mapping information; the mapping information being used to represent the mapping relationship between the bright spot position information and the position of the user's eyeball gaze, the mapping information being obtained based on training with multiple sample data, any sample data including: a position sample of the eyeball gazing and a bright spot position information sample determined based on image samples, wherein the image sample is an image acquired when the eyeball gazing at the position sample.

[0006] In this embodiment of the application, during eye tracking, an illumination source (such as a first illumination source) is turned on, and a first image containing the user's eyeball is acquired (i.e., an image of the user's eye at this moment is acquired). Then, based on the first image, position information of a first bright spot representing the bright spot in the user's eyeball and the pupil is determined. Then, based on the position information of the first bright spot and the trained mapping information, the target position of the user's eyeball is determined. In this method, since it is not necessary to turn on each illumination source and acquire images of the user's eye multiple times, eye tracking can be achieved, thereby greatly saving the time delay caused by turning on the illumination source and acquiring images of the user's eye.

[0007] Furthermore, the mapping information in this embodiment is trained based on multiple sample data, and this mapping information can accurately reflect the mapping relationship between the bright spot position information and the position of the user's eye gaze. Therefore, after determining the first bright spot position information based on the first image described above, the target position of the user's eye gaze can be quickly and accurately determined based on the first bright spot position information and the mapping information.

[0008] In one possible implementation, before acquiring a first image of the user's eye portion when a first illumination source in at least one illumination source is turned on and the other illumination sources are turned off, the method further includes: acquiring first sample data from the plurality of sample data, including: turning on each of the at least one illumination source in a light source cycle, wherein the eye sample gazes at a fixed position during the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on; acquiring a corresponding second image when each illumination source is turned on, wherein any second image includes a bright spot formed in the eye sample by light emitted from the corresponding illumination source; determining the position of the bright spot in each second image relative to the pupil of the eye sample; determining a target second image that is closest to the position of the bright spot from the pupil of the eye sample; using the position of the bright spot in the target second image relative to the pupil of the eye sample as a bright spot position information sample; and using the physical position of the target illumination source corresponding to the target second image as a position sample of the eye sample's gaze.

[0009] In this embodiment, at least one of the illumination sources can be turned on during a light source cycle (i.e., other illumination sources are turned off when any one illumination source is turned on). When each illumination source is turned on, a corresponding second image is acquired. Then, the position of the bright spot in each second image relative to the pupil is determined. Furthermore, the target second image with the bright spot closest to the pupil is determined, and the physical position of the target light source corresponding to the target second image is taken as the position of eye gaze. Therefore, this method can not only track the user's eyeball, but also use the position of the bright spot in the target second image from the pupil and the corresponding position of eye gaze as sample data for training to obtain accurate mapping information.

[0010] In this embodiment of the application, when determining the target position of the user's eye based on the first bright spot position information and mapping information, it can be specifically implemented in the following, but not limited to, two possible ways:

[0011] In a first possible implementation, the mapping information includes a one-to-one correspondence between multiple bright spot position information and multiple user eye gaze positions; determining the target position of the user eye gaze based on the first bright spot position information and the mapping information includes: determining the target bright spot position information that is closest to the first bright spot position information among the multiple bright spot position information included in the mapping information; and determining the corresponding target position of the user eye gaze based on the target bright spot position information in the mapping information.

[0012] Optionally, among the multiple bright spot location information contained in the mapping information, the K-nearest neighbor (KNN) algorithm can be used to determine the target bright spot location information that is closest to the first bright spot location information.

[0013] Optionally, the first mapping table is stored using a hash table structure. This implementation allows for quick and accurate retrieval of the first mapping table when used.

[0014] With this implementation, after determining the position information of the first bright spot in the first image, the target position of the user's eye can be quickly determined according to the formula satisfied by the position information of the first bright spot and the mapping information.

[0015] In the second possible implementation, the mapping information satisfies the following formula:

[0016]

[0017] in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

[0018] By means of implementation, after determining the position information of the first bright spot in the first image, the target position of the user's eye can be quickly and accurately determined according to the formula satisfied by the position information of the first bright spot and the mapping information.

[0019] In one possible implementation, before determining the location information of the first bright spot based on the first image, the method further includes: determining that the first image includes the first bright spot and the pupil of the user's eyeball.

[0020] Optionally, the first image can be determined by examining a local histogram of the first image to determine whether it includes the first bright spot and the pupil of the user's eye. If both are included, the first image is considered normal; if not, the first image is considered abnormal.

[0021] This implementation method ensures that the target direction of the user's eye gaze can be accurately determined. When the first image acquired based on the first illumination source is abnormal, other illumination sources, such as a second illumination source, can also be used to ensure that the target direction of the user's eye gaze can be accurately determined.

[0022] Secondly, this application provides an eye-tracking device, including: a processing unit and an acquisition unit;

[0023] The acquisition unit acquires a first image containing the user's eyeball when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off; wherein, the user's eyeball in the first image contains a first bright spot, which is formed by light emitted from the first illumination source; the processing unit is configured to determine the position information of the first bright spot based on the first image, the position information of the first bright spot being used to represent the position of the first bright spot relative to the pupil of the user's eyeball; and determine the target position of the user's eyeball gaze based on the position information of the first bright spot and mapping information; the mapping information is used to represent the mapping relationship between the position information of the bright spot and the position of the user's eyeball gaze, the mapping information being obtained based on training of multiple sample data, any sample data including: a position sample of the eyeball sample gaze and a bright spot position information sample determined based on the image sample, wherein the image sample is an image containing the eyeball sample collected when the eyeball sample gazes at the position sample.

[0024] In one possible implementation, the acquisition unit is further configured to acquire first sample data from the plurality of sample data before acquiring a first image of the user's eye portion when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off; the processing unit is further configured to turn on each of the at least one illumination source in one light source cycle, wherein the eye sample gazes at a fixed position during the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on; the acquisition unit is further configured to acquire a corresponding second image when each illumination source is turned on, wherein any second image includes a bright spot formed in the eye sample by light emitted from the corresponding illumination source; the processing unit determines the position of the bright spot in each second image relative to the pupil of the eye sample; determines a target second image that is closest to the pupil position of the eye sample; uses the position of the bright spot in the target second image relative to the pupil of the eye sample as a bright spot position information sample; and uses the physical position of the target illumination source corresponding to the target second image as a position sample of the eye sample's gaze.

[0025] In one possible implementation, the mapping information includes a one-to-one correspondence between multiple bright spot position information and multiple user eye gaze positions; the processing unit, in determining the target position of the user eye gaze based on the first bright spot position information and the mapping information, includes: determining the target bright spot position information that is closest to the first bright spot position information among the multiple bright spot position information included in the mapping information; and determining the corresponding target position of the user eye gaze in the mapping information based on the target bright spot position information.

[0026] In one possible implementation, the mapping information is stored using a hash table structure.

[0027] In one possible implementation, the mapping information satisfies the following formula:

[0028]

[0029] in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

[0030] In one possible implementation, the processing unit is further configured to determine, before determining the location information of the first bright spot based on the first image, that the first image includes the first bright spot and the pupil of the user's eyeball.

[0031] Thirdly, this application provides an eye-tracking system, comprising: at least one illumination source, a light source controller, an imaging device, and a first controller; wherein, the light source controller is used to control the first illumination source among the at least one illumination source to be turned on, and the other illumination sources to be turned off; the imaging device is used to acquire a first image containing a user's eyeball; wherein, the user's eyeball in the first image contains a first bright spot, the first bright spot being formed based on light emitted by the first illumination source; the first controller is used to acquire the first image; determine the position information of the first bright spot based on the first image, the first bright spot position information being used to represent the position of the first bright spot relative to the pupil of the user's eyeball; determine the target position of the user's eyeball gaze based on the first bright spot position information and mapping information; the mapping information is used to represent the mapping relationship between the bright spot position information and the position of the user's eyeball gaze, the mapping information being obtained based on training with multiple sample data, any sample data including: a position sample of the eyeball sample gaze and a bright spot position information sample determined based on the image sample, wherein the image sample is an image containing the eyeball sample acquired when the eyeball sample gazes at the position sample.

[0032] In one possible implementation, the first controller is further configured to, when acquiring the first sample data among the plurality of sample data, specifically: turn on each of the at least one illumination source in a light source cycle, wherein the eye sample gazes at a fixed position during the light source cycle, and other illumination sources are turned off when any one illumination source is turned on; when each illumination source is turned on, acquire a corresponding second image, wherein any second image includes a bright spot formed in the eye sample by light emitted from the corresponding illumination source; determine the position of the bright spot in each second image relative to the pupil of the eye sample; determine a target second image that is closest to the position of the bright spot from the pupil of the eye sample; use the position of the bright spot in the target second image relative to the pupil of the eye sample as a bright spot position information sample; and use the physical position of the target illumination source corresponding to the target second image as a position sample of the eye sample's gaze.

[0033] In one possible implementation, the first controller is further configured to: send a first signal to the light source controller, the first signal being configured to instruct the light source controller to control the first of the at least one lighting source to turn on and the other lighting sources to turn off.

[0034] In one possible implementation, the first controller is further configured to: send a second signal to the imaging device, the second signal being used to instruct the imaging device to acquire the first image; or

[0035] The light source controller is further configured to: send a third signal to the imaging device, the third signal being used to instruct the imaging device to acquire the first image.

[0036] In one possible implementation, the mapping information includes a one-to-one correspondence between multiple bright spot position information and multiple user eye gaze positions; when the first controller determines the target position of the user's eye gaze based on the first bright spot position information and the mapping information, it is specifically used for:

[0037] Among the multiple bright spot position information contained in the mapping information, the target bright spot position information that is closest to the first bright spot position information is determined; based on the target bright spot position information, the corresponding target position of the user's eye is determined in the mapping information.

[0038] In one possible implementation, the mapping information satisfies the following formula:

[0039]

[0040] in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

[0041] In one possible implementation, the first controller is further configured to: determine that the first image includes the first bright spot and the pupil of the user's eyeball before determining the location information of the first bright spot based on the first image.

[0042] Fourthly, this application provides an eye-tracking device, which includes: at least one processor and an interface circuit; the interface circuit is used to provide input and / or output of a program or instructions to the at least one processor; the at least one processor is used to execute the program or instructions so that the eye-tracking device can implement the method provided in the first aspect or any of the possible implementations described above.

[0043] Fifthly, this application provides a computer storage medium storing a software program, which, when read and executed by one or more processors, can implement the method provided in the first aspect or any of the possible implementations described above.

[0044] Sixthly, this application provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect or any of the possible implementations described above.

[0045] In a seventh aspect, this application provides a chip system including a processor for supporting devices to implement the functions involved in the first aspect above.

[0046] In one possible design, the chip system further includes a memory for storing necessary program instructions and data. The chip system can be composed of chips or may include chips and other discrete devices.

[0047] Eighthly, this application also provides a chip system comprising a processor and an interface, the interface being used to acquire a program or instructions, and the processor being used to invoke the program or instructions to implement or support the device in implementing the functions involved in the first aspect, such as determining or processing at least one of the data and information involved in the above methods.

[0048] In one possible design, the chip system further includes a memory for storing necessary program instructions and data for the terminal device. The chip system can be composed of chips or may include chips and other discrete components.

[0049] The technical effects that can be achieved by any of the second to eighth aspects or any of the second to eighth aspects can be described with reference to the technical effects that can be achieved by the first aspect or any of the possible implementations, and will not be repeated here. Attached Figure Description

[0050] Figure 1 A schematic diagram illustrating possible application scenarios for an eye-tracking method provided in this application embodiment;

[0051] Figure 2 A schematic diagram illustrating the relative position between the center of a user's pupil and a bright spot, provided for the purposes of this application;

[0052] Figure 3 This is a schematic diagram of the structure of an eye-tracking system provided in an embodiment of this application;

[0053] Figure 4This application provides a schematic diagram illustrating the specific workflow of an eye-tracking system as an embodiment of the present application.

[0054] Figure 5 A timing diagram showing the LED lighting cycle and camera capture provided for an embodiment of this application;

[0055] Figure 6 A schematic diagram illustrating the relative position coordinates of the pupil center and the bright spot, provided for an embodiment of this application;

[0056] Figure 7 A schematic diagram of a visual coordinate mapping table provided for an embodiment of this application;

[0057] Figure 8 A schematic flowchart illustrating an eye-tracking method provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of an eye-tracking device provided in an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of another eye-tracking device provided in an embodiment of this application;

[0060] Figure 11 This is a schematic diagram of a chip device structure provided in an embodiment of this application. Detailed Implementation

[0061] This application provides an eye-tracking method and system. The method and system are based on the same or similar technical concepts. Since the methods and systems solve problems in similar ways, their implementations can be referred to each other, and repeated details will not be repeated.

[0062] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0063] The scenarios in which the embodiments of this application are typically applied may include, but are not limited to, contactless interaction in intelligent cockpits of vehicles, multi-screen terminals, laptops, and tablets.

[0064] Figure 1 This illustrates possible specific application scenarios for an eye-tracking method provided in this application embodiment. See also... Figure 1 As shown, multiple devices, such as device 0, device 1, device 2, and device 3, are placed in front of the user. Each device includes a light source, such as the built-in light function of a mobile phone. In the embodiments of the application, the light source may not be limited to being located within the device, or it may be independent of the device, such as... Figure 1As shown, device 0 corresponds to lighting source 0, device 1 corresponds to lighting source 1, device 2 corresponds to lighting source 2, and device 3 corresponds to lighting source 3. Each lighting source can be placed above, below, or next to its corresponding device, and the positional relationship between each lighting source and its corresponding device is known, or the distance between each lighting source and its corresponding device is negligible.

[0065] like Figure 1 The scenario shown also includes a host (not shown) and a camera. The host (which can be equivalent to the first controller in this embodiment) is mainly responsible for control, management, and calculation functions. The host can be a standalone device or one of devices 0-2. A light source controller is connected to devices 0-2 and the host. When the light source controller receives a command from the host, it can control the lighting sources of devices 0-2 to turn on or off. The light source controller can be a standalone device, located within the host, or located within one of devices 0-2. The host is also connected to a camera (e.g., a camera) to control the camera to take pictures of the user. In this embodiment, the host can primarily capture photos of the user's eyes through the camera to capture and extract eye information, thereby determining eye movement and estimating the user's gaze direction or eye position.

[0066] Existing eye-tracking technologies mainly include the Pugin field method, contact lens method, scleral reflex method, pupillary-corneal reflex method, and image-based methods. For the pupillary-corneal reflex method, the following approaches are currently available.

[0067] refer to Figure 2As shown, an illumination source is placed within the user's line of sight. This source reflects off the user's cornea, forming a bright spot. A camera captures an image of the user's eye, and an algorithm is used to obtain the position of the pupil center and the position of the corneal reflection spot. Further, based on these positions, their relative positions are determined. Finally, the direction of the user's gaze can be determined using their relative positions. This method achieves high accuracy in determining the user's gaze direction; for example, the direction error can be controlled within 1 degree. However, to avoid obstructing the user's vision, this method typically requires an infrared point light source, necessitating an infrared camera to capture the user's eye image, thus increasing the cost of the eye-tracking device. Furthermore, due to differences in eye structure among different users, the mapping relationship between the relative positions (x, y) of the pupil center and the bright spot and the user's gaze position (X, Y) also differs among users. Therefore, in the early stages, a calibration process needs to be performed for specific users, that is, to record several mapping relationships between the relative positions (x, y) of the pupil center and the bright spot and the user's gaze position (X, Y) for that user, and then use a regression algorithm to calculate the mapping relationship. However, this calibration process takes a certain amount of time, making this approach unsuitable in some scenarios.

[0068] The current solution addresses the time-consuming calibration process in the aforementioned pupillary corneal reflex method by referring to... Figure 1 As shown, all the lighting sources corresponding to the devices are turned on sequentially at an extremely high speed, and photos are taken simultaneously using a camera. For example, a complete light source cycle consists of 4 frames, each lasting 33.33ms; therefore, a complete light source cycle lasts 133.32ms. Continuing... Figure 1 Taking the four lighting sources shown as an example, referring to Table 1, in frame 1, lighting source 0 is turned on and lit, while the other lighting sources are turned off. In frame 2, lighting source 1 is turned on and lit, while the other lighting sources are turned off. In frame 3, lighting source 2 is turned on and lit, while the other lighting sources are turned off. In frame 4, lighting source 3 is turned on and lit, while the other lighting sources are turned off. In frames 5 to 8 (i.e., the next light source cycle), the state of frames 1 to 4 is repeated, and so on.

[0069] Table 1

[0070]

[0071] When different light sources are turned on in the above manner, the camera captures images of the user's eyes at the same frame rate, acquiring one frame per light source illumination time. For example, in frame 1, after light source 0 is turned on, the camera begins capturing images of the user's eyes and stops capturing images before light source 0 is turned off. Therefore, in each frame captured by the camera, only one light source is illuminated. Based on the photos captured by the camera when different light sources are illuminated within a light source cycle, it is clear that the bright spots in each image are located at different positions of the human eye. By comparing the bright spots in each image that are closest to the center of the pupil, the approximate location of the user's gaze can be determined. This indicates that the user's gaze is directed at the device with that light source. For example, if the bright spot closest to the center of the pupil corresponds to light source 1, then the user's gaze is directed at device 1.

[0072] This scheme avoids the time consumed by the calibration process. However, it determines the user's eye position or direction based on the criterion of proximity to the pupil center. But when the bright spots corresponding to different lighting sources are all relatively close to the pupil center, it becomes difficult to determine the user's eye position or direction. Therefore, the accuracy of this scheme is relatively low, typically with an error of less than 5 degrees in determining the user's eye position or direction. Furthermore, this scheme requires a complete light source cycle to obtain the position of the bright spot relative to the pupil center for each lighting source, and then determines the user's eye position or direction based on the proximity criterion. Since each frame has a duration of d, if there are n lighting sources, the total delay required is at least n. d and n are positive integer values, where d is a value greater than 0. The symbol is for multiplication. Therefore, this scheme requires a relatively large time delay.

[0073] In summary, although current eye-tracking methods can pinpoint the location or direction of a user's gaze, their accuracy remains relatively low, and they also incur significant latency.

[0074] Based on this, this application provides an eye-tracking method, which includes: acquiring a first image containing a user's eyeball when a first illumination source in at least one illumination source is turned on and other illumination sources are turned off; the user's eyeball in the first image contains a first bright spot, which is formed based on light emitted by the first illumination source; determining the position information of the first bright spot based on the first image, the position information of the first bright spot being used to represent the position of the first bright spot relative to the pupil of the user's eyeball; determining the target position of the user's eyeball gaze based on the position information of the first bright spot and mapping information; the mapping information being used to represent the mapping relationship between the position information of the bright spot and the position of the user's eyeball gaze, the mapping information being obtained based on training on multiple sample data. Therefore, this method can accurately and quickly determine the position of the user's eyeball gaze.

[0075] This application provides an eye-tracking method that can be applied to an eye-tracking system. (See reference...) Figure 3 As shown, the eye-tracking system 300 includes a first controller 301, a light source controller 302, at least one illumination source 303, and an imaging device 304; the first controller 301 is connected to the light source controller 302 and the imaging device 304 respectively, and the light source controller 302 is connected to at least one illumination source 303 respectively.

[0076] The following describes in detail an eye-tracking system provided by this application, with reference to specific embodiments.

[0077] like Figure 3 As shown, the eye-tracking system 300 includes: a first controller 301, a light source controller 302, at least one illumination source 303, such as illumination source 303(1), illumination source 303(2)...illumination source 303(N), and a photographing device 304. Each illumination source 303 can be mounted on an electronic device and numbered separately. The electronic device can be a terminal device, such as a mobile phone, a display device, or a vehicle central control screen. Each illumination source 303 is placed independently and numbered separately. Each illumination source 303 can be an 850nm infrared LED, and the photographing device can be a 5-megapixel CMOS camera with an 850nm infrared filter. The light source controller 302 can be an LED controller, mainly used to control the illumination sources 303 to turn on or off respectively. The first controller 301 can be regarded as a host, which can be the aforementioned electronic device or an independent electronic device.

[0078] This application does not limit the specific location and form of the equipment or apparatus in the system used in this application, and can be flexibly set according to actual needs.

[0079] In one possible implementation, the electronic device in this application embodiment may be a portable electronic device including functions such as a personal digital assistant and / or a music player, such as a mobile phone, tablet computer, wearable device with wireless communication capabilities (such as a smartwatch, smart bracelet, etc.), in-vehicle device, etc. Exemplary embodiments of the portable electronic device include, but are not limited to, portable electronic devices running iOS®, Android®, Microsoft®, or other operating systems. The aforementioned portable electronic device may also be a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, the aforementioned electronic device may also be a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0080] The following is a detailed description of the specific operation of an eye-tracking system provided in an embodiment of this application:

[0081] Figure 4 This paper illustrates the specific workflow of an eye-tracking system provided in an embodiment of this application. For example, the system includes a host (i.e., a first controller), electronic device 0 (such as a mobile phone), electronic device 1 (such as a head-up display), and electronic device 2 (such as a vehicle central control screen). Each electronic device has an 850nm infrared LED mounted below it, or each electronic device contains an 850nm infrared LED. The LEDs of electronic devices 0 through 2 are numbered L0, L1, and L2 (i.e., at least one illumination source). A 5-megapixel CMOS camera (i.e., a photographing device) equipped with an 850nm infrared filter is used to photograph the user, who is positioned in front of the device. The host executes the eye-tracking algorithm. The system also includes an LED controller, which controls the infrared LEDs numbered L0, L1, and L2 to turn on or off. This LED controller can be located in the host or be a separate control device. This embodiment of the application does not specifically limit the specific form and assembly method of each device or component in the eye-tracking system; this embodiment is merely an example.

[0082] refer to Figure 4 As shown, the specific workflow of the eye-tracking system is as follows:

[0083] S401: Establish a communication connection between devices.

[0084] refer to Figure 4 The flowchart shown indicates that step S401 belongs to the initialization phase. Specifically, it may include:

[0085] S401(1): Host builds server.

[0086] The host establishes a TCP server, and the master and slave devices establish connections. That is, the host establishes connections with electronic device 0, electronic device 1, electronic device 2, LED controller, and camera respectively to achieve communication.

[0087] S401(2): LED controller client creation.

[0088] The LED controller and electronic device 0-2 establish a client connection, that is, the LED controller establishes a connection with electronic device 0-2 to achieve communication.

[0089] S401(3): Electronic device 0 client.

[0090] Electronic device 0 establishes client connections with both the LED controller and the host, that is, electronic device 0 establishes connections with both the LED controller and the host to achieve communication.

[0091] S401(4): Electronic device 1 builds a client.

[0092] Electronic device 1 establishes client connections with both the LED controller and the host, that is, electronic device 1 establishes connections with both the LED controller and the host to achieve communication.

[0093] S401(5): Electronic device 2 builds client.

[0094] Electronic device 2 establishes client connections with both the LED controller and the host, that is, electronic device 2 establishes connections with both the LED controller and the host to achieve communication.

[0095] S402: The host sends the first indication signal to periodically turn on the LED.

[0096] Optionally, the host can send a first indication signal to the LED controller to periodically turn on the LED via wired, Wi-Fi, or Bluetooth network communication.

[0097] This means that the LEDs are controlled to turn on sequentially according to a preset light source cycle, controlling the L0 of electronic device 0, the L1 of electronic device 1, and the L2 of electronic device 2. When any one of the LEDs is turned on, the other LEDs are turned off.

[0098] S403: The LED controller listens to the first indication signal and periodically turns on (i.e. lights up) L0-L2.

[0099] For example, the LED controller listens for a first indication signal. After the LED controller hears the first indication signal, it sequentially lights up L0, L1, and L2 through the LED driving circuit within a preset light source cycle.

[0100] It should be noted that when any one LED is lit, the other LEDs are in the off (turned off) state. Furthermore, the same operation as in the previous preset light source cycle is performed in the next preset light source cycle.

[0101] S404: The host computer acquires images captured by the camera.

[0102] After the LED controller detects the first instruction signal sent by the host, it controls each LED to turn on. At the same time, the host or the LED controller can control the camera to take a picture of the user's eyes and send the picture to the host.

[0103] Optionally, the host or LED controller can use an external trigger circuit to control the camera to take a picture of the user's eyes and obtain an image of the user's eyes.

[0104] Figure 5 This diagram illustrates the timing of an LED lighting cycle (i.e., a preset light source cycle) and a camera capture. (Reference) Figure 5 As shown, a preset light source cycle can include 3 frames, each lasting 33ms. In the first frame, the host controls the LED controller to turn L0 on (i.e., light up) and L1 and L2 off (i.e., turn off), and takes a picture of the user using the camera to acquire the image. In the second frame, the host controls the LED controller to turn L1 on (i.e., light up) and L0 and L2 off (i.e., turn off), and takes a picture of the user using the camera to acquire the image. In the third frame, the host controls the LED controller to turn L2 on (i.e., light up) and L0 and L1 off (i.e., turn off), and takes a picture of the user using the camera to acquire the image. The same operation is performed in the next preset light source cycle, which will not be described in detail here. It should be noted that in the first preset light source cycle, when the LED controller controls L0 to turn on and L1 and L2 to turn off, it controls the camera to take a picture of the user's eyes, obtaining image 0 of the user's eyes, and simultaneously sends image 0 to the host. Normally, when L0 is turned on, the light emitted by L0 will form a bright spot 0 on the pupil of the user's eyeball. Therefore, the bright spot 0 is included in the user's eyeball in image 0.

[0105] S405: LED controller reports LED status.

[0106] Through step S405, the LED controller can report the LED status to the host in real time, so that the host knows the status of the LEDs (i.e., L0-L2), such as LED0 being on.

[0107] S406: The host obtains the LED status.

[0108] The host obtains the status of the LEDs (i.e., L0-L2) to determine whether L0-L2 is currently on or off, so as to execute the next step.

[0109] Optionally, the host can also obtain the location information corresponding to L0-L2.

[0110] S407: The host executes the eye-tracking algorithm.

[0111] In step S407, the host computer can execute an eye-tracking algorithm for each LED. For each LED that is turned on, the host computer acquires the corresponding image from the camera and then executes the eye-tracking algorithm to determine the correspondence (i.e., mapping information) between the relative position of the pupil center and the bright spot in the image and the object being looked at by the user (electronic device 0-electronic device 2).

[0112] For example, a set light source cycle consists of 3 frames. In each frame, the LED controller controls only one LED to be turned on (lit up). For instance, in a preset light source cycle, in the first frame, the LED controller controls L0 to be turned on (lit up), while L1 and L2 are turned off (extinguished). The camera takes a picture of the user's eyes, obtaining the corresponding image 0, which is then sent to the host. In the second frame, the LED controller controls L1 to be turned on (lit up), while L0 and L2 are turned off (extinguished). The camera takes a picture of the user's eyes, obtaining the corresponding image 1, which is then sent to the host. In the third frame, the LED controller controls L2 to be turned on (lit up), while L0 and L3 are turned off (extinguished). The camera takes a picture of the user's eyes, obtaining the corresponding image 3, which is then sent to the host. The host then executes the following eye-tracking algorithm on the obtained images 0, 1, and 2.

[0113] For a preset light source period image, the host computer executes an eye-tracking algorithm, which may include the following steps:

[0114] Step 1: Obtain the relative position coordinates between the bright spot and the center of the pupil in each image.

[0115] refer to Figure 6 As shown, image 0 is obtained by taking a picture of the user's eye at time t (e.g., the time corresponding to the first frame). The position of the center of the pupil in the user's eyeball in image 0 is represented as ( , ),in, Let x be the x-coordinate of the center of the pupil of the user's eye in image 0. Let be the ordinate of the center of the pupil of the user's eye in image 0; and let the position of the bright spot in image 0 be represented as ( , ),in, Let x be the x-coordinate of the bright spot in image 0. Let i be the vertical coordinate of the bright spot in image 0, where i represents the number 0 of the LED that is turned on (lit up) at time t, or i represents the corresponding image sequence number. For example, if i=0, the host performs an eye-tracking algorithm on image 0.

[0116] Optionally, the relative position coordinates between the bright spot in image 0 and the center of the pupil can satisfy the following formula:

[0117]

[0118] Therefore, this method can be used to obtain the relative position coordinates between the bright spot in image 0 and the center of the pupil within a preset light source period. The relative coordinates between the bright spot in Image 1 and the center of the pupil And the relative position coordinates between the bright spot in Image 2 and the center of the pupil. .

[0119] Step 2: Determine the corresponding distance value based on the relative position coordinates between the bright spot in the image and the center of the pupil.

[0120] Based on the relative position coordinates between the bright spots in images 0, 1, and 2 and the center of the pupil, the corresponding distance values ​​are determined. The following formula can be satisfied:

[0121] .

[0122] Using the distance formula described above, the distance values ​​for image 0 are calculated respectively. The distance value of image 1 is The distance value of image 2 is .

[0123] Step 3: Select the electronic device (or location) where the LED corresponding to the image with the smallest distance value is located as the target electronic device (location) that the user's eye is focused on.

[0124] By following the three steps above, the target electronic device (location) that the user's eye is currently focused on can be determined, thereby enabling eye tracking.

[0125] For example, if the image with the smallest distance value is determined to correspond to electronic device 1, and electronic device 1 is taken as the target electronic device, then the host can send a command to electronic device 1 to wake up the screen of electronic device 1.

[0126] It should be noted that in each preset light source cycle, the host computer executes the eye-tracking algorithm, which can be achieved by referring to the steps of the host computer executing the eye-tracking algorithm in the first preset light source cycle, as described above. It will not be elaborated on here.

[0127] S408: The host determines whether a preset light source cycle is met.

[0128] If the condition is not met, return to step S404 and repeat steps S404-S407.

[0129] It should be noted that within a preset light source cycle, the LED controller sequentially controls each LED to turn on, and for the image acquired for each LED, the eye-tracking algorithm is executed according to the above step S407.

[0130] If satisfied, proceed to step S409.

[0131] S409: The host sends a second instruction signal to the device that the user is looking at.

[0132] When the host determines that a preset light source cycle is met, it can send a second instruction message to the electronic device that the user's eye is looking at (i.e., electronic device 0 and / or electronic device 1 and / or electronic device 2).

[0133] S410: The electronic device that the user is looking at listens to the second instruction signal and performs the corresponding operation.

[0134] For example, electronic device 0 and / or electronic device 1 and / or electronic device 2 listen for the second indication signal, and when the second indication signal is heard, perform a corresponding operation, such as reporting location information to the host.

[0135] S411: The host computer will record all image samples acquired from the camera into mapping information (such as a coordinate mapping table).

[0136] For example, the host establishes a correspondence between the relative position coordinates of the bright spot and the pupil center in each image and the target device (such as electronic device 1). Specifically, this can be:

[0137] The correspondence between image 0 and the target electronic device is as follows: ;

[0138] The correspondence between image 1 and the target electronic device is as follows: ;

[0139] The correspondence between image 2 and the target electronic device is as follows: .

[0140] Furthermore, the relative position coordinates between the bright spot in the image corresponding to each LED and the center of the pupil, as well as the aforementioned correspondence, are recorded as sample data in the mapping information.

[0141] Soon , ; , ; , Recorded in mapping information (such as Figure 6 (as shown in the coordinate mapping table).

[0142] At this point, a preset light source cycle operation is complete, and the next preset light source cycle can begin to be repeated.

[0143] Figure 7 This example demonstrates a visualization of a coordinate mapping table at time i=1 in a two-dimensional coordinate system, where the coordinates of each point are... Different colored dots represent different objects that the user is looking at. (i.e., different devices the user is looking at), meaning one color corresponds to one object (one device).

[0144] It should be noted that, in Figure 7 In the coordinate mapping table shown, different colored points represent different LEDs, such as... Figure 7 The coordinates of all red dots represent the positions of the bright spots obtained with L0 enabled relative to the user's pupil; the coordinates of all purple dots represent the positions of the bright spots obtained with L1 enabled relative to the user's pupil; the coordinates of all cyan dots represent the positions of the bright spots obtained with L2 enabled relative to the user's pupil; the coordinates of all green dots represent the positions of the bright spots obtained with L3 enabled relative to the user's pupil; and the coordinates of all blue dots represent outliers, which can be ignored in this embodiment. Since this embodiment uses L1 enabled as an example, therefore, in Figure 7 The main task is to find the closest point among all the purple points.

[0145] S412: The host determines whether the number of image samples meets the preset requirement.

[0146] It should be noted that the preset quantity can be a pre-defined number of image samples, such as 100 images.

[0147] If the preset quantity is not met, return to step S404 and repeat steps S404-S411; if the preset number of images is met, proceed to step S413.

[0148] Steps S404-S412 above constitute the object-oriented eye tracking process, through which complete mapping information can be obtained (e.g., Figure 7 The coordinate mapping table shown can be stored in the host or a storage device connected to the host using a hash structure, so as to be used for subsequent execution of the self-calibrated eye tracking process (which can be understood as the real-time eye tracking process).

[0149] Optionally, the storage device may be located in the host or in other devices in the system and may communicate with the host. This application does not specifically limit the specific form and location of the storage device.

[0150] The following describes the self-calibrated eye-tracking process, which includes the following steps:

[0151] S413: The host sends a third indication signal to the LED controller.

[0152] For example, the third indication signal is used to instruct the LED controller to keep L1 on (lit) and the other LEDs off (turned off).

[0153] S414: The LED controller listens to this third indication signal and controls L1 to remain on (lit up).

[0154] Through step S414, when the LED controller detects the third indication signal, it controls L1 to remain on (i.e., lit), while the other LEDs are off (i.e. turned off).

[0155] S415: The host computer acquires the current image captured by the camera.

[0156] In step S414 above, the host controls L1 to be continuously turned on (lit up) through the LED controller. At the same time, the host triggers the camera to take a picture of the user's eye area to obtain the first image, and the host obtains the first image from the camera.

[0157] When L1 is turned on, the light emitted by L1 will form a bright spot on the pupil of the user's eye, namely the first bright spot. Therefore, the pupil of the user's eye in the first image includes the first bright spot.

[0158] S416: The host executes an eye-tracking algorithm to determine the target electronic device (location) that the user's eyes are looking at.

[0159] For example, the host computer executes an eye-tracking algorithm to determine the target electronic device that the user's eyes are looking at, which may specifically include the following:

[0160] Based on the first image, the host computer can determine the relative position coordinates of the first bright spot in the first image with respect to the center of the user's pupil. Then, the KNN algorithm is used in... Figure 6 In the coordinate mapping table shown, find the coordinate point closest to it. Furthermore, referring to this coordinate point ( ) and the coordinates of the point ( The correspondence between the target device and the sample is determined to establish the coordinates of the point. The target electronic device corresponding to the sample is =0, which means electronic device 0. Therefore, electronic device 0 is taken as the target electronic device currently being looked at by the user's eye.

[0161] Through step S416, the host can determine the target electronic device that the user's eye is looking at, and thus can track the user's eye.

[0162] S417: The host sends a fourth instruction signal to the target electronic device that the user's eye is looking at.

[0163] For example, through the above step S416, the host determines that the target electronic device that the user's eye is looking at is electronic device 0, and the host sends a fourth indication signal to electronic device 0.

[0164] In addition, since the object being gazed at by the user's eyes (i.e., the electronic device) changes in real time, the host also returns to step S415 and repeats steps S415-S418 to acquire the currently updated image in real time, thereby quickly and accurately determining the electronic device being gazed at by the user's eyes in order to track the user's eyes.

[0165] S418: The target electronic device that the user is looking at listens to the fourth instruction signal and performs the corresponding operation.

[0166] For example, when the target electronic device that the user's eye is focused on is electronic device 0, electronic device 0 hears the fourth indication signal and performs the corresponding operation. In this embodiment of the application, the specific operation performed by the electronic device that the user's eye is focused on is not limited. For example, the fourth indication signal may instruct electronic device 0 to play video or play music.

[0167] Through the self-calibrated eye tracking described above, the target electronic device (location) that the user's eye is looking at can be quickly and accurately determined, thereby effectively tracking the user's eye.

[0168] The technical solution of this application is described below with reference to specific embodiments.

[0169] Based on the above Figure 4 The self-calibrated eye-tracking process in the middle, Figure 8This is a schematic flowchart illustrating an eye-tracking method according to an embodiment of this application. The method can be executed by a first controller or by a chip corresponding to the first controller. This embodiment can also be implemented by a device connected to the first controller. This application does not specifically limit the specific configuration of the device executing this embodiment. Please refer to... Figure 8 The specific process of this method is as follows:

[0170] S801: The first controller acquires a first image containing the user's eyeball.

[0171] The first controller acquires a first image containing the user's eyeball when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off.

[0172] Optionally, the first controller may be a host, or the first controller may be located within a host, or the first controller may be a stand-alone device. This application does not specifically limit the first controller.

[0173] In one possible implementation, when the first lighting source in at least one lighting source is turned on and the other lighting sources are turned off, the first controller sends a first signal to the light source controller, the first signal instructing the light source controller to control the first lighting source in at least one lighting source to be turned on and the other lighting sources to be turned off. Optionally, this implementation can be carried out with reference to the steps S413-S414 described above.

[0174] In one possible implementation, when the first lighting source in at least one lighting source is turned on and the other lighting sources are turned off, the first controller is further configured to: send a second signal to the imaging device, the second signal being used to instruct the imaging device to capture the first image; or the light source controller is further configured to: send a third signal to the imaging device, the third signal being used to instruct the imaging device to capture the first image. Optionally, this implementation can be carried out with reference to step S415 described above.

[0175] In one possible implementation, before acquiring a first image of the user's eye when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off, the first controller further includes: acquiring first sample data from a plurality of sample data, including: turning on each of the at least one illumination source in a light source cycle, wherein the eye sample gazes at a fixed position in the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on; acquiring a corresponding second image when each illumination source is turned on, wherein any second image includes a bright spot formed in the eye sample by the light emitted by the corresponding illumination source; determining the position of the bright spot in each second image relative to the pupil of the eye sample; determining a target second image that is closest to the pupil position of the eye sample; using the position of the bright spot in the target second image relative to the pupil of the eye sample as a bright spot position information sample; and using the physical position of the target illumination source corresponding to the target second image as a position sample of the eye sample's gaze. This implementation can be carried out with reference to the above steps S401-S412, that is, acquiring multiple sample data and training to obtain mapping information.

[0176] For example, the system includes three lighting sources, numbered L0, L1, and L2. During one lighting cycle, the light source controller turns on L0 (light source number 0) while the other lighting sources remain closed. Simultaneously, a photograph is taken of the user's eyes, resulting in image 0. Similarly, the light source controller turns on L1 (light source number 1) while the other lighting sources remain closed, and the photograph is taken of the user's eyes, resulting in image 1. Finally, the light source controller turns on L1 (light source number 21) while the other lighting sources remain closed, and the photograph is taken of the user's eyes, resulting in image 2.

[0177] After acquiring images 0, 1, and 2, the host computer determines the position of the bright spot in image 0 relative to the pupil of the eye within it, based on image 0. For example, the position coordinates are... Based on Image 1, determine the position of the bright spot in Image 1 relative to the pupil of the eyeball within it, such as the position coordinates as... Based on Image 2, determine the position of the bright spot in Image 2 relative to the pupil of the eyeball within it, such as the position coordinates as... Specifically, to determine the position coordinates of the bright spots in each image relative to the pupil of the eyeball, refer to the formula in step one of the above steps S408 when the host executes the eye-tracking algorithm (and...). Figure 6 ).

[0178] Furthermore, the host computer determines the position of image 0, such as its position coordinates. Determine the distance between the bright spot in image 0 and the pupil of the eye within it. The host computer determines the position of image 1 based on its coordinates. Determine the distance between the bright spot in image 1 and the pupil of the eye within it. The host computer determines the position of image 2 based on its coordinates. Determine the distance between the bright spot in image 1 and the pupil of the eye within it. For specific distance calculations, please refer to the formula in step two of the host computer executing the eye-tracking algorithm in step S408 above (and...). Figure 6 The host determines that the nearest (i.e., smallest) distance value corresponds to image 1, that is, image 1 is the aforementioned target second image.

[0179] Finally, the host computer will determine the coordinates of the bright spot in image 0 relative to the pupil of the eye. The coordinates of the bright spot in Image 1 relative to the pupil of the eye. The coordinates of the bright spot in Image 2 relative to the pupil of the eye. and coordinates Correspondence between L1 and image 1 ,coordinate Correspondence between L1 and image 1 ,coordinate Correspondence between L1 and image 1 This data serves as sample data (such as the first sample data). Optionally, the host may record this sample data in mapping information, such as a coordinate mapping table. For details, please refer to step S411 above; it will not be described in detail here.

[0180] In this embodiment, to ensure that the first controller can accurately process different photos captured by the imaging device, the size, pixels, resolution, etc., of the images of the user's eyes captured by the imaging device should be the same under different lighting conditions. Furthermore, the size and position of the user's eyes in the image should also be the same in different images.

[0181] S802: The first controller determines the position information of the first bright spot based on the first image.

[0182] The first bright spot position information is used to indicate the position of the first bright spot relative to the pupil of the user's eyeball.

[0183] Optionally, the position information of the first bright spot can be represented by coordinates. Therefore, when performing step S802, the above can be referred to. Figure 4 Step S407, that is, the formula in step one when the host executes the eye-tracking algorithm, and Figure 6 .

[0184] In one possible implementation, before the first controller determines the location information of the first bright spot based on the first image, the method further includes: determining that the first image includes the first bright spot and the pupil of the user's eyeball.

[0185] S803: The first controller determines the target position that the user's eye is looking at based on the position information of the first bright spot and the mapping information.

[0186] In one possible implementation, the mapping information includes a one-to-one correspondence between multiple bright spot position information and multiple user eye gaze positions; the first controller determines the target position of the user's eye gaze based on the first bright spot position information and the mapping information, including: determining the target bright spot position information that is closest to the first bright spot position information among the multiple bright spot position information included in the mapping information; and determining the corresponding target position of the user's eye gaze in the mapping information based on the target bright spot position information. Optionally, this implementation can be carried out with reference to step S416 above.

[0187] Optionally, the mapping information can be stored using a hash table structure. This allows for quick retrieval of the mapping information when needed.

[0188] In another possible implementation, the mapping information satisfies the following formula:

[0189]

[0190] in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. This represents the ordinate of the target position that the user's eye is looking at at time t. This represents the x-coordinate of the position of the first bright spot at time t. This represents the ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

[0191] This implementation method is also implemented with reference to the first implementation method described below, which will not be described in detail here.

[0192] Therefore, this implementation method can accurately calculate the coordinates of the target position that the user's eye is looking at based on the coordinates of the first bright spot position.

[0193] It should be noted that before executing the embodiments of this application, the mapping information can also be trained and learned by referring to the above steps S401-S412 and stored. However, the embodiments of this application do not specifically limit the method used to learn the mapping information, such as machine learning methods.

[0194] In addition, the above-mentioned mapping information can also be represented using machine learning algorithms and their learned parameters. This application does not impose specific limitations on this, and it can be flexibly set according to actual needs.

[0195] In summary, this application provides an eye-tracking method, which includes: in this embodiment, during eye-tracking, turning on an illumination source (such as a first illumination source) and acquiring a first image containing the user's eyeball (i.e., acquiring an image of the user's eye at this time); then determining first bright spot position information representing the bright spot in the user's eyeball and the pupil based on the first image; and then determining the target position of the user's eyeball gaze based on the first bright spot position information and trained mapping information. In this method, since it is not necessary to turn on each illumination source and acquire images of the user's eye multiple times to track the user's eyeball, the time delay caused by turning on the illumination source and acquiring images of the user's eyeball can be greatly reduced. Furthermore, the mapping information in this embodiment is trained based on multiple sample data, and this mapping information can accurately reflect the mapping relationship between the bright spot position information and the position of the user's eyeball gaze. Therefore, after determining the first bright spot position information based on the first image, the target position of the user's eyeball gaze can be quickly and accurately determined based on the first bright spot position information and the mapping information.

[0196] Implementation Method 1

[0197] Based on the above Figure 8 In the eye-tracking method shown in step S803, another implementation method for determining the formula satisfied by the mapping information can be achieved as follows:

[0198] For example, the devices used to perform eye tracking (i.e., all the devices the user's eyes might be looking at) are in fixed positions, and each device is equipped with a lighting source, such as an LED. In this case, numbering each device can be represented as follows: Each number corresponds to a physical coordinate on a line-of-sight plane. This can be considered the center position of the device. Existing calibration methods can be used to establish the physical coordinates of the device as the user's eye is focused on. The relative position coordinates of the bright spot in the image with respect to the user's pupil The mapping relationship between them can be expressed as: physical coordinates Relative coordinates of bright spot and pupil The mapping relationship between them.

[0199] For example, with 5 devices, namely device 1 (L1) to device 5 (L5), when only device 1's L1 is turned on, the user's eyes are focused on device 1, the physical coordinates 1 of device 1 are obtained, an image 1 of the user's eye is obtained through a camera, and based on image 1, a sample 1 of the relative position coordinates of the bright spot and the pupil is determined. When only device 2's L2 is turned on, the user's eyes are focused on device 2, the physical coordinates 2 of device 2 are obtained, an image 2 of the user's eye is obtained through a camera, and based on image 2, a sample 2 of the relative position coordinates of the bright spot and the pupil is determined. This process is repeated for each device.

[0200] Five sets of sample data can be obtained through the above method, namely (physical coordinate 1, bright spot-pupil relative position coordinate sample 1), (physical coordinate 2, bright spot-pupil relative position coordinate sample 2), (physical coordinate 3, bright spot-pupil relative position coordinate sample 3), (physical coordinate 4, bright spot-pupil relative position coordinate sample 4), and (physical coordinate 5, bright spot-pupil relative position coordinate sample 5).

[0201] physical coordinates Relative coordinates of pupil-bright spot The mapping formula between them can satisfy the following:

[0202]

[0203] in, This represents the physical x-coordinate of the device that the user's eye is looking at at the current time t. This represents the physical ordinate of the device that the user's eye is looking at at time t. This represents the x-coordinate of the bright spot position at time t. The ordinate represents the position of the bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the device (LED) that the user's eyes are focused on.

[0204] For device 1 (corresponding to L1), the polynomial coefficients in the mapping formula, i.e., a set of parameters, can be calculated by acquiring multiple sample data of device 1 (i.e., physical coordinate 1, bright spot-pupil relative position coordinate sample 1). .

[0205] Other devices can refer to the method described for device 1 above to obtain the corresponding set of parameters.

[0206] It is important to note that each set of parameters includes 5 pairs of binary variables. Therefore, at least 5 sets of sample data are needed to obtain the polynomial coefficients of the above mapping formula. When more than 5 sets of sample data are obtained, the least squares method can be used to solve the overdetermined system of equations to obtain the optimal parameter values ​​of the above mapping formula.

[0207] When determining a set of parameters corresponding to each device, when executing the above S803, the target physical coordinates of the user's eye can be accurately calculated based on the position coordinates of the first bright spot and the corresponding mapping formula, that is, the target device that the user's eye is looking at can be determined.

[0208] This implementation method allows the calibration process described above to be completed during user operation without requiring the user to explicitly perform the calibration process, and enables high-precision tracking of the user's eyeballs.

[0209] Implementation Method Two:

[0210] This second embodiment is an execution method proposed based on the step of determining in step S802 above that the first image includes the first bright spot and the pupil of the user's eyeball.

[0211] For example, glare detection is added to the eye-tracking algorithm steps. When a user wears glasses or sunglasses, the specular reflection caused by the LEDs on the glasses may cause glare in the image captured by the camera, making it impossible to identify the user's pupils and bright spots in the image. Therefore, glare near the user's eyes can be detected using local histograms or other methods. If glare is present, the first controller can switch to other LEDs and keep them constantly lit. For example, if the currently lit LED is number 1, when glare is detected, a command can be sent to notify the LED controller to turn off LED1 and turn on LED2. Then, a new cycle is restarted. When performing the lookup table mapping operation, the mapping information corresponding to LED2 (the mapping table corresponding to LED2) is used instead. The specific execution method is the same as that for LED1, and will not be elaborated here.

[0212] In this second embodiment, when a user wears glasses or sunglasses, the specular reflection generated by the LEDs on the glasses causes glare in the image of the user's eyes captured by the camera. Therefore, the lit LEDs can be automatically switched to eliminate glare, ensuring that the image acquired by the camera is normal and avoiding the impact of glare on the eye-tracking system. This further makes the eye-tracking system of this application more universal, such as supporting situations where the user is wearing glasses or sunglasses.

[0213] Based on the same technical concept, this application provides an eye-tracking device that can be used to perform the operations executed by the first controller in the above-described method embodiments. The device can also be the first controller, its processor, or a chip. The device includes modules or units corresponding to the methods / operations / steps / actions described by the first controller in the above embodiments. These modules or units can be hardware circuits, software, or a combination of hardware circuits and software. The eye-tracking device can have the following characteristics: Figure 9 The structure shown.

[0214] refer to Figure 9 As shown, the device 900 includes an acquisition unit 901 and a processing unit 902; wherein, the acquisition unit 901 is used to acquire a first image of the current user's eye portion when a first illumination source in at least one illumination source is turned on and the other illumination sources are turned off; wherein, the first image contains a first bright spot in the user's eyeball, the first bright spot being formed based on light emitted by the first illumination source; the processing unit 902 is used to determine the position information of the first bright spot based on the first image, the first bright spot position information being used to indicate the position of the first bright spot relative to the pupil of the user's eyeball; and to determine the target direction of the user's gaze based on the first bright spot position information.

[0215] In one possible implementation, when the processing unit 902 determines the target direction of the user's eye gaze based on the first bright spot position information, it is specifically configured to: determine the target bright spot position information that is closest to the first bright spot position information from among the multiple bright spot position information contained in the first mapping table; the first mapping table includes: a one-to-one mapping relationship between multiple bright spot position information and multiple gaze directions; and determine the target direction of the user's eye gaze corresponding to the target bright spot position information in the first mapping table.

[0216] In one possible implementation, the first mapping table is stored using a hash table structure.

[0217] In one possible implementation, when determining the target direction of the user's eye gaze based on the first bright spot position information, the processing unit 902 specifically performs the following steps: calculating the target position of the user's eye gaze based on the first bright spot position information and a first mapping formula; the first mapping formula represents the correspondence between the bright spot position information and the position of the user's eye gaze. The target direction of the user's eye gaze is then determined based on the target position of the user's eye gaze.

[0218] In one possible implementation, the first mapping formula satisfies the following formula:

[0219]

[0220] in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. This represents the ordinate of the target position that the user's eye is looking at at time t. This represents the x-coordinate of the position of the first bright spot at time t. This represents the ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

[0221] In one possible implementation, the processing unit 902 is further configured to determine, before determining the location information of the first bright spot based on the first image, that the first image includes the first bright spot and the pupil of the user's eyeball.

[0222] Optionally, the device 900 may further include a storage unit 903 for storing data or instructions (also referred to as code or program). Each of the aforementioned units can interact with or be coupled to the storage unit 903 to implement corresponding methods or functions. For example, the processing unit 902 can read data or instructions from the storage unit, enabling the device 900 to implement the methods described in the above embodiments.

[0223] Based on the same inventive concept, embodiments of this application also provide an eye-tracking device, such as... Figure 10 The diagram shown is a schematic of an eye-tracking device provided in this application. The device 1000 can be a first controller, a processor of the first controller, or a chip in the above embodiments. The device 1000 can be used to execute the operations performed by the first controller in the above method embodiments. The device 1000 includes a processor 1002. Optionally, the device 1000 may also include a communication interface 1001, a memory 1003, and a communication line 1004. The processor 1002, communication interface 1001, and memory 1003 can be interconnected via the communication line 1004. The communication line 1004 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication line 1004 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0224] The processor 1002 may be a CPU, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of programs according to the present application.

[0225] Communication interface 1001 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), wired access network, etc.

[0226] The memory 1003 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 1004. The memory may also be integrated with the processor.

[0227] The memory 1003 stores computer execution instructions for implementing the scheme of this application, and the processor 1002 controls the execution. The processor 1002 executes the computer execution instructions stored in the memory 1003, thereby implementing the eye-tracking method provided in the above embodiments of this application.

[0228] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0229] Figure 11 This is a schematic diagram of a chip device structure provided in an embodiment of this application. The chip 1100 includes an interface circuit 1101 and one or more processors 1102. Optionally, the chip 1100 may also include a bus. Wherein:

[0230] The processor 1102 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the aforementioned eye-tracking method can be completed through integrated logic circuits in the hardware of the processor 1102 or through software instructions. The processor 1102 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0231] The interface circuit 1101 can be used to send or receive data, instructions or information. The processor 1102 can use the data, instructions or other information received by the interface circuit 1101 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1101.

[0232] Optionally, the chip also includes memory 1103, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of memory 1103 may also include non-volatile random access memory (NVRAM).

[0233] Optionally, the memory stores executable software modules or data structures, and the processor can perform corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0234] Optionally, the chip can be used in the first controller (or a host including the first controller) involved in the embodiments of this application. Optionally, the interface circuit 1101 can be used to output the execution result of the processor 1102. For details regarding the eye-tracking method provided by one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0235] It should be noted that the functions of the interface circuit 1101 and the processor 1102 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0236] This application also provides a computer-readable storage medium storing computer instructions for implementing the method executed by the first controller in the above method embodiments.

[0237] For example, when the computer program is executed by the computer, it enables the computer to implement the method executed by the first controller in the above method embodiment.

[0238] This application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to implement the method executed by the first controller in the above method embodiments.

[0239] This application also provides a chip device, including a processor, configured to call computer programs or computer instructions stored in the memory, so that the processor executes the above-described... Figure 4 or Figure 8 An eye-tracking method according to the embodiment shown.

[0240] In one possible implementation, the input of the chip device corresponds to the above. Figure 4 or Figure 8 The receiving operation in the illustrated embodiment corresponds to the output of the chip device described above. Figure 4 or Figure 8 The sending operation in the illustrated embodiment.

[0241] Optionally, the processor is coupled to the memory via an interface.

[0242] Optionally, the chip device further includes a memory storing computer programs or computer instructions.

[0243] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), or one or more devices used to control the above. Figure 5 The illustrated embodiment is an integrated circuit for program execution of an eye-tracking method. The memory mentioned above can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0244] It should be noted that, for the sake of convenience and brevity, the explanations and beneficial effects of the relevant contents in any of the eye-tracking devices provided above can be referred to the corresponding eye-tracking method embodiments provided above, and will not be repeated here.

[0245] In this application, the devices may further include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer may include hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system layer may be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer may include applications such as browsers, address books, word processing software, and instant messaging software.

[0246] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0247] Through the above description of the embodiments, those skilled in the art will clearly understand that the embodiments of this application can be implemented in hardware, firmware, or a combination thereof. When implemented in software, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a computer. For example, but not limited to, computer-readable media can include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible to a computer. Furthermore, any connection can suitably be a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used in embodiments of this application, disks and discs include compact discs (CDs), laser discs, optical discs, digital video discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while discs optically copy data using lasers. The combinations above should also be included within the scope of protection for computer-readable media.

[0248] In summary, the above descriptions are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. An eye-tracking method, characterized in that, include: When the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off, a first image containing the user's eyeball is acquired; wherein the user's eyeball in the first image contains a first bright spot, the first bright spot being formed based on light emitted from the first illumination source; Based on the first image, the position information of the first bright spot is determined, which is used to indicate the position of the first bright spot relative to the pupil of the user's eyeball; Based on the first bright spot location information and the mapping information, the target location of the user's eye is determined; the mapping information is used to represent the mapping relationship between the bright spot location information and the location of the user's eye, and the mapping information is obtained by training based on multiple sample data. Each sample data includes: a location sample of the eye sample's gaze and a bright spot location information sample determined based on the image sample, wherein the image sample is an image containing the eye sample collected when the eye sample gazes at the location sample; Before acquiring the first image of the user's eye portion when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off, the method further includes: acquiring the plurality of sample data; wherein, acquiring the first sample data among the plurality of sample data includes the following: During a light source cycle, each of the at least one illumination source is turned on, wherein the eye sample gazes at a fixed position during the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on; When each illumination source is turned on, a corresponding second image is acquired, and any second image includes a bright spot formed in the eye sample by the light emitted by the corresponding illumination source; Determine the position of the bright spot in each second image relative to the pupil of the eyeball sample; Determine the target second image whose bright spot is closest to the pupil position of the eyeball sample; The position of the bright spot in the second target image relative to the pupil of the eye sample is used as the bright spot position information sample; and the physical position of the target illumination source corresponding to the second target image is used as the position sample of the eye sample's gaze.

2. The method according to claim 1, characterized in that, The mapping information includes a one-to-one correspondence between multiple bright spot location information and multiple user eye gaze positions; Determining the target position of the user's eye gaze based on the first bright spot position information and mapping information includes: Among the multiple bright spot position information contained in the mapping information, the target bright spot position information that is closest to the first bright spot position information is determined; Based on the target bright spot location information, the corresponding target location that the user's eye is looking at is determined in the mapping information.

3. The method according to claim 2, characterized in that, The mapping information is stored using a hash table structure.

4. The method according to claim 1, characterized in that, The mapping information satisfies the following formula: in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

5. The method according to any one of claims 1 to 4, characterized in that, Before determining the location information of the first bright spot based on the first image, the method further includes: The first image is determined to include the first bright spot and the pupil of the user's eye.

6. An eye-tracking device, characterized in that, include: Acquisition unit and processing unit; The acquisition unit acquires a first image containing the user's eyeball when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off; wherein, the user's eyeball in the first image contains a first bright spot, which is formed based on the light emitted by the first illumination source; The processing unit is configured to determine the position information of the first bright spot based on the first image, wherein the position information of the first bright spot is used to indicate the position of the first bright spot relative to the pupil of the user's eyeball; Based on the first bright spot location information and the mapping information, the target location of the user's eye is determined; the mapping information is used to represent the mapping relationship between the bright spot location information and the location of the user's eye gaze, and the mapping information is obtained based on training with multiple sample data. Each sample data includes: a location sample of the eye sample's gaze and a bright spot location information sample determined based on the image sample, wherein the image sample is an image containing the eye sample collected when the eye sample gazes at the location sample; the acquisition unit is further configured to acquire the multiple sample data before acquiring the first image of the user's eye portion when the first illumination source in at least one illumination source is turned on and the other illumination sources are turned off; wherein, when acquiring the first sample data among the multiple sample data, the acquisition unit is specifically configured to: The processing unit turns on each of the at least one illumination source during one light source cycle, wherein the eye sample gazes at a fixed position during the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on. The acquisition unit is used to acquire a corresponding second image when each illumination source is turned on, and any second image includes a bright spot formed in the eye sample by the light emitted by the corresponding illumination source; The processing unit determines the position of the bright spot in each second image relative to the pupil of the eye sample; determines the target second image that is closest to the position of the bright spot relative to the pupil of the eye sample; and uses the position of the bright spot in the target second image relative to the pupil of the eye sample as a bright spot position information sample; and uses the physical position of the target illumination source corresponding to the target second image as a position sample of the eye sample's gaze.

7. The apparatus according to claim 6, characterized in that, The mapping information includes a one-to-one correspondence between multiple bright spot location information and multiple user eye gaze positions; The processing unit determines the target position of the user's eye gaze based on the first bright spot position information and mapping information, including: Among the multiple bright spot position information contained in the mapping information, the target bright spot position information that is closest to the first bright spot position information is determined; Based on the target bright spot location information, the corresponding target location that the user's eye is looking at is determined in the mapping information.

8. The apparatus according to claim 7, characterized in that, The mapping information is stored using a hash table structure.

9. The apparatus according to claim 6, characterized in that, The mapping information satisfies the following formula: in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The processing unit is further configured to determine, before determining the location information of the first bright spot based on the first image, that the first image includes the first bright spot and the pupil of the user's eyeball.

11. An eye-tracking system, characterized in that, include: At least one lighting source, a light source controller, a photographic device, and a first controller; The first controller is connected to both the light source controller and the imaging device, and the light source controller is connected to both of the at least one lighting source. The light source controller is used to control the first lighting source among the at least one lighting source to turn on and the other lighting sources to turn off; The photographing device is used to capture a first image containing a user's eyeball; wherein, the user's eyeball in the first image contains a first bright spot, the first bright spot being formed based on light emitted from the first illumination source; The first controller is configured to acquire the first image; and determine the position information of the first bright spot based on the first image, wherein the position information of the first bright spot is used to indicate the position of the first bright spot relative to the pupil of the user's eyeball; Based on the first bright spot location information and the mapping information, the target location of the user's eye is determined; the mapping information is used to represent the mapping relationship between the bright spot location information and the location of the user's eye, and the mapping information is obtained by training based on multiple sample data. Each sample data includes: a location sample of the eye sample's gaze and a bright spot location information sample determined based on the image sample, wherein the image sample is an image containing the eye sample collected when the eye sample gazes at the location sample; The first controller is further configured to acquire the plurality of sample data; wherein, when acquiring the first sample data among the plurality of sample data, the first controller is specifically configured to: During a light source cycle, each of the at least one illumination source is turned on, wherein the eye sample gazes at a fixed position during the light source cycle, and the other illumination sources are turned off when any one illumination source is turned on; When each illumination source is turned on, a corresponding second image is acquired, and any second image includes a bright spot formed in the eye sample by the light emitted by the corresponding illumination source; Determine the position of the bright spot in each second image relative to the pupil of the eyeball sample; Determine the target second image whose bright spot is closest to the pupil position of the eyeball sample; The position of the bright spot in the second target image relative to the pupil of the eyeball sample is used as the bright spot position information sample; The physical location of the target illumination source corresponding to the second target image is used as the position sample of the eyeball's gaze.

12. The system according to claim 11, characterized in that, The first controller is further configured to: send a first signal to the light source controller, the first signal being configured to instruct the light source controller to control the first of the at least one lighting light source to turn on and the other lighting light sources to turn off.

13. The system according to claim 11 or 12, characterized in that, The first controller is further configured to: send a second signal to the imaging device, the second signal being used to instruct the imaging device to capture the first image; or The light source controller is further configured to: send a third signal to the imaging device, the third signal being used to instruct the imaging device to acquire the first image.

14. The system according to claim 11 or 12, characterized in that, The mapping information includes a one-to-one correspondence between multiple bright spot location information and multiple user eye gaze positions; When the first controller determines the target position of the user's eye based on the first bright spot position information and mapping information, it is specifically used for: Among the multiple bright spot position information contained in the mapping information, the target bright spot position information that is closest to the first bright spot position information is determined; Based on the target bright spot location information, the corresponding target location that the user's eye is looking at is determined in the mapping information.

15. The system according to claim 11 or 12, characterized in that, The mapping information satisfies the following formula: in, This represents the x-coordinate of the target position that the user's eye is looking at at the current time t. The vertical coordinate represents the position of the target being gazed upon by the user's eye at the current time t. This represents the x-coordinate of the position of the first bright spot at time t. The ordinate of the position of the first bright spot at time t; parameter group is a constant, representing the parameter set corresponding to the first lighting source.

16. The system according to claim 11 or 12, characterized in that, The first controller is further configured to: determine, before determining the location information of the first bright spot based on the first image, that the first image includes the first bright spot and the pupil of the user's eyeball.

17. A chip system, characterized in that, The chip system includes a processor and an interface circuit, the interface circuit being used to acquire a program or instructions, and the processor being used to invoke the program or instructions to cause the method described in any one of claims 1 to 5 to be executed.

18. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the method as described in any one of claims 1 to 5 to be performed.

19. A non-volatile computer-readable storage medium, characterized in that, A computer program is stored that, when run on a processor, causes the method described in any one of claims 1 to 5 to be executed.

Images