Eyeball tracking method and device, equipment and storage medium

Abstract

The invention discloses an eyeball tracking method and device, equipment and a storage medium. Comprising the following steps: acquiring an eye image and / or an infrared signal returned by an eye; determining a sight line direction and first fixation point information according to the eye image; determining second fixation point information according to the infrared signal; and fusing the first fixation point information and the second fixation point information according to the line-of-sight direction to obtain target fixation point information. According to the eyeball tracking method provided by the embodiment of the invention, the two pieces of fixation point information are fused based on the sight line direction to obtain the final fixation point information, so that the eyeball tracking precision can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of eye tracking technology, and in particular to an eye tracking method, device, equipment and storage medium. Background Art

[0002] At present, a single eye tracking technology is usually used for eye tracking, for example, using one of the pupil-corneal reflection method, image contour extraction algorithm or micro-electro-mechanical system (MEMS) method for eye tracking. The eye tracking accuracy of the pupil-corneal reflection method or image contour extraction algorithm will be affected by the direction of eye gaze; the MEMS method is suitable for low-precision eye tracking, but not for high-precision eye tracking. Summary of the invention

[0003] Embodiments of the present invention provide an eye tracking method, an apparatus, a device and a storage medium, which can improve the accuracy of eye tracking.

[0004] In a first aspect, an embodiment of the present invention provides an eye tracking method, comprising:

[0005] Acquiring eye images and / or infrared signals returned by the eyes;

[0006] Determining a sight line direction and first gaze point information according to the eye image;

[0007] Determining second gaze point information according to the infrared signal;

[0008] The first gaze point information and the second gaze point information are fused according to the line of sight direction to obtain target gaze point information.

[0009] In a second aspect, an embodiment of the present invention further provides an eye tracking device, comprising:

[0010] An acquisition module, used for acquiring eye images and / or infrared signals returned by the eyes;

[0011] A first gaze point information determination module, used to determine a sight line direction and first gaze point information according to the eye image;

[0012] A second gaze point information determination module, used to determine the second gaze point information according to the infrared signal;

[0013] The target gaze point information acquisition module is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain the target gaze point information.

[0014] In a third aspect, an embodiment of the present invention further provides an eye tracking system, comprising: a first eye tracking module, a second eye tracking module and a gaze point determination module; wherein the first eye tracking module performs eye tracking based on a pupil corneal reflection method or an image contour extraction algorithm; and the second eye tracking module performs eye tracking based on a micro-electromechanical system (MEMS) method;

[0015] The first eye tracking module is used to collect eye images and determine the sight direction and first gaze point information according to the eye images;

[0016] The second eye tracking module is used to collect infrared signals returned by the eyes, and determine the second gaze point information according to the infrared signals;

[0017] The gaze point determination module is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain target gaze point information.

[0018] In a fourth aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the eye tracking method described in the embodiment of the present invention.

[0022] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the eye tracking method described in the embodiment of the present invention when executed.

[0023] The embodiment of the present invention discloses an eye tracking method, device, equipment and storage medium. Obtain an eye image and / or an infrared signal returned by the eye; determine the sight direction and first gaze point information according to the eye image; determine the second gaze point information according to the infrared signal; fuse the first gaze point information and the second gaze point information according to the sight direction to obtain the target gaze point information. The eye tracking method provided by the embodiment of the present invention fuses the two gaze point information based on the sight direction to obtain the final gaze point information, which can improve the accuracy of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of an eye tracking method in Embodiment 1 of the present invention;

[0025] Figure 2 This is an example diagram of a distribution of sight lines in the first embodiment of the present invention;

[0026] Figure 3 This is an example diagram of a distribution of sight lines in the first embodiment of the present invention;

[0027] Figure 4 is a structural schematic diagram of an eye tracking system in Embodiment 2 of the present invention;

[0028] Figure 5 is a schematic diagram of the structure of an eye tracking device in Embodiment 3 of the present invention;

[0029] Figure 6 It is a structural schematic diagram of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Eye tracking, also known as line of sight tracking, is a technology that estimates the line of sight and / or gaze point of the eye by measuring the movement of the eye. Among them, the line of sight can be understood as a three-dimensional vector, and the gaze point can be understood as the two-dimensional coordinates of the above three-dimensional vector projected on a plane. At present, the optical recording method is widely used: using a camera or a video camera to record the user's eye movement, that is, obtaining an eye image reflecting the eye movement, and extracting eye features from the obtained eye image to establish a line of sight / gaze point estimation model. Among them, eye features may include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot (also called Purkinje spot) position, etc.

[0032] Among the optical recording methods, the most mainstream eye tracking methods currently include: pupil-corneal reflection method and image contour extraction algorithm. Other methods may also include methods not based on eye images, such as: inferring eye movement based on contact / non-contact sensors (such as electrodes, capacitive sensors), and MEMS methods.

[0033] For the pupil-corneal reflection method, the hardware requirements include: light source and image acquisition equipment. Among them, the light source can be an infrared light source, and can be multiple infrared light sources, arranged in a predetermined manner, such as a herringbone shape, a straight line shape, etc. The image acquisition device can be an infrared camera device, an infrared image sensor, a camera or a video camera, etc. The specific process can be: 1. The light source is directed to the eye, and the image acquisition device takes a picture of the eye, and the reflection point of the light source on the cornea is photographed accordingly, so as to obtain an eye image with a light spot; 2. As the eyeball rotates, the relative position relationship between the pupil center and the light spot changes, and the correspondingly collected several eye images with light spots can reflect the position change relationship between the pupil center and the light spot; the line of sight or gaze point is estimated based on the position change relationship.

[0034] This method has a good eye tracking effect in most positions, but when the eye turns to a large angle with the camera, the positioning of the pupil and the light spot (due to severe pupil deformation) may be inaccurate. Due to factors such as the deformation of the pupil contour, the eye tracking effect of this solution in specific positions will be affected, making the local use effect poor.

[0035] For the image contour extraction algorithm, the hardware device can use a single-camera eye tracking system, use the first camera of the head-mounted device worn by the user to capture the user's left eye image; use the second camera of the head-mounted device to capture the user's right eye image; use the image contour extraction algorithm to obtain the edge contour of the pupil, and then based on the spatial geometry (back projection), to obtain a pupil orientation in space, and multiple pupil orientation information can establish an eyeball 3D center model. This method requires the establishment of a human eye 3D center model in advance. The establishment process can be: presenting a first stimulus (which can be understood as a calibration point) to a user wearing a head-mounted device, when the user responds to the first stimulus or is expected to respond to the first stimulus, the first camera of the head-mounted device captures the first left image of the user's left eye. The second camera captures the first right image of the user's right eye. A data connection is established between the head-mounted device and the database. Generate a first data set including a first left image, a first right image and eyeball-related parameters. The first data set is related to the first stimulus. By analogy, a second, third...Nth and other multiple stimulation points will be performed, and then multiple data sets are formed, and the data sets include corresponding left images, corresponding right images and corresponding corresponding eyeball-related parameters. Among them, the eye-related parameters include pupil information (including: pupil outline, pupil size, pupil diameter, etc.) and eye 3D center coordinates. Finally, the user's final eye 3D center position will be determined by an algorithm based on multiple eye 3D center coordinates. When performing eye tracking, the user's gaze direction can be approximately calculated based on the user's eye 3D center position coordinates and the pupil center coordinates calculated in real time, thereby predicting the user's line of sight and gaze point.

[0036] This method has the following defects: when the eyeball turns to an angle that is too large with the camera, the outline of the pupil may be inaccurate. Due to factors such as the deformation of the pupil outline, the eye tracking effect at that specific position will be affected, resulting in poor local use effect; when the eyeball looks straight ahead, that is, the pupil outline is a perfect circle, the pupil outline image does not change much when it is slightly deflected by a few degrees, and the difference in the image is very small, which means that the algorithm is not accurate in this position.

[0037] For the MEMS method, a low-power infrared diode is used to emit infrared light, which is then directed to the scanner's plane mirror. The scanner reflects the infrared light to the user's eyes, and then reflects it from the corneal surface to the optical sensor. The optical sensor converts the optical signal into an electrical signal, and finally identifies part of the eye data (the user's cornea, iris, and pupil information data, etc.), and finally determines the gaze point information based on the eye data. This method can only obtain low-precision eye tracking results, because its eye tracking accuracy is directly restricted by the number of optical signal receivers, and if too many optical signal receivers are deployed, the overall signal refresh rate will be affected.

[0038] Embodiment 1

[0039] Figure 1 This is a flow chart of an eye tracking method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of tracking eyes. The method can be performed by an eye tracking device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server. Specifically, it includes the following steps:

[0040] S110, acquiring an eye image and / or an infrared signal returned by the eye.

[0041] Among them, the eye image can be obtained by the image acquisition device in the eye tracking device to collect the user's eyes. Please refer to the eye image acquisition process when eye tracking is performed based on the pupil-corneal reflection method or the image contour extraction algorithm in the above embodiment, which will not be repeated here. The infrared signal can be understood as an infrared electrical signal converted by the optical sensor in the eye tracking device from the infrared light signal reflected back from the user's cornea. Please refer to the infrared signal acquisition process when eye tracking is performed based on the MEMS method in the above embodiment, which will not be repeated here.

[0042] S120, determining a sight line direction and first gaze point information according to the eye image.

[0043] The sight direction can be understood as a three-dimensional vector, and the gaze point information can be understood as the two-dimensional coordinates of the three-dimensional vector projected on a certain plane.

[0044] Optionally, if the pupil-corneal reflection method is used for eye tracking, the method for determining the line of sight direction and the first gaze point information based on the eye image can be: determining the pupil center and multiple light spot position information based on the eye image; determining the line of sight direction and the first gaze point information based on the pupil center and multiple light spot position information.

[0045] Among them, the light spot can be understood as the reflection point of multiple infrared light sources on the cornea. The method of determining the pupil center and multiple light spot position information based on the eye image can be: performing image analysis on the eye image, extracting the pupil area and the light spot area in the eye image, and then performing ellipse fitting on the pupil area, determining the center of the fitted ellipse as the pupil center, performing circle fitting on the light spot area, and determining the position of the center of the fitted circle as the light spot position. After obtaining the pupil center and multiple light spot position information, the user's line of sight direction and the first gaze point information are determined based on the relative position relationship between the pupil center and the light spot. In this embodiment, the process of determining the line of sight direction and the first gaze point information based on the eye image can refer to the relevant technology in the pupil-corneal reflection method, which is not limited here.

[0046] Optionally, if an image contour extraction algorithm is used for eye tracking, the method for determining the line of sight direction and the first gaze point information based on the eye image can be: determining the pupil center based on the eye image; determining the line of sight direction and the first gaze point information based on the pupil center and a pre-built eye 3D center model.

[0047] Among them, the method of determining the pupil center based on the eye image can be: performing image analysis on the eye image, extracting the pupil area in the eye image, and then performing ellipse fitting on the pupil area, and determining the center of the fitted ellipse as the pupil center. The principle of the eyeball 3D center model can be: obtaining the edge contour of the pupil through an image contour extraction algorithm, and then obtaining a pupil orientation in space based on spatial geometry (back projection). Multiple pupil orientation information can establish an eyeball 3D center model. The specific establishment process can refer to the relevant content of the image contour extraction algorithm in the above embodiment, which will not be repeated here. Specifically, after obtaining the pupil center, the pupil center is connected to the eyeball 3D center. According to the connection line, the user's gaze direction can be approximately calculated, thereby predicting the user's line of sight direction and gaze point information.

[0048] S130, determining second gaze point information according to the infrared signal.

[0049] In this embodiment, the process of determining the second gaze point information according to the infrared signal may be: first determining eye data such as the user's cornea, iris and pupil information according to the infrared signal, and then determining the second gaze point information according to the eye data.

[0050] S140, fusing the first gaze point information and the second gaze point information according to the sight direction to obtain target gaze point information.

[0051] In the above embodiment, when the pupil-corneal reflection method or the image contour extraction algorithm is used for eye tracking, the accuracy of the determined gaze point is related to the line of sight direction. The accuracy of eye tracking using the MEMS method is also relatively low. Therefore, it is necessary to fuse the first gaze point information and the second gaze point information according to the line of sight direction to improve the accuracy of the determined gaze point.

[0052] The user's eyes have a field of view, which may be in the shape of a cone. The field of view may be characterized by a field of view angle, for example, the field of view angle of the eye may be any value between 100 and 150 degrees.

[0053] Specifically, the first gaze point information and the second gaze point information are fused according to the line of sight direction to obtain the target gaze point information. The method can be: determining a first weight corresponding to the first gaze point information and a second weight corresponding to the second gaze point information according to the line of sight direction; performing weighted summation of the first gaze point information and the second gaze point information based on the first weight and the second weight to obtain the target gaze point information.

[0054] Among them, the line of sight direction represents the rotation angle of the user's eyeball. When the pupil-corneal reflection method or the image contour extraction algorithm is used for eye tracking, the accuracy of the determined gaze point is related to the line of sight direction (that is, the rotation angle of the eyeball). Therefore, it is necessary to determine the weights corresponding to the first gaze point information and the second gaze point information respectively according to the line of sight direction, so as to fuse the two gaze point information based on the determined weights.

[0055] Optionally, when the pupil-corneal reflection method is used for eye tracking, the method for determining the first weight corresponding to the first gaze point information and the second weight corresponding to the second gaze point information according to the line of sight direction can be: if the straight line where the line of sight direction is located is in the first field of view, then the first weight corresponding to the determined first gaze point information is greater than the second weight corresponding to the second gaze point information; if the straight line where the line of sight direction is located is in the second field of view, then the first weight corresponding to the determined first gaze point information is less than the second weight corresponding to the second gaze point information.

[0056] The first field of view area is a field of view area with a field of view angle less than the first set value; the second field of view area is a field of view area with a field of view angle greater than or equal to the first set value. In this application scenario, when the user looks straight ahead, the field of view angle is 0 degrees. The first set value can be any value less than the field of view angle of the user's eyes. For example: assuming that the field of view angle of the user's eyes is 110 degrees, the first set value can be set to 100 degrees. Exemplarily, Figure 2 is an example diagram of the distribution of sight lines in this embodiment. Figure 2 As shown, when the straight line where the line of sight direction is located is in the first field of view area, the first weight corresponding to the determined first gaze point information is greater than the second weight corresponding to the second gaze point information. The ratio between the first weight and the second weight can be fixed, or the ratio between the first weight and the second weight changes with the change of the eye rotation angle, for example, inversely proportional to the eye rotation angle. Exemplarily, when the user is facing forward, the line of sight direction is in the direction of the field of view angle of 0 degrees, the first weight can be set to 1, and the second weight is set to 0, that is, only the pupil-corneal reflection method is used to calculate the gaze point. As shown Figure 2 As shown, when the straight line where the sight line is located is in the second field of view, the first weight corresponding to the determined first gaze point information is less than the second weight corresponding to the second gaze point information. The ratio between the first weight and the second weight may be fixed, or the ratio between the first weight and the second weight may change with the change of the eyeball rotation angle, for example, inversely proportional to the eyeball rotation angle. When the straight line where the sight line is located is in the second field of view, it indicates that the rotation of the eyeball is lower and larger. At this time, the pupil in the eye image captured by the camera is deformed. Therefore, it is necessary to supplement the calculation of the user's gaze point information based on the MEMS method. Exemplarily, when the straight line where the sight line is located is in the second field of view, the first weight may be set to 0.2 and the second weight may be set to 0.8.

[0057] Optionally, when an image contour extraction algorithm is used for eye tracking, the method for determining the first weight corresponding to the first gaze point information and the second weight corresponding to the second gaze point information according to the line of sight direction can be: if the straight line where the line of sight direction is located is in the third field of view area or the fourth field of view area, then the first weight corresponding to the first gaze point information is determined to be less than the second weight corresponding to the second gaze point information; if the straight line where the line of sight direction is located is in the fifth field of view area, then the first weight corresponding to the first gaze point information is determined to be greater than the second weight corresponding to the second gaze point information.

[0058] Among them, the third field of view area is a field of view area with a field of view angle less than or equal to the second set value; the fourth field of view area is a field of view area with a field of view angle greater than or equal to the third set value. The fifth field of view area is a field of view area with a field of view angle greater than the second set value and less than the third set value. The third field of view area can be understood as an area with a smaller eyeball rotation angle, and the fourth field of view area can be understood as an area with a larger eyeball rotation angle. The second setting value can be any value between 5-10 degrees, and the third setting value can be any value less than the field of view angle of the user's eyes and greater than the second setting value. For example: assuming that the field of view angle of the user's eyes is 110, the third setting value can be any value between 90-100. Exemplarily, Figure 3 is an example diagram of the distribution of sight lines in this embodiment. Figure 3As shown, when the straight line where the sight line is located is in the third field of view, the user's pupil contour is close to a perfect circle. When it is slightly deflected by a few degrees, the pupil contour in the eye image does not change much, and the difference cannot be seen in the image, that is, it is not sensitive to the angle and the calculated gaze point is not accurate. It is necessary to use the MEMS method to supplement the calculation of the gaze information. Therefore, the first weight corresponding to the first gaze point information is determined to be less than the second weight corresponding to the second gaze point information. Among them, the ratio between the first weight and the second weight can be fixed, or the ratio between the first weight and the second weight changes with the change of the eyeball rotation angle, for example, it is proportional to the eyeball rotation angle. Exemplarily, when the user is facing forward, the sight line direction here is in the direction of the field of view angle of 0 degrees, the first weight can be set to 0.3, and the second weight is set to 0.7.

[0059] like Figure 3 As shown, when the straight line where the sight line is located is in the fourth field of view, the eye rotation angle is too large, and the pupil in the eye image captured by the camera is deformed. It is necessary to use the MEMS method to supplement the calculation of the gaze point information. Therefore, the first weight corresponding to the first gaze point information is determined to be less than the second weight corresponding to the second gaze point information. Among them, the ratio between the first weight and the second weight can be fixed, or the ratio between the first weight and the second weight changes with the change of the eyeball rotation angle, for example, inversely proportional to the eyeball rotation angle. Exemplarily, when the straight line where the sight line is located is in the fourth field of view, the first weight can be set to 0.2 and the second weight can be set to 0.8.

[0060] like Figure 3 As shown, when the straight line where the sight line is located is in the fifth field of view, the image contour extraction algorithm can be used to accurately track the eye, so the first weight corresponding to the first gaze point information is greater than the second weight corresponding to the second gaze point information. The ratio between the first weight and the second weight can be fixed, or the ratio between the first weight and the second weight changes with the change of the eye rotation angle, for example, inversely proportional to the eye rotation angle. Exemplarily, when the straight line where the sight line is located is in the fifth field of view, the first weight can be set to 0.8 and the second weight can be set to 0.2.

[0061] The technical solution of this embodiment obtains an eye image and / or an infrared signal returned by the eye; determines the sight direction and the first gaze point information according to the eye image; determines the second gaze point information according to the infrared signal; and fuses the first gaze point information and the second gaze point information according to the sight direction to obtain the target gaze point information. The eye tracking method provided by the embodiment of the present invention fuses the two gaze point information based on the sight direction to obtain the final gaze point information, which can improve the accuracy of eye tracking.

[0062] Embodiment 2

[0063] Figure 4 is a schematic diagram of the structure of an eye tracking system provided by Embodiment 2 of the present invention, such as Figure 4 As shown, the system includes: a first eye tracking module, a second eye tracking module and a gaze point determination module.

[0064] The first eye tracking module performs eye tracking based on a pupil corneal reflection method or an image contour extraction algorithm; the second eye tracking module performs eye tracking based on a micro-electromechanical system (MEMS) method;

[0065] The first eye tracking module is used to collect eye images and determine the line of sight direction and the first gaze point information based on the eye images; the second eye tracking module is used to collect the infrared signal returned by the eye and determine the second gaze point information based on the infrared signal; the gaze point determination module is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain the target gaze point information.

[0066] In this embodiment, when the first eye tracking module performs eye tracking based on the pupil cornea reflection method, the process of the gaze point determination module fusing the first gaze point information and the second gaze point information can be referred to the contents of the above embodiment and will not be repeated here.

[0067] In this embodiment, when the first eye tracking module performs eye tracking based on the image contour extraction algorithm, the process of the gaze point determination module fusing the first gaze point information and the second gaze point information can be referred to the contents of the above embodiment and will not be repeated here.

[0068] The technical solution of this embodiment realizes eye tracking by configuring and using a first eye tracking module and a second eye tracking module, which can improve the precision and accuracy of eye tracking.

[0069] Embodiment 3

[0070] Figure 5 is a schematic diagram of the structure of an eye tracking device provided in Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0071] An acquisition module 510 is used to acquire an eye image and / or an infrared signal returned by the eye;

[0072] A first gaze point information determination module 520, configured to determine a sight line direction and first gaze point information according to an eye image;

[0073] A second gaze point information determination module 530, configured to determine the second gaze point information according to the infrared signal;

[0074] The target gaze point information acquisition module 540 is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain the target gaze point information.

[0075] Optionally, the target gaze point information acquisition module 540 is further used to:

[0076] Determine, according to the line of sight direction, a first weight corresponding to the first gaze point information and a second weight corresponding to the second gaze point information;

[0077] The first gaze point information and the second gaze point information are weightedly summed based on the first weight and the second weight to obtain the target gaze point information.

[0078] Optionally, the target gaze point information acquisition module 540 is further used to:

[0079] If the straight line where the sight line is located is in the first field of view area, the first weight corresponding to the first gaze point information is determined to be greater than the second weight corresponding to the second gaze point information; wherein the first field of view area is a field of view area with a field of view angle less than a first set value;

[0080] If the straight line where the line of sight is located is in the second field of view area, the first weight corresponding to the determined first gaze point information is less than the second weight corresponding to the second gaze point information; wherein the second field of view area is a field of view area with a field of view angle greater than or equal to the first set value.

[0081] Optionally, the first gaze point information determining module 520 is further configured to:

[0082] Determine the pupil center and multiple light spot position information according to the eye image;

[0083] The sight direction and the first gaze point information are determined according to the pupil center and the position information of multiple light spots.

[0084] Optionally, the target gaze point information acquisition module 540 is further used to:

[0085] If the straight line where the sight line direction is located is in the third field of view area or the fourth field of view area, the first weight corresponding to the determined first gaze point information is less than the second weight corresponding to the second gaze point information; wherein the third field of view area is a field of view area with a field of view angle less than or equal to the second set value; and the fourth field of view area is a field of view area with a field of view angle greater than or equal to the third set value;

[0086] If the straight line where the line of sight is located is in the fifth field of view area, the first weight corresponding to the first gaze point information is determined to be greater than the second weight corresponding to the second gaze point information; wherein the fifth field of view area is a field of view area where the field of view angle is greater than the second set value and less than the third set value.

[0087] Optionally, the first gaze point information determining module 520 is further configured to:

[0088] Determining the pupil center based on the eye image;

[0089] The line of sight direction and the first gaze point information are determined based on the pupil center and the pre-built 3D center model of the eyeball.

[0090] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0091] Embodiment 4

[0092] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0093] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0094] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0095] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the eye tracking method.

[0096] In some embodiments, the eye tracking method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the eye tracking method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the eye tracking method in any other suitable manner (e.g., by means of firmware).

[0097] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0099] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0102] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0104] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An eye tracking method, characterized in that: include: Acquiring eye images and / or infrared signals returned by the eyes; Determining a sight line direction and first gaze point information according to the eye image; Determining second gaze point information according to the infrared signal; The first gaze point information and the second gaze point information are fused according to the line of sight direction to obtain target gaze point information.

2. The method according to claim 1, characterized in that The first gaze point information and the second gaze point information are fused according to the line of sight direction to obtain target gaze point information, including: Determine, according to the line of sight direction, a first weight corresponding to the first gaze point information and a second weight corresponding to the second gaze point information; The first gaze point information and the second gaze point information are weightedly summed based on the first weight and the second weight to obtain target gaze point information.

3. The method according to claim 2, characterized in that Determining a first weight corresponding to the first gaze point information and a second weight corresponding to the second gaze point information according to the line of sight direction includes: If the straight line where the sight line direction is located is in a first field of view area, the first weight corresponding to the first gaze point information is determined to be greater than the second weight corresponding to the second gaze point information; wherein the first field of view area is a field of view area with a field of view angle less than a first set value; If the straight line where the line of sight is located is in the second field of view area, the first weight corresponding to the first gaze point information is determined to be less than the second weight corresponding to the second gaze point information; wherein the second field of view area is a field of view area whose field of view angle is greater than or equal to the first set value.

4. The method according to claim 1, characterized in that: Determining the sight direction and first gaze point information according to the eye image includes: Determine the pupil center and multiple light spot position information according to the eye image; The sight direction and the first gaze point information are determined according to the pupil center and the multiple light spot position information.

5. The method according to claim 2, characterized in that: Determining a first weight corresponding to the first gaze point information and a second weight corresponding to the second gaze point information according to the line of sight direction includes: If the straight line where the sight line direction is located is in the third field of view area or the fourth field of view area, the first weight corresponding to the first gaze point information is determined to be less than the second weight corresponding to the second gaze point information; wherein the third field of view area is a field of view area with a field of view angle less than or equal to a second set value; and the fourth field of view area is a field of view area with a field of view angle greater than or equal to a third set value; If the straight line where the line of sight is located is in the fifth field of view area, the first weight corresponding to the first gaze point information is determined to be greater than the second weight corresponding to the second gaze point information; wherein the fifth field of view area is a field of view area where the field of view angle is greater than the second set value and less than the third set value.

6. The method according to claim 5, characterized in that Determining the sight direction and first gaze point information according to the eye image includes: determining a pupil center based on the eye image; The sight direction and the first gaze point information are determined based on the pupil center and a pre-constructed 3D center model of the eyeball.

7. An eye tracking device, characterized in that: include: An acquisition module, used for acquiring eye images and / or infrared signals returned by the eyes; A first gaze point information determination module, used to determine a sight line direction and first gaze point information according to the eye image; A second gaze point information determination module, used to determine the second gaze point information according to the infrared signal; The target gaze point information acquisition module is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain the target gaze point information.

8. An eye tracking system, characterized in that: include: A first eye tracking module, a second eye tracking module and a gaze point determination module; wherein the first eye tracking module performs eye tracking based on a pupil corneal reflection method or an image contour extraction algorithm; and the second eye tracking module performs eye tracking based on a micro-electromechanical system (MEMS) method; The first eye tracking module is used to collect eye images and determine the sight direction and first gaze point information according to the eye images; The second eye tracking module is used to collect infrared signals returned by the eyes, and determine the second gaze point information according to the infrared signals; The gaze point determination module is used to fuse the first gaze point information and the second gaze point information according to the line of sight direction to obtain target gaze point information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the eye tracking method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the eye tracking method according to any one of claims 1 to 6 when executed.

Images