Electronic device and eye tracking method

By combining infrared IR images and RGB images in electronic devices, the problem of low eye tracking accuracy caused by optical reflection when wearing glasses is solved, and a more efficient and accurate eye tracking effect is achieved.

CN117707330BActive Publication Date: 2024-11-08HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310570322.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-08
Estimated Expiration
2043-05-19

AI Technical Summary

Technical Problem

When existing electronic devices wear glasses, reflections caused by infrared light or near-infrared irradiation lead to low eye tracking accuracy, and even unable to achieve eye tracking.

Method used

The first camera is used to acquire infrared IR images and the second camera is used to acquire RGB images, determine the position of the reflected spot through the processor, and switch to use IR images or RGB images for eye tracking according to different situations to avoid the influence of reflected spots.

Benefits of technology

Improves the accuracy and reliability of eye tracking, overcomes the optical reflection problem when wearing glasses, and ensures accurate tracking under different lighting and distance conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117707330B_ABST
    Figure CN117707330B_ABST
Patent Text Reader

Abstract

The present application provides an electronic device, including a first camera, a second camera and a processor, wherein the first camera is used to obtain an IR image to be processed including first binocular information and first facial information of a target user, and the second camera is used to obtain a first RGB image including second binocular information and second facial information of the target user. The processor is used to determine the position of a reflection spot in the IR image to be processed, and when the reflection spot position is located in the eye region of the target user, control the second camera to obtain the first RGB image. According to the second binocular information and second facial information of the target user in the first RGB image, the gaze point position is obtained. Since there is no reflection spot in the RGB image, when the electronic device uses the RGB image without the reflection spot for eye tracking, the reflection spot can be avoided when the IR image is used for eye tracking, thereby improving the tracking accuracy of the eye tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an electronic device and an eye tracking method. Background Art

[0002] Eye tracking technology is a technology that enables electronic devices to track eye movements by measuring the location of the target user's eyes or the movement of the eyeballs relative to the head. Electronic devices can interact with target users through eye tracking technology, and then achieve specific functions, such as unlocking the screen of an electronic device or expanding the information bar of an electronic device.

[0003] Electronic devices generally use the pupil-corneal reflection method to implement eye tracking technology. The pupil-corneal reflection method is a method for estimating the gaze direction of the target user's eyes by extracting the relative displacement between the pupil center and the center of the corneal reflection highlight in the human eye image.

[0004] The human eye image is acquired based on the principle of infrared imaging. However, when the target user wears glasses, infrared light or near-infrared light will be irradiated on the surface of the glasses lens, which will produce severe reflections. When forming a portrait image, the glasses lens part will form a reflection spot with a brightness value higher than other parts. When the reflection spot blocks the human eye area in the human eye image, the electronic device can only recognize the high-brightness reflection spot formed by the glasses lens, and cannot recognize the human eye part. Therefore, the eye tracking effect of the electronic device is low in accuracy, or even cannot achieve eye tracking. Summary of the invention

[0005] The purpose of this application is to provide an electronic device and an eye tracking method to improve the accuracy of eye tracking effects.

[0006] In a first aspect, the present application provides an electronic device, comprising: a first camera, a second camera, and a processor;

[0007] The first camera is used to obtain an infrared IR image to be processed including first binocular information and first facial information of a target user; the second camera is used to obtain a first RGB image including second binocular information and second facial information of the target user; the processor is used to determine the position of a reflected light spot in the IR image to be processed; when the reflected light spot is located in the human eye region of the target user, the second camera is controlled to obtain the first RGB image; and the gaze point position is obtained according to the second binocular information and the second facial information of the target user in the first RGB image; when the reflected light spot is not in the human eye region of the target user, the gaze point position is obtained according to the first binocular information and the first facial information of the target user in the IR image to be processed.

[0008] Since RGB images are images obtained by reflecting visible light, they do not contain reflection spots. When electronic devices use RGB images without reflection spots for eye tracking, they can avoid the technical problem of low accuracy of eye tracking results or even failure to achieve eye tracking due to reflection spots, thereby improving the tracking accuracy of eye tracking. When the reflection spot is not in the human eye area or does not exist, using the IR image to be processed for eye tracking can overcome the problem of inaccurate eye tracking results caused by pupil reflection of visible light under normal lighting.

[0009] Optionally, the processor is further used to determine the glasses wearing state of the target user according to the IR image to be processed; when the glasses wearing state is wearing glasses, the IR image to be processed is binarized to obtain a binarized IR image; and the position of the reflected light spot is determined according to the binarized IR image. When the target user wears glasses, the glasses lenses emit infrared light, resulting in the existence of reflected light spots in the acquired IR image to be processed. Therefore, the glasses wearing state of the target user is first determined, and when it is determined that the target user is wearing glasses, the position of the reflected light spot is determined to simplify the processing flow.

[0010] Optionally, the processor is further configured to obtain the gaze point position according to the first binocular information and the first face information of the target user in the IR image to be processed when the glasses wearing state is no glasses. When the target user is not wearing glasses, the IR image to be processed does not contain a reflected light spot, and using the IR image instead of the RGB image can overcome the problem that the pupil reflects visible light under normal lighting, resulting in inaccurate eye tracking results using the RGB image.

[0011] Optionally, the processor is specifically used to control the first camera to obtain the IR image to be processed when the tracking distance is greater than or equal to a first preset threshold and less than or equal to a second preset threshold; the tracking distance is the distance between the human eye of the target user and the screen of the electronic device. When performing eye tracking, the distance from the human eye of the target user to the screen of the electronic device is considered. When the screen distance is too far, eye tracking cannot be performed using IR images. When the distance is too close, the target user is not within the camera's perception field of view, and the processor cannot obtain the human eye image of the target user, nor can eye tracking be performed. The tracking distance is limited to further ensure the tracking accuracy of eye tracking.

[0012] Optionally, the processor is further configured to: when the tracking distance is greater than a second preset threshold, control the second camera to obtain a second RGB image, wherein the second resolution of the second RGB image is greater than the first resolution of the first RGB image; and obtain the gaze point position according to the third binocular information and the third face information of the target user in the second RGB image. The high-resolution second RGB image is used for eye tracking, and the eye tracking accuracy is higher.

[0013] In an optional manner, before tracking is performed, the processor is further used to determine the illumination state of the ambient light according to the current light intensity of the ambient light; the illumination state includes a normal light state and an abnormal light state; the normal light state is the illumination state when acquiring an image of a first definition, and the abnormal light state is the illumination state when acquiring an image of a second definition, the first definition being greater than the second definition;

[0014] When the illumination state of the ambient light is a normal light state, the first camera and the second camera are turned on to calibrate the eye movement mapping relationship; the eye movement mapping relationship is a mapping relationship between a feature vector corresponding to an eye rotation image of a target user and a gaze point; the eye rotation image includes an IR image to be processed, a first RGB image and / or a second RGB image.

[0015] For different target users, the shape, size and structure of the human eye are different. When the target user's eye gaze point is at the same position, the eye movement information of different target users is different. That is, there is a difference in the correspondence between the eye movement information of different target users and the screen position of the landing point. When the correspondence between A's eye movement information and the screen position of the landing point is used to track B's eye movement, the eye tracking accuracy will be low. Through calibration, the above technical problems can be overcome and the accuracy of eye tracking can be further improved. In addition, during the calibration process, the illumination state of the ambient light is limited to the normal light state, and the first camera and the second camera are turned on for calibration. The purpose is to ensure that the electronic device is in the normal light state and the acquired image clarity is high, thereby improving the tracking accuracy of eye tracking. In addition, the camera data of the RGB camera and the TOF camera can be calibrated at one time, so the target user experience can be improved.

[0016] Optionally, the processor is further configured to prompt the target user to adjust the ambient light when the ambient light state is the abnormal light state; and when it is determined that the adjusted ambient light state is the normal light state, simultaneously start the first camera and the second camera to calibrate the eye movement mapping relationship, so as to ensure that the electronic device works under the normal light state.

[0017] Optionally, the processor is further configured to simultaneously activate the first camera and the second camera to calibrate the eye movement mapping relationship when the calibration distance is within a preset calibration range; the calibration distance is the distance from the target user's eyes to the screen of the electronic device during calibration. By limiting the distance, the accuracy of eye movement tracking during tracking is further improved.

[0018] Optionally, the processor is also used to simultaneously turn on the first camera and the second camera, and display at least one calibration point on the screen of the electronic device in turn; save the RGB image and the IR image with timestamp alignment, as well as the first feature vector corresponding to the RGB image and the second feature vector corresponding to the IR image, to complete the calibration process; the RGB image is acquired by the second camera mode; the IR image is acquired by the first camera mode; the first feature vector is used to represent the eye movement mapping relationship of the target user under the RGB image; the second feature vector user represents the eye movement mapping relationship of the target user under the IR image.

[0019] Optionally, when the first camera is a time-of-flight TOF camera, the second camera is an RGB camera.

[0020] In a second aspect, the present application provides an eye tracking method, comprising:

[0021] Obtain an infrared IR image to be processed including first binocular information and first facial information of the target user; determine the position of the reflected light spot in the IR image to be processed; when the reflected light spot is located in the human eye region of the target user, obtain the first RGB image, and obtain the gaze point position according to the second binocular information and second facial information of the target user in the first RGB image; when the reflected light spot is not in the human eye region of the target user, obtain the gaze point position according to the first binocular information and first facial information of the target user in the IR image to be processed. When the electronic device uses an RGB image without a reflected light spot for eye tracking, it can avoid the technical problem of low accuracy of eye tracking results or even failure to achieve eye tracking due to the reflected light spot, thereby improving the tracking accuracy of eye tracking. When the reflected light spot is not in the human eye region or does not exist, using the IR image to be processed for eye tracking can overcome the problem of inaccurate eye tracking results caused by pupil reflection of visible light under normal lighting.

[0022] Optionally, the glasses wearing state of the target user is determined according to the IR image to be processed; when the glasses wearing state is wearing glasses, the IR image to be processed is binarized to obtain a binarized IR image; and the position of the reflected light spot is determined according to the binarized IR image.

[0023] Optionally, when the glasses wearing state is not wearing glasses, the gaze point position is obtained according to the first pair of eyes information and the first face information of the target user in the IR image to be processed.

[0024] Optionally, when the tracking distance is greater than or equal to a first preset threshold and less than or equal to a second preset threshold, an IR image to be processed is acquired; the tracking distance is the distance between the human eyes of the target user and the screen of the electronic device.

[0025] Optionally, when the tracking distance is greater than a second preset threshold, a second RGB image is acquired, and a second resolution of the second RGB image is greater than the first resolution of the first RGB image; and the gaze point position is obtained based on binocular information and face information of the target user in the second RGB image.

[0026] Optionally, the illumination state of the ambient light is determined according to the current light intensity of the ambient light; the illumination state includes a normal light state and an abnormal light state; the normal light state is the illumination state when acquiring an image of a first definition, and the abnormal light state is the illumination state when acquiring an image of a second definition, the first definition being greater than the second definition;

[0027] When the illumination state of the ambient light is a normal light state, the first camera and the second camera are turned on to calibrate the eye movement mapping relationship; the eye movement mapping relationship is a mapping relationship between a feature vector corresponding to an eye rotation image of the target user and a gaze point, and the eye rotation image includes an IR image to be processed, a first RGB image and / or a second RGB image. The first camera is used to generate an IR image to be processed; and the second camera is used to generate a first RGB and / or a second RGB image.

[0028] Optionally, when the lighting state of the ambient light is an abnormal light state, the target user is prompted to adjust the ambient light; when it is determined that the lighting state of the adjusted ambient light is a normal light state, the first camera and the second camera are turned on at the same time to calibrate the eye movement mapping relationship.

[0029] Optionally, when the calibration distance is within a preset calibration range, the first camera and the second camera are turned on simultaneously to calibrate the eye movement mapping relationship; the calibration distance is the distance from the human eyes of the target user to the screen of the electronic device during calibration.

[0030] Optionally, the first camera and the second camera are turned on at the same time, and at least one calibration point is displayed on the screen of the electronic device in turn; the RGB image and the IR image with timestamp alignment, as well as the first feature vector corresponding to the RGB image and the second feature vector corresponding to the IR image are saved to complete the calibration process; the RGB image is acquired by the second camera mode; the IR image is acquired by the first camera mode; the first feature vector is used to represent the eye movement mapping relationship of the target user under the RGB image; the second feature vector user represents the eye movement mapping relationship of the target user under the IR image.

[0031] Optionally, when the first camera is a time-of-flight TOF camera, the second camera is an RGB camera.

[0032] In a third aspect, the present application provides a storage medium, comprising: computer-readable instructions, which, when executed on a computing device, cause the computing device to execute a method as described in any one of the second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A scene diagram for expanding and retracting the content of a text message information bar provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of an eye unlocking scenario provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of eye-controlled shooting provided in an embodiment of the present application;

[0036] Figure 4 Provide a naked eye 3D application scenario for the embodiment of the present application;

[0037] Figure 5 A schematic diagram of the structure of the electronic device 100 provided in this application;

[0038] Fig. 6A A schematic diagram of a method for opening eye tracking provided in an embodiment of the present application;

[0039] Figure 6B A flow chart of an eye tracking method provided in an embodiment of the present application;

[0040] Fig. 7A A flow chart of a calibration process provided in an embodiment of the present application;

[0041] Figure 7B A schematic diagram showing "reducing (or enhancing) the illumination intensity range of ambient light to be mlx to nlx" on an electronic device terminal interface;

[0042] Figure 7C It is a schematic diagram of 5 calibration points displayed in sequence on the terminal interface;

[0043] Figure 8 A schematic diagram showing the completion of a calibration process for an electronic device screen display provided in an embodiment of the present application;

[0044] Fig. 9 A flow chart of an eye tracking process provided in an embodiment of the present application;

[0045] Fig.10 An IR image to be processed with a reflection spot provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of a method for obtaining a gaze point position provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of a method for obtaining a gaze point position provided in an embodiment of the present application;

[0048] Fig.131 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to facilitate technical personnel to understand the technical solutions of the embodiments of the present application, the application scenarios of the eye tracking method provided by the present application are first introduced.

[0050] Application scenario 1: Expanding and retracting the content of the information bar.

[0051] Eye tracking technology can be applied to the information bar of electronic devices (such as the information bar of instant messaging software or SMS information bar, etc.) to realize the expansion and retraction of the content of the information bar. Figure 1 As shown, a scene diagram of expanding and retracting the content of a text message information bar provided in an embodiment of the present application is provided.

[0052] Figure 1 Part (a) shows that the target user's eyes are focused on the SMS information bar area 1000. Part (a) also shows the content displayed in the SMS information bar area 1000, including the message receiving time "Message XXXX / XX / XXXX:XX", the message source "159*******" and part of the message content "Dear customer, XXXX", and "Reply" and "Mark as read" prompt boxes.

[0053] When the preset gaze condition is met, the electronic device expands the short message information bar area 1000 to obtain (b). (b) shows the entire content of the information corresponding to the partial content of the information in (a) "Dear customer, XXXX, XXXX, please confirm as soon as possible, ***".

[0054] The preset gaze condition is a pre-stored expansion response condition in the electronic device. For example, the preset gaze condition may be that the duration of the landing point in the text message bar area exceeds a preset dwell time, or that the target user blinks at least twice within a preset time period, so that the landing point falls in the text message bar area 1000 at least twice, etc.

[0055] When the target user's eyes are positioned in areas other than the short message information bar area 1000, all the information content shown in (b) is automatically retracted, and the display content shown in (a) is obtained.

[0056] Application scenario 2: Eye unlocking

[0057] See also Figure 2 As shown, it is a schematic diagram of an eye unlocking scenario provided in an embodiment of the present application. Figure 2 (a) shows an interface display of an electronic device in a password locked state. The target user can unlock the electronic device without a trace by using the eye tracking technology to obtain the decoded display interface shown in (c).

[0058] Optionally, when the landing point is located in a preset area of ​​the screen, the electronic device can be directly unlocked without a trace to obtain the decoded display interface shown in (c).

[0059] Alternatively, when the landing point is located in the preset area of ​​the screen, the electronic device first enters the password input display interface shown in (b). When the landing point of the target user falls in the preset digital area, the electronic device can be unlocked without a trace, and the decoded display interface shown in (c) is obtained.

[0060] The number of preset areas can be one or more, and can be set as needed. The number of preset digital areas can be one or more, and can be set as needed. When there are multiple preset digital areas, the electronic device pre-stores the execution order of multiple preset digital areas, and when the landing point falls on each preset digital area in turn according to the execution order, the electronic device is unlocked without a trace.

[0061] Application scenario three: eye-controlled shooting.

[0062] See also Figure 3 3 is a schematic diagram of eye-controlled shooting provided by an embodiment of the present application. The electronic device completes camera focusing according to the location 3001 of the eyes of the target user.

[0063] When the electronic device completes the camera focus, the electronic device delays for a preset time and then triggers the camera to automatically shoot, or the electronic device triggers the camera to automatically shoot after recognizing that the target user has blinked a preset number of times. In this way, the electronic device can use the landing point position in the eye tracking technology to achieve automatic camera focus and shooting without manual operation.

[0064] In addition, eye tracking technology can also be used in scenarios such as naked-eye 3D, eye-movement browsing, identity recognition, and eye-controlled games.

[0065] Example description: See Figure 4 , the embodiment of the present application provides a naked eye 3D application scenario, the electronic device detects the spatial position of the human eye according to the eye tracking technology. According to the spatial position of the human eye, the 3D picture is adjusted in real time to enhance the viewing freedom. The spatial position of the human eye includes the distance from the human eye to the screen of the electronic device, the angle between the human eye's gaze line and the screen of the electronic device, etc.

[0066] Eye tracking technology is a technology that tracks eye movements by measuring the position of the target user's eyes or the movement of the eyes relative to the head.

[0067] In related schemes, electronic devices often use pupil-corneal reflection method to implement eye tracking technology. The pupil-corneal reflection method is a method for estimating the gaze direction of the target user's eyes by extracting the relative offset between the center of the pupil and the center of the corneal reflection highlight in the human eye image (that is, the target offset). The human eye image is formed by infrared light or near-infrared light irradiating the corneal surface and reflecting from the corneal surface.

[0068] However, when the target user wears glasses, infrared light or near-infrared light will cause severe reflections when it hits the surface of the glasses lens. When forming a portrait image, the glasses lens part will form a reflection spot with a brightness value higher than other parts. When the reflection spot blocks the human eye area in the human eye image, the electronic device can only recognize the high-brightness reflection spot formed by the glasses lens, and cannot recognize the human eye part. Therefore, the eye tracking effect of the electronic device is low in accuracy, or even cannot achieve eye tracking.

[0069] Based on this, an embodiment of the present application provides an electronic device, which includes a first camera, a second camera and a processor, wherein the first camera is used to obtain an infrared (IR) image to be processed including first binocular information and first facial information of a target user, and the second camera is used to obtain a first RGB image including second binocular information and second facial information of the target user. The processor is used to determine the position of the reflected light spot in the IR image to be processed, and when the reflected light spot position is located in the human eye region of the target user, control the second camera to obtain the first RGB image. The processor obtains the gaze point position according to the second binocular information and second facial information of the target user in the first RGB image. The processor is also used to obtain the gaze point position according to the first binocular information and first facial information of the target user in the IR image to be processed when the reflected light spot is not in the human eye region of the target user. When the electronic device uses an RGB image without a reflected light spot for eye tracking, it can avoid the technical problem that the reflected light spot exists when using an IR image for eye tracking, resulting in low accuracy of eye tracking results, or even inability to achieve eye tracking, thereby improving the tracking accuracy of eye tracking.

[0070] The electronic device provided in the embodiments of the present application may be a mobile phone with eye tracking function, a laptop computer, a wearable electronic device (such as a smart watch), a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, and a vehicle-mounted device, etc.

[0071] Figure 5 A schematic structural diagram of an electronic device 100 provided in the present application is shown.

[0072] The electronic device may include a processor 110, a display screen 194, a camera 193, an ambient light sensor 180L, and the like.

[0073] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0074] In an embodiment of the present application, the processor is used to determine the position of the reflected light spot in the IR image to be processed. When the reflected light spot is located in the human eye region of the target user, the second camera is controlled to obtain the first RGB image; and the gaze point position is obtained according to the second binocular information and the second face information of the target user in the first RGB image; when the reflected light spot is not in the human eye region of the target user, the gaze point position is obtained according to the first binocular information and the first face information of the target user in the IR image to be processed.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0076] The electronic device 100 can realize the shooting function through an image signal processor (ISP), a camera 193, a video codec, a GPU, a display screen 194 and an application processor.

[0077] ISP is used to process the data fed back by camera 193. In the embodiment of the present application, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In an embodiment of the present application, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0079] In the embodiment of the present application, the electronic device 100 includes a first camera 120 and a second camera 130. The first camera 120 includes a camera head 193 and an infrared light sensor 121. When eye tracking is performed, the shutter is opened, infrared light or near-infrared light is transmitted to the infrared light sensor 121 through the lens, and the optical signal is converted into an electrical signal. The infrared light sensor 121 transmits the electrical signal to the ISP for processing and converts it into an IR image to be processed that is visible to the naked eye. The second camera 130 includes a camera head 193 and a visible light sensor 131. When eye tracking is performed, the shutter is opened, and visible light including at least red light, green light and blue light is transmitted to the visible light sensor 131 through the lens, and the optical signal is converted into an electrical signal. The visible light sensor 131 transmits the electrical signal to the ISP for processing and converts it into a first RGB image and / or a second RGB image that is visible to the naked eye.

[0080] The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The embodiment of the present application can utilize an ambient light sensor to obtain the current intensity of the ambient light, and trigger a processor to determine the lighting state of the ambient light according to the current intensity of the ambient light. The lighting state includes a normal light state and an abnormal light state. Among them, the normal light state is the lighting state when acquiring an image of a first clarity, and the abnormal light state is the lighting state when acquiring an image of a second clarity, and the first clarity is greater than the second clarity.

[0081] The following uses the above-mentioned electronic device as an example to introduce the eye tracking method provided by the embodiment of the present application.

[0082] First, let’s introduce how to turn on the eye tracking function of electronic devices.

[0083] See also Fig. 6A , which is a schematic diagram of a method for opening eye tracking provided in an embodiment of the present application.

[0084] In a possible implementation, an "eye tracking" logo is set on the mobile phone shown in (a). The target user clicks the "eye tracking" logo, and the electronic device opens the eye tracking interface shown in (c) to perform eye tracking. The eye tracking interface includes a prompt box that displays functional controls such as eye cursor and function experience. For example, the prompt box prompts the target user to "look at the screen for auxiliary operation, 20-50 cm away from the screen, with your eyes facing the screen, and look to expand the banner notification: stay for a while to enter the details."

[0085] In another achievable manner, the target user clicks on the "Eye Tracking" application. In response to receiving the target user's operation to start the eye tracking application, the mobile phone first enters the calibration interface and displays the eye calibration interface in (b). The eye calibration interface displays a prompt box, prompting the target user to "make sure that the eyes and face are not blocked, follow the instructions, and look at 5 positions in the interface in turn to complete the calibration." When the target user completes the calibration, the mobile phone opens the "Eye Tracking" application and can display the eye tracking interface shown in (c).

[0086] When the "eye tracking" application is turned on on an electronic device, the following eye tracking method can be performed.

[0087] See also Figure 6B , is a flow chart of an eye tracking method provided in an embodiment of the present application, the method comprising:

[0088] S61: The first camera obtains an IR image to be processed and sends it to a processor.

[0089] The first camera refers to using infrared light or near infrared light to illuminate the surface of the target user, and the target user surface reflects to form an IR image to be processed. The first camera mode may be a Time Of Flight (TOF) camera.

[0090] The IR image to be processed refers to an IR image including first binocular information and first facial information of a target user, which is obtained under a first camera.

[0091] S62: The processor determines the position of the reflected light spot in the IR image to be processed.

[0092] The processor receives the IR image to be processed acquired by the first camera, and determines the position of the emission spot in the IR image to be processed.

[0093] In the embodiment of the present application, the reflected light spot is caused by the target user wearing glasses or there are other reflective components on the target user's face, which makes the infrared light or near-infrared light irradiated on the surface of the object to form a strong reflection, resulting in a light spot on the image surface with a brightness value higher than the brightness value of other parts.

[0094] In the IR image to be processed, the brightness value of the reflected light spot is higher than the brightness values ​​of other areas. Therefore, the electronic device can determine whether the reflected light spot exists and the position of the reflected light spot according to the brightness value in the IR image to be processed.

[0095] Optionally, in order to confirm the position of the reflected light spot, the electronic device first performs binarization processing on the IR image to be processed to obtain a binarized IR image. Binarization processing refers to setting the grayscale value of the pixel points on the image to 0 or 255, so that the entire image is clearly black and white. That is, the grayscale values ​​of the pixel points of the binary IR image only include 0 and 255.

[0096] In the embodiment of the present application, since the reflection spot is caused by the reflection of infrared light by the glasses lens, the brightness of the pixels in the reflection spot area is higher than the brightness of the pixels in other areas on the IR image to be processed.

[0097] The gray value of a pixel is used to represent the brightness of the pixel on the IR image to be processed. The larger the gray value of a pixel, the larger the brightness of the pixel. Therefore, the gray value of the pixel in the reflection spot area is greater than the gray value of the pixel in other areas on the IR image to be processed.

[0098] Optionally, the electronic device determines the pixel points whose grayscale values ​​on the IR image to be processed are greater than or equal to the grayscale threshold as the pixel points of the reflection spot area and sets the value to 255. The electronic device sets the pixel points whose grayscale values ​​on the IR image to be processed are less than the grayscale threshold to 0. The electronic device obtains a binarized IR image, and the binarized IR image can effectively obtain the reflection spot area.

[0099] S63: When the reflected light spot is located in the eye area of ​​the target user, the processor sends an acquisition instruction to the second camera.

[0100] The target user's eye region includes the target user's left eye region and the target user's right eye region. Optionally, the target user's eye region in the to-be-processed IR image may be located by facial key points.

[0101] When the determined reflected light spot is located in the eye area of ​​the target user, the processor sends an acquisition instruction to the second camera.

[0102] Optionally, the acquisition instruction includes information such as a first resolution of the image and image elements, so that the second camera acquires a first RGB image corresponding to the first resolution.

[0103] The second camera uses visible light to illuminate the surface of the target user, and the RGB image is formed by reflection from the surface of the target user. For example, the second camera mode can be an RGB camera mode.

[0104] S64: The second camera acquires the first RGB image and sends it to the processor.

[0105] The first RGB image refers to an RGB image including the second binocular information and the second face information of the target user. The image resolution of the first RGB image is the first resolution.

[0106] When there is a reflection spot on the IR image to be processed, and the reflection spot is located in the human eye area, the reflection spot in the IR image to be processed blocks the human eye because the brightness value of the reflection spot is greater than the brightness value of the human eye area. This makes it impossible for the electronic device to identify the eye movement information of the target user through the IR image to be processed, thereby causing the electronic device to be unable to achieve eye movement tracking. The eye movement information of the target user includes but is not limited to the gaze point, gaze time, gaze number, eye saccade distance or pupil size.

[0107] Therefore, when there is a reflection spot on the IR image to be processed and the reflection spot is located in the human eye area, the electronic device switches the camera from the first camera to the second camera. Under the second camera, a first RGB image without the reflection spot is obtained and sent to the processor, so that the processor uses the first RGB image to perform eye tracking.

[0108] S65: The processor obtains the gaze point position according to the second pair of eyes information and the second face information of the target user in the first RGB image.

[0109] The fixation point is the point on the screen where the target user's gaze falls when the user's eyes are fixed on the screen of the electronic device. The electronic device obtains the fixation point position according to the first RGB image.

[0110] Optionally, the processor obtains second binocular information and second facial information of the target user based on the first RGB image, and inputs the second binocular information and the second facial information into the RGB eye movement AI model to obtain the gaze point position.

[0111] The RGB eye-tracking AI model is a convolutional neural network model that is used to track the gaze point position of the target user on the screen of an electronic device in real time.

[0112] The input of the RGB eye movement AI model is the second binocular information and second facial information of the target user. The second binocular information and the second facial information can be RGB images, or can be binocular feature vectors and facial feature vectors.

[0113] Optionally, when the second binocular information and the second face information are RGB images, the electronic device can obtain the binocular information and face information of the target user by preprocessing the first RGB image and cropping the preprocessed first RGB image. The purpose of preprocessing is to avoid the influence of noise, etc., specifically including: denoising the first RGB image to reduce the first RGB image generated during the shooting or transmission process; smoothing the first RGB image to smooth the non-edge area and protect the image edge.

[0114] The output of the RGB eye movement AI model is the gaze point position. The specific output method can be the landing point mark on the screen or the two-dimensional coordinates of the gaze point on the screen.

[0115] The RGB eye movement AI model can be a non-intrusive eye movement AI model, such as a pupil-corneal reflection eye movement AI model, or an intrusive eye movement AI model. This application does not limit the specific form of the RGB eye movement AI model.

[0116] S66: When the reflected light spot is not located in the eye region of the target user, the processor obtains the gaze point position according to the first pair of eyes information and the first face information of the target user in the IR image to be processed.

[0117] The reflected light spot is not located in the human eye area, including the reflected light spot does not exist, or the reflected light spot exists but is not located in the human eye area. When the reflected light spot does not exist, or the reflected light spot exists but is not in the human eye area, the electronic device obtains the gaze point position according to the IR image to be processed.

[0118] Optionally, the electronic device obtains the first pair of eyes information and the first face information of the target user according to the IR image to be processed, and inputs the first pair of eyes information and the first face information into the IR eye movement AI model to obtain the gaze point position.

[0119] The IR eye movement AI model is a convolutional neural network model that is used to track the gaze point position of the target user on the screen of an electronic device in real time.

[0120] The input of the IR eye movement AI model is the first pair of eyes information and the first face information of the target user. The first pair of eyes information and the first face information can be an IR image or a pair of eyes feature vector and a face feature vector.

[0121] Optionally, when the first pair of eyes information and the first face information are IR images, the electronic device can obtain the first pair of eyes information and the first face information of the target user by preprocessing the IR image to be processed and cropping the preprocessed IR image to be processed. The purpose of preprocessing is to avoid the influence of noise, etc., specifically including: denoising the IR image to be processed to reduce the noise caused by the IR image to be processed during the shooting or transmission process; smoothing and filtering the IR image to be processed to smooth the non-edge area and protect the image edge.

[0122] The output of the IR eye movement AI model is the gaze point position. The specific output method can be the landing point mark on the screen or the two-dimensional coordinates of the gaze point on the screen.

[0123] The IR eye movement AI model can be a non-intrusive eye movement AI model, such as a pupil-corneal reflection eye movement AI model, or an intrusive eye movement AI model. This application does not limit the specific form of the RGB eye movement AI model.

[0124] Using IR images instead of RGB images for eye tracking can overcome the problem that the pupil reflects visible light under normal lighting, which leads to inaccurate eye tracking results using RGB images.

[0125] The electronic device first obtains the IR image to be processed, and determines whether the reflected light spot is in the human eye area based on the IR image to be processed. If the electronic device determines that the reflected light spot is in the human eye area, it obtains the first RGB image, and obtains the gaze point position based on the first RGB image to achieve eye tracking. Since the RGB image is an image obtained by reflecting visible light, the RGB image does not contain a reflected light spot. When the electronic device uses the RGB image without a reflected light spot for eye tracking, it can avoid the technical problem of low accuracy of eye tracking results or even failure to achieve eye tracking due to the reflected light spot, thereby improving the tracking accuracy of eye tracking. When the reflected light spot is not in the human eye area, or there is no reflected light spot, the IR image to be processed is used for eye tracking to overcome the problem of inaccurate eye tracking results caused by the pupil reflecting visible light under normal lighting.

[0126] When performing eye tracking, in addition to being affected by the reflected light spot, which leads to low tracking accuracy, other factors also affect it, such as individual differences of target users, the influence of ambient light, the distance between the target user and the screen of the electronic device, etc., which can also lead to low tracking accuracy. Based on this, an embodiment of the present application provides another eye tracking method on the electronic device.

[0127] Combine the following Figure 7A to Figure 12 , combined with the eye tracking method provided in the embodiments of the present application.

[0128] Among them, the first camera is a TOF camera, and the second camera is an RGB camera.

[0129] Since the shape, size and structure of human eyes are different for different target users, when the target user's eye gaze point is at the same position, the eye movement information of different target users is different. That is, the correspondence between the eye movement information of different target users and the screen position of the point of gaze is different. When the correspondence between A's eye movement information and the screen position of the point of gaze is used to track B's eye movement, the eye tracking accuracy will be low. Therefore, calibration is performed first before eye tracking.

[0130] The calibration process is also called the calibration process, which is used to calibrate the eye movement mapping relationship. The eye movement mapping relationship is the mapping relationship between the feature vector corresponding to the eye rotation image of the target user and the gaze point.

[0131] Recalibration is required when the target user uses the electronic device for the first time, the target user previously chose to exit the calibration process, the calibration process was unsuccessful when the target user last used the electronic device, the target user has deleted his calibration profile, or the electronic device was turned off and then back on.

[0132] See also Fig. 7A , is a flow chart of a calibration process provided in an embodiment of the present application, specifically including:

[0133] S71: The ambient light sensor obtains the current light intensity of the ambient light and sends it to the processor.

[0134] Ambient light refers to the light used to illuminate the environment in which the electronic device is located. The current light intensity of the ambient light is used to indicate the light intensity of the ambient light at the current moment.

[0135] The electronic device can obtain the current intensity of the ambient light in a variety of ways. For example, the electronic device can detect the ambient light using an ambient light sensor that it carries with it to obtain the current intensity of the ambient light.

[0136] In addition, the present application can also utilize other photosensitive sensors, such as an illumination sensor, to detect ambient light, and send the detection results to an electronic device so that the electronic device can obtain the current intensity of the ambient light, etc.

[0137] S72: The processor determines whether the current ambient light illumination state is a normal light state according to the current light intensity of the ambient light. If not, execute S73; if yes, execute S74.

[0138] The illumination state of the ambient light includes a normal light state and an abnormal light state.

[0139] The normal light state refers to the ambient light required for the electronic device to acquire an image of the first definition.

[0140] In the embodiment of the present application, the electronic device also includes an abnormal light state. The abnormal light state refers to the ambient light required for the electronic device to obtain an image of the second definition. The second definition is less than the first definition. That is, the electronic device obtains a low definition of the human eye image in the abnormal light state.

[0141] Abnormal light includes two types: a dark light state and a bright light state. A dark light state refers to a light state in which the light intensity is lower than the light intensity required for normal light. A bright light state refers to a light state in which the light intensity is higher than the light intensity required for normal light.

[0142] When the processor uses low-resolution human eye images for eye tracking, the tracking result is of low accuracy or even fails to achieve eye tracking. Therefore, in the actual calibration process, in order to ensure the effect of eye tracking, the processor needs to determine whether the current ambient light is normal.

[0143] The processor can determine this in the following ways:

[0144] The processor determines whether the current light intensity of the ambient light is within a preset light intensity range. If so, the lighting state of the ambient light is determined to be a normal light state; if not, the lighting state of the ambient light is determined to be an abnormal light state. Further, when the ambient light is determined to be an abnormal light state, if the current light intensity of the ambient light is lower than the minimum value of the preset light intensity range, the abnormal light state is determined to be a dark light state. If the current light intensity of the ambient light is greater than the maximum value of the preset light intensity range, the abnormal light state is determined to be a bright light state.

[0145] Exemplary description: Assume that the preset light intensity range set by the target user is [50lx, 20000lx]. When the current light intensity of the ambient light obtained by the electronic device is 3000lx, the current light intensity of the ambient light is within the preset light intensity range, and the lighting state of the ambient light is determined to be a normal light state. When the current light intensity of the ambient light obtained by the electronic device is 40lx, the current light intensity of the ambient light is not within the preset light intensity range, and the lighting state of the ambient light is determined to be an abnormal light state. Furthermore, since the current light intensity of the ambient light is lower than the minimum value of the preset light intensity range of 50lx, the abnormal light state is determined to be a dim light state.

[0146] In the embodiment of the present application, when the ambient light illumination state is normal light state, the electronic device executes the next calibration process of S74. When the ambient light illumination state is abnormal light state, the electronic device executes S73 to prompt the target user to adjust the ambient light.

[0147] S73: The processor prompts the target user to adjust the ambient light and executes S71.

[0148] When the ambient light is in a dim state, the target user is prompted to increase the ambient light intensity. When the ambient light is in a bright state, the target user is prompted to reduce the ambient light intensity.

[0149] The embodiment of the present application can display the adjustment content through the terminal interface of the electronic device. The adjustment content may include weakening (or strengthening) the illumination intensity of the ambient light, weakening (or strengthening) the specific illumination intensity range.

[0150] For example, see Figure 7B , the electronic device terminal interface displays a schematic diagram of "reducing (or increasing) the illumination intensity range of the ambient light to be mlx to nlx". The target user adjusts the ambient light according to the display result of the electronic device terminal interface.

[0151] Optionally, the processor sets a delay after the prompt. When the delay reaches a first preset time, the processor triggers the ambient light sensor to execute S71 again, reacquires the current intensity of the ambient light, and re-judges the ambient light to ensure that the adjusted ambient light is normal light.

[0152] Optionally, the processor can set a first preset number of cycles to prevent the electronic device from wasting memory space by looping judgments when the target user does not make adjustments. The electronic device records the number of acquisitions of the current light intensity of the ambient light during the calibration process. When the number of acquisitions is greater than the first preset number of cycles and the ambient light is still abnormal light, the electronic device turns off the eye tracking function.

[0153] S74: The processor starts the TOF camera.

[0154] When the illumination state of the ambient light is normal light, the processor starts the TOF camera.

[0155] The TOF camera is an active distance measurement method that directly measures the time difference between the round trip of infrared or near-infrared light to obtain the depth information of the object. Compared with the object depth obtained by RGB binocular cameras and structured light cameras, the TOF camera can measure the depth information of objects at a longer distance (less than or equal to 100m) and is not affected by ambient light and object texture. Therefore, the electronic device first starts the TOF camera to determine the distance from the target user's eyes to the electronic device.

[0156] S75: The TOF camera obtains a TOF depth map and sends it to the processor.

[0157] A TOF depth map is a real-time image obtained by a TOF camera that contains the depth information of an object. Each pixel value in a TOF depth map includes the distance from the pixel in the scene to the camera (i.e., the screen of the electronic device).

[0158] S76: The processor obtains a calibration distance according to the TOF depth map.

[0159] The calibration distance is the distance from the human eye to the screen of the electronic device during the calibration process.

[0160] S77: The processor determines whether the calibration distance is within a preset calibration range. If so, execute S79; otherwise, execute S78.

[0161] During calibration, if the calibration distance is too far or too close, the electronic device cannot accurately obtain the gaze point position when performing eye tracking. Therefore, the embodiment of the present application limits the calibration distance to within a preset calibration range.

[0162] In the embodiment of the present application, a calibration distance greater than the maximum value of the preset calibration range is regarded as a calibration distance that is too far, and a calibration distance less than the minimum value of the preset calibration range is regarded as a calibration distance that is too close. Exemplary explanation: Assume that the preset calibration range is 30cm to 50cm. The first distance is too close means that the distance from the human eye to the screen of the electronic device is less than 30cm, and the first distance is too far means that the distance from the human eye to the screen of the electronic device is greater than 50cm.

[0163] To improve the accuracy of obtaining the gaze point position, when the calibration distance is too close or too far, the processor executes S75 to prompt the target user to adjust the distance to the electronic device screen. When the calibration distance is within the preset calibration range, the processor executes the calibration process of S78.

[0164] It is worth noting that the preset calibration range is pre-stored in the processor. In actual application, the preset calibration range can be adjusted according to actual needs.

[0165] S78: The processor sends a prompt instruction to the calibration interface, and the calibration interface prompts the target user to adjust the distance to the electronic device, and executes S75.

[0166] When the calibration distance is too close, the processor sends a prompt instruction to the calibration interface, prompting the target user's eyes to move away from the screen of the electronic device. When the calibration distance is too far, the processor sends a prompt instruction to the calibration interface, prompting the target user's eyes to move closer to the screen of the electronic device. In this way, it is ensured that during the tracking process, the distance from the target user's eyes to the screen of the electronic device is within the preset calibration range, thereby improving the accuracy of obtaining the gaze point position.

[0167] Furthermore, in order to facilitate the target user to adjust the distance, the terminal interface displays the content that needs to be adjusted. The adjustment content includes moving away from (or close to) the screen of the electronic device, and the distance range of moving away (or close). For example, when the first distance is too close, the electronic device display interface may display "The target user's eyes are away from the screen of the electronic device, and the distance range of moving away (or close) is X1cm to X2cm".

[0168] Optionally, the processor may set a prompt delay. When the delay reaches a second preset time, the processor re-executes S75 and re-acquires the calibrated distance for judgment based on the real-time TOF depth map.

[0169] Optionally, the processor can set a second preset number of cycles to prevent the electronic device from wasting memory space by looping the judgment when the target user does not make any adjustments. The electronic device records the number of times the first distance is determined, and when the number of times the determination is greater than the second preset number of cycles, and the first distance is still not within the preset calibration range, the electronic device turns off eye tracking.

[0170] S79: The processor turns on the RGB camera and the TOF camera at the same time and sends the information to the terminal interface.

[0171] When the calibration is within the preset calibration range, the processor turns on the RGB camera and the TOF camera at the same time, and after the turning on is completed, sends a display instruction to the terminal interface, so that the terminal interface displays multiple points according to the display instruction.

[0172] An RGB camera is a camera mode that uses multiple visible light rays to illuminate the surface of an object and reflect from the surface of the object to form an RGB image. The multiple visible light rays include at least green light, red light, and blue light. It can be seen that, compared to a TOF camera, an RGB camera can obtain high-resolution RGB images that are not affected by reflected light. It is understandable that an RGB camera, when using an RGB stream to obtain an RGB image, does not have a reflected light spot.

[0173] S710: The terminal interface displays N calibration points in sequence, and when the calibration points are displayed, sends a display completion instruction to the processor.

[0174] The calibration point is also called a calibration point or a calibration target. The processor pre-stores the calibration point and the screen position of the calibration point on the screen. The number of calibration points is N, N ≥ 1, and N is an integer, such as N can be 1, 3, 5, 9 or 13. When N is multiple, the display order of multiple calibration points is preset. The processor stores the display order of multiple calibration points.

[0175] The terminal interface displays the calibration points in sequence. Figure 7CSchematic diagram of the five calibration points displayed in sequence on the terminal interface. Five calibration points ⑤ to ① are displayed in sequence from (a) to (e). The specific display method is as follows: when the target user's binocular gaze is at the calibration point ⑤ in (a), the terminal interface displays the calibration point ④ in (b). When the target user's binocular gaze is at the calibration point ④ in (b), the terminal interface displays the calibration point ③ in (c). When the target user's binocular gaze is at the calibration point ③ in (c), the terminal interface displays the calibration point ② in (d). When the target user's binocular gaze is at the calibration point ② in (d), the terminal interface displays the calibration point ① in (e).

[0176] As shown in the figure above, when the gaze points of the target user's eyes fall on the pre-stored calibration points in sequence, the terminal interface can calibrate the eye movement mapping relationship of the current target user based on the eye movement information of the current target user and the screen position of the calibration points.

[0177] The landing point screen position is the screen position of the landing point of the binocular gaze on the screen of the electronic device.

[0178] For different target users, the shape, size and structure of human eyes are different. When the target users' eyes are fixed on the same position, the eye movements of different target users are different. That is, the mapping relationship between the change of the fixation point caused by the eye movement of different target users and the target offset is also different. When the eye movement mapping relationship of A is used to track the eye movement of B, the eye tracking accuracy will be low. Therefore, the eye tracking accuracy can be improved by using the calibrated correspondence relationship for eye tracking.

[0179] S711: The processor saves the RGB image and the IR image with aligned timestamps, as well as the first eigenvector corresponding to the RGB image and the second eigenvector corresponding to the IR image, completing the calibration process.

[0180] The first feature vector is used to represent the eye movement mapping relationship of the target user under the RGB image. The RGB image is an image of the human eye obtained in the RGB camera mode. The electronic device inputs the RGB image into the first preset AI model, and the first preset AI model processes it to obtain the first feature vector.

[0181] The first preset AI model may be a feature extraction model based on a Histogram of Oriented Gradient (HOG), or a feature extraction model based on a Local Binary Pattern (LBP), etc. The embodiment of the present application does not limit the specific first preset AI model.

[0182] The second eigenvector is used to represent the eye movement mapping relationship of the target user under the IR image. The IR image is an image of the human eye obtained in the TOF camera mode. The electronic device inputs the IR image into the second preset AI model, and processes it through the second preset AI model to obtain the second eigenvector.

[0183] The second preset AI model may be a feature extraction model based on a Histogram of Oriented Gradient (HOG), or a feature extraction model based on a Local Binary Pattern (LBP), etc. The embodiment of the present application does not limit the specific second preset AI model.

[0184] Since the RGB camera mode and the TOF camera mode are turned on at the same time, the electronic device can obtain the RGB image and the IR image at the same time. The electronic device saves a plurality of consecutive RGB images and IR images at the same time, that is, the electronic device saves the RGB images and IR images with consecutive time stamp alignment.

[0185] The electronic device inputs the RGB image and the IR image at the same time into the corresponding first preset AI model and the second preset AI model, and can obtain the first feature vector and the second feature vector aligned with the timestamp. The electronic device saves the first feature vector and the second feature vector aligned with the timestamp.

[0186] The electronic device completes the calibration process. The target user can calibrate the camera data of the RGB camera and the TOF camera at one time in combination with the ambient light, thereby improving the target user experience.

[0187] See also Figure 8 , is a schematic diagram of completing the calibration process of an electronic device screen display provided in an embodiment of the present application. Figure 8 As shown, the electronic device screen displays the "calibration completed" logo. In addition, Figure 8 The screen of the electronic device can also display "Complete" and "Experience Now" prompt boxes. When the target user clicks the "Experience Now" prompt box, the electronic device performs the eye tracking experience process. When the target user clicks the "Complete" prompt box, the electronic device enters the eye tracking process.

[0188] See also Fig. 9 , is a flow chart of an eye tracking process provided by an embodiment of the present application. Specifically, it includes:

[0189] S91: The processor obtains the tracking distance.

[0190] The tracking distance is the distance between the target user's eyes and the screen of the electronic device during the eye tracking process. When the processor enters the eye tracking process, the tracking distance can be directly obtained based on the real-time TOF depth map.

[0191] The processor may enter the eye tracking process in a variety of ways, for example, it may enter the eye tracking process for the first time after calibration is completed, or it may enter the eye tracking process again after the eye tracking process is completed.

[0192] S92: The processor determines whether the tracking distance is less than a first preset threshold value. If so, S919 is executed; if not, S93 is executed.

[0193] The first preset threshold is the narrowest field of view perceived by the camera of the electronic device. The first preset threshold is a value less than the minimum value of the preset calibration range. Exemplary description: When the preset calibration range is 30cm to 50cm, the narrowest field of view perceived by the camera is generally 20cm, that is, the first preset threshold is 30cm.

[0194] To ensure the effectiveness of eye tracking, the processor must limit the tracking distance to no less than a first preset threshold. This is because when the tracking distance is less than the first preset threshold, the target user is not within the camera's field of view, and the processor cannot obtain the target user's eye image, nor can it perform eye tracking.

[0195] Therefore, when the processor determines that the second distance is less than the first preset threshold, the eye tracking is terminated. When the processor determines that the second distance is not less than the first preset threshold, the next step of the eye tracking process is performed.

[0196] S93: The processor determines whether the tracking distance is less than a second preset threshold value. If so, S94 is executed; if not, S916 is executed.

[0197] When the tracking distance is too far, the IR image obtained by the TOF camera has a low resolution or cannot be obtained due to the irradiation distance limitation of infrared light or near-infrared light. Therefore, when the tracking distance is too far, the processor uses the IR image obtained by the TOF camera to perform eye tracking, but the tracking accuracy is low or even cannot be achieved.

[0198] Tracking distance too far is used to indicate that the tracking distance is greater than the second preset threshold. The second preset threshold is the maximum value of the preset calibration range. For example, the preset calibration range is 30cm to 50cm, and the second preset threshold is set to 50cm. If the second distance is greater than 50cm, it means that the second distance is too far.

[0199] When the tracking distance is not greater than the second preset threshold, under the RGB camera, since the pupil of the target user's eye emits visible light, the tracking accuracy of the processor using the RGB image for eye tracking is lower than that using the IR image for eye tracking.

[0200] Therefore, the processor needs to determine the relationship between the tracking distance and the second preset threshold. When the tracking distance is too far, the processor triggers the camera to switch from the TOF camera to the RGB camera, otherwise the electronic device starts the TOF camera. In this way, the tracking accuracy of the processor for eye tracking is further improved.

[0201] S94: The processor starts the TOF camera.

[0202] When the tracking distance is greater than or equal to the first preset threshold and the tracking distance is less than or equal to the second preset threshold, the processor starts the TOF camera.

[0203] S95: The TOF camera obtains the IR image to be processed and sends it to the processor.

[0204] The TOF camera obtains the IR image to be processed according to the TOF stream. The IR image to be processed is an image including the first pair of eyes information and the first face information of the target user.

[0205] The TOF stream consists of multiple image frames arranged in time order. Each image frame is determined by subtracting the end frame (also called TX_OFF frame) from the start frame (also called TX_ON frame). The TOF camera obtains the IR image to be processed based on multiple consecutive image frames.

[0206] Optionally, the TOF camera extracts the eye area in the image based on pixel difference for two adjacent image frames to obtain the IR image to be processed. Optionally, the TOF camera can also use a continuous difference method for multiple image frames to obtain the IR image to be processed. The continuous difference method refers to respectively performing pixel difference between the current frame image and the previous frame image, and the next frame image and the current frame image to obtain two differential images. Based on the two differential images, the IR image to be processed is obtained. The IR image to be processed includes the same eye area and face area in the two differential images.

[0207] The embodiment of the present application may also obtain the IR image to be processed by other methods, which will not be described here.

[0208] S96: The processor determines whether the target user wears glasses. If yes, execute S97; if no, execute S99.

[0209] When the target user wears glasses, the glasses lenses emit infrared light, resulting in reflected light spots in the acquired IR image to be processed. Fig.10 , which is an IR image to be processed with a reflection spot provided in an embodiment of the present application. Fig.10 In the embodiment, the reflected light spot 1001 is located in the human eye region, blocking the eye. This makes the IR image to be processed not contain the eye movement information, resulting in the electronic device being unable to implement eye movement tracking.

[0210] The processor can determine whether the target user is wearing glasses in the following ways:

[0211] The IR image to be processed is input into the face attribute algorithm, and processed by the face attribute algorithm to determine whether the target user wears glasses. Optionally, the face attribute algorithm can be a face attribute algorithm based on a local Gabor binary pattern, or based on an AdaBoost algorithm, etc.

[0212] When the processor determines that the target user is not wearing glasses, the eye tracking is performed directly using the IR image to be processed. When the processor determines that the target user is wearing glasses, it further determines whether the reflected light spot is located in the target user's eye area.

[0213] S97: The processor extracts multiple facial key points from the IR image to be processed and obtains the eye area.

[0214] The key points of the face are used to locate the eye area of ​​the target user in the IR image to be processed. The key points of the face include at least 2 eye centers.

[0215] Optionally, the number of facial key points may be 5, 28, or 128, etc. The eye region may be located according to the facial key points.

[0216] The processor can extract the facial key points of the IR image to be processed in a variety of ways, such as using a Deep Alignment Network (DNA) to extract the facial key points of the IR image to be processed, or using a Multi-task Cascaded Convolutional Networks (MTCNN) to extract the facial key points of the IR image to be processed. This application does not limit the method of extracting facial key points.

[0217] S98: The processor determines whether the reflected light spot area is within the human eye area. If so, execute S911; if not, execute S99.

[0218] The electronic device determines whether the reflected light spot area is in the human eye area. If the reflected light spot area is in the human eye area, the reflected light spot blocks the target user's eyes, affecting the accuracy of eye tracking. At this time, the electronic device switches from the TOF camera mode to the RGB camera mode to avoid the problem of low tracking accuracy of eye tracking caused by the reflected light spot. If the reflected light spot area is not in the human eye area, that is, the reflected light spot area has no effect on eye tracking, the electronic device continues to perform eye tracking in the TOF camera mode.

[0219] In the embodiment of the present application, the reflection light spot area is in the human eye area, including one of the two situations that a part of the reflection light spot area is in the human eye area, or the whole of the reflection light spot area is in the human eye area.

[0220] S99: The processor extracts first binocular information and first face information of the IR image to be processed.

[0221] The first binocular information includes a left eye IR image and a right eye IR image of the target user, etc. The first face information is used to provide head posture information, including a face IR image and a face position, etc.

[0222] The processor crops the IR image to be processed to obtain the first pair of eyes information and the first face information. Fig.11 , is a schematic diagram of a method for obtaining the gaze point position provided in an embodiment of the present application. (a) is the IR image to be processed, and (b), (c), (d) and (e) are obtained by cropping. (b) is the right eye IR image, (c) is the left eye IR image, (d) is the face IR image, and (e) is the face position.

[0223] In order to avoid the influence of noise, the processor can pre-process the IR image to be processed before cropping it, including: denoising the IR image to be processed to reduce the noise caused by the IR image to be processed during shooting or transmission; smoothing and filtering the IR image to be processed to smooth the non-edge area and protect the image edge.

[0224] S910: The processor obtains a gaze point position according to the first pair of eyes information and the first face information.

[0225] The processor obtains the gaze point position according to the first pair of eyes information and the first face information.

[0226] In an optional manner, the processor inputs the left eye IR image and the right eye IR image in the first binocular information, and the face IR image and face position in the first face information into the IR eye movement AI model, and obtains the gaze point position through processing by the IR eye movement AI model.

[0227] Example description: Fig.11 Take the following example. The cropped results (b), (c), (d) and (e) are input into the IR eye-tracking AI model. Through the eye-tracking AI model processing, the electronic device can obtain the gaze point position.

[0228] The IR eye movement AI model can be a non-intrusive eye movement AI model, such as a pupil-corneal reflection eye movement AI model, or an intrusive eye movement AI model, which is not limited in this application.

[0229] S911: The processor switches the camera from the TOF camera to the RGB camera.

[0230] S912: The RGB camera acquires a first RGB image and sends it to the processor.

[0231] The first RGB image is an image of a first resolution acquired by an RGB camera, and the first RGB image includes second binocular information and second face information of the target user. The first resolution is greater than the resolution of the IR image to be processed.

[0232] The second binocular information includes the target user's left eye RGB image and right eye RGB image, etc. The second face information is used to provide head posture information, including face RGB image and face position, etc.

[0233] The electronic device uses an RGB camera to obtain a first RGB image. Compared with using a TOF camera to obtain an IR image to be processed, the first sub-RGB image does not include a reflected light spot. Therefore, using the first RGB image for eye tracking can obtain a higher accuracy of the gaze point position.

[0234] S913: The processor extracts second binocular information and second face information from the first RGB image.

[0235] The electronic device crops the first RGB image to obtain second binocular information and second face information. Fig.12 , is a schematic diagram of a method for obtaining the gaze point position provided in an embodiment of the present application. (a) is the RGB image to be processed, and (b), (c), (d) and (e) are obtained by cropping. (b) is the right eye IR image, (c) is the left eye IR image, (d) is the face IR image, and (e) is the face position. Among them, the RGB image to be processed does not contain reflected light spots.

[0236] In order to avoid the influence of noise, etc., the electronic device may pre-process the first RGB image before cropping the first RGB image, including: denoising the first RGB image to reduce the noise caused by the first RGB image during shooting or transmission; smoothing filtering the first RGB image to smooth the non-edge area and protect the image edge.

[0237] S914: The processor obtains the gaze point position according to the second pair of eyes information and the second face information.

[0238] The processor obtains the gaze point position according to the second pair of eyes information and the second face information.

[0239] In an optional manner, the processor inputs the left eye RGB image and the right eye RGB image in the second binocular information, and the face RGB image and face position in the first face information into the RGB image eye movement AI model, and obtains the gaze point position through processing by the RGB image eye movement AI model.

[0240] Example description: Fig.12Taking (b), (c), (d) and (e) as an example, input (b), (c), (d) and (e) into the RGB image eye movement AI model, and process them through the RGB image eye movement AI model to obtain the gaze point position.

[0241] The RGB eye movement AI model can be a non-intrusive eye movement AI model, such as an eye movement AI model based on the pupil-corneal reflection method, or it can be an intrusive eye movement AI model, which is not limited in this application.

[0242] S915: Processor converts the camera from TOF camera to RGB camera.

[0243] When the tracking distance is greater than or equal to a second preset threshold, the processor switches the camera from the TOF camera to the RGB camera.

[0244] S916: The RGB camera acquires a second RGB image and sends it to the processor.

[0245] The second RGB image is an image with a second resolution obtained by the RGB camera, and the second RGB image includes third binocular information and third facial information of the target user.

[0246] The second resolution is greater than the first resolution. In this way, eye tracking is performed using the high-resolution second RGB image, and the eye tracking accuracy is higher. Optionally, the second resolution can be 1080P and the first resolution is 640*680.

[0247] S917: The processor extracts third binocular information and third face information of the second RGB image.

[0248] The third binocular information includes the target user's left eye RGB image and right eye RGB image, etc. The third face information is used to provide head posture information, including face RGB image and face position, etc.

[0249] The processor crops the second RGB image to obtain third binocular information and third face information. The specific cropping method is the same as that of obtaining the second binocular information and the second face information, which will not be repeated here.

[0250] S918: The processor obtains the gaze point position according to the third pair of eyes information and the third face information.

[0251] The processor obtains the gaze point position according to the third pair of eyes information and the third face information.

[0252] In an optional manner, the processor inputs the left eye RGB image and the right eye RGB image in the third binocular information, and the face RGB image and the face position in the third face information into the RGB eye movement AI model, and generates a two-dimensional coordinate, which is the gaze point position, through processing by the RGB eye movement AI model.

[0253] The RGB eye movement AI model is the same as step S914 and will not be discussed here.

[0254] S919: The processor ends the tracking process.

[0255] The processor can end the tracking process in a variety of ways. For example, the eye tracking device is turned off to end the tracking process, or after obtaining the gaze point position, the electronic device screen displays an "end tracking" prompt box, and the target user clicks "end tracking" to end the tracking process. The eye tracking device can be turned off by clicking the "off" mark of the eye tracking device to turn off the eye tracking device and end the tracking process, or a preset number of blinks is set. After obtaining the gaze point position, the target user blinks the preset number of blinks, triggering the eye tracking device to automatically turn off and end the tracking, etc.

[0256] An eye tracking method provided in an embodiment of the present application first performs calibration and then performs tracking. The tracking process fully considers conditions such as the distance between the target user's eyes and the screen of the electronic device, whether the user wears glasses, and whether the reflected light spot blocks the eyes. When the distance between the target user's eyes and the screen of the electronic device is too far, the electronic device switches to the RGB camera, obtains a high-resolution second RGB image, and uses the high-resolution second sub-RGB image for eye tracking. When the distance between the target user's eyes and the screen of the electronic device is between the first preset threshold and the second preset threshold, the TOF camera is first used to obtain an IR image to determine whether the target user wears glasses and whether the reflected light spot blocks the eye area. Then the electronic device switches the camera based on whether the reflected light spot blocks the eye area. When the reflected light spot blocks the eye area, the electronic device switches to the RGB camera to obtain the RGB image for eye tracking. When the reflected light spot does not block the eye area or the target user does not wear glasses, the IR image is directly used for eye tracking. In this way, when performing eye tracking, the technical problems of using only TOF cameras, severe glare from the target user's eyes, resulting in poor eye tracking effect, and eye tracking not being able to work properly when the target user's eyes are far away from the screen of the electronic device are solved. The problem of using only RGB cameras instead of TOF cameras, which will affect the accuracy of eye tracking under normal circumstances due to the pupil reflecting visible light, is overcome, thereby improving the accuracy of eye tracking in various extreme scenarios.

[0257] In addition, the embodiment of the present application combines depth information to dynamically adjust the resolution of the RGB image, which can effectively improve the target user's usage distance and usage experience. When calibrating, combined with the ambient light information, the target user can calibrate the RGB and TOF dual-camera calibration data in turn, further improving the target user's experience.

[0258] Fig.13 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0259] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0260] The application layer can include a series of application packages.

[0261] like Fig.13 As shown, the application package may include applications such as camera eye tracking.

[0262] The first camera is used to obtain an infrared IR image to be processed including first binocular information and first facial information of the target user. The second camera is used to obtain a first RGB image including second binocular information and second facial information of the target user.

[0263] In another optional manner, the second camera is further used to obtain a second RGB image including third binocular information and third face information of the target user.

[0264] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0265] like Fig.13 As shown, the application framework layer may include eye tracking methods, face attribute algorithms, eye movement AI models, etc.

[0266] The eye tracking method may be: obtaining an infrared IR image to be processed including first binocular information and first facial information of a target user; determining the position of a reflected light spot in the IR image to be processed; when the reflected light spot is located in the human eye region of the target user, obtaining a first RGB image, and obtaining a gaze point position based on second binocular information and second facial information of the target user in the first RGB image; when the reflected light spot is not in the human eye region of the target user, obtaining a gaze point position based on the first binocular information and first facial information of the target user in the IR image to be processed.

[0267] The face attribute algorithm is used to determine whether the target user in the IR image is wearing glasses.

[0268] The eye-tracking AI model user processes the target image, such as the IR image to be processed, the first RGB image, and the second RGB image, to obtain the gaze point position.

[0269] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0270] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0271] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0272] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0273] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0274] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0275] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0276] A 2D graphics engine is a drawing engine for 2D drawings.

[0277] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0278] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0279] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An electronic device, characterized in that: include: a first camera, a second camera, and a processor; The first camera is used to obtain an infrared IR image to be processed including first binocular information and first facial information of a target user; The second camera is used to obtain a first RGB image including second binocular information and second face information of the target user; The processor is used to determine the position of the reflected light spot in the IR image to be processed; when the reflected light spot is located in the human eye area of ​​the target user, control the second camera to obtain the first RGB image; and obtain the gaze point position according to the second binocular information and the second facial information of the target user in the first RGB image; when the reflected light spot is not in the human eye area of ​​the target user, obtain the gaze point position according to the first binocular information and the first facial information of the target user in the IR image to be processed.

2. The device according to claim 1, characterized in that: The processor is further configured to: Determining the glasses wearing state of the target user according to the IR image to be processed; When the glasses wearing state is wearing glasses, the IR image to be processed is binarized to obtain a binarized IR image; and the position of the reflected light spot is determined according to the binarized IR image.

3. The device according to claim 2, characterized in that: The processor is further configured to: When the glasses wearing state is not wearing glasses, the gaze point position is obtained according to the first binocular information and the first face information of the target user in the IR image to be processed.

4. The device according to claim 1, characterized in that: The processor is specifically used for: When the tracking distance is greater than or equal to a first preset threshold and less than or equal to a second preset threshold, the first camera is controlled to acquire the IR image to be processed; the tracking distance is the distance between the human eyes of the target user and the screen of the electronic device.

5. The device according to claim 4, characterized in that: The processor is further configured to: When the tracking distance is greater than the second preset threshold, controlling the second camera to acquire a second RGB image, wherein a second resolution of the second RGB image is greater than a first resolution of the first RGB image; The processor is further configured to obtain the gaze point position according to third binocular information and third face information of the target user in the second RGB image.

6. The device according to any one of claims 1 to 5, characterized in that: The processor is further configured to: Determine, according to the current light intensity of the ambient light, the illumination state of the ambient light; the illumination state includes a normal light state and an abnormal light state; the normal light state is the illumination state when acquiring an image of a first definition, and the abnormal light state is the illumination state when acquiring an image of a second definition, the first definition being greater than the second definition; When the illumination state of the ambient light is the normal light state, turning on the first camera and the second camera, Calibrate the eye movement mapping relationship; the eye movement mapping relationship is the mapping relationship between the feature vector corresponding to the eye rotation image of the target user and the gaze point; the eye rotation image includes the IR image to be processed, the first RGB image and / or the second RGB image.

7. The device according to claim 6, characterized in that: The processor is further configured to: When the illumination state of the ambient light is the abnormal light state, prompting the target user to adjust the ambient light; When it is determined that the adjusted illumination state of the ambient light is the normal light state, the first camera and the second camera are turned on simultaneously to calibrate the eye movement mapping relationship.

8. The device according to claim 6, characterized in that: The processor is further configured to: When the calibration distance is within a preset calibration range, the first camera and the second camera are turned on at the same time to calibrate the eye movement mapping relationship; the calibration distance is the distance from the human eyes of the target user to the screen of the electronic device during calibration.

9. The device according to claim 6, characterized in that: The processor is further configured to: Simultaneously turning on the first camera and the second camera, and sequentially displaying at least one calibration point on the screen of the electronic device; Saving the RGB image and the IR image with aligned timestamps, as well as the first eigenvector corresponding to the RGB image and the second eigenvector corresponding to the IR image, to complete the calibration process; The RGB image is acquired by the second camera mode; the IR image is acquired by the first camera mode; the first feature vector is used to represent the eye movement mapping relationship of the target user under the RGB image; the second feature vector user represents the eye movement mapping relationship of the target user under the IR image.

10. The device according to claim 1, characterized in that: When the first camera is a time-of-flight TOF camera, the second camera is an RGB camera.

11. A method for eye tracking, characterized in that: The method comprises: Acquire an infrared IR image to be processed including first pair of eyes information and first face information of a target user; Determining the position of the reflected light spot in the IR image to be processed; When the reflected light spot is located in the eye region of the target user, a first RGB image is acquired, and a gaze point position is obtained according to second binocular information and second face information of the target user in the first RGB image; When the reflected light spot is not in the eye region of the target user, the gaze point position is obtained according to the first binocular information and the first face information of the target user in the IR image to be processed.

12. The method according to claim 11, characterized in that: The determining the position of the reflected light spot in the IR image to be processed comprises: Determining the glasses wearing state of the target user according to the IR image to be processed; When the glasses wearing state is wearing glasses, the IR image to be processed is binarized to obtain a binarized IR image; and the position of the reflected light spot is determined according to the binarized IR image.

13. The method according to claim 12, characterized in that: The method further comprises: When the glasses wearing state is not wearing glasses, the gaze point position is obtained according to the first binocular information and the first face information of the target user in the IR image to be processed.

14. The method according to claim 11, characterized in that: The step of obtaining an infrared IR image to be processed including first binocular information and first facial information of a target user includes: When the tracking distance is greater than or equal to a first preset threshold and less than or equal to a second preset threshold, the IR image to be processed is acquired; the tracking distance is the distance between the human eyes of the target user and the screen of the electronic device.

15. The method according to claim 14, characterized in that: The method further comprises: When the tracking distance is greater than the second preset threshold, acquiring a second RGB image, wherein a second resolution of the second RGB image is greater than a first resolution of the first RGB image; The gaze point position is obtained according to the binocular information and face information of the target user in the second RGB image.

16. The method according to any one of claims 11 to 15, characterized in that: The method further comprises: Determine, according to the current light intensity of the ambient light, the illumination state of the ambient light; the illumination state includes a normal light state and an abnormal light state; the normal light state is the illumination state when acquiring an image of a first definition, and the abnormal light state is the illumination state when acquiring an image of a second definition, the first definition being greater than the second definition; When the illumination state of the ambient light is the normal light state, turning on the first camera and the second camera, Calibrate the eye movement mapping relationship; the eye movement mapping relationship is the mapping relationship between the feature vector corresponding to the eye rotation image of the target user and the gaze point; the eye rotation image includes the IR image to be processed, the first RGB image and / or the second RGB image; the first camera is used to generate the IR image to be processed; the second camera is used to generate the first RGB and / or second RGB image.

17. The method according to claim 16, characterized in that: The method further comprises: When the illumination state of the ambient light is the abnormal light state, prompting the target user to adjust the ambient light; When it is determined that the adjusted illumination state of the ambient light is the normal light state, the first camera and the second camera are turned on simultaneously to calibrate the eye movement mapping relationship.

18. The method according to claim 17, characterized in that: The simultaneously starting the first camera and the second camera and calibrating the eye movement mapping relationship includes: When the calibration distance is within a preset calibration range, the first camera and the second camera are turned on at the same time to calibrate the eye movement mapping relationship; the calibration distance is the distance from the human eyes of the target user to the screen of the electronic device during calibration.

19. The method according to claim 17, characterized in that: The simultaneously starting the first camera and the second camera and calibrating the eye movement mapping relationship includes: Simultaneously turning on the first camera and the second camera, and sequentially displaying at least one calibration point on the screen of the electronic device; Saving the RGB image and the IR image with aligned timestamps, as well as the first eigenvector corresponding to the RGB image and the second eigenvector corresponding to the IR image, to complete the calibration process; The RGB image is acquired by the second camera mode; the IR image is acquired by the first camera mode; the first feature vector is used to represent the eye movement mapping relationship of the target user under the RGB image; the second feature vector user represents the eye movement mapping relationship of the target user under the IR image.

20. The method according to claim 16, characterized in that: When the first camera is a time-of-flight TOF camera, the second camera is an RGB camera.