Eye tracking method and device, and storage medium

Abstract

The present application provides an eye tracking method and device, and a storage medium. The method is applied to an electronic device comprising a first infrared light source and a camera module. The camera module comprises a receiver and a transmitter that integrates a second infrared light source. The electronic device of this structure determines a current service scenario thereof, and then when the electronic device is in a high-precision eye tracking scenario, controls the two infrared light sources to both emit infrared light to a target object having the eyes of a user, and receives, by means of the receiver, reflected infrared light, which is first infrared light and second infrared light reflected from the target object, to generate a first infrared image of the eyes, so that the electronic device can perform eye tracking on the basis of the first infrared image. Because the camera module itself comprises an infrared light source, high-precision eye tracking can be achieved by adding only one additional infrared light source, thereby ensuring the precision of eye tracking, and reducing the number of holes on a screen of the electronic device.

Description

Eye tracking method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 24, 2023, with application number 202311391303.6 and invention name “Eye tracking method, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of human-computer interaction technology, and in particular to an eye tracking method, device, and storage medium. Background Art

[0003] Eye tracking is an application technology that uses sensors, computers, machine vision, and other technologies to track human eye movements. This technology is widely used in fields such as virtual reality (VR), augmented reality (AR), healthcare, education, intelligent driving, and psychological analysis, and is currently a research hotspot in human-computer interaction.

[0004] Currently, common eye tracking devices include telemetry (screen-based) and head-mounted devices. The former requires a long, strip-shaped acquisition and computing device (including multiple infrared light sources and infrared cameras) to be placed above or below the electronic device (external), while the latter integrates a series of hardware into glasses. While both can effectively achieve eye tracking, they are generally expensive and inconvenient to carry.

[0005] In addition, if telemetry devices are integrated into electronic devices such as mobile phones and tablets, it will not only increase the power consumption of the hardware, but also increase the number of openings on the screen.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides an eye tracking method, device and storage medium, which aims to use the camera module that can emit infrared light that is originally integrated in the electronic device, and then add at least one additional infrared light source. It can control the operation of different infrared light sources according to the actual business scenarios of the electronic device, and then obtain different infrared images, thereby taking into account the hardware power consumption and achieving eye tracking with different degrees of accuracy.

[0008] In a first aspect, the present application provides an eye tracking method, which is applied to an electronic device, the electronic device comprising: a first infrared light source and a camera module, the camera module comprising a transmitter and a receiver, and the transmitter comprising a second infrared light source. The eye tracking method comprises: the electronic device determining the current business scenario; when the electronic device is in the first business scenario, controlling the first infrared light source to emit a first infrared light toward a target object, and controlling the second infrared light source to emit a second infrared light toward the target object, the target object comprising the user's eye, and the first business scenario being a high-precision eye tracking scenario; the receiver receiving a first infrared reflected ray of the first infrared light reflected by the target object, and a second infrared reflected ray of the second infrared light reflected by the target object, to generate a first infrared image of the eye; and the electronic device performing eye tracking based on the first infrared image.

[0009] The number of the first infrared light sources may be one or more.

[0010] Among them, the number of the second infrared light source is 1.

[0011] Among them, electronic devices include mobile phones, tablet computers and other portable devices.

[0012] Among them, the camera module is the hardware that the electronic device originally has, and the first infrared light source is an additional infrared light source.

[0013] The transmitter in the camera module may include not only a second infrared light source but also a driving control unit for driving the second infrared light source.

[0014] Among them, the receiver in the camera module can be regarded as an image sensor, which can not only receive the infrared reflected light reflected by the target object, but also generate an infrared image of the current object in the current scene based on the received infrared reflected light.

[0015] High-precision eye tracking scenarios, for example, require high eye tracking accuracy but are not constrained by power consumption (i.e., they don't require long-term eye tracking), such as identifying the app icon being looked at and eye focusing. These scenarios require an infrared camera coupled with multiple infrared light sources to capture infrared images, and then implement eye tracking based on pupil-corneal reflection (PCCR).

[0016] The operation of determining the current business scenario of the electronic device may be implemented, for example, by the application layer of the electronic device.

[0017] Among them, the operation of performing eye tracking based on the obtained first infrared image can be implemented, for example, by a processing unit in an electronic device processor, such as one or more of an application processor (AP), an image signal processor (ISP), and a digital signal processor (DSP).

[0018] Therefore, for an electronic device with a camera module, and the camera module itself has an infrared light source, by adding one or more infrared light sources (first infrared light sources), when the electronic device of this structure turns on the eye tracking function, by determining the current business scenario of the electronic device, and then in the case of high-precision eye tracking, the infrared light source built into the camera module and the newly added infrared light source are controlled to emit infrared light toward the target object, and the receiver receives the infrared reflected light reflected by the target object to generate a first infrared image of the eye, so that the electronic device can analyze and process the first infrared image according to the PCCR, thereby realizing eye tracking. Since the camera module itself has an infrared light source, high-precision eye tracking can be achieved by adding at least one additional infrared light source, thereby ensuring the accuracy of eye tracking and reducing the number of screen openings of the electronic device.

[0019] According to the first aspect, a receiver receives a first infrared reflected ray of the first infrared light reflected by the target object, and a second infrared reflected ray of the second infrared light reflected by the target object, to generate a first infrared image of the eye, including: the receiver receives the first infrared reflected ray of the first infrared light reflected by the cornea and retina of the eye, and the second infrared reflected ray of the second infrared light reflected by the cornea and retina; wherein the position of the corneal light spot formed by the cornea reflecting the first infrared reflected ray and the second infrared reflected ray is fixed, and the light reflected on the retina indicates the direction of the pupil of the eye.

[0020] According to the first aspect, or any implementation of the first aspect above, the electronic device performs eye tracking based on the first infrared image, including: the electronic device uses the pupil corneal reflection method to determine the direction of eye movement based on the vector between the corneal light spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil.

[0021] According to the first aspect, or any implementation method of the first aspect above, the electronic device adopts the pupil-corneal reflection method to determine the direction of movement of the eye according to the vector between the corneal spot formed by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil, including: the electronic device corrects the user's head posture according to the 3D depth information provided by the camera module; the electronic device adopts the pupil-corneal reflection method to determine the direction of movement of the eye according to the vector between the corneal spot formed by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil.

[0022] Understandably, a camera module with a second infrared light source, such as a TOF module or a structured light module, can not only capture infrared images but also provide 3D depth images. Therefore, by integrating the 3D depth information provided by the camera module, the generalization of poses can be improved, making eye tracking results based on infrared images corrected for 3D depth information more accurate.

[0023] According to the first aspect, or any implementation of the first aspect above, the method also includes: when the electronic device is in a second business scenario, controlling the second infrared light source to emit second infrared light to the target object, the second business scenario being a low-precision eye tracking scenario; the receiver receives a third infrared reflected light reflected by the target object from the second infrared light, and generates a second infrared image of the eye; and the electronic device performs eye tracking based on the second infrared image.

[0024] Low-precision eye tracking scenarios, for example, require low accuracy but high power consumption (requiring prolonged use of the eye tracking function), such as eye-controlled page turning / swiping when reading e-books or watching short videos. These scenarios do not require additional infrared light sources and only utilize the infrared light source built into the camera module's emitter for illumination. Subsequent analysis and processing of the infrared image using apparent artificial intelligence (AI) eye tracking technology enables eye tracking.

[0025] Therefore, in low-precision eye tracking scenarios, there is no need to use an additional first infrared light source. Only the second infrared light source built into the emitter in the camera module is used for lighting. Then, the obtained infrared image is analyzed and processed based on the apparent AI eye tracking technology, which can not only achieve eye tracking but also ensure the hardware power consumption of the electronic device.

[0026] According to the first aspect, or any implementation of the first aspect above, the receiver receives the third infrared reflected light of the second infrared light reflected by the target object to generate a second infrared image of the eye, including: the receiver receives the third infrared reflected light of the second infrared light reflected by the user's face; and generates a second infrared image based on the third infrared reflected light.

[0027] According to the first aspect, or any implementation method of the first aspect above, the electronic device performs eye tracking based on the second infrared image, including: the electronic device determines the key points of the area where the eyes are located in the second infrared image; the electronic device intercepts the image area including the key from the second infrared image to obtain an infrared image of the eye; the electronic device uses apparent artificial intelligence eye tracking technology to determine the direction of eye movement based on the appearance of the user's face and eyes in the second infrared image and the position of the eye's gaze point in the infrared image of the eye.

[0028] According to the first aspect, or any implementation method of the first aspect above, a first infrared light source and a camera module are arranged under the screen of an electronic device; when the electronic device is in a first business scenario, the first infrared light source is controlled to emit a first infrared light toward the target object, and the second infrared light source is controlled to emit a second infrared light toward the target object, including: when the electronic device is in the first business scenario, the first infrared light source is controlled to emit a first infrared light toward the target object with a first intensity, and the second infrared light source is controlled to emit a second infrared light toward the target object with a first intensity; wherein the first intensity is higher than the second intensity, and the second intensity is the luminous intensity of the first infrared light source and the second infrared light source when the first infrared light source and the camera module are arranged in a through hole opened in the screen.

[0029] Therefore, by hiding the camera module and the first infrared light source under the screen, there is no need to open a hole in the screen, and positions for the camera module and the first infrared light source are reserved, so that the electronic device can achieve a higher screen-to-body ratio.

[0030] In addition, since the camera module and the first infrared light source are set under the screen, when the first infrared light source and the second infrared light source are needed for lighting, the first infrared light source and the second infrared light source are set to work with a stronger luminous intensity, thereby ensuring that the obtained infrared image can include the effective information required for eye tracking.

[0031] According to the first aspect, or any implementation of the first aspect above, a through hole is provided on the screen of the electronic device, and a first infrared light source and a camera module are arranged in the through hole; when the electronic device is in the first business scenario, the first infrared light source is controlled to emit a first infrared light toward the target object, and the second infrared light source is controlled to emit a second infrared light toward the target object, including: when the electronic device is in the first business scenario, the first infrared light source is controlled to emit the first infrared light toward the target object with a second intensity, and the second infrared light source is controlled to emit the second infrared light toward the target object with a second intensity; wherein the second intensity is lower than the first intensity, and the first intensity is the luminous intensity of the first infrared light source and the second infrared light source when the first infrared light source and the camera module are arranged under the screen.

[0032] Therefore, by opening a through hole on the screen of the electronic device and setting the camera module and the first infrared light source in the through hole, so that the first infrared light source and the second infrared light source operate at a luminous intensity lower than the first intensity, it can be ensured that the obtained infrared image can include effective information required for eye tracking, thereby reducing the hardware power consumption of the electronic device.

[0033] According to the first aspect, or any implementation of the first aspect, there is at least one first infrared light source.

[0034] In this way, electronic devices can operate in high-precision eye tracking scenarios while taking into account implementation costs, and the number of openings on the screen of the electronic device can be reduced as much as possible, thereby reducing hardware power consumption.

[0035] According to the first aspect, or any implementation of the first aspect above, the distance between each first infrared light source and the second infrared light source is greater than or equal to 1 cm, and the distance between any two first infrared light sources is greater than or equal to 1 cm.

[0036] This ensures that the infrared image can include the effective information required for eye tracking, making the eye tracking results more accurate.

[0037] In a second aspect, the present application provides an electronic device. The electronic device includes: a memory, a processor, a first infrared light source, and a camera module, the camera module including a transmitter and a receiver, and the transmitter including a second infrared light source; the processor is coupled to the memory, the first infrared light source, and the camera module, respectively; the memory stores program instructions, which, when executed by the processor, cause the electronic device to execute instructions of the method of the first aspect or any possible implementation of the first aspect.

[0038] According to the second aspect, the camera module is a Time of Flight (TOF) module or a structured light module.

[0039] According to the second aspect, or any implementation of the second aspect above, the first infrared light source and the camera module are arranged under the screen of the electronic device.

[0040] According to the second aspect, or any implementation of the second aspect above, a through hole is provided on the screen of the electronic device, and the first infrared light source and the camera module are arranged in the through hole.

[0041] According to the second aspect, or any implementation of the second aspect, there is at least one first infrared light source.

[0042] According to the second aspect, or any implementation of the second aspect above, the distance between each first infrared light source and the second infrared light source is greater than or equal to 1 cm, and the distance between any two first infrared light sources is greater than or equal to 1 cm.

[0043] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0044] In a third aspect, the present application provides a computer-readable storage medium comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute instructions of the method in the first aspect or any possible implementation of the first aspect.

[0045] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0046] In a fourth aspect, the present application provides a computer program comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0047] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1A is a schematic diagram illustrating the structure of a remote eye tracking device;

[0049] FIG1B is a schematic diagram illustrating an exemplary bright pupil tracking effect;

[0050] FIG1C is a schematic diagram illustrating an exemplary dark pupil tracking effect;

[0051] FIG1D is a schematic diagram illustrating an exemplary scenario of eye tracking based on the remote eye tracking device shown in FIG1A ;

[0052] FIG2 is a schematic diagram illustrating a hardware structure of an electronic device;

[0053] FIG3A is a schematic diagram illustrating the relationship between a camera module and a first infrared light source in an electronic device;

[0054] FIG3B is a schematic diagram showing exemplary placement of a camera module and a first infrared light source;

[0055] FIG4 is a schematic diagram illustrating a software structure of an electronic device;

[0056] 5A and 5B are schematic diagrams of user interfaces for exemplarily activating the eye tracking function from a setting portal;

[0057] FIG5C is a diagram illustrating an exemplary user interface for enabling the eye tracking function from the pull-down notification bar;

[0058] FIG6 is a flow chart illustrating an exemplary embodiment of an eye tracking method provided by an embodiment of the present application;

[0059] 7A and 7B are schematic diagrams of user interfaces for exemplarily illustrating high-precision eye tracking scenarios;

[0060] FIG7C is a diagram illustrating a user interface of a low-precision eye tracking scenario. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0063] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0064] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0065] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0066] Currently, manufacturers of electronic devices such as mobile phones, tablets, personal computers (PCs), and smart screens are gradually integrating eye tracking functions into these electronic devices, enabling users to better interact with electronic devices, improving user experience, and thus seizing more user markets.

[0067] For example, in some possible implementations, it is possible to build a model based on apparent AI eye tracking technology (hereinafter described as: apparent AI method) by utilizing facial images containing eye images in various scenarios collected by a big data platform, and then analyze and process the images containing eyes taken by the camera module based on the constructed model to achieve eye tracking.

[0068] It should be noted that when implementing eye tracking based on the apparent AI method, it is only necessary to use the camera module in the electronic device. The hardware is simple and the power consumption is low.

[0069] However, eye tracking based on the appearance AI method has low accuracy and cannot meet the requirements of high-precision eye tracking scenarios. Therefore, in other possible implementation methods, eye tracking can be achieved with the help of eye tracking equipment.

[0070] Currently, commonly used eye tracking devices include remote measurement (screen type) and head-mounted types.

[0071] Among them, head-mounted eye tracking devices, such as the common eye trackers with glasses, usually integrate a series of hardware, such as multiple infrared lighting sources and one (or more) infrared cameras, into the glasses.

[0072] Among them, remote eye tracking devices are usually long strip-shaped structures. The hardware structure of a remote eye tracking device can be shown in Figure 1A. Referring to Figure 1A, for example, a remote eye tracking device can include two sets of dark pupil infrared light sources, one (or more) sets of bright pupil infrared light sources, two eye movement sensors (i.e., image sensors, which can also be understood as receivers in camera modules), and three computing chips.

[0073] It should be noted that for the computing chip in the remote eye tracking device, the eye tracking algorithm it adopts can specifically be the pupil corneal reflection method (hereinafter referred to as: PCCR method).

[0074] Furthermore, the bright pupil infrared light source referred to in this embodiment can be understood as an infrared light source aligned with the visual axis, as shown in Figure 1B . Because this infrared light source is aligned with the visual axis, when the infrared light emitted by this infrared light source strikes the user's eye, the retina reflects the light (as shown in Figure 1B , the infrared light emitted by the infrared light source and the reflected infrared light are aligned), resulting in the pupil appearing bright while the iris is relatively dark, creating a bright pupil tracking effect, as shown in Figure 1B . Therefore, the infrared light source located in the middle area of ​​Figure 1A , i.e., close to the eye tracking sensor, is referred to as the bright pupil infrared light source.

[0075] Furthermore, the dark-pupil infrared light source referred to in this embodiment can be understood as an infrared light source that is not aligned with the visual axis, as shown in Figure 1C. Because this infrared light source is not aligned with the visual axis, when the infrared light emitted by this source strikes the user's eye, the pupil appears dark, while the iris is relatively bright, creating a dark-pupil tracking effect, as shown in Figure 1C. Therefore, the infrared light sources at the two ends of Figure 1A, i.e., those farther away from the eye tracking sensor, are referred to as dark-pupil infrared light sources.

[0076] For the use of the long strip-shaped remote eye tracking device shown in Figure 1A, it is usually necessary to place (externally) it above or below the electronic device, so that eye tracking can be achieved with the help of the external remote eye tracking device while the user is using the electronic device.

[0077] The use of the long, strip-shaped remote eye tracking device shown in FIG1A can be illustrated in FIG1D . Referring to FIG1D , for example, using a laptop computer as an example, in some possible implementations, the remote eye tracking device can be placed below the laptop screen. Thus, when the user focuses on a fixation point on the laptop screen, the dark-pupil infrared light source and bright-pupil infrared light source in the remote eye tracking device illuminate, enabling the eye sensor (infrared camera) in the remote eye tracking device to capture eye-related information and generate an infrared image. Accordingly, the computing chip in the remote eye tracking device can use the PCCR method to analyze and process the infrared image captured by the eye sensor, thereby accurately estimating the eye's position in space and the fixation point. Finally, the computing chip calculates details such as eye position and fixation point using a 3D running model, and can feed the eye tracking results back to the electronic device, allowing the electronic device to respond to instructions corresponding to the eye tracking results, achieving human-computer interaction.

[0078] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0079] Since both telemetry-based and head-mounted eye tracking devices analyze and process captured infrared images based on the PCCR method, they can effectively implement eye tracking. However, they are generally expensive and difficult to carry. Furthermore, due to the multiple infrared light sources, the hardware power consumption is relatively high.

[0080] In addition, if telemetry devices are integrated into electronic devices such as mobile phones, tablets, PCs, and smart screens, it will not only increase hardware power consumption, but also increase the number of openings on the screen, affecting the screen-to-body ratio.

[0081] In view of this, an embodiment of the present application provides an eye tracking solution, which aims to make use of the camera module that can emit infrared light that is originally integrated in the electronic device, and then add at least one additional infrared light source. It can control the operation of different infrared light sources according to the actual business scenarios of the electronic device, and then adopt different eye tracking methods, such as the PCCR method or the apparent AI method, thereby taking into account both hardware power consumption and achieving eye tracking with different degrees of accuracy.

[0082] The following is a detailed description of the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems with specific embodiments.

[0083] Before describing in detail the eye tracking solution provided in the embodiments of the present application, the hardware structure of the electronic device applicable to this case is first described.

[0084] Among them, electronic devices can be, for example, mobile phones, tablet computers, smart wearable devices, PCs, smart large screens, etc., which are not listed here one by one and are not limited in this application. The following describes the solution by taking the mobile phone as an example.

[0085] 2 , the mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, a first infrared light source 196, and the like.

[0086] Among them, the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor (Modem), a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc., which are not listed one by one here and this application does not limit this.

[0087] Specifically, in the eye tracking solution provided in the embodiment of the present application, the image of the target object (including the user's eyes) captured by the camera 193 can be transmitted to the processing unit in the processor 110, and the processing unit analyzes and processes the obtained image according to the corresponding method, such as the PCCR method or the apparent AI method, to achieve eye tracking.

[0088] Among them, according to the PCCR method or the apparent AI method, the obtained image is analyzed and processed, and then the processing unit for realizing eye tracking can be, for example, any one or more of AP, GPU, ISP, and DSP, and this application does not impose any restrictions on this.

[0089] This embodiment uses the ISP processing data fed back by camera 193 as an example. In an eye tracking scenario, when infrared light strikes a target object, such as the infrared reflected light from a user's eye, it is transmitted through the lens to the camera's photosensitive element. The optical signal is converted into an electrical signal, which is then transmitted to the ISP for processing and converted into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some implementations, the ISP can be located within camera 193.

[0090] It should be noted that the camera photosensitive element mentioned in this embodiment can be regarded as the eye movement sensor (image sensor) mentioned in Figure 1A.

[0091] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0092] In addition, the processor 110 may further include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., which are not listed here one by one and are not limited in this application.

[0093] In addition, processor 110 may also include a memory for storing instructions and data. In some implementations, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor 110 latency, and thus improves system efficiency.

[0094] Continuing with FIG2 , the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0095] Continuing to refer to Figure 2, the internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, and the eye tracking function mentioned in the embodiment of the present application, etc.). The data storage area can store data created during the use of the mobile phone 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0096] 2 , the charging management module 140 is configured to receive charging input from a charger. While charging the battery 142 , the charging management module 140 can also power the electronic device through the power management module 141 .

[0097] Continuing with Figure 2 , the power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the first infrared light source, the wireless communication module 160, and the like. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance), among others.

[0098] 2 , the wireless communication function of the mobile phone 100 can be implemented through the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , the modem processor and the baseband processor.

[0099] 2 , the mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G for the mobile phone 100. The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. for the mobile phone 100.

[0100] 2 , the audio module 170 may include a speaker 170A, a receiver 170B, a microphone 170C, an earphone jack 170D, etc. For example, the mobile phone 100 may implement audio functions, such as audio recording and video recording, through the application processor and the speaker 170A, the receiver 170B, the microphone 170C, the earphone jack 170D, etc. in the audio module 170.

[0101] Continuing to refer to Figure 2, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., which are not listed here one by one and are not limited in this application.

[0102] Continuing with FIG2 , buttons 190 include a power button, a volume button, and the like. Buttons 190 may be mechanical buttons or touch buttons. Mobile phone 100 may receive key input and generate signal input related to user settings and function control of mobile phone 100. Motor 191 may generate vibration prompts. Motor 191 may be used for incoming call vibration prompts or for touch vibration feedback. Indicator 192 may be an indicator light that may be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0103] Continuing with FIG2 , the display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. In some implementations, the mobile phone 100 may include one or N display screens 194 , where N is a positive integer greater than 1. The mobile phone 100 may implement display functions using a GPU, the display screen 194 , and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0104] Continuing with FIG2 , the camera 193 is used to capture still images or videos. The mobile phone 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, and the application processor. In some implementations, the mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0105] In addition, it should be noted that, specifically in the eye tracking solution provided in the embodiment of the present application, the camera of the electronic device is a camera with a built-in infrared light source.

[0106] For example, in one possible implementation, the camera with its own infrared light source may be, for example, a TOF module. In other possible implementations, the camera module with its own infrared light source may be, for example, a structured light module.

[0107] Thus, by operating the camera 193 with its own infrared light source and the additional first infrared light source 196 together, it is possible to implement eye tracking using the PCCR method. By operating the camera 193 with its own infrared light source alone, it is possible to implement eye tracking using the apparent AI method. In this way, according to different eye tracking scenarios, it is possible to reasonably control the first infrared light source 196 and the infrared light source in the camera 193 (subsequently described as the second infrared light source) to operate together, or to control the second infrared light source alone, thereby taking into account both the accuracy requirements of eye tracking and the hardware power consumption requirements.

[0108] The positional relationship between the camera 193 and the first infrared light source can be shown as (1) in Figure 3A. Referring to (1) in Figure 3A, for example, when the camera module is located in the upper left corner of the screen of the mobile phone 100, the first infrared light source can be located at different positions shown in (1) in Figure 3A.

[0109] It should be noted that, in order to ensure hardware power consumption, in some possible implementations, the first infrared light source may be one. In this case, the first infrared light source may be located at any position of the first infrared light source shown in (1) in FIG. 3A .

[0110] For example, in some other possible implementations, such as a mobile phone 100 with a higher requirement for accuracy, multiple first infrared light sources may be additionally provided. For example, a first infrared light source may be provided at each of the four positions shown in (1) in FIG3A .

[0111] It should be understood that the above description is only an example listed for a better understanding of the technical solution of this embodiment and is not intended to be the sole limitation of this embodiment. In actual applications, the camera module and the first infrared light source can be set according to business needs. In order to ensure the eye tracking effect, any two first infrared light sources can be set, and the distance between any one infrared light source and the second infrared light source can meet the set distance (for example, greater than or equal to 1 cm), thereby ensuring that the captured image information can fully include the feature information required for eye tracking.

[0112] Continuing to refer to (1) in FIG3A , for example, a camera (camera module) with its own infrared light source may include a transmitter and a receiver. The hardware structure of this type of camera module may be shown in (2) in FIG3A .

[0113] Referring to (2) in FIG3A , for example, the transmitter portion of the camera module may include an infrared light source, namely the second infrared light source mentioned above, and a drive control unit for controlling the operation of the second infrared light source. The receiver portion of the camera module may be considered as an infrared camera, or an eye movement sensor, or an image sensor.

[0114] It should be understood that the above description is merely an example for better understanding the technical solution of this embodiment and is not intended to be the sole limitation on this embodiment. In actual applications, the camera module may include more components than those shown in (2) in FIG3A , and this application does not impose any limitation on this.

[0115] Continuing to refer to (1) in FIG3A , illustratively, the camera module and the first infrared light source can be arranged in a through hole opened on the screen of the mobile phone 100. For this arrangement, in one possible implementation, a through hole of a relatively long length can be opened on the screen, and the camera module and the first infrared light source are both arranged in the same through hole.

[0116] Understandably, since the distance between any two first infrared light sources, and between any one infrared light source and the second infrared light source, needs to meet a set distance, such as not less than 1 cm, the through hole opened on the screen will occupy a large area, affecting the screen-to-body ratio of the mobile phone 100.

[0117] For example, in another possible implementation, multiple through-holes can be opened on the screen, and the camera module and the first infrared light source can be placed in different through-holes. Although this method can reduce the size of each through-hole, it increases the number of holes in the screen. Therefore, it also affects the screen-to-body ratio of the mobile phone 100.

[0118] In view of this, in one possible implementation, the camera module and the first infrared light source can be placed under the screen of the mobile phone 100, that is, the screen does not have a through hole, as shown in (1) in Figure 3B. In this way, the number of screen holes can be reduced and the screen ratio can be increased.

[0119] In addition, it should be noted that in some possible implementations, one of the camera module and the first infrared light source can be placed under the screen, and the other can be placed in the through-hole. This can also reduce the number of screen openings and increase the screen-to-body ratio.

[0120] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0121] Referring to (2) in FIG. 3B , for example, in some other possible implementations, the camera module can be normally set in the through hole opened in the screen, and the additional first infrared light source can be set on the non-AA of the screen or on an accessory of the mobile phone 100 .

[0122] Generally speaking, the screen of an electronic device can be divided into the View Area (VA), the Active Area (AA, also known as the effective area), and the Black Matrix (BM). The VA is the visible portion of the screen, the AA is the area that can be programmed to display, and the BM is located at the outermost edge of the VA (commonly known as the black border of the screen).

[0123] In addition, it is also understandable that the screen-to-body ratio is usually for AA. Therefore, in some possible implementations, in order to improve the screen-to-body ratio, the first infrared light source can be set on non-AA, that is, VA or BM, as shown in (2) in Figure 3B.

[0124] Furthermore, it should be noted that, for scenes where the first and second infrared light sources are positioned below a screen, the infrared light emitted by the first and second infrared light sources will be affected by the screen's material and thickness, affecting transmittance, reflection, and refraction. Therefore, to ensure that the resulting infrared image fully includes the feature information required for eye tracking, the first and second infrared light sources can be configured to operate at a higher luminous intensity (hereinafter referred to as the first intensity), thereby ensuring that the resulting infrared image includes valid information required for eye tracking.

[0125] In addition, it should be noted that for the scenario where the first infrared light source and the second infrared light source are set with a through hole opened on the screen, or on non-AA, or on accessories of the mobile phone 100, the first infrared light source and the second infrared light source can be set to operate with a luminous intensity lower than the first intensity (subsequently represented by the second intensity), thereby ensuring that the obtained infrared image can include the effective information required for eye tracking and reducing hardware power consumption.

[0126] It should be understood that the above description is merely an example for a better understanding of the technical solution of this embodiment and is not intended to be the sole limitation of this embodiment. In actual applications, the additional infrared light source, such as the first infrared light source, can be placed under the screen or on the screen at the same time as the camera module, or one can be placed on the screen and the other under the screen. This application does not impose any restrictions on this. However, it should be understood that in order to ensure eye tracking, the luminous intensity of the infrared light source placed under the screen must be higher than that of the infrared light source placed on the screen.

[0127] This concludes the introduction to the hardware structure of the mobile phone 100. It should be understood that the mobile phone 100 shown in FIG2 is merely an example. In a specific implementation, the mobile phone 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in FIG2 may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0128] 2 , the software structure of the mobile phone 100 is described below. Before describing the software structure of the mobile phone 100 , the architecture that the software system of the mobile phone 100 can adopt is first described.

[0129] Specifically, in actual applications, the software system of the mobile phone 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.

[0130] In addition, it is understandable that the software systems used by current mainstream electronic devices include but are not limited to Windows systems, Android systems, and iOS systems. For ease of explanation, the embodiment of the present application takes the layered architecture Android system as an example to exemplify the software structure of the mobile phone 100.

[0131] In addition, the eye tracking solution provided in the subsequent embodiments of the present application is also applicable to other systems in specific implementations.

[0132] See FIG4 , which is a software structure block diagram of the mobile phone 100 according to an embodiment of the present application.

[0133] As shown in Figure 4, the layered architecture of mobile phone 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some implementations, the Android system is divided into five layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer (HAL layer), and the kernel layer.

[0134] The application layer may include a series of application packages. As shown in FIG4 , the application package may include applications such as camera, settings, maps, WLAN, Bluetooth, gallery, music, etc., which are not listed here one by one and are not limited in this application.

[0135] It should be noted that in order for the mobile phone 100 to achieve eye tracking and thus perform human-computer interaction with the user, the user needs to enable the eye tracking function in advance, so that when the mobile phone 100 is in an eye tracking scenario, such as when the screen is on, or when a notification message is received on the lock screen interface, there is no need for manual operation by the user. The first infrared light source and the second infrared light source are directly controlled to work together according to the current business scenario, or the second infrared light source is controlled to work alone. Then, the eye tracking module analyzes and processes the collected infrared image based on the method corresponding to the current business scenario, such as the PCCR method or the apparent AI method, so that eye tracking can be performed.

[0136] For example, in some possible implementations, the entry for enabling the eye tracking function may be set in a settings application in the application layer, for example.

[0137] For example, in order to enable the eye tracking function through the settings application, the user can click the icon of the settings application in the interface 10a shown in FIG5A (1), so that the mobile phone 100 responds to the operation behavior, starts the settings application, and then switches the user interface from the interface 10a to the interface 10b shown in FIG5A (2).

[0138] For example, in some possible implementations, the eye tracking option can be provided directly in the interface 10b, such as option 10b-1 shown in (2) of Figure 5B. In other possible implementations, the eye tracking option can also be provided in an interface corresponding to other functional options of the interface 10b, such as in a smart assistant.

[0139] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0140] For ease of explanation, this embodiment takes the example of interface 10b directly displaying the eye tracking option 10b-1. For example, when the user clicks on the eye tracking option 10b-1, the mobile phone 100 responds to the operation by loading the interface for enabling the eye tracking function, and the user interface switches from interface 10b to interface 10c shown in (1) of FIG5B.

[0141] Referring to (1) in FIG5B , for example, by default, the eye tracking function may be in an off state. In this case, the control for turning on the eye tracking function may be in the style of the control 10c-1 shown in (1) in FIG5B .

[0142] For example, when the user clicks on the control 10c-1, the mobile phone 100 will turn on the eye tracking function in response to the operation. At this time, the control 10c-1 will switch to the style of the control 10c-1' in the interface 10c' shown in Figure 5B (2).

[0143] As can be seen from the above description, the second infrared light source in the camera module can operate independently. In this case, the eye tracking results are relatively less accurate, but the hardware power consumption is low. When the additional first infrared light source and the second infrared light source in the camera module operate in conjunction, the eye tracking results are relatively more accurate, but the hardware power consumption is also higher. Therefore, to better meet user needs, after the eye tracking function is enabled, the interface 10c' also displays a mode for the user to select.

[0144] Continuing to refer to (2) in FIG5B , exemplary eye tracking modes for user selection may include, for example, a first mode, a second mode, and a third mode.

[0145] When the user selects the first mode, the mobile phone 100 can automatically control the first mode of operation of the first infrared light source and the second infrared light source according to the current business scenario. For example, in the current business scenario (high-precision eye tracking scenario), the first infrared light source and the second infrared light source are controlled to work together. In the current business scenario (low-precision eye tracking scenario), only the second infrared light source is controlled to work.

[0146] Among them, when the user selects the second mode, the mobile phone 100 does not distinguish the current business scenario, that is, whether it is a high-precision eye tracking scenario or a low-precision eye tracking scenario, it controls the first infrared light source and the second infrared light source to work together.

[0147] Among them, when the user selects the third mode, the mobile phone 100 does not distinguish the current business scenario, that is, whether it is a high-precision eye tracking scenario or a low-precision eye tracking scenario, it only controls the second infrared light source to work alone.

[0148] Specifically in the technical solution provided in the embodiment of the present application, the eye tracking mode selected by the user is taken as the first mode as an example.

[0149] In addition, it should be noted that in some possible implementations, it can be set that after the eye tracking function is turned on, the default selected eye tracking mode is the first mode.

[0150] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0151] For example, in some other possible implementations, the entry for opening the eye tracking function may be set in a pull-down notification bar, for example.

[0152] For example, in order to enable the eye tracking function by pulling down the notification bar, the user can make a sliding gesture on the screen along the direction of the arrow in the interface 10a shown in (1) in FIG5C , so that the mobile phone 100 responds to the operation, replaces the pull-down notification bar, and then switches the user interface from interface 10a to interface 10d shown in (2) in FIG5C .

[0153] Referring to (2) in FIG. 5C , illustratively, after the user clicks the eye tracking function option shown in the interface 10d, the mobile phone 100 responds to the operation behavior, turns on the eye tracking function, and selects the default eye tracking mode, such as the first mode, or the eye tracking mode previously selected by the user.

[0154] For example, in some other possible implementations, after the user clicks the eye tracking function option shown in interface 10d, the mobile phone 100 can jump to interface 10c shown in (1) of Figure 5B in response to the operation behavior. Then, the eye tracking function is turned on through the control 10c-1 shown in interface 10c, and the eye tracking mode displayed in interface 10c' shown in (2) of Figure 5B is automatically selected.

[0155] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0156] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. In some implementations, these programming interfaces and programming frameworks can be described as functions. As shown in Figure 4, the application framework layer may include functions such as a camera service, an eye tracking service, a window manager, a content provider, a resource manager, and a notification manager. These functions are not listed here and are not limited in this application.

[0157] The Android Runtime includes the core library and virtual machine. The Android Runtime is responsible for scheduling and management of the Android system.

[0158] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0159] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0160] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL).

[0161] The HAL layer is used to encapsulate the hardware driver and provide the same interface to the upper framework. Referring to Figure 4, the HAL layer may include a camera hardware abstraction layer, an audio hardware abstraction layer, etc., which are not listed here one by one and are not limited in this application.

[0162] Among them, the kernel layer is the layer between hardware and software, including various hardware drivers, such as sensor driver, display driver, microphone driver, WLAN driver, camera driver, etc.

[0163] Based on the software structure and hardware layer shown in Figure 4, in the technical solution provided in the embodiment of the present application, the camera service can be used to enable the eye tracking function on the mobile phone 100, and when it is currently in the eye tracking scenario, according to the business scenario, the camera hardware abstraction layer in the HAL layer is notified to trigger the camera driver in the kernel layer, and then the camera driver drives the camera in the hardware layer, such as the second infrared light source in the TOF module or the structured light module to illuminate alone, or drive the additional first infrared light source and the second yellow external light source in the camera module to cooperate in lighting, thereby obtaining an infrared image of the target object, so that the eye tracking service can realize analysis and processing of the infrared image according to the integrated PCCR method or the apparent AI method, thereby realizing eye tracking.

[0164] This concludes the introduction to the software structure of the mobile phone 100. It should be understood that the layers in the software structure shown in FIG4 and the components contained in each layer do not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not impose any limitations thereon.

[0165] To better understand the eye tracking method provided in the embodiments of this application, this application uses the example of a user enabling the eye tracking function and selecting the first eye tracking mode as described in the above embodiment. When this eye tracking method is applied to an electronic device with the hardware structure shown in FIG2 and the software structure shown in FIG3, its processing flow can be shown in FIG6.

[0166] 6 , which exemplifies a flow chart of an eye tracking method. In this embodiment, the eye tracking method may include:

[0167] 101. The electronic device determines the current business scenario.

[0168] Among them, electronic devices, such as a camera module with a built-in infrared light source, have added at least one infrared light source device. In actual applications, it can be a mobile phone, tablet computer, PC, smart large screen, etc., which are not listed here one by one and are not limited in this application.

[0169] It should be noted that, in order to minimize the hardware power consumption of the electronic device, in actual applications, a trigger condition for step 101 can be set. Specifically, after the user turns on the eye tracking function using the method shown in Figures 5A and 5B, or the method shown in Figure 5C, in one possible implementation, step 101 can be triggered when the screen of the electronic device is in the bright screen state.

[0170] Exemplarily, in another possible implementation, the electronic device may be configured to trigger execution of step 101 when receiving a notification message, for example, when a message management window pops up with a received message.

[0171] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0172] Exemplarily, regarding the way in which the electronic device determines the current business scenario in step 101, in one possible implementation, the current business scenario can be determined, for example, by obtaining the user interface currently displayed on the screen and then performing image recognition on the content of the user interface.

[0173] For example, in another possible implementation, the current business scenario can be determined based on the properties of the currently running foreground application by obtaining application identification information of the currently running foreground application.

[0174] Specifically in the embodiment of the present application, the business scenarios in which the electronic device is located can be divided into a first business scenario (high-precision eye tracking scenario) and a second business scenario (low-precision eye tracking scenario).

[0175] The first business scenario, which requires high eye tracking accuracy but not high power consumption (no need for long-term eye tracking), includes scenarios such as identifying the app icon being looked at and eye focusing. This scenario requires an infrared camera with multiple infrared light sources to capture infrared images, and then implement eye tracking based on the PCCR method.

[0176] For example, in the first business scenario, for example, in FIG7A , a user gazes at icon 10a-2 of the Gallery application in interface 10a and desires to open the Gallery application. In this scenario, eye tracking based on the PCCR method can accurately determine that the user is gazing at icon 10a-2. Based on the eye tracking results, the Gallery application is launched, causing the user interface to switch from interface 10a to interface 10e shown in FIG7B .

[0177] For example, in the first business scenario, for example, a user gazes at option 10e-1 in interface 10e in FIG7B and desires to open an interface displaying all photos taken with the camera, such as interface 10f in FIG7C . In this scenario, eye tracking based on the PCCR method can accurately determine that the user is gazing at option 10e-1. Then, in response to the eye tracking results, the interface displaying photos taken with the camera is loaded, causing the user interface to switch from interface 10e to interface 10f shown in FIG7C .

[0178] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0179] The second business scenario, which requires low eye tracking accuracy but high power consumption (requiring prolonged use of the eye tracking function), includes eye-controlled page turning / swiping when reading e-books or watching short videos. This scenario does not require additional infrared light sources and only utilizes the infrared light source built into the emitter of the camera module for illumination. Subsequent analysis and processing of the infrared image based on the apparent AI method enables eye tracking.

[0180] For example, in the second business scenario, for example, in FIG7C , a user gazes at a sliding control 10f-1 in interface 10f and desires to slide the photos displayed in interface 10f. In this scenario, eye tracking can be implemented using an appearance AI method without increasing hardware power consumption, allowing the user to control the sliding control 10f-1 by sliding it up or down along the arrow to slide other photos in interface 10f.

[0181] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0182] 102 , control a first infrared light source to emit a first infrared light toward a target object, and control a second infrared light source to emit a second infrared light toward the target object.

[0183] Specifically, when the current business scenario is determined to be the first business scenario, in order to ensure the user experience and quickly and accurately complete the human-computer interaction actions expected by the user, the PCCR method with high eye tracking accuracy can be used to perform eye tracking.

[0184] As can be seen from the above embodiments, the PCCR method requires that the captured infrared image be generated based on at least two infrared light sources irradiating a target object, such as infrared reflected light from a user's eyes. Therefore, when the electronic device is in the first service scenario, the first infrared light source can be controlled to emit a first infrared light toward the target object, and the second infrared light source can be controlled to emit a second infrared light toward the target object.

[0185] It should be noted that in the first business scenario, the first infrared light source and the second infrared light source can be controlled to operate simultaneously (emit infrared light) through hard synchronization. This method can, for example, be achieved by connecting the first infrared light source and the second infrared light source to different pins of the same drive control unit, and then controlling the drive control unit to send the same drive signal to the pins connected to the two infrared light sources at the same time, thereby driving the first infrared light source and the second infrared light source to operate simultaneously.

[0186] For example, in another possible implementation, the first infrared light source and the second infrared light source can be controlled to operate simultaneously (emit infrared light) through soft synchronization. In this method, for example, instructions can be sent to the first infrared light source and the second infrared light source simultaneously during the same processing period, thereby driving the first infrared light source and the second infrared light source to operate simultaneously.

[0187] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0188] In addition, it should be noted that for users wearing glasses, the infrared light emitted by infrared light sources in different positions may cause the position of corneal reflection to be affected by the reflection of the lens, resulting in a highlighted area. Therefore, in the first business scenario, the first infrared light source and the second infrared light source can also operate alternately. By alternately driving the first infrared light source and the second infrared light source, an image containing complete information can be processed from two images.

[0189] Furthermore, it should be noted that, to ensure the user experience is not affected, the alternating period can be set to a time interval that is imperceptible to the user. This means that, to the user, the first and second infrared light sources are constantly emitting infrared light. This not only ensures that the infrared image of the user's eye wearing the glasses contains complete information, but also reduces the hardware power consumption of the electronic device through alternating operation.

[0190] 103. A receiver receives a first infrared reflected light beam caused by the target object reflecting the first infrared light, and a second infrared reflected light beam caused by the target object reflecting the second infrared light, to generate a first infrared image of the eye.

[0191] The first infrared reflected light received by the receiver is specifically the infrared reflected light reflected by the cornea and retina of the eye. The second infrared reflected light received by the receiver is specifically the infrared reflected light reflected by the cornea and retina of the eye.

[0192] It can be understood that the receiver is essentially an image sensor, so the receiver can generate a first infrared image according to the first infrared reflected light and the second infrared reflected light.

[0193] 104. The electronic device uses a pupil-corneal reflection method to determine the direction of eye movement based on a vector between a corneal spot formed on the cornea by a first infrared reflected light ray and a second infrared reflected light ray in a first infrared image and a center of the pupil.

[0194] It can be understood that the basic principle of the PCCR method to achieve eye tracking is based on the reflection generated by the cornea when the infrared light source illuminates the eye to achieve eye tracking. Specifically, if the infrared light source and the image acquisition device are fixed, the pupil position will change when the eyeball rotates, and the cornea reflects different infrared light. For example, the position of the corneal spot formed by the first infrared reflected light and the second infrared reflected light in this embodiment will not change. Therefore, it can be used as a reference point for pupil movement, and the movement of the eyeball can be estimated based on the relative position change between the center of the pupil and the corneal spot (corneal reflection point). Therefore, the electronic device adopts the pupil-corneal reflection method, and according to the vector between the corneal spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil, it can determine the direction of eye movement, that is, achieve eye tracking.

[0195] Furthermore, it should be understood that, in practical applications, the estimation of the sight direction can be achieved, for example, through a mapping model (such as a nonlinear polynomial model) or a geometric model.

[0196] For details on how to implement eye tracking based on the PCCR method, please refer to the relevant literature on the PCCR method. Detailed explanation is not provided here.

[0197] In addition, it should be noted that the camera module that has its own second infrared light source can generally provide 3D depth information in addition to providing infrared images. Therefore, in some possible implementations, the electronic device can first correct the user's head posture based on the 3D depth information provided by the camera module. Then, the electronic device uses the pupil corneal reflection method to calculate the vector between the corneal spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil. In this way, by fusing the 3D depth information provided by the camera module, the generalization of posture can be improved, making the eye tracking results based on the infrared image corrected with 3D depth information more accurate.

[0198] Therefore, for an electronic device with a camera module, and the camera module itself has an infrared light source, by adding one or more infrared light sources (first infrared light sources), when the electronic device of this structure turns on the eye tracking function, by determining the current business scenario of the electronic device, and then in the case of high-precision eye tracking, the infrared light source built into the camera module and the newly added infrared light source are controlled to emit infrared light toward the target object, and the receiver receives the infrared reflected light reflected by the target object to generate a first infrared image of the eye, so that the electronic device can analyze and process the first infrared image according to the PCCR, thereby realizing eye tracking. Since the camera module itself has an infrared light source, high-precision eye tracking can be achieved by adding at least one additional infrared light source, thereby ensuring the accuracy of eye tracking and reducing the number of screen openings of the electronic device.

[0199] 105 , control the second infrared light source to emit second infrared light toward the target object.

[0200] Specifically, when it is determined that the current business scenario is the second business scenario, in order to reduce the hardware power consumption of the electronic device, the apparent AI method can be used, and then eye tracking can be used.

[0201] As can be seen from the above embodiments, the apparent AI method does not require that the captured infrared image be generated based on at least two infrared light sources irradiating the target object, such as infrared light reflected from the user's eyes. Therefore, when the electronic device is in the second service scenario, to reduce hardware power consumption, only the second infrared light source can be controlled to operate, that is, to emit the second infrared light toward the target object.

[0202] 106. The receiver receives a third infrared reflected light of the second infrared light reflected by the target object to generate a second infrared image of the eye.

[0203] Specifically, in the second service scenario, since only the second infrared light source operates independently, the infrared light it emits is flood light. Therefore, the captured second infrared image includes the user's face. The second infrared image is generated by the receiver receiving the third infrared reflected light from the second infrared light reflected by the user's face, and then generating the second infrared image based on the third infrared reflected light.

[0204] 107. The electronic device uses apparent artificial intelligence eye tracking technology to determine the direction of eye movement based on the second infrared image.

[0205] It is understandable that the basic principle of the apparent AI method to achieve eye tracking is that the apparent AI method uses the coupling relationship between key points of the eye area such as the pupil center, the appearance of the eye image and facial image, and the position of the gaze point to perform eye tracking through a large amount of data training.

[0206] Therefore, when implementing step 107, the electronic device can first determine the key points of the eye region in the second infrared image. Then, the image region containing the key information is captured from the second infrared image to obtain an infrared image of the eye. Finally, using an appearance AI method, based on the appearance of the user's face and eyes in the second infrared image and the location of the eye's gaze point in the infrared image of the eye, the direction of eye movement can be determined, thereby achieving eye tracking.

[0207] For the specific processing details of eye tracking based on the apparent AI method, please refer to the relevant literature on the apparent AI method, which will not be explained in detail here.

[0208] Therefore, in low-precision eye tracking scenarios, there is no need to use an additional first infrared light source. Only the second infrared light source built into the emitter in the camera module is used for lighting. Then, the obtained infrared image is analyzed and processed based on the apparent AI eye tracking technology, which can not only achieve eye tracking but also ensure the hardware power consumption of the electronic device.

[0209] Furthermore, it's important to note that in practical applications, the infrared image resolution can be flexibly configured based on the eye tracking accuracy and hardware power consumption requirements of different business scenarios. Understandably, higher resolutions improve accuracy but also increase power consumption. Conversely, lowering the infrared image resolution can reduce hardware power consumption, but this also reduces eye tracking accuracy.

[0210] Through the eye tracking method provided in the embodiments of the present application, for electronic devices whose camera modules have their own infrared light source (the second infrared light source in the above embodiments), at least one additional infrared light source (the first infrared light source in the above embodiments) can be added. This allows the electronic device to reasonably control the first infrared light source and the second infrared light source to operate individually or collaboratively in actual business scenarios, thereby selecting different eye tracking methods and realizing layered and graded eye tracking processing, which can not only meet the user's needs for high-precision eye tracking, but also meet the user's needs for low power consumption of electronic devices, better meet the different usage needs of users, and enhance the user experience.

[0211] In addition, it is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0212] Furthermore, it should be noted that the eye tracking methods provided in the above embodiments, implemented by electronic devices in actual application scenarios, can also be executed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system may be coupled to a memory so that when the chip system is running, it calls a computer program stored in the memory to implement the steps performed by the above electronic device. The processor in the chip system may be an application processor or a processor other than an application processor.

[0213] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the eye tracking method in the above-mentioned embodiment.

[0214] In addition, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the eye tracking method in the above-mentioned embodiment.

[0215] In addition, an embodiment of the present application also provides a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the eye tracking method in the above-mentioned embodiment to control the receiving pin to receive signals, so as to control the transmitting pin to send signals.

[0216] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0217] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An eye tracking method, characterized in that: Applied to an electronic device, the electronic device comprises: a first infrared light source and a camera module, the camera module comprises a transmitter and a receiver, and the transmitter comprises a second infrared light source; The method comprises: The electronic device determines the current business scenario; When the electronic device is in a first business scenario, controlling the first infrared light source to emit a first infrared light to a target object, and controlling the second infrared light source to emit a second infrared light to the target object, the target object includes an eye of a user, and the first business scenario is a high-precision eye tracking scenario; The receiver receives a first infrared reflected light beam reflected by the target object from the first infrared light, and a second infrared reflected light beam reflected by the target object from the second infrared light, to generate a first infrared image of the eye; The electronic device performs eye tracking according to the first infrared image.

2. The method according to claim 1, characterized in that The receiver receives a first infrared reflected light beam reflected by the target object from the first infrared light, and a second infrared reflected light beam reflected by the target object from the second infrared light, and generates a first infrared image of the eye, including: The receiver receives the first infrared reflected light beam formed by the cornea and retina of the eye reflecting the first infrared light, and the second infrared reflected light beam formed by the cornea and the retina reflecting the second infrared light; wherein the position of the corneal light spot formed by the cornea reflecting the first infrared reflected light beam and the second infrared reflected light beam remains fixed, and the light beam reflected on the retina identifies the pupil direction of the eye; A first infrared image of the eye is generated according to the first infrared reflected light and the second infrared reflected light.

3. The method according to claim 2, characterized in that The electronic device performs eye tracking according to the first infrared image, including: The electronic device adopts a pupil-corneal reflection method to determine the movement direction of the eye according to a vector between a corneal spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil.

4. The method according to claim 3, characterized in that The electronic device uses a pupil corneal reflection method to determine the movement direction of the eye according to a vector between a corneal spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil, including: The electronic device corrects the head posture of the user according to the 3D depth information provided by the camera module; The electronic device adopts a pupil-corneal reflection method to determine the movement direction of the eye according to a vector between a corneal spot formed on the cornea by the first infrared reflected light and the second infrared reflected light in the first infrared image and the center of the pupil.

5. The method according to claim 1, characterized in that The method further comprises: When the electronic device is in a second business scenario, controlling the second infrared light source to emit the second infrared light to the target object, the second business scenario being a low-precision eye tracking scenario; The receiver receives a third infrared reflected light beam from the target object reflecting the second infrared light, and generates a second infrared image of the eye; The electronic device performs eye tracking according to the second infrared image.

6. The method according to claim 5, characterized in that The receiver receives a third infrared reflected light beam from the target object reflecting the second infrared light, and generates a second infrared image of the eye, including: The receiver receives a third infrared reflected light beam which is the second infrared light beam reflected by the face of the user; The second infrared image is generated according to the third infrared reflected light.

7. The method according to claim 6, characterized in that The electronic device performs eye tracking according to the second infrared image, including: The electronic device determines key points of the area where the eye is located in the second infrared image; The electronic device intercepts the key image area from the second infrared image to obtain an infrared image of the eye; The electronic device uses apparent artificial intelligence eye tracking technology to determine the direction of movement of the eye based on the appearance of the user's face and eyes in the second infrared image and the position of the eye's gaze point in the eye infrared image.

8. The method according to any one of claims 1 to 7, characterized in that: The first infrared light source and the camera module are arranged under the screen of the electronic device; When the electronic device is in a first business scenario, controlling the first infrared light source to emit a first infrared light to a target object, and controlling the second infrared light source to emit a second infrared light to the target object, includes: When the electronic device is in the first service scenario, controlling the first infrared light source to emit the first infrared light to the target object at a first intensity, and controlling the second infrared light source to emit the second infrared light to the target object at the first intensity; Among them, the first intensity is higher than the second intensity, and the second intensity is the luminous intensity of the first infrared light source and the second infrared light source when the first infrared light source and the camera module are arranged in the through hole opened in the screen.

9. The method according to any one of claims 1 to 7, characterized in that: A through hole is provided on the screen of the electronic device, and the first infrared light source and the camera module are arranged in the through hole; When the electronic device is in a first business scenario, controlling the first infrared light source to emit a first infrared light to a target object, and controlling the second infrared light source to emit a second infrared light to the target object, includes: When the electronic device is in the first service scenario, controlling the first infrared light source to emit the first infrared light to the target object at a second intensity, and controlling the second infrared light source to emit the second infrared light to the target object at the second intensity; The second intensity is lower than the first intensity, and the first intensity is the luminous intensity of the first infrared light source and the second infrared light source when the first infrared light source and the camera module are arranged under the screen.

10. The method according to any one of claims 1 to 7, characterized in that: The number of the first infrared light source is at least one.

11. The method according to claim 10, characterized in that The distance between each of the first infrared light sources and the second infrared light sources is greater than or equal to 1 cm, and the distance between any two of the first infrared light sources is greater than or equal to 1 cm.

12. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, a first infrared light source and a camera module, the camera module comprises a transmitter and a receiver, and the transmitter comprises a second infrared light source; The processor is coupled to the memory, the first infrared light source, and the camera module respectively; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the eye tracking method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the eye tracking method as claimed in any one of claims 1 to 11.

Images