Eye tracking method, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application provide an eye movement tracking method, an electronic device and a computer readable storage medium. In the method, the electronic device first calibrates user eye movement data and head movement data. The electronic device can collect images of the user gazing at different positions on the display screen in the same position and the same head posture, and images of the user gazing at the same position on the display screen in different positions and / or different head postures. Then the electronic device can extract eye features and head posture features from the above images. When it is necessary to track the eye movement of the user, the electronic device can collect user images, and then determine the eye movement of the user based on the collected images and the above eye features and head posture features. Implementing the technical solutions provided by the embodiments of the present application can improve the accuracy and head posture robustness of eye movement tracking, and thus accurately predict the gazing position of the user in multiple head postures.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to eye-tracking methods, electronic devices, and computer-readable storage media. Background Technology

[0002] In recent years, eye tracking has become an important human-computer interaction technology. Electronic devices can track a user's eye movements through a camera to determine the user's gaze position. The electronic device can then provide feedback based on the user's gaze position. However, how to accurately track a user's eye movements and reduce misjudgments is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides an eye-tracking method, an electronic device, and a computer-readable storage medium, which can improve the accuracy of electronic devices in tracking human eye movements, thereby providing more accurate feedback based on the user's eye movements.

[0004] In a first aspect, embodiments of this application provide an eye-tracking method applied to an electronic device, the electronic device including a display screen and a camera, the method comprising: the electronic device displaying a first identifier at multiple locations on the display screen and capturing multiple first image frames via the camera, the multiple first image frames containing images of a user gazing at the first identifier at different locations on the display screen under the same head posture; the electronic device displaying a second identifier at one location on the display screen and capturing multiple second image frames, the multiple second image frames containing images of a user gazing at the second identifier under different head postures; the electronic device acquiring a third image frame via the camera, and then determining the user's first gaze position on the display screen based on the multiple first image frames, the multiple second image frames, and the third image frame.

[0005] In other words, electronic devices can capture images of a user gazing at multiple points in one head posture, as well as images of the user gazing at the same point in different head postures. These head postures include both head position and head pose. This fully considers the distortion caused by head movements to the eye images. By calibrating the user's eye movement data under different head postures, the electronic device can obtain a more comprehensive image of the user gazing at points on the display screen, effectively eliminating individual differences and reducing deviations in eye tracking caused by factors such as strabismus or varying head shapes. Based on these images, the electronic device can accurately predict the user's gaze position under various head postures, improving the accuracy of eye tracking and the robustness of head posture.

[0006] In conjunction with the first aspect, in some embodiments, before the electronic device sequentially displays the first identifier at multiple locations on the display screen, the method further includes: the electronic device receiving a first user operation, the first user operation being used to begin a first calibration of eye features and / or head posture features when the user views different locations on the display screen; before the electronic device acquires a third image frame via a camera, the method further includes: the electronic device indicating that the first calibration is complete.

[0007] In other words, electronic devices can provide users with an entry point to calibrate their head posture and eye features during eye movements. Users can then use this entry point to correct their eye and / or head posture features when viewing a display screen. The electronic device can then prompt the user upon completion of the calibration. This provides users with the option to choose whether to correct their eye and / or head posture features, improving the user experience. It also encourages users to cooperate with the electronic device in acquiring images, helping the device obtain images that better meet the requirements for feature extraction.

[0008] In conjunction with the first aspect, in some embodiments, before the electronic device captures multiple first image frames through the camera, the method further includes: the electronic device prompting the user to keep their head fixed and look at the first identifier displayed on the screen; before the electronic device captures multiple second image frames, the method further includes: the electronic device prompting the user to change their head position to look at the second identifier.

[0009] In other words, the electronic device can issue prompts to instruct the user to fix the head position to acquire the first image frame and change the head position to acquire the second image frame. In this way, the electronic device can obtain images that better meet the requirements of feature extraction and help the user complete the acquisition of the above images more quickly, thereby improving the user experience.

[0010] In conjunction with the first aspect, in some embodiments, after the electronic device determines the user's first gaze position on the display screen, the method further includes: the electronic device unfolding and displaying a message at the first gaze position.

[0011] In other words, electronic devices can unfold messages based on the user's eye gaze position. This way, users do not need to operate with their fingers; they only need to move their eyes and look at the message to quickly unfold it, reducing the user's operational burden and improving the user experience.

[0012] In conjunction with the first aspect, in some embodiments, the electronic device determines the user's first gaze position on the display screen based on multiple first image frames, multiple second image frames, and a third image frame, including: the electronic device determines the gaze point deviation based on a first eye movement deviation, a first head posture spatial deviation, and / or a second head posture spatial deviation, wherein the first eye movement deviation is determined based on the first image frame, the first head posture spatial deviation is determined based on a second image frame that does not carry head depth information, and the second head posture spatial deviation is determined based on a second image frame that carries head depth information; the electronic device determines the first gaze position based on the gaze point deviation and a first calibration position, wherein the first calibration position is the position where the electronic device displays a first identifier in the first reference image frame, and a second identifier is displayed at the first calibration position, and the first reference image frame is selected from multiple first image frames.

[0013] In other words, the electronic device can determine the first eye movement deviation based on the multiple first image frames acquired above, determine the first head posture spatial deviation based on the second image frames that do not carry head depth information, and determine the second head posture spatial deviation based on the second image frames that carry head depth information. The electronic device can then determine the user's gaze deviation relative to the first calibration position using these deviations, thereby determining the user's first gaze position. The first calibration position is either the position where the electronic device displays the second identifier or the position where the electronic device displays the first identifier in the first reference image frame. By determining the gaze deviation based on the first and second image frames, the electronic device can fully consider the impact of the user's head posture on the eye image when gazing at the display screen, thus more accurately determining the aforementioned gaze deviation.

[0014] In conjunction with the first aspect, in some embodiments, the method further includes: determining a first eye movement deviation amount based on a first image frame according to the following steps: the electronic device extracts image features from a first reference image frame to obtain a first eye feature, extracts image features from first image frames other than the first reference image frame to obtain multiple second eye features, and extracts image features from a third image frame to obtain a third eye feature; the electronic device fuses the first eye feature with the multiple second eye features to obtain multiple eye movement deviation features; and the electronic device obtains the first eye movement deviation amount based on the third eye feature and the multiple eye movement deviation features.

[0015] In other words, the electronic device can extract eye features separately, and then fuse the first eye features with the second eye features. This yields eye deviation features relative to the user's gaze at different points on the display screen under the same head posture, as well as the user's gaze at the first calibrated position on the display screen. The electronic device can then determine the amount of eye movement deviation in the third image frame to be predicted based on the third eye features and the aforementioned multiple eye deviation features.

[0016] In conjunction with the first aspect, in some embodiments, the method further includes: determining a first head pose spatial deviation based on multiple second image frames that do not carry head depth information according to the following steps: the electronic device extracts image features from the second image frames that do not carry head depth information to obtain multiple fourth eye features; the electronic device fuses the first eye features with the multiple fourth eye features to obtain multiple first head pose deviation features; the electronic device obtains the first head pose spatial deviation based on the third eye features and the multiple head pose deviation features.

[0017] The second image frame, which does not carry head depth information, can be a two-dimensional image captured by a camera. It contains multiple two-dimensional images of the user looking at the first calibrated position on the display screen under different head postures. The electronic device can then determine the first head posture deviation feature based on the second image frame. In this way, the electronic device can determine the user's head posture deviation in the third image frame from the perspective of two-dimensional images, thereby determining the deviation position of the user's gaze point in multiple dimensions and improving accuracy.

[0018] In conjunction with the first aspect, in some embodiments, the multiple second image frames further include a second reference image frame carrying head depth information. The method further includes: determining a second head pose spatial deviation based on the multiple second image frames carrying head depth information according to the following steps: the electronic device extracts image features from the second reference image frame to obtain a first head pose feature; extracts image features from the second image frame carrying head depth information to obtain multiple second head pose features; extracts image features from the fourth image frame to obtain a third head pose feature, the fourth image frame carrying head depth information of the third image frame; the electronic device fuses the first head pose feature with the multiple second head pose features to obtain multiple second head pose deviation features; the electronic device obtains the second head pose spatial deviation feature based on the fourth image frame and the multiple second head pose deviation features.

[0019] The second image frame, carrying head depth information, can be acquired by a depth sensor and contains multiple depth images of the user looking at the first calibrated position on the display screen under different head postures. The electronic device can determine the second head posture deviation characteristics based on these depth images, thus determining the user's head posture deviation from the perspective of depth image information. This allows for the determination of the deviation position of the user's gaze point in multiple dimensions, improving accuracy.

[0020] In conjunction with the first aspect, in some embodiments, the camera is a front-facing camera.

[0021] The front-facing camera and the display panel are located on the same side of the electronic device. When a user is using the electronic device, their eyes are focused on the display screen. The front-facing camera can more easily capture images of the user looking at the display screen, thereby determining the user's position when looking at the display screen. Based on the user's gaze position on the display screen, various functions can be provided to the user, improving the user experience.

[0022] In a second aspect, embodiments of this application provide an electronic device, which includes a memory and a processor coupled to the memory; the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method described in any one of the first aspects.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program or computer instructions, which are executed by a processor to implement the method described in any of the first aspects above.

[0024] Fourthly, embodiments of this application provide a computer program product, which, when executed by a processor, implements the method described in any of the first aspects above.

[0025] Fifthly, embodiments of this application provide a chip including a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, causing the chip to perform the method described in any of the first aspects above.

[0026] The solutions provided in the second to fifth aspects above are used to implement or cooperate with the methods provided in the first aspect above, and therefore can achieve the same or corresponding beneficial effects as the methods in the first aspect, which will not be elaborated here. Attached Figure Description

[0027] Figure 1A This is a schematic diagram of the eye movement direction vector of the human eye provided in an embodiment of this application;

[0028] Figure 1B These are schematic diagrams of faces with different head orientation vectors provided in the embodiments of this application;

[0029] Figure 1C These are schematic diagrams of some different eye direction vectors provided in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application;

[0031] Figures 3A-3KThis is a schematic diagram of the user interface involved in the calibration of user eye features and head posture features by some electronic devices 100 provided in the embodiments of this application;

[0032] Figures 4A-4C This is a schematic diagram of a user interface for an electronic device 100 that provides message expansion functionality based on user eye movement, as provided in the embodiments of this application.

[0033] Figures 5A-5F These are schematic diagrams illustrating some ways users can change their head position or posture, as provided in the embodiments of this application.

[0034] Figure 6 This is a schematic flowchart of a method for determining a user's gaze position using an electronic device according to an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the process for calculating fixation point deviation when the electronic device provided in this application only uses the front-facing camera to collect user eye movement images;

[0036] Figure 8 This is a schematic diagram of the process for calculating gaze deviation when an electronic device uses a front-facing camera and a depth sensor to collect user eye movement images, as provided in the embodiments of this application.

[0037] Figure 9 This is a schematic diagram of another electronic device provided in this application that calculates gaze deviation when using a front-facing camera and a depth sensor to acquire user eye movement images;

[0038] Figure 10 This is a flowchart illustrating how the electronic device 100 provided in this application extracts user eye features from a two-dimensional image;

[0039] Figure 11 This is a flowchart illustrating how an electronic device 100, according to an embodiment of this application, extracts user head pose features from a depth image;

[0040] Figure 12 This is a flowchart illustrating the eye-tracking method provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of the structure of an eye-tracking device 1300 provided in an embodiment of this application;

[0042] Figure 14 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0043] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0044] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0045] In recent years, eye tracking has become an important human-computer interaction technology, widely used in various fields such as healthcare, education, consumer electronics, robotics, virtual reality (VR), and augmented reality (AR). Electronic devices can record and track a user's eye movements using a camera to determine the user's gaze position. The electronic device can then provide feedback based on the user's gaze position. Accurate eye tracking is crucial for providing accurate feedback to the user.

[0046] Figure 1A This is a schematic diagram of the eye movement direction vector of a human eye provided in an embodiment of this application. The human eye movement direction vector can be decomposed into a head direction vector and an eye direction vector. For example, Figure 1B As shown, Figure 1B These are schematic diagrams of faces with different head orientation vectors provided in embodiments of this application. The head orientation vectors can be used to reflect the orientation of the user's head. Figure 1C This is a schematic diagram of some different eye direction vectors provided in the embodiments of this application. The eye direction vector can be used to reflect the orientation of the user's eyes. In some eye-tracking methods, the electronic device only calibrates the user's eye features during eye movements based on the eye image when the user is viewing the display screen in the same head posture and position, but ignores the distortion or warping of the eye image caused by changes in the user's head posture and / or head position. That is, when determining the user's eye movement direction, only the user's eye direction vector is considered, but the user's head direction vector is not considered. This may result in the electronic device calibrating the user's eye features incompletely, leading to errors in subsequent eye movement tracking based on the calibrated eye features.

[0047] To address the aforementioned problems, embodiments of this application provide an eye-tracking method, an electronic device, and a computer-readable storage medium. In this method, the electronic device can acquire images of a user viewing different positions on a display screen while in the same head posture, and can also acquire images of a user viewing the same position on the display screen while in different head postures and / or different head positions. This allows the electronic device to acquire images of the user's eyes more comprehensively across various head positions and postures. The electronic device can more accurately extract image features from these images when the user views different positions on the display screen, thereby more accurately determining the user's eye movement direction based on the extracted image features during subsequent use of the electronic device.

[0048] The electronic device 100 provided in the embodiments of this application will be introduced first below.

[0049] Figure 2 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0050] Electronic device 100 may be equipped with Or other portable terminal devices with different operating systems, such as mobile phones, tablets, desktop computers, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.

[0051] The following description uses electronic device 100 as an example to illustrate the embodiment. It should be understood that electronic device 100 may have more than Figure 2 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0052] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, sensor module 140, camera 150, display screen 130, etc. Sensor module 140 may include pressure sensor 140A and depth sensor 140B.

[0053] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0054] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0055] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0056] Electronic device 100 implements display functions through a GPU, display screen 130, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 130 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0057] The display screen 130 is used to display images, videos, etc. The display screen 130 includes a display panel.

[0058] Electronic device 100 can perform shooting functions through ISP, camera 150, video codec, GPU, display screen 130 and application processor.

[0059] The ISP (Image Signal Processor) is used to process data fed back from the camera 150. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 150.

[0060] Camera 150 is used to capture still images or videos. Camera 150 may include a front-facing camera 150A. Figure 2 (Not shown in the image). The front-facing camera 150A can be located on the same side of the electronic device 100 as the display panel, so that when the user views the user interface displayed on the screen, the front-facing camera 150A can capture image data of the user's head and face. The front-facing camera 150A may include a shutter, a lens, and a photosensitive element. When the shutter opens, an optical image of the object is generated through the lens and projected onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the front-facing camera 150A can be an RGB camera, an infrared camera, etc.

[0061] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable applications such as intelligent cognition in electronic devices, for example, extracting user head pose and eye features, and tracking user eye movements based on these features.

[0062] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.

[0063] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as eye tracking function), etc. The data storage area may store data created during the use of electronic device 100 (such as user eye characteristics, head posture characteristics, eye deviation characteristics, head posture deviation characteristics, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0064] Pressure sensor 140A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 140A may be disposed on display screen 130. When a touch operation is applied to display screen 130, electronic device 100 detects the intensity and location of the touch operation based on pressure sensor 140A.

[0065] Depth sensor 140B can be used to acquire depth information. Depth sensor 140B can be, for example, a laser sensor. Depth sensor 140B can acquire the distance between an object and electronic device 100, i.e., acquire depth information, through time-of-flight (TOF). Depth sensor 140B may include, for example, an illumination unit, an optical lens, a photosensitive element, and a computing unit. The illumination unit can emit modulated light pulses, which can be infrared light. After the light pulse hits the object, it is reflected by the optical lens, which filters the received light, allowing only light with the same wavelength as the light emitted by the illumination unit to pass through. The filtered light reaches the photosensitive element. The photosensitive element converts the light signal into an electrical signal for the computing unit to calculate. The computing unit can determine the flight distance of the light pulse based on the time difference between the light pulse emitted by the illumination unit and the light pulse received by the photosensitive element, thus determining the distance between the object and depth sensor 140B.

[0066] The depth sensor 140B can be positioned on the same side of the electronic device 100 as the camera 150, such as the front-facing camera 150A, and within a certain distance range from the front-facing camera 150A, such as 1 centimeter. In this way, when the camera 150 captures a two-dimensional image, the depth sensor 140B can also provide depth information of the object being photographed.

[0067] In some embodiments, camera 150 may be a depth camera capable of providing depth information. Specifically, camera 150 may be a camera coupled to depth sensor 140B, such as a time-of-flight (TOF) depth camera or a structured light depth camera, so that camera 150 can acquire both two-dimensional images and depth images (i.e., depth information). Alternatively, camera 150 may also include multiple cameras, and camera 150 may utilize the parallax of multiple cameras at the same point to acquire the depth information of an object; for example, camera 150 may be a binocular depth camera.

[0068] For ease of description, the following embodiments of this application will use a front-facing camera to track the user's eye movements to introduce the eye-tracking method. It is not limited to the front-facing camera, but can also be a camera at other locations on the electronic device 100. The embodiments of this application do not limit this.

[0069] Figures 3A-3K This is a schematic diagram of a user interface involved in the calibration of user eye features and head posture features by some electronic devices 100 provided in this application embodiments. The purpose of calibration is to acquire images of the user gazing at various locations on the display screen with the same head posture and position, as well as gazing at the same location on the display screen with different head postures and / or different head positions. The electronic device 100 can extract image features based on these images, including the user's eye features and head posture features. These eye features and head posture features can be used by the electronic device to subsequently track the user's eye movements, thereby determining the user's gaze position on the display screen.

[0070] like Figure 3A As shown, the electronic device 100 can display a settings interface 300, which may include one or more function options for the electronic device 100, such as mobile network function options, desktop and wallpaper function options, display and brightness function options, smart assistant function options 301, etc. These function options are used to jump to the management interface of the corresponding function. The electronic device 100 can receive user actions on the smart assistant function options 301 and respond to these actions, such as... Figure 3BAs shown, the electronic device 100 can access the smart assistant management interface 310, which is used to manage one or more smart functions. These smart functions may include, for example, smart sensing functions, quick launch and gesture functions, and eye-tracking functions. The eye-tracking function tracks the user's eye movements to determine the user's gaze position on the display screen. The smart assistant management interface 310 may include options for these smart functions, such as the eye-tracking function option 311, which is used to set the parameters of the eye-tracking function and its start / stop function, etc. Initially, the electronic device 100 does not store the user's eye features and / or head posture features, so the eye-tracking function cannot be directly enabled. At this time, the right side of the eye-tracking function option 311 displays "Uncalibrated". The electronic device 100 first needs to acquire facial and head images of the user viewing different positions on the display screen, extract image features based on these facial and head images, and store the image features (this process can also be called calibration). In this way, when the electronic device 100 subsequently acquires images of the user viewing the display screen, it can determine the user's gaze position based on the aforementioned image features.

[0071] Electronic device 100 can receive user input on eye-tracking function option 311, such as... Figure 3C As shown, in response to this operation, the electronic device 100 can enter the eye-tracking calibration interface 320, which is used to acquire images of the user viewing the display screen and complete the calibration process. The eye-tracking calibration interface 320 may include a start button 321, which is used to start the calibration process.

[0072] like Figure 3C As shown, the electronic device 100 can receive the user's operation on the start button 321. In response to this operation, the electronic device 100 can begin to calibrate the user's facial and head image features. The electronic device 100 can first acquire facial images of the user viewing different positions on the display screen under the same head posture and position (which can be simply referred to as different points in the same posture). Figure 3D As shown, the electronic device 100 can first display prompts 322 on the eye-tracking calibration interface 320. These prompts 322 can guide the user to fix their head and look at different positions on the display screen. For example, the prompts 322 could include: "Please keep your head still and look at the point displayed on the screen," making it easier for the user to understand and perform corresponding actions based on the prompts 322, thus completing the acquisition of the user's facial and head images more quickly. Optionally, the electronic device 100 can also instruct the user to face the front-facing camera.

[0073] The electronic device 100 can receive a user's action on the next button 323, and in response to this action, the electronic device 100 can begin acquiring images of the user looking at different positions on the display screen while maintaining the same head posture and head position. For example... Figure 3E As shown, with the user facing the display screen, the electronic device 100 displays a mark 324 in the upper left corner of the screen. The mark 324 may be circular, and numbers may be displayed within it. These numbers indicate how many marks the user needs to look at during the calibration process. Optionally, the electronic device 100 may also display a prompt next to the mark 324 to instruct the user to view it. Figure 3E (Not shown in the image). Furthermore, the electronic device 100 can capture one or more two-dimensional images of the user viewing the sign 324 using the front-facing camera 150A. In other embodiments, the electronic device 100 also uses a depth sensor 140B to obtain depth information of the user's image while capturing the aforementioned two-dimensional images, thus allowing the electronic device 100 to also obtain a depth image of the user viewing the sign 324. The electronic device 100 can hold the sign 324 for a preset duration, such as two seconds, three seconds, etc., and then display the sign in other locations. For example... Figure 3F As shown, the electronic device 100 can display a mark 324 in the center of the display screen and hold it there for a preset duration. Here, the center of the display screen can also be referred to as the reference position. Optionally, the electronic device 100 can also display instruction information, instructing the user to look at the mark 324 in the center of the display screen. During the display of the mark 324 in the center of the display screen, the electronic device 100 can capture a two-dimensional image and / or depth image of the user using a front-facing camera 150A. After the mark 324 has been displayed in the center of the display screen for the preset duration, the electronic device 100 can change the display position of the mark 324. For example... Figure 3G As shown, the electronic device 100 can also display an identifier 324 in the lower right corner of the display screen and hold the display for a preset duration. Furthermore, the electronic device 100 can capture one or more frames of a two-dimensional image and / or depth image of the user during the period the identifier 324 is displayed. In this way, the electronic device 100 acquires two-dimensional images and / or depth images of the user viewing multiple points on the display screen while maintaining the same head posture and head position.

[0074] in Figures 3E-3G The number and position of the identifiers 324 shown are merely examples. The electronic device 100 may also display the identifiers 324 in more or fewer positions and guide the user to look at the identifiers 324. In this way, the electronic device 100 can obtain two-dimensional images and / or depth images when the user views more points on the display screen while the head posture and head position remain unchanged. Furthermore, there may be more positions where the identifiers 324 are displayed on the display screen. This application embodiment does not limit this.

[0075] Next, the electronic device 100 can capture images of the user viewing a reference position on the display screen when the user's head posture and / or head position changes (which can be simply referred to as the same point but different postures).

[0076] like Figure 3H As shown, the electronic device 100 can display prompts on the eye-tracking calibration interface 320, instructing the user to move their head and maintain focus on the same point on the display screen throughout the movement. The electronic device 100 can receive user input to the next button 323 and respond to that input as follows: Figure 3I As shown, the electronic device 100 can display a mark 325 in the center of the display screen. The mark 325 can also display the remaining time displayed on the screen. For example, if the mark 325 displays "3", it indicates that the mark 325 will remain on the screen for another 3 seconds, meaning that the calibration of the image when the user looks at the same position on the screen from different head positions will continue for another three seconds. The position where the mark 325 is displayed can also be called the reference position. The electronic device 100 can then determine the user's gaze position based on the reference position. The user can move their head left or right and focus their eyes on the mark 325 displayed on the screen. Figure 3I As shown, the user is initially positioned facing the front-facing camera 150A directly on the screen. Then, the user moves their head to the left, as... Figure 3J As shown, the user changes position to the left of the front-facing camera 150A on the display screen. The position of the identifier 325 on the display screen remains constant. The electronic device 100 can capture one image frame at preset intervals, thus acquiring images of the user looking at the same point on the display screen in left-right translational directions. Optionally, the electronic device 100 can also acquire depth information of the user's head posture using the depth sensor 140B while capturing the aforementioned image frames.

[0077] When the duration displayed by identifier 325 reaches the preset duration, the electronic device 100 has completed image acquisition. The electronic device 100 can extract image features based on the acquired image frames. The method by which the electronic device 100 extracts image features can be referred to in the following embodiments, and will not be elaborated here. After extracting the image features, the electronic device 100 can store the image features in its internal memory.

[0078] like Figure 3K As shown, the electronic device 100 can display a calibration completion prompt 326, which indicates that the electronic device 100 has successfully acquired images of the user looking at the display screen. This allows the user to use the eye-tracking function. The electronic device 100 can extract image features based on the acquired images to track the user's eye movements.

[0079] Figures 3H-3J The images of a user looking at a point on the display screen when the electronic device 100 is in different head positions are only examples and are not limited to the point in one position mentioned above. The electronic device 100 can also guide the user to look at other points on the display screen in the same head / head posture change mode, and then collect images and depth information of the user looking at other points on the display screen in different head postures / head positions. This application embodiment does not limit this.

[0080] After acquiring the aforementioned image, the electronic device 100 can track the user's eye movements based on the image features of the image, and then provide one or more functions based on the user's eye movements. For example, Figures 4A-4C This is a schematic diagram of a user interface for an electronic device 100 that provides message expansion functionality based on user eye movement, as provided in the embodiments of this application.

[0081] like Figure 4A As shown, the electronic device 100 displays a short message notification box 401 on top of the eye-tracking calibration interface 320. The short message notification box 401 can be displayed after a chat application receives a new message. The short message notification box 401 can display the message contact ("Mom") and the message content ("Are you coming home for dinner today?"). The electronic device 100 can capture images through the front-facing camera 150A and track the user's eye movements based on the captured images.

[0082] like Figure 4B As shown, the electronic device 100 can also display the user's identified gaze position on the screen and display a gaze indicator 402 at that gaze position. When the electronic device 100 determines that the user's gaze position is located at the location of the short message notification box 401, and the user gazes at that location for a duration exceeding a specified time, the electronic device 100 can expand and display the short message notification box 401. Alternatively, the electronic device 100 can jump to the chat conversation interface of that contact in a chat application. Figure 4C As shown, when the electronic device 100 detects that the user's gaze at the short message notification box 401 exceeds a preset duration, the electronic device 100 can jump to the chat interface 410 provided by the chat application. The chat interface 410 can be the chat interface of the contact corresponding to the newly received message in the short message notification box 401, used to chat with that contact. The chat interface 410 can include chat history and a message editing box 411. The chat history can contain received chat messages, and the message editing box 411 is used to edit new messages and send messages to contacts. In other words, the user can jump to the user interface without clicking their finger, thereby reducing the user's operational burden and improving the user experience.

[0083] It should be noted that, Figures 3H-3JThe illustrated embodiment is provided for ease of description and better understanding, using the example of a user moving their head to illustrate how the electronic device 100 can acquire images of the user in different head positions or postures. The electronic device 100 can also guide the user to adopt more head postures or move their head in other directions to look at a point on the display screen, thereby acquiring user images. This application embodiment does not limit this.

[0084] For example, Figures 5A-5F This is a schematic diagram illustrating some ways users can change their head position or posture, as provided in the embodiments of this application.

[0085] For ease of description, we select the state where the user's head is facing the display screen (i.e., facing the front-facing camera) and construct a coordinate system based on the user's head position, where the horizontal direction relative to the display screen is taken as the x-axis, the vertical direction relative to the display screen is taken as the y-axis, and the direction perpendicular to the display screen is taken as the z-axis.

[0086] like Figure 5A As shown, the electronic device 100 can guide the user to face the front-facing camera 150A and move their head horizontally (or along the x-axis) relative to the display screen. The electronic device 100 can display a sign at a reference position on the display screen and guide the user to look at the sign during the display period. Furthermore, during the user's head movement, the electronic device can capture multiple two-dimensional images through the front-facing camera 150A, thereby obtaining a two-dimensional image of the user looking at the same point on the display screen during lateral movement.

[0087] like Figure 5B As shown, the electronic device 100 can also guide the user to face the front-facing camera 150A and move their head vertically (or along the y-axis) relative to the display screen. The electronic device 100 can display a marker of a preset duration at a reference position on the display screen and guide the user to move their head vertically and look at the marker during the display period. During the user's head movement, the electronic device can capture two-dimensional images through the front-facing camera 150A, thereby obtaining a two-dimensional image of the user looking at the same point on the display screen during the vertical movement.

[0088] like Figure 5C As shown, the electronic device 100 can also display a marker of a preset duration at a reference position on the display screen, and then guide the user to translate their head back and forth (or along the z-axis) relative to the display screen during the display period of the marker. The user keeps looking at the marker displayed on the display screen while translating their head, and the electronic device 100 can capture a two-dimensional image of the user during the user's head translation (i.e., during the display of the marker), thereby obtaining a two-dimensional image of the user looking at the same point on the display screen while translating their head back and forth.

[0089] like Figure 5DAs shown, the electronic device 100 can also display a preset duration indicator at a reference position on the display screen, and guide the user to rotate their head along the scroll angle (or along the z-axis) and look at the indicator displayed on the display screen during the rotation. Furthermore, the electronic device 100 can capture a two-dimensional image of the user looking at the same point on the display screen while rotating their head along the z-axis.

[0090] like Figure 5E As shown, the electronic device 100 can also guide the user to rotate their head along a yaw angle (or along the y-axis) and look at a marker displayed on the screen at a reference position during the rotation. Furthermore, it captures a two-dimensional image of the user looking at the same point on the screen while rotating their head along the y-axis.

[0091] like Figure 5F As shown, the electronic device 100 can also display a preset duration indicator at a reference position on the display screen, and guide the user to rotate their head along a pitch angle (or along the x-axis) and look at the indicator on the display screen during the rotation. Furthermore, the electronic device 100 can capture a two-dimensional image of the user looking at the same point on the display screen while rotating their head along the x-axis.

[0092] Optionally, in the above Figures 5A-5F When the user changes their head position or posture, the electronic device 100 can acquire two-dimensional images through the front-facing camera 150A and also obtain depth information through the depth sensor 140B.

[0093] Among them, Figures 5A-5F The method for displaying indicators on electronic devices 100 in scenarios where the user changes head position or posture can be referred to. Figures 3H-3J The embodiments shown are not described in detail here.

[0094] Not limited to Figures 5A-5F The electronic device 100 can also guide the user to change head position or head posture in other ways, as shown in the embodiments of this application.

[0095] Figure 6 This is a schematic flowchart of a method for determining a user's gaze position using an electronic device, as provided in an embodiment of this application.

[0096] like Figure 6 As shown, the electronic device 100 can extract the user's eye features and / or head posture features from the images acquired during the calibration process described above, and then determine the gaze point deviation between the user's gaze position and the reference position. Furthermore, the electronic device 100 can determine the user's gaze position based on the gaze point deviation and the reference position. The calculation methods for the reference position and gaze point deviation can be found in the following embodiments, and will not be elaborated upon here.

[0097] The following section details how electronic device 100 calculates gaze deviation.

[0098] When the electronic device 100 uses different types of front-facing cameras to track the user's eye movements, the electronic device 100 may have different methods for calculating gaze deviation.

[0099] 1. Electronic devices use only the front-facing camera 150A to track the user's eye movements.

[0100] Figure 7 This is a schematic diagram of the process for calculating gaze deviation when the electronic device provided in this application uses only the front-facing camera to collect user eye movement images.

[0101] like Figure 7 As shown, the electronic device 100 first acquires a calibration two-dimensional image set A and a calibration two-dimensional image set B through a front-facing camera. The calibration two-dimensional image set A contains multiple images of the user viewing different positions on the display screen while in the same head posture and position. The method by which the electronic device 100 acquires the calibration two-dimensional image set A can be referred to... Figure 3D - Figure 3G The illustrated embodiment. The calibration two-dimensional image set B contains multiple two-dimensional images of the user viewing a reference position on the display screen under different head postures and / or different head positions. The method for the electronic device 100 to acquire the calibration two-dimensional image set B can be referred to... Figures 3H-3J The embodiments shown are not described in detail here.

[0102] For ease of description, the image acquired by electronic device 100 in calibration two-dimensional image set A when displaying marker 324 at the reference position is called the reference calibration two-dimensional image. The n images acquired by electronic device 100 in calibration two-dimensional image set A when displaying marker 324 at a non-reference position are called reference calibration images A, and are identified by serial numbers in the lower right corner. The m images acquired by electronic device 100 in calibration two-dimensional image set B are called reference calibration images B, and are identified by serial numbers in the lower right corner, where n and m are positive integers.

[0103] like Figure 7 As shown, the electronic device 100 can extract features from the baseline calibration two-dimensional image, reference calibration images A1 to An, and reference calibration images B1 to Bm, respectively. Feature extraction methods can include, but are not limited to, corner detection, edge detection, statistical distribution information extraction, and neural network extraction. Furthermore, the electronic device 100 can extract the user's eye features from the aforementioned images. The features extracted from the baseline calibration two-dimensional image are called baseline eye features, and the features extracted by the electronic device from reference calibration images A1 to An are respectively denoted as eye features F1 to eye features Fm. n The electronic device is aligned with reference calibration diagram B1—reference calibration diagram B. mThe extracted features are denoted as eye feature T1 and eye feature T1, respectively. m Furthermore, the electronic device 100 can compare the reference eye features with the aforementioned eye features F1—eye features F n Feature fusion is performed to obtain the eye deviation feature F'1 — eye deviation feature F' n The aforementioned eye features can be represented as multi-dimensional tensors to describe image features extracted from the image across different dimensions. Eye deviation features can be used to reflect the eye image deviation when the user looks at different points compared to a reference point. Feature fusion can be achieved by calculating the difference between two features, or by merging the features by channel and then extracting them using a convolutional kernel. Alternatively, when using a neural network for feature extraction, feature fusion can involve connecting the fully connected layers of the convolutional neural network used in the feature extraction process. The electronic device 100 can also connect the reference eye features to the aforementioned eye features T1 and T1 respectively. m Feature fusion is performed to obtain the head pose deviation feature T'1 – the head pose deviation feature T' m Head posture deviation feature T'1 — Head posture deviation feature T' m It can be used to reflect the deviation between the eye image of a user looking at a reference position under different head postures and / or different head positions and the eye image of a user looking at the reference position directly in front of the front camera.

[0104] When performing eye tracking, the electronic device 100 can continuously capture images of the user's eyes through its front-facing camera. For example, the electronic device 100 captures a two-dimensional image C, which is a two-dimensional image of the user's gaze position to be predicted. This image can be... Figures 4A-4C In the scenario shown, the electronic device 100 can collect data from the two-dimensional image C, and then extract features from the image to obtain the eye features of the two-dimensional image under test. The electronic device 100 can then convert the eye deviation features F'1 to the eye deviation features F'. n The eye features of the two-dimensional image to be tested are input into neural network model 1 to obtain the eye movement deviation, and the head posture deviation features T'1 and T' are used to obtain the head posture deviation features T'1. m The eye features of the two-dimensional image under test are input into neural network model 2 to obtain the head pose spatial deviation 1. Both the eye movement deviation and the head pose spatial deviation 1 can be represented in tensor form. Finally, electronic device 100 inputs the aforementioned eye movement deviation and head pose spatial deviation 1 into neural network model 3 to obtain the fixation point deviation.

[0105] 2. Electronic devices simultaneously use a front-facing camera 150A and a depth sensor 140B to track the user's eye movements.

[0106] In other words, the electronic device 100 uses both the front-facing camera 150A to capture two-dimensional images and the depth sensor 140B to capture depth images (depth information). The electronic device 100 then uses the captured two-dimensional images and depth images together to determine the user's eye movements.

[0107] Figure 8 This is a schematic diagram illustrating the process of calculating gaze deviation when an electronic device, provided in this application embodiment, uses a front-facing camera and a depth sensor to acquire user eye-movement images.

[0108] The method for determining eye movement deviation based on a calibrated two-dimensional image set A in electronic device 100 can be referenced. Figure 7 The description of the illustrated embodiments will not be repeated here.

[0109] The calibration of the two-dimensional image set H can be done by electronic device 100. Figures 3H-3J The depth information was collected by the depth sensor in the scenario shown. That is, when the user changes head position and looks at the mark 325 on the display screen, the electronic device 100 uses the depth sensor 140B to collect the user's depth information.

[0110] For ease of description, the j images acquired by electronic device 100 in the calibration two-dimensional image set H are called reference calibration images H, and are identified by a number in the lower right corner, where j is a positive integer. The depth image acquired by electronic device 100 through the depth sensor when the user is facing the display screen is called the reference calibration depth image. Then, electronic device 100 can respectively calibrate the reference calibration images H1 to H2. j Feature extraction is performed to obtain head pose features K1 — head pose features K j The electronic device can also extract features from the reference calibration depth map to obtain reference head pose features. These head pose features can be represented as multidimensional tensors, which can be used to reflect the shape, position, depth, and other multidimensional features of the user's head pose. Furthermore, the electronic device 100 can use the reference head pose features and the aforementioned head pose features K1—head pose features K... j Feature fusion is performed to obtain head pose feature K'1 — head pose feature K' j Head posture feature K'1 — Head posture feature K' j This can be used to reflect the deviation between a depth image of a reference position on the display screen when the user is looking at it under different head postures and / or different head positions, and a depth image of a reference position on the display screen when the user is looking directly at the camera. Among them, electronic device 100 has a pair of reference calibration maps H1 and H2. j Methods for feature extraction can include, but are not limited to, methods based on convolutional neural networks for feature extraction, methods for head pose calculation based on facial landmark detection, and so on.

[0111] When performing eye tracking, the electronic device 100 acquires user images via a front-facing camera and a depth sensor at preset time intervals. At each time point, the electronic device 100 can acquire images of different dimensions using the front-facing camera and the depth sensor respectively. For example, the electronic device 100 acquires a two-dimensional image C via the front-facing camera while simultaneously obtaining depth information via the depth sensor, resulting in a depth image D. The electronic device 100 can then perform feature extraction on the depth image D to obtain the head pose features of the depth image to be measured, and then combine these head pose features with head pose feature K'1—head pose feature K'. j The input to neural network model 4 yields the head pose spatial deviation 2. Then, electronic device 100 can input the eye movement deviation and the head pose spatial deviation 2 into neural network model 5 to obtain the fixation point deviation. Alternatively, electronic device 100 can add the eye movement deviation and the head pose spatial deviation 2 to obtain the fixation point deviation. The aforementioned head pose spatial deviation 2 can also be represented as a multidimensional tensor. Adding the eye movement deviation and the head pose spatial deviation 2 is equivalent to adding corresponding elements of the tensor.

[0112] Figure 9 This is a schematic diagram illustrating the process of calculating gaze deviation when an electronic device uses a front-facing camera and a depth sensor to acquire user eye-movement images, as provided in another embodiment of this application.

[0113] like Figure 9 As shown, the electronic device 100 can acquire a calibration two-dimensional image set A and a calibration two-dimensional image set B through a front-facing camera, and acquire a calibration depth image set H through a depth sensor. The methods by which the electronic device 100 determines the eye movement deviation based on the calibration two-dimensional image set A, the head posture spatial deviation 1 based on the calibration two-dimensional image set B, and the head posture spatial deviation 2 based on the calibration depth image set H can be referred to the description in the above embodiments, and will not be repeated here.

[0114] Finally, the electronic device 100 can input the eye movement deviation, head posture spatial deviation 1, and head posture spatial deviation 2 into the neural network model 6 to obtain the fixation point deviation.

[0115] The following is an example of a method for extracting eye features from a two-dimensional image by an electronic device 100. This method can be applied to extracting reference eye features from a two-dimensional reference calibration image, and can also be applied to extracting eye features from a two-dimensional reference calibration image (such as reference calibration image A1 – reference calibration image A). n Or refer to calibration diagram B1 — refer to calibration diagram B m It can extract eye features and can also be applied to extract eye features from a two-dimensional image C of the gaze position to be predicted.

[0116] Figure 10This is a flowchart illustrating how an electronic device 100, as provided in this application embodiment, extracts user eye features from a two-dimensional image.

[0117] like Figure 10 As shown, the electronic device 100 can first extract an eye region image from a two-dimensional image, and then extract an eye mask from the region where the eye is located. The eye mask can be a binary image, where the eye region is marked as 1 and the region outside the eye is marked as 0. Then, the electronic device 100 can input the eye region image into a neural network model 7 to obtain intermediate feature 1, and input the eye mask into a neural network model 8 to obtain intermediate feature 2. The aforementioned intermediate feature 1 and intermediate feature 2 can be tensors used to represent a series of features such as the shape of the human eye contour and the position of the eyeball. Then, the electronic device 100 can concatenate intermediate feature 1 and intermediate feature 2, and input the concatenated feature into a neural network model 9 to obtain the eye features.

[0118] The following is an example of a method for extracting head pose features from depth images by an electronic device 100. This method can be applied to extracting reference depth features from a reference calibration depth map by the electronic device 100, and can also be applied to extracting head pose features from a reference calibration map H1 to a reference calibration map H2. j Head pose feature extraction can also be applied to electronic devices 100 to extract head pose features from depth images D of the gaze position to be predicted, etc.

[0119] Figure 11 This is a flowchart illustrating how an electronic device 100, as provided in this application embodiment, extracts user head pose features from a depth image.

[0120] like Figure 11 As shown, the electronic device 100 can input the user's head image acquired by the depth sensor into the neural network model 10, and then the neural network model can output head pose features, wherein the head pose features can be tensors that can reflect the user's head posture, head position, head shape and other features.

[0121] The neural network models mentioned in this application embodiment may include, but are not limited to, convolutional neural networks (CNN) models, recurrent neural networks (RNN) models, long short-term memory (LSTM) models, deep neural networks (DNN) models, etc. This application embodiment does not limit the type of neural network model.

[0122] Figure 12 This is a flowchart illustrating the eye-tracking method provided in the embodiments of this application.

[0123] like Figure 12 As shown, the method includes, but is not limited to, the following steps:

[0124] S1201. The electronic device displays a first identifier at multiple locations on the display screen and captures multiple first image frames by a camera. The multiple first image frames contain images of the first identifier at different locations on the display screen when the user is in the same head position.

[0125] refer to Figures 3E-3G In the illustrated embodiment, the first identifier may be, for example, identifier 324. The electronic device may sequentially display the first identifier (or simultaneously display it) at multiple locations on the display screen, and then capture multiple first image frames using a camera. The first image frame is a two-dimensional image, and may be, for example, the reference calibration two-dimensional image set A, or reference calibration image A1—reference calibration image A1 from the aforementioned calibration two-dimensional image set A. n Any one of them.

[0126] S1202, The electronic device displays a second logo at a location on the display screen and captures multiple second image frames, the multiple second image frames containing images of the user looking at the second logo in different head positions.

[0127] refer to Figures 3I-3J In the illustrated embodiment, the second identifier may be, for example, identifier 325. The electronic device may display the second identifier at a location on the display screen, which may be referred to as a reference position. During the display of the second identifier, the electronic device may capture multiple second image frames, which are also two-dimensional images, and may be, for example, the aforementioned reference calibration diagram B1 – Reference Calibration Diagram B. m Reference calibration diagram H1 — Reference calibration diagram H j Any one of the reference calibration depth maps. Among them, reference calibration map B1—reference calibration map B m It can be captured by a camera, see calibration diagram H1 – reference calibration diagram H j The reference calibration depth map can be captured by a depth sensor.

[0128] Head positioning includes both head position and head posture. Users can refer to [the relevant documentation / reference] to change their head positioning. Figures 5A-5F The embodiments shown are not described in detail here.

[0129] S1203. The electronic device acquires a third image frame through a camera, and then determines the user's first gaze position on the display screen based on multiple first image frames, multiple second image frames, and the third image frame.

[0130] The electronic device can capture a third image frame via a camera. This third image frame can be, for example, a two-dimensional image C of the gaze position to be predicted, as described in the foregoing embodiments. Exemplarily, the third image frame can be... Figures 4A-4C The data collected by the electronic device in the scenario shown.

[0131] In some embodiments, before the electronic device sequentially displays the first identifier at multiple locations on the display screen, the method further includes: the electronic device receiving a first user operation, the first user operation being used to begin a first calibration of eye features and / or head posture features when the user views different locations on the display screen; before the electronic device acquires a third image frame via a camera, the method further includes: the electronic device indicating that the first calibration is complete.

[0132] refer to Figure 3C In the illustrated embodiment, the first user operation can be, for example, the user clicking the start button 321. This first user operation is used to begin calibrating the user's eye features and / or head posture features when viewing different positions on the display screen; this calibration can also be referred to as the first calibration. After receiving the first user operation, the electronic device can display a first identifier and a second identifier according to preset rules and capture the user's image. The user image can include the user's eye image and head image. When the electronic device 100 completes the calibration, it can indicate that the first calibration is complete. The electronic device 100 can prompt the user through a display screen, see reference [reference missing]. Figure 3K The embodiment shown is not limited to this. The electronic device 100 may also prompt the user through voice broadcast that the calibration of eye and head posture features has been successfully completed, or the electronic device 100 may also prompt the user through other means such as vibration. This application embodiment does not limit this.

[0133] In some embodiments, before the electronic device captures multiple first image frames via a camera, the method further includes: the electronic device prompting the user to keep their head fixed and look at a first identifier displayed on a screen; before the electronic device captures multiple second image frames, the method further includes: the electronic device prompting the user to change their head position to view a second identifier. Wherein, as... Figure 3D , Figure 3H In the illustrated embodiment, the electronic device 100 can display prompts on a screen. However, it is not limited to this; the electronic device 100 can also prompt the user through voice broadcasts or other means, and this embodiment does not impose any limitations on this.

[0134] In some embodiments, the method further includes: the electronic device unfolding to display a message at the first gaze position. (See reference...) Figures 4A-4CIn the illustrated embodiment, after the electronic device 100 determines that a message is displayed at the first gaze position, the electronic device 100 can expand and display the message at the first gaze position. For example, the electronic device 100 can enter the detailed interface of the message, i.e., the chat dialog interface 410. In this way, the message can be quickly expanded without the user moving their finger, reducing the user's operational burden.

[0135] In some embodiments, the electronic device determines a user's first gaze position on the display screen based on multiple first image frames, multiple second image frames, and a third image frame, including: the electronic device determining a gaze point deviation based on a first eye movement deviation, a first head posture spatial deviation, and / or a second head posture spatial deviation, wherein the first eye movement deviation is determined based on the first image frame, the first head posture spatial deviation is determined based on a second image frame without head depth information, and the second head posture spatial deviation is determined based on a second image frame with head depth information; the electronic device determines the first gaze position based on the gaze point deviation and a first calibration position, wherein the first calibration position is the position where the electronic device displays a first identifier in the first reference image frame, and a second identifier is displayed at the first calibration position, and the first reference image frame is selected from multiple first image frames.

[0136] refer to Figure 6 In the illustrated embodiment, the electronic device can determine the user's gaze position based on gaze deviation and a reference position. The reference position can also be referred to as the first gaze position. The first reference image frame can, for example, be the reference calibration two-dimensional graph in the above embodiment. The first eye movement deviation can be as described above. Figure 7 , Figure 8 , Figure 9 In the illustrated embodiment, the eye movement deviation, specifically the first head posture spatial deviation, can be as described above. Figure 7 , Figure 9 In the illustrated embodiment, the head posture space deviation amount 1, and the second head posture space deviation amount can be the above-mentioned... Figure 8 , Figure 9 The head posture spatial deviation 2 in the illustrated embodiment.

[0137] In some embodiments, the method further includes: determining a first eye movement deviation based on a first image frame according to the following steps: the electronic device extracts image features from a first reference image frame to obtain a first eye feature, extracts image features from a first image frame other than the first reference image frame to obtain multiple second eye features, and extracts image features from a third image frame to obtain a third eye feature; the electronic device fuses the first eye feature with the multiple second eye features to obtain multiple eye deviation features; and the electronic device obtains the first eye movement deviation based on the third eye feature and the multiple eye deviation features.

[0138] refer to Figures 7-9In the illustrated embodiment, the electronic device extracts image features from multiple first image frames to obtain multiple second eye features. These second eye features can be, for example, eye feature F1 – eye feature F... n The electronic device can also extract image features from the first reference image frame to obtain a first eye feature, such as a reference eye feature. The third image frame is, for example, a two-dimensional image C of the gaze position to be predicted, and the third eye feature is, for example, the eye feature of the aforementioned two-dimensional image to be tested. The method by which the electronic device obtains the first eye movement deviation based on the aforementioned first eye feature and multiple second eye features can be referred to the description in the above embodiments, and will not be repeated here.

[0139] In some embodiments, the method further includes: determining a first head pose spatial deviation based on multiple second image frames that do not carry head depth information according to the following steps: the electronic device extracts image features from the second image frames that do not carry head depth information to obtain multiple fourth eye features; the electronic device fuses the first eye features with the multiple fourth eye features to obtain multiple first head pose deviation features; the electronic device obtains the first head pose spatial deviation based on the third eye features and the multiple head pose deviation features.

[0140] The second image frame that does not carry head depth information is, for example, the one described above. Figure 7 , Figure 9 Reference calibration diagram B1 of the illustrated embodiment — Reference calibration diagram B m This can be captured by a camera. The fourth eye feature is, for example, eye feature T1 – eye feature T. m The first head posture deviation feature is, for example, the head posture deviation features T'1 to T'm. The method for determining the first head posture spatial deviation based on the first eye feature and the fourth eye feature in electronic devices can refer to the above. Figure 7 , Figure 9 The description of the illustrated embodiments will not be repeated here.

[0141] In some embodiments, the multiple second image frames further include a second reference image frame carrying head depth information, and the method further includes: determining a second head pose spatial deviation based on the multiple second image frames carrying head depth information according to the following steps: the electronic device extracts image features from the second reference image frame to obtain a first head pose feature, extracts image features from the second image frame carrying head depth information to obtain multiple second head pose features, extracts image features from the fourth image frame to obtain a third head pose feature, the fourth image frame carrying head depth information of the third image frame; the electronic device fuses the first head pose feature with the multiple second head pose features to obtain multiple second head pose deviation features; the electronic device obtains the second head pose spatial deviation feature based on the fourth image frame and the multiple second head pose deviation features.

[0142] Among them, reference Figure 8 , Figure 9 In the illustrated embodiment, the second reference image frame carrying head depth information is, for example, a reference calibration depth map. The second image frame carrying head depth information also includes, for example, a reference calibration map H1—a reference calibration map H. j The second head pose feature is, for example, head pose feature K1 – head pose feature K. j The fourth image frame is, for example, the depth image D of the gaze position to be predicted, which can be acquired by the depth sensor simultaneously with the acquisition of the two-dimensional image C by the camera. The second head pose deviation feature can be, for example, head pose deviation feature K'1—head pose deviation feature K' j The method for obtaining the second head pose spatial deviation features based on the fourth image frame and multiple second head pose deviation features of the electronic device can be referred to the description in the above embodiments, and will not be repeated here.

[0143] The aforementioned head depth information can be collected by the depth sensor of the electronic device 100, and the aforementioned camera can be a front-facing camera.

[0144] Figure 13 This is a schematic diagram of the structure of an eye-tracking device 1300 provided in an embodiment of this application.

[0145] Figure 13 The eye-tracking device 1300 shown may be the electronic device described above, or it may be a chip or circuit disposed in the aforementioned electronic device. This eye-tracking device 1300 may be used to perform steps or functions performed by the electronic device in any of the embodiments of the above-described eye-tracking method.

[0146] The eye-tracking device 1300 includes a display unit 1301, an image acquisition unit 1302, and a processing unit 1303. The display unit 1301 displays a first identifier at multiple locations. The image acquisition unit 1302 captures multiple first image frames while the display unit 1301 displays the first identifier; these multiple first image frames contain images of the user gazing at the first identifier at different locations with the same head posture. The display unit 1301 also displays a second identifier at one location. The image acquisition unit 1302 further captures multiple second image frames while the display unit 1301 displays the second identifier; these multiple second image frames contain images of the user gazing at the second identifier with different head postures. The image acquisition unit 1302 also captures a third image frame. The processing unit 1303 determines the user's first gaze position based on the multiple first image frames, multiple second image frames, and the third image frame.

[0147] In some embodiments, the eye-tracking device 1300 further includes an operation receiving unit 1304 and a prompting unit 1305. The operation receiving unit 1304 is used to receive a first user operation before the display unit 1301 displays a first identifier. The first user operation is used to begin a first calibration of eye features and / or head posture features when the user views different positions on the display unit 1301. The prompting unit 1305 is used to prompt that the first calibration is complete before the image acquisition unit 1302 acquires a third image frame.

[0148] In some embodiments, the prompting unit 1305 is further configured to prompt the user to keep their head fixed and look at the first identifier displayed by the display unit 1301 before the image acquisition unit 1302 takes multiple first image frames; the prompting unit 1305 is further configured to prompt the user to change their head position to look at the second identifier before the image acquisition unit 1302 takes multiple second image frames.

[0149] In some embodiments, the processing unit 1303 is further configured to display a message at the first viewing position on the display unit 1301.

[0150] In some embodiments, the processing unit 1303 is configured to determine a user's first gaze position based on multiple first image frames, multiple second image frames, and a third image frame, including: the processing unit 1303 is configured to determine a gaze point deviation based on a first eye movement deviation, a first head posture spatial deviation, and / or a second head posture spatial deviation, wherein the first eye movement deviation is determined based on the first image frame, the first head posture spatial deviation is determined based on the second image frame without head depth information, and the second head posture spatial deviation is determined based on the second image frame with head depth information; the processing unit 1303 is configured to determine the first gaze position based on the gaze point deviation and a first calibration position, wherein the first calibration position is the position where the display unit 1301 displays a first identifier in the first reference image frame, and the second identifier is displayed at the first calibration position, and the first reference image frame is selected from multiple first image frames.

[0151] In some embodiments, the processing unit 1303 determines a first eye movement deviation based on a first image frame according to the following steps: the processing unit 1303 extracts image features from a first reference image frame to obtain a first eye feature, extracts image features from first image frames other than the first reference image frame to obtain multiple second eye features, and extracts image features from a third image frame to obtain a third eye feature; the processing unit 1303 fuses the first eye feature with the multiple second eye features to obtain multiple eye deviation features; the processing unit 1303 obtains the first eye movement deviation based on the third eye feature and the multiple eye deviation features.

[0152] In some embodiments, the processing unit 1303 is further configured to determine a first head pose spatial deviation based on multiple second image frames that do not carry head depth information according to the following steps: the processing unit 1303 extracts image features from the second image frames that do not carry head depth information to obtain multiple fourth eye features; the processing unit 1303 fuses the first eye features with the multiple fourth eye features to obtain multiple first head pose deviation features; the processing unit 1303 obtains the first head pose spatial deviation based on the third eye features and the multiple head pose deviation features.

[0153] In some embodiments, the multiple second image frames further include a second reference image frame carrying head depth information. The processing unit 1303 is further configured to determine a second head pose spatial deviation based on the multiple second image frames carrying head depth information according to the following steps: the processing unit 1303 extracts image features from the second reference image frame to obtain a first head pose feature, extracts image features from the second image frame carrying head depth information to obtain multiple second head pose features, extracts image features from the fourth image frame to obtain a third head pose feature, the fourth image frame carrying head depth information of the third image frame; the processing unit 1303 fuses the first head pose feature with the multiple second head pose features to obtain multiple second head pose deviation features; the processing unit 1303 obtains the second head pose spatial deviation feature based on the fourth image frame and the multiple second head pose deviation features.

[0154] In some embodiments, the eye-tracking device 1300 further includes a depth information acquisition unit 1306, which is used to acquire the aforementioned head depth information.

[0155] Figure 14 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. For example... Figure 14 As shown, chip 1400 includes one or more (including two) processors 1401, bus 1402 and communication interface 1403. Optionally, chip 1400 also includes memory 1404.

[0156] In some implementations, memory 1404 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0157] The communication methods implemented by the user equipment, such as the first user equipment, or the network device in any of the embodiments described above in this application can be applied to or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by the integrated logic circuitry in the hardware of the processor 1401 or by instructions in software form. The processor 1401 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. The processor 1401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0158] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 1404, and processor 1401 reads information from memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0159] The processor 1401, memory 1404 and communication interface 1403 can communicate with each other via bus 1402.

[0160] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0161] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the paging method implemented by the first user equipment or the paging method implemented by a non-terrestrial network device according to the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0162] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instructions. When executed by a processor, the computer program or instructions implement the paging method implemented by the first user equipment or the paging method implemented by a non-terrestrial network device. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted on the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0163] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0164] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0165] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0166] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An eye-tracking method, characterized in that, The method is applied to an electronic device, the electronic device including a display screen and a camera, and the method includes: The electronic device displays a first identifier at multiple locations on the display screen and captures multiple first image frames via the camera. The multiple first image frames contain images of the first identifier viewed by the user at different locations on the display screen with the same head posture. The electronic device displays a second identifier at a location on the display screen and captures multiple second image frames, the multiple second image frames containing images of the user viewing the second identifier in different head positions; The electronic device acquires a third image frame through the camera, and then determines the user's first gaze position on the display screen based on the multiple first image frames, the multiple second image frames, and the third image frame.

2. The method according to claim 1, characterized in that, Before the electronic device sequentially displays the first identifier at multiple locations on the display screen, the method further includes: the electronic device receiving a first user operation, the first user operation being used to begin a first calibration of the user's eye features and / or head posture features when viewing different locations on the display screen; Before the electronic device acquires the third image frame via the camera, the method further includes: The electronic device indicates that the first calibration is complete.

3. The method according to claim 1 or 2, characterized in that, Before the electronic device captures multiple first image frames through the camera, the method further includes: the electronic device prompting the user to keep their head fixed and look at the first identifier displayed on the screen; Before the electronic device captures multiple second image frames, the method further includes: the electronic device prompting the user to change their head position to view the second identifier.

4. The method according to any one of claims 1-3, characterized in that, After the electronic device determines the user's first gaze position on the display screen, the method further includes: The electronic device unfolds to display the message at the first gaze location.

5. The method according to any one of claims 1-4, characterized in that, The electronic device determines the user's first gaze position on the display screen based on the plurality of first image frames, the plurality of second image frames, and the third image frame, including: The electronic device determines the fixation point deviation based on a first eye movement deviation, a first head posture spatial deviation, and / or a second head posture spatial deviation, wherein the first eye movement deviation is determined based on the first image frame, the first head posture spatial deviation is determined based on the second image frame without head depth information, and the second head posture spatial deviation is determined based on the second image frame with head depth information. The electronic device determines the first gaze position based on the gaze point deviation and the first calibration position, wherein the first calibration position is the position in the first reference image frame where the electronic device displays the first identifier, the second identifier is displayed at the first calibration position, and the first reference image frame is selected from the plurality of first image frames.

6. The method according to claim 5, characterized in that, The method further includes: determining the first eye movement deviation based on the first image frame according to the following steps: The electronic device extracts image features from the first reference image frame to obtain a first eye feature, extracts image features from the first image frame other than the first reference image frame to obtain multiple second eye features, and extracts image features from the third image frame to obtain a third eye feature; The electronic device fuses the first eye feature with the plurality of second eye features to obtain a plurality of eye deviation features; The electronic device obtains the first eye movement deviation based on the third eye feature and the plurality of eye deviation features.

7. The method according to claim 6, characterized in that, The method further includes: determining the first head pose spatial deviation based on the plurality of second image frames that do not carry head depth information, according to the following steps: The electronic device extracts image features from the second image frame, which does not carry head depth information, to obtain multiple fourth eye features; The electronic device fuses the first eye feature with the plurality of fourth eye features to obtain a plurality of first head posture deviation features; The electronic device obtains a first head posture spatial deviation based on the third eye feature and the plurality of head posture deviation features.

8. The method according to any one of claims 5-7, characterized in that, The plurality of second image frames also includes a second reference image frame carrying head depth information, and the method further includes: The second head pose spatial deviation is determined based on the multiple second image frames carrying head depth information according to the following steps: The electronic device extracts image features from the second reference image frame to obtain a first head pose feature, extracts image features from the second image frame carrying head depth information to obtain multiple second head pose features, and extracts image features from the fourth image frame to obtain a third head pose feature, wherein the fourth image frame carries the head depth information of the third image frame. The electronic device performs feature fusion between the first head pose feature and the plurality of second head pose features to obtain a plurality of second head pose deviation features; The electronic device obtains the second head pose spatial deviation feature based on the fourth image frame and the plurality of second head pose deviation features.

9. The method according to any one of claims 5-8, characterized in that, The head depth information was acquired by a depth sensor.

10. The method according to any one of claims 1-9, characterized in that, The camera is a front-facing camera.

11. An electronic device, characterized in that, The electronic device includes: a display screen, a camera, a memory, and a processor coupled to the memory; the display screen is used to display a user interface, the camera is used to capture image frames, the memory stores a computer program, and the processor executes the computer program to cause the electronic device to implement the method as described in any one of claims 1-10.

12. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed on the processor, they cause the method as described in any one of claims 1-10 to be performed.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-10.

Images