Gaze point estimation method, electronic device and computer readable storage medium

Abstract

The present application relates to the field of image processing, and provides a gaze point estimation method, an electronic device, and a computer-readable storage medium. The gaze point estimation method comprises: when a user is monitored to be gazing at a screen, acquiring a first image of the user's face; acquiring a second image, wherein the second image is an image of the user's face captured when the user is gazing at a first gaze point position on the screen; calculating image distinguishing features between the first image and the second image; and determining the second gaze point position of the user gazing at the screen when the first image is acquired based on the image distinguishing features and the first gaze point position. Through the above method, the influence of the shooting environment on the image quality can be effectively overcome, the error of the gaze point estimation can be effectively reduced, and the stability and reliability of the gaze point estimation result can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a gaze point estimation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Eye control is a new type of human-computer interaction, which is becoming more and more widely used. Eye control includes gaze point control based on electronic devices with screens such as mobile phones and tablets, that is, the gaze point is estimated first, and then corresponding control is performed based on the estimated gaze point position.

[0003] In the related art, the gaze point estimation is realized by collecting images of the user gazing at the screen and performing image recognition on the collected images. Since the shooting environment has a great influence on the image quality of the collected images, the gaze point position estimated by the collected images has a large error, and the stability and reliability of the estimation results are also low. Summary of the invention

[0004] The present application provides a gaze point estimation method, an electronic device and a computer-readable storage medium, which solve the problems in the prior art of large errors, low stability and low reliability of gaze point estimation results.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a method for estimating a gaze point is provided, the method comprising:

[0007] When it is detected that the user is looking at the screen, a first image of the user's face is acquired;

[0008] Acquire a second image, where the second image is a facial image of the user captured when the user is gazing at a first gaze point on the screen;

[0009] calculating image distinguishing features between the first image and the second image;

[0010] The position of a second gaze point at which the user gazes at the screen when acquiring the first image is determined according to the image distinguishing feature and the first gaze point position.

[0011] In the embodiment of the present application, calculating the image distinguishing features between the first image and the second image is equivalent to obtaining the image distinguishing features between the calibration image and the image to be identified. Since the first image and the second image correspond to the same user, the image distinguishing features can represent the difference in the shooting environment of the first image and the second image. Then, the second gaze point position corresponding to the first image is estimated based on the image distinguishing features and the first gaze point position corresponding to the second image, which is equivalent to pre-calibrating a gaze point position (first gaze point position), taking the gaze point position as a reference, and adjusting the gaze point position according to the difference in the shooting environment of the two images, thereby determining the gaze point position corresponding to the image to be identified. Through the method in the embodiment of the present application, the influence of the shooting environment on the image quality can be effectively overcome, and the error of the gaze point estimation can be effectively reduced, thereby effectively improving the stability and reliability of the gaze point estimation result.

[0012] In an implementation of the first aspect, the calculating image distinguishing features between the first image and the second image includes:

[0013] Acquire first feature information of the first image;

[0014] Acquire second feature information of the second image;

[0015] The image distinguishing feature between the first image and the second image is calculated according to the first feature information and the second feature information.

[0016] Through the above method, the correlation features between pixels in the image can be obtained, so that the calculated image distinguishing features can more accurately reflect the difference in shooting environment and shooting content between the two images, thereby providing a reliable data basis for the subsequent determination of the gaze point position. In addition, the above method is equivalent to calculating the image distinguishing features only based on feature information, which is more targeted and less computationally intensive.

[0017] In an implementation manner of the first aspect, the first feature information includes a first plane feature, and the second feature information includes a second plane feature;

[0018] The process of acquiring the first plane feature of the first image includes:

[0019] Acquire eye features of the face in the first image;

[0020] Acquire position features of the face in the first image;

[0021] The first plane feature is calculated according to the eye feature and the position feature.

[0022] In the above method, the eye features and the position features of the face are extracted from the first image, which is equivalent to extracting the local feature information related to the sight line direction, which is more targeted. There is no need to extract features from the entire image, which can effectively reduce the amount of calculation.

[0023] In an implementation of the first aspect, acquiring eye features of the face in the first image includes:

[0024] Acquire a left eye feature of the face in the first image;

[0025] Acquire a right eye feature of the face in the first image;

[0026] The eye feature is calculated according to the left eye feature and the right eye feature.

[0027] When the user looks at the screen, the sight lines of the left and right eyes are different. In the above method, the features of the left and right eyes are extracted separately, which is conducive to the subsequent accurate estimation of the sight lines.

[0028] In an implementation manner of the first aspect, the first feature information includes a first plane feature and a first depth feature, and the second feature information includes a second plane feature and a second depth feature;

[0029] The calculating the image distinguishing feature between the first image and the second image according to the first feature information and the second feature information includes:

[0030] The image distinguishing feature between the first image and the second image is calculated according to the first plane feature, the second plane feature, the first depth feature, and the second depth feature.

[0031] In the embodiment of the present application, the depth feature is added, which is equivalent to taking into account the changes in the user's facial posture. It can not only overcome the influence of the shooting environment on the image quality, but also overcome the influence of the user's own posture changes, effectively reducing the error of the gaze point estimation, thereby effectively improving the stability and reliability of the gaze point estimation results.

[0032] In an implementation manner of the first aspect, calculating the image distinguishing feature between the first image and the second image according to the first plane feature, the second plane feature, the first depth feature, and the second depth feature includes:

[0033] Calculating a first distinguishing feature between the first image and the second image according to the first depth feature and the second depth feature;

[0034] The image distinctive feature between the first image and the second image is calculated based on the first plane feature, the second plane feature and the first distinctive feature.

[0035] In the above implementation, the change factors of the depth features are taken into account in the process of calculating the plane distinguishing features, which is equivalent to focusing on the plane features and taking into account the influence of posture at the same time, and paying more attention to the correlation between the plane features and the depth features, which is beneficial to the subsequent estimation of the gaze point position.

[0036] In an implementation of the first aspect, calculating a first distinguishing feature between the first image and the second image according to the first depth feature and the second depth feature includes:

[0037] Calculating a first variation according to the first depth feature and the second depth feature, wherein the first variation represents a translation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired;

[0038] Calculating a second variation according to the first depth feature and the second depth feature, wherein the second variation represents a rotation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired;

[0039] The first distinguishing feature is calculated according to the first variation and the second variation.

[0040] In the above implementation, not only the translation change but also the rotation change is considered, so that the calculated first distinguishing feature is more accurate.

[0041] In an implementation of the first aspect, determining, according to the image distinguishing feature and the first gaze point position, a second gaze point position at which the user gazes at the screen when acquiring the first image includes:

[0042] Calculating a gaze point deviation according to the image distinguishing feature, wherein the gaze point deviation represents a distance between the first gaze point position and the second gaze point position;

[0043] The second gaze point position is determined according to the gaze point deviation and the first gaze point position.

[0044] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device including a processor, the processor being used to run a computer program stored in a memory to implement a method provided in any possible implementation manner of the first aspect.

[0045] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions, which, when executed on a computer or a processor, enable the computer or the processor to execute a method provided in any possible implementation of the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer or a processor, enables the computer or the processor to execute a method provided in any possible implementation of the first aspect.

[0047] It can be understood that the electronic device described in the second aspect, the computer storage medium described in the third aspect, or the computer program product described in the fourth aspect provided above are all used to execute the method provided in the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of gaze point estimation provided by an embodiment of the present application;

[0049] Figure 2 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0050] Figure 3 It is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of an application interface for selecting an eye control function provided in an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of an application scenario of electronic book reading and page turning provided by an embodiment of the present application;

[0053] Figure 6 is a flow chart of a method for estimating a gaze point provided in an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of an application scenario for capturing a second image provided in an embodiment of the present application;

[0055] Figure 8 is a framework diagram of a gaze point estimation method provided in an embodiment of the present application;

[0056] Fig. 9 It is a framework diagram of a gaze point estimation method provided by another embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0058] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0059] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0060] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0061] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0062] Eye control is a new type of human-computer interaction, and its application is becoming more and more widespread. Eye control includes gaze point control based on electronic devices with screens such as mobile phones, tablets, and smart screens, that is, the gaze point is estimated first, and then the corresponding control is performed based on the estimated gaze point position.

[0063] In some eye movement control application scenarios, gaze point estimation can be used to control the interactive interface. For example, when a user browses a software interface, the user's eye movements can be determined by continuously estimating the user's gaze point, and then the software interface can perform corresponding actions based on the determined eye movements.

[0064] In other eye movement control application scenarios, gaze point estimation can also be used for data mining. For example, when a user browses a software interface, the user's gaze point is estimated to obtain the interface content corresponding to the user's gaze point position, and the interface content is marked as the content that the user is interested in. Then, content related or similar to the interface content is searched and pushed to the user.

[0065] Among them, gaze point estimation refers to estimating the point where a person's binocular sight is focused. Figure 1 As shown, by collecting an image 11 of the user gazing at the screen and performing image recognition on the collected image, the point 12 where the user's binocular sight is focused on the screen is obtained, thereby realizing the estimation of the gaze point. Since the shooting environment has a great influence on the image quality of the image, the error of the gaze point position estimated by the shot image is large, and the stability and reliability of the estimation result are also low.

[0066] Based on this, the embodiments of the present application provide a method and electronic device for estimating the gaze point, which uses the image distinguishing features between the calibration image and the captured image, and the known gaze point position corresponding to the calibration image, to estimate the gaze point position corresponding to the captured image. The method in the embodiments of the present application can effectively overcome the influence of the shooting environment on the image quality, effectively reduce the error of the gaze point estimation, and thus effectively improve the stability and reliability of the gaze point estimation result.

[0067] The gaze point estimation method provided in the embodiment of the present application can be applied to electronic devices with screens and shooting functions. Electronic devices include terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a smart screen, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the electronic device.

[0068] See also Figure 2 , is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.

[0069] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0070] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the gaze point estimation method in the embodiment of the present application.

[0071] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0072] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0073] The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a memory other than the memory of the electronic device and the cache of the processor, and the memory is generally a non-volatile memory.

[0074] The internal memory 121, which may also be referred to as "memory", may be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), and the like.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In some embodiments, the electronic device 100 displays a user interface through the display screen 194.

[0076] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0077] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. Exemplarily, the camera 193 is used to capture an image of a user looking at the display screen 194.

[0079] The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals. Exemplarily, the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The air pressure sensor 180C is used to measure the air pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. The bone conduction sensor 180M can obtain a vibration signal.

[0080] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0081] Exemplarily, in an embodiment of the present application, the touch sensor 180K can detect a user's click operation on an application icon, and transmit the detected click operation to the application processor, determine that the click operation is used to start or run the application, and then execute the running operation of the application.

[0082] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0083] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0084] The above is a specific description of the embodiment of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0085] In addition, an operating system runs on the above components, such as the iOS operating system, the Android open source operating system, and the Windows operating system, etc. Application programs can be installed and run on the operating system.

[0086] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0087] Figure 3 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0088] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library and the Android runtime (Android runtime), and the kernel layer.

[0089] The application layer can include a series of application packages.

[0090] like Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, e-book, shopping, Bluetooth, music, video, short message and other applications. For example: the e-book application may include an eye control function. When an e-book application including an eye control function is installed on an electronic device, the electronic device may monitor the user's gaze point through the eye control function, thereby realizing eye control of turning pages of the e-book. Another example: the shopping application may include an eye control function. When an electronic device includes a shopping application including an eye control function, the electronic device may monitor the user's gaze point through the eye control function, obtain the content that the user is interested in, thereby realizing content recommendation for the user.

[0091] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0092] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0093] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0094] The content provider is used to store and obtain data and make the data accessible to the application. The data may include monitoring data obtained by the sensor module 180 (such as acceleration data obtained by the acceleration sensor 180E), video, image, audio, calls made and received, browsing history and bookmarks, phone book, etc.

[0095] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0096] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0097] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0098] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders (such as travel messages, etc.). The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0099] In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface is used to provide an application programming interface and a programming framework for the camera application.

[0100] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0101] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0102] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system layer may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0104] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0105] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0106] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0107] A 2D graphics engine is a drawing engine for 2D drawings.

[0108] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver. Among them, the camera driver is used to drive the camera to collect images.

[0109] The following introduces the application scenarios of the eye movement control function involved in the embodiments of the present application in combination with the above-mentioned hardware structure and software structure.

[0110] In some application scenarios, the camera application of the electronic device has an eye control function. When the application software installed on the electronic device needs to use the eye control function, the application software calls the camera application. The camera application calls the camera access interface of the application framework layer to start the camera application, so as to capture the image of the user looking at the screen in real time through the camera (such as the front camera), and estimate the gaze point based on the image, obtain the gaze point position of the user's line of sight on the screen, and send the gaze point position to the application software. The application software controls the user interface according to the received gaze point position.

[0111] In other application scenarios, the application software installed on the electronic device has an eye control function. When the application software installed on the electronic device (such as an e-book or shopping software, etc.) turns on the eye control function, the application software calls the camera application. The camera application calls the camera access interface of the application framework layer to start the camera application, thereby capturing an image of the user looking at the screen through a camera (such as a front camera) and sending the captured image to the application software. The application software estimates the gaze point based on the acquired image, obtains the gaze point position of the user's line of sight on the screen, and then controls the user interface based on the gaze point position.

[0112] The eye control function of the application software can be set to be turned on by default. For example, when the application software is opened, the eye control function is turned on.

[0113] The eye control function of the application software can also be manually selected by the user. For example, see Figure 4 , is a schematic diagram of an application interface for selecting an eye control function provided in an embodiment of the present application. Figure 4 As shown, it is a setting interface 40 of the application software. The setting interface 40 may include a selection information area 401 and a selection control 402. Among them:

[0114] Select information area 401 to display the name of the eye control function. Figure 4 As shown in (a) in FIG. 4 , “Eye Control” is displayed in the selection information area 401 . It should be noted that Figure 4 The names of the eye movement control functions shown in are only examples and are not intended to be specific limitations. In actual applications, the names of the eye movement control functions may be defined as other contents as long as they are easy for users to distinguish.

[0115] Selection control 402 is used to implement corresponding software functions when the user clicks / touches it. The selection control corresponds to the above selection information. Figure 4 As shown in (a) of FIG. 1 , the selection information “eye control” corresponds to a selection control. For example, when the user clicks / touches the selection control corresponding to “eye control”, the electronic device 100 displays the following Figure 4The interface shown in (b) in the figure, at this time, the application software turns on the eye movement control function.

[0116] For example, see Figure 5 , is a schematic diagram of an application scenario of electronic book reading and page turning provided by an embodiment of the present application. When the electronic book application software turns on the eye control function, the electronic book application software monitors the image of the user gazing at the screen through the camera application, and then performs image recognition on the image to obtain the user's gaze point position on the screen. Figure 5 Point a is the position of the user's gaze point on the electronic device screen at time t1, and point b is the position of the user's gaze point on the electronic device screen at time t2 (t2 is the time after t1). Then the e-book software determines the user's eye movement through the changes in the gaze point positions corresponding to the previous and next frame images, and performs corresponding control. When it is determined that the user's eye movement is moving up and down, the e-book stays on the current page; when it is determined that the user's eye movement is moving left and right, the e-book page is turned. Figure 5 As shown, the user's gaze point position changes from time t1 to time t2, that is, the gaze point positions corresponding to the previous and next frame images move from point a to point b. From this, it can be determined that the user's eye movement is left and right, and the e-book page turning action is executed.

[0117] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0118] The execution subject of the gaze point estimation method provided in the embodiment of the present application can be the above-mentioned electronic device (such as a mobile phone), or it can be a functional module and / or functional entity in the electronic device that can implement the gaze point estimation method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. Taking an electronic device as an example, the gaze point estimation method provided in the embodiment of the present application is exemplarily described in combination with the accompanying drawings.

[0119] See also Figure 6 , is a flow chart of the method for estimating the gaze point provided in the embodiment of the present application. As an example and not a limitation, Figure 6 As shown, the gaze point estimation method may include the following steps:

[0120] S601, when it is detected that a user is looking at the screen, a first image of the user's face is acquired.

[0121] When the eye control function is turned on, the camera application monitors the screen through the front camera of the electronic device. When a face appears in the screen, the face image is collected and the initial line of sight direction is identified. If the identified line of sight is pointing to the screen, the user is detected to be looking at the screen. If the identified line of sight is not pointing to the screen, the screen continues to be monitored. The monitoring process can continue in the background.

[0122] The process of obtaining the first image of the user's face in step 601 is continuous. When the user is detected looking at the screen, the first image of the user's face can be obtained at a preset monitoring period. For example, when the eye movement function in the application software is turned on, an image is captured by the front camera of the mobile phone at a preset time interval.

[0123] The above-mentioned monitoring period can be a fixed period automatically set by the application software. It can also be automatically selected according to the current state of the electronic device. For example, the application software is set with 3 monitoring periods A, B and C (A>B>C). When the electronic device is currently in low power mode, the application software selects monitoring period C; when the electronic device is currently in high power mode, the application software selects monitoring period A. For another example, when the electronic device is currently in high refresh rate mode, the application software selects monitoring period A; when the electronic device is currently in low refresh rate mode, the application software selects monitoring period C. Of course, the monitoring period can also be manually selected by the user.

[0124] In one implementation, the process of monitoring the user's gaze at the screen and the process of acquiring the first image can be two parallel processing threads. For example, the first thread continuously monitors the image in front of the screen, and when it detects that the user is gazing at the screen, it sends a first signal to the second thread; after receiving the first signal, the second thread acquires the first image at a preset monitoring period; when the first thread detects that the user stops gazing at the screen, it sends a second signal to the second thread; after receiving the second signal, the second thread stops acquiring the first image.

[0125] In another implementation, S601 may be executed by a processing thread. For example, the thread monitors the screen in front of the screen in the first cycle, and when a face appears in the screen, collects the face image, and performs preliminary sight direction recognition on the face image; if the recognized sight direction points to the screen, it is detected that the user is looking at the screen, and the current face image is used as the first image, and then the image in front of the screen is acquired in the second cycle; each time an image is acquired, the sight direction of the image is recognized; if the recognized sight direction points to the screen, the image in front of the screen is acquired in the second cycle; if the recognized sight direction points to the screen, the image in front of the screen is acquired in the first cycle; if the recognized sight direction does not point to the screen, the screen in front of the screen is monitored in the first cycle.

[0126] S602: Acquire a second image, where the second image is a facial image of the user captured when the user gazes at a first gaze point on the screen.

[0127] The second image in this step may be pre-collected and stored. For example, see Figure 7 , is a schematic diagram of an application scenario for capturing a second image provided in an embodiment of the present application.

[0128] In one implementation, a second image may be collected in advance. When the user turns on the eye control function in the application software, the application jumps to the following example: Figure 7 The user interface 70 shown in (a) in FIG. 7 may include an image acquisition area 701, a gaze point 702 located in the image acquisition area, a prompt box 703, and an operation control 704. The user gazes at the gaze point 702 in the image acquisition area according to the prompt content in the prompt box 703, and clicks / touches the operation control 704 during the gaze process. In response to the user clicking / touching the operation control 704, the electronic device 100 captures the user's face image in front of the screen through the front camera. The image can be used as the second image, and the position of the gaze point 702 on the screen is the first gaze point position.

[0129] In another implementation, multiple second images may be collected in advance, and different second images correspond to different gaze point positions. Figure 7 The user interface 71 shown in (b) of FIG. 7 may include an image acquisition area 711, multiple gaze marks 712, a prompt box 713, and an operation control 714. After the user clicks / touches the operation control 714, the electronic device 100 displays the gaze marks 712 (such as Figure 7 The electronic device 100 captures the user's facial image in front of the screen through the front camera. The multiple captured user facial images can be used as the second image, and the position of each gaze mark on the screen is a first gaze point position. In the above manner, whenever a gaze mark is displayed, the electronic device can obtain a second image corresponding to the gaze point position of the gaze mark. In this implementation, multiple gaze marks can be distributed in various directions in the image acquisition area, so that the second image of each direction the user is looking at can be captured.

[0130] It should be noted that Figure 7 What is shown is only an example of capturing the second image, and is not intended to specifically limit the capture interface, image capture area, gaze mark, and the like.

[0131] In practical applications, the second image can be collected when the user turns on the eye control function for the first time, or each time the user turns on the eye control function. Of course, when the eye control function is turned on, the user can also choose to re-collect the second image.

[0132] S603: Calculate image distinguishing features between the first image and the second image.

[0133] In the embodiment of the present application, the image distinguishing feature is used to indicate the difference between the feature information of the image, for example, the RGB information, grayscale information, depth information, key point features, etc. of the image.

[0134] Since the user in the first image and the second image is the same, the distinguishing features of the two images can indicate the difference between the shooting environments of the first image and the second image.

[0135] The implementation method of this step is detailed in the description of the following embodiment and will not be repeated here.

[0136] S604: Determine, according to the image distinguishing feature and the first gaze point position, a second gaze point position at which the user gazes at the screen when acquiring the first image.

[0137] In one implementation of this step, a gaze point deviation is calculated based on image distinguishing features, wherein the gaze point deviation represents the distance between the first gaze point position and the second gaze point position; and the second gaze point position is determined based on the gaze point deviation and the first gaze point position.

[0138] Optionally, the image distinguishing features may be input into a trained recognition model to obtain the gaze point deviation, wherein the recognition model may adopt a neural network or other algorithm models.

[0139] Optionally, the first feature information and the second feature information may be input into a trained detection model to output the gaze point deviation. The detection model may adopt a neural network or other algorithm models.

[0140] The difference between the above two methods is that the input of the recognition model is the image distinguishing feature, and the input of the detection model is the first feature information and the second feature information.

[0141] Figure 6In the gaze point estimation method described in the embodiment, calculating the image distinguishing features between the first image and the second image is equivalent to obtaining the image distinguishing features between the calibration image and the image to be identified. Since the first image and the second image correspond to the same user, the image distinguishing features can represent the difference in the shooting environment of the first image and the second image. Then, the second gaze point position corresponding to the first image is estimated based on the image distinguishing features and the first gaze point position corresponding to the second image, which is equivalent to pre-calibrating a gaze point position (first gaze point position), taking the gaze point position as a reference, and adjusting the gaze point position according to the difference in the shooting environment of the two images, thereby determining the gaze point position corresponding to the image to be identified. Through the method in the embodiment of the present application, the influence of the shooting environment on the image quality can be effectively overcome, and the error of the gaze point estimation can be effectively reduced, thereby effectively improving the stability and reliability of the gaze point estimation result.

[0142] It should be noted that, in S603, each time a first image is acquired, the image distinguishing features between the first image and the second image are calculated.

[0143] As described in S602, if there are multiple second images, each time a first image is obtained, the image distinguishing features between the first image and each second image are calculated. Correspondingly, for S604, a third gaze point position is calculated based on each group of calculated image distinguishing features and its corresponding first gaze point position; then the final second gaze point position is determined based on all the calculated third gaze point positions. For example, the average value of all third gaze point positions can be calculated, and the average value is determined as the second gaze point position. For another example, the median value of all third gaze point positions can be calculated, and the median value is determined as the second gaze point position.

[0144] Since the multiple second images can reflect the gaze point positions corresponding to each line of sight of the user, by comparing the first image with the multiple second images respectively, not only the influence of the shooting environment on the gaze point estimation result can be overcome, but also the influence of the user's own gaze posture on the gaze point estimation result can be overcome, thereby effectively improving the stability and reliability of the gaze point estimation result.

[0145] One implementation of S603 is: calculating a pixel difference between pixels corresponding to the first image and the second image; and using the pixel difference as an image distinguishing feature between the first image and the second image.

[0146] Although this method is simple to calculate, the image distinguishing features obtained can only reflect the RGB difference of a single pixel. If the focus point of a person's binocular vision is detected based on the image, it is also necessary to understand the correlation features between the pixels in the image. However, the above method ignores the correlation features between the pixels in the image.

[0147] In order to solve the above problem, another implementation method of S603 is provided in the embodiment of the present application:

[0148] I. Obtain first feature information of the first image.

[0149] II. Obtain second feature information of the second image.

[0150] III. Calculate image distinguishing features between the first image and the second image based on the first feature information and the second feature information.

[0151] In the embodiment of the present application, the first feature information of the first image and the second feature information of the second image can be obtained by using a trained feature extraction model, wherein the feature extraction model can be a neural network model or other algorithm model.

[0152] In the embodiment of the present application, the first feature information and the second feature information can be input into the trained detection model to output image distinguishing features.

[0153] Through the above method, the correlation features between pixels in the image can be obtained, so that the calculated image distinguishing features can more accurately reflect the difference in shooting environment and shooting content between the two images, thereby providing a reliable data basis for the subsequent determination of the gaze point position. In addition, the above method is equivalent to calculating the image distinguishing features only based on feature information, which is more targeted and less computationally intensive.

[0154] Embodiment 1

[0155] In some embodiments, the first image and the second image may be RGB images. The first feature information includes a first plane feature, and the second feature information includes a second plane feature.

[0156] Optionally, in step I, obtaining the first plane feature of the first image may be taking the first image as a whole to obtain the first plane feature of the first image as a whole.

[0157] This method easily ignores local feature information in the image. To solve the above problem, optionally, the process of obtaining the first plane feature of the first image includes:

[0158] Acquire eye features of the face in the first image;

[0159] Obtaining position features of a face in the first image;

[0160] The first plane feature is calculated according to the eye feature and the position feature.

[0161] In an embodiment of the present application, a method for obtaining eye features includes: performing eye recognition on a first image, obtaining a partial image of the eye in the first image, and then extracting feature information of the partial image of the eye as the eye feature.

[0162] The method of obtaining the position feature includes: dividing the first image into grids to obtain a grid map; identifying a partial image of a face in the grid map; and then calculating the position of the partial image of the face in the grid map as the position feature.

[0163] In one implementation, the local image of the eye and the grid map can be input into different feature extraction models to obtain the eye features of the first image and the position features of the face in the first image. The feature extraction model can be a neural network model or other algorithm model.

[0164] In the embodiment of the present application, the eye feature and the position feature can be spliced ​​into a vector or matrix as the first plane feature. In one implementation, the eye feature and the position feature can be spliced ​​into a plane feature through a fully connected layer in a neural network.

[0165] In the above method, the eye features and the position features of the face are extracted from the first image, which is equivalent to extracting the local feature information related to the sight line direction, which is more targeted. There is no need to extract features from the entire image, which can effectively reduce the amount of calculation.

[0166] Similarly, the process of acquiring the second plane feature of the second image is the same as the process of acquiring the first plane feature of the first image, and will not be repeated here.

[0167] In some implementations, obtaining eye features of the face in the first image includes:

[0168] Obtaining a left eye feature of the face in the first image;

[0169] Acquire the right eye feature of the face in the first image;

[0170] The eye feature is calculated based on the left eye feature and the right eye feature.

[0171] In the embodiment of the present application, the method for obtaining the left eye feature includes: performing left eye recognition on the first image, obtaining a partial image of the left eye in the first image, and then extracting feature information of the partial image of the left eye as the left eye feature. The method for obtaining the right eye feature is the same as the method for obtaining the left eye feature, which will not be repeated here.

[0172] In one implementation, the partial image of the left eye and the partial image of the right eye can be input into different feature extraction models to obtain the left eye feature and the right eye feature. The feature extraction model can be a neural network model or other algorithm model.

[0173] In an embodiment of the present application, the left eye feature and the right eye feature may be concatenated into a vector or matrix as an eye feature. In one implementation, the left eye feature and the right eye feature may be concatenated into an eye feature through a fully connected layer in a neural network.

[0174] When the user looks at the screen, the sight lines of the left and right eyes are different. In the above method, the features of the left and right eyes are extracted separately, which is conducive to the subsequent accurate estimation of the sight lines.

[0175] Similarly, the process of obtaining the eye features of the face in the second image is the same as the process of obtaining the eye features of the face in the first image, and will not be repeated here.

[0176] See also Figure 8 , is a framework diagram of the gaze point estimation method provided in the embodiment of the present application. Figure 8 As shown, for the first image, the left eye feature and the right eye feature of the face in the first image are obtained, and the eye feature in the first image is calculated according to the left eye feature and the right eye feature in the first image; the position feature of the face in the first image is obtained; and the first plane feature of the first image is calculated according to the eye feature and the position feature of the face in the first image. For the second image, the left eye feature and the right eye feature of the face in the second image are obtained, and the eye feature in the second image is calculated according to the left eye feature and the right eye feature in the second image; the position feature of the face in the second image is obtained; and the second plane feature of the second image is calculated according to the eye feature and the position feature of the face in the second image. The image distinguishing feature between the first plane feature and the second plane feature is calculated. The gaze point deviation is calculated according to the image distinguishing feature. Finally, the second gaze point position corresponding to the first image is calculated according to the first gaze point position corresponding to the second image and the gaze point deviation.

[0177] from Figure 8As can be seen in the example, the method for estimating the gaze point provided in the embodiment of the present application uses the second image with a known gaze point position as a calibration image to calculate the image distinguishing features between the first image and the second image, and the image distinguishing features can represent the difference in the shooting environment of the first image and the second image. Then, the second gaze point position corresponding to the first image is estimated based on the image distinguishing features and the first gaze point position corresponding to the second image, which is equivalent to pre-calibrating a gaze point position (first gaze point position), taking the gaze point position as a reference, and adjusting the gaze point position according to the difference in the shooting environment of the two images, thereby determining the gaze point position corresponding to the image to be identified. Through the method in the embodiment of the present application, the influence of the shooting environment on the image quality can be effectively overcome, and the error of the gaze point estimation can be effectively reduced, thereby effectively improving the stability and reliability of the gaze point estimation result. In addition, in the process of calculating the image distinguishing features, the eye features of the left eye and the right eye and the position features of the face are extracted respectively, which is equivalent to fully extracting the local feature information related to the line of sight direction, without the need to extract features for the entire image, which is more targeted and can effectively reduce the amount of calculation.

[0178] In the above embodiment, both the first image and the second image are RGB images. In this case, the image distinguishing features between the first image and the second image can only reflect the RGB distinguishing features of the two. However, in actual applications, the posture of the user when looking at the screen may change, and the RGB distinguishing features cannot reflect the changes in the facial posture in the image, which will affect the final gaze point estimation result.

[0179] Embodiment 2

[0180] In some other embodiments, the first image may include a first RGB image and a first depth image, and similarly, the second image may include a second RGB image and a second depth image. Accordingly, the first feature information includes a first plane feature and a first depth feature, and the second feature information includes a second plane feature and a second depth feature.

[0181] For example, the RGB image can be obtained by taking an RGB camera of the electronic device, and the depth image can be obtained by taking a depth camera of the electronic device. For another example, the depth image can be a three-dimensional point cloud image, which can be obtained by a radar device on the electronic device detecting the front of the screen.

[0182] It is understandable that, in practical applications, the first image may be a first RGB image, and the first image carries the three-dimensional point cloud information corresponding to each pixel in the first RGB image. The second image may be a second RGB image, and the second image carries the three-dimensional point cloud information corresponding to each pixel in the second RGB image.

[0183] Correspondingly, the above step I of acquiring the first feature information of the first image may include the following steps:

[0184] Acquire a first plane feature of a first RGB image;

[0185] A first depth feature of the first depth image is obtained.

[0186] The process of obtaining the first plane feature of the first RGB image is the same as the process of obtaining the first feature information of the first image in the first embodiment, and will not be repeated here.

[0187] The first depth feature may include the three-dimensional coordinates of the pixel point (of course, it may also include only one-dimensional coordinates for indicating the distance from the screen).

[0188] The step of obtaining the second characteristic information of the second image in step II is the same as the step of obtaining the first characteristic information of the first image in step I, and will not be repeated here.

[0189] Correspondingly, step III calculates the image distinguishing features between the first image and the second image according to the first feature information and the second feature information, including:

[0190] An image distinguishing feature between the first image and the second image is calculated according to the first plane feature, the second plane feature, the first depth feature, and the second depth feature.

[0191] In the embodiment of the present application, the depth feature is added, which is equivalent to taking into account the changes in the user's facial posture. It can not only overcome the influence of the shooting environment on the image quality, but also overcome the influence of the user's own posture changes, effectively reducing the error of the gaze point estimation, thereby effectively improving the stability and reliability of the gaze point estimation results.

[0192] In one implementation, a first distinguishing feature between the first image and the second image is calculated based on the first depth feature and the second depth feature; a second distinguishing feature between the first image and the second image is calculated based on the first plane feature and the second plane feature; and an image distinguishing feature between the first image and the second image is calculated based on the first distinguishing feature and the second distinguishing feature.

[0193] Optionally, the image distinguishing features can be calculated by the trained detection model. Specifically: the first depth feature and the second depth feature are input into the first unit of the detection model, and the first distinguishing feature is output; the first plane feature and the second plane feature are input into the second unit of the detection model, and the second distinguishing feature is output; then the first distinguishing feature and the second distinguishing feature are input into the third unit of the detection model, and the image distinguishing feature is output. The detection model can be a neural network model or other algorithm model. When the detection model is a neural network model, the third unit can be a fully connected layer.

[0194] The above implementation method calculates the plane features and the depth features separately, focusing more on the difference between the plane features and the depth features, while ignoring the correlation between the plane features and the depth features.

[0195] In another implementation, a first distinguishing feature between the first image and the second image is calculated based on the first depth feature and the second depth feature; and an image distinguishing feature between the first image and the second image is calculated based on the first plane feature, the second plane feature and the first distinguishing feature.

[0196] Optionally, the first depth feature and the second depth feature are input into a first unit of the detection model to output a first distinguishing feature; the first distinguishing feature, the first plane feature and the second plane feature are input into a second unit of the detection model to output an image distinguishing feature.

[0197] In this implementation, the first plane feature, the second plane feature and the first distinguishing feature are input into the same detection model unit. Compared with the first implementation in which the plane feature and the depth feature are calculated separately, the second implementation takes into account the change factor of the depth feature in the process of calculating the plane distinguishing feature, which is equivalent to taking the plane feature as the main factor and taking into account the influence of the posture at the same time, and paying more attention to the correlation between the plane feature and the depth feature, thereby facilitating the subsequent estimation of the gaze point position.

[0198] In the embodiment of the present application, the calculation method of the first distinguishing feature may include:

[0199] Calculating a first variation according to the first depth feature and the second depth feature, wherein the first variation represents a translation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired;

[0200] Calculating a second variation according to the first depth feature and the second depth feature, wherein the second variation represents a rotation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired;

[0201] The first distinguishing feature is calculated based on the first variation and the second variation.

[0202] As described above, the depth feature may include the three-dimensional coordinates of the pixel point. Exemplarily, the calculation method of the first change amount includes:

[0203] X2=X1+x t ;

[0204] Y2=Y1+y t ;

[0205] Z2=Z1+z t .

[0206] Among them, (X1, Y1, Z1) is the first depth feature, (X2, Y2, Z2) is the second depth feature, T = (x t ,y t , z t ) represents the translation vector or translation matrix, that is, the first change.

[0207] The calculation method of the second change amount includes: P=RP0, wherein P1=(X1, Y1, Z1) is the first depth feature, P2=(X2, Y2, Z2) is the second depth feature, and R is the rotation matrix.

[0208] The point cloud rotates around the x, y, and z axes respectively, and the corresponding rotation angles are α, β, and γ respectively. The calculation method of the rotation matrix of the three rotations is as follows:

[0209]

[0210]

[0211]

[0212] If we follow the internal rotation method, i.e. zyx (first rotate around its own z-axis, then rotate around its own y-axis, and finally rotate around its own x-axis), we can get the rotation matrix R = R x (α)R y (β)R z (γ).

[0213] After obtaining the translation matrix T and the rotation matrix R, the first distinguishing feature is calculated according to the first change amount and the second change amount as follows:

[0214] In the above implementation, not only the translation change but also the rotation change is considered, so that the calculated first distinguishing feature is more accurate.

[0215] See also Fig. 9 , is a framework diagram of a method for estimating a gaze point provided by another embodiment of the present application. Fig. 9As shown, for the first image, the left eye feature and the right eye feature of the face in the first image are obtained, and the eye feature in the first image is calculated according to the left eye feature and the right eye feature in the first image; the position feature of the face in the first image is obtained; the first plane feature of the first image is calculated according to the eye feature and the position feature of the face in the first image; and the first depth feature of the first image is obtained. For the second image, the left eye feature and the right eye feature of the face in the second image are obtained, and the eye feature in the second image is calculated according to the left eye feature and the right eye feature in the second image; the position feature of the face in the second image is obtained; the second plane feature of the second image is calculated according to the eye feature and the position feature of the face in the second image; and the second depth feature of the second image is obtained. The first distinguishing feature between the first depth feature and the second depth feature is calculated. The image distinguishing feature between the first image and the second image is calculated according to the first distinguishing feature, the first plane feature and the second plane feature. The gaze point deviation is calculated according to the image distinguishing feature. Finally, the second gaze point position corresponding to the first image is calculated according to the first gaze point position corresponding to the second image and the gaze point deviation.

[0216] and Figure 8 Compared with the embodiment, Fig. 9 In the embodiment, depth features are added. In the process of calculating the plane distinguishing features, the variation factors of the depth features are taken into account, which is equivalent to taking the plane features as the main factor and taking the influence of the face posture into account at the same time. Therefore, Fig. 9 The method described in the embodiment can not only overcome the influence of the shooting environment on the image quality, but also overcome the influence of the user's own posture changes, effectively reducing the error of the gaze point estimation, thereby effectively improving the stability and reliability of the gaze point estimation result.

[0217] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0218] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0219] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0220] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0221] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the gaze point estimation method in the above-mentioned embodiment.

[0222] This embodiment also provides a computer program product, where the computer-readable storage medium stores program code. When the computer program product runs on a computer, the computer executes the above-mentioned related steps to implement the gaze point estimation method in the above-mentioned embodiment.

[0223] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the gaze point estimation method in the above-mentioned method embodiments.

[0224] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0225] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0226] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for estimating a gaze point, characterized in that: include: When it is detected that the user is looking at the screen, acquiring a first image of the user's face; Acquire a second image, where the second image is a facial image of the user captured when the user is gazing at a first gaze point on the screen; Acquire first feature information of the first image; wherein the first feature information includes a first plane feature and a first depth feature, and the first depth feature includes a three-dimensional coordinate of a pixel point in the first image; Acquire second feature information of the second image; wherein the second feature information includes a second plane feature and a second depth feature, and the second depth feature includes the three-dimensional coordinates of a pixel point in the second image; Calculating a first variation according to the first depth feature and the second depth feature, wherein the first variation represents a translation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired; Calculating a second variation according to the first depth feature and the second depth feature, wherein the second variation represents a rotation variation between a posture of the user's face when the first image is acquired and a posture of the user's face when the second image is acquired; Calculating a first distinguishing feature according to the first variation and the second variation; Calculating an image distinguishing feature between the first image and the second image according to the first plane feature, the second plane feature and the first distinguishing feature; Determining, according to the image distinguishing feature and the first gaze point position, a second gaze point position of the user gazing at the screen when acquiring the first image; The process of acquiring the first plane feature of the first image includes: Acquire left eye features and right eye features of the face in the first image, and calculate eye features of the face in the first image according to the left eye features and the right eye features; Divide the first image into grids to obtain a grid map; identify a partial image of a human face in the grid map; calculate a position of the partial image in the grid map to obtain a position feature of the human face in the first image; Calculate the first plane feature according to the eye feature and the position feature; Wherein, the method further comprises: Monitor the image in front of the screen in the first cycle; When a face appears on the screen, a first face image is collected; Performing sight line direction recognition on the first facial image; If the identified sight line direction points to the screen, it is determined that the user is looking at the screen, and the first face image is used as the first image; Acquire a second facial image of the person in front of the screen in a second cycle; Each time a second facial image is obtained, performing gaze direction recognition on the second facial image; When the identified sight line direction points to the screen, continue to acquire the image of the face in front of the screen in the second cycle; When the identified sight line direction is not pointing to the screen, the image in front of the screen is monitored in the first cycle; Wherein, the method further comprises: When there are multiple second images, determining, according to image distinguishing features between the first image and each of the second images, and a first gaze point position corresponding to each of the second images, a third gaze point position at which the user gazes at the screen when acquiring the first image; The final second gaze point position is determined based on all the calculated third gaze point positions.

2. The method for estimating the gaze point according to claim 1, wherein: The determining, according to the image distinguishing feature and the first gaze point position, a second gaze point position of the user gazing at the screen when acquiring the first image comprises: Calculating a gaze point deviation according to the image distinguishing feature, wherein the gaze point deviation represents a distance between the first gaze point position and the second gaze point position; The second gaze point position is determined according to the gaze point deviation and the first gaze point position.

3. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to run a computer program stored in a memory to implement the method according to claim 1 or 2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to claim 1 or 2 is implemented.

Images