Line of sight estimation method and electronic device

Abstract

The present application discloses a line of sight estimation method and electronic device, which relate to the field of terminals. The method includes: the electronic device collects a first image including a first user's face through a first camera. The electronic device determines the gaze direction of the first user's line of sight in a first coordinate system based on the first image. The electronic device determines the gaze direction of the first user in a second coordinate system based on the gaze direction of the first user's line of sight in the first coordinate system and the position and posture of the first user's head relative to the first camera. The electronic device determines that the first user's line of sight is focused on a first position of the display screen based on the gaze direction of the first user in the second coordinate system and the coordinates of the display screen in the second coordinate system. When the electronic device determines that the first position is located in a preset area of ​​the display screen, the electronic device executes a first instruction.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a line of sight estimation method and electronic equipment. Background Art

[0002] With the development of terminal technology and the maturity of device interaction technology, users have gradually moved away from traditional interaction methods such as mouse and keyboard input or touching the display screen to control electronic devices to execute corresponding instructions, and began to explore more convenient human-computer interaction methods, such as controlling electronic devices based on line of sight interaction, voice interaction or gesture interaction. In the above-mentioned line of sight interaction method, the electronic device can determine the corresponding control instructions based on the user's eye movement to execute the user's intended operation. However, in the current field of line of sight interaction technology, the user's line of sight estimation needs to obtain eye movement information with the help of a head-mounted device. Therefore, the application scenario of this line of sight interaction method is not wide enough, the operation is very cumbersome, and the method of obtaining eye movement information is also relatively complicated. Summary of the invention

[0003] The present application provides a line of sight estimation method and electronic device, which realizes that the electronic device can determine the user's line of sight focus position on the display screen on the electronic device 100 without the help of other devices other than the electronic device (such as the aforementioned head-mounted device). The application scenarios are more extensive and the operation is very convenient, which greatly improves the computing efficiency of the electronic device 100 and saves the computing resources of the electronic device 100.

[0004] In a first aspect, the present application provides a sight estimation method, which is applied to an electronic device including a first camera and a display screen, wherein the first camera and the display screen are located on the same side of the electronic device, and the method includes: the electronic device collects a first image through the first camera, and the first image includes a facial image of a first user. The electronic device determines the gaze direction of the first user's sight in a first coordinate system based on the first image. The first coordinate system is a coordinate system with a preset point of the first user's head as the origin. The electronic device determines the gaze direction of the first user in a second coordinate system based on the gaze direction of the first user's sight in the first coordinate system and the position and posture of the first user's head relative to the first camera. The second coordinate system is a coordinate system with the optical center of the first camera as the origin and the optical axis of the first camera as the Z axis of the coordinate system. The electronic device determines that the first user's sight is focused on a first position of the display screen based on the gaze direction of the first user in the second coordinate system and the coordinates of the display screen in the second coordinate system. When the electronic device determines that the first position is located in a preset area of ​​the display screen, the electronic device executes a first instruction.

[0005] In a possible implementation, the electronic device collects a first image through the first camera, and the first image includes a facial image of the first user, specifically including: the electronic device detects a display state of the display screen. When the electronic device detects that the display screen is in a bright screen state, the electronic device collects a first image through the first camera, and the first image includes a facial image of the first user.

[0006] In a possible implementation, the electronic device determines the gaze direction of the first user's sight line in a first coordinate system based on the first image, specifically including: the electronic device performs distortion correction on the first image. The electronic device extracts the facial key points of the first user based on the undistorted first image. The electronic device determines the head posture of the first user based on the facial key points of the first user. The electronic device performs perspective transformation on the undistorted first image based on the head posture of the first user. The electronic device performs grayscale equalization processing on the first image after perspective transformation to obtain a first standard image. The electronic device extracts the head features and facial features of the first user based on the first standard image. The electronic device determines the gaze direction of the first user's sight line in the first coordinate system based on the head features and facial features of the first user.

[0007] In a possible implementation, the electronic device performs a perspective transformation on the undistorted first image based on the head posture of the first user, specifically including: the electronic device transforms the first image in the second coordinate system to an image in a preset coordinate system based on the head posture of the first user.

[0008] In a possible implementation, the facial feature of the first user indicates the position of the first user's eyes, the open / closed state of the first user's eyes, etc. The head feature of the first user indicates the head rotation posture of the first user.

[0009] In a possible implementation, based on the gaze direction of the first user in the second coordinate system and the coordinates of the display screen in the second coordinate system, determining that the first user's line of sight is focused on the first position of the display screen specifically includes: the electronic device determines the intersection of the gaze direction of the first user in the second coordinate system and the display screen. The electronic device determines the three-dimensional coordinates of the intersection in the second coordinate system. The electronic device determines the two-dimensional coordinates of the intersection in the screen coordinate system based on the three-dimensional coordinates of the intersection in the second coordinate system and the first mapping relationship. The two-dimensional coordinates are the coordinates of the first position, and the first mapping relationship is used to map the coordinates in the second coordinate system to the coordinates in the screen coordinate system.

[0010] In a possible implementation, the screen coordinate system is a coordinate system with the upper left corner of the display screen as the origin and the plane where the display screen is located as the xoy plane.

[0011] In a possible implementation, the first instruction is: displaying a notification interface, opening a specified application, or displaying a desktop interface.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors, one or more memories, and a display screen. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method in any possible implementation of the first aspect.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a method in any possible implementation of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a chip or a chip system, comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to run the code instructions to execute the method in any possible implementation of the first aspect above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product is run on an electronic device, the electronic device executes the method in any possible implementation of the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A A schematic diagram of an application scenario provided for an embodiment of the present application;

[0017] Figure 1B A schematic diagram of a user interface provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a specific flow chart of a line of sight estimation method provided in an embodiment of the present application;

[0019] Figure 3A A schematic diagram of an electronic device configuration provided in an embodiment of the present application;

[0020] Figure 3B A schematic diagram of another application scenario provided for an embodiment of the present application;

[0021] Figure 3CA schematic diagram of a user image provided in an embodiment of the present application;

[0022] Figure 3D A schematic diagram of a process of image standardization processing provided in an embodiment of the present application;

[0023] Figure 3E A schematic diagram of image distortion imaging provided by an embodiment of the present application;

[0024] Figure 3F A schematic diagram of extracting key points of a face provided in an embodiment of the present application;

[0025] Figure 3G A schematic diagram of a process for determining a user's head posture provided in an embodiment of the present application;

[0026] Figure 3H A schematic diagram of a method for establishing a normalized camera coordinate system provided in an embodiment of the present application;

[0027] Fig. 3I A schematic diagram of generating a new image after normalizing an original image provided in an embodiment of the present application;

[0028] Figure 3J A schematic diagram of a process for determining a user's gaze position provided in an embodiment of the present application;

[0029] Figure 3K A coordinate conversion diagram provided in an embodiment of the present application;

[0030] Figure 3L Another coordinate transformation schematic diagram provided for an embodiment of the present application;

[0031] Figure 3M A schematic diagram of an electronic device executing instructions based on a user's gaze position provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a software module of an electronic device provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the attached claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear contrary indication in its context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations including one or more of the listed striking ones. In the embodiments of the present application, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as implying or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can expressly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise stated, the meaning of "multiple" is two or more.

[0035] In some application scenarios, the electronic device may not need to receive the user's touch operation on the display screen and / or the user's input based on the keyboard / mouse, but may determine the corresponding control instruction based on the coordinate position of the user's line of sight focused on the display screen (also referred to as the user's gaze position) to execute the user's intended operation. The above interaction method may be referred to as a line of sight interaction method.

[0036] For example, an electronic device displays a notification interface based on a line of sight interaction, and the applicable scenario of the above line of sight interaction is specifically described. The notification interface refers to a user interface that can display one or more notification messages received by the electronic device. Figure 1AAs shown, the user can wear glasses 110 and look at the display screen of the electronic device 100. The electronic device 100 can display a desktop 1001. One or more application icons and a status bar area 1001A can be displayed in the desktop 1001. Among them, one or more application icons can include a weather application icon, a stock application icon, a calculator application icon, a settings application icon, a mail application icon, a theme application icon, a calendar application icon, and a video application icon, etc. The status bar area 1001A can include one or more signal strength indicators of a mobile communication signal (also known as a cellular signal), a signal strength indicator of a wireless fidelity (wireless fidelity, Wi-Fi) signal, a battery status indicator, a time indicator, etc. Optionally, the desktop 1001 can also include a page indicator and a tray icon area. Among them, the page indicator can be used to indicate the positional relationship between the currently displayed page and other pages. The tray icon area includes a plurality of tray icons (such as a dial application icon, an information application icon, a contact application icon, a camera application icon, etc.), and the tray icon remains displayed when the page is switched. The above page may also include multiple application icons and page indicators. The page indicator may not be a part of the page but may exist independently. The above tray icon is also optional, and the embodiment of the present application does not limit this.

[0037] like Figure 1A As shown, the electronic device 100 (that is, the electronic device mentioned above) can determine the position of the display screen where the user's line of sight is focused through the glasses 110 worn by the user. At this time, the position of the display screen where the user's line of sight is focused is the status bar area 1001A. In response to the line of sight interaction operation, the electronic device 100 can display a notification interface.

[0038] like Figure 1B As shown, the electronic device 100 may display a notification interface 1002. The notification interface 1002 may display one or more notification bars, each of which may display a corresponding notification message. Figure 1B Notification bar 1002A may display notification messages of information applications; notification bar 1002B may display notification messages of weather applications; and notification bar 1002C may display notification messages of system updates.

[0039] However, through the above Figure 1A-1B It can be seen from the scene shown that in the eye-line interaction method shown in this embodiment, the electronic device needs to use a head-mounted device (for example, Figure 1AThe glasses 110 shown in the figure determine the position of the display screen where the user's line of sight is looking, and then the electronic device performs corresponding operations based on the above-determined display screen position. Therefore, the application scenarios of the above-mentioned line of sight interaction method are not wide enough, the operation is very cumbersome, and the way in which the electronic device obtains the user's eye movement information is also relatively complicated.

[0040] Therefore, the embodiment of the present application provides a line of sight estimation method. The method can be applied to electronic devices such as mobile phones and tablet computers. Electronic devices such as mobile phones and tablet computers that implement the above method can be referred to as electronic devices 100. In subsequent embodiments, electronic device 100 will refer to the above electronic devices such as mobile phones and tablet computers.

[0041] Not limited to mobile phones and tablet computers, the electronic device 100 may also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific types of the above-mentioned electronic devices.

[0042] Specifically, in the line of sight estimation method, the electronic device 100 can capture a first image through a camera (for example, a front camera and / or a rear camera, etc.) provided on the electronic device 100. The first image may include a facial image of the first user. The electronic device 100 can determine the line of sight gaze feature of the first user based on the facial image of the first user in the first image. The electronic device 100 can determine the three-dimensional gaze direction vector of the first user based on the line of sight gaze feature. Then, the electronic device 100 can determine that the line of sight of the first user is focused on the first position of the display screen on the electronic device 100 based on the three-dimensional gaze direction vector of the first user. The electronic device 100 can determine the corresponding control instruction based on the first position and then execute the user's intended operation.

[0043] It can be seen from the above process that the line of sight estimation method can determine the user's line of sight focus position on the display screen on the electronic device 100 without the aid of other devices except the electronic device 100 (such as the aforementioned head-mounted device). It has a wider range of application scenarios and is very easy to operate. It greatly improves the computing efficiency of the electronic device 100 and saves the computing resources of the electronic device 100.

[0044] The following describes the specific process of the line of sight estimation method provided in the embodiment of the present application.

[0045] Combination Figure 2 The process diagram shown and Figure 3A-3M The application scenario examples shown specifically illustrate the implementation process of the line of sight estimation method provided by the present application and the application scenarios applicable to the method.

[0046] like Figure 2 As shown, the specific process of the sight line estimation method can be:

[0047] S201. The electronic device 100 detects the display status of the display screen.

[0048] Specifically, the electronic device 100 can obtain the display status of the current display screen through a preset mechanism (for example, a broadcast mechanism, etc.). The display status of the display screen may include an off-screen state, an off-screen AOD (Always on Display) state, and a bright screen state. The bright screen state may include a lock screen interface display state and a bright screen and unlocked state.

[0049] The screen-off state refers to a state in which, when the electronic device 100 is in a display-locked state, the display of the electronic device 100 is in sleep mode and becomes a black screen, and the display screen does not display interface elements, but other devices and programs operate normally; the AOD state refers to a state in which, when the electronic device 100 is in a display-locked state, a portion of the display screen on the electronic device 100 remains always on to display time, notifications and other information; the lock screen interface display state refers to the user interface of the electronic device 100 when it is in a lock screen state; the screen-on and unlocked state refers to a state in which the electronic device 100 can receive user input based on the interface displayed in this state, and run one or more applications in response to the input, which is the unlocked state of the electronic device referred to in the prior art.

[0050] When the display screen of the electronic device 100 is in the off state / AOD state, the electronic device 100 can receive and respond to the user's input to light up the display screen, and display the lock screen interface. Among them, the lock screen interface may refer to an interface in which some functions of the electronic device 100 are locked, that is, the electronic device 100 does not provide some functions. The functions provided by the electronic device 100 under the lock screen interface have lower requirements for data security. Exemplarily, the functions provided by the electronic device 100 under the lock screen interface may include: answering calls, hanging up calls, adjusting music volume, starting the camera application, turning on / off airplane mode, etc.

[0051] When the display screen of the electronic device 100 is in the lock screen interface display state, the electronic device 100 can verify the identity of the user through a biometric authentication method to unlock the display screen, so that the display screen is in a bright screen and unlocked state. Among them, the biometric authentication method can include one or more of password authentication, face authentication, fingerprint authentication, iris authentication, voiceprint authentication, etc. Bright screen can refer to the state in which all areas of the display screen on the electronic device 100 are lit up.

[0052] In some examples, when the display screen of the electronic device 100 is in the off state / AOD state, the electronic device 100 can verify the identity of the user through biometric authentication to unlock the display screen, so that the display screen is in the on and unlocked state. That is to say, the present application does not limit the switching between the display states of the above display screen.

[0053] S202. When the electronic device 100 detects that the display screen is in the on-screen state, the electronic device 100 captures a first image including a facial image of a first user through the first camera.

[0054] Among them, the screen-on status can refer to the instructions in the previous steps and will not be repeated here.

[0055] In some examples, the first camera may be a front camera of the electronic device 100. Figure 1B As shown, the first camera may be the front camera 120 on the electronic device 100. The electronic device 100 may capture the first image through the front camera 120 and implement the sight estimation method provided in the present application to determine that the sight of the first user is focused on the display screen on the same side as the front camera 120 (for example, Figure 1B The first position on the display screen of the notification interface 1002 is displayed in the display screen. Then, the electronic device 100 can execute the corresponding control instruction based on the first position.

[0056] In some examples, the first camera may be a rear camera of the electronic device 100. For example, Figure 3A As shown, the electronic device 100 may be configured with a display screen 130 at the back cover. A user interface 1003 may be displayed on the display screen 130. The user interface 1003 may include one or more signal strength indicators of mobile communication signals, signal strength indicators of Wi-Fi signals, battery status indicators, and time indication information. In this example, the first camera may be the rear camera 140. The electronic device 100 may capture the first image through the rear camera 140 and implement the line of sight estimation method provided in the present application to determine that the line of sight of the first user is focused on the first position on the display screen 130 on the same side as the rear camera 140. Then, the electronic device 100 may execute the corresponding control instruction based on the first position.

[0057] That is to say, the embodiment of the present application does not limit the setting position of the first camera on the electronic device 100.

[0058] For example, Figure 3B The implementation method of this step is explained by taking the example of the first user looking at the electronic device 100.

[0059] like Figure 3B As shown, in this application scenario, the first user can look at the electronic device 100. Among them, the electronic device 100 is equipped with a front camera 120 (that is, the first camera). At this time, the electronic device 100 can display the desktop 1001, and the display screen of the electronic device 100 is in a bright screen state. For the description of the desktop 1001, please refer to the above description and will not be repeated here. The electronic device 100 can detect that the display screen is in a bright screen state through a broadcast mechanism, and then the electronic device 100 can capture the first image through the front camera 120.

[0060] like Figure 3C As shown, the first image may include a facial image area 1004 of the first user.

[0061] It can be seen from the above process that when the electronic device 100 determines that the display screen is in the on-screen state, the electronic device 100 will capture the first image including the first user's facial image through the first camera. In this way, the electronic device 100 will not frequently enable the first camera to capture images, so the first camera will not always be in a working state, which can avoid excessive power consumption and privacy security issues.

[0062] S203. The electronic device 100 performs standardization processing on the first image to obtain a first standard image.

[0063] Specifically, the specific implementation process of the electronic device 100 performing standardization processing on the first image to obtain the first standard image can be as follows: Figure 3D As shown:

[0064] 1). Optionally, the electronic device 100 performs distortion calibration on the distorted first image to obtain a distortion-free first image.

[0065] In some examples, when the electronic device 100 captures a designated image (e.g., a first image) for a designated object (e.g., a first user) through the first camera, due to the manufacturing accuracy of the optical lens on the first camera and the deviation of the assembly process, the imaging of the designated object on the designated image is often distorted, resulting in distortion of the designated image. This phenomenon can be referred to as distortion of the designated image. For example, Figure 3EAs shown, there is a rectangular object (a) in the real world. The electronic device 100 can capture a designated image 1 including the rectangular object (a) through the first camera. Based on the above reasons, the designated image 1 captured by the electronic device 100 through the first camera for the rectangular object (a) can be as follows: Figure 3E The concave distortion imaging shown in (b) or, Figure 3E The convex distortion imaging shown in (c) above. The above-mentioned concave distortion imaging and convex distortion imaging are both distortions of the specified image 1. The distortion of the first image can be similar to the above example. The distortion of the specified image can also include other types, which can be specifically referred to the prior art and will not be repeated here.

[0066] It can be seen from the above description that when the first image is distorted to the point where the first user is distorted on the first image, the user gaze position determined by the electronic device 100 based on the distorted first image is not accurate enough, which results in a large deviation between the first position determined by the electronic device 100 and the display screen position actually gazed by the first user. Therefore, in order to improve the calculation accuracy of the first position so that the electronic device 100 can more accurately perform the intended operation of the first user, the electronic device 100 can perform distortion correction on the distorted first image to obtain a distortion-free first image.

[0067] Specifically, the electronic device 100 can obtain the distortion coefficient of the first camera through the checkerboard calibration method or other methods. That is to say, the present application does not limit the method of obtaining the distortion coefficient of the first camera. Then, the electronic device 100 can map each coordinate on the distorted first image to the corresponding standard coordinate through the distortion coefficient of the first camera, thereby obtaining the first image without distortion. For example, the multiple coordinate points on the distorted first image include the coordinate point (u1, v1) and the coordinate point (u2, v2). The electronic device 100 can map the coordinate point (u1, v1) to the corresponding standard coordinate point (u3, v3) and the coordinate point (u2, v2) to the corresponding standard coordinate point (u4, v4) through the distortion coefficient of the first camera. The other coordinate points are similarly mapped to the corresponding standard coordinate points. The set of the above standard coordinate points can constitute the first image without distortion, that is, the electronic device 100 obtains the first image without distortion.

[0068] In some examples, all parameter values ​​in the distortion coefficient of the first camera are 0, that is, the first image captured by the electronic device 100 through the first camera is not distorted, and the electronic device 100 can directly capture the undistorted first image based on the first camera. Therefore, the electronic device 100 may not perform the above-mentioned distortion correction step, and the electronic device 100 may perform processing based on the undistorted first image in the subsequent process.

[0069] 2). The electronic device 100 extracts key points of the first user's face based on the undistorted first image.

[0070] Specifically, the electronic device 100 can extract the facial key points of the first user based on the undistorted first image obtained in the aforementioned step 1), or the undistorted first image acquired by the first camera, based on a general face 3D model key point detection algorithm (for example, a general 6-point face 3D model key point detection algorithm, a general 48-point face 3D model key point detection algorithm, etc.) / a face key point recognition algorithm based on Kinect. The present application does not limit the method for extracting facial key points. Among them, the facial key points of the first user may include: key points of one or more parts such as the left eye, the right eye, the nose, the left lip corner, the right lip corner, and the facial contour.

[0071] For example, the electronic device 100 may be based on Figure 3C The first image shown in FIG. 1 is used to extract the facial key points of the first user according to the general 6-point facial 3D model key point detection algorithm. Figure 3F As shown, the facial key points of the first user may include: left eye a, right eye b, nose c, left lip corner d, right lip corner e, and jaw point f, and the two-dimensional coordinates of each key point are determined.

[0072] 3). The electronic device 100 determines the head posture of the first user based on the facial key points of the first user.

[0073] Among them, the position and posture are the position and posture, and the head posture of the first user refers to: the position and posture of the first user's head relative to the first camera, that is, the position and orientation of the first user's head relative to the first camera. The electronic device 100 can show the head posture of the first user through parameters representing the spatial position relationship and parameters representing the spatial posture relationship.

[0074] Specifically, Figure 3G The example shows the implementation process of the electronic device 100 determining the head posture of the first user:

[0075] a. The electronic device 100 determines the internal parameters of the first camera.

[0076] The electronic device 100 can determine the internal parameters of the first camera (for example, the focal length and pixel size of the first camera) by a checkerboard calibration method or other methods. That is to say, the present application does not limit the method of obtaining the internal parameters of the first camera. Among them, the internal parameters of the first camera can be used to map the three-dimensional coordinates of a certain three-dimensional point in the camera coordinate system of the first camera to two-dimensional coordinates on the first image. The camera coordinate system of the first camera (also referred to as the second coordinate system) refers to a coordinate system with the optical center of the first camera as the origin and the optical axis of the first camera as the Z axis of the coordinate system.

[0077] b. The electronic device 100 obtains the two-dimensional coordinates of each of the key points of the first user's face on the first image.

[0078] c. The electronic device 100 obtains the three-dimensional coordinates of each key point of the first user's face.

[0079] The electronic device 100 can obtain the three-dimensional coordinates of each point in the key points of the first user's face in the world coordinate system based on the preset origin. The above-mentioned world coordinate system refers to a coordinate system with a specified point as the preset origin. Exemplarily, in the embodiment of the present application, the preset origin of the world coordinate system can be the center of the right eye of the first user, the X-axis is the line connecting the two eyes and points from the left eye to the right eye, the Y-axis is perpendicular to the Z-axis and points downward in the face plane, and the Z-axis is determined according to the right-hand rule of spatial coordinates. The world coordinate system can also be the head coordinate system in subsequent embodiments.

[0080] d. The electronic device 100 determines the head posture of the first user.

[0081] Specifically, the electronic device 100 can determine the translation vector t and the rotation matrix Rr based on the internal parameters of the first camera, the two-dimensional coordinates of each point in the facial key points of the first user on the first image, and the three-dimensional coordinates of each point in the facial key points of the first user. The translation vector t is the aforementioned parameter used to represent the spatial position relationship, which can be used to characterize the position of the first user's head relative to the first camera. The rotation matrix Rr is the aforementioned parameter used to represent the spatial posture relationship, which can be used to characterize the orientation of the first user's head relative to the first camera.

[0082] In some examples, the electronic device 100 may also determine the head posture of the first user through a deep learning algorithm. That is, the present application does not limit the manner in which the electronic device 100 determines the head posture of the first user.

[0083] 4). The electronic device 100 performs perspective transformation on the first image without distortion.

[0084] The electronic device 100 performs perspective transformation on the first image without distortion, including: the electronic device 100 performs rotation processing and scaling processing on the first image without distortion, so that the first image without distortion is transformed into a normalized image captured by a normalized camera. The parameters of the normalized camera (e.g., posture, focal length, and distance from the photographed object, etc.) are pre-set fixed values. In other words, the electronic device 100 performs perspective transformation on the first image without distortion, that is, the angle, position, and distance of the head of the first user on the first image from the camera meet the preset standards. In addition, the normalized image satisfies the following three conditions: a. The center of the eye of the first user on the normalized image is the center of the image. b. The image of a straight line on the X-axis of the head coordinate system on the normalized image is still a straight line. c. The normalized camera is located at a preset fixed distance (expressed as dn) from the center of the eye.

[0085] In this way, the computational complexity caused by different head postures of the user can be eliminated, the computational efficiency of the electronic device 100 can be improved, and the computational accuracy of the electronic device 100 can also be improved.

[0086] Specifically, first, the electronic device 100 can establish a normalized camera coordinate system based on the position and posture of the first user's head relative to the first camera and the camera coordinate system of the first camera. The normalized camera coordinate system (also referred to as a preset coordinate system) is a coordinate system with the optical center of the normalized camera as the origin and the optical axis of the normalized camera as the Z axis of the coordinate system. Exemplarily, the establishment of the normalized camera coordinate system and the relationship between the normalized camera coordinate system and the head coordinate system can be as follows: Figure 3H As shown. The normalized camera coordinate system can be established as follows: the axis where the aforementioned translation vector t is located is the Zc axis of the normalized camera coordinate system; the axis where the vector on the X axis in the head coordinate system is obtained by cross-producting the vector on the Zc axis of the normalized camera coordinate system is the Yc axis of the normalized camera coordinate system; the axis where the vector on the Zc axis of the normalized camera coordinate system is obtained by cross-producting the vector on the Yc axis is the Xc axis of the normalized camera coordinate system. Based on this step, the rotation matrix R of the normalized camera coordinate system can be obtained, which is recorded as:

[0087]

[0088] in, and They are respectively the vectors on the Xc axis, Yc axis and Zc axis in the normalized camera coordinate system. and is the modulus of the above vectors.

[0089] Then, the Zc axis of the normalized camera coordinate system is scaled to a preset fixed value dn. Based on this step, the scaling matrix S of the normalized camera coordinate system can be obtained. The scaling matrix is ​​a diagonal matrix and is recorded as:

[0090]

[0091] Among them, dn is the aforementioned preset fixed value dn, and t is the aforementioned translation vector.

[0092] Therefore, the transformation matrix from the first image to the normalized image is M = SR. Based on the transformation matrix M, the electronic device 100 can transform the first image into a normalized image in the normalized camera coordinate system.

[0093] For example, Fig. 3I As shown, Figure 3C Taking the first image shown as an example, the electronic device 100 can convert the first image into a normalized image d in the normalized camera coordinate system based on the aforementioned transformation matrix M. It can be seen that the head state of the first user in the first image before the conversion is tilted, and the head state of the first user in the normalized image d after the conversion is upright.

[0094] 5). The electronic device 100 performs equalization processing based on the first image obtained in step 4) to obtain a first standard image.

[0095] In order to make the facial image of the first user appear more clearly on the first image, thereby improving the prediction accuracy of the subsequent user's gaze position, the electronic device 100 can perform equalization processing on the first image obtained in step 4). The equalization of the image means that the electronic device 100 adjusts the contrast of the first image based on the gray value distribution histogram of the first image.

[0096] Specifically, the electronic device 100 performs equalization processing on the first image through a histogram equalization algorithm. First, the electronic device 100 can normalize the grayscale values ​​in the first image and map the grayscale value range from [0,255] to [0,1]. Then, the electronic device 100 can count the number of occurrences of each grayscale value in the first image (that is, how many pixels each grayscale value includes) to obtain a grayscale value distribution histogram of the first image. Then, the electronic device 100 can calculate the probability of each grayscale value and calculate the cumulative probability density function based on the probability of each grayscale value. The electronic device 100 can re-divide the number of pixels included in each grayscale value based on the cumulative probability density function. Without limitation to this, the electronic device 100 can also perform equalization processing on the first image by other methods, and the present application does not limit this.

[0097] In this way, the electronic device 100 performs standardization processing on the first image through step S203. After acquiring the first standard image, the electronic device 100 can perform subsequent processing based on the first standard image to reduce the computational complexity of the electronic device 100 and improve the computational efficiency of the electronic device 100.

[0098] S204. The electronic device 100 determines the gaze feature of the first user based on the first standard image.

[0099] Among them, the electronic device 100 can determine the sight gaze feature of the first user from the first standard image through a feature extraction network composed of a neural network structure. Among them, the neural network structure can be a recurrent neural network (RNN), a deep neural network (DNN), a dual-path recurrent neural network (DPRNN), etc. The feature extraction network may include a lightweight encoding network and a lightweight decoding network, and the lightweight encoding network and the lightweight decoding network can be used for encoding and decoding of vectors. Among them, the "lightweight" in the lightweight encoding network and the lightweight decoding network can refer to: the network contains fewer convolutional layers (for example, 5 layers, 6 layers, etc.) containing parameter values ​​(for example, weight values ​​and / or bias values), and the amount of calculation is small.

[0100] Specifically, the electronic device 100 can extract the facial features (also referred to as Appearance features) and head features (also referred to as Head features) of the first user after the rotation transformation in step 4) in the aforementioned S203 from the first standard image through the feature extraction network. Among them, the Appearance feature can be represented by a 64-dimensional vector, which characterizes the facial features of the first user, such as the position of the first user's eyes, the open / closed state of the first user's eyes, etc.; the Head feature can be represented by the Euler angle of a 32-dimensional vector, which characterizes the head rotation posture of the first user. The Euler angle can include a pitch parameter representing the angle between the first user and the horizontal X-axis, and a yall parameter representing the angle between the first user's head and the vertical Y-axis. It should be noted that, since the roll parameter representing the angle between the first user's head and the Z-axis does not affect the extraction of subsequent sight gaze features, after the rotation transformation in step 4) in the aforementioned S203, the roll parameter of the angle between the first user's head and the Z-axis has been eliminated, reducing the amount of calculation of subsequent steps of the electronic device 100 and improving the calculation efficiency of the electronic device 100. Then, the electronic device 100 can determine the first user's line of sight gaze feature (also referred to as Gaze feature) through a feature extraction network based on the first user's Appearance feature and the first user's Head feature. The Gaze feature can be represented by an Euler angle, which characterizes the gaze direction of the first user in the first coordinate system. Among them, the first coordinate system is a coordinate system with a preset point on the first user's head (for example, the tip of the first user's nose, the center of the first user's binocular connection, etc.) as the origin, and the Euler angle may include a pitch parameter of the angle between the first user and the horizontal X-axis, and a yall parameter of the angle between the first user and the vertical Y-axis.

[0101] S205. The electronic device 100 determines a three-dimensional gaze direction vector of the first user based on the gaze characteristics of the first user.

[0102] Specifically, the electronic device 100 can determine the three-dimensional gaze direction vector of the first user based on the first user's line of sight gaze features and the first user's head features through the aforementioned feature extraction network. The three-dimensional gaze direction vector in this step can be used to characterize the gaze direction of the first user in the normalized camera coordinate system. In some examples, the electronic device 100 normalizes the three-dimensional gaze direction vector, that is, divides the parameters in the three-dimensional gaze direction vector by its own modulus, so as to reduce the subsequent computational complexity of the electronic device 100 and improve the computational efficiency of the electronic device 100.

[0103] It should be noted that before implementing the line of sight estimation method provided by the present application, the developer can construct and train the line of sight estimation network. In the first step, the developer can obtain images of multiple users, implement the above steps through a third-party electronic device, and determine the three-dimensional gaze direction vector of each user. In the second step, the third-party electronic device can obtain the three-dimensional gaze direction vector (also referred to as the real three-dimensional gaze direction vector) corresponding to the real line of sight of each user. Then, the third-party electronic device can calculate the difference between the three-dimensional gaze direction vector determined above and the real three-dimensional gaze direction vector through the loss function. The loss function can be preset by the developer. Exemplarily, the loss function can be a superposition of L1Res Loss, Embed Loss, and Across Loss, or it can be other loss functions, which is not limited by the present application. In the third step, the third-party electronic device adjusts the weight value and / or bias value in the feature extraction network to maximize the loss function (for example, to maximize or minimize the loss function). In the fourth step, the third-party electronic device can preset the feature extraction network that maximizes the loss function into the electronic device 100, so that the electronic device 100 can implement the line of sight estimation method provided by the present application.

[0104] S206 . The electronic device 100 determines, based on the three-dimensional gaze direction vector of the first user, that the line of sight of the first user is focused on a first position of the display screen on the electronic device 100 .

[0105] Specifically, the process of the electronic device 100 implementing this step can be as follows: Figure 3J As shown:

[0106] a). Convert the three-dimensional gaze direction vector obtained in step S205 to the camera coordinate system of the first camera.

[0107] Specifically, since the three-dimensional gaze direction vector obtained in step S205 is obtained based on the first standard image, the three-dimensional gaze direction vector is a vector representation in the normalized camera coordinate system. The electronic device 100 can perform an inverse transformation based on the aforementioned rotation matrix R to convert the three-dimensional gaze direction vector to the camera coordinate system of the first camera. Among them, the three-dimensional gaze direction vector in this step can be used to characterize the gaze direction of the first user in the camera coordinate system of the first camera. The rotation matrix R is related to the position and posture of the first user's head relative to the first camera.

[0108] b) In the camera coordinate system of the first camera, determine the intersection point of the three-dimensional gaze direction vector and the display screen.

[0109] Specifically, the electronic device 100 can obtain the coordinates of each point on the display screen of the electronic device 100 in the camera coordinate system of the first camera. Then, the electronic device 100 can determine the intersection of the three-dimensional gaze direction vector and the display screen, and obtain the three-dimensional coordinates of the intersection in the camera coordinate system of the first camera.

[0110] Example type, such as Figure 3K As shown above Figure 3B Taking the application scenario shown as an example, after processing based on the above steps, the three-dimensional gaze direction vector G of the first user can be obtained. The electronic device 100 can obtain the coordinates of each point in the display screen of the electronic device 100 under the camera coordinate system (CCS) of the front camera 120. Then, the electronic device 100 can determine that the intersection of the three-dimensional gaze direction vector G and the display screen is p. In the camera coordinate system of the front camera 120, the three-dimensional coordinates of the intersection p are (x1, y1, z1). In this example, the camera coordinate system of the front camera 120 is also the camera coordinate system of the first camera (also referred to as the second coordinate system).

[0111] c). Calculate the two-dimensional coordinates of the intersection point in the screen coordinate system.

[0112] Specifically, the screen coordinate system is a coordinate system with the upper left corner of the display screen as the origin and the plane where the display screen is located as the xoy plane. The coordinates of each point on the screen coordinate system are expressed based on pixels. The screen coordinate system and the camera coordinate system of the first camera can be converted to each other through the rotation matrix Rs and the translation matrix Ts and their inverse matrices. That is to say, in the screen coordinate system, the electronic device 100 can convert the two-dimensional coordinates of a specified point on the display screen into the three-dimensional coordinates in the camera coordinate system of the first camera through the rotation matrix Rs and the translation matrix Ts; similarly, the electronic device 100 can also convert the three-dimensional coordinates of the specified point in the camera coordinate system of the first camera into the two-dimensional coordinates in the screen coordinate system through the inverse matrix of the rotation matrix Rs and the inverse matrix of the translation matrix Ts. Among them, the rotation matrix Rs and the translation matrix Ts can be preset by the electronic device 100. The inverse matrix of the rotation matrix Rs and the inverse matrix of the translation matrix Ts can be called the first mapping relationship.

[0113] For example, Figure 3L As shown, Figure 3KTaking the scene shown as an example, the screen coordinate system (SCS) and the camera coordinate system of the front camera 120 can be converted to each other through the rotation matrix Rs and the translation matrix Ts and their inverse matrices. The intersection point of the three-dimensional gaze direction vector G of the first user obtained in the above step b) and the display screen is p, and its three-dimensional coordinates in the camera coordinate system of the front camera 120 are (x1, y1, z1). The electronic device 100 can convert the coordinates (x1, y1, z1) into two-dimensional coordinates (u, v) in the screen coordinate system based on the inverse matrix of the rotation matrix Rs and the inverse matrix of the translation matrix Ts. The position represented by the two-dimensional coordinates (u, v) is the first position.

[0114] S207. When the electronic device 100 determines that the first position is located in a preset area of ​​the display screen, the electronic device 100 executes a first instruction.

[0115] For example, Figure 3M As shown above Figure 3B Taking the application scenario shown as an example, after the processing of step S201 to step S207, the electronic device 100 determines that the first user is looking at the first position p(u,v) on the display screen, and the first position p is in the status bar area (that is, the preset area) on the desktop 1001. Therefore, the electronic device 100 can execute the first instruction based on the first position, that is, display the notification interface 1002. For the description of the notification interface 1002, please refer to the above description, which will not be repeated here.

[0116] In some examples, when the first user is using the mobile phone, the first user's line of sight can be directed to any place on the display screen of the electronic device 100 at any time. Therefore, in the process of using the mobile phone, the first user's line of sight may accidentally scan the first position. If the electronic device 100 still executes the first instruction corresponding to the first position in this case, the operation triggered is very likely not the user's intended operation, thereby causing a false trigger and reducing the user's experience.

[0117] Therefore, after the electronic device 100 captures the first image through the first camera, it captures the second image including the facial image of the first user again after a specified time period 1 (for example, 1 second, 2 seconds, etc.). The specified time period 1 is greater than or equal to the preset time period 1. Similar to the processing of the first image, the electronic device 100 can process the second image through the aforementioned steps S201-S207. If the gaze position of the first user determined by the electronic device 100 based on the first image and the gaze position of the first user determined by the electronic device 100 based on the second image are both the first position on the display screen, and the first position is located in the preset area of ​​the display screen, the electronic device 100 can determine that the time the first user gazes at the first position is greater than or equal to the preset time period 1, and the first user wants the electronic device 100 to execute the first instruction. Therefore, the electronic device 100 can execute the first instruction based on the first position to realize the intention of the first user.

[0118] In some examples, the first position may also be other preset areas other than the status bar area. The first instruction executed by the electronic device 100 based on the first position may also be other operations other than displaying the notification interface, such as opening a specified application, displaying a desktop interface, etc. This application does not limit this.

[0119] It can be seen from the implementation process of the above-mentioned line of sight estimation method that the method does not require the use of other equipment besides the electronic device 100 (such as the aforementioned head-mounted device), has a wider range of application scenarios, and is very convenient to operate; it does not require understanding of the structure of the human eyeball, and is simple to implement; combined with the posture of the first user's head and eyes, the extracted features are more targeted and accurate; at the same time, the image captured by the first camera is standardized, which reduces the computational complexity of the electronic device 100, improves the computational efficiency of the electronic device 100, and saves computing resources of the electronic device 100.

[0120] Next, a software module applied to the electronic device 100 provided in an embodiment of the present application is introduced.

[0121] Please refer to Figure 4 , Figure 4 The schematic diagram of a software module applied to the electronic device 100 is exemplarily shown. The software module can be used to implement a sight line estimation method provided by the present application.

[0122] like Figure 4 As shown, the software modules on the electronic device 100 may include: an image processing module 401, a feature extraction module 402, a sight estimation module 403 and an instruction execution module 404. Among them:

[0123] The image processing module 401 can be used to perform standardization processing on the first image collected by the first camera to obtain a first standard image. Figure 2 The illustrated embodiments will not be described in detail here.

[0124] The feature extraction module 402 can be used to extract the appearance feature, head feature and gaze feature of the first user based on the first standard image. Figure 2 The illustrated embodiments will not be described in detail here.

[0125] The sight estimation module 403 can be used to determine the three-dimensional gaze direction vector of the first user based on the Gaze feature, thereby determining that the first user's sight is focused on the first position on the display screen of the electronic device 100. The specific implementation process can refer to the aforementioned Figure 2 The illustrated embodiments will not be described in detail here.

[0126] The instruction execution module 404 can be used to, when the electronic device 100 determines that the first user is looking at the first position on the display screen, the electronic device 100 executes the first instruction. Figure 2 The illustrated embodiments will not be described in detail here.

[0127] Next, the hardware structure of an electronic device 100 provided in an embodiment of the present application is introduced.

[0128] Please refer to Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device 100 is exemplarily shown.

[0129] The electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0130] like Figure 5As shown, the electronic device 100 may include a processor 501 , a memory 502 , a wireless communication module 503 , a display screen 504 , a camera 505 , an audio module 506 and a speaker 507 .

[0131] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0132] The processor 501 may include one or more processor units, for example, the processor 501 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0133] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0134] The processor 501 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 501 is a cache memory. The memory may store instructions or data that the processor 501 has just used or circulated. If the processor 501 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 501, and thus improves the efficiency of the system.

[0135] In some embodiments, the processor 501 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.

[0136] The memory 502 is coupled to the processor 501 and is used to store various software programs and / or multiple groups of instructions. In a specific implementation, the memory 502 may include a volatile memory, such as a random access memory (RAM); it may also include a non-volatile memory, such as a ROM, a flash memory, a hard disk drive (HDD) or a solid state drive (SSD); the memory 502 may also include a combination of the above-mentioned types of memories. The memory 502 may also store some program codes so that the processor 501 can call the program codes stored in the memory 502 to implement the implementation method of the embodiment of the present application in the electronic device 100. The memory 502 may store an operating system, such as an embedded operating system such as uCOS, VxWorks, RTLinux, etc.

[0137] In the embodiment of the present application, the processor 501 can be used to: perform standardization processing on the collected first image to obtain a first standard image; extract the gaze feature of the first user based on the first standard image, and determine the first position where the first user is looking at on the display screen 504 according to the gaze feature of the first user; and execute the corresponding first instruction based on the first position. The specific implementation method can refer to the aforementioned Figure 2 The embodiments shown will not be described in detail here.

[0138] The wireless communication module 503 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 503 can be one or more devices integrating at least one communication processing module. The wireless communication module 503 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 501. The wireless communication module 503 can also receive signals to be sent from the processor 501, modulate and amplify them, and convert them into electromagnetic waves for radiation through the antenna. In some embodiments, the electronic device 100 can also communicate with the processor 501 through the Bluetooth module ( Figure 5 Not shown), WLAN module ( Figure 5 The WLAN module may provide a solution including one or more Bluetooth communication in basic rate / enhanced data rate (BR / EDR) or Bluetooth low energy (BLE), and a solution including one or more WLAN communication in Wi-Fi direct, Wi-Fi LAN or Wi-Fi softAP.

[0139] The display screen 504 can be used to display images, videos, etc. The display screen 504 may include a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 504, where N is a positive integer greater than 1.

[0140] The camera 505 can be used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some examples, the electronic device 100 may include 1 or N cameras 505, where N is a positive integer greater than 1.

[0141] In the embodiment of the present application, the camera 505 can be used to collect an image (eg, a first image) including a face image of the first user. The camera 505 can be a front camera or a rear camera. Figure 2 In the process shown, camera 505 is the first camera. For details, please refer to Figure 2 The illustrated embodiments will not be described in detail here.

[0142] The audio module 506 can be used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signal. The audio module 506 can also be used to encode and decode audio signals. In some embodiments, the audio module 506 can also be arranged in the processor 501, or some functional modules of the audio module 506 can be arranged in the processor 501.

[0143] The speaker 507 , which may also be called a “speaker”, is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or a hands-free phone call through the speaker 507 .

[0144] The electronic device 100 may further include a sensor module ( Figure 5 Not shown) and / or a touch sensor ( Figure 5 (not shown). The touch sensor may also be referred to as a "touch control device". The touch sensor may be provided on the display screen 504, and the touch sensor and the display screen 504 form a touch screen, also referred to as a "touch control screen". The touch sensor may be used to detect a touch operation applied thereto or thereabout. Optionally, the sensor module may also include a gyroscope sensor ( Figure 5 Not shown), acceleration sensor ( Figure 5 The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the electronic device 100 can determine the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) through the gyroscope sensor. The acceleration sensor can be used to detect the acceleration magnitude of the electronic device 100 in various directions (generally x, y, and z axes), and can also detect the magnitude and direction of gravity when the electronic device 100 is stationary.

[0145] The electronic device 100 may further include a mobile communication module ( Figure 5 (not shown). The mobile communication module can provide solutions for wireless communications including 2G / 3G / 4G / 5G etc. applied to the electronic device 100.

[0146] It should be noted that Figure 5 The electronic device 100 shown in the figure is only used to exemplify the hardware structure of the electronic device provided by the present application, and does not constitute a specific limitation to the present application.

[0147] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0149] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A line of sight estimation method, applied to an electronic device comprising a first camera and a display screen, wherein the first camera and the display screen are located on the same side of the electronic device, characterized in that: The method comprises: The electronic device collects a first image through the first camera, where the first image includes a face image of a first user; The electronic device performs distortion correction on the first image; The electronic device extracts key points of the first user's face based on the first undistorted image; The electronic device determines the head posture of the first user based on the facial key points of the first user; The electronic device establishes a normalized camera coordinate system based on the head posture of the first user and the camera coordinate system of the first camera; wherein the camera coordinate system of the first camera is a coordinate system with the optical center of the first camera as the origin and the optical axis of the first camera as the Z axis of the coordinate system, and the normalized camera coordinate system is a coordinate system with the optical center of the normalized camera as the origin and the optical axis of the normalized camera as the Z axis of the coordinate system; The electronic device transforms the first undistorted image into the first image in the normalized camera coordinate system based on a first transformation matrix; The electronic device performs grayscale equalization processing based on the first image in the normalized camera coordinate system to obtain a first standard image; The electronic device extracts head features and facial features of the first user based on the first standard image; The electronic device determines, based on the head features and facial features of the first user, a gaze direction of the first user's line of sight in a first coordinate system; wherein the first coordinate system is a coordinate system with a preset point of the first user's head as an origin; The electronic device determines the gaze direction of the first user in the normalized camera coordinate system based on the gaze direction of the first user's sight in the first coordinate system and the first user's head rotation posture; The electronic device determines, based on the gaze direction of the first user in the normalized camera coordinate system and the orientation of the first user's head relative to the first camera, the gaze direction of the first user's sight in the camera coordinate system of the first camera; The electronic device determines an intersection point between a gaze direction of the first user in a camera coordinate system of the first camera and the display screen; The electronic device determines the three-dimensional coordinates of the intersection point in the camera coordinate system of the first camera; The electronic device determines the two-dimensional coordinates of the intersection point in the screen coordinate system based on the three-dimensional coordinates of the intersection point in the camera coordinate system of the first camera and the first mapping relationship; wherein the two-dimensional coordinates are coordinates of the first position, and the first mapping relationship is used to map the coordinates of the first camera in the camera coordinate system to coordinates in the screen coordinate system; When the electronic device determines that the first position is located in a preset area of ​​the display screen, the electronic device executes a first instruction.

2. The method according to claim 1, characterized in that The electronic device collects a first image through the first camera, where the first image includes a face image of a first user, specifically including: The electronic device detects a display state of the display screen; When the electronic device detects that the display screen is in a light-on state, the electronic device collects a first image through the first camera, where the first image includes a facial image of a first user.

3. The method according to claim 1, characterized in that The facial features of the first user represent the positions of the first user's eyes and the open / closed states of the first user's eyes; the head features of the first user represent the first user's head rotation posture.

4. The method according to claim 1, characterized in that: The screen coordinate system is a coordinate system with the upper left corner of the display screen as the origin and the plane where the display screen is located as the xoy plane.

5. The method according to claim 1, characterized in that: The first instruction is: Display the notification interface, open a specified application, or display the desktop interface.

6. An electronic device, characterized in that: include: One or more processors, one or more memories and a display screen; the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, when the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 5.

8. A chip or a chip system, characterized in that: The method comprises a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit the code instructions to the processing circuit, and the processing circuit is used to run the code instructions to execute the method according to any one of claims 1 to 5.

Images