Method, system and device for estimating three-dimensional gaze direction of human eye based on appearance

Abstract

Disclosed is a method, system and device for estimating the three-dimensional gaze direction of a human eye based on appearance. The method comprises: acquiring an image or video containing a face; performing face detection in the image or video frame, introducing a face key point detection algorithm to detect and locate the face key points; cutting out the face image and left and right eye images according to the face detection result and the positions of the key points; and estimating the three-dimensional gaze of the human eye in the image using a gaze estimation model, wherein the training process of the gaze estimation model comprises: using a Base-CNNs network to extract features of the face image to obtain a full-face image feature vector, using two asymmetric branch structures of an EE-Net network to extract features of the left and right eye images to obtain double-eye image feature vectors, estimating a head posture vector in combination with the face key point detection algorithm, and finally using a full connection layer to regress the full-face image feature vector, the double-eye image feature vector and the head posture vector for the final gaze estimation task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a human eye three-dimensional gaze direction estimation method, system and device based on appearance. BACKGROUND

[0002] Human eye gaze contains rich information and is one of important behavior characteristics of human beings. Research on human eye gaze estimation can be applied to many fields such as social interaction, human-computer interaction, virtual reality and augmented reality, consumer behavior analysis, and mental disorder research. Gaze tracking represents an important task in biometric applications. Research and application in recent years show that the movement of iris and pupil contains a large amount of information, and with the continuous deepening and development of gaze tracking technology research, its application field is also more and more extensive. Therefore, human eye gaze estimation has been an important research topic in recent decades and has attracted attention of the safety science, psychology, robotics, and neurology.

[0003] Research on gaze estimation technology is mainly divided into two categories: model-based gaze estimation method and appearance-based gaze estimation method. The model-based method constructs a three-dimensional gaze estimation model by anatomical structure of human eyes and face. By using the geometric relationship between different facial features and eye features (facial landmarks, cornea, pupil, etc.), the gaze direction of human eyes can be calculated. The model-based method has high accuracy and the ability to handle head movement, and has been widely used in many professional eye trackers. Patent documents CN108875524A and CN107392156A construct a gaze estimation model by using eye and pupil features. However, since the model-based method needs related parameters of human eyes, it is necessary to calibrate individuals to obtain good accuracy. In addition, the model-based method has high requirements for image resolution, and existing researches are mostly based on wearable instruments to obtain clearer picture information for eye image modeling, which is not conducive to the popularization and promotion of this technology.

[0004] The appearance-based gaze estimation method uses the powerful learning technology of computer, directly takes the image information of human eyes as input, learns the mapping function from eye appearance to gaze direction or screen drop point from a large number of training samples, and calculates the corresponding screen gaze drop point from the appearance of human eyes in the collected new image. Compared with the model-based method, this method avoids modeling the complex eye structure, and only needs a monocular camera to obtain the appearance image information of human eyes, and the amount of statistical sample information is large, so the robustness of the algorithm is stronger than that of the model-based gaze tracking algorithm. And with the collection and publication of some data sets, the appearance-based gaze estimation method has gradually solved the problems of head free movement and light change, and has a large research space.

[0005] Early work on appearance-based methods was limited to laboratory environments with fixed head poses. With the advent of some large-scale datasets, these constraints have gradually decreased, and a variety of learning-based methods have emerged, including artificial neural networks (ANNs), random forest methods, Gaussian process regression methods, support vector regressions (SVRs), reinforcement learning, and convolutional neural networks (CNNs). Among them, CNNs currently have higher accuracy and robustness in gaze estimation, and can perform non-calibrated gaze estimation with sufficient scale and diversity of training data support. For example, patent document CN113807330A uses a deep learning method for gaze estimation, which improves the accuracy of gaze estimation to some extent. Dr. Xucong Zhang and others of the Max Planck Institute in Germany were the first to try to use LeNet and VGG16 network architecture for monocular gaze estimation, and to splice head pose information with extracted eye features to learn human eye gaze. However, using monocular images as input does not fully utilize the complementary information of binoculars. Yihua Cheng of Beijing University of Aeronautics and Astronautics proposed an asymmetric regression method based on binoculars. Using monocular / binaryocular images as input has two defects: 1) it cannot fully utilize the features of the full face for gaze estimation; 2) it needs an additional module to estimate the head pose. Therefore, some scholars have begun to explore using full-face images or full-face plus binocular images for gaze estimation. The disadvantage is that they mostly use the same network model to extract features from full-face images and left and right eye images, ignoring the fact that eye images contain more fine-grained feature information in gaze estimation, while the focus on full-face images is more on the impact of overall facial features on gaze estimation accuracy. SUMMARY

[0006] The present application provides a human eye gaze direction estimation method, system and device. The present application uses two different network structures to process full-face images and left and right eye images respectively, while adding a head pose vector, which can help extract more fine-grained gaze estimation feature information and improve the accuracy and robustness of gaze estimation.

[0007] In a first aspect, a method for estimating a gaze direction of a human eye based on appearance is provided, including: obtaining an image or a video containing a face; performing face detection in the image or video frame using an MTCNN method, and introducing a face key point detection algorithm to detect and locate face key points; cropping a face image and left and right eye images according to the face detection result and the positions of the face key points; and estimating a three-dimensional gaze of the human eye in the image using a gaze estimation model to obtain an estimated result of the gaze direction of the human eye, wherein the training process of the gaze estimation model includes: extracting a full-face image feature vector from the face image containing less detailed information using a Base-CNNs network, extracting double-eye image feature vectors from the left and right eye images using two asymmetric branch structures of an EE-Net network, estimating a head posture vector in combination with the face key point detection algorithm, and finally performing regression on the full-face image feature vector, the double-eye image feature vectors and the head posture vector using a fully connected layer for the final gaze estimation task.

[0008] In a second aspect, a system for estimating a gaze direction of a human eye based on appearance is provided, including: an image or video obtaining module configured to obtain an image or a video containing a face; a face detection module configured to perform face detection in the image or video frame using an MTCNN method, and introduce a face key point detection algorithm to detect and locate face key points; an image cropping module configured to crop a face image and left and right eye images according to the face detection result and the positions of the face key points; and a gaze estimation model capable of obtaining an estimated result of the gaze direction of the human eye, wherein the training process of the gaze estimation model includes: extracting a full-face image feature vector from the face image containing less detailed information using a Base-CNNs network, extracting double-eye image feature vectors from the left and right eye images using two asymmetric branch structures of an EE-Net network, estimating a head posture vector in combination with the face key point detection algorithm, and finally performing regression on the full-face image feature vector, the double-eye image feature vectors and the head posture vector using a fully connected layer for the final gaze estimation task.

[0009] In the first aspect and / or the second aspect described above, the head posture vector calculation method includes: extracting 68 key point pixel information of the face through the face key point detection algorithm; calculating a rotation matrix and a translation matrix between a 3D face coordinate system and a camera coordinate system using a Perspective-n-Point algorithm according to a mapping relationship between pixel coordinates of 14 points and corresponding key point coordinate values in a standard 3D face model, wherein the calculated rotation matrix of the face coordinate system to the camera coordinate system is the head posture vector, and the head posture vector includes three dimensions of a pitch angle, a yaw angle and a roll angle of the head posture.

[0010] In the first and / or second aspects described above, the EE-Net network is a composite scaling of the Base-CNNs network in three dimensions of model depth, width and input image resolution, and the scaling rules are as follows: within limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting the composite scaling coefficients.

[0011] In a third aspect, an apparatus is provided, comprising: a camera that captures images or videos containing human faces; a processor; a memory comprising one or more program modules; wherein the one or more program modules are stored in the memory and configured to be executed by the processor, and the one or more program modules comprise instructions for implementing the appearance-based human eye gaze landing point estimation method.

[0012] In a fourth aspect, a storage medium is provided for storing non-transitory instructions that, when executed by a processor, can implement the appearance-based human eye gaze landing point estimation method.

[0013] In a fifth aspect, a server is provided, characterized in that the gaze estimation model trained using the appearance-based human eye gaze landing point estimation method is arranged thereon.

[0014] The present application has the following advantages:

[0015] 1. The appearance-based human eye three-dimensional gaze direction estimation method, system and apparatus of the present application use a common monocular camera as an image acquisition device, and compared with feature-based gaze estimation methods, the method has the advantages of simple operation, low price, high method universality, etc.

[0016] 2. The present application uses a large-scale data set as a training sample to estimate the human eye gaze direction, without the need for additional calibration work on different individuals, which brings convenience to user operation and makes the measurement method more flexible.

[0017] 3. Most existing methods are committed to exploring the use of the same multi-source network structure to extract features from binocular images and full-face images. Compared with other appearance-based human eye gaze estimation methods, the present application uses two different network models to extract feature information conducive to gaze estimation from full-face images and binocular images, respectively. The EE-Net model used to extract eye image features is a composite scaling of the face image feature extraction network Base-CNNs in three dimensions of depth, width and input image resolution. This method enables the model to focus on both overall features and fully extract finer-grained feature information, thereby improving the accuracy of gaze estimation.

[0018] 4. The verification makes the application estimate the human eye sight line, when the composite scaling factor μ of the left eye and the right eye is 0, the test error is 4.16°, and when μ increases to 3, the sight line estimation error is only 2.94°. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings of the embodiments will be briefly introduced below.

[0020] Figure 1 is a flow chart of the appearance-based human eye three-dimensional sight line direction estimation method provided by an embodiment of the application.

[0021] Figure 2 is a flow chart of the sight line estimation model training provided by an embodiment of the application.

[0022] Figure 3 is a schematic diagram of the human eye sight line landing point estimation model provided by an embodiment of the application, (a) is a Base-CNNs network structure diagram for face image feature extraction, and (b) is a Base-CNNs network shrinking example.

[0023] Figure 4 is a general structure diagram of the appearance-based human eye sight line direction estimation method provided by an embodiment of the application. DETAILED DESCRIPTION

[0024] Figure 1 An end-to-end appearance-based human eye sight line direction estimation method is shown. The method shown below will be described in detail. Figure 1

[0025] Step 1: Image acquisition.

[0026] An image or video containing a face is acquired by a camera or other image acquisition device. If the image does not contain a face or the face image is incomplete, the following steps cannot be performed, and the image needs to be reacquired or the next frame of picture in the video is intercepted.

[0027] Step 2: Image preprocessing to obtain face images and left and right eye images that meet the input requirements of the model.

[0028] The sight line estimation is a very fine signal and plays an important role in the study of human behavior. In addition to the fact that the eye region image is directly related to the sight line direction, the sight line direction is also closely related to other regions of the face and the head posture, so the sight line estimation network model proposed in the application takes the full face image, the left and right eye images and the head posture vector as the input. Therefore, the collected image needs to be preprocessed to extract the left and right eye images and the full face image and the head posture information in the original image before the sight line estimation. The preprocessing process of the input image will be described in detail below. ​

[0029] Step 2-1: the MTCNN method is used to realize the detection of the face in the image or video frame, and the face image is cropped according to the detection frame.

[0030] MTCNN, Multi-task convolutional neural network, is an existing high-precision face detection and face key point detection algorithm, and the method is used for face detection in the application, which provides a basis for the cropping of full-face images and left and right eye images. The network can be divided into three network structures of P-Net, R-Net and O-Net in general, wherein P-Net is used to quickly generate a candidate window, R-Net is used to filter and select a high-precision candidate window, and O-Net is used to generate a final bounding box and a face key point.

[0031] Step 2-2: the face key point detection algorithm is used to realize the detection and positioning of 68 face key points. Since the face key point is needed for eye region detection and head pose estimation in the application, a lightweight face key point detection algorithm is introduced to detect the face key point.

[0032] Step 2-3: the positions of the left and right eye center points (the average value of the left and right eye corner coordinates) and the side length of the cropping frame (1.7 times the distance between the left and right eye corners) are calculated according to the positions of the face key points. The center and the side length are used to determine the cropping rectangular frame, and then the left and right eye images in the face image are cropped.

[0033] Step 2-4: finally, the picture size is redefined according to the model input requirements. The required face image size for gaze estimation is 224x224x3, wherein 224x224 represents the size of the face image, and 3 represents the channel number of the RGB image. The required left and right eye image size is 224*r x 224*r x 3, wherein 224*r x 224*r represents the size of the left and right eye image, and 3 represents the channel number of the RGB image.

[0034] Step 3: a preset gaze estimation model is used to estimate the three-dimensional gaze of the eyes in the image, and the gaze direction estimation result of the eyes is obtained. The training process of the gaze estimation model is as follows: the face image features and eye image features are extracted using the eye and face image feature extraction method; the head pose vector is estimated by combining the face key point detection method; finally, the full-face image feature vector, the double-eye image feature vector and the head pose vector are regressed using the full connection layer, which is used for the final gaze estimation task.

[0035] Since the full-face image and the double-eye image have different effects on the estimation of the human eye gaze direction, the present application uses two different networks to extract the features of the face image and the eye image respectively, that is, the Base-CNNs network with relatively simple model structure is used to extract the features of the full-face image containing less detailed information (the network structure is shown in Figure 3 (a)), and the EE-Net network with relatively complex model structure is used to extract the features of the left and right eye images with finer granularity, wherein the EE-Net network is a composite scaling of the Base-CNNs network in the model depth, width and input image resolution (as shown in Figure 3 (b)). The training process of the network model will be described in detail below. Figure 2

[0036] Step 3-1: Data set selection and preprocessing.

[0037] MPIIFaceGaze data set is a widely used three-dimensional gaze estimation data set, which contains 213659 face images collected from 15 subjects under different light, head posture and background, and these images are collected using the built-in camera of a notebook computer. The data set also contains 2D gaze landing point and 3D gaze angle information, 6 face key point coordinates, head posture vector, face center point coordinate information and other label data, so the present application selects MPIIFaceGaze data set as the training and testing data set of the network model, and pre-processes the data set. However, the present application is not limited to using other data sets.

[0038] According to the model input requirements, the present application calculates the positions of the eyes and the face center respectively from the 6 face key point coordinates in the label data of the data set, and takes 1.7 times the distance between the corners of the eyes and 1.5 times the farthest distance between the 6 face key points as the edge length of the eye region cropping frame and the full-face cropping frame, respectively. The left eye image, the right eye image and the face image are cropped from the original data set picture, and they are adjusted to the image size required by the model, wherein the face image required for gaze estimation is 224x224x3, and the left and right eye image size is a dynamic value 224xr224xr3, wherein 224xr224xr represents the size of the left and right eye image, and 3 represents the channel number of the RGB image.

[0039] Step 3-2: Extracting full-face image features using Base-CNNs network.

[0040] The Base-CNNs network is composed of four convolutional layers, all of which are followed by ReLU operation, and the maximum pooling and local response normalization unit is used after the first two convolutional layers and ReLU operation to reduce the image size and enhance the generalization ability of the model, and the Base-CNNs network architecture is as follows​Figure 3 (a) shows, finally, a full connection operation is performed on the extracted feature vector to form a 64-dimensional full face image feature vector.

[0041] Step 3-3: Use the EE-Net network to extract left and right eye image features respectively, and fuse the double eye image features.

[0042] Another branch of the eye and face image feature extraction network is a network named EE-Net, which takes left and right eye images as input. In order to capture more subtle changes in eye appearance from images, the present application realizes the EE-Net network by adjusting the scaling factor to scale the Base-CNNs, which specifically includes scaling in three dimensions of network width (#channels), depth (#layers) and resolution (#image size) (as shown in (b)). Figure 3

[0043] Under limited computing resources, the present application adjusts the composite scaling factor to uniformly scale the three dimensions of the network, and the calculation principle is as follows:

[0044] depth: d = a μ

[0045] width: w = b μ

[0046] resolution: r = r μ (1)

[0047] Where d, w, r are the scaling factors of the EE-Net network model in the three dimensions of depth, width and input image resolution, respectively, and μ represents the composite scaling factor of the EE-Net network model, whose value is determined by the number of available resources. a, b, r are the depth, width and resolution scaling constants of the model, which can be determined by the method of Bayesian optimization.

[0048] In theory, if the model depth is scaled by a times, the floating point operation number FLOPs will increase by the same multiple, but if the model width is increased by b times or the resolution is increased by r times, the FLOPs will increase by b 2 or r 2 times. Therefore, using the rules specified by formula (1) to scale the EE-Net network FLOPs will increase: 2 x (a b 2 r 2 ) μ In order to simplify the calculation process, the present application further restricts a, b, r:

[0049] a b 2 r 2 ≈2 ​

[0050] a > 1, b > 1, r > 1 (2)

[0051] Thus, for any device, the EE-Net is scaled using formula (1), and the FLOPs will increase by 2 μ+1 times, and users can specify the scaling factor μ according to the number of device computing resources. The present application takes μ = 3 as an example to scale the model, and the Base-CNNs and EE-Net network model structures are shown in Table 1.

[0052] Table 1 Base-CNNs and EE-Net network model structures when μ = 3

[0053]

[0054]

[0055] The EE-Net network is used to extract 64-dimensional features of left and right eye images respectively, and the extracted feature vectors are fused to form a 128-dimensional binocular image feature vector.

[0056] Step 3-4: The data set used by the present application is a data set containing head pose labels, so there is no need to estimate the head pose during model training. For data sets that do not contain head poses or when calling the trained model to estimate the line of sight, the head pose vector needs to be calculated. The head pose vector calculation method is: using the face key point detection method to extract the pixel coordinates of 6 or more points of the face, according to the coordinates of the standard face 3D model corresponding to these key points, using the classic Perspective-n-Point algorithm to calculate the rotation matrix and translation matrix between the 3D face coordinate system and the camera coordinate system, wherein the calculated rotation matrix is the head pose vector, which includes the pitch angle, yaw angle and roll angle of the head pose.

[0057] Step 3-5: The 64-dimensional feature vectors extracted from the full face image using the Base-CNNs network and the 128-dimensional feature vectors extracted from the left and right eye images using the EE-Net network are spliced, and a 3-dimensional head pose vector is added, and two fully connected layers are used for regression for the final line of sight angle estimation, and the overall network framework is shown in Figure 4 .

[0058] Step 4: Model deployment. The trained line of sight estimation model is stably deployed to the cloud in real time, and the deployment task of the application platform is realized. Taking the driver's line of sight estimation in the intelligent cockpit as an example, the trained network model is stably deployed to the cloud in real time, and the vehicle-mounted platform is deployed, for example, it can be deployed to the Qualcomm 820A chip platform.

[0059] In an embodiment, a gaze direction estimation system based on appearance is also provided, which comprises an image or video acquisition module, a face detection module, an image cropping module, and a gaze estimation model.

[0060] The image or video acquisition module is configured to acquire an image or video containing a face.

[0061] The face detection module is configured to detect a face in an image or video frame using the MTCNN method, and introduce a face key point detection algorithm to detect and locate face key points.

[0062] The image cropping module is configured to crop a face image and left and right eye images according to the face detection result and the positions of the face key points.

[0063] The gaze estimation model can obtain a gaze direction estimation result of the human eye. The training process of the gaze estimation model is as follows: a Base-CNNs network is used to extract features of the face image containing less detailed information to obtain a full-face image feature vector, two asymmetric branch structures of an EE-Net network are used to extract features of the left and right eye images in a more detailed manner to obtain a double-eye image feature vector, a head pose vector is estimated in combination with a face key point detection algorithm, and finally a full connection layer is used to regress the full-face image feature vector, the double-eye image feature vector, and the head pose vector for the final gaze estimation task.

[0064] For more detailed implementation methods of the image or video acquisition module, the face detection module, the image cropping module, and the gaze estimation model of the gaze direction estimation system based on appearance, see steps 1-3 of the above method part, which will not be repeated here.

[0065] In some embodiments, an apparatus is also provided. The apparatus comprises a camera, a processor, and a memory. The memory is configured to store non-transitory instructions (e.g., one or more program modules). The processor is configured to execute the non-transitory instructions, which, when executed by the processor, can perform one or more steps of the gaze direction estimation method of the human eye described above. The memory and the processor can be interconnected through a bus system and / or other forms of connection mechanism.

[0066] For example, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capability and / or program execution capability. For example, the central processing unit (CPU) can be of X86 or ARM architecture, etc. The processor can be a general-purpose processor or a special-purpose processor, and can control other components in the electronic device to perform desired functions.

[0067] For example, the memory can be volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), and / or a cache, etc. The non-volatile memory, for example, can include read-only memory (ROM), a hard disk, erasable programmable read-only memory (EPROM), compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more program modules can be stored on the memory, and the processor can run the one or more program modules to implement various functions of the electronic device.

[0068] In an embodiment, a storage medium is also provided for storing non-transitory instructions which, when executed by an electronic device, can implement one or more steps of the above-mentioned human eye gaze direction estimation method. That is, when the human eye gaze direction estimation method and system provided by the embodiments of the present application are implemented in the form of software and sold or used as an independent product, they can be stored in a storage medium. The relevant description of the storage medium can be referred to the corresponding description of the memory of the electronic device above, and will not be described here again.

Claims

1. A method for estimating a direction of a line of sight of a human eye based on appearance, characterized by, The method comprises: obtaining an image or video containing a face; performing face detection in the image or video frame by using an MTCNN method, and introducing a face key point detection algorithm to realize detection and positioning of face key points; cutting out a face image and left and right eye images according to the face detection result and the positions of the face key points; estimating a three-dimensional line of sight of the eyes in the image by using a line of sight estimation model to obtain an eye line of sight direction estimation result, wherein the training process of the line of sight estimation model comprises: extracting features of the face image containing less detailed information by using a Base-CNNs network to obtain a full-face image feature vector, performing more fine-grained feature extraction on the left and right eye images by using two asymmetric branch structures of an EE-Net network to obtain double-eye image feature vectors, estimating a head posture vector in combination with the face key point detection algorithm, and finally performing regression on the full-face image feature vector, the double-eye image feature vector and the head posture vector by using a full connection layer for a final line of sight estimation task; the EE-Net network is a composite scaling of the Base-CNNs network in three dimensions of model depth, width and input image resolution, and the scaling rules are as follows: in limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting the composite scaling coefficients, and the calculation principles are as follows: depth: width: resolution: (1) wherein , , are the scaling coefficients of the EE-Net network in the three dimensions of model depth, width and input image resolution, respectively; represents the composite scaling coefficient of the EE-Net network, the value of which is determined by the number of available resources; , , are the depth, width and resolution scaling constants of the model, respectively, determined by the method of Bayesian optimization.

2. The appearance-based human eye gaze direction estimation method of claim 1, wherein the head posture vector calculation method comprises: extracting 68 key point pixel information of the face by using the face key point detection algorithm; Based on the mapping relationship between the pixel coordinates of 14 points and the corresponding key point coordinates in the standard face 3D model, using... Perspective-n-Point The algorithm calculates the rotation matrix and translation matrix between the 3D face coordinate system and the camera coordinate system. The calculated rotation matrix between the face coordinate system and the camera coordinate system is the head pose vector, which includes three dimensions: pitch angle, yaw angle and roll angle.

3. An appearance-based human eye gaze direction estimation system, characterized by, The method comprises: an image or video acquisition module configured to obtain an image or video containing a face; a face detection module configured to perform face detection in the image or video frame by using an MTCNN method, and introducing a face key point detection algorithm to realize detection and positioning of face key points; an image cutting module configured to cut out a face image and left and right eye images according to the face detection result and the positions of the face key points; a line of sight estimation model capable of obtaining an eye line of sight direction estimation result, wherein the training process of the line of sight estimation model comprises: extracting features of the face image containing less detailed information by using a Base-CNNs network to obtain a full-face image feature vector, performing more fine-grained feature extraction on the left and right eye images by using two asymmetric branch structures of an EE-Net network to obtain double-eye image feature vectors, estimating a head posture vector in combination with the face key point detection algorithm, and finally performing regression on the full-face image feature vector, the double-eye image feature vector and the head posture vector by using a full connection layer for a final line of sight estimation task; the EE-Net network is a composite scaling of the Base-CNNs network in three dimensions of model depth, width and input image resolution, and the scaling rules are as follows: in limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting the composite scaling coefficients, and the calculation principles are as follows: depth: width: resolution: (1) wherein , , are the scaling coefficients of the EE-Net network in the three dimensions of model depth, width and input image resolution, respectively; represents the composite scaling coefficient of the EE-Net network, the value of which is determined by the number of available resources; , , are the depth, width and resolution scaling constants of the model, respectively, which are determined by the method of Bayesian optimization.

4. The appearance-based human eye gaze direction estimation system of claim 3, wherein the head posture vector calculation method comprises: extract 68 key points pixel information of the face through a face key point detection algorithm; Based on the mapping relationship between the pixel coordinates of 14 points and the corresponding key point coordinates in the standard face 3D model, using... Perspective-n-Point The algorithm calculates the rotation matrix and translation matrix between the 3D face coordinate system and the camera coordinate system. The calculated rotation matrix between the face coordinate system and the camera coordinate system is the head pose vector, which includes three dimensions: pitch angle, yaw angle and roll angle.

5. An apparatus, comprising: comprise: a camera that captures an image or a video containing a face; a processor; a memory comprising one or more program modules; wherein the one or more program modules are stored in the memory and configured to be executed by the processor, and the one or more program modules comprise instructions for implementing the appearance-based human eye gaze direction estimation method according to any one of claims 1-2.

6. A storage medium for storing non-transitory instructions, the instructions comprising: The non-transitory instructions, when executed by the processor, can implement the appearance-based human eye gaze direction estimation method according to any one of claims 1-2.

7. A server, characterized by The gaze estimation model trained by using the appearance-based human eye gaze direction estimation method according to any one of claims 1-2 is arranged thereon. The gaze estimation model trained by using the appearance-based human eye gaze direction estimation method according to any one of claims 1-2 is arranged thereon.

Images