Human eye sight line landing point estimation method, system and electronic device

Abstract

The application discloses a kind of human eye sight point of view estimation method, system, electronic equipment.The method comprises: collecting image or video containing face;Then using face detection algorithm to detect the face in the collected video or image, while using key point detection algorithm to carry out face key point positioning;According to the position of key point and face detection result left and right eye image and face image;These images and face grid image capable of representing the relative position of face in image are input to human eye sight point of view estimation model, for the estimation of final human eye sight point of view.Estimation model of human eye sight point of view includes Base-CNNs network and EE-Net network, Base-CNNs network extracts the feature of full face image containing less detailed information, EE-Net network containing two asymmetric branches carries out more fine-grained feature extraction to left and right eye image respectively, finally using fully connected layer to regress full face image feature vector, binocular image feature vector and face grid image, for the final sight estimation task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a human eye gaze point estimation method and system and an electronic device. BACKGROUND

[0002] Human eye gaze contains rich information and is one of important behavior characteristics of human beings. Research on human eye gaze estimation can be applied to many fields such as social interaction, human-computer interaction, virtual reality and augmented reality, consumer behavior analysis, and mental disorder research. Gaze tracking represents an important task in biometric applications. Research and application in recent years show that the movement of the iris and the pupil contains a large amount of information, and with the continuous deepening and development of gaze tracking technology research, its application field is also more and more extensive. Therefore, human eye gaze estimation has been an important research topic in recent decades and has attracted attention from the fields of safety science, psychology, robotics, and neurology.

[0003] Research on gaze estimation can be divided into three categories according to different application scenarios, namely, fixation target estimation, gaze point estimation, and three-dimensional gaze estimation. Gaze point estimation focuses on estimating the gaze point of the human eye on a two-dimensional plane, which can be a mobile phone screen, a tablet computer screen, a computer screen, etc., and the input of the model is the image collected by the front camera of these devices. Gaze point estimation can be applied to the research on visual attention mechanism, consumer behavior analysis in electronic shopping platforms, advertisement location placement, and other scenarios, and has very important significance.

[0004] Existing gaze estimation techniques mainly fall into two categories: model-based gaze estimation methods and appearance-based gaze estimation methods. Model-based methods require high image resolution, and are mostly based on wearable instruments to obtain clearer picture information for eye image modeling, which is not conducive to the popularization and promotion of this technology. Appearance-based gaze estimation methods use the powerful learning technology of computers to directly use human eye image information as input, learn the mapping function from eye appearance to gaze direction or screen landing point from a large number of training samples, and thus calculate the corresponding screen gaze landing point from the appearance of the human eye in the collected new image. Compared with model-based methods, this method avoids modeling the complex eye structure, and only needs a monocular camera to obtain human eye appearance image information, and the amount of statistical sample information is large, so the robustness of the algorithm is stronger than that of model-based gaze tracking algorithms.

[0005] Early work on appearance-based methods was limited to laboratory environments with fixed head poses. With the advent of some large-scale datasets, these constraints have gradually decreased, and a variety of learning-based methods have emerged, including artificial neural networks (ANNs), random forest methods, Gaussian process regression methods, support vector regressions (SVRs), reinforcement learning, and convolutional neural networks (CNNs). Among them, CNNs currently have higher accuracy and robustness in gaze estimation, and can perform non-calibrated gaze estimation with sufficient scale and diversity of training data support. For example, the gaze estimation method using deep learning in patent document CN113807330A improves the gaze estimation accuracy to some extent. Dr. Xucong Zhang et al. of Max Planck Institute in Germany first tried to use LeNet and VGG16 network architecture for monocular gaze estimation, and spliced the head pose information with the extracted eye features to learn the human eye gaze. However, with monocular image as input, the complementary information of binoculars is not fully utilized, and Yihua Cheng of Beijing University of Aeronautics and Astronautics proposed an asymmetric regression method based on binoculars. With monocular / binaryocular image as input, there are two defects: 1) cannot fully utilize the features of the whole face for gaze estimation; 2) needs additional module to estimate head pose. Therefore, some scholars begin to explore the use of full-face image or full-face plus binocular image for gaze estimation. The deficiency is that they mostly use the same network model to extract features from full-face image and left and right eye image, ignoring the fact that eye image contains more fine-grained feature information in gaze estimation, while the focus on full-face image is more on the influence of overall facial features on gaze estimation accuracy. SUMMARY

[0006] The present application provides a human eye gaze point estimation method, system and electronic device. The present application uses two different network structures to process full-face image and left and right eye image respectively, which can help to extract more fine-grained gaze estimation feature information; in addition, to solve the problems of low gaze estimation accuracy and poor robustness caused by head pose, illumination and eye region occlusion, the present application proposes to use a binocular asymmetric network structure to extract left and right eye image features, effectively improving the accuracy and robustness of gaze estimation.

[0007] In a first aspect, a method for estimating a gaze landing point of a human eye is provided, which includes: acquiring an image or a video containing a human face; performing human face detection in the image or video frame by using an MTCNN method, and introducing a human face key point detection algorithm to realize detection and positioning of human face key points; cutting out a human face image and left and right eye images according to the human face detection result and the positions of the human face key points; generating a face grid image according to the human face detection result; using a human eye gaze landing point estimation model to extract image features and perform a gaze estimation task, wherein the model includes a Base-CNNs network and an EE-Net network, the Base-CNNs network extracts features of the human face image containing less detailed information to obtain a full face image feature vector, the EE-Net network has two asymmetric branch structures, which respectively extract more detailed features of the left and right eye images to obtain double eye image feature vectors, and a full connection layer is used to regress the full face image feature vector, the double eye image feature vectors and the face grid image for the final gaze estimation task.

[0008] In a second aspect, a system for estimating a gaze landing point of a human eye is provided, which includes: an image or video acquisition module configured to acquire an image or a video containing a human face; a human face detection module configured to perform human face detection in the image or video frame by using an MTCNN method, and introduce a human face key point detection algorithm to realize detection and positioning of human face key points; an image cutting module configured to cut out a human face image and left and right eye images according to the human face detection result and the positions of the human face key points; a face grid image generation module configured to generate a face grid image according to the human face detection result; a human eye gaze landing point estimation model including a Base-CNNs network and an EE-Net network, the Base-CNNs network extracts features of the human face image containing less detailed information to obtain a full face image feature vector, the EE-Net network has two asymmetric branch structures, which respectively extract more detailed features of the left and right eye images to obtain double eye image feature vectors, and a full connection layer is used to regress the full face image feature vector, the double eye image feature vectors and the face grid image for the final gaze estimation task.

[0009] In the first aspect and / or the second aspect described above, the EE-Net network scales the Base-CNNs network in three dimensions of model depth, width and input image resolution according to a rule, and the scaling rule is as follows: within limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting a composite scaling coefficient.

[0010] In the first and / or second aspects, the composite scaling factor is dynamically adjusted according to the performance of the left and right eye images in the gaze estimation, so as to adjust the allocation of the computing resources in the feature extraction of the binocular images. Under the fixed computing resource limit, a relatively high scaling factor is given to the eye image with relatively high quality, so that the model pays more attention to the features extracted from the eye with relatively high quality, and a relatively small scaling factor is given to the eye image with relatively low quality, so that the model pays less attention to the features extracted from the eye with relatively low quality.

[0011] In a third aspect, an electronic device is provided, comprising: a processor; a memory comprising one or more program modules; wherein the one or more program modules are stored in the memory and configured to be executed by the processor, and the one or more program modules comprise instructions for implementing the method for estimating the gaze landing point of a human eye.

[0012] In a fourth aspect, a storage medium is provided for storing non-transitory instructions capable of implementing the method for estimating the gaze landing point of a human eye when executed by a processor.

[0013] The beneficial effects of the present application are:

[0014] 1. Compared with the feature-based gaze estimation method, the present application uses a common monocular camera as an image acquisition device for human eye gaze estimation based on the appearance of full-face images and binocular images. The present application has the advantages of simple device operation, low price, high universality, etc. due to the low requirement for image resolution.

[0015] 2. The present application estimates the gaze direction of a human eye using a large-scale data set as a training sample, without the need for additional calibration work on different individuals, which brings convenience to user operation and makes the measurement method more flexible.

[0016] 3. Most of the existing technologies are committed to exploring the use of the same branch network structure to extract features of binocular images and full-face images. Compared with other appearance-based human eye gaze estimation methods, the present application uses two different network models to extract feature information conducive to gaze estimation in full-face images and binocular images, respectively. The EE-Net model used to extract eye image features is a composite scaling of the face image feature extraction network Base-CNNs in depth, width and input image resolution. This method enables the model to pay attention to both overall features and fully extract finer-grained feature information, thereby improving the accuracy of gaze estimation.

[0017] 4. In order to solve the problems of low accuracy and poor robustness of gaze estimation caused by head pose, illumination and eye region occlusion, the application proposes to use EE-Net double-eye asymmetric network structure to extract left and right eye image features, thereby effectively improving the accuracy and robustness of gaze estimation within fixed computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings of the embodiments will be briefly introduced below.

[0019] Figure 1 is a flow chart of a human eye gaze point estimation method provided by an embodiment of the application.

[0020] Figure 2 is a flow chart of a gaze estimation model training provided by an embodiment of the application.

[0021] Figure 3 is a schematic diagram of a human eye gaze point estimation model provided by an embodiment of the application, (a) is a Base-CNNs network structure diagram for face image feature extraction, and (b) is a Base-CNNs network shrinkage example.

[0022] Figure 4 is a general structure diagram of a human eye gaze point estimation method provided by an embodiment of the application. DETAILED DESCRIPTION

[0023] Figure 1 A human eye gaze point estimation method is shown. The method shown below will be described in detail. Figure 1

[0024] Step 1: Image acquisition.

[0025] An image or video containing a face is acquired by an electronic device. If the image does not contain a face or the face image is incomplete, the following steps cannot be performed, and the image needs to be reacquired or the next frame of picture in the video is intercepted. The electronic device in the application includes but is not limited to a smart phone, a tablet computer or other devices with an embedded camera.

[0026] Step 2: Use the MTCNN method to detect the face in the image or video frame, and introduce a face key point detection algorithm to realize the detection and positioning of the face key point.

[0027] ​MTCNN, Multi-task convolutional neural network, is an existing high-precision face detection and face key point detection algorithm, and the face detection and face key point detection are performed by using the method, so as to provide a basis for face image and left and right eye image cutting, and the network can be divided into three network structures of P-Net, R-Net and O-Net in the whole, wherein the P-Net is used for quickly generating a candidate window, the R-Net is used for filtering and selecting a high-precision candidate window, and the O-Net is used for generating a final boundary box and a face key point.

[0028] The implementation of MTCNN includes the following steps:

[0029] Step 2-1: Transforming the original image or video frame collected in step 1 in different scales to construct an image pyramid, and realizing face detection tasks of different sizes in the image.

[0030] Step 2-2: Using the P-Net network to quickly generate a face candidate window. The P-Net network takes the image pyramid generated in step 2-1 as input, performs rough feature extraction through a full convolutional network, and then uses the features to judge whether there is a face in the image, and simultaneously generates multiple predicted windows in which a face may exist through boundary box regression and face key point locator.

[0031] Step 2-3: Using the R-Net network to select a high-precision candidate window. The predicted window generated in step 2-2 is input into the R-Net network, and the feature is precisely extracted through 3 convolutional layers and 1 fully connected layer, and the boundary box regression and key point positioning of the face area are performed again through the boundary box regression and face key point locator, and then a more accurate face area is output.

[0032] Step 2-4: Using the O-Net network to generate a final boundary box and a face key point, and the O-Net network performs regression on the face key feature points through four convolutional layers and one fully connected layer, and finally outputs a face area detection box and five feature points;

[0033] Step 2-5: Since the present application needs to use more face key points for eye area detection, a lightweight face key point detection algorithm is introduced for face key point detection, which generally involves detection and positioning of 68 face key points.

[0034] Step 3: According to the face detection result and the position of the key point, the face image and the left and right eye image are cut respectively, and the picture size is redefined according to the model input requirement.

[0035] The gaze estimation as a very fine signal plays an important role in the study of human behavior. In addition to the eye region image being directly related to the gaze direction, the gaze direction is also closely related to other regions of the face and the head posture, so the gaze estimation network model proposed in the application takes the full face image, left and right eye images and head posture vector as input. Therefore, the collected image needs to be preprocessed to extract the left and right eye and full face images and head posture information in the original image before gaze estimation, which specifically includes the following steps:

[0036] Step 3-1: The left upper corner coordinate and the right lower corner coordinate of the face region are finally output by the MTCNN face detection method in step 2-4, and the face region in the original image or video is cropped according to the left upper corner and the right lower corner coordinates of the face region.

[0037] Step 3-2: The center positions of the left and right eyes (i.e. the average of the left and right eye corner coordinates of the eyes) are calculated according to the key points detected by the face key point detection algorithm in step 2-5, and the side length of the cropped frame (i.e. 1.7 times the distance between the left and right eye corners) is also calculated. According to the center and the side length, a cropped rectangular frame is determined, and then the left and right eye images in the face image are cropped.

[0038] Step 3-3: Finally, the picture size is redefined according to the model input requirements. The required face image size for gaze estimation is 224x224x3, where 224x224 represents the size of the face image, and 3 represents the channel number of the RGB image; the required left eye image size is 224xr l x224xr l x3, where 224xr l x224xr l represents the size of the left eye image, and 3 represents the channel number of the RGB image; and the required right eye image size is 224xr r x224xr r x3, where 224xr r x224xr r represents the size of the right eye image, and 3 represents the channel number of the RGB image.

[0039] Step 4: Generate a face grid image according to the face detection result.

[0040] The face grid is a method for representing the relative position of the face in the image first proposed by Krafka et al., and the image contains head posture information, avoiding the complex operation of head posture extraction. The specific method for generating the face grid image is as follows: according to the face detection algorithm in step 2, the face is detected, and the pixel values in the face region of the original image are assigned as 1, and the other regions are 0. Finally, according to the model input requirements, the face grid image is defined as a 25x25 single-channel image;

[0041] Step 5: Extracting left eye image features, right eye image features and face image features respectively using preset eye and face image feature extraction model.

[0042] The eye and face image feature extraction model mainly includes two network structures, Base-CNNs network and EE-Net network, that is, using the Base-CNNs network with relatively simple model structure to extract the full face image features containing less detailed information (network structure as shown in Figure 3 (a)); using the EE-Net network with relatively complex model structure containing two asymmetric branch structures to extract more fine-grained features of left and right eye images respectively.

[0043] EE-Net is a network structure containing two asymmetric branches, and the two branch structures are used for extracting features of left and right eye images respectively. The network has two properties of scalability and asymmetry.

[0044] The scalability of EE-Net mainly reflects that the network is scaled according to certain rules in three dimensions of model depth, width and input image resolution (as shown in Figure 3 (b)). That is, in order to capture more subtle changes in eye appearance from images, the present application scales the Base-CNNs by adjusting the scaling coefficients to realize the EE-Net network, which includes scaling in three dimensions of network width (#channels), depth (#layers) and resolution (#image size). The specific scaling rules are as follows:

[0045] Under limited computing resources, the present application adjusts the composite scaling coefficient to uniformly scale the three dimensions of the network, and the calculation principle is as follows:

[0046] depth:

[0047] width:

[0048] resolution:

[0049] Where d l , ω l , r l are the scaling coefficients of the left eye EE-Net branch network model in three dimensions of depth, width and input image resolution. d r , ω r , r rare the scaling coefficients of the right eye EE-Net branch network model in the depth, width and input image resolution dimensions respectively. l and r respectively represent the composite scaling coefficients of the EE-Net network model in the left and right eye branch structures, and their values are determined by the number of computer available resources. l , β l , γ l and α r , β r , γ r are the depth, width and resolution scaling constants of the left eye model and the right eye model respectively, which can be determined by the method of Bayesian optimization.

[0050] In theory, if the model depth is scaled by α times, the floating point operation FLOP s will increase by the same multiple, but if the model width is increased by β times or the resolution is increased by r times, the FLOPs will increase by β 2 times or r 2 . Therefore, using the rules specified by formula (1) to scale the EE-Net network, the FLOPs will increase by: Since the EE-Net is scaled by adjusting different scaling coefficients μ l and μ r to Base-CNNs, therefore, by referring to the experience of EfficientNet network hyperparameter setting, the same network depth, width and resolution scaling constants are set for the left eye and the right eye, that is, α l = α r = 1.2, β l = β r = 1.1, γ l = γ r = 1.15. In order to simplify the calculation process, the present application further constrains α l , β l , γ l and α r , β r , γ r :

[0051] α l · β l 2 · γ l 2 + α r · β r 2 · γ r 2 ≈ 4

[0052] α l ≥ 1, β l ≥ 1, γl ≥1, a r ≥1, b r ≥1, g r ≥1 (2)

[0053] Thus, for any device, the FLOPs will increase times using formula (1) to scale EE-Net, and users can specify the scaling factor according to their own device computing resources Here we define it as 2 ε , where e is a constant.

[0054] The present application takes μ = 3 (μ can be μ l or μ r ) as an example to scale the EE-Net branch network, and the Base-CNNs and EE-Net network model structures are shown in Table 1. Note that Table 1 is an example and does not represent that the EE-Net model is only the structure listed in Table 1. In actual model training, the scaling factor needs to be dynamically adjusted according to the computer computing performance and the "quality" of the left and right eye images.

[0055] Table 1 Base-CNNs and EE-Net network model structures when μ = 3

[0056]

[0057] The asymmetry of EE-Net mainly reflects that according to the performance of left and right eye images in gaze estimation, the model scaling factors μ l and μ r are dynamically adjusted according to certain rules, so as to adjust the allocation of resources in the feature extraction of binocular images.

[0058] Due to the influence of factors such as light, head posture, and eye region occlusion, the left and right eyes often show asymmetry in gaze estimation, that is, there is a situation that the "quality" of the image of one eye is relatively high in gaze estimation, and its performance in gaze estimation task is better than that of the other eye image. Therefore, under the limitation of fixed computing resources, the present application will give the eye image a larger scaling factor, so that the model pays more attention to the features extracted from the image of the eye with relatively high "quality", and gives the image of the eye with relatively low "quality" a smaller scaling factor, so that the model pays less attention to the features extracted from the image of the eye.

[0059] The calculation method of the model scaling factors μ l and μ r is as follows:

[0060]

[0061]

[0062] where ε is the remaining resource constant of the computer, L l and L r are the L2 loss values when the left and right eye images are used for gaze estimation using the Base-CNNs network.

[0063] The training process of the network model will be described in detail below. Figure 2 The training process of the network model will be described in detail below.

[0064] Step 5-1: Data set selection and cleaning.

[0065] GazeCapture is a data set published by MIT, which contains 1471 participants' full face images and left and right eye, face position labels, etc. The data set collects the gaze landing points of people on mobile phones and tablets, containing a total of 2.5 million frames of images, which can be applied to estimate the gaze point on a two-dimensional plane. The data set has variability in head pose, lighting, and eye / facial appearance, which helps us train a robust and general-purpose model. The data set contains the following information:

[0066] (1) The image of the participant collected using the front camera of the mobile phone or tablet;

[0067] (2) Face and left and right eye detection frame;

[0068] (3) Gaze landing point vector (x and y direction gaze landing point coordinates relative to the camera center, unit: centimeter).

[0069] Therefore, the present application first selects the GazeCapture data set as the training and testing data set of the network model, and pre-processes the data set. The inventors select about 1.5 million frames from the complete data set, remove pictures with incomplete face or eye images in the original data set, and ensure that at least one valid frame is selected for all 1471 subjects. Then the data set is divided into training set, test set and validation set, containing 1271, 50 and 150 subjects respectively.

[0070] Step 5-2: Image cropping and redefinition size.

[0071] According to the model input requirements, the present application crops the face image and left and right eye images from the original data set picture through the face and double eye detection frame in the data set label data, and adjusts them to the required image size of the model, wherein the required face image size for gaze estimation is 224x224x3, wherein 224x224 represents the size of the face image, and 3 represents the channel number of the RGB image; the required left eye image size is 224x224x3, wherein 224x224 represents the size of the left eye image, and 3 represents the channel number of the RGB image. l l l 224x224x3, wherein 224x224 represents the size of the left eye image, and 3 represents the channel number of the RGB image.​​l denotes the size of the left eye image, 3 denotes the channel number of the RGB image; the required right eye image size is 224 r r x 224 r r x 3, wherein 224 r r x 224 r r denotes the size of the right eye image, 3 denotes the channel number of the RGB image.

[0072] Step 5-3: Generate a face grid image.

[0073] Similar to step 4, a face grid image also needs to be generated according to the original image during the image feature extraction process. The specific generation method of the face grid is as follows: according to the face detection frame in the data set label, the pixel value in the face area of the original image is assigned as 1, and the other areas are 0. Finally, according to the model input requirements, the face grid image is defined as a 25x25 single-channel image.

[0074] Step 5-4: Extract full face image features using Base-CNNs network.

[0075] The Base-CNNs network is composed of four convolutional layers, all of which are accompanied by ReLU operations. Max pooling and local response normalization units are used after the first two layers to reduce the image size and enhance the generalization ability of the model. The Base-CNNs network architecture is shown in Figure 3 (a), and finally the extracted feature vector is fully linked to form a 64-dimensional full face image feature vector.

[0076] Step 5-5: Extract left and right eye image features using EE-Net network, and fuse the double eye image features to form a 128-dimensional double eye image feature vector.

[0077] Step 5-6: Use a fully connected layer to regress the face grid image to form a 128-dimensional face grid image feature vector.

[0078] Step 6: Use a fully connected layer to regress the full face image feature vector, double eye image feature vector and face grid image for the final gaze estimation task.

[0079] The 64-dimensional feature vector extracted from the full face image using the Base-CNNs network and the 128-dimensional feature vector extracted from the left and right eye images using the EE-Net network are spliced, and the 128-dimensional face grid feature vector is added. Two fully linked layers are used for regression for the final gaze angle estimation, and the overall network framework is shown in Figure 4 .

[0080] In summary, the aforementioned human eye gaze point estimation method can be summarized as follows: collecting images or videos containing human faces; then using a face detection algorithm to detect the human face in the collected video or image, and using a key point detection algorithm to locate the human face key points; according to the positions of the key points and the face detection results, left and right eye images and face images that meet the input requirements of the model can be further cropped; these images and face grid images that can represent the relative position of the face in the image are input into the human eye gaze point estimation model for the final human eye gaze point estimation. The human eye gaze point estimation model includes a Base-CNNs network and an EE-Net network. The Base-CNNs network extracts features of the full-face image containing less detailed information. The EE-Net network with two asymmetric branches extracts more granular features of the left and right eye images. The model scaling coefficient is dynamically adjusted according to the performance of the left and right eye images in gaze estimation, so as to adjust the network structure of the left and right eye image feature extraction. Finally, a fully connected layer is used to regress the full-face image feature vector, the double eye image feature vector and the face grid image for the final gaze estimation task.

[0081] In an embodiment, a human eye gaze point estimation system is also provided, which includes an image or video acquisition module, a face detection module, an image cropping module, a face grid image generation module, and a human eye gaze point estimation model.

[0082] The image or video acquisition module is configured to acquire images or videos containing human faces.

[0083] The face detection module is configured to use the MTCNN method for human face detection in the image or video frame, and introduce a face key point detection algorithm to detect and locate the face key points.

[0084] The image cropping module is configured to crop the face image and the left and right eye images according to the face detection results and the positions of the face key points. The face grid image generation module is configured to generate a face grid image according to the face detection results.

[0085] The human eye gaze point estimation model includes a Base-CNNs network and an EE-Net network. The Base-CNNs network extracts features of the face image containing less detailed information to obtain a full-face image feature vector. The EE-Net network with two asymmetric branch structures extracts more granular features of the left and right eye images to obtain a double eye image feature vector. Then, a fully connected layer is used to regress the full-face image feature vector, the double eye image feature vector and the face grid image for the final gaze estimation task.

[0086] The image or video acquisition module, the face detection module, the image cropping module, the face grid image generation module, and the human eye gaze landing point estimation model of the human eye gaze landing point estimation system are described in more details in steps 1-6 of the method section above, which will not be repeated here.

[0087] In some embodiments, an electronic device is also provided, which can be a smartphone, a tablet computer, or other device with an embedded camera (e.g., a game console). The electronic device includes a camera, a processor, and a memory. The memory is configured to store non-transitory instructions (e.g., one or more program modules). The processor is configured to execute the non-transitory instructions, which, when executed by the processor, can perform one or more steps of the human eye gaze landing point estimation method described above. The memory and the processor can be interconnected by a bus system and / or other form of connection mechanism.

[0088] For example, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit having data processing and / or program execution capabilities. For example, the central processing unit (CPU) can be of X86 or ARM architecture, etc. The processor can be a general purpose processor or a special purpose processor, which can control other components in the electronic device to perform desired functions.

[0089] For example, the memory can be a volatile memory and / or a non-volatile memory. The volatile memory can include, for example, a random access memory (RAM), a cache, or the like. The non-volatile memory can include, for example, a read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a compact disk read-only memory (CD-ROM), a USB memory, a flash memory, or the like. One or more program modules can be stored on the memory, and the processor can execute the one or more program modules to implement various functions of the electronic device.

[0090] In an embodiment, a storage medium is also provided, which is configured to store non-transitory instructions that, when executed by an electronic device, can implement one or more steps of the human eye gaze landing point estimation method described above. That is, when the human eye gaze landing point estimation method, system, and the like provided by the embodiments of the present application are implemented in the form of software and sold or used as an independent product, they can be stored in a storage medium. The relevant description of the storage medium can be referred to the corresponding description of the memory of the electronic device above, which will not be repeated here.

Claims

1. A method of estimating a point of impact of a line of sight of a human eye, characterized by, The method comprises the following steps: obtaining an image or a video containing a face; performing face detection in the image or the video frame by using an MTCNN method, and introducing a face key point detection algorithm to realize detection and positioning of face key points; cutting out a face image and left and right eye images according to the face detection result and the positions of the face key points; generating a face grid image according to the face detection result; extracting image features and performing gaze estimation by using a human eye gaze point estimation model, wherein the model comprises a Base-CNNs network and an EE-Net network, the Base-CNNs network extracts features of the face image containing less detailed information to obtain a full-face image feature vector, two asymmetric branch structures of the EE-Net network extract features of the left and right eye images in a more fine-grained manner to obtain double-eye image feature vectors, and a full connection layer is used to regress the full-face image feature vector, the double-eye image feature vectors and the face grid image for the final gaze estimation task; the EE-Net network scales the Base-CNNs network in three dimensions of model depth, width and input image resolution according to rules, and the scaling rules are as follows: under the limitation of limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting a composite scaling coefficient, and the calculation principle is as follows: depth: width: resolution: (1) wherein , , are the scaling coefficients of the left eye branch structure of the EE-Net network for extracting features of left eye images in the three dimensions of depth, width and input image resolution, respectively; , , are the scaling coefficients of the right eye branch structure of the EE-Net network for extracting features of right eye images in the three dimensions of depth, width and input image resolution, respectively; and represent the composite scaling coefficients of the EE-Net network on the left eye branch structure, the left and right eye branch structure, respectively; , , , , , , and are determined by the number of available resources; , , and , , are the depth, width and resolution scaling constants of the left eye branch structure and the left and right eye branch structure, respectively, determined by the method of Bayesian optimization.

2. The method of estimating a point of gaze of a human eye according to claim 1, wherein, According to the performance of the left and right eye images in the line-of-sight estimation, the composite scaling factor is dynamically adjusted according to the rules and , so as to adjust the allocation of computing resources in the binocular image feature extraction. under the limitation of fixed computing resources, a relatively higher scaling coefficient is given to an eye image with relatively higher quality, so that the model pays more attention to the features extracted from the eye image with relatively higher quality, and a relatively smaller scaling coefficient is given to an eye image with relatively lower quality, so that the model pays less attention to the features extracted from the eye image with relatively lower quality.

3. The human eye gaze point estimation method according to claim 2, comprising: Composite scaling factor And The calculation method is: (2) (3) wherein is a constant of the remaining computing resources, and are L2 loss values when the left and right eye images use the Base-CNNs network for gaze estimation, respectively.

4. A human eye visual line impact point estimation system, characterized by, an image or video acquisition module configured to acquire an image or a video containing a face; a face detection module configured to perform face detection in the image or the video frame by using an MTCNN method, and introduce a face key point detection algorithm to realize detection and positioning of face key points; an image cutting module configured to cut out a face image and left and right eye images according to the face detection result and the positions of the face key points; a face grid image generation module configured to generate a face grid image according to the face detection result; a human eye gaze point estimation model comprising a Base-CNNs network and an EE-Net network, the Base-CNNs network extracts features of the face image containing less detailed information to obtain a full-face image feature vector, the two asymmetric branch structures of the EE-Net network extract features of the left and right eye images in a more fine-grained manner to obtain double-eye image feature vectors, and then a full connection layer is used to regress the full-face image feature vector, the double-eye image feature vectors and the face grid image for the final gaze estimation task; ​ The EE-Net network scales the Base-CNNs network in three dimensions of model depth, width and input image resolution according to rules as follows: In limited computing resources, the three dimensions of the Base-CNNs network are uniformly scaled by adjusting the composite scaling coefficient, and the calculation principle is as follows: depth: width: resolution: (1) wherein , , are the scaling coefficients of the left eye branch structure of the EE-Net network for extracting features of left eye images in the three dimensions of depth, width and input image resolution, respectively; , , are the scaling coefficients of the right eye branch structure of the EE-Net network for extracting features of right eye images in the three dimensions of depth, width and input image resolution, respectively; and respectively represent the composite scaling coefficients of the EE-Net network on the left eye branch structure, the left and right eye branch structure; , , , , , , and the values of which are determined by the number of available resources; , , and , , are the depth, width and resolution scaling constants of the left eye branch structure and the left and right eye branch structure, respectively, which are determined by the method of Bayesian optimization.

5. The human eye view ray impact point estimation system of claim 4, wherein, According to the performance of the left and right eye images in the line-of-sight estimation, the composite scaling factor is dynamically adjusted according to the rules and , so as to adjust the allocation of computing resources in the binocular image feature extraction. Under the fixed computing resource limit, a relatively higher quality eye image is given a larger scaling coefficient, so that the model pays more attention to the features extracted from the image of the relatively higher quality eye, and a relatively lower quality eye image is given a smaller scaling coefficient, so that the model pays less attention to the features extracted from the image of the relatively lower quality eye.

6. The human eye visual line landing point estimation system according to claim 5, characterized in that, Composite scaling factor And The calculation method is: (2) (3) wherein is a constant of the remaining computing resources, and are L2 loss values when the Base-CNNs network is used for gaze estimation for left and right eye images, respectively.

7. An electronic device, comprising: comprises: a camera that captures images or videos containing human faces; a processor; a memory including one or more program modules; wherein the one or more program modules are stored in the memory and configured to be executed by the processor, and the one or more program modules include instructions for implementing the human eye visual line landing point estimation method of any one of claims 1-3.

8. A storage medium for storing non-transitory instructions, the instructions comprising: The non-transitory instructions can implement the human eye visual line landing point estimation method of any one of claims 1-3 when executed by the processor.

Images