A method for estimating a line of sight of a human eye based on appearance

Abstract

The application discloses a human eye line of sight estimation method based on appearance, and a line of sight estimation model used by the method comprises an improved ResNet network and full connection layers; a multi-scale hollow convolution block and a feature selection block are respectively embedded after a second convolution layer, a third convolution layer and a fourth convolution layer of the ResNet network, and an effective area attention block is embedded after a fifth convolution layer; output features of the multi-scale hollow convolution block are used as input features of a next convolution layer; output features of all the feature selection blocks are spliced with output features of the fifth convolution layer to be used as input features of the effective area attention block, and then an improved ResNet network is obtained; a face image is subjected to spatial gaze feature extraction through the improved ResNet network; the spatial gaze feature is subjected to two full connection layers respectively, and a yaw angle and a pitch angle of the line of sight are obtained; finally, the trained line of sight estimation model is used for human eye line of sight estimation. The method extracts more global feature information, enhances effective area information, weakens redundant information that may contain blurred facial details in the features, and has higher estimation precision.

Description

Technical Field

[0001] This invention belongs to the field of gaze estimation technology, specifically a method for estimating human eye gaze based on appearance. Background Technology

[0002] In computer vision, gaze estimation is a research branch of face detection. Depending on the application scenario, it can be broadly categorized into three types: gaze point estimation, gaze target estimation, and 3D gaze estimation. Gaze point estimation estimates the focal point where the gazes of both eyes converge on the screen, primarily performed in a two-dimensional plane. Gaze target estimation detects the target being gazed at; currently, it can detect both the gaze target and the gazed object within the same image, as well as across frames. 3D gaze estimation estimates the gaze direction of the human eye in the world coordinate system using eye or face images.

[0003] Existing methods are mainly divided into model-based and appearance-based gaze estimation methods. Model-based gaze estimation primarily detects the appearance and shape features of the eyes, and calculates and estimates the gaze based on these features. Appearance-based gaze estimation employs machine learning strategies, learning from the texture information of the face image to study the eye image and gaze direction, and inferring the gaze. However, appearance-based gaze estimation suffers from drawbacks. Traditional convolutional neural networks extract shallow features with small receptive fields, leading to the loss of contextual information and the inability to distinguish between different visual fields. This results in insufficient context awareness and weak semantic information representation. Furthermore, to achieve better results in gaze estimation based on the entire face image, it is necessary to extract as much useful regional information as possible from the input image. However, the importance of different regions varies significantly in the gaze estimation task. Therefore, this invention proposes an appearance-based gaze estimation method that extracts more global feature information, enhances effective regional information, and weakens redundant information that may contain blurred facial details. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method for estimating human eye gaze based on appearance.

[0005] The present invention solves the aforementioned technical problem by adopting the following technical solution:

[0006] A method for estimating human eye gaze based on appearance, characterized by the following steps:

[0007] Step 1: Acquire several face images, convert the 3D gaze direction in the face images into a 2D gaze direction, and mark the yaw angle and pitch angle of the gaze based on the 2D gaze direction;

[0008] Step 2: Construct a gaze estimation model; The gaze estimation model includes an improved ResNet network and fully connected layers. The face image is processed by the improved ResNet network to extract spatial gaze features. The spatial gaze features are then processed by two fully connected layers to obtain the yaw angle and pitch angle of the gaze.

[0009] After the second, third, and fourth convolutional layers of the ResNet network, multi-scale dilated convolutional blocks and feature selection blocks are embedded respectively. After the fifth convolutional layer, effective region attention blocks are embedded. The output features of the multi-scale dilated convolutional blocks are used as the input features of the next convolutional layer. The output features of all feature selection blocks are concatenated with the output features of the fifth convolutional layer and used as the input features of the effective region attention blocks, thus obtaining the improved ResNet network.

[0010] For the feature selection block, the input features are divided into four sub-groups according to the number of channels. The value of each sub-group feature is calculated, and the output features of the feature selection block are obtained according to equations (1) to (3).

[0011]

[0012] T mdn =T0+T1+T2-T min -S max (2)

[0013] T min =Min[T0, T1, T2] (3)

[0014] In the formula, T output S is the output feature of the feature selection block. max T is the feature of the subgroup with the largest value. mdn T0, T1, and T2 are the features of the median subgroup, excluding the smallest value. min The feature with the smallest value among the remaining three subgroup features;

[0015] For the effective region attention block, the input features are average pooled along the horizontal and vertical directions to obtain horizontal global pooling features and vertical global pooling features. Simultaneously, the input features are max pooled along the horizontal and vertical directions to obtain horizontal max pooling features and vertical max pooling features. The horizontal global pooling features and horizontal max pooling features are concatenated to obtain a one-dimensional horizontal feature. The vertical global pooling features and vertical max pooling features are concatenated to obtain a one-dimensional vertical feature. The one-dimensional horizontal feature and one-dimensional vertical feature are then subjected to one-dimensional convolution, group normalization, and activation operations, respectively, before being multiplied with the input features to obtain the output feature of the effective region attention block.

[0016] Step 3: Use the face images from Step 1 to train the gaze estimation model, and then use the trained gaze estimation model for human eye gaze estimation.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] 1. This invention introduces multi-scale dilated convolutional blocks into the ResNet network, enabling the model to capture a wider receptive field while extracting shallow features. Moreover, the same network layer can also extract features from multiple receptive fields, thereby extracting more global feature information.

[0019] 2. A region enhancement technique combining one-dimensional convolution and group normalization, namely the effective region attention block, is proposed. This block is primarily responsible for selecting information more relevant to the current task from diverse data. It effectively encodes one-dimensional horizontal and one-dimensional vertical features without requiring dimensionality reduction to accurately locate the region of interest, while also allowing for lightweight implementation. The use of multi-scale dilated convolutional blocks and the effective region attention block addresses the limitations of feature extraction regions in unconstrained environments and the scarcity of gaze-related feature information, particularly in the eye region, where limited feature information hinders model prediction.

[0020] 3. The features extracted by the convolutional layers of the ResNet network may contain redundant information of blurred facial details. The presence of redundant information will affect gaze estimation. Therefore, a feature selection block is introduced to extract key features. All the extracted key features are concatenated with the output features of the fifth convolutional layer of the ResNet network to guide the model to estimate gaze more accurately.

[0021] 4. This invention utilizes a lightweight network model, which has a fast prediction speed, can be deployed on a variety of devices, and only requires ordinary camera devices to collect face images. Attached Figure Description

[0022] Figure 1 This is a structural diagram of the line-of-sight estimation model of the present invention;

[0023] Figure 2 This is a structural diagram of the multi-scale dilated convolution block of the present invention;

[0024] Figure 3 This is a structural diagram of the feature selection block of the present invention;

[0025] Figure 4 This is a structural diagram of the effective region attention block of the present invention;

[0026] Figure 5 This is a diagram illustrating the framework for constructing the loss function of this invention. Detailed Implementation

[0027] Specific embodiments are given below with reference to the accompanying drawings. These specific embodiments are only used to describe the technical solution of the present invention in detail, and are not intended to limit the scope of protection of this application.

[0028] This invention provides an appearance-based method for estimating human eye gaze (hereinafter referred to as the method, see below). Figures 1-5 The process includes the following steps:

[0029] Step 1: Acquire several face images, convert the 3D gaze direction in the face images into a 2D gaze direction, and label the yaw and pitch angles of the 2D gaze direction to obtain the dataset;

[0030] Step 2: Construct a gaze estimation model based on the ResNet network;

[0031] The gaze estimation model consists of an improved ResNet network and fully connected layers. The face image is processed by the improved ResNet network to extract spatial gaze features. The spatial gaze features are then processed by two fully connected layers to obtain the yaw angle and pitch angle of the gaze, which are the outputs of the gaze estimation model.

[0032] This embodiment takes ResNet-18 as an example. The initial layer (conv1) of ResNet-18 is replaced with a depth initial layer. Each residual block of ResNet-18 is downsampled using average pooling. After the second convolutional layer (conv2_x), the third convolutional layer (conv3_x), and the fourth convolutional layer (conv4_x), a multi-scale dilated convolutional block and a feature selection block are embedded respectively. At the same time, after the fifth convolutional layer (conv5_x), an effective region attention block is embedded. The output features of the multi-scale dilated convolutional block are used as the input features of the next convolutional layer. The output features of all feature selection blocks are concatenated with the output features of the fifth convolutional layer and used as the input features of the effective region attention block, thus obtaining the improved ResNet-18.

[0033] The initial depth layer includes a convolutional layer and an average pooling layer. The convolutional layer has a size of 7×7, 64 channels, and a stride of 2. The average pooling layer has a size of 2×2, a stride of 2, and zero padding.

[0034] like Figure 2As shown, the multi-scale dilated convolutional block comprises five parallel branches. The input features of the multi-scale dilated convolutional block are the input features of each branch. The first four branches use convolutions with different padding, stride, dilation rate, and kernel size to extract features at multiple scales. The fifth branch uses residual connections, employing the idea of ​​identity mapping to better preserve the original information of the input features. Specifically, the first branch includes a 3×3 convolution with a dilation rate of 1, the second branch includes a 3×1 convolution with a dilation rate of 1, and a 3×3 convolution with a kernel size of 1. The first branch has a dilated convolution with a dilation rate of 3. The third branch includes a 1×3 convolution with a dilation rate of 1 and a 3×3 dilated convolution with a dilation rate of 3. The fourth branch includes a 1×3 convolution with a dilation rate of 1, a 3×1 convolution with a dilation rate of 1, and a 3×3 dilated convolution with a dilation rate of 5. The output features of the first four branches are concatenated. After the concatenated features are processed by a 1×1 convolution, they are added element-wise with the input features of the multi-scale dilated convolution block to obtain the output of the multi-scale dilated convolution block.

[0035] like Figure 3 As shown, in the feature selection block, the input features are first divided into four subgroups S0~S4∈R according to the number of channels. (H×W) The algorithm calculates the feature values ​​for each subgroup, discards the subgroup feature with the smallest value, calculates the median subgroup feature based on the remaining three subgroup features, adds the subgroup feature with the largest value to the median subgroup feature, and calculates the average to obtain the output feature of the feature selection block. The feature selection block can be represented as:

[0036]

[0037] T mdn =T0+T1+T2-T min -S max (2)

[0038] T min =Min[T0, T1, T2] (3)

[0039] In the formula, T output S is the output feature of the feature selection block. max T is the feature of the subgroup with the largest value. mdn T0, T1, and T2 are the features of the median subgroup, excluding the smallest value. min It is the feature with the smallest value among the remaining three subgroup features.

[0040] like Figure 4As shown, the input features of the effective region attention block are the output features of the fifth convolutional layer of ResNet-18. In the effective region attention block, the input features are subjected to average pooling along the horizontal and vertical directions to obtain horizontal global pooling features and vertical global pooling features, respectively. On the other hand, the features of each channel are subjected to max pooling along the horizontal and vertical directions to obtain horizontal max pooling features and vertical max pooling features, respectively. The horizontal global pooling features and the horizontal max pooling features are concatenated to obtain a one-dimensional horizontal feature. The vertical global pooling features and the vertical max pooling features are concatenated to obtain a one-dimensional vertical feature. The one-dimensional horizontal feature and the one-dimensional vertical feature are then subjected to one-dimensional convolution, group normalization and activation operations, respectively, and then multiplied with the input features to obtain the output features of the effective region attention block.

[0041] The effective region attention block can be represented as:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] y h =sigmoid(G n (conv1d(P h (12)

[0051] y w =sigmoid(G n (convld(P w (13)

[0052] Y = X × y h ×y w (14)

[0053] In the formula, X c (h, i), X c (j, w) represent the features X of the c-th channel of the input features, respectively. cThe horizontal and vertical directions are represented by H and W, respectively, which represent the height and width of the input feature. X represents the feature of the c-th channel, respectively. c The results of horizontal and vertical global pooling are given, with Maxpool representing the max pooling operation. X represents the feature of the c-th channel, respectively. c The horizontal and vertical max pooling results are given, with `concat` representing the concatenation operation. X represents the feature of the c-th channel, respectively. c Horizontal pooling results and vertical pooling results, P h P w These represent one-dimensional horizontal features and one-dimensional vertical features, respectively. C represents the number of channels, convld represents a one-dimensional convolution operation, and G represents a vertical feature. n represents the group normalization operation, sigmoid represents the activation operation, and X and Y represent the input and output features of the effective region attention block, respectively.

[0054] The effective region attention block employs a region enhancement mechanism, processing sequential signals in features through one-dimensional convolutions instead of two-dimensional convolutions. One-dimensional convolutions are not only better at processing sequential signals but also more lightweight than two-dimensional convolutions. The kernel size of the one-dimensional convolution is 7, effectively enhancing the interactive capability of embedding localization information, enabling the entire region enhancement mechanism to accurately locate the region of interest. Batch normalization (BN) largely depends on the size of the mini-batch. When the mini-batch is too small, the mean and variance calculated by BN may not adequately represent the entire dataset, potentially harming the overall model performance. Therefore, replacing BN with group normalization (GN) and embedding it into a smaller network architecture immediately and significantly improves model performance.

[0055] The third step is to train the gaze estimation model by inputting a sequence of face images or videos into the trained gaze estimation model to estimate the gaze of the human eyes.

[0056] The face image is input into the gaze estimation model. The improved ResNet network extracts spatial gaze features from the face image. The spatial gaze features are passed through two fully connected layers to predict the yaw and pitch angles of the gaze. In order to calculate the model training loss, the yaw and pitch angles of the gaze are converted into probability distributions by the Softmax function to obtain the yaw angle probability distribution and the pitch angle probability distribution. Based on the yaw angle probability distribution and the pitch angle probability distribution, the classification loss and L1 loss of the yaw angle and the pitch angle are calculated respectively. Finally, all losses are added together to obtain the total loss. That is, during the training process, the total training loss is calculated by equations (12) to (14).

[0057]

[0058]

[0059]

[0060] In the formula, Indicates total training loss. The classification loss represents the difference between yaw and pitch angles. The L1 loss represents the yaw and pitch angles, where λ is a hyperparameter and p k Let y be the probability of the k-th sample. k Let y be the true label of the k-th sample. n , Let n represent the true label and the predicted value of sample n, respectively, and N represent the number of samples.

[0061] The initial learning rate was set to 0.01, and training was conducted for 40 epochs. The first 5 epochs used a quadratic function for learning rate warm-up, epochs 5-30 used cosine annealing, and the learning rate remained constant in the last 10 epochs. This was done to ensure the model learns facial image features more smoothly in the initial training phase, avoiding training instability or gradient explosion caused by an excessively large learning rate. Using cosine annealing to set the learning rate allows for faster convergence in the early stages of training, while a gradually decreasing learning rate in the later stages allows for more precise parameter fine-tuning, resulting in better performance.

[0062] Optimizer configuration: The type is SGD (Stochastic Gradient Descent), the momentum parameter is set to 0.9, and the weight decay parameter is set to 5e-4; Nesterov momentum is used, and the weights and biases of the normalized layer are not decayed; the maximum gradient norm is set to 35, which means that the norm of the gradient will be clipped to 35, and the norm of the gradient is calculated using the L2 norm.

[0063] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method of estimating a line of sight of a human eye based on appearance, characterized by, The method comprises the following steps: First step: obtaining a plurality of face images, converting the three-dimensional gaze direction in the face images into a two-dimensional gaze direction, and labeling the yaw angle and pitch angle of the gaze line according to the two-dimensional gaze direction; Second step: constructing a gaze estimation model; the gaze estimation model comprises an improved ResNet network and a full connection layer, the face image is subjected to spatial gaze feature extraction through the improved ResNet network, and the spatial gaze feature is subjected to the yaw angle and pitch angle of the gaze line through two full connection layers respectively; A multi-scale hollow convolution block and a feature selection block are respectively embedded after the second convolution layer, the third convolution layer and the fourth convolution layer of the ResNet network, and an effective region attention block is embedded after the fifth convolution layer; the output feature of the multi-scale hollow convolution block is used as the input feature of the next convolution layer; the output features of all the feature selection blocks and the output feature of the fifth convolution layer are spliced to be used as the input feature of the effective region attention block, so as to obtain the improved ResNet network; For the feature selection block, the input feature is divided into four sub-group features according to the channel number, the values of the sub-group features are calculated, and the output feature of the feature selection block is obtained according to formula (1)-(3); T mdn = T0+ T1+ T2- T min -S max (2) T min = Min [T0, T1, T2] (3) In the formula, T output is the output feature of the feature selection block, S max is the sub-group feature with the maximum value, T mdn are three sub-group features other than the minimum value, T min is the sub-group feature with the minimum value among the remaining three sub-group features For the effective region attention block, the input feature is subjected to average pooling of the features of each channel along the horizontal direction and the vertical direction to obtain horizontal global pooling features and vertical global pooling features; meanwhile, the input feature is subjected to maximum pooling of the features of each channel along the horizontal direction and the vertical direction to obtain horizontal maximum pooling features and vertical maximum pooling features; the horizontal global pooling features and the horizontal maximum pooling features are spliced to obtain one-dimensional horizontal features; the vertical global pooling features and the vertical maximum pooling features are spliced to obtain one-dimensional vertical features; the one-dimensional horizontal features and the one-dimensional vertical features are subjected to one-dimensional convolution, group normalization and activation operation in sequence, and then multiplied by the input feature to obtain the output feature of the effective region attention block; Third step: training the gaze estimation model by using the face images in the first step, and using the trained gaze estimation model for eye gaze estimation.

2. The appearance-based human eye gaze estimation method of claim 1, wherein, The multi-scale hollow convolution block extracts multi-scale features by using a plurality of branches, splices the output features of all the branches, and then adds the spliced features and the input features of the multi-scale hollow convolution block element by element after convolution to obtain the output of the multi-scale hollow convolution block.

3. The appearance-based human eye gaze estimation method of claim 2, wherein, The multi-scale hollow convolution block comprises four branches, the first branch comprises a convolution with a size of 3*3 and a dilation rate of 1, the second branch comprises a convolution with a size of 3*1 and a dilation rate of 1 and a hollow convolution with a size of 3*3 and a dilation rate of 3, the third branch comprises a convolution with a size of 1*3 and a dilation rate of 1 and a hollow convolution with a size of 3*3 and a dilation rate of 3, and the fourth branch comprises a convolution with a size of 1*3 and a dilation rate of 1, a convolution with a size of 3*1 and a dilation rate of 1, and a hollow convolution with a size of 3*3 and a dilation rate of 5.

4. The appearance-based human eye gaze estimation method according to any one of claims 1 to 3, characterized in that, The ResNet network is one of ResNet-18, ResNet-34 and ResNet-50.

Images