Appearance-based human eye sight line estimation method

Abstract

The invention discloses a human eye sight line estimation method based on appearance. A sight line estimation model used in the method comprises an improved ResNet network and a full connection layer. A multi-scale cavity convolution block and a feature selection block are embedded behind a second convolution layer, a third convolution layer and a fourth convolution layer of the ResNet network respectively, an effective area attention block is embedded behind a fifth convolution layer, and output features of the multi-scale cavity convolution block serve as input features of a next convolution layer. The output features of all the feature selection blocks and the output features of the fifth convolutional layer are spliced to serve as the input features of an effective region attention block, and then an improved ResNet network is obtained; spatial fixation features are extracted from the face image through an improved ResNet network, and the spatial fixation features respectively pass through two full connection layers to obtain a yaw angle and a pitch angle of a sight line; and finally, the trained sight line estimation model is used for human eye sight line estimation. According to the method, more global feature information is extracted, effective area information is enhanced, redundant information possibly containing fuzzy face details in the features is weakened, and higher estimation precision is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of sight line estimation, and in particular is a method for estimating the sight line of a human eye based on appearance. Background Art

[0002] In computer vision technology, gaze estimation is a research branch of face detection. According to different application scenarios, it can be roughly divided into three categories: gaze point estimation, gaze target estimation and 3D gaze estimation. Gaze point estimation is to estimate the focus of the gaze of both eyes on the screen, mainly on a two-dimensional plane. Gaze target estimation is to detect the target that the human eye is looking at. Currently, it can detect the human eye and the gaze target in the same picture, and can also detect the gaze target across frames. 3D gaze estimation is to estimate the gaze direction of the human eye in the world coordinate system through eye pictures or face pictures.

[0003] Existing methods are mainly divided into model-based and appearance-based sight line estimation methods. Model-based sight line estimation mainly detects the appearance shape features of the eyes, and calculates and estimates the sight line based on these features. Appearance-based sight line estimation adopts a machine learning strategy to learn the eye image and gaze direction based on the texture information of the face image, and infer the sight line of the human eye. For the extraction of facial feature information, the receptive field of the shallow features extracted by the traditional convolutional neural network is small, resulting in the loss of some context information in the feature extraction process, the loss of the ability to distinguish different fields of view, and the defects of insufficient context perception and weak semantic information representation. In addition, in order to obtain better results for sight line estimation based on the full face image, it is necessary to extract more regional information useful for sight line estimation from the input image as much as possible, and the importance of each part of the regional information to the sight line estimation task is quite different. Therefore, the present invention proposes a human eye sight line estimation method based on appearance, which extracts more global feature information, enhances the effective regional information, and weakens the redundant information that may contain blurred facial details in the features. Summary of the invention

[0004] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a method for estimating the sight line of a human eye based on appearance.

[0005] The present invention solves the technical problem by adopting the following technical solution:

[0006] A method for estimating human eye sight lines based on appearance, characterized in that the method comprises the following steps:

[0007] Step 1: Obtain several face images, convert the three-dimensional gaze direction in the face images into a two-dimensional gaze direction, and mark the yaw angle and pitch angle of the line of sight according to the two-dimensional gaze direction;

[0008] Step 2: Build a gaze estimation model; the gaze estimation model includes an improved ResNet network and a fully connected layer. The face image is passed through the improved ResNet network to extract spatial gaze features. The spatial gaze features are passed through two fully connected layers to obtain the yaw angle and pitch angle of the gaze.

[0009] A multi-scale hole convolution block and a feature selection block are respectively embedded after the second, third and fourth convolution layers of the ResNet network, and an effective area attention block is embedded after the fifth convolution layer. The output features of the multi-scale hole convolution block are used as the input features of the next convolution layer. The output features of all feature selection blocks are concatenated with the output features of the fifth convolution layer as the input features of the effective area attention block, thereby obtaining an improved ResNet network.

[0010] For the feature selection block, the input features are evenly divided into four subgroup features according to the number of channels, and the values ​​of each subgroup feature are calculated. The output features of the feature selection block are obtained according to equations (1) to (3);

[0011]

[0012] T mdn =T0+T1+T2-T min -S max (2)

[0013] T min =Min[T0, T1, T2] (3)

[0014] Where, T output is the output feature of the feature selection block, S max is the subgroup feature with the largest value, T mdn is the median subgroup characteristic, T0, T1, and T2 are the three subgroup characteristics except the one with the smallest value, T min It is the subgroup characteristic with the smallest value among the remaining three subgroup characteristics;

[0015] For the effective area attention block, the input features are average pooled for the features of each channel in the horizontal and vertical directions respectively to obtain horizontal global pooling features and vertical global pooling features; at the same time, the input features are max-pooled for the features of each channel in the horizontal and vertical directions respectively to obtain horizontal max-pooling features and vertical max-pooling features; the horizontal global pooling features and the horizontal max-pooling features are concatenated to obtain one-dimensional horizontal features; the vertical global pooling features and the vertical max-pooling features are concatenated to obtain one-dimensional vertical features; the one-dimensional horizontal features and the one-dimensional vertical features are respectively subjected to one-dimensional convolution, group normalization and activation operations, and then multiplied with the input features to obtain the output features of the effective area attention block.

[0016] Step 3: Use the face image in the first step to train the gaze estimation model, and use the trained gaze estimation model for human eye gaze estimation.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention introduces a multi-scale dilated convolution block into the ResNet network, so that the model can capture a wider receptive field as much as possible while extracting shallow features, and the same layer of the network can also extract features of multi-scale receptive fields, thereby extracting more global feature information.

[0019] 2. A regional enhancement technique combining one-dimensional convolution and group normalization is proposed, namely the effective region attention block, which is mainly responsible for selecting more effective information for the current task from a variety of information. It effectively encodes one-dimensional horizontal features and one-dimensional vertical features without dimensionality reduction to accurately locate the region of interest, while allowing lightweight implementation. The multi-scale hole convolution block and the effective region attention block solve the problem of limited area for feature extraction in an unconstrained environment and little feature information related to the line of sight, especially the lack of feature information in the eye area, which makes the model prediction unable to proceed normally.

[0020] 3. The features extracted by the convolutional layer of the ResNet network may contain redundant information that blurs facial details. The presence of redundant information will affect the line of sight estimation. Therefore, a feature selection block is introduced to extract key features. All the extracted key features are concatenated with the output features of the fifth convolutional layer of the ResNet network to guide the model to estimate the line of sight more accurately.

[0021] 4. The present invention utilizes a lightweight network model with fast prediction speed, can be deployed on a variety of devices, and only requires ordinary camera equipment to collect facial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of the line of sight estimation model of the present invention;

[0023] Figure 2 : This is a structural diagram of the multi-scale hole convolution block of the present invention;

[0024] Figure 3 It is a structural diagram of the feature selection block of the present invention;

[0025] Figure 4 It is a structural diagram of the effective area attention block of the present invention;

[0026] Figure 5 This is a diagram of the construction framework of the loss function of the present invention. DETAILED DESCRIPTION

[0027] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the protection scope of the present application.

[0028] The present invention provides a method for estimating the sight line of a human eye based on appearance (hereinafter referred to as method, see Figures 1 to 5 ), including the following steps:

[0029] Step 1: Obtain several face images, convert the three-dimensional gaze direction in the face images into a two-dimensional gaze direction, annotate the yaw angle and pitch angle of the two-dimensional gaze direction, and obtain a data set;

[0030] Step 2: Build a sight estimation model based on the ResNet network;

[0031] The gaze estimation model includes an improved ResNet network and a fully connected layer. The face image is passed through the improved ResNet network to extract spatial gaze features. The spatial gaze features are passed through two fully connected layers to obtain the yaw angle and pitch angle of the gaze, which is the output of the gaze estimation model.

[0032] This embodiment takes ResNet-18 as an example, replaces the initial layer (conv1) of ResNet-18 with a deep initial layer, downsamples each residual block of ResNet-18 using average pooling, and embeds a multi-scale hole convolution block and a feature selection block after the second convolution layer (conv2_x), the third convolution layer (conv3_x), and the fourth convolution layer (conv4_x), respectively. At the same time, an effective area attention block is embedded after the fifth convolution layer (conv5_x). The output features of the multi-scale hole convolution block are used as the input features of the next convolution layer, and the output features of all feature selection blocks are concatenated with the output features of the fifth convolution layer as the input features of the effective area attention block, thereby obtaining an improved ResNet-18.

[0033] The depth initial layer includes a convolution layer and an average pooling layer. The size of the convolution layer is 7×7, the number of channels is 64, and the step size is 2. The size of the average pooling layer is 2×2, the step size is 2, and the padding is 0.

[0034] like Figure 2As shown in the figure, the multi-scale dilated convolution block includes five parallel branches. The input features of the multi-scale dilated convolution block are the input features of each branch. The first four branches use convolutions with different padding, step size, dilation rate, and convolution kernel size to extract multi-scale features. The fifth branch uses residual connection and the idea of ​​identity mapping to better retain the original information of the input features. Specifically, the first branch includes a convolution with a size of 3×3 and a dilation rate of 1, and the second branch includes a convolution with a size of 3×1 and a dilation rate of 1 and a convolution with a size of 3×3 and a dilation rate of 1. The dilated convolution with a dilation rate of 3, the third branch includes a convolution with a size of 1×3 and a dilation rate of 1 and a dilated convolution with a size of 3×3 and a dilation rate of 3, the fourth branch includes a convolution with a size of 1×3 and a dilation rate of 1, a convolution with a size of 3×1 and a dilation rate of 1, and a dilated convolution with a size of 3×3 and a dilation rate of 5. The output features of the first four branches are spliced, and the spliced ​​features are convolved with a size of 1×1, and then element-by-element added to the input features of the multi-scale dilated convolution block to obtain the output of the multi-scale dilated convolution block.

[0035] like Figure 3 As shown in the feature selection block, the input features are first divided into four sub-group features S0~S4∈R according to the number of channels. (H×W) , calculate the value of each subgroup feature, discard the subgroup feature with the smallest value, calculate the median subgroup feature based on the remaining three subgroup features, add the subgroup features with the largest value and the median subgroup features and average them to get the output feature of the feature selection block. The feature selection block can be expressed as:

[0036]

[0037] T mdn =T0+T1+T2-T min -S max (2)

[0038] T min =Min[T0, T1, T2] (3)

[0039] Where, T output is the output feature of the feature selection block, S max is the subgroup feature with the largest value, T mdn is the median subgroup characteristic, T0, T1, and T2 are the three subgroup characteristics except the one with the smallest value, T min It is the subgroup characteristic with the smallest value among the remaining three subgroup characteristics.

[0040] like Figure 4As shown, the input features of the effective area attention block are the output features of the fifth convolutional layer of ResNet-18. In the effective area attention block, the input features, on the one hand, perform average pooling on the features of each channel along the horizontal and vertical directions, respectively, to obtain horizontal global pooling features and vertical global pooling features; on the other hand, perform maximum pooling on the features of each channel along the horizontal and vertical directions, respectively, to obtain horizontal maximum pooling features and vertical maximum pooling features; the horizontal global pooling features and the horizontal maximum pooling features are concatenated to obtain one-dimensional horizontal features; the vertical global pooling features and the vertical maximum pooling features are concatenated to obtain one-dimensional vertical features; the one-dimensional horizontal features and the one-dimensional vertical features are respectively subjected to one-dimensional convolution, group normalization and activation operations, and then multiplied with the input features to obtain the output features of the effective area attention block.

[0041] The effective region attention block can be expressed as:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] y h =sigmoid(G n (conv1d(P h ))) (12)

[0051] y w =sigmoid(G n (convld(P w ))) (13)

[0052] Y=X×y h ×y w (14)

[0053] Where, X c (h, i), X c (j, w) represent the feature X of the cth channel of the input feature respectively cThe horizontal and vertical indicators, H and W are the height and width of the input feature respectively. Represents the feature X of the cth channel respectively c The horizontal global pooling result and the vertical global pooling result of , Maxpool represents the maximum pooling operation, Represents the feature X of the cth channel respectively c The horizontal maximum pooling result and the vertical maximum pooling result of , concat represents the splicing operation, Represents the feature X of the cth channel respectively c The horizontal pooling result and vertical pooling result of P h , P w They represent one-dimensional horizontal features and one-dimensional vertical features respectively, C represents the number of channels, convld represents one-dimensional convolution operation, G n represents the group normalization operation, sigmoid represents the activation operation, and X and Y represent the input and output features of the effective area attention block, respectively.

[0054] The effective regional attention block adopts a regional enhancement mechanism to process the sequence signal in the feature through one-dimensional convolution instead of two-dimensional convolution. One-dimensional convolution is not only good at processing sequence signals, but also lighter than two-dimensional convolution. The convolution kernel size of one-dimensional convolution is 7, which effectively enhances the interactive ability of localization information embedding, so that the entire regional enhancement mechanism can accurately locate the region of interest. Batch normalization (BN) depends heavily on the size of the mini-batch. When the mini-batch is too small, the mean and variance calculated by BN may not fully represent the entire dataset, which may damage the overall performance of the model. Therefore, group normalization (GN) is used instead of BN. When embedded in a smaller network architecture, the model performance immediately shows significant improvement.

[0055] Step 3: Train the gaze estimation model, input the facial image sequence or video into the trained gaze estimation model to estimate the gaze of the human eye.

[0056] The face image is input into the gaze estimation model. The spatial gaze features are extracted from the face image through the improved ResNet network. The spatial gaze features are passed through two fully connected layers to predict the yaw angle and pitch angle of the gaze. In order to calculate the model training loss, the yaw angle and pitch angle of the gaze are converted into probability distributions through the Softmax function to obtain the yaw angle probability distribution and the pitch angle probability distribution; based on the yaw angle probability distribution and the pitch angle probability distribution, the classification loss and L1 loss of the yaw angle and the pitch angle are calculated respectively, and finally all losses are added to obtain the total loss. That is, during the training process, the total training loss is calculated by equations (12) to (14);

[0057]

[0058]

[0059]

[0060] In the formula, represents the total training loss, represents the classification loss of yaw angle and pitch angle, represents the L1 loss of yaw angle and pitch angle, λ is a hyperparameter, and p k is the probability of the kth sample, y k is the true label of the kth sample, y n , They represent the true label and predicted value of sample n respectively, and N represents the number of samples.

[0061] The initial learning rate is set to 0.01, and the training is repeated for 40 rounds. The quadratic function is used to warm up the learning rate in the first 5 rounds, and the learning rate is annealed according to the cosine function in the 5th to 30th rounds. The learning rate remains unchanged in the last 10 rounds. The purpose is to allow the model to learn the features of face images more smoothly in the initial training stage and avoid training instability or gradient explosion problems caused by excessive learning rates. Cosine annealing is used to set the learning rate. In the early stages of training, a larger learning rate helps the model converge quickly; in the later stages of training, a gradually reduced learning rate enables the model to fine-tune its parameters more accurately, thereby achieving better performance.

[0062] Configuration of the optimizer: The type uses SGD (stochastic gradient descent), the momentum parameter is set to 0.9, and the weight decay parameter is set to 5e-4; Nesterov momentum is used, and the weights and bias terms of the normalization layer are not decayed; the maximum gradient norm is set to 35, indicating that the norm of the gradient will be clipped to 35, and the L2 norm is used to calculate the norm of the gradient.

[0063] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A method for estimating human eye sight based on appearance, characterized in that: The method comprises the following steps: Step 1: Obtain several face images, convert the three-dimensional gaze direction in the face images into a two-dimensional gaze direction, and mark the yaw angle and pitch angle of the line of sight according to the two-dimensional gaze direction; Step 2: Build a gaze estimation model; the gaze estimation model includes an improved ResNet network and a fully connected layer. The face image is passed through the improved ResNet network to extract spatial gaze features. The spatial gaze features are passed through two fully connected layers to obtain the yaw angle and pitch angle of the gaze. A multi-scale hole convolution block and a feature selection block are respectively embedded after the second, third and fourth convolution layers of the ResNet network, and an effective area attention block is embedded after the fifth convolution layer. The output features of the multi-scale hole convolution block are used as the input features of the next convolution layer. The output features of all feature selection blocks are concatenated with the output features of the fifth convolution layer as the input features of the effective area attention block, thereby obtaining an improved ResNet network. For the feature selection block, the input features are evenly divided into four subgroup features according to the number of channels, and the values ​​of each subgroup feature are calculated. The output features of the feature selection block are obtained according to equations (1) to (3); T mdn =T0+T1+T2-T min -S max (2) T min =Min[T0,T1,T2] (3) Where, T output is the output feature of the feature selection block, S max is the subgroup feature with the largest value, T mdn is the median subgroup characteristic, T0, T1, and T2 are the three subgroup characteristics except the one with the smallest value, T min It is the subgroup characteristic with the smallest value among the remaining three subgroup characteristics; For the effective area attention block, the input features are average pooled for the features of each channel in the horizontal and vertical directions respectively to obtain horizontal global pooling features and vertical global pooling features; at the same time, the input features are max-pooled for the features of each channel in the horizontal and vertical directions respectively to obtain horizontal max-pooling features and vertical max-pooling features; the horizontal global pooling features and the horizontal max-pooling features are concatenated to obtain one-dimensional horizontal features; the vertical global pooling features and the vertical max-pooling features are concatenated to obtain one-dimensional vertical features; the one-dimensional horizontal features and the one-dimensional vertical features are respectively subjected to one-dimensional convolution, group normalization and activation operations, and then multiplied with the input features to obtain the output features of the effective area attention block. Step 3: Use the face image in the first step to train the gaze estimation model, and use the trained gaze estimation model for human eye gaze estimation.

2. The method for estimating human sight lines based on appearance according to claim 1, characterized in that: The multi-scale atrous convolution block uses multiple branches to extract multi-scale features, splices the output features of all branches, and after convolution, adds the spliced ​​features element by element with the input features of the multi-scale atrous convolution block to obtain the output of the multi-scale atrous convolution block.

3. The method for estimating human sight lines based on appearance according to claim 2, characterized in that: The multi-scale dilated convolution block includes four branches, the first branch includes a convolution with a size of 3×3 and a dilation rate of 1, the second branch includes a convolution with a size of 3×1 and a dilation rate of 1 and a dilated convolution with a size of 3×3 and a dilation rate of 3, the third branch includes a convolution with a size of 1×3 and a dilation rate of 1 and a dilated convolution with a size of 3×3 and a dilation rate of 3, and the fourth branch includes a convolution with a size of 1×3 and a dilation rate of 1, a convolution with a size of 3×1 and a dilation rate of 1, and a dilated convolution with a size of 3×3 and a dilation rate of 5.

4. The method for estimating human sight lines based on appearance according to any one of claims 1 to 3, characterized in that: The ResNet network is one of ResNet-18, ResNet-34 and ResNet-50.

Images