A line-of-sight estimation method and system based on convolutional neural network and self-attention model

Abstract

The application provides a gaze estimation method and system based on a convolutional neural network and a self-attention model; a neural network is built to process face images and eye images, so that high-precision yaw angles and pitch angles of the gaze direction can be regressed in an unconstrained environment; in view of the problem that an ordinary convolutional neural network cannot capture fine-grained features of the eye images, the ResNet neural network model is modified, the down-sampling layer is removed, and an expanded convolution kernel is added to the neural network; visual features extracted from the eye images and the face images are input into a visual Transformer network model, a multi-head self-attention mechanism is used to calculate the correlation between coarse-grained features and fine-grained features; and the gaze angle regression process is improved, so that the yaw angle and the pitch angle of the gaze are regressed from the full connection layer respectively, and the precision of the gaze estimation is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a gaze estimation method and system based on a convolutional neural network and a self-attention model. BACKGROUND

[0002] The eye provides important visual information for people's life and is an indispensable information collector for human beings. Compared with language and body movements, the gaze is a relatively small but more direct information, which contains rich intention information of human beings. Gaze estimation refers to a process of determining the position of the gaze by analyzing the eye movement and gaze direction of a person. The gaze information captured by the gaze estimation method can provide a new interaction mode for human-computer interaction. This method enables the computer to better understand the intention and needs of human beings, and provides more personalized services and support. In addition, it can also help some disabled people overcome interaction barriers and improve the quality of life.

[0003] In existing research, the gaze estimation method under laboratory conditions has reached high accuracy, but these methods require a large number of high-precision cameras and infrared light sources and other hardware facilities, and the head of the test personnel cannot be freely rotated. These methods cannot be applied in mobile and portable devices. Ideal human-computer interaction is human-computer interaction in an unconstrained environment, so it is of great significance to study the improvement of gaze estimation accuracy in an unconstrained environment.

[0004] The existing mainstream gaze estimation methods are divided into contact type and non-contact type. The contact type gaze estimation method includes mechanical recording method, electrooculography method, contact lens method, and sclera search coil method. These methods need to directly contact the human eye, which will bring discomfort to the user.

[0005] With the development of computer technology, imaging technology and infrared technology, non-contact gaze estimation technology has developed rapidly. It can reduce the interference to the user while maintaining high accuracy, and has become the mainstream technology. The non-contact method is divided into feature-based method, model-based method and appearance-based method. The feature-based method and the model-based method both need to detect the key positions of the face image and the eye image. The estimation accuracy of the gaze angle is highly related to the number of detected feature points, so multiple infrared cameras with high resolution and multiple infrared light sources are needed to collect face images and eye images. Since these two methods have strict requirements for gaze estimation conditions, they cannot handle complex image changes and can only be used in dedicated scenes and hardware conditions.

[0006] In the appearance-based method, when using traditional machine learning algorithms such as linear regression (Linear Regression, LR), random forest (RandomForest, RF), Gaussian process (Gaussion Process, GP) and support vector machine (Support Vector Machine, SVM) and the like for gaze estimation tasks, if there are large angle head movements, illumination changes and the like affecting the appearance of the eye, the accuracy of gaze estimation will be greatly reduced, which is not suitable for use in unconstrained environments and mobile and portable devices. For gaze estimation tasks, the integrity of eye features is crucial, and in existing gaze estimation methods using deep learning, the feature extraction network used reduces the size of the input image through the pooling layer to improve the network visual receptive field, but this will cause the loss of eye features and cannot extract complete eye fine-grained features. In addition, in addition to eye features, there are also features on the face that can help improve the accuracy of gaze estimation, and existing methods simply concatenate face features and eye features, which cannot fully utilize the features in the face image that are beneficial to the gaze estimation task.

[0007] In the field of gaze estimation, the mainstream using deep learning methods is to use convolutional neural network (CNN) as the backbone network. Traditional CNN models have some limitations in gaze estimation tasks. CNN models extract features through local perception convolution operations, which makes the model have weak global information perception ability for the whole image, and cannot fully capture the subtle relationship and context information between the eyes and the face, nor can it capture the fine-grained features of the input image, thereby affecting the accuracy of gaze estimation. At the same time, CNN models usually use fixed convolution structure and input size. In the gaze estimation task, different environments and devices may produce images with different resolutions, which leads to the need for additional preprocessing or adjustment of the model structure when CNN deals with eye and face images with different resolutions, which increases the difficulty of applying the model to the actual environment. When using large-scale data sets for training, CNN networks are prone to overfitting, which affects the generalization of the gaze estimation model.

[0008] The visual Transformer model is rarely applied as an innovative method in the gaze estimation task. The method uses the visual Transformer model as the backbone network of the gaze estimation model, overcoming the limitations in the CNN model. The method uses CNN as a feature extraction network instead of the Patch Embedding layer in the visual Transformer, uses the self-attention mechanism of the visual Transformer, can make the model extract the global and local features of the input image, and thus better capture the fine-grained features of the eye image and the face image. The self-attention mechanism processing of the left and right eye images and the face image can improve the ability of the model to establish the correlation between the face image and the eye image in the gaze estimation task, thereby improving the accuracy and robustness of the gaze estimation task. The visual Transformer model can flexibly process input images of different resolution sizes, so that the model can cope with the gaze estimation task under different devices and environmental conditions, and is more adaptable. Based on the characteristics of the visual Transformer model, when a large-scale data set is used for training, the generalization of the model can be improved, and the overfitting phenomenon is not prone to occur. SUMMARY

[0009] The present application aims at the problem of insufficient fine-grained feature capture of eye images by ordinary convolutional neural networks, and proposes a gaze estimation method and system based on convolutional neural networks and self-attention models, which regresses the angle value of the gaze from the appearance image. The existing public data set is processed, the image is divided into three parts of face image, left eye image and right eye image, and different neural networks are used for feature extraction, and the extracted features are input into the visual Transformer network for processing, and finally the yaw angle and pitch angle of the gaze are regressed from the full connection layer.

[0010] The present application is realized by the following technical solutions:

[0011] A gaze estimation method based on convolutional neural networks and self-attention models:

[0012] The method specifically comprises the following steps:

[0013] Step 1: Select the data set according to the requirements of the use scene;

[0014] Step 2: Expand the data set using the gaze redirection method;

[0015] Step 3: Perform pose detection and normalization processing on the data set;

[0016] Step 4: Modify the ResNet network based on the features of the human eye and the features of the face, respectively;

[0017] Step 5, the extracted face and eye features are input into the visual Transformer model for global self-attention mechanism processing;

[0018] Step 6, the gaze angle is regressed to obtain the gaze direction.

[0019] Further, in step 1, the MPIIFaceGaze dataset and the Gaze360 dataset are selected, wherein the MPIIFazeGaze dataset collects images using the camera of a notebook computer, including eye images and full-face images, and is attached with artificial labeling; the Gaze360 dataset includes full-body images and cropped images of the face region, as a supplement to the MPIIFaceGaze dataset;

[0020] According to the three Euler angles of the rotation of an object in space, the head posture angle and the gaze direction angle are named;

[0021] In the head posture angle, the Yaw angle (yaw angle) is used as the angle of left and right shaking of the head, the Pitch angle (pitch angle) is used as the angle of up and down tilting of the head, and the Roll angle (roll angle) is used as the angle of rolling movement of the head;

[0022] The Yaw angle (yaw angle) is used as the angle of left and right movement of the eyes, and the Pitch angle (pitch angle) is used as the angle of up and down movement of the eyes.

[0023] Further, in step 2, the gaze is generated by using the Glow generation model based on the Normalizing Flows;

[0024] The affine coupling layer in the Glow model can complete the encoding transformation F of the input image I0 and the inverse transformation F -1 Process, decouple the eye image and the gaze angle, and map to the feature space to generate z0, z0 is the encoding of the decoupled eye image; the probability density function of the mapping process is shown in formula (2-1):

[0025]

[0026] Where p z (z) is the basic probability distribution, x is the input value, c is the conditional value, q is the parameter of the mapping function, is the Jacobian determinant of z with respect to x and c, f q (x, c) is the process of decoupling the eye image and the gaze angle z = f q (x, c);

[0027] In the affine coupling layer, a conditional layer is additionally added, and the conditional vector c = [E YawE Pitch H Yaw H Pitch V]input into Glow model, where E Yaw and E Pitch represent the yaw and pitch angles of eye gaze, H Yaw and H Pitch represent the yaw and pitch angles of head pose, and V represents the position of the eye.

[0028] Further, the line-of-sight redirection method specifically comprises:

[0029] After inputting the original image I0 into the Glow model, the image scale is first adjusted by a compression layer (SqueezeLayer) to convert the size (HxWxC) of the original image into Next, the ActNorm layer is used to perform zero-mean normalization and linear transformation on the data of each channel input.

[0030] The formula for zero-mean normalization is shown in (2-2):

[0031]

[0032] where x is the input data, x scale is the normalized data, x mean is the mean of the data, and s is the standard deviation of the data; the formula for linear transformation is shown in (2-3):

[0033] y = s1x + b1 (2-3)

[0034] where s1 is the scaling weight, i.e., the Jacobian matrix, y, x, and b1 are vectors with the same length and channel size, x is the input of each layer, y is the output of each layer, and b1 is the bias;

[0035] After passing through the reversible 1x1 convolution, the random channel arrangement order is learned; in the affine coupling layer, the input x is split into x1 and x2 according to the channel number, and the two parts have the same size. x1 and the condition vector are input into the neural network for processing, and the neural network outputs a scaling weight s2 and a bias b2 to convert x2 into z2, the original x1 remains unchanged, and z2 is directly concatenated with the feature as the output z0, as shown in formula (2-4):

[0036] z2 = exp(G q ,s2(x1,g(c)))x2 + G q ,b2(x1,g(c))

[0037] z0 = concat(x1, z2) (2-4)

[0038] where G q is an arbitrary neural network, g is a conditional encoder;

[0039] Finally z0 is combined with the output conditional variable c j Inverse transformation to generate an image I with a new view angle j , the process is shown in equation (2-5):

[0040] I j = F -1 (z0, c j ) (2-5)

[0041] Further, in step 3, specifically comprising the following steps:

[0042] Step 3.1, pose detection on the data set: head pose estimation using CNN network;

[0043] According to the head coordinate system (HCS), take the center points of the left and right eyes and the center point of the lips as the basis points of the plane, and set the position between the two eyes as the origin, the direction from the center of the two eyes to the left eye in the image as the positive direction of the X H axis, the direction from the origin to the lips and perpendicular to the X H axis as the Y H axis, and the Z H axis according to the right-hand rule, use the face detection module in the dlib library in Python to detect the face and the feature points of the face, and detect 68 feature points on the face,

[0044] After the feature points are detected, six key points are selected for feature fitting,

[0045] Use the Hopenet network based on ResNet to extract the features in the face image, use three separate losses to classify the yaw angle, pitch angle and roll angle based on the softmax layer and cross-entropy loss, get the classification error, calculate the expected value of each angle output, and calculate the regression error by comparing the expected value with the actual value, and the final total error is equal to the weight sum of the classification error and the regression error;

[0046] Step 3.2, normalize the data set: the normalization of image data is completed through coordinate system rotation, scale transformation and cropping:

[0047] The coordinate system rotation refers to rotating the virtual camera coordinate system so that the Z axis of the virtual camera coordinate system is directed to the center of the eye image or the center of the face image. The X axis, Y axis and Z axis of the virtual camera coordinate system before rotation are X C , Y C and Z C respectively. The yaw angle, pitch angle and roll angle of the head posture are obtained through the above head posture detection network, and the matrix R H of the head posture is obtained through conversion.

[0048] When the eye is taken as the center, the X axis of the R H matrix, i.e. the first column vector of the matrix, is the X axis vector r X of the virtual camera rotation matrix R, and the eye center after rotation is on the Z axis of the camera coordinate system, so the unit vector of the eye position is used as the Z axis vector r Z of the rotation matrix R. According to the right-hand rule, the Y axis vector r Y of the rotation matrix R is r X × r Z . Finally, the three direction vectors are unitized to obtain the rotation matrix:

[0049] R = [r X r Y r Z ] (3-1)

[0050] The X axis, Y axis and Z axis of the virtual camera coordinate system after rotation are , Y and Z respectively. The scale transformation adjusts the image scale through the scale matrix S n so that the distance between the camera Z axis direction and the center of the eye or face is a fixed distance d s . The scale matrix S n is:

[0051]

[0052] wherein d is the distance between the center position of the eye or face and the original point of the camera.

[0053] The original camera intrinsic matrix K r and the virtual intrinsic matrix K s are introduced. The intrinsic matrix K is:

[0054]

[0055] wherein f x and f y are the focal lengths in pixels, c x and c y are the principal points.represents the offset of the camera optical axis in the image coordinate system, and the focal length and offset of the original image intrinsic matrix are provided in the data set; the transformation matrix M is represented as:

[0056]

[0057] Finally, the portrait image is transformed and cropped to obtain the eye image and the face image; the original gaze direction vector g i also needs to be calculated after corresponding normalization transformation to the converted virtual camera coordinate system, that is, g i ' = Mg i , and the inverse transformation is performed from the virtual camera coordinate system to the camera coordinate system during gaze estimation, g i ' is the converted gaze direction vector.

[0058] Further, in step 4, the ResNet-18 network pre-trained on ImageNet is used to extract the face image features:

[0059] The dilated convolution network based on the improved ResNet-18 network is used to extract the eye image features.

[0060] Further, in step 5, the gaze estimation is processed as a regression task, and the face features and eye features extracted by the CNN method are input into the Transformer to obtain the global relationship of the features; the image is restored to an N-dimensional feature matrix Token of HxW, and a 1*1 N-dimensional feature mark T m is embedded in each Token to realize the self-attention mechanism aggregation between each Token;

[0061] That is, an image I is input, and the corresponding features are extracted by the CNN to obtain a feature matrix T I The image feature matrix and the feature mark matrix are concatenated to obtain the final output matrix T O .

[0062] T O is represented as: T O = [T I T m ] (5-1).

[0063] Further, in step 6, the feature matrix T O is processed by the Encoder layer of the visual Transformer model to obtain the output feature vector of the model, and the multi-layer perception MLP is used to regress the gaze angle, and the l1loss is used as the loss function of the gaze estimation, and the regression of the gaze angle g f is represented as follows:

[0064] gf MLP(E n (T O ))(6-1)

[0065] wherein E n represents an Encoder layer in the Transformer model;

[0066] The preprocessed image dataset and label file are input into the built feature extraction network and feature fusion network for parameter training, and the trained network parameters are saved;

[0067] In the final line-of-sight regression process, the yaw angle and pitch angle of the line-of-sight are separately regressed to improve the regression accuracy; Finally, the actual predicted angle of the line-of-sight is obtained through inverse normalization processing.

[0068] A line-of-sight estimation system based on a convolutional neural network and a self-attention model:

[0069] The line-of-sight estimation system comprises a dataset, a line-of-sight redirection module, a preprocessing module, a feature extraction module, a global self-attention mechanism processing module and a regression module;

[0070] The line-of-sight redirection module uses a line-of-sight redirection method to expand the dataset;

[0071] The preprocessing module is used for posture detection and normalization processing of the dataset;

[0072] The feature extraction module respectively modifies the ResNet network based on human eye features and human face features;

[0073] The global self-attention mechanism processing module transmits the extracted human face and human eye features into a visual Transformer model for global self-attention mechanism processing;

[0074] The regression module regresses the line-of-sight angle to obtain the line-of-sight direction.

[0075] An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the above method when executing the computer program.

[0076] A computer-readable storage medium for storing computer instructions, the computer instructions being executed by a processor to implement the steps of the above method.

[0077] Advantages of the present application

[0078] The present application can process face images and eye images by building a neural network, and can regress high-precision line-of-sight direction yaw angle and pitch angle in an unconstrained environment.

[0079] In view of the problem that the general convolutional neural network is insufficient in capturing fine-grained features of eye images, the ResNet neural network model is modified, the down-sampling layer (pooling layer) is removed, and an expanded convolution kernel is added to the neural network; the receptive field of the network is improved without increasing the parameter amount, and the extraction of the detailed features of the eye image is more complete; the visual features extracted from the eye image and the face image are input into the visual Transformer network model, the face features and the eye features are fused by the self-attention mechanism, the multi-head self-attention mechanism is used to calculate the correlation between the coarse-grained features and the fine-grained features, and the face features and the eye features are fused by the self-attention mechanism; the gaze angle regression process is improved, the yaw angle and the pitch angle of the gaze are respectively regressed from the full connection layer, and finally the yaw angle and the pitch angle are respectively output, so that the precision of the gaze estimation is improved.

[0080] The face image and the eye image are input into the network, the appearance-based gaze estimation task under unconstrained conditions can be completed, the gaze estimation angle is regressed, no additional hardware device is needed, only a common computer camera can complete the gaze estimation task, and the precision of the gaze estimation is improved.

[0081] The visual redirection method is used for data set expansion, so that the visual Transformer model is trained using a large number of expanded image data sets, the model has higher prediction precision for unobserved samples in different scene and condition gaze estimation tasks, and the transferability and cross-data set prediction accuracy of the model are improved. BRIEF DESCRIPTION OF DRAWINGS

[0082] Figure 1 The method flowchart of the present application is shown in the figure;

[0083] Figure 2 The head posture angle in the space of the present application is shown in the figure;

[0084] Figure 3 The gaze redirection flowchart of the present application is shown in the figure;

[0085] Figure 4 The data set preprocessing flowchart of the present application is shown in the figure;

[0086] Figure 5 The head coordinate system (HCS) of the present application is shown in the figure;

[0087] Figure 6 The face feature point detection result of the present application is shown in the figure;

[0088] Figure 7 The head posture estimation network of the present application is shown in the figure;

[0089] Figure 8For the virtual camera coordinate system rotation and scale transformation process of the application;

[0090] Figure 9 For the ResNet-18 network;

[0091] Figure 10 For the line of sight estimation flowchart. DETAILED DESCRIPTION

[0092] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0093] In combination Figures 1 to 10 .

[0094] A line of sight estimation method based on a convolutional neural network and a self-attention model, specifically comprising the following steps:

[0095] Step 1: Select a data set according to the requirements of the use scenario;

[0096] Step 2: Expand the data set using the line of sight redirection method;

[0097] Step 3: Data set preprocessing, pose detection and normalization processing of the data set;

[0098] Step 4: ResNet network is respectively modified based on human eye features and human face features;

[0099] Step 5: The extracted human face and human eye features are transmitted into the visual Transformer model for global self-attention mechanism processing;

[0100] Step 6: Regress the line of sight angle to obtain the line of sight direction.

[0101] In step 1, MPIIFaceGaze data set and Gaze360 data set are selected, wherein the MPIIFazeGaze data set collects images using the camera of a notebook computer, including eye images and full face images, and is attached with artificial labeling; the Gaze360 data set includes full body images and cropped images of the face region, serving as a supplement to the MPIIFaceGaze data set;

[0102] According to the requirements of the use scene, the MPIIFaceGaze dataset and the Gaze360 dataset are selected for training and testing. The MPIIFaceGaze dataset is an extension of the public dataset MPIIGaze commonly used in the field of gaze estimation, which is collected by 15 participants including 6 females, 9 males, and 5 people wearing glasses, using the camera of a notebook computer in daily life and work, including 213659 double-eye images and 4500 full-face images, and manually annotated; however, since the MPIIFaceGaze dataset is collected by a computer camera, the gaze angle and head posture angle are limited, so the Gaze360 dataset is selected for supplementation. The Gaze360 dataset collects 172000 images of 238 participants, including full-body images and cropped images of the face region, which has a wide range of head posture, gaze angle, indoor and outdoor scenes, gender and sample changes, and can be used to train and test the algorithm model for gaze estimation of any posture in an unconstrained environment.

[0103] The head posture angle and the gaze direction angle are named according to the three Euler angles of the rotation of the object in space;

[0104] As shown in Figure 2 , in the head posture angle, the Yaw angle (yaw angle) is used as the angle of left and right shaking of the head, the Pitch angle (pitch angle) is used as the angle of up and down tilting of the head, and the Roll angle (roll angle) is used as the angle of rolling movement of the head;

[0105] The naming of the head posture angle is also referred to the naming of the gaze angle, the Yaw angle (yaw angle) is used as the angle of left and right movement of the eyes, and the Pitch angle (pitch angle) is used as the angle of up and down movement of the eyes.

[0106] In step 2, training the gaze estimation model using a large number of samples can improve the accuracy of estimation, and the present application uses a generation model Glow based on standardization flow (Normalizing Flows) to generate the redirection of the gaze;

[0107] The affine coupling layer in the Glow model can complete the encoding transformation F and inverse transformation F -1 of the input image I0, decouple the eye image and the gaze angle, and map to the feature space to generate z0, which is the encoding of the decoupled eye image. The probability density function of the mapping process is shown in formula (2-1):

[0108]

[0109] where p z(z) is a base probability distribution, x is an input value, c is a conditional value, q is a parameter of a mapping function, is a Jacobian of z with respect to x and c, f q (x, c) is a process of decoupling eye image and gaze angle, z = f q (x, c).

[0110] In the affine coupling layer, a conditional layer is additionally added, and the conditional vector c = [E Yaw E Pitch H Yaw H Pitch V] related to the gaze is input into the Glow model, where E Yaw and E Pitch represent the yaw angle and the pitch angle of the eye fixation, H Yaw and H Pitch represent the yaw angle and the pitch angle of the head pose, and V represents the position of the eye.

[0111] The flowchart of the gaze redirection is shown in Figure 2 .

[0112] where c I0 represents the conditional vector of the input image, c j represents the conditional vector of the pre-generated image, and I j is the generated eye image with a new gaze angle.

[0113] The gaze redirection method is specifically as follows:

[0114] After the original image I0 is input into the Glow model, the image scale is first adjusted through the compression layer (SqueezeLayer) to convert the size (H x W x C) of the original image into Next, the ActNorm layer is used to perform zero-mean normalization and linear transformation on the data of each channel input.

[0115] The formula of the zero-mean normalization is shown in (2-2):

[0116]

[0117] where x is the input data, x scale is the normalized data, x mean is the mean of the data, and s is the standard deviation of the data. The formula of the linear transformation is shown in (2-3):

[0118] y = s1 x + b1 (2-3)

[0119] where s1 is the scaling weight, That is, the Jacobian matrix, y, x, b1 are vectors with the same length and channel size, x is the input of each layer, y is the output of each layer, and b1 is the bias.

[0120] After the reversible 1x1 convolution, the random channel arrangement order is learned. In the affine coupling layer, the input x is split into x1 and x2 according to the channel number, and the two parts have the same size. x1 and the condition vector c I0 are input into the neural network for processing. The neural network outputs a scaling weight s2 and a bias b2 to convert x2 into z2, and the original x1 remains unchanged and is directly concatenated with z2 as the output z0. The process is shown in equation (2-4):

[0121] z2 = exp(G q ,s2(x1,g(c)))·x2+G q ,b2(x1,g(c))

[0122] z0 = concat(x1,z2) (2-4)

[0123] where G q is an arbitrary neural network, and g is a conditional encoder.

[0124] Finally, z0 is combined with the output condition variable c j to generate an image I j with a new view angle through inverse transformation. The process is shown in equation (2-5):

[0125] I j = F -1 (z0,c j ) (2-5)

[0126] In step 3, the following steps are included:

[0127] Step 3.1, pose detection on the dataset: head pose estimation using a CNN network.

[0128] As Figure 5 shown in the head coordinate system (HCS), take the center points of the left and right eyes and the center point of the lips as the basis points of the plane, and set the position between the two eyes as the origin. The direction from the center of the two eyes to the left eye in the image is the positive direction of the X H axis, the direction from the origin to the lips and perpendicular to the X H axis is the positive direction of the Y H axis, and finally the Z H axis is determined according to the right-hand rule, i.e., place the right hand at the origin, with the thumb, index finger and middle finger forming a right angle, and point the thumb in the positive direction of the X H axis, and point the index finger in the positive direction of the Y HThe positive direction of the axis, the direction indicated by the middle finger is Z H The positive direction of the axis. The face detection and face feature point detection are performed by using the face detection module in the dlib library in the Python language, 68 feature points on the face are detected, and the detection result is as shown in Figure 6

[0129] After the feature points are detected, six key points are selected for feature fitting. This time, the tip of the nose (0.0, 0.0, 0.0), the lower jaw (0.0, -330.0, -65.0), the left eye corner (-225.0, 170.0, -135.0), the right eye corner (225.0, 170.0, -135.0), the left mouth corner (-150.0, -150.0, -125.0), and the right mouth corner (150.0, -150.0, -125.0) are selected. The six coordinate points are fitted with a general 3D face model.

[0130] The Hopenet network based on ResNet is used to extract the features in the face image. Three separate losses are used to classify the yaw angle, pitch angle, and roll angle based on the softmax layer and cross-entropy loss. The classification error is obtained, and the expected value of each angle output is calculated. The expected value and the actual value are calculated by the mean square error to obtain the regression error. The final total error is equal to the weight sum of the classification error and the regression error. The head pose estimation process is as shown in Figure 7

[0131] Step 3.2, normalize the data set: the normalization of image data is completed by coordinate system rotation, scale transformation and cropping:

[0132] The rotation transformation and scale transformation process of the virtual coordinate system is as shown in Figure 8 , the coordinate system rotation refers to rotating the virtual camera coordinate system so that the Z axis of the virtual camera coordinate system is perpendicular to the eye image center or the face image center. The X axis, Y axis and Z axis of the virtual camera coordinate system before rotation are X C , Y C and Z C . The yaw angle, pitch angle and roll angle of the head pose are obtained by the above head pose detection network, and the matrix R H of the head pose is obtained by conversion.

[0133] When the eye is taken as the center, the X axis of the R H matrix, i.e. the first column vector of the matrix, is the X axis vector r X of the virtual camera rotation matrix R, and the eye center after rotation is on the Z axis of the camera coordinate system, so the unit vector of the eye position is used as the Z axis vector r Z ​​, and finally according to the right-hand rule, the Y-axis vector of the rotation matrix R is r Y X Z Finally, the three direction vectors are unitized to obtain the rotation matrix:

[0134] R = [r X r Y r Z ] (3-1)

[0135] The X-axis, Y-axis and Z-axis of the virtual camera coordinate system after rotation are and The scale transformation adjusts the image scale through the scale matrix S n to make the distance between the camera Z-axis direction and the eye or face center point a fixed distance d s , and the scale matrix S n is:

[0136]

[0137] where d is the distance between the eye or face center position and the camera origin.

[0138] Next, in order to apply the same perspective transformation to the planar image, overcome the problem of different camera focal lengths and different eye distances, the original camera intrinsic matrix K r and the virtual intrinsic matrix K s are introduced, and the intrinsic matrix K is:

[0139]

[0140] where f x and f y are the focal lengths in pixels, c x and c y represent the offset of the camera optical axis in the image coordinate system, and the parameter values of the focal length and offset of the original image intrinsic matrix provided in the data set. The transformation matrix M is represented as:

[0141]

[0142] Finally, the function cv2.warpPerspective in the OpenCV library is used to transform and crop the portrait image, and in the present application, the distance d s of the virtual camera is set to 600 mm, the focal length f x and f y are set to 960, c x and c y are 30 and 18 pixels respectively, to obtain an eye image of 36x60 pixels and a face image of 224x224 pixels. The original gaze direction vector g i ​​Also, the corresponding normalization transformation to the converted virtual camera coordinate system is performed for calculation, that is, g i ' = Mg i In the line-of-sight estimation, the inverse transformation is performed to convert from the virtual camera coordinate system to the camera coordinate system, g i ' is the converted line-of-sight direction vector.

[0143] In step 4, the face image features are extracted using the ResNet-18 network pre-trained on ImageNet:

[0144] The structure of the face feature extraction network of the application is as shown in the figure Figure 9 Specifically, the face image with a size of 3*224*224 is input, the input image enters the ConvNet1(7*7, 64) layer, the ConvNet1(7*7, 64) layer is connected with the Max pool(2*2) layer, the Max pool(2*2) layer is connected with the ConvNet2_1(3*3, 64)*2 layer, the ConvNet2_1(3*3, 64)*2 layer is connected with the ConvNet2_2(3*3, 64)*2 layer, the ConvNet2_2(3*3, 64)*2 layer is connected with the ConvNet3_1(3*3, 128)*2 layer, the ConvNet3_1(3*3, 128)*2 layer is connected with the ConvNet3_2(3*3, 128)*2 layer, the ConvNet3_2(3*3, 128)*2 layer is connected with the ConvNet4_1(3*3, 256)*2 layer, the ConvNet4_1(3*3, 256)*2 layer is connected with the ConvNet4_2(3*3, 256)*2 layer, the ConvNet4_2(3*3, 256)*2 layer is connected with the ConvNet5_1(3*3, 512)*2 layer, the ConvNet5_1(3*3, 512)*2 layer is connected with the ConvNet5_2(3*3, 512)*2 layer, the ConvNet5_2(3*3, 512)*2 layer outputs a feature image with a size of 7*7*512, an Avgpooling layer with a 1*1 convolution kernel is used to reduce the dimension of the feature, and finally a feature map is output.

[0145] The dilated convolution network based on the ResNet-18 network is used to extract the features of the eye image:

[0146] Because the down-sampling layer and the pooling layer in the ResNet network can cause the loss of the detailed features of the image, the down-sampling layer and the pooling layer in the ResNet network are removed, eight layers at the end of the original ResNet network are added with expansion coefficients to improve the extraction ability of the network to the detailed features, and a reasonable structure is designed to eliminate the gridding artifacts of the dilated convolution network.

[0147] Specifically, a 36*60 eye image is connected with a Conv-1(7*7, 16) layer, then the Conv-1(7*7, 16) layer is connected with a Conv-2(3*3, 64)*2 layer, the Conv-2(3*3, 64)*2 layer is connected with a Conv-3(3*3, 128)*2 layer, the Conv-3(3*3, 128)*2 layer is connected with a DilatedConv1_2(3*3, 256)*2 layer, the DilatedConv1_2(3*3, 256)*2 layer is connected with a DilatedConv2_4(3*3, 512)*2 layer, the DilatedConv2_4(3*3, 512)*2 layer is connected with a DilatedConv3_2(3*3, 512)*2 layer, and finally the DilatedConv3_2(3*3, 512)*2 layer is connected with a DilatedConv4_1(3*3, 512)*2 layer. The expansion coefficients used in the above convolution layers are 2, 4, 2 and 1 respectively from the DilatedConv1 layer to the DilatedConv4 layer. All the convolution layers except the Dilatedconv3 layer and the Dilatedconv4 layer use the residual structure connection.

[0148] In step 5, the gaze estimation is processed as a regression task, and because the PatchEmbedding link of the visual Transformer model will slice the image, unlike the classification task, the cutting of the eye details will affect the regression task. Therefore, the PatchEmbedding link is removed, the face features and eye features extracted by the CNN method are input into the Transformer to obtain the global relationship of the features, the image is restored to an H*W N-dimensional feature matrix Token, and a 1*1 N-dimensional feature marker T m is embedded in each Token to realize the self-attention mechanism aggregation between each Token.

[0149] That is, an image I is input, the corresponding features are extracted by the CNN to obtain a feature matrix T I , and the image feature matrix and the feature marker matrix are concatenated to obtain a final output matrix T O .

[0150] TO is represented as: T O = [T I T m ](5-1).

[0151] In step 6, the feature matrix T O After processing through the Encoder layer of the visual Transformer model, the output feature vector of the model is obtained, and a multi-layer perception (MLP) is used to regress the gaze angle, with an l1loss used as the loss function for gaze estimation. The regression of the gaze angle g f is represented as follows:

[0152] g f = MLP(E n (T O ))(6-1)

[0153] where E n represents the Encoder layer in the Transformer model.

[0154] The preprocessed image dataset and label file are input into the built feature extraction network and feature fusion network for parameter training, and the trained network parameters are saved.

[0155] In the final gaze regression process, the yaw angle and pitch angle of the gaze are separately regressed to improve the accuracy of the regression. Finally, the actual predicted angle of the gaze is obtained after inverse normalization processing.

[0156] A gaze estimation system based on a convolutional neural network and a self-attention model:

[0157] The gaze estimation system includes a dataset, a gaze redirection module, a preprocessing module, a feature extraction module, a global self-attention mechanism processing module, and a regression module.

[0158] The gaze redirection module uses a gaze redirection method to expand the dataset.

[0159] The preprocessing module is used for pose detection and normalization processing of the dataset.

[0160] The feature extraction module modifies the ResNet network based on human eye features and human face features.

[0161] The global self-attention mechanism processing module transmits the extracted human face and human eye features into a visual Transformer model for global self-attention mechanism processing.

[0162] The regression module regresses the gaze angle to obtain the gaze direction.

[0163] An electronic device includes a memory storing a computer program and a processor implementing the steps of the above method when executing the computer program.

[0164] A computer readable storage medium for storing computer instructions, the computer instructions implementing the steps of the above method when executed by a processor.

[0165] The memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct Rambus RAM (DRRAM). It should be noted that the memory of the method described in the present application is intended to include, but not be limited to, these and any other suitable types of memory.

[0166] The above describes a line-of-sight estimation method and system based on a convolutional neural network and a self-attention model, and the principle and implementation of the present application are described. The above description of the embodiments is only used to help understand the method of the present application and its core idea. For those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A gaze estimation method based on convolutional neural networks and a self-attention model, characterized in that: The method specifically includes the following steps: Step 1: Select the dataset according to the needs of the use case; Step 2: Expand the dataset using gaze redirection methods; In step 2, the Glow generative model based on Normalizing Flows is used to redirect and generate the gaze. The affine coupling layer in the Glow model can perform input image processing. Encoding transformation Inverse Transform The process decouples the eye image from the gaze angle and maps it into the feature space to generate... , The decoupled eye image is encoded; the probability density function of the mapping process is shown in Equation (2-1): （2-1） in For the basic probability distribution, For input values, For conditional values, For the parameters of the mapping function, for about and Jacobian determinant, The process of decoupling eye images from the viewing angle ; Add an additional conditional layer to the affine coupling layer to handle the view-dependent conditional vectors. The input is fed into the Glow model, where and These represent the yaw angle and pitch angle at which the eye is looking, respectively. and These represent the yaw and pitch angles of the head posture, respectively, and V represents the position of the eyes; Step 3: Perform pose detection and normalization on the dataset; Step 4: Modify the ResNet network based on human eye features and facial features respectively; Step 5: Input the extracted face and eye features into the visual Transformer model for global self-attention processing; Step 6: Regress the line of sight angle to obtain the line of sight direction.

2. The line-of-sight estimation method according to claim 1, characterized in that: In step 1, the MPIIFazeGaze dataset and the Gaze360 dataset are selected. The MPIIFazeGaze dataset uses a laptop camera to collect images, including images of both eyes and the full face, with added manual annotations. The Gaze360 dataset includes full-body images and cropped images of facial regions, as a supplement to the MPIIFaceGaze dataset; The head posture angle and the line of sight angle are named according to the three Euler angles of the object's rotation in space; In head posture angles, the yaw angle is used as the angle of left and right head swaying, the pitch angle is used as the angle of up and down head pitching, and the roll angle is used as the angle of head rolling. Use the yaw angle, or yaw angle, as the angle for left-right eye movement, and the pitch angle, or pitch angle, as the angle for up-down eye movement.

3. The line-of-sight estimation method according to claim 2, characterized in that: The gaze redirection method is specifically as follows: When the original image is input into the Glow model Then, the image scale is first adjusted using a compression layer (Squeeze Layer) to reduce the original image size. Transform into Next, the ActNorm layer is used to perform zero-mean normalization and linear transformation on the data of each input channel. The formula for zero-mean normalization is shown in (2-2): （2-2） in For data input, For the normalized data, The mean of the data. Let be the standard deviation of the data; the formula for the linear transformation is shown in (2-3): （2-3） in To scale the weights, That is, the Jacobian matrix. , , It is a vector with the same length as the number of channels. For the input of each layer, For the output of each layer, This is the bias value; After a reversible 1×1 convolution, the random channel arrangement is learned; the input is then processed in an affine coupling layer. Split into based on the number of channels and The two parts that were split off have the same scale. With condition vector The input is fed into a neural network for processing, and the neural network will output a scaling weight. and bias Will Convert to ,Original remain unchanged and Directly concatenate features as output The process is shown in formula (2-4): （2-4） in For any neural network, It is a condition encoder; at last By combining the output condition variable Inverse transform generates an image with a new viewing angle. The process is shown in formula (2-5): (2-5)。 4. The line-of-sight estimation method according to claim 3, characterized in that: Step 3 specifically includes the following steps: Step 3.1, perform pose detection on the dataset: use a CNN network to estimate head pose; Using the head coordinate system (HCS), the center points of the left eye, right eye, and lips are taken as the base points of the plane. The origin is set at the midpoint between the eyes, and the direction from the center of the eyes to the left eye in the image is defined as... The positive direction of the axis points from the origin to the lips and is perpendicular to the axis. The direction of the axis is The positive direction of the axis is finally determined by the right-hand rule. The algorithm uses the face detection module from the dlib library in Python to perform face detection and facial feature point detection, detecting 68 feature points on the face. After detecting the feature points, six key points are selected for feature fitting. The features in the face image are extracted using the ResNet-based Hopenet network. Three separate losses are used to classify the yaw angle, pitch angle and roll angle based on the softmax layer and cross-entropy loss respectively, and the classification error is obtained. The expected value of the output for each angle is calculated. The mean square error of the expected value and the actual value is calculated to obtain the regression error. The final total error is equal to the weighted sum of the classification error and the regression error. Step 3.2, normalize the dataset: Image data normalization is accomplished through three parts: coordinate system rotation, scale transformation, and cropping. The coordinate system rotation refers to rotating the virtual camera coordinate system so that the virtual camera coordinate system... The axis is aligned with the center of the eye image or the center of the face image; the virtual camera coordinate system before rotation. axis, shaft and The axes are respectively , and The head pose detection network described above is used to obtain the yaw, pitch, and roll angles of the head pose. These angles can then be transformed to obtain the head pose matrix. ; When the eye is the center point, take Matrix The axis, i.e., the first column vector of the matrix, is the virtual camera rotation matrix. of Axis vectors The rotated eye center is in the camera coordinate system. Since it's on the axis, the unit vector of the eye position is used as the rotation matrix. of Axis vectors Finally, according to the right-hand rule, the rotation matrix... of The axis vector is Finally, normalizing the three direction vectors yields the rotation matrix: (3-1) The rotated virtual camera coordinate system axis, shaft and The axes are respectively , and ; Scaling transformation is performed through the scaling matrix Adjusting the image scale to the camera The axial distance from the center point of the eye or face is a fixed distance. Scale matrix for: （3-2） in The distance from the center of the eye or face to the camera origin; Introducing the original camera intrinsic parameter matrix and virtual intrinsic parameter matrix Intrinsic parameter matrix for: （3-3） in and Focal length is measured in pixels. and This represents the offset of the camera's optical axis in the image coordinate system. The dataset provides the focal length and offset parameter values ​​of the original image's intrinsic parameter matrix; the transformation matrix... Represented as: (3-4) Finally, the portrait image is transformed and cropped to obtain the eye image and face image; the original gaze direction vector... It also needs to undergo a corresponding normalization transformation to be calculated in the transformed virtual camera coordinate system, that is... During line-of-sight estimation, an inverse transformation is performed to convert from the virtual camera coordinate system to the camera coordinate system. This is the transformed line-of-sight vector.

5. The line-of-sight estimation method according to claim 4, characterized in that: In step 4, the ResNet-18 network pre-trained on ImageNet is used to extract facial image features: We use a dilated convolutional network based on a modified ResNet-18 network to extract features from human eye images.

6. The line-of-sight estimation method according to claim 5, characterized in that: In step 5, gaze estimation is treated as a regression task. The facial and eye features extracted using a CNN method are input into a Transformer to obtain the global relationship of the features; the image is then restored to its original state. of A 1x1 feature matrix is ​​generated for each token, and a 1x1 feature matrix is ​​embedded in each token. Dimensional feature labeling This enables the aggregation of tokens through a self-attention mechanism. That is, input an image The feature matrix is ​​obtained by extracting the corresponding features through CNN. The final output matrix is ​​obtained by concatenating the image feature matrix and the feature label matrix. ; Represented as: (5-1).

7. The line-of-sight estimation method according to claim 6, characterized in that: In step 6, the feature matrix After processing by the encoder layer of the visual Transformer model, the output feature vector is obtained. A multilayer perceptron (MLP) is then used to regress the gaze angle. The loss function is used for line-of-sight estimation, where the line-of-sight angle is the loss function. The regression is represented as follows: （6-1） in This represents the Encoder layer in the Transformer model; The preprocessed image dataset and label files are input into the constructed feature extraction network and feature fusion network for parameter training, and the trained network parameters are saved. In the final line-of-sight regression process, the yaw angle and pitch angle of the line of sight are regressed separately to improve the accuracy of the regression; finally, the actual predicted angle of the line of sight is obtained through inverse normalization.

8. A gaze estimation system based on convolutional neural networks and a self-attention model, characterized in that: The system is used to perform the gaze estimation method based on convolutional neural networks and self-attention models as described in any one of claims 1 to 7; The gaze estimation system includes a dataset, a gaze redirection module, a preprocessing module, a feature extraction module, a global self-attention mechanism processing module, and a regression module; The gaze redirection module uses gaze redirection methods to expand the dataset; The preprocessing module is used to perform pose detection and normalization on the dataset; The feature extraction module modifies the ResNet network based on human eye features and facial features respectively; The global self-attention mechanism processing module inputs the extracted face and eye features into the visual Transformer model for global self-attention mechanism processing; The regression module regresses the viewing angle to obtain the viewing direction.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images