A fine-grained human eye gaze estimation method based on text information

Abstract

The application discloses a fine-grained human eye gaze estimation method based on text information, and the gaze estimation model is divided into two parts of face coarse-grained feature extraction and eye fine-grained feature extraction. The face coarse-grained feature extraction part comprises an improved CLIP module and a Transformer encoder, and the eye fine-grained feature extraction part comprises a feature extraction network, a channel attention module and a hybrid stripe pooling block. The face image is input into the improved CLIP module to extract text information reflecting the approximate gaze direction. The text information reflecting the approximate gaze direction is input into the Transformer encoder to obtain the face coarse-grained feature. The eye fine-grained feature extraction part comprises the feature extraction network, the channel attention module and the hybrid stripe pooling block. The left and right eye images are subjected to the feature extraction network to obtain the left and right eye features, the channel attention module gives the left and right eye features channel attention weights, and then the eye fine-grained feature is obtained. The hybrid stripe pooling block is used for dimension reduction processing of the eye fine-grained feature. Finally, the face coarse-grained feature and the eye fine-grained feature are fused to obtain a gaze estimation vector. The text information is fully utilized to guide the representation of the context information, the effective fusion of the two kinds of granularity features is realized, and the estimation accuracy is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and gaze estimation, and particularly relates to a fine-grained human eye gaze estimation method based on text information. BACKGROUND

[0002] Gaze contains rich human intention, emotion and other information, and can be used to deeply understand human cognition and behavior. Gaze estimation is an important research direction in the field of computer vision, and has been widely applied in many fields such as human-computer interaction, head-mounted devices, and assisted driving.

[0003] In recent years, gaze estimation based on deep learning has made great progress. Its input can be a face image, an eye image or a combination of the two. Through a network model, the gaze features are extracted, and the mapping relationship between the input and the gaze angle is learned to realize gaze estimation. If the face and eye images are considered as independent of each other and are input into the network for gaze estimation, the intrinsic relationship between them at the feature granularity level will be ignored, which will reduce the gaze estimation accuracy. Since the face contains more redundant information than the eye image, coarse-grained features can be extracted from the face image to estimate the approximate gaze direction, and then fine-grained features can be extracted from the eye image to refine the gaze direction. Finally, the coarse-grained features extracted from the face image and the fine-grained features extracted from the eye image are fused to obtain the final gaze direction. In the fusion process of coarse-grained features and fine-grained features, the effective fusion between different granularity features needs to be considered. The conventional fusion method is to splice different granularity features with weights, but this will make the parameter quantity too large, resulting in slow estimation speed, mismatched feature information and other problems. In addition, the traditional convolutional neural network has the problems of small feature receptive field and poor context information representation capability in feature extraction.

[0004] Therefore, the present application proposes a fine-grained human eye gaze estimation method based on text information, which extracts features of different granularities of face and eye, and effectively fuses feature information of different granularities, fully utilizes the advantages of text information in context information representation, increases the feature reuse rate while appropriately reducing the parameter quantity, and improves the prediction speed and accuracy of the network. SUMMARY

[0005] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a fine-grained human eye gaze estimation method based on text information.

[0006] The technical problem of the present application is solved by adopting the following technical solution:

[0007] A fine-grained human eye gaze estimation method based on text information, characterized in that the method comprises the following steps:

[0008] The first step is to obtain a face image, pre-process the face image, and obtain a face image and left and right eye images;

[0009] The second step is to build a gaze estimation model; the gaze estimation model is divided into two parts: face coarse-grained feature extraction and eye fine-grained feature extraction; the face coarse-grained feature extraction part includes an improved CLIP module and a Transformer encoder, and the eye fine-grained feature extraction part includes a feature extraction network, a channel attention module, and a hybrid stripe pooling block;

[0010] The improved CLIP module includes two text encoders and a visual encoder; the gaze direction is divided into two groups, the first group includes three directions of "up, down, and front", and the second group includes three directions of "left, right, and front"; two kinds of semantic prompt texts are predefined for the face image, one kind of semantic prompt text contains a gaze direction in the first group of gaze directions, and the other kind of semantic prompt text contains a gaze direction in the second group of gaze directions, so each kind of semantic prompt text contains three semantic prompt texts containing different gaze directions; the two kinds of semantic prompt texts are input into the two text encoders respectively to obtain two groups of text features; the face image is subjected to feature extraction by the visual encoder to obtain face features; the cosine similarity of the face features and each text feature is calculated, and the text feature with the maximum cosine similarity is selected from the two groups of text features; the semantic prompt texts corresponding to the two text features with the maximum cosine similarity are input into the Transformer encoder for feature extraction to obtain face coarse-grained features;

[0011] The left and right eye images are subjected to feature extraction by the feature extraction network to obtain left and right eye features; the left and right eye features are subjected to channel attention by the channel attention module to obtain their respective channel attention weight vectors; the two channel attention weight vectors are respectively multiplied with the left and right eye features in the channel dimension, and then concatenated to obtain eye fine-grained features; the eye fine-grained features are subjected to key feature information screening by the hybrid stripe pooling block to obtain reduced eye fine-grained features;

[0012] The face coarse-grained features are sequentially subjected to a linear layer and a Sigmod activation function to obtain a face gaze probability vector; at the same time, the reduced eye fine-grained features are subjected to a fully connected layer to obtain an eye fine-grained feature vector; the face gaze probability vector and the eye fine-grained feature vector are multiplied element by element, and then added to the eye fine-grained feature vector to obtain a gaze estimation vector;

[0013] The third step is to train the gaze estimation model, and the trained gaze estimation model is used for eye gaze estimation.

[0014] Further, the feature extraction network is improved from the bottleneck layer of the MobileNetV2 network, and the improved bottleneck layer includes three parallel branches, the first branch includes convolution, batch normalization and activation operation; the second branch includes convolution and depth separable convolution, and batch normalization and activation operation are followed after the convolution and depth separable convolution; the third branch includes convolution, depth separable convolution and convolution, and batch normalization and activation operation are followed after the convolution and depth separable convolution; the outputs of the three branches are spliced, and then 1*1 convolution is performed, and residual connection is performed with the input of the bottleneck layer to obtain the output of the bottleneck layer.

[0015] Further, the mixed stripe pooling block is divided into two branches, and horizontal and vertical average pooling operations are performed on the eye fine-grained features to obtain horizontal average pooling features and vertical average pooling features; the horizontal average pooling features and the vertical average pooling features are subjected to 1*1 convolution and activation function respectively to obtain horizontal position weight vectors and vertical position weight vectors; the horizontal position weight vectors and the vertical position weight vectors are multiplied with the corresponding elements of the eye fine-grained features respectively and then added to obtain the stripe-pooled eye fine-grained features; the stripe-pooled eye fine-grained features are subjected to maximum pooling operation to obtain the reduced eye fine-grained features.

[0016] Further, one kind of semantic prompt text is "an image of a face staring (category 1)", and the other kind of semantic prompt text is "an image of a face staring (category 2)", and category 1 and category 2 are one of the first and second groups of gaze directions.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] 1. Since gaze estimation is a continuous regression task, ordinary binary classification tasks cannot meet the accuracy requirements, so multiple prompt texts are needed for learning, and the prompt texts will affect the learning effect, the present application selects five more detailed gaze directions as semantic prompt texts, and predefines two kinds of semantic prompt texts, the two kinds of semantic prompt texts are subjected to feature extraction by two text encoders of the improved CLIP module to obtain two groups of text features; the two groups of text features are screened to obtain text information reflecting the approximate gaze direction, the text information reflecting the approximate gaze direction is used to coarsely extract gaze-related features to obtain face coarse-grained features, the preliminary estimation of the gaze direction is realized, and the gaze direction is preliminarily limited to the corresponding quadrant. In the fusion process of the face coarse-grained features and the eye fine-grained features, the face coarse-grained features are converted into probability values, and the eye fine-grained features are constrained, so that the face coarse-grained features and the eye fine-grained features are effectively fused.

[0019] 2. The bottleneck layer of the MobileNet V2 network is improved, deep separable convolution and one-dimensional convolution operation are used to extract eye features from multiple channels, the number of parallel branches and output features of the network is increased, and more features related to the line of sight are contained in the feature space; at the same time, the residual connection between the input features and the output features is used to add the input feature information directly to the feature information output by the bottleneck layer, improve the feature reuse rate, and prevent the generation of gradient disappearance and explosion phenomenon to a certain extent, and better adapt to complex data samples. Under the premise of only increasing a small part of the parameter amount, the accuracy of the line of sight estimation is improved; the model structure is easy to realize, and the feature reuse rate is increased.

[0020] 3. In order to be able to retain the effective features extracted from the eyes, and filter out as much as possible the feature information irrelevant to the line of sight, a hybrid stripe pooling block is introduced to reduce the dimensionality of the eye fine-grained features, effectively retain key features, weaken unimportant feature information, capture the dependency relationship between long-distance features, and at the same time increase the receptive field of the network, which can improve the accuracy of line of sight estimation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The overall flowchart of the present application;

[0022] Figure 2 The structure diagram of the line of sight estimation model of the present application;

[0023] Figure 3 The structure diagram of the hybrid stripe pooling block of the present application;

[0024] Figure 4 The structure diagram of the improved MobileNet V2 network of the present application;

[0025] Figure 5 The structure diagram of the improved bottleneck layer of the present application. DETAILED DESCRIPTION

[0026] The specific embodiments are described below in conjunction with the drawings, which are only used to specifically introduce the technical scheme of the present application, and do not limit the protection scope of the present application.

[0027] The present application provides a fine-grained human eye line of sight estimation method based on text information (referred to as method, see Figures 1 to 5 ), comprising the following steps:

[0028] First step: obtaining a face image, performing preprocessing operations including normalization and cropping on the face image to obtain a face image and left and right eye images, and then constructing a data set;

[0029] This embodiment uses face images from the public datasets MPIIFaceGaze and Gaze360. First, the face images are normalized. Then, each face image is cropped into a 224×224×3 face image and left and right eye images. Each image is labeled.

[0030] Step 2: Build a line-of-sight estimation model; such as... Figure 2 As shown, the gaze estimation model is divided into two parts: coarse-grained facial feature extraction and fine-grained eye feature extraction. The coarse-grained facial feature extraction part includes an improved CLIP module and a Transformer encoder. The facial image is input into the improved CLIP module, and text information reflecting the approximate gaze direction is extracted by constructing multiple pairs of correspondences between text information and visual information. The Transformer encoder extracts features from the text information reflecting the approximate gaze direction to obtain coarse-grained facial features, thus achieving a preliminary estimation of the gaze direction. The fine-grained eye feature extraction part includes an improved MobileNet V2 network, a channel attention module, and a hybrid stripe pooling block. The left and right eye images are processed by the improved MobileNet V2 network to extract left and right eye features. The channel attention module assigns channel attention weights to the left and right eye features, thereby obtaining fine-grained eye features. The hybrid stripe pooling block is used for dimensionality reduction processing of the fine-grained eye features.

[0031] like Figure 2 As shown, the improved CLIP module includes two text encoders and one vision encoder. The gaze direction is divided into two groups: the first group includes three directions (up, down, and forward), and the second group includes three directions (left, right, and forward). Two semantic cue texts are predefined for each facial image: one is "an image of a face gazing (Category 1)", and the other is "an image of a face gazing (Category 2)". Category 1 and Category 2 represent one of the first and second gaze directions, respectively. Therefore, each semantic cue text contains three semantic cue texts representing different gaze directions. The two semantic cue texts are input into the two text encoders to obtain two sets of text features; the facial image is input into the vision encoder to extract facial features.

[0032] The cosine similarity between facial features and each text feature is calculated. The text feature with the highest cosine similarity is selected from the two sets of text features to obtain the approximate gaze direction, such as upper left, lower left, upper right, lower right, etc. The semantic prompt text corresponding to the two text features with the highest cosine similarity (e.g., "an image of a face gazing, upper left") is used as text information reflecting the approximate gaze direction and input into the Transformer encoder for feature extraction to obtain coarse-grained facial features and complete the preliminary estimation of the gaze direction.

[0033] The text encoder adopts the existing BERT structure based on Transformer, which can effectively capture the context information contained in the semantic prompt text of the input; the visual encoder adopts the existing Vision Transformer structure, which is different from the traditional Transformer structure in that it prefers to divide the input image into blocks, expand the image into a sequence, and add the positional relationship between the image blocks to the sequence, and then perform feature extraction by the Transformer.

[0034] For the extraction of eye features, the left and right eye images are respectively input into the improved MobileNet V2 network for feature extraction to obtain left and right eye features; the left and right eye features respectively pass through the channel attention module (CAM) to make the network pay more attention to the information on the channel. First, the left eye feature with a dimension of HxWxC is compressed according to the spatial dimension to obtain a one-dimensional left eye feature with a dimension of 1x1xC; then, the one-dimensional left eye feature passes through a fully connected layer and an activation function to obtain a channel attention weight vector ε1; finally, the channel attention weight vector ε1 is multiplied with the left eye feature in the channel dimension to obtain the left eye feature fused with channel attention; similarly, the right eye feature fused with channel attention is obtained; the left and right eye features and the left and right eye features fused with channel attention are spliced to obtain eye fine-grained features fused with information of both eyes.

[0035] Since the eye fine-grained features at this time are relatively large, they need to be reduced in dimension through a pooling operation to filter out more critical features, so the mixed stripe pooling block (MSP) is used to filter the eye fine-grained features to obtain the eye fine-grained features after dimension reduction. As shown in Figure 3 The mixed stripe pooling block has two branches, which capture long-range context information from the horizontal and vertical dimensions respectively; first, the eye fine-grained features are respectively subjected to horizontal average pooling and vertical average pooling operations to establish the connection between the features in the feature space, to obtain horizontal average pooling features and vertical average pooling features; then, the horizontal average pooling features and the vertical average pooling features are respectively subjected to 1x1 convolution for dimension expansion, and then subjected to an activation function to obtain a horizontal position weight vector and a vertical position weight vector; the horizontal position weight vector and the vertical position weight vector are respectively multiplied with the eye fine-grained features corresponding to the elements and then added to obtain the eye fine-grained features subjected to stripe pooling; finally, the eye fine-grained features subjected to stripe pooling are subjected to a max-pooling operation to obtain the eye fine-grained features after dimension reduction; the max-pooling operation is used to retain key information and weaken relatively unimportant information, and at the same time, it also increases the receptive field of the network, which can improve the accuracy of gaze estimation.

[0036] In order to realize the fusion between the face coarse-grained features and the eye fine-grained features, the face coarse-grained features are used to constrain the eye fine-grained features, the face coarse-grained features are converted into one-dimensional face coarse-grained vectors through a linear layer, and then each element in the one-dimensional face coarse-grained vectors is converted into a probability value between 0 and 1 through a Sigmod activation function to obtain a face gaze probability vector; meanwhile, the eye fine-grained features after dimension reduction are converted into a one-dimensional vector through a full connection layer to obtain an eye fine-grained feature vector; the face gaze probability vector and the eye fine-grained feature vector are multiplied element by element, in order to prevent the adverse effects of inaccurate gaze direction estimation from the face image on the overall performance, an identity mapping is added to the gaze direction obtained from the eye fine-grained features, that is, the multiplication element by element is added to the eye fine-grained feature vector to obtain a gaze estimation vector, and the final gaze estimation is completed.

[0037] For the improved MobileNet V2 network, the structure is as shown in Figure 4 The left and right eye images first undergo 3x3 convolution for preliminary feature extraction, then pass through 6 improved bottleneck layers, and then pass through 1x1 convolution to change the output dimension.

[0038] As shown in Figure 5 The improved bottleneck layer includes three parallel branches, the first branch includes convolution, batch normalization and activation operation, and the convolution kernel is 1x1; the second branch includes a 1x1 convolution and a 3x3 depth separable convolution, and batch normalization and activation operation are followed after the convolution and the depth separable convolution; the third branch includes two convolutions and a depth separable convolution between the two convolutions, and batch normalization and activation operation are followed after the convolution and the depth separable convolution, the size of the convolution is 1x1, and the size of the depth separable convolution is 3x3; the output features of the three branches are spliced, then pass through a 1x1 convolution to change the feature dimension to the same size as the input feature of the bottleneck layer, and finally the residual connection is performed with the input feature of the improved bottleneck layer to increase the feature reuse rate, and the output feature of the improved bottleneck layer is obtained. Among them, the activation operation adopts PReLU activation function, and the batch normalization and activation operation enhances the non-linear characteristics of the network. The 1x1 convolution of the first branch does not extract features, but expands the channel number of the 3x3 depth separable convolution, because too many radial structures in the network will destroy the parallelism of the network, and therefore increasing the number of branches can to some extent preserve relatively important features while increasing the network efficiency. The input of the improved MobileNet V2 network is the left and right images, which contain many features related to the gaze, therefore, appropriately increasing the channel number in the preliminary feature extraction stage can preserve more important features, which is undoubtedly beneficial to improving the gaze estimation accuracy.

[0039] Third step: train the gaze estimation model using the training set in the data set, update the parameters of the improved CLIP module and the improved MobileNet V2 network respectively through the loss functions of formulas (1) and (2), until the loss converges, and the trained gaze estimation model is obtained;

[0040]

[0041] In the formula, s i is the facial feature extracted by the visual encoder, z i and z k are the text features extracted by two text encoders respectively, sim(·) represents the cosine similarity between two features, K is the number of text features, τ is a constant, is the gaze estimation vector, y n ∈R 3 is the real gaze direction, and N is the number of samples.

[0042] The trained gaze estimation model is evaluated and tested using the test set in the data set, and the gaze angle error evaluation model of formula (3) is used.

[0043]

[0044] The unmentioned part of the application is applicable to the prior art.

Claims

1. A fine-grained human eye gaze estimation method based on text information, characterized in that, The method comprises the following steps: First step: obtaining a face image, pre-processing the face image to obtain a face image and left and right eye images; Second step: building a gaze estimation model; the gaze estimation model comprises two parts, namely face coarse-grained feature extraction and eye fine-grained feature extraction; the face coarse-grained feature extraction part comprises an improved CLIP module and a Transformer encoder, and the eye fine-grained feature extraction part comprises a feature extraction network, a channel attention module and a hybrid stripe pooling block; The improved CLIP module comprises two text encoders and a visual encoder; the gaze directions are divided into two groups, the first group comprising three directions of "up, down and front", and the second group comprising three directions of "left, right and front"; two kinds of semantic prompt texts are predefined for the face image, one kind of semantic prompt text comprising a gaze direction in the first group of gaze directions, and the other kind of semantic prompt text comprising a gaze direction in the second group of gaze directions, so that each kind of semantic prompt text comprises three semantic prompt texts containing different gaze directions; the two kinds of semantic prompt texts are input into the two text encoders respectively to obtain two groups of text features; the face image is subjected to feature extraction by the visual encoder to obtain face features; the cosine similarity of the face features and each text feature is calculated, and the text feature with the maximum cosine similarity is selected from the two groups of text features; the semantic prompt texts corresponding to the two text features with the maximum cosine similarity are input into the Transformer encoder for feature extraction to obtain face coarse-grained features; The left and right eye images are subjected to feature extraction by the feature extraction network to obtain left and right eye features; the left and right eye features are subjected to channel attention by the channel attention module to obtain respective channel attention weight vectors; the two channel attention weight vectors are respectively multiplied with the left and right eye features in the channel dimension, and then spliced to obtain eye fine-grained features; the eye fine-grained features are subjected to key feature information screening by the hybrid stripe pooling block to obtain reduced eye fine-grained features; The face coarse-grained features are sequentially subjected to a linear layer and a Sigmod activation function to obtain a face gaze probability vector; at the same time, the reduced eye fine-grained features are subjected to a fully connected layer to obtain an eye fine-grained feature vector; the face gaze probability vector and the eye fine-grained feature vector are multiplied element by element, and then added to the eye fine-grained feature vector to obtain a gaze estimation vector; Third step: training the gaze estimation model, and using the trained gaze estimation model for eye gaze estimation.

2. The method of claim 1, wherein, The feature extraction network is improved from a bottleneck layer of a MobileNet V2 network, and the improved bottleneck layer comprises three parallel branches, the first branch comprises convolution, batch normalization and activation operation; the second branch comprises convolution and depth separable convolution, and the convolution and the depth separable convolution are followed by batch normalization and activation operation; the third branch comprises convolution, depth separable convolution and convolution, and the convolution and the depth separable convolution are followed by batch normalization and activation operation; the outputs of the three branches are spliced, then pass through 1*1 convolution, and are connected with the input of the bottleneck layer in a residual manner to obtain the output of the bottleneck layer.

3. The method of claim 2, wherein, The convolution kernel of the convolution is 1*1, and the convolution kernel of the depth separable convolution is 3*3.

4. The method of claim 1, wherein, The mixed stripe pooling block comprises two branches, horizontal and vertical average pooling operations are performed on the eye fine-grained features to obtain horizontal average pooling features and vertical average pooling features; the horizontal average pooling features and the vertical average pooling features pass through 1*1 convolution and an activation function respectively to obtain a horizontal position weight vector and a vertical position weight vector; the horizontal position weight vector and the vertical position weight vector are multiplied with the eye fine-grained features respectively and then added to obtain stripe-pooled eye fine-grained features; The stripe-pooled eye fine-grained features pass through a maximum pooling operation to obtain reduced eye fine-grained features.

5. The method of claim 1-4, wherein, One kind of semantic prompt text is "an image of a face staring (category 1)", and the other kind of semantic prompt text is "an image of a face staring (category 2)", and category 1 and category 2 are one of the first and second groups of gaze directions respectively.

6. The method of claim 1, wherein, The text encoder adopts a BERT structure, and the visual encoder adopts a Vision Transformer structure.

Images