Sight line estimation method based on small sample learning

Abstract

The invention discloses a sight line estimation method based on small sample learning, belongs to the technical field of computer vision, and aims to estimate the sight line direction of a person through face appearance. The method mainly comprises the following steps: (1) providing a line-of-sight estimation model based on a conditional nerve process, and improving the line-of-sight estimation precision for a specific individual; (2) a label-feature fusion module based on an attention mechanism is provided, prior information provided by a small number of calibration samples is fully utilized, and the sight line estimation precision is effectively improved; and (3) providing a small sample learning-based sight line estimation model training mode, training a small sample learning model on the basis of the general model, and rapidly training to obtain a human-specific sight line estimation model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a line of sight estimation method based on small sample learning. Background Art

[0002] Line of sight is one of the important non-verbal information in human communication. It contains rich visual attention information of humans and is an important clue to understand human behavior and intention. Line of sight estimation refers to the analysis and processing of human eye movement, line of sight and other information to infer the direction or position of line of sight focus, and then assist in analyzing people's attention and interest performance. As a key way to understand human cognition and behavior, line of sight estimation has broad application prospects in real-world scenarios. In the field of smart medical care, it can detect mental or psychological diseases by judging people's gaze behavior patterns; in the field of traffic safety, it can be used to detect whether the driver is in a state of fatigue driving, thereby reducing the risk of traffic accidents; in the field of human-computer interaction, it allows users to interact directly with computers only through line of sight, providing convenience for people with disabilities. In addition, line of sight estimation also has rich practical value in education, business, and entertainment.

[0003] Gaze estimation technology is the process of estimating the direction of human gaze using certain technical means. Most early gaze estimation technologies require the use of special equipment such as additional sensors or infrared cameras for detection. This method can achieve a high accuracy rate, but the cumbersome operation steps and high hardware costs make it difficult to popularize and apply. Therefore, people have proposed a gaze estimation method based on facial appearance, that is, to infer the direction of a person's gaze only through the facial appearance image taken by a single camera. The rapid development of deep learning has greatly promoted the development of the field of gaze estimation, making the prediction of gaze more accurate. In recent years, researchers have carried out extensive and in-depth research in the field of gaze estimation based on deep learning methods. The paper "Zhang X, Sugano Y, Fritz M, et al. Appearance-based gaze estimation in the wild [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 4511-4520." first proposed a CNN-based gaze estimation method to regress the gaze angle from the eye image; the paper "Krafka K, Khosla A, Kellnhofer P, et al. Eye tracking foreveryone [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2176-2184." proposed a method to regress the two-dimensional coordinates of the gaze point using four items as input: left and right eye images, facial images, and facial grid proportions.

[0004] However, due to the differences in eyeball shape and facial appearance between people, the accuracy of the general model proposed in the above method is difficult to further improve. Researchers have found that training independent models for specific individuals may help further improve the accuracy of line of sight estimation. However, training models requires a large number of data samples, and in real-world scenarios, collecting thousands of samples for each target individual for training is very expensive and not suitable for widespread promotion. Therefore, some researchers have integrated small sample learning into the field of line of sight estimation, that is, training the learning model with a small number of labeled samples and assuming that the model can be generalized to other people, so that the accuracy of line of sight estimation for a specific individual can be improved with a small number of labeled samples. The paper "Park S, Mello SD, Molchanov P, et al. Few-shot adaptive gaze estimation[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2019:9368-9377" first integrated small sample learning into the field of gaze estimation, and learned the meta-model through the model-agnostic meta-learning method (MAML, Model-Agnostic Meta-Learning), which improved the accuracy of gaze estimation, but its prediction accuracy still has room for further improvement; the paper "Ma J, Zhang X, Wu Y, et al. Few-Shot Gaze Estimation with Model Offset Predictors[C] / / ICASSP 2022-2022IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP).IEEE,2022:4893-4897" further improved the accuracy of gaze estimation through the offset predictor, but its method of training the offset predictor for each individual separately makes the model training process more complicated. The present invention proposes a line of sight estimation method based on small sample learning and a line of sight estimation method based on conditional neural process, which makes full use of the prior information in a small number of sample labels and simplifies the model training method while ensuring the accuracy of line of sight estimation. Summary of the invention

[0005] This paper proposes a line of sight estimation method based on small sample learning, aiming to improve the line of sight estimation accuracy for a specific individual, which mainly includes the following contents:

[0006] (1) A gaze estimation model based on conditional neural processes is proposed.

[0007] (2) A label-feature fusion module based on attention mechanism is proposed.

[0008] (3) A line of sight estimation model training method based on small samples is proposed.

[0009] The specific contents are as follows:

[0010] (1) A gaze estimation model based on conditional neural process is proposed: A gaze estimation model is constructed based on conditional neural process. The main structure includes: image encoder, feature-label fusion module, specific individual encoder, aggregator and decoder, etc. The network structure is as follows Figure 1 shown.

[0011] The image encoder module 110 is used for feature representation of eye images. Specifically, it is used to extract features from the original image data to obtain the original feature representation of the image, capture the important information related to the line of sight in the image through encoding, generate high-quality feature representation, and encode the context set and the target set to convert the high-dimensional image data into a low-dimensional feature vector. The specific acquisition steps are as follows: the first step is to collect a large number of eye images with line of sight angle labels, and train a general line of sight estimation model based on the convolutional neural network Resnet18 and a three-layer multi-layer perceptron; the second step is to remove the last two layers of the multi-layer perceptron in the model after the training is completed, and use Resnet18 and the first layer of the multi-layer perceptron as the image encoder.

[0012] The feature-label fusion module 120 is used to obtain the fused feature representation of the label and the image feature. The specific application is to divide the training set into two parts: a context set and a target set. The context set is a small number of annotated eye images, and the target set is a large number of unannotated eye images. For the context set, the feature representation obtained by the encoder and the corresponding label are fused through the feature fusion module to maximize the utilization of the label information. The formal representation is shown in Formula 1, where L represents the label, F represents the image feature, MLP represents the multi-layer perceptron, ALFF represents the feature fusion module, and Z represents the fused feature.

[0013] Z=ALFF(MLP(L),F) (1)

[0014] The specific individual encoder module 130 is used to further encode the obtained fusion features. Its network structure consists of a linear layer and a ReLU function. It regards the features from the same individual as a task, comprehensively processes the information of different tasks, and generates corresponding context features to better capture individual characteristics.

[0015] Aggregator module 140 is used to aggregate context features from the same specific individual. The aggregator module aggregates different feature information from the same specific individual to smooth the impact of individual differences and generate a comprehensive and robust global encoding representation. By aggregating these diverse information, the aggregator better captures global information and generates more accurate aggregated sample features, that is, aggregated sample features.

[0016] The decoder module 150 is used to convert the coded representation into a specific line of sight estimation output. The specific application is to concatenate the embedded features obtained by the image encoder of the input image of the target set with the aggregated sample features obtained above. The concatenation operation combines the original information with the individual-specific information, and the concatenated feature vector is input to the decoder, which decodes the global coded representation into the final line of sight estimation result.

[0017] (2) A label-feature fusion module based on the attention mechanism is proposed: the feature representation obtained by the encoder in the context set is fused with the corresponding label information. Specifically, it is implemented based on the attention mechanism, which is used to maximize the use of the prior information in the label, thereby improving the effect of feature fusion. The label and image feature vector are the two inputs of the module. The structure of the feature fusion module is as follows: Figure 2 shown.

[0018] The label information is mapped by a multi-layer perceptron (MLP) to obtain a vector representation of the same dimension as the feature vector. The multi-layer perceptron here consists of multiple layers of fully connected layers, each of which is followed by a nonlinear activation function (such as ReLU), which is used to perform nonlinear transformations on the labels to capture the complex features in the labels. Through this mapping, the label information is converted into a feature representation and fused with the image features in the same space. Its formal representation is shown in Formula 2, where L represents the label, W1 and b1 represent the weight matrix and bias of the first linear layer, respectively, W2 and b2 represent the weight matrix and bias of the second linear layer, respectively, and δ represents the ReLU function.

[0019] MLP(L)=W2(δ(W1x+b1))+b2 (2)

[0020] The two branches are subjected to point-wise convolution respectively. Point-wise convolution is a special convolution performed through a convolution kernel of size 1x1, and each output channel depends only on the linear combination of the input channels. While keeping the spatial dimension unchanged, the channel dimension of the feature is changed to effectively capture the relationship between the features. Next, the nonlinear activation of the ReLU layer is performed to further enhance the expressive power of the feature. Subsequently, point-wise convolution is performed again to enrich the feature representation. Its formulaic representation is shown in Formula 3, where PWConv1 and PWConv2 represent point-wise convolution, and δ represents the ReLU function.

[0021] L(x)=PWConv2(δ(PWConv1(x))) (3)

[0022] Before fusing feature representations, the weight of each feature representation needs to be calculated. To this end, the summed feature representations are processed through a Sigmoid layer, the input is mapped to between 0 and 1, and the corresponding weight representation w is obtained. In this model, the weight representation w is used to measure the importance of each feature in the final fused representation. The calculation method of w value is shown in Formula 4, where represents broadcast addition, and σ represents the Sigmoid function.

[0023]

[0024] Next, the feature representations obtained from the two branches are assigned weights w and (1-w) respectively, and the weights are added to obtain the final fused feature representation. The fused feature representation contains both important information in the original image and prior information in the label, which is used to provide auxiliary information for line of sight angle prediction. Its formula is shown in Formula 5, where represents the Hadamard product, and x and y represent the input features of the two branches.

[0025]

[0026] (3) A small sample-based gaze estimation model training method is proposed: First, a large number of eye images with gaze angle labels are provided as training data sets. These annotated eye images are trained using a general feature extractor based on a convolutional neural network to obtain an encoder for feature extraction. The encoder consists of a pre-trained convolutional neural network (e.g., ResNet18) and a multi-layer perceptron, which is trained using a large-scale labeled data set to extract important feature information from the image.

[0027] During the training process, the images in the training dataset are first classified according to individuals. Each individual category contains more than 1,000 eye images. These classified images are used in the subsequent meta-training process. In order to achieve context learning and target learning in the conditional neural process, 1, 5, 10 or 15 images are randomly selected from each individual category as the context set. The images in the context set are labeled with the sight angle to provide prior information. 500 of the remaining unselected images are taken as the target set. The images in the target set are not labeled and serve as samples that the model needs to predict.

[0028] In the meta-training process, each training batch is defined to contain images from 20 different individuals. The images in each batch are randomly extracted from the context set and the target set according to the above method. In each round of training iteration, 20 individuals are randomly selected. For each individual category, the encoder extracts features from the images in the context set to obtain feature representation, and then obtains context sample features through feature fusion, aggregation and other operations. In the meta-testing process, the images in the target set are also extracted by the encoder, and the extracted features are concatenated with the context sample features, and then input into the decoder for regression to obtain a two-dimensional vector. The absolute value difference between the predicted value and the true value is used as the loss function for training. The loss function formula is shown in Formula 6, where y # represents the predicted value of the sight angle, Represents the true value of the sight angle, and n represents the number of samples.

[0029]

[0030] The model obtained through this training can be used as a gaze estimator for a specific individual. When the gaze angle of an individual needs to be predicted, a small number of calibration samples with gaze labels from the individual are collected as the context set. Similarly, after encoding, feature fusion and aggregation, context aggregation features containing prior information are obtained. A large number of unlabeled images that need to be predicted are taken as the target set. The features obtained after encoding are concatenated with the context aggregation features and input into the decoder to obtain the final prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of the architecture of a line of sight estimation model based on a conditional neural process in an embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of a label-feature fusion module based on an attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by ordinary technicians in the technical field to which the present application belongs.

[0034] The present invention is a line of sight estimation method based on small sample learning. The development language is Python 3.7, the deep learning framework is Pytorch 1.10, and the operating system is Windows 10. The specific steps are as follows:

[0035] Step 1: Prepare the data for model training and testing. Select individuals with more than 1,000 images in the GazeCapture dataset for model training and testing, of which 795 people are used as training sets and 109 people are used as test sets. Use the method in the literature "Xucong Zhang, Yusuke Sugano, and Andreas Bulling. Revisiting data normalization for appearance-based gaze estimation. In ETRA, 2018" to preprocess the images. Ensure that the public virtual camera points to the same reference point in space with the head upright by rotating, tilting, and translating the virtual camera. Use the RetinaFace model to detect facial landmarks, crop the eye area, and scale the image to 64×256.

[0036] Step 2: General line of sight estimation model training. All image data in the training set are used for training. The model architecture adopts the Resnet18+three-layer MLP architecture, and the number of training rounds is set to 15. After the training is completed and the general model is obtained, the last two layers of its MLP are removed, and the parameters of the remaining network layers are frozen as a general encoder for subsequent image feature extraction.

[0037] Step 3: Network structure construction. The network structure is as follows Figure 1 As shown, it includes an image encoder, a label-feature fusion module, a specific individual encoder, an aggregator and a decoder, where the context set input is an image with a sight label, the target set input is an image without a sight label, and the output is the sight angle.

[0038] Step 4: Network model training. During the training process, 20 IDs are randomly selected from 795 individuals in each iteration, which are regarded as 20 training tasks. K (k = 1 / 5 / 10 / 15) images are randomly selected from the sample of each individual as the context set, and 500 images are randomly selected as the target set. Under this parameter setting, after 50,000 iterations of training, an accurate and robust small sample line of sight estimation model is obtained.

[0039] Step 5: Model effect evaluation. During the test, the samples of each of the 109 individuals in the test set are sorted by file name, and the last 500 images are used as the target set. K images are randomly selected from the remaining samples as the context set. The context set and the target set are input into the model respectively, and the sight angle corresponding to the image is obtained after decoding. The experimental verification shows that the sight angle error predicted by this scheme is 3.08 degrees when k=1. When the context set exceeds 10 images, the sight angle error is within 3 degrees, which confirms the effectiveness of this scheme.

[0040] In summary, the present invention designs a line of sight estimation method based on small sample learning, which improves the accuracy of line of sight estimation with the help of prior information provided by a small number of calibration samples.

[0041] Those skilled in the art will appreciate that the protection scope of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after the changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A line of sight estimation method based on small sample learning, characterized in that: It is a line of sight estimation model based on conditional neural process. Its main structure includes: image encoder, feature-label fusion module, specific individual encoder, aggregator and decoder. The image encoder obtains the feature representation of the eye image. The acquisition steps are as follows: first, a large number of eye images with sight angle labels are collected, and a general sight estimation model is trained based on the convolutional neural network Resnet18 and a three-layer multi-layer perceptron; second, after the training is completed, the last two layers of the multi-layer perceptron in the model are removed, and the Resnet18 and the first layer of the multi-layer perceptron are used as the image encoder. The feature-label fusion module obtains the fusion feature representation of the label and the image feature. Its formula is as follows: Z=ALFF(MLP(L),F) Where L represents the label, F represents the image feature, MLP represents the multi-layer perceptron, ALFF represents the feature fusion module, and Z represents the fusion feature. The specific individual encoder further encodes the obtained fusion features. Its network structure is a stack of linear layers and ReLU layers. It regards the features from the same individual as a task and comprehensively processes the information of different tasks. The aggregator aggregates contextual features from the same specific individual. The decoder converts the encoded representation into a specific line of sight estimation output. The embedded features of the input image of the target set obtained by the image encoder are concatenated with the aggregated sample features obtained above. The concatenated feature vector is input to the decoder, which decodes the global encoded representation into the final line of sight estimation result.

2. The label-feature fusion method according to claim 1, characterized in that: It is implemented based on the attention mechanism, which applies the attention mechanism to maximize the use of prior information in the label, thereby improving the effect of feature fusion. The label information is mapped by a multi-layer perceptron to obtain a vector representation with the same dimension as the feature vector. Through this mapping, the label information is converted into a feature representation and fused with the image features in the same space. Its formulaic representation is as follows: MLP(L)=W2(δ(W1x+b1))+b2 Where L represents the label, W " and b " They represent the weight matrix and bias of the first linear layer, W2 and b2 represent the weight matrix and bias of the second linear layer, and δ represents the ReLU function. The two branches are subjected to pointwise convolution respectively, that is, special convolution is performed through a convolution kernel of size 1x1. Next, the nonlinear activation of the ReLU layer is performed to further enhance the feature expression ability. Subsequently, the pointwise convolution is performed again to make the feature representation richer. Its formulaic expression is as follows: L(x)=PWConv2(δ(PWConv1(x))) Among them, PWConv1 and PWConv2 represent two point-wise convolutions, and δ represents the ReLU function. Before fusing feature representations, the weight of each feature representation needs to be calculated. To this end, the summed feature representations are processed through a Sigmoid layer, the input is mapped to between 0 and 1, and the corresponding weight representation w is obtained. The calculation method of w is as follows: in represents broadcast addition, and σ represents the Sigmoid function. The feature representations obtained from the two branches are assigned weights w and (1-w) respectively, and added together to obtain the final fused feature representation. Its formula is as follows: in represents the Hadamard product, and x and y represent the input features of the two branches.

3. The method according to claim 1, characterized in that: The model training method is: The images in the training dataset are classified by individuals. Each individual category contains more than 1,000 eye images. These classified images are used in the subsequent meta-training process. One, five, ten, or fifteen images are randomly selected from each individual category as the context set. The images in the context set are labeled with the gaze angle to provide prior information. 500 of the remaining unselected images are taken as the target set. The images in the target set are not labeled and are used as samples that the model needs to predict. In the meta-training process, each training batch is defined to contain images from 20 different individuals. The images in each batch are randomly extracted from the context set and the target set according to the above method. In each round of training iteration, 20 individuals are randomly selected. For each individual category, the encoder extracts features from the images in the context set to obtain feature representation, and then obtains context sample features through feature fusion and aggregation. In the meta-testing process, the images in the target set are also extracted by the encoder, and the extracted features are concatenated with the context sample features, and then input into the decoder for regression to obtain a two-dimensional vector. The absolute value difference between the predicted value and the true value is used as the loss function for training. The loss function is as follows: where y # represents the predicted value of the sight angle, represents the true value of the sight angle, and n represents the number of samples.

Images