A line-of-sight estimation method based on multi-task feature processing

Abstract

Embodiments of the present disclosure disclose a line-of-sight estimation method based on multi-task feature processing. A specific implementation of the method comprises: training a multi-task model capable of simultaneously performing line-of-sight estimation and line-of-sight independent factor estimation on training data; for each line-of-sight independent factor, synthesizing interference data thereof, generating a corresponding line-of-sight feature thereof using the interference data, referred to as interference feature; and obtaining line-of-sight related features using a line-of-sight estimation algorithm on the training data. The implementation can improve the line-of-sight estimation accuracy of the model on target domain data without the aid of any target domain data by interfering with line-of-sight independent features and ensuring that the model can extract line-of-sight consistent features.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically to a gaze estimation method based on multi-task feature processing. Background Technology

[0002] A person's gaze indicates the direction they are looking. Gaze information has wide applications in human-computer interaction, virtual reality, and saliency detection. Gaze estimation techniques are used to calculate the direction of a person's gaze and can be categorized into model-based and appearance-based methods. Model-based methods typically require complex or expensive equipment, such as eye trackers; while appearance-based methods only require inexpensive webcams. With the development of deep learning technology, appearance-based gaze estimation methods have rapidly evolved. Appearance-based gaze estimation methods typically use manually designed features or features extracted using deep learning techniques to estimate gaze direction. The latest gaze estimation methods generally use deep learning techniques to extract features. They are usually trained and tested on the same dataset. However, if these methods are tested on a dataset different from the training set, their performance will significantly decrease. The problems with these methods limit the application of gaze estimation because in real-world scenarios, gaze-independent features such as identity, facial expressions, and lighting often change, and deep learning models often overfit to these irrelevant features, leading to performance degradation on datasets different from the training set. This problem is known as the cross-domain gaze estimation problem, which is very challenging. To address this problem, the academic community has proposed several solutions, such as unsupervised domain adaptation. This involves training the model first in the source domain and then fine-tuning it on a small amount of unlabeled target domain data, aiming to improve the model's testing accuracy in the target domain. However, this type of method typically requires a small amount of unlabeled target domain data and fine-tuning for each target domain, thus limiting its application. Another strategy is domain generalization. In this strategy, the model is usually trained in the source domain using certain methods and then directly tested on target domain data, aiming to improve the model's testing accuracy in the target domain. This strategy does not require target domain data during the training phase, greatly expanding its application scenarios. However, this type of method currently lacks research. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure propose a gaze estimation method based on multi-task feature processing to solve one or more of the technical problems mentioned in the background section above.

[0005] Some embodiments of this disclosure provide a gaze estimation method based on multi-task feature processing. The method includes: training a multi-task model capable of simultaneously estimating gaze and gaze-independent factors on training data; for gaze-independent factors, synthesizing interference data for each factor, and using the interference data to generate its corresponding gaze feature, called interference feature; and obtaining gaze-related features on the training data using a gaze estimation algorithm. The method is characterized in that the process of extracting gaze-related features depends on minimizing the feature distance between interference features and gaze-related features. Attached Figure Description

[0006] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0007] Figure 1 This is a flowchart of some embodiments of the gaze estimation method based on multi-task feature processing according to the present disclosure;

[0008] Figure 2 This is a schematic diagram of the line-independent feature interference stage of some embodiments of the line-of-sight estimation method based on multi-task feature processing according to the present disclosure;

[0009] Figure 3 This is a schematic diagram of the gaze consistency feature extraction stage according to some embodiments of the gaze estimation method based on multi-task feature processing of this disclosure;

[0010] Figure 4 This is a schematic diagram of a line-independent feature interference stage adversarial attack based on some embodiments of the line-of-sight estimation method based on multi-task feature processing disclosed herein;

[0011] Figure 5 This is a schematic diagram of the gaze consistency feature extraction stage according to some embodiments of the gaze estimation method based on multi-task feature processing of this disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Figure 1 A flow 100 of some embodiments of a gaze estimation method based on multi-task feature processing according to the present disclosure is shown. This gaze estimation method based on multi-task feature processing includes the following steps:

[0019] Step 101: Train a multi-task model on the training data that can simultaneously perform gaze estimation and gaze-independent factor estimation.

[0020] The aforementioned multi-task model consists of a gaze estimation branch and a gaze-independent factor estimation branch. Different branches share parameters in the shallow layer of the aforementioned multi-task model, but have independent parameters in the deep layer of the aforementioned multi-task model. The gaze-independent factors include, but are not limited to, one of the following: identity, expression, lighting, and hue.

[0021] Step 102: For line-of-sight-independent factors, synthesize interference data for each factor, and use the interference data to generate its corresponding line-of-sight features, which are called interference features.

[0022] The aforementioned interference data is obtained through adversarial attacks or data augmentation. The steps of the adversarial attack are as follows: In the line-of-sight-independent factor estimation branch, the training data is input and the line-of-sight-independent factor value is output. Based on a given adversarial attack method, adversarial noise is added to the training data by causing the line-of-sight-independent factor estimation branch to produce an incorrect output. The training data with added adversarial noise is the interference data.

[0023] The aforementioned adversarial attack method is specifically implemented as follows:

[0024] On the line-independent factor estimation branch corresponding to a specific line-independent factor, the gradient of the loss function of the line-independent factor estimation branch with respect to the training data is calculated, and the sign of the gradient is calculated (the sign is -1 when the gradient is negative, 1 when the gradient is positive, and 0 when the gradient is 0). The training data is then subtracted from the value of the gradient sign multiplied by ∈ (∈ represents the strength of the adversarial noise, with a value range of [0, 1]) to obtain the interference data. This ensures that the line-independent factors in the interference data have been interfered with, and the line-independent information is correctly preserved.

[0025] The aforementioned data enhancements include, but are not limited to, one of the following:

[0026] The brightness of the image is changed by a random amplitude, but the line-of-sight information in the image is not changed.

[0027] The contrast of the image is changed by a random amplitude, but the gaze information in the image is not changed.

[0028] The saturation of the image is changed by a random amplitude, but the gaze information in the image is not changed.

[0029] The image's hue is altered by a random amplitude, but the gaze information within the image remains unchanged.

[0030] Data augmentation ensures that gaze information in the image is correctly preserved, while gaze-independent factors such as brightness, contrast, saturation, and hue are not affected.

[0031] Step 103: Use a gaze estimation algorithm to obtain gaze-related features on the training data. The process of extracting gaze-related features depends on minimizing the feature distance between the interference features and the gaze-related features.

[0032] The aforementioned minimization of the feature distance between the aforementioned interference features and the aforementioned line-of-sight related features includes:

[0033] The interference data is input into the line-of-sight estimation branch to obtain interference features, and it is ensured that the line-of-sight irrelevant features in the interference features cannot be overfitted by the model.

[0034] The training data is input into the gaze estimation branch to obtain gaze-related features;

[0035] The maximum mean difference between interference features and line-of-sight related features is calculated, and the maximum mean difference is minimized through backpropagation and gradient descent to ensure that the model does not overfit to line-of-sight unrelated factors in the training data.

[0036] The aforementioned feature distance is obtained through the following steps:

[0037] For each line-of-sight-independent factor, calculate the interference data corresponding to the line-of-sight-independent factor, and calculate the aforementioned feature distances corresponding to the line-of-sight-independent factor;

[0038] The total feature distance is obtained by summing all the calculated feature distances. This total feature distance is then used to minimize the total feature distance.

[0039] Some embodiments of the gaze estimation method based on multi-task feature processing disclosed herein are divided into two stages:

[0040] The first stage involves perturbing gaze-independent features: This stage attempts to perturb information from gaze-independent factors (identity, facial expression, lighting, color tone, etc.) to prevent the model from overfitting. This is achieved by performing adversarial attacks or data augmentation on specific gaze-independent factors in the training data to synthesize data that has been perturbed by these factors. When training with this data, the model will not overfit to gaze-independent features. A diagram illustrating the adversarial attack in the gaze-independent feature perturbation stage is shown below. Figure 4 As shown.

[0041] The second stage, gaze consistency feature extraction, ensures that the gaze-related features in the synthetic data remain correct and learnable. The synthetic data from the previous step, along with the original data, are input into the model to guarantee that the features extracted by the model are consistent across gaze-related factors. A diagram illustrating gaze consistency feature extraction is shown below. Figure 5 As shown.

[0042] Figure 2 The flowchart 200 of the gaze-independent feature interference stage of some embodiments of the gaze estimation method based on multi-task feature processing of this disclosure is shown:

[0043] Step 201: Randomly sample training data from the source domain.

[0044] Step 202: Estimate identity information through the identity recognition branch, calculate the identity recognition loss function, and use this loss function to perform adversarial attacks on the input training data.

[0045] Step 203: Estimate facial expression information through the facial expression recognition branch, calculate the facial expression recognition loss function, and use this function to perform adversarial attacks on the input training data.

[0046] Step 204: Estimate illumination information through the illumination recognition branch, calculate the illumination estimation loss function, and use this function to perform adversarial attacks on the input training data.

[0047] Step 205: Randomly adjust the tone of the input training image to obtain training data where line-of-sight-independent factors are disturbed.

[0048] Network Structure: ResNet is used as the backbone network for feature extraction, handling both gaze estimation and gaze-independent factor estimation tasks. For different tasks, the network shares the first layer (layer1), and each task has independent second layer (layer2), third layer (layer3), fourth layer (layer4), and prediction layer.

[0049] The gaze estimation branch is used to estimate the gaze direction, as the gaze estimation task is a regression task. The prediction layer uses a multilayer perceptron and employs the following loss function. To train the gaze estimation branch:

[0050]

[0051] Where y is the actual line-of-sight label. The direction of the line of sight estimated by the model. for The 1-norm.

[0052] Identity Recognition Branch: The identity recognition task is a classification task. The identity recognition branch is used to predict identity labels. The prediction layer uses the CosFace method (face recognition / identity recognition method) and the following loss function is used. To train the identity recognition branch:

[0053]

[0054] Where CE represents the cross-entropy loss, y ide For real identity labels, The identity labels estimated by the model.

[0055] Facial Expression Recognition Branch: Since facial expression recognition is a classification task, the facial expression recognition branch is used to predict facial expression labels. These labels can include: anger, disgust, fear, happiness, sadness, surprise, and neutral. The prediction layer uses a multilayer perceptron and employs the following loss function. To train the facial expression recognition branch:

[0056]

[0057] Where CE represents the cross-entropy loss, y exp For real emoji tags, The facial expression labels estimated by the model.

[0058] Illumination estimation branch: The illumination estimation task is a regression task. The illumination estimation branch is used to estimate the average illumination intensity of the image. The prediction layer uses a multilayer perceptron to convert the input image with RGB channels into HSV (Hue, Saturation, and Value) channels, and uses the average value of the Value channel as the average illumination intensity of the image. The following loss function is used. To train the illumination estimation branch:

[0059]

[0060] Among them, y illu To calculate the average illumination intensity of the image using the Value channel in HSV, The average illumination intensity of the image estimated by the model.

[0061] Tone adjustment: Use PyTorch's ColorJitter to randomly adjust the tone of the input image, thereby interfering with the tone.

[0062] Eye-independent feature perturbation: The purpose of this stage is to interfere with the training data by targeting specific eye-independent factors, such as identity, facial expression, lighting, and color tone, so that these factors cannot be overfitted by the model. This is achieved through adversarial attacks or data augmentation to perturb the training data by targeting specific eye-independent factors.

[0063] For identity, facial expression, and lighting, adversarial attacks are used to generate perturbed data. Without loss of generality, we will use identity as an example; the same principle applies to facial expression and lighting. The adversarial attack method of the Fast Gradient Sign Method (FGSM) is used to generate the perturbed data:

[0064]

[0065] Where x represents the original input image, This represents adversarial examples, which are the training data that has been interfered with, preventing the model from overfitting to eye-independent features. `slgn` represents the sign of the gradient: -1 for a negative gradient, 1 for a positive gradient, and 0 for a zero gradient. `∈` is set to 0.1.

[0066] For hue adjustment, PyTorch's ColorJitter function is used to randomly adjust the hue of the input image, generating... This process perturbs the color tone. It helps obtain images with different color tones, which have their contrast, saturation, etc., randomly jittered. The final perturbed training data contains combinations of images with different color tones, allowing the model to learn images with different color tones that share the same viewing direction.

[0067] For each line-of-sight-independent factor, distorted training data is generated independently.

[0068] Figure 3 The flowchart 300 of the gaze consistency feature extraction stage of some embodiments of the gaze estimation method based on multi-task feature processing of this disclosure is shown:

[0069] Step 301: Calculate the gaze estimation loss function based on the original input training data.

[0070] Step 302: Calculate the line-of-sight-independent loss function based on the original input training data.

[0071] Step 303: Calculate the feature distance based on the original input training data and the disturbed training data.

[0072] Step 304: Calculate the total loss function and train the entire network to obtain the trained gaze estimation branch.

[0073] The purpose of this stage is to ensure that the features extracted by the model can fit gaze-related factors, rather than overfitting to gaze-irrelevant factors such as identity, facial expression, lighting, and color tone. In other words, it extracts gaze-consistent features, ensuring that gaze information in the disturbed dataset is accurate and learnable. The disturbed data generated during the gaze-irrelevant feature disturbance stage... The original data X is input into the gaze estimation branch to obtain gaze features. And Z. Then, minimize the feature distance between these two features to ensure that the gaze estimation model can extract gaze consistency features.

[0074] Specifically, this involves interfering with the data generated during the line-of-sight-independent feature interference phase. The original data x is input together into the gaze estimation branch F to obtain gaze features. And Z, that is:

[0075]

[0076] To ensure that the gaze estimation model can extract features of gaze consistency, minimize The feature distance between Z and Z is used to align these two features. For this purpose, the feature distance common in domain adaptation tasks, namely Maximum Mean Discrepancy (MMD), is used to constrain the alignment. The feature distance between and Z, i.e.

[0077]

[0078] Minimize Feature distance between Z and

[0079] Generate using more than one line-independent feature At this time, it is necessary to use different line-independent features to obtain The average value is used for training. For example, it is necessary to generate [data] based on identity and facial expressions. set up For identity corresponding For the corresponding emoji Then the final for

[0080] Total loss function:

[0081]

[0082] in, The loss function for line-of-sight estimation. Let the loss function be for identity recognition. Let be the loss function for facial expression recognition. Let be the loss function for illumination estimation. λ is the loss function used for minimizing feature distance. ide , λ exp , λ illu and λ ali These are all hyperparameters used to balance the terms of the loss function, and the value of each hyperparameter is set to 1.0.

[0083] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A gaze estimation method based on multi-task feature processing, comprising: Train a multi-task model on training data that can simultaneously perform gaze estimation and gaze-independent factor estimation; For gaze-independent factors, for each factor, interference data is synthesized, and gaze-related features corresponding to each factor are generated using the interference data, called interference features. Specifically, training data is randomly sampled from the source domain; identity information is estimated through the identity recognition branch, the identity recognition loss function is calculated, and adversarial attacks are performed on the input training data accordingly; expression information is estimated through the expression recognition branch, the expression recognition loss function is calculated, and adversarial attacks are performed on the input training data accordingly; illumination information is estimated through the illumination recognition branch, the illumination estimation loss function is calculated, and adversarial attacks are performed on the input training data accordingly; and the tone of the input training image is randomly adjusted to obtain training data with gaze-independent factors interfered with. Different branches share parameters in the shallow layers of the multi-task model, but have independent parameters in the deeper layers of the multi-task model. Target domain data is not required during the training phase. A gaze-related feature is obtained using a gaze estimation algorithm on the training data. The process of extracting the gaze-related feature depends on minimizing the feature distance between the interference features and the gaze-related features.

2. The method according to claim 1, wherein, The multi-task model consists of a line-of-sight estimation branch and a line-of-sight-independent factor estimation branch.

3. The method according to claim 1, wherein, The factors unrelated to vision include one of the following: identity, facial expression, lighting, and hue.

4. The method according to claim 2, wherein, The interference data is obtained through adversarial attacks or data augmentation.

5. The method according to claim 4, wherein, The steps to counter the attack are as follows: In the line-of-sight-independent factor estimation branch, the input is training data and the output is the value of line-of-sight-independent factors. Based on the given adversarial attack method, adversarial noise is added to the training data by causing the line-of-sight-independent factor estimation branch to produce erroneous outputs. The training data with added adversarial noise is called interfering data.

6. The method according to claim 5, wherein, The given adversarial attack methods include: For the line-independent factor estimation branch corresponding to a specific line-independent factor, calculate the gradient of the loss function of the line-independent factor estimation branch with respect to the training data, and calculate the sign of the gradient, where the sign is -1 when the gradient is negative, 1 when the gradient is positive, and 0 when the gradient is 0; subtract the sign multiplier of the gradient from the training data. The value of , where, This represents the strength of the noise suppression, with a value range of [0, 1]. The interference data is obtained, ensuring that line-of-sight irrelevant factors in the interference data have been interfered with and line-of-sight information is correctly preserved.

7. The method according to claim 4, wherein, The data enhancement includes one of the following: The brightness of the image is changed by a random amplitude, but the line-of-sight information in the image is not changed. The contrast of the image is changed by a random amplitude, but the gaze information in the image is not changed. The saturation of the image is changed by a random amplitude, but the gaze information in the image is not changed. The image's hue is altered by a random amplitude, but the gaze information within the image remains unchanged.

8. The method according to claim 2, wherein, The feature distance between the minimized interference features and the line-of-sight related features includes: The interference data is input into the line-of-sight estimation branch to obtain interference features, and it is ensured that the line-of-sight irrelevant features in the interference features cannot be overfitted by the model. The training data is input into the gaze estimation branch to obtain gaze-related features; The maximum mean difference between interference features and line-of-sight related features is calculated, and the maximum mean difference is minimized through backpropagation and gradient descent to ensure that the model does not overfit to line-of-sight unrelated factors in the training data.

9. The method according to claim 8, wherein, The feature distance is obtained through the following steps: For each line-of-sight-independent factor, calculate the interference data corresponding to the line-of-sight-independent factor, and calculate the feature distance corresponding to the line-of-sight-independent factor; The total feature distance is obtained by summing all the calculated feature distances, and this total feature distance is used to minimize the feature distance.

Images