A method for estimating the gaze directions of multiple people in an image

Abstract

The embodiments of the present disclosure disclose a multi-person gaze direction estimation method in an image, comprising: a new multi-person gaze direction estimation method in an image, which can accurately and timely predict the gaze direction of a single or multiple face regions in an image; a new multi-task learning network structure, which can simultaneously predict the gaze direction of multiple face regions in an image through one-time calculation; a self-supervised loss function based on two-dimensional projection, which can be used to supervise three-dimensional gaze direction estimation and improve calculation accuracy; a new gaze direction replacement data generation method for multiple face regions, which can quickly generate a large amount of realistic data with labels for the training and testing of a deep learning model; and the model can predict the gaze direction in the deployment test process through end-to-end training of the proposed network on the constructed data.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing, specifically to a method for estimating the gaze direction of multiple people in an image. Background Technology

[0002] Human gaze direction is a crucial channel for expressing intent and has significant applications in human-computer interaction, virtual and augmented reality, and other fields. Calculating gaze direction using facial images is currently a hot research topic. However, existing methods typically assume that the input image contains only a cropped and calibrated face. If the application scenario involves multiple subjects, it is impossible to calculate the gaze direction for multiple people in real time. To address this issue, this technology aims to simultaneously estimate the gaze direction of multiple subjects in an image through a single calculation, ultimately enabling real-time multi-person gaze direction estimation in video. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion that follows. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Currently, appearance-based gaze direction estimation is a popular research topic, and various strategies have been applied to improve estimation accuracy, such as coarse-to-fine estimation strategies, adversarial learning methods, and self-attention mechanisms. Meanwhile, several large-scale gaze estimation image datasets have been published. Most of these images were acquired in laboratory environments with strict multi-view camera setups, fixed-position participants and gazing targets, etc. Such acquisition procedures typically result in these datasets containing only single-face images from limited scenes. Correspondingly, proposed gaze estimation methods all assume that there is only one calibrated face in the input image. However, this leads to a drawback: the speed of current gaze direction estimation methods is directly proportional to the number of faces in the input image. When multiple people are present in the image, most methods cannot process video data in real-time.

[0005] The task of understanding facial images has received considerable attention due to its wide range of applications. Many practical methods, such as face localization, facial expression recognition, and head pose estimation, have been proposed. With the development of object detection methods, one-time computation methods for parallel multi-face computation are favored for real-time applications due to their lightweight design and high accuracy. For example, new face detection methods employ a single-stage structure and design more efficient modules for facial features. Meanwhile, many large-scale face datasets have been published, many of which are constructed through extensive manual annotation. Furthermore, it has been found that performing multi-task learning (facial landmark, head pose, gender, etc.) simultaneously with face detection is an efficient method because the computation of these tasks relies on common facial features. Inspired by these works, this paper aims to develop a one-time computation-based gaze direction estimation method based on faces.

[0006] Based on the aforementioned practical needs and technical challenges, the objectives of this invention are: a novel method for one-time computation of gaze directions in multiple face regions, capable of accurately predicting the gaze directions of single or multiple face regions in an image in real time; a novel method for generating multiple-person gaze direction replacement data, capable of rapidly generating large amounts of realistic labeled data for training and testing deep learning models; based on the generated dataset, a novel multi-task learning network structure is proposed, which can simultaneously predict the gaze directions of multiple face regions in an image through one-time computation and supports end-to-end training; and a self-supervised loss function based on two-dimensional projection can be used to supervise three-dimensional gaze direction estimation, improving computational accuracy.

[0007] Some embodiments of this disclosure provide a method for determining the gaze direction of multiple people in an image. The method includes: acquiring a face image, wherein the face image includes at least one face region; constructing a deep network model for multi-task learning, wherein the deep network model includes multiple parallel computing multi-task processing structures, wherein the multiple parallel computing multi-task processing structures output the gaze direction of each face region in the face image and the face position information of each face region in a single calculation; training the deep network model end-to-end on a dataset to obtain a trained deep network model, wherein the trained deep network model is used to determine the gaze direction of multiple people; and inputting the face image into the trained deep network model to obtain the gaze direction of at least one face region included in the face image.

[0008] The method for determining the gaze direction of multiple face regions in an image disclosed in this invention has the following advantages compared with other gaze direction estimation methods: (1) It has invented a novel method for calculating the gaze direction of multiple face regions at once. Its running speed is not affected by the number of faces in the image and it can accurately predict the gaze direction of one or more face regions in the image in real time; (2) It has designed a novel multi-task learning network structure, which can simultaneously predict the gaze direction of multiple face regions in the image by processing the image at once, and can also predict information such as face position and head pose; (3) It has proposed a novel self-supervised loss function based on two-dimensional projection, which can specifically supervise the estimation of three-dimensional gaze direction and improve the accuracy of calculation; (4) It has designed a novel method for generating replacement data for the gaze direction of multiple face regions, which can quickly generate a large amount of realistic data with labels for training and testing of deep learning models; (5) The proposed network can be trained end-to-end on the constructed data, and then the model can accurately predict the gaze direction in real time during the deployment and testing process. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a flowchart of some embodiments of the method for estimating the direction of gaze of multiple people in an image according to the present disclosure;

[0011] Figure 2 This is a flowchart of some further embodiments of the method for estimating the direction of gaze of multiple people in an image according to the present disclosure;

[0012] Figure 3 This is a schematic diagram of a face image based on a method for estimating the gaze direction of multiple people in an image according to some embodiments of the present disclosure;

[0013] Figure 4 This is a schematic diagram of a dataset construction and generation framework for a method for estimating the gaze direction of multiple people in an image according to some embodiments of the present disclosure;

[0014] Figure 5 This is a schematic diagram of a deep network model for a method of estimating the gaze direction of multiple people in an image according to some embodiments of the present disclosure;

[0015] Figure 6 This is a schematic diagram of a projection-based self-supervised loss function for a method of estimating the gaze direction of multiple people in an image according to some embodiments of the present disclosure;

[0016] Figure 7This is a schematic diagram of the gaze direction of a face region in an image according to a method for estimating the gaze direction of multiple people in an image based on some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Figure 1 This is a flowchart of some embodiments of a method for estimating the gaze direction of multiple people in an image according to the present disclosure. The method for estimating the gaze direction of multiple people in an image includes the following steps:

[0022] Step 101: Obtain the face image.

[0023] In some embodiments, the entity performing the multi-person gaze direction estimation method in an image can acquire face images via a wired or wireless connection. The face image includes at least one face region.

[0024] Optionally, the face region in at least one of the above face regions may include, but is not limited to, at least one of the following: face size information, pose information, expression information, and gender information.

[0025] Step 102: Construct a deep network model for multi-task learning.

[0026] In some embodiments, the aforementioned execution entity can construct a deep network model for multi-task learning. This deep network model includes multiple parallelizable multi-task processing structures. These multiple parallelizable multi-task processing structures simultaneously compute and output the gaze direction of each facial region in the face image, as well as the facial position information of each facial region.

[0027] Optionally, the aforementioned deep network model is a multi-task single-stage deep network model, which includes an encoder and multiple decoders. The decoders among these decoders simultaneously perform different types of tasks. The deep network model is used to simultaneously determine the gaze direction of multiple face regions and output the face location information and key point information of these multiple face regions.

[0028] Step 103: Train the deep network model end-to-end on the dataset to obtain the trained deep network model.

[0029] In some embodiments, the execution entity can train the deep network model end-to-end on a dataset to obtain a trained deep network model. This trained deep network model is then used to determine the gaze direction of multiple users.

[0030] Optionally, the aforementioned dataset is generated from multi-person gaze direction images with eye region replacement and a generation framework, using two types of data as input. One type of data consists of single-person image data with gaze direction labels. The other type consists of multi-person image data with multiple face regions. The generation framework is used to automatically cluster the single-person image data based on at least one of the following information: gender, age, and head pose, to facilitate retrieval. The generation framework is also used to, for each face region in the multi-person image data, retrieve the single-person image data that most closely matches the aforementioned face region, perform eye region replacement, and generate the corresponding gaze direction.

[0031] Alternatively, the overall loss function during the end-to-end training process can be obtained using the following formula:

[0032] .

[0033] in, This represents the overall loss function. This indicates the first adjustable hyperparameter. This represents the loss function related to face location information and key point information. This indicates the second adjustable hyperparameter. This represents the loss function related to the line-of-sight direction. The first and second adjustable hyperparameters can be set according to actual needs.

[0034] in, It can be obtained through the following formula:

[0035] .

[0036] in, This represents the loss function related to the direction of the line of sight. This represents the first hyperparameter used to balance the different loss terms. This represents the self-supervised loss function. This represents the second hyperparameter used to balance different loss terms. Indicates the direction of gaze in the face area. Indicates a truth value label. This represents the L1 norm. The first hyperparameter used to balance different loss terms and the second hyperparameter used to balance different loss terms can be set according to actual needs.

[0037] Step 104: Input the face image into the trained deep network model to obtain the gaze direction of at least one face region included in the face image.

[0038] In some embodiments, the execution entity can input the face image into the trained deep network model in real time in the deployment environment to accurately obtain the gaze direction of at least one face region included in the face image in one go.

[0039] Optionally, the above method may further include: for the gaze direction of each face region among the gaze directions of the at least one face region, the following determination steps may be performed:

[0040] The first step is to determine the direction of gaze in the aforementioned facial area. The positions of the line-of-sight projection points on the front, top, and side projection planes. .in, It indicates the position of the line-of-sight projection point in the positive direction. This indicates the position of the line-of-sight projection point in the top direction. This indicates the position of the line-of-sight projection point in the lateral direction. Indicates the positive direction. Indicates the top direction. Indicates the lateral direction. This indicates the direction of gaze in the aforementioned face area. Nucleation angle, indicating the direction of the line of sight. The rotation angle indicates the direction of the line of sight.

[0041] The second step is to determine the positions of the line-of-sight projection points on the three projection planes: front, top, and side. Whether it is equal to the three projections of the 3D view prediction value. The three projections of the aforementioned 3D view prediction value can be obtained using the following formula:

[0042] .

[0043] in, This represents the projection function. Indicates the positive direction. Nucleation angle, indicating the direction of the line of sight. The rotation angle indicates the direction of the line of sight. This indicates that the direction of the line of sight is projected onto the front plane. express The sine value. express The cosine value. Indicates the top direction. This indicates that the line of sight is projected onto the top plane. express The cosine value. Indicates the lateral direction. This indicates that the direction of the line of sight is projected onto the side plane. express The sine value.

[0044] Optionally, the above deep network model includes a self-supervised loss function, which is obtained by the following formula:

[0045] .

[0046] in, This represents the self-supervised loss function. It indicates the positive direction, the top direction, or the side direction. The value is Indicates the positive direction. Indicates the top direction. Indicates the lateral direction. It indicates the direction of sight, whether it is the top or side direction. This represents the projection function. Indicates the direction of gaze in the face area. This represents the projection function from three dimensions to two dimensions. This represents the L1 norm. Represents the natural constant. This represents the trainable parameters. express of Power of 1. Indicating targeting Correction factor for projection.

[0047] See Figure 2 The basic operation of the image-based multi-person gaze direction estimation method of the present invention is to input a face image containing one or more face regions. For example... Figure 3 As shown, the head pose, facial expression, gender, and age information of each person in the image are not limited. The output is the gaze direction of each person, such as... Figure 7As shown, the direction of each person's gaze is indicated by an arrow with a unit vector at the center of their forehead.

[0048] The system operation flowchart of this invention is as follows: Figure 2 As shown, the process includes: inputting a single image, using a deep network model based on multi-task learning, model training, calculating the gaze information of multiple faces in the image at once, and outputting the gaze directions of multiple people. Among these:

[0049] (1) During the training process, a dataset of multiple images with gaze direction labels is constructed by performing image fusion on single-person image datasets with gaze direction labels and multi-person image datasets without gaze direction labels.

[0050] (2) The image fusion process is as follows Figure 4 As shown, the input first includes an image containing multiple faces, where each face can be a facial region. Then, each face is processed, including face regularization and facial feature calculation. Simultaneously, an image dataset with gaze directions is input, and facial features are calculated, followed by facial feature attribute clustering. Then, the results of facial feature calculations on the input image containing multiple faces are used for face matching and retrieval within the clustered facial feature attributes, ultimately generating an eye region to replace the gaze direction. Therefore, by inputting an image containing multiple target tasks and a dataset with gaze direction labels, a dataset is constructed that includes both multiple target individuals and corresponding ground truth labels for gaze directions, thus combining the advantages of both face datasets and gaze direction datasets.

[0051] (3) Using the data generated in (2) as the training set, design a single-stage, multi-task neural network model framework. The process of the model framework is as follows: Figure 5 As shown, the model framework includes inputting an image of a specific size, extracting feature information at different scales through a feature pyramid structure, a context module aggregating multi-scale feature information, multi-task parallel computation, and outputting gaze direction information and face information. The model's input is a complete image containing any number of face regions, and the output is the 3D gaze direction of each detected face region. This model can be trained end-to-end on the training set.

[0052] (4) During the test, input an initial image, such as... Figure 3 As shown, the image can also be scaled to a specific size.

[0053] (5) Input the scaled image from (4) into the deep network model trained in (3) for testing, and obtain information such as the gaze direction and face position of the target face region in the image. Visualize the results in the input image. The results are as follows: Figure 7 As shown.

[0054] The innovations made in each step are illustrated with relevant accompanying diagrams, as detailed below:

[0055] Innovation Point (1) A method for estimating the direction of sight for multiple people in a single calculation

[0056] This invention employs a one-time computation method for estimating the gaze of multiple individuals. Unlike other gaze estimation models that process each face region individually, this model is the first to estimate the gaze of multiple individuals through a single computation. This model can accurately estimate the gaze directions of multiple target objects in an image in real time during testing.

[0057] Innovation Point (2) Deep Neural Network Structure Based on Multi-Task Learning

[0058] First, for the input image, such as Figure 3 As shown, this network learns through multi-task training and outputs the gaze directions of multiple objects in an image. The visualization results are as follows. Figure 7 As shown. Figure 5 As shown, the network structure mainly consists of three parts: a multi-scale feature extraction module, an information aggregation module, and a multi-task parallel processing module. During training, this deep neural network is trained end-to-end on a pre-constructed dataset.

[0059] Specifically, the multi-scale feature extraction module can be implemented using a feature pyramid structure. The extracted features are calculated from the outputs of different stages of a depthwise separable convolutional MobileNet model or a ResNet residual network using top-down and lateral connections. This feature pyramid structure includes two modules: a bottom-up feature extraction module and a top-down feature aggregation module. In the bottom-up process, features are extracted from the image through multiple stages of convolutional modules. Then, the feature aggregation module performs feature aggregation. For the output of each feature layer, this technique adds a context module to increase the receptive field. In this way, features rich in semantic information are extracted from the image, allowing faces at different scales to be processed equally.

[0060] After feature extraction, it is used in the multi-task parallel processing module. Convolution is used for computation in different downstream tasks. For the face detection task, three computational modules are used in the implementation process: a classification module, a localization module, and a keypoint module, which are used to estimate the probability of face presence. Location of the face area And the location of key facial features This technology also includes a three-dimensional line-of-sight estimation module for calculation. And three supplementary two-dimensional line-of-sight projection point estimation modules for computation For each training anchor box The goal of this technique is to minimize the loss. .in, This represents the overall loss function. This indicates the first adjustable hyperparameter. This represents the loss function related to face location information and key point information. This indicates the second adjustable hyperparameter. This represents the loss function related to the viewing direction. Wherein, .in, This represents the loss function related to face location information and key point information. This represents the loss function used for classification. This represents the training anchor box. Indicates the first The possibility of a personal face exists. Represents the first labeled data set. The possibility of a personal face exists. This represents the first hyperparameter used to balance the different loss terms. express. Indicates the first The location of the individual's face area. Represents the first labeled data set. The location of the individual's face area. This represents the second hyperparameter used to balance different loss terms. This represents the loss function used to calculate key points. The network prediction of the first The location of key facial features. This indicates the location of the i-th facial landmark labeled in the dataset.

[0061] Innovation point (3) Self-supervised loss function based on projection

[0062] This invention designs a novel projection-based self-supervised loss function, which can improve the prediction accuracy of the gaze direction. The loss function is as follows: , The definition can be:

[0063] .

[0064] in, This represents the loss function related to the direction of the line of sight. This represents the first hyperparameter used to balance the different loss terms. Represents the self-supervised loss function. This represents the second hyperparameter used to balance different loss terms. Indicates the direction of gaze in the face area. Indicates a truth value label. This represents the L1 norm.

[0065] here A projection-based self-supervised loss designed for 3D view directions. Specifically, such as... Figure 6 As shown, this technique proposes projecting the 3D viewing direction onto three planes—front, side, and top—to form three 2D viewing projection points. Calculating the positions of these three 2D points is then used as a subtask to achieve self-supervision, enhancing the accuracy of the 3D viewing direction calculation. Mathematically, this technique introduces 2D viewing sensitivity (…). GS The concept of )

[0066] .

[0067] in, This indicates two-dimensional line-of-sight sensitivity. Indicates the direction of the three-dimensional view. This represents the projection of the line of sight into the image coordinate system. express and The projection ratio of the two lines of sight. Indicates the angle of rotation. express and The ratio. This represents the radius of the unit circle.

[0068] Two-dimensional line-of-sight sensitivity defines the following The changes caused by The rate of change, the formula indicates The farther the position is from the origin, the lower the sensitivity. GS The larger the value, the better. At the implementation level, the objective function for gaze direction estimation not only predicts the 3D gaze direction but also simultaneously calculates the positions of the gaze projection points on the three projection planes. The calculation is performed using the following formula:

[0069] .

[0070] in, This represents the projection function. Indicates the positive direction. Nucleation angle, indicating the direction of the line of sight. The rotation angle indicates the direction of the line of sight. This indicates that the direction of the line of sight is projected onto the front plane. express The sine value. express The cosine value. Indicates the top direction. This indicates that the line of sight is projected onto the top plane. express The cosine value. Indicates the lateral direction. This indicates that the direction of the line of sight is projected onto the side plane. express The sine value.

[0071] By restricting the positions of the three two-dimensional line-of-sight projection points With 3D line-of-sight prediction value The projection positions are the same, and this can be achieved using the following loss function:

[0072] .

[0073] in, This represents the self-supervised loss function. It indicates the positive direction, the top direction, or the side direction. The value is Indicates the positive direction. Indicates the top direction. Indicates the lateral direction. It indicates the direction of sight, whether it is the top or side direction. This represents the projection function. Indicates the direction of gaze in the face area. This represents the projection function from three dimensions to two dimensions. This represents the L1 norm. Represents the natural constant. This represents the trainable parameters. express of Power of 1. Indicating targeting Correction factor for projection.

[0074] Innovation Point (4) Generation of Data for Multiple Persons' Line of Sight Replacement

[0075] In innovation point (2), a novel multi-person gaze replacement process framework is proposed. Existing datasets either contain only facial information (such as bounding boxes, key points, etc.) or only calibrated single faces with gaze direction labels. Therefore, the first problem faced by this technology is to construct a dataset that includes multiple target individuals and corresponding ground truth labels for gaze directions, thereby combining the advantages of both face datasets and gaze datasets. The framework of this process is as follows: Figure 4 As shown. In the implementation process, in order to synthesize a multi-person gaze replacement dataset, this invention selected the largest and most common existing gaze dataset ETH-XGaze, as well as the face detection dataset Widerface.

[0076] First, facial attributes are calculated for each suitable face in Widerface and ETH-XGaze:

[0077] .

[0078] in, This represents the set of facial attributes calculated from the input image. This represents a feature extractor. This represents the input image.

[0079] in It is a set of facial attributes calculated from a single face image, including facial landmark information, head pose information, age information, and gender information.

[0080] Next, for each Wideface dataset, appropriate faces are selected. The algorithm retrieves the closest face in ETH-XGaze. During the search, a scoring formula guides the results. The scoring formula is shown below:

[0081] .

[0082] in, This represents the image that needs to be queried. This represents the images in the dataset. This represents the scoring function. Representing an image Its characteristics. Representing an image Its characteristics. For key points or head posture. It is determined by the results of comparative experiments. It indicates the nature of a human face. This represents the facial properties corresponding to the image retrieved from the dataset. This indicates the facial features extracted from the retrieved image.

[0083] Then, during the replacement of the glasses region with truth labels for the gaze direction, a gaze replacement method is designed to generate gaze direction labels with truth values. Synthetic face images Through affine transformation, The truth value of the gaze direction can be preserved.

[0084] In constructing the data, this invention used 24,282 faces from the Widerface training set for gaze replacement, while other faces that were too small, too blurry, or had excessive facial occlusion were not replaced. Compared with other known gaze datasets, the newly proposed synthetic dataset has the advantage that each image contains one or more human objects, the entire training set contains approximately 20,000 human objects, and it also has the advantage of a face detection dataset, namely, its data collection environment is highly diverse.

[0085] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for estimating the gaze direction of multiple people in an image, comprising: Acquire a face image, wherein the face image includes at least one face region; Construct a deep network model for multi-task learning, wherein the deep network model includes multiple parallel computing multi-task processing structures, and the multiple parallel computing multi-task processing structures output the gaze direction of each face region in the face image and the face position information of each face region in a single calculation. The deep network model is trained end-to-end on the dataset to obtain a trained deep network model, wherein the trained deep network model is used to determine the gaze direction of multiple people. The face image is input into the trained deep network model to obtain the gaze direction of at least one face region included in the face image; For the gaze direction of each face region among the gaze directions of the at least one face region, the following determination steps are performed: Determine the gaze direction of the face region The positions of the line-of-sight projection points on the front, top, and side projection planes. ,in, This indicates the position of the projection point of the line of sight in the positive direction. This indicates the position of the projection point of the line of sight in the top direction. Indicates the position of the line-of-sight projection point in the lateral direction. Indicates the positive direction. Indicates the top direction. Indicates lateral direction. Indicates the direction of gaze in the facial region. Nucleation angle, indicating the direction of the line of sight. The rotation angle indicating the direction of the line of sight; Determine the positions of the line-of-sight projection points on the front, top, and side projection planes. Whether it is equal to the three projections of the three-dimensional view prediction value, wherein the three projections of the three-dimensional view prediction value are obtained by the following formula: ， in, Represents the projection function. This indicates that the direction of the line of sight is projected onto the front plane. express The sine value, express cosine value, This indicates that the line of sight is projected onto the top plane. express cosine value, This indicates that the direction of the line of sight is projected onto the side plane. express The sine value; The deep network model includes a self-supervised loss function, which is obtained by the following formula: ， in, Represents the self-supervised loss function. Indicates the positive direction, top direction, or side direction. The value is , Indicates the direction of sight, either directly overhead or laterally. The projection function from three dimensions to two dimensions. Describing the L1 norm, Represents the natural constant. Represents the trainable parameters. express of Power of 1 Indicating targeting Correction factor for projection.

2. The method according to claim 1, wherein, The face region in the at least one face region includes at least one of the following: face size information, pose information, expression information, and gender information.

3. The method according to claim 2, wherein, The deep network model is a multi-task single-stage deep network model, which includes an encoder and multiple decoders. The decoders among the multiple decoders are decoders that simultaneously perform different types of tasks. The deep network model is used to simultaneously determine the gaze direction of multiple face regions and output the face position information and key point information of multiple face regions.

4. The method according to claim 3, wherein, The dataset is generated from multiple images with replaced eye regions and a generative framework. Two types of data are input: one type is single-person image data with eye direction labels, and the other type is multi-person image data with multiple face regions. The generative framework automatically clusters the single-person image data based on at least one of the following information: gender, age, and head pose, to facilitate retrieval. The generative framework also retrieves the single-person image data that most closely matches each face region in the multi-person image data, replaces the eye regions, and generates the corresponding eye direction.

5. The method according to claim 4, wherein, The overall loss function during end-to-end training is obtained using the following formula: ， in, Represents the overall loss function. This indicates the first adjustable hyperparameter. This represents the loss function related to face location information and key point information. This indicates the second adjustable hyperparameter. Let represent the loss function related to the line-of-sight direction, where It is obtained through the following formula: ， in, This represents the loss function related to the direction of the line of sight. This represents the first hyperparameter used to balance the different loss terms. Represents the self-supervised loss function. This represents the second hyperparameter used to balance the different loss terms. Indicates the direction of gaze in the facial region. Indicates the truth label, This represents the L1 norm.

6. The method according to claim 5, wherein, In the deployment environment, the face image is input into the trained deep network model in real time to obtain the gaze direction of at least one face region included in the face image.

Images