Line-of-sight estimation method, apparatus, electronic device, and storage medium

Abstract

The application provides a line-of-sight estimation method and device, electronic equipment and storage medium, wherein the method comprises: determining a face image to be estimated and an eye image in the face image; performing feature extraction on the face image and the eye image, performing mixed attention interaction based on the face features and the eye features obtained through the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and face posture features; and performing line-of-sight estimation based on the eye fusion features and the face posture features to obtain a line-of-sight direction corresponding to the face image, overcoming the defect that the current line-of-sight estimation is easily disturbed by various factors, resulting in poor estimation result accuracy, deeply mining complementary information through mixed attention interaction, realizing information complementation through complementary feature fusion, and performing line-of-sight estimation on this basis, which not only maximizes the use of information, but also improves the fineness and accuracy of line-of-sight estimation, and an accurate and reliable line-of-sight direction can be obtained.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a gaze estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Eye tracking, as an important branch of computer vision, has shown great potential in various fields such as virtual reality, game control, medical device operation, and security systems. It achieves non-contact interactive control by capturing and analyzing eye movements, thereby improving the interactive experience and ease of operation.

[0003] Current gaze estimation schemes mainly fall into two categories: eye input and face input. Eye input relies on the visual features of the eyes, estimating gaze direction by capturing changes in eye appearance. However, this method is susceptible to limitations such as lighting and occlusion, affecting the stability and accuracy of the estimation. Face input schemes use facial images as input, extracting global facial features for gaze estimation, but they are also affected by factors such as facial expressions and head posture, resulting in less accurate estimations. Summary of the Invention

[0004] This invention provides a line-of-sight estimation method, apparatus, electronic device, and storage medium to address the shortcomings of existing line-of-sight estimation, such as susceptibility to interference and insufficient comprehensiveness and accuracy, thereby achieving more refined line-of-sight estimation and improving estimation accuracy.

[0005] This invention provides a line-of-sight estimation method, comprising:

[0006] Identify the face image to be estimated, and the eye image within the face image;

[0007] Feature extraction is performed on the face image and the eye image, and a hybrid attention interaction is performed on the face features and eye features obtained from the feature extraction, and a self-attention interaction is performed on the face features to obtain eye fusion features and face pose features.

[0008] Based on the eye fusion features and the facial pose features, gaze direction is estimated to obtain the gaze direction corresponding to the face image.

[0009] According to a gaze estimation method provided by the present invention, the steps of extracting features from the face image and the eye image, performing hybrid attention interaction based on the face features and eye features obtained from the feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial pose features include:

[0010] The face image and the eye image are cropped to obtain multiple face regions and multiple eye regions;

[0011] Based on the feature extraction layer in the gaze estimation model, features are extracted from each face region and each eye region to obtain the features of each face region and each eye region.

[0012] Based on the attention interaction layer in the gaze estimation model, hybrid attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial pose features.

[0013] The gaze estimation model is trained based on sample face images and their corresponding gaze directions and face poses, using a visual converter.

[0014] According to a gaze estimation method provided by the present invention, the attention interaction layer includes an eye branch and a face branch;

[0015] The eye branch is used to determine the hybrid attention parameters based on the features of each face region and the features of each eye region, and to determine the eye fusion features based on the hybrid attention parameters and the features of each eye region;

[0016] The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the face pose feature based on the self-attention parameter and the features of each face region.

[0017] According to the present invention, a gaze estimation method is provided, wherein the gaze estimation model is trained based on the following steps:

[0018] Identify a sample face image, and a sample eye image within the sample face image;

[0019] Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features;

[0020] Based on the initial attention interaction layer in the initial model, the sample face features and the sample eye features are subjected to mixed attention interaction, and the sample face features are subjected to self-attention interaction to obtain sample eye fusion features and sample face pose features.

[0021] Based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed using the sample eye fusion features and the sample face pose features to obtain the predicted gaze direction and predicted facial pose corresponding to the sample face image.

[0022] Based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, the parameters of the initial model are iterated to obtain the gaze estimation model.

[0023] According to a line-of-sight estimation method provided by the present invention, the predicted line-of-sight direction includes a first predicted line-of-sight direction and a second predicted line-of-sight direction;

[0024] The initial gaze estimation layer based on the initial model applies the sample eye fusion features and the sample face pose features to perform gaze estimation and face pose prediction, obtaining the predicted gaze direction and predicted face pose corresponding to the sample face image, including:

[0025] Based on the gaze estimation branch of the initial gaze estimation layer, gaze estimation is performed using the sample eye fusion features to obtain the first predicted gaze direction;

[0026] Based on the pose prediction branch of the initial gaze estimation layer, the facial pose features of the sample are applied to predict the facial pose, and the predicted facial pose is obtained.

[0027] Based on the global prediction branch of the initial gaze estimation layer, gaze estimation is performed using the sample facial pose features and the sample eye fusion features to obtain the second predicted gaze direction.

[0028] According to a gaze estimation method provided by the present invention, the method involves iterating the parameters of an initial model based on the predicted gaze direction and predicted facial pose corresponding to the sample face image, and the sample gaze direction and sample facial pose corresponding to the sample face image, to obtain the gaze estimation model, comprising:

[0029] Based on the first predicted gaze direction and the sample gaze direction, a first loss is determined;

[0030] Based on the second predicted gaze direction and the sample gaze direction, a second loss is determined;

[0031] Based on the predicted face pose and the sample face pose, a third loss is determined;

[0032] Based on the first loss, the second loss, and the third loss, the parameters of the initial model are iterated to obtain the gaze estimation model.

[0033] According to a gaze estimation method provided by the present invention, the sample face image and its corresponding sample face pose, as well as the sample eye image in the sample face image, are determined based on the following steps:

[0034] Obtain sample human body images;

[0035] Face detection is performed on the sample human body image, and based on the face detection results, the sample face image is determined from the sample human body image;

[0036] Facial landmark detection is performed on the sample face image, and based on the facial landmark detection results, the sample eye image is determined from the sample face image;

[0037] Facial pose detection is performed based on the sample face image to obtain the sample face pose corresponding to the sample face image.

[0038] The present invention also provides a line-of-sight estimation device, comprising:

[0039] An image determination unit is used to determine a face image to be estimated, and an eye image in the face image;

[0040] An attention interaction unit is used to extract features from the face image and the eye image, perform mixed attention interaction based on the face features and eye features obtained from the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial pose features.

[0041] The gaze estimation unit is used to estimate the gaze based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the gaze estimation method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the line-of-sight estimation method as described above.

[0044] The present invention provides a gaze estimation method, apparatus, electronic device, and storage medium that extracts features from a face image and the eye image within the face image. Based on the extracted face and eye features, it performs hybrid attention interaction and self-attention interaction to obtain eye fusion features and facial pose features. Based on these features, gaze estimation is performed to obtain the gaze direction corresponding to the face image. This overcomes the shortcomings of traditional methods where gaze estimation is easily affected by various factors, leading to poor accuracy. By deeply mining complementary information through hybrid attention interaction and achieving information complementarity through complementary feature fusion, gaze estimation is performed on this basis. This not only maximizes information utilization but also improves the precision and accuracy of gaze estimation, resulting in an accurate and reliable gaze direction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the line-of-sight estimation method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the initial model provided by the present invention;

[0048] Figure 3 This is an example diagram of facial key points provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the line-of-sight estimation device provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Eye-tracking technology, as an important branch of computer vision, has demonstrated enormous potential and value in many fields. For example, in virtual reality, games can be controlled using the eyes instead of the hands, thus reducing fatigue; in the medical field, some patients with limited mobility can control devices using their eyes instead of their hands; in the security field, eye-tracking technology can be used to develop eye-tracking unlocking devices. Unlike iris recognition unlocking, users can customize their eye movements to replace passwords for verification.

[0053] Current gaze estimation schemes are mainly divided into two categories: eye input and face input. Eye input schemes primarily rely on the visual features of the eyes, estimating gaze direction by capturing subtle changes in the appearance of the eyes. These methods typically use convolutional neural networks to extract features from grayscale images of the eyes for gaze estimation. However, relying solely on eye information can be limited by factors such as lighting and occlusion, affecting the stability and accuracy of the estimation.

[0054] In contrast, facial input methods estimate gaze by extracting global facial features. This approach demonstrates better robustness and performance in complex scenes, but it is also affected by factors such as facial expression changes and head posture, which can impact the accuracy of the estimation.

[0055] In response, this invention provides a gaze estimation method that combines facial and eye inputs and uses a hybrid attention mechanism for gaze estimation. This method fully utilizes the complementarity of eye and facial information to estimate gaze, achieving not only in-depth information mining and maximizing the use of complementary information, but also improving the precision and accuracy of gaze estimation, resulting in more accurate and reliable gaze estimation results.

[0056] Figure 1 This is a flowchart illustrating the line-of-sight estimation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0057] Step 110: Determine the face image to be estimated, and the eye image within the face image;

[0058] Step 120: Extract features from the face image and eye image, and perform hybrid attention interaction based on the face features and eye features obtained from the feature extraction, as well as self-attention interaction based on the face features, to obtain eye fusion features and face pose features.

[0059] Step 130: Based on eye fusion features and facial pose features, gaze estimation is performed to obtain the gaze direction corresponding to the face image.

[0060] Specifically, considering the problems of low estimation accuracy and poor precision in current gaze estimation schemes that rely on both eye and face input, this invention proposes to combine face and eye input for gaze estimation. In this process, Mixed Attention (MA) is used to effectively extract and integrate facial and eye features, achieving more refined predictions. This not only maximizes the use of facial and eye information for estimation but also improves the accuracy and stability of gaze estimation.

[0061] Understandably, in practical applications, before performing gaze estimation, it is first necessary to determine the image to be estimated, namely the "face input" and the "eye input," referred to here as the face image to be estimated, and the eye image of the face image. The eye image here refers to the image of the eye region in the face image, which is cropped from the face image. Specifically, this can be achieved by performing facial landmark detection on the face image and cropping the eye image from the face image based on the detection results; alternatively, the eye region can be manually defined in the face image and cropped to obtain the eye image; or the eye region can be located through eye features, cropped, or obtained through other methods. This embodiment of the invention does not specifically limit these methods.

[0062] The face image can be a real-life image containing a face, an image downloaded from the internet, an image containing a face region found in a database of pre-stored images, or an image containing a face obtained through other means. This embodiment of the invention does not specifically limit the types of images.

[0063] For the initially obtained image containing a face, in this embodiment of the invention, it is usually necessary to perform further processing, such as face detection to obtain the face region, and face key point detection to further obtain the eye region, thereby obtaining the face image and the corresponding eye image.

[0064] Here, the face image and the eye image correspond to each other. Corresponding to the human body structure, the eye image can include the left eye image and the right eye image. Since the left and right eyes are not absolutely symmetrical and completely identical, in order to ensure the accuracy of the estimation, the left eye image and the right eye image are used simultaneously for gaze estimation to avoid the direction deviation caused by monocular estimation.

[0065] After determining the face image and eye image, in this embodiment of the invention, feature extraction can be performed on these images to deeply mine their respective information. Feature fusion is then performed using mixed attention to obtain complementary information from the face and eyes. Based on this, estimation is performed, maximizing information utilization and improving the precision and accuracy of the estimation. Specifically, feature extraction is first performed on the face image and eye image to extract rich facial and eye information, thus obtaining facial features and eye features. Here, the eye image contains rich gaze detail information, while the face image contains global information, including not only gaze information but also head posture information, facial attribute information, etc. The face image can compensate for the lack of global information in the eye image, thereby improving the accuracy of subsequent estimation.

[0066] After this, attentional interaction can be performed on facial features and eye features to obtain eye fusion features and facial pose features. Specifically, mixed attentional interaction and self-attentional interaction are performed on facial features and eye features respectively to perform feature fusion through mixed attentional interaction and self-attentional interaction, thereby learning information about the face and eyes, and exploring the correlation and complementarity between the two to obtain the fusion features of the eyes and face, i.e., eye fusion features, and learning the unique head pose information of the face to obtain head pose features, which can also be called facial pose features.

[0067] Here, in the hybrid attention interaction process, eye features can provide fine features centered on gaze, while facial features provide global features containing rich contextual information. Based on this, hybrid attention interaction can deeply explore the complementary information between facial pose and eye gaze, thereby enabling more accurate prediction in the subsequent gaze estimation process.

[0068] Furthermore, after obtaining independent, uncoupled, but complementary eye fusion features and facial pose features through attention interaction, in this embodiment of the invention, feature fusion can be performed on the complementary features, and the direction of gaze can be predicted in a refined and accurate manner based on the fused features.

[0069] Specifically, this method estimates gaze direction based on eye fusion features and facial pose features obtained through attention interaction, thus determining the gaze direction corresponding to the face image. This process involves fusing the features obtained through attention interaction, specifically the eye fusion features and facial pose features, to obtain a fused feature set. This feature fusion achieves information complementarity, resulting in a more complete and comprehensive reflection of gaze, head pose, and other information. Feature fusion can be achieved through addition, concatenation, weighted fusion, etc., depending on the specific requirements. Next, gaze direction is estimated based on the fused features, yielding an accurate and reliable gaze estimation result—the gaze direction corresponding to the face image.

[0070] Compared to solutions that rely solely on eye and face input, this embodiment of the invention employs both face and eye input for gaze estimation, achieving superior results. Furthermore, by using a hybrid attention mechanism and a self-attention mechanism for initial feature fusion, learning can proceed from two perspectives: separately learning head pose information, and separately learning face and eye information. This allows for the mining of complementary information between the two, followed by complementary feature fusion and gaze estimation, significantly improving the precision and accuracy of gaze estimation.

[0071] The gaze estimation method provided by this invention extracts features from a face image and the eye image within the face image. It then performs hybrid attention interaction and self-attention interaction based on the extracted face and eye features to obtain eye fusion features and facial pose features. Based on these features, gaze estimation is performed to obtain the gaze direction corresponding to the face image. This method overcomes the shortcomings of traditional methods where gaze estimation is easily affected by various factors, leading to poor accuracy. By deeply mining complementary information through hybrid attention interaction and achieving information complementarity through complementary feature fusion, gaze estimation is performed on this basis. This not only maximizes information utilization but also improves the precision and accuracy of gaze estimation, resulting in an accurate and reliable gaze direction.

[0072] Based on the above embodiments, step 120 includes:

[0073] Image cropping is performed on face and eye images to obtain multiple face regions and multiple eye regions;

[0074] Based on the feature extraction layer in the gaze estimation model, features are extracted from each face region and each eye region to obtain the features of each face region and each eye region.

[0075] Based on the attention interaction layer in the gaze estimation model, hybrid attention interaction is performed on the features of each face region and each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial pose features.

[0076] The gaze estimation model is trained based on the visual converter, using sample face images and their corresponding gaze directions and face poses.

[0077] Specifically, the process of extracting features from face and eye images, performing hybrid attention interaction based on the extracted face and eye features, and performing self-attention interaction based on face features can be implemented through a gaze estimation model. The specific process includes:

[0078] First, the face image and the eye image can be cropped separately to divide them into multiple blocks, thus obtaining multiple corresponding image blocks. Each image block corresponds to a region in the original image. Therefore, the multiple image blocks obtained can also be called multiple image regions. That is, by cropping the image, we can obtain multiple face regions corresponding to the face image and multiple eye regions corresponding to the eye image.

[0079] The number of face regions and eye regions obtained can be either 16 or 64, and the specific number can be determined according to the actual situation. This embodiment of the invention does not impose a specific limitation on this. However, the number of face regions and eye regions is the same. For example, there are 8*8 face regions, left eye regions, and right eye regions.

[0080] Here, the process of cropping face and eye images can be achieved through a gaze estimation model, or it can be achieved through other methods or algorithms before processing with the gaze estimation model. Then, the cropped face and eye regions are input into the gaze estimation model for feature extraction and attention interaction.

[0081] Subsequently, a gaze estimation model can be used to extract features from multiple face regions and multiple eye regions to obtain the region features corresponding to each face region and the region features corresponding to each eye region, i.e., face region features and eye region features. In this embodiment of the invention, the gaze estimation model includes multiple layers; here, a feature extraction layer is used to extract features from each face region and each eye region to obtain the features of each face region and each eye region.

[0082] Here, in the gaze estimation model, the feature extraction layer first performs linear projection, that is, inputting each face region and each eye region into the linear projection layer of the feature extraction layer to perform feature mapping on the input, thereby obtaining the features corresponding to each face region and each eye region. Then, position encoding and class tokens can be added, that is, borrowing the class token idea from the Vision Transformer (ViT), adding their respective position encoding and class tokens to the features of each face region and each eye region obtained by feature mapping, thereby obtaining the face region features corresponding to each face region and the eye region features corresponding to each eye region.

[0083] Then, attentional interaction can continue through the gaze estimation model. Specifically, through the model's attentional interaction layer, hybrid attentional interaction is performed on the features of each facial region and each eye region, as well as self-attentional interaction on the features of each facial region, to obtain eye fusion features and facial pose features. In this process, the attentional interaction from both angles is constrained, so that self-attentional interaction only focuses on head pose information, while hybrid attentional interaction focuses on both eye gaze information and head pose information, in order to explore the correlation and complementarity between the two as much as possible, thereby obtaining facial pose features and eye fusion features.

[0084] However, it is worth noting that the gaze estimation model needs to be pre-trained before applying it for feature extraction and attention interaction. Specifically, this can be achieved by pre-training the gaze estimation model using sample face images and their corresponding labels. However, considering the limitations of single-eye and face input, this embodiment of the invention introduces an auxiliary task on top of the gaze estimation task to improve model performance when dealing with a combination of eye and face input. This auxiliary task can be head pose prediction, facial feature point detection, etc., which helps the model perform gaze estimation, resulting in better performance on this task.

[0085] As a preferred option, considering the consistency between gaze direction and head posture under normal circumstances, in this embodiment of the invention, head posture prediction is selected as an auxiliary task, and the model is trained through a multi-task approach, so that the model is more accurate in gaze estimation.

[0086] Specifically, this can begin by collecting a large number of sample face images and obtaining their corresponding labels for the gaze estimation task, i.e., the sample gaze direction. Simultaneously, it's also necessary to determine their labels for the auxiliary task, i.e., the sample head pose / sample face pose. Then, based on these sample face images, along with their corresponding sample gaze directions and sample face poses, the initial model can be trained to obtain the model, which is the gaze estimation model.

[0087] Here, the initial model can be built on the basis of the visual transformer ViT. Specifically, it borrows the architecture of the visual transformer ViT, improves the transformer, and adopts a hybrid attention mechanism for attention interaction to better capture and fuse features, thereby obtaining more representative, complete and comprehensive features.

[0088] Based on the above embodiments, the attention interaction layer includes an eye branch and a face branch;

[0089] The eye branch is used to determine the mixed attention parameters based on the features of each face region and each eye region, and to determine the eye fusion features based on the mixed attention parameters and the features of each eye region.

[0090] The face branch is used to determine the self-attention parameters based on the features of each face region, and to determine the face pose features based on the self-attention parameters and the features of each face region.

[0091] Specifically, in this embodiment of the invention, the attention interaction layer has two branches: an eye branch and a face branch. The eye branch is used to perform mixed attention interaction on the features of each face region and the features of each eye region to obtain eye fusion features; the face branch is used to perform self-attention interaction on the features of each face region to obtain facial pose features.

[0092] It's worth noting that during the attention interaction process, the parameters used in the face branch's attention interaction—namely, the self-attention parameters Q, K, and V (query, key, and value)—are derived solely from the face region features (face token). Therefore, the face branch's self-attention aims to focus solely on head pose information. Conversely, the eye branch's hybrid attention parameters come from both eye region features and face region features. Specifically, Q (query) comes from the eyes token, while K and V (key and value) come from both the eyes token and the face token. Therefore, the eye branch's hybrid attention interaction aims to pay attention to both eye gaze information and complementary information from the face.

[0093] Based on the above embodiments, the gaze estimation model is trained using the following steps:

[0094] Identify the sample face image and the sample eye image within the sample face image;

[0095] Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and sample eye image to obtain sample face features and sample eye features;

[0096] Based on the initial attention interaction layer in the initial model, the sample face features and sample eye features are subjected to mixed attention interaction, and the sample face features are subjected to self-attention interaction to obtain the sample eye fusion features and sample face pose features.

[0097] Based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed by applying sample eye fusion features and sample face pose features to obtain the predicted gaze direction and predicted facial pose corresponding to the sample face image.

[0098] Based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, the parameters of the initial model are iterated to obtain the gaze estimation model.

[0099] Specifically, the training process of the gaze estimation model may include the following steps:

[0100] First, it is necessary to identify the sample face images and the eye regions within them, i.e., the sample eye images. These sample face images can be pre-collected, specifically obtained from open-source datasets. While acquiring the sample face images, the corresponding gaze estimation task label, i.e., the sample gaze direction, can be simultaneously determined. This gaze direction can be manually labeled or selectively chosen during the acquisition of the sample face images, selecting those with gaze direction labels. This allows for the simultaneous acquisition of both the sample face image and its corresponding gaze direction.

[0101] At the same time, an initial model for training also needs to be built. Figure 2 This is a schematic diagram of the structure of the initial model provided by the present invention, as shown below. Figure 2 As shown, the initial model is built on the Vision Transformer (ViT) architecture, specifically by replacing the transformer in ViT with a Mixed Attention Unit (MAU). Furthermore, while using facial and eye inputs can produce better results than a single input, the inconsistency between head pose and gaze direction inevitably arises during model training, affecting not only training speed but also significantly reducing training effectiveness.

[0102] Based on this, in this embodiment of the invention, when constructing the initial model, the model structure is further improved, specifically the network head is improved, and multiple network heads are designed to correspond to different prediction tasks and obtain their respective prediction results. This can achieve decoupling between different tasks, not only maximizing the use of facial and eye information, but also decoupling the learning of gaze direction and head posture, thereby implicitly modeling spatial geometric relationships.

[0103] In detail, when the model is trained based on a multi-task approach, namely head pose prediction and gaze estimation, the head pose in the sample face image is consistent with the gaze direction, and the head pose closely reflects the gaze direction. In this case, the head pose can provide great help in gaze estimation. Conversely, in some cases, the head pose and gaze direction are not consistent. That is, although the head pose changes significantly, the gaze direction remains unchanged, or the gaze direction changes significantly but the head pose remains basically unchanged. In this case, the head pose is prone to misleading gaze estimation. To address this, in this embodiment of the invention, multiple network heads are selected, and each network head adopts a two-layer multilayer perceptron to correspond to different prediction tasks, so as to avoid mutual interference during the prediction process, which would lead to inaccurate prediction results and poor model training performance.

[0104] After determining the initial model and sample face images, in this embodiment of the invention, model training can be performed. That is, feature extraction is first performed using the feature extraction layer in the initial model. Since the feature extraction layer is not yet trained, it is called the initial feature extraction layer. Based on this, feature extraction is performed on the sample face images and sample eye images. Specifically, image cropping can be performed first, and then feature mapping can be performed on each cropped image block, and position encoding and category tokens can be added to obtain the corresponding sample face features and sample eye features.

[0105] Next, attention interactions can be performed on the extracted sample face features and sample eye features. Specifically, an initial attention interaction layer is applied to perform mixed attention interactions on the sample face features and sample eye features, as well as self-attention interactions on the sample face features. Here, specifically, the eye branch in the initial attention interaction layer performs mixed attention interactions on the sample face features and sample eye features, and the face branch performs self-attention interactions on the sample face features. This ensures that the face branch learns only information related to head pose, while the eye branch learns not only information related to eye gaze but also complementary information between head pose and gaze. Ultimately, the sample eye fusion features (sample left eye fusion features and sample right eye fusion features) and sample face pose features are output by the two branches respectively.

[0106] Then, gaze estimation and facial pose prediction can be performed using the sample eye fusion features and sample facial pose features. Specifically, based on the sample eye fusion features and sample facial pose features, gaze estimation and facial pose prediction are performed separately through the initial gaze estimation layer in the initial model, thereby obtaining the predicted gaze direction and predicted facial pose corresponding to the sample face image. Here, different network heads can be used to perform gaze estimation and facial pose prediction to predict the gaze direction and facial pose respectively, and the prediction results are output, thus obtaining the predicted gaze direction and predicted facial pose.

[0107] Here, in the hybrid attention mechanism, the face branch, obtained through self-attention, only learns information related to head pose. The eye branch, through mix-attention, learns not only information related to eye gaze but also complementary information between head pose and gaze, improving gaze estimation accuracy. Building on this, an explicit dual-supervision constraint mechanism for head pose and gaze estimation is added. This decouples the features of the face and eye branches, making them more complementary. Essentially, the gaze estimation and head pose prediction tasks are decoupled to obtain independent features, which are then fused in a complementary manner to achieve more refined predictions.

[0108] Then, the model can be trained based on the predicted gaze direction and predicted face pose output by the model, as well as the labels of the sample face images. That is, based on the predicted gaze direction and predicted face pose corresponding to the sample face images, and the sample gaze direction and sample face pose, the parameters of the initial model are iterated. That is, the parameters of the initial model are adjusted according to the output prediction results and the labels of the samples, so that the prediction results output by the model after parameter adjustment are consistent with the labels, or as close as possible to the labels. Finally, a trained initial model can be obtained, which is the gaze estimation model. Here, it should be noted that the labels of the head pose prediction task corresponding to the sample face images, i.e., the sample face pose, can be obtained by performing face pose detection on the sample face images, or by manual annotation, or by other methods. This embodiment of the invention does not specifically limit this.

[0109] Based on the above embodiments, the predicted line of sight includes a first predicted line of sight and a second predicted line of sight.

[0110] Based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed using sample eye fusion features and sample facial pose features. This yields the predicted gaze direction and predicted facial pose corresponding to the sample face image, including:

[0111] Based on the gaze estimation branch of the initial gaze estimation layer, gaze estimation is performed using sample eye fusion features to obtain the first predicted gaze direction;

[0112] Based on the pose prediction branch of the initial gaze estimation layer, the facial pose features of the sample are applied to predict the facial pose, and the predicted facial pose is obtained.

[0113] Based on the global prediction branch of the initial gaze estimation layer, gaze estimation is performed using sample facial pose features and sample eye fusion features to obtain the second predicted gaze direction.

[0114] Specifically, the process of applying an initial gaze estimation layer to estimate gaze and predict facial pose based on the sample eye fusion features and sample facial pose features can be implemented as follows: In this embodiment of the invention, the gaze estimation layer includes three branches: a gaze estimation branch, a pose prediction branch, and a global prediction branch. Here, gaze estimation can be performed through the gaze estimation branch based on the sample eye fusion features to obtain a first predicted gaze direction; facial pose prediction can be performed through the pose prediction branch based on the sample facial pose features to obtain a predicted facial pose; and the global prediction branch can be applied to estimate gaze based on the sample facial pose features and sample eye fusion features to obtain a second predicted gaze direction.

[0115] In detail, after MAU, both the face image and the eye image have a corresponding feature Class Tokens, namely the sample eye fusion feature (sample left eye fusion feature and sample right eye fusion feature) and the sample face pose feature, denoted as follows: , and These represent information about the entire image. In this embodiment of the invention, a dual-supervision constraint is designed using three network heads, each employing a two-layer multilayer perceptron structure. The dual-supervision constraint includes supervision of the eye's gaze direction and head pose constraint of the face.

[0116] Among them, the line-of-sight estimation head / line-of-sight estimation branch is spliced. and The obtained features are used as input to perform line-of-sight estimation, resulting in the first predicted line-of-sight direction (including pitch and yaw angles), denoted as . The attitude prediction head / attitude prediction branch only uses... As input, face pose prediction is performed to obtain the predicted face pose (including pitch and yaw angles), denoted as . The global monitoring effect is applied to the third prediction head, which aggregates... , and That is, the global prediction branch is spliced ​​together. , and The obtained features are used as input to predict the final gaze direction, and the output is a second predicted gaze direction (including pitch and yaw angles), denoted as . .

[0117] It should be noted that during model training, three branches are used for prediction. However, after training, in practical applications, only the global prediction branch is used for gaze estimation, and the output of this branch is the final result of the application process.

[0118] Based on the above embodiments, the initial model is iterated by parameters based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, to obtain a gaze estimation model, including:

[0119] The first loss is determined based on the first predicted line of sight and the sample line of sight.

[0120] The second loss is determined based on the second predicted line of sight and the sample line of sight.

[0121] A third loss is determined based on the predicted face pose and the sample face pose;

[0122] Based on the first loss, the second loss, and the third loss, the parameters of the initial model are iterated to obtain the line-of-sight estimation model.

[0123] Specifically, the process of iterating the parameters of the initial model to obtain the gaze estimation model based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, may include:

[0124] First, loss measures can be performed based on the first predicted gaze direction and the sample gaze direction, as well as based on the second predicted gaze direction and the sample gaze direction, to determine the model's loss on the gaze estimation task, thus obtaining the first loss and the second loss. Specifically, the distance / closeness between the first predicted gaze direction and the sample gaze direction, and the distance / closeness between the second predicted gaze direction and the sample gaze direction, are calculated to determine the first loss and the second loss.

[0125] Simultaneously, the loss of the model on the head text prediction task can be measured based on the predicted face pose and the sample face pose, resulting in a third loss. That is, the distance / closeness between the predicted face pose and the sample face pose is calculated, and the third loss is determined accordingly.

[0126] Then, the first loss, the second loss, and the third loss can be combined to determine the overall loss of the model. Specifically, the three losses can be weighted and summed to obtain the overall loss. Based on this overall loss, the parameters of the initial model can be iterated to obtain the line-of-sight estimation model.

[0127] In this embodiment of the invention, in addition to adding a hybrid attention mechanism at the network design level to improve feature extraction and fusion, the loss function is also explicitly constrained, forcing the face branch to learn only information related to head pose, while the eye branch can learn complementary information of head pose and gaze. Finally, the decoupled features are spliced ​​together to achieve more refined prediction.

[0128] Based on the above embodiments, the overall loss of the initial model can be expressed by the following formula:

[0129]

[0130] In the formula, and The weights for the first and third losses are 1, respectively. and These represent the direction of the sample's gaze and the text on the sample's face, respectively.

[0131] Based on the above embodiments, the sample face image and its corresponding sample face pose, as well as the sample eye image in the sample face image, are determined based on the following steps:

[0132] Obtain sample human body images;

[0133] Face detection is performed on the sample human body images, and the sample face images are determined from the sample human body images based on the face detection results;

[0134] Facial landmark detection is performed on the sample face images, and based on the facial landmark detection results, the sample eye images are determined from the sample face images;

[0135] Facial pose detection is performed based on sample face images to obtain the sample face pose corresponding to the sample face image.

[0136] Specifically, the sample face image, its corresponding sample face pose, and the sample eye image within the sample face image can be determined through the following steps:

[0137] First, sample human images can be obtained, which can be obtained from major public websites and open source datasets.

[0138] Next, face detection can be performed on the sample human image to detect face regions within the image, thus obtaining face detection results. These results can include face bounding boxes and confidence scores. Then, based on the face detection results, face regions can be determined from the sample human image, serving as the sample face image. Face detection in the sample human image can be achieved using current mature face detection technologies and models. The detected sample face image can be scaled down to a uniform size of 128*128.

[0139] Subsequently, facial landmark detection can be performed on the sample face image to detect facial landmarks, thereby locating the eye region in the face and obtaining the sample eye image. Figure 3 This is an example diagram of facial key points provided by the present invention, such as... Figure 3 As shown, facial landmark detection can yield the results of facial landmark detection, namely 68 landmarks in the face, of which the left eye and the right eye each contain six landmarks. Based on this, the eye region can be determined from the sample face image as the sample eye image.

[0140] The eye region cropping process specifically includes: calculating the center point (mean) and major axis distance of key points 37~42 of the left eye and key points 43~48 of the right eye respectively; using 1.5 times the major axis distance as the side length and the center point as the center, cropping a square region and uniformly scaling it to 128*128 to obtain sample eye images (sample left eye image and sample right eye image).

[0141] Next, facial pose detection can be performed based on the sample face image to obtain the corresponding sample face pose. Here, 3DMM (3D Morphable Models) is used for facial pose detection to obtain the sample face pose corresponding to the sample face image.

[0142] Based on the above embodiments, the overall process of the line-of-sight estimation method includes the following steps:

[0143] First, identify the face image to be estimated, and the eye image within the face image.

[0144] Subsequently, feature extraction was performed on the face image and eye image, and a hybrid attention interaction was performed based on the face features and eye features obtained from the feature extraction, as well as a self-attention interaction based on the face features, to obtain eye fusion features and face pose features.

[0145] Here, feature extraction is performed on face and eye images. Based on the extracted face and eye features, a hybrid attention interaction and a self-attention interaction are performed to obtain eye fusion features and facial pose features. Specifically, this includes: cropping face and eye images to obtain multiple face regions and multiple eye regions; extracting features from each face and eye region based on the feature extraction layer of the gaze estimation model to obtain features for each face region and each eye region; and performing a hybrid attention interaction and a self-attention interaction on the features for each face and eye region based on the attention interaction layer of the gaze estimation model to obtain eye fusion features and facial pose features. The gaze estimation model is trained on sample face images and their corresponding sample gaze directions and sample facial poses based on a visual converter.

[0146] The attention interaction layer includes an eye branch and a face branch. The eye branch is used to determine the hybrid attention parameters based on the features of each face region and each eye region, and to determine the eye fusion features based on the hybrid attention parameters and the features of each eye region. The face branch is used to determine the self-attention parameters based on the features of each face region, and to determine the facial pose features based on the self-attention parameters and the features of each face region.

[0147] Subsequently, gaze direction is estimated based on eye fusion features and facial pose features to obtain the gaze direction corresponding to the face image.

[0148] The gaze estimation model is trained based on the following steps: First, a sample face image and sample eye images within the sample face image are determined. Second, features are extracted from the sample face image and sample eye image using the initial feature extraction layer in the initial model to obtain sample face features and sample eye features. Third, a hybrid attention interaction and self-attention interaction are performed on the sample face features and sample eye features using the initial attention interaction layer in the initial model to obtain sample eye fusion features and sample face pose features. Fourth, gaze estimation and face pose prediction are performed using the sample eye fusion features and sample face pose features using the initial gaze estimation layer in the initial model to obtain the predicted gaze direction and predicted face pose corresponding to the sample face image. Fifth, the parameters of the initial model are iterated based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, to obtain the gaze estimation model.

[0149] Here, the predicted line of sight includes a first predicted line of sight and a second predicted line of sight.

[0150] Specifically, based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed using sample eye fusion features and sample facial pose features to obtain the predicted gaze direction and predicted facial pose corresponding to the sample face image. This includes: based on the gaze estimation branch of the initial gaze estimation layer, gaze estimation is performed using sample eye fusion features to obtain the first predicted gaze direction; based on the pose prediction branch of the initial gaze estimation layer, facial pose prediction is performed using sample facial pose features to obtain the predicted facial pose; and based on the global prediction branch of the initial gaze estimation layer, gaze estimation is performed using sample facial pose features and sample eye fusion features to obtain the second predicted gaze direction.

[0151] Furthermore, based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, the initial model is iterated to obtain a gaze estimation model, including: determining a first loss based on the first predicted gaze direction and the sample gaze direction; determining a second loss based on the second predicted gaze direction and the sample gaze direction; determining a third loss based on the predicted face pose and the sample face pose; and iterating the parameters of the initial model based on the first loss, the second loss, and the third loss to obtain the gaze estimation model.

[0152] The sample face image and its corresponding sample face pose, as well as the sample eye image in the sample face image, are determined based on the following steps: acquiring a sample human body image; performing face detection on the sample human body image, and determining the sample face image from the sample human body image based on the face detection results; performing facial landmark detection on the sample face image, and determining the sample eye image from the sample face image based on the facial landmark detection results; and performing face pose detection on the sample face image to obtain the sample face pose corresponding to the sample face image.

[0153] The method provided in this invention extracts features from a face image and the eye image within the face image. It then performs hybrid attention interaction based on the extracted face and eye features, and self-attention interaction based on the face features to obtain eye fusion features and facial pose features. Based on these features, it estimates the gaze direction corresponding to the face image. This method overcomes the shortcomings of traditional methods where gaze estimation is easily affected by various factors, leading to poor accuracy. By deeply mining complementary information through hybrid attention interaction and achieving information complementarity through complementary feature fusion, gaze estimation is performed on this basis. This not only maximizes information utilization but also improves the precision and accuracy of gaze estimation, resulting in an accurate and reliable gaze direction.

[0154] The line-of-sight estimation device provided by the present invention is described below. The line-of-sight estimation device described below can be referred to in correspondence with the line-of-sight estimation method described above.

[0155] Figure 4 This is a schematic diagram of the line-of-sight estimation device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0156] Image determination unit 410 is used to determine the face image to be estimated, and the eye image in the face image;

[0157] The attention interaction unit 420 is used to extract features from the face image and the eye image, perform mixed attention interaction based on the face features and eye features obtained from the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and face pose features.

[0158] The gaze estimation unit 430 is used to estimate the gaze based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image.

[0159] The gaze estimation device provided by this invention extracts features from a face image and the eye image within the face image. It then performs hybrid attention interaction and self-attention interaction based on the extracted face and eye features to obtain eye fusion features and facial pose features. Based on these features, gaze estimation is performed to obtain the gaze direction corresponding to the face image. This overcomes the shortcomings of traditional methods where gaze estimation is easily affected by various factors, leading to poor accuracy. By deeply mining complementary information through hybrid attention interaction and achieving information complementarity through complementary feature fusion, gaze estimation is performed on this basis. This not only maximizes information utilization but also improves the precision and accuracy of gaze estimation, resulting in an accurate and reliable gaze direction.

[0160] Based on the above embodiments, the attention interaction unit 420 is used for:

[0161] The face image and the eye image are cropped to obtain multiple face regions and multiple eye regions;

[0162] Based on the feature extraction layer in the gaze estimation model, features are extracted from each face region and each eye region to obtain the features of each face region and each eye region.

[0163] Based on the attention interaction layer in the gaze estimation model, hybrid attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial pose features.

[0164] The gaze estimation model is trained based on sample face images and their corresponding gaze directions and face poses, using a visual converter.

[0165] Based on the above embodiments, the attention interaction layer includes an eye branch and a face branch;

[0166] The eye branch is used to determine the hybrid attention parameters based on the features of each face region and the features of each eye region, and to determine the eye fusion features based on the hybrid attention parameters and the features of each eye region;

[0167] The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the face pose feature based on the self-attention parameter and the features of each face region.

[0168] Based on the above embodiments, the device further includes a model training unit, used for:

[0169] Identify a sample face image, and a sample eye image within the sample face image;

[0170] Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features;

[0171] Based on the initial attention interaction layer in the initial model, the sample face features and the sample eye features are subjected to mixed attention interaction, and the sample face features are subjected to self-attention interaction to obtain sample eye fusion features and sample face pose features.

[0172] Based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed using the sample eye fusion features and the sample face pose features to obtain the predicted gaze direction and predicted facial pose corresponding to the sample face image.

[0173] Based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, the parameters of the initial model are iterated to obtain the gaze estimation model.

[0174] Based on the above embodiments, the predicted line of sight includes a first predicted line of sight and a second predicted line of sight;

[0175] The model training unit is used for:

[0176] Based on the gaze estimation branch of the initial gaze estimation layer, gaze estimation is performed using the sample eye fusion features to obtain the first predicted gaze direction;

[0177] Based on the pose prediction branch of the initial gaze estimation layer, the facial pose features of the sample are applied to predict the facial pose, and the predicted facial pose is obtained.

[0178] Based on the global prediction branch of the initial gaze estimation layer, gaze estimation is performed using the sample facial pose features and the sample eye fusion features to obtain the second predicted gaze direction.

[0179] Based on the above embodiments, the model training unit is used for:

[0180] Based on the first predicted gaze direction and the sample gaze direction, a first loss is determined;

[0181] Based on the second predicted gaze direction and the sample gaze direction, a second loss is determined;

[0182] Based on the predicted face pose and the sample face pose, a third loss is determined;

[0183] Based on the first loss, the second loss, and the third loss, the parameters of the initial model are iterated to obtain the gaze estimation model.

[0184] Based on the above embodiments, the model training unit is also used for:

[0185] Obtain sample human body images;

[0186] Face detection is performed on the sample human body image, and based on the face detection results, the sample face image is determined from the sample human body image;

[0187] Facial landmark detection is performed on the sample face image, and based on the facial landmark detection results, the sample eye image is determined from the sample face image;

[0188] Facial pose detection is performed based on the sample face image to obtain the sample face pose corresponding to the sample face image.

[0189] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a gaze estimation method, which includes: determining a face image to be estimated, and eye images within the face image; extracting features from the face image and the eye images, performing hybrid attention interaction based on the extracted face features and eye features, and performing self-attention interaction based on the face features to obtain eye fusion features and facial pose features; and estimating gaze based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image.

[0190] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the gaze estimation method provided by the above methods, the method comprising: determining a face image to be estimated, and an eye image in the face image; performing feature extraction on the face image and the eye image, and performing hybrid attention interaction based on the face features and eye features obtained by feature extraction, and performing self-attention interaction based on the face features to obtain eye fusion features and facial pose features; and performing gaze estimation based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image.

[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the gaze estimation method provided by the methods described above. The method includes: determining a face image to be estimated, and an eye image in the face image; extracting features from the face image and the eye image, performing a hybrid attention interaction based on the face features and eye features obtained from the feature extraction, and performing a self-attention interaction based on the face features to obtain eye fusion features and facial pose features; and estimating gaze based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A line-of-sight estimation method, characterized in that, include: Identify the face image to be estimated, and the eye image within the face image; Feature extraction is performed on the face image and the eye image, and a hybrid attention interaction is performed on the face features and eye features obtained from the feature extraction, and a self-attention interaction is performed on the face features to obtain eye fusion features and face pose features. Based on the eye fusion features and the facial pose features, gaze estimation is performed to obtain the gaze direction corresponding to the face image; The process of extracting features from the face image and the eye image, performing hybrid attention interaction based on the extracted face and eye features, and performing self-attention interaction based on the face features to obtain eye fusion features and facial pose features includes: The face image and the eye image are cropped to obtain multiple face regions and multiple eye regions; Based on the feature extraction layer in the gaze estimation model, features are extracted from each face region and each eye region to obtain the features of each face region and each eye region. Based on the attention interaction layer in the gaze estimation model, hybrid attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial pose features. The gaze estimation model is trained based on sample face images and their corresponding gaze directions and face poses, using a visual converter.

2. The line-of-sight estimation method according to claim 1, characterized in that, The attention interaction layer includes an eye branch and a face branch; The eye branch is used to determine the hybrid attention parameters based on the features of each face region and the features of each eye region, and to determine the eye fusion features based on the hybrid attention parameters and the features of each eye region; The face branch is used to determine the self-attention parameter based on the features of each face region, and to determine the face pose feature based on the self-attention parameter and the features of each face region.

3. The line-of-sight estimation method according to claim 1, characterized in that, The gaze estimation model is trained based on the following steps: Identify a sample face image, and a sample eye image within the sample face image; Based on the initial feature extraction layer in the initial model, feature extraction is performed on the sample face image and the sample eye image to obtain sample face features and sample eye features; Based on the initial attention interaction layer in the initial model, the sample face features and the sample eye features are subjected to mixed attention interaction, and the sample face features are subjected to self-attention interaction to obtain sample eye fusion features and sample face pose features. Based on the initial gaze estimation layer in the initial model, gaze estimation and facial pose prediction are performed using the sample eye fusion features and the sample face pose features to obtain the predicted gaze direction and predicted facial pose corresponding to the sample face image. Based on the predicted gaze direction and predicted face pose corresponding to the sample face image, as well as the sample gaze direction and sample face pose corresponding to the sample face image, the parameters of the initial model are iterated to obtain the gaze estimation model.

4. The line-of-sight estimation method according to claim 3, characterized in that, The predicted line of sight includes a first predicted line of sight and a second predicted line of sight. The initial gaze estimation layer based on the initial model applies the sample eye fusion features and the sample face pose features to perform gaze estimation and face pose prediction, obtaining the predicted gaze direction and predicted face pose corresponding to the sample face image, including: Based on the gaze estimation branch of the initial gaze estimation layer, gaze estimation is performed using the sample eye fusion features to obtain the first predicted gaze direction; Based on the pose prediction branch of the initial gaze estimation layer, the facial pose features of the sample are applied to predict the facial pose, and the predicted facial pose is obtained. Based on the global prediction branch of the initial gaze estimation layer, gaze estimation is performed using the sample facial pose features and the sample eye fusion features to obtain the second predicted gaze direction.

5. The line-of-sight estimation method according to claim 4, characterized in that, The initial model is iterated based on the predicted gaze direction and predicted facial pose corresponding to the sample face image, as well as the sample gaze direction and sample facial pose corresponding to the sample face image, to obtain the gaze estimation model, including: Based on the first predicted gaze direction and the sample gaze direction, a first loss is determined; Based on the second predicted gaze direction and the sample gaze direction, a second loss is determined; Based on the predicted face pose and the sample face pose, a third loss is determined; Based on the first loss, the second loss, and the third loss, the parameters of the initial model are iterated to obtain the gaze estimation model.

6. The line-of-sight estimation method according to any one of claims 1 to 5, characterized in that, The sample face image and its corresponding sample face pose, as well as the sample eye image in the sample face image, are determined based on the following steps: Obtain sample human body images; Face detection is performed on the sample human body image, and based on the face detection results, the sample face image is determined from the sample human body image; Facial landmark detection is performed on the sample face image, and based on the facial landmark detection results, the sample eye image is determined from the sample face image; Facial pose detection is performed based on the sample face image to obtain the sample face pose corresponding to the sample face image.

7. A line-of-sight estimation device, characterized in that, include: An image determination unit is used to determine a face image to be estimated, and an eye image in the face image; An attention interaction unit is used to extract features from the face image and the eye image, perform mixed attention interaction based on the face features and eye features obtained from the feature extraction, and perform self-attention interaction based on the face features to obtain eye fusion features and facial pose features. A gaze estimation unit is used to estimate the gaze based on the eye fusion features and the facial pose features to obtain the gaze direction corresponding to the face image. The process of extracting features from the face image and the eye image, performing hybrid attention interaction based on the extracted face and eye features, and performing self-attention interaction based on the face features to obtain eye fusion features and facial pose features includes: The face image and the eye image are cropped to obtain multiple face regions and multiple eye regions; Based on the feature extraction layer in the gaze estimation model, features are extracted from each face region and each eye region to obtain the features of each face region and each eye region. Based on the attention interaction layer in the gaze estimation model, hybrid attention interaction is performed on the features of each face region and the features of each eye region, and self-attention interaction is performed on the features of each face region to obtain eye fusion features and facial pose features. The gaze estimation model is trained based on sample face images and their corresponding gaze directions and face poses, using a visual converter.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the line-of-sight estimation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the line-of-sight estimation method as described in any one of claims 1 to 6.

Images